/segment-anything

description: Foundation model for image segmentation with zero-shot transfer. Use
  when you need to segment any object in images using points, boxes, or masks as prompts,
  or automatically generate all object masks in an image.
version: 1.0.0
author: Orchestra Research
license: MIT
dependencies:
- segment-anything
- transformers>=4.30.0
- torch>=1.7.0
metadata:
  catalog:
    tags:
    - Multimodal
    - Image Segmentation
    - Computer Vision
    - SAM
    - Zero-Shot

Kheish Compatibility

This skill is repo-local and stays inactive until explicitly activated.

When the original instructions refer to legacy tool names, use these Kheish mappings:

• `terminal` => `bash`
• `web_extract` => `web_fetch`, plus `web_search` when discovery is needed
• `search_files` => `grep_search` and `glob_search`
• `browser_*` tools require a browser-capable surfaced tool or MCP; if none is available, use the closest available surface and say so explicitly

When the instructions mention local helper files, resolve them from `${KHEISH_SKILL_DIR}`.

Segment Anything Model (SAM)

Comprehensive guide to using Meta AI's Segment Anything Model for zero-shot image segmentation.

When to use SAM

**Use SAM when:**

• Need to segment any object in images without task-specific training
• Building interactive annotation tools with point/box prompts
• Generating training data for other vision models
• Need zero-shot transfer to new image domains
• Building object detection/segmentation pipelines
• Processing medical, satellite, or domain-specific images

**Key features:**

• **Zero-shot segmentation**: Works on any image domain without fine-tuning
• **Flexible prompts**: Points, bounding boxes, or previous masks
• **Automatic segmentation**: Generate all object masks automatically
• **High quality**: Trained on 1.1 billion masks from 11 million images
• **Multiple model sizes**: ViT-B (fastest), ViT-L, ViT-H (most accurate)
• **ONNX export**: Deploy in browsers and edge devices

**Use alternatives instead:**

• **YOLO/Detectron2**: For real-time object detection with classes
• **Mask2Former**: For semantic/panoptic segmentation with categories
• **GroundingDINO + SAM**: For text-prompted segmentation
• **SAM 2**: For video segmentation tasks

Quick start

Installation

# From GitHub
pip install git+https://github.com/facebookresearch/segment-anything.git

# Optional dependencies
pip install opencv-python pycocotools matplotlib

# Or use HuggingFace transformers
pip install transformers

Download checkpoints

# ViT-H (largest, most accurate) - 2.4GB
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth

# ViT-L (medium) - 1.2GB
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_l_0b3195.pth

# ViT-B (smallest, fastest) - 375MB
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth

Basic usage with SamPredictor

import numpy as np
from segment_anything import sam_model_registry, SamPredictor

# Load model
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda")

# Create predictor
predictor = SamPredictor(sam)

# Set image (computes embeddings once)
image = cv2.imread("image.jpg")
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
predictor.set_image(image)

# Predict with point prompts
input_point = np.array([[500, 375]])  # (x, y) coordinates
input_label = np.array([1])  # 1 = foreground, 0 = background

masks, scores, logits = predictor.predict(
    point_coords=input_point,
    point_labels=input_label,
    multimask_output=True  # Returns 3 mask options
)

# Select best mask
best_mask = masks[np.argmax(scores)]

HuggingFace Transformers

import torch
from PIL import Image
from transformers import SamModel, SamProcessor

# Load model and processor
model = SamModel.from_pretrained("facebook/sam-vit-huge")
processor = SamProcessor.from_pretrained("facebook/sam-vit-huge")
model.to("cuda")

# Process image with point prompt
image = Image.open("image.jpg")
input_points = [[[450, 600]]]  # Batch of points

inputs = processor(image, input_points=input_points, return_tensors="pt")
inputs = {k: v.to("cuda") for k, v in inputs.items()}

# Generate masks
with torch.no_grad():
    outputs = model(**inputs)

# Post-process masks to original size
masks = processor.image_processor.post_process_masks(
    outputs.pred_masks.cpu(),
    inputs["original_sizes"].cpu(),
    inputs["reshaped_input_sizes"].cpu()
)

Core concepts

Model architecture

SAM Architecture:
┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│  Image Encoder  │────▶│ Prompt Encoder  │────▶│  Mask Decoder   │
│     (ViT)       │     │ (Points/Boxes)  │     │ (Transformer)   │
└─────────────────┘     └─────────────────┘     └─────────────────┘
        │                       │                       │
   Image Embeddings      Prompt Embeddings         Masks + IoU
   (computed once)       (per prompt)             predictions

Model variants

| Model | Checkpoint | Size | Speed | Accuracy | |-------|------------|------|-------|----------| | ViT-H | `vit_h` | 2.4 GB | Slowest | Best | | ViT-L | `vit_l` | 1.2 GB | Medium | Good | | ViT-B | `vit_b` | 375 MB | Fastest | Good |

Prompt types

| Prompt | Description | Use Case | |--------|-------------|----------| | Point (foreground) | Click on object | Single object selection | | Point (background) | Click outside object | Exclude regions | | Bounding box | Rectangle around object | Larger objects | | Previous mask | Low-res mask input | Iterative refinement |

Interactive segmentation

Point prompts

# Single foreground point
input_point = np.array([[500, 375]])
input_label = np.array([1])

masks, scores, logits = predictor.predict(
    point_coords=input_point,
    point_labels=input_label,
    multimask_output=True
)

# Multiple points (foreground + background)
input_points = np.array([[500, 375],