/multimodal-segment-anything

name: multimodal-segment-anything
description: Foundation model for image segmentation with zero-shot transfer. Use when you need to segment any object in images using points, boxes, or masks as prompts, or automatically generate all object masks in an image.
license: MIT
metadata:
  version: 1.0.0
  author: Orchestra Research
  tags: [Multimodal, Image Segmentation, Computer Vision, SAM, Zero-Shot]
  dependencies: [segment-anything, transformers>=4.30.0, torch>=1.7.0]

Segment Anything Model (SAM)

Comprehensive guide to using Meta AI's Segment Anything Model for zero-shot image segmentation.

When to use SAM

**Use SAM when:**

• Need to segment any object in images without task-specific training
• Building interactive annotation tools with point/box prompts
• Generating training data for other vision models
• Need zero-shot transfer to new image domains
• Building object detection/segmentation pipelines
• Processing medical, satellite, or domain-specific images

**Key features:**

• **Zero-shot segmentation**: Works on any image domain without fine-tuning
• **Flexible prompts**: Points, bounding boxes, or previous masks
• **Automatic segmentation**: Generate all object masks automatically
• **High quality**: Trained on 1.1 billion masks from 11 million images
• **Multiple model sizes**: ViT-B (fastest), ViT-L, ViT-H (most accurate)
• **ONNX export**: Deploy in browsers and edge devices

**Use alternatives instead:**

• **YOLO/Detectron2**: For real-time object detection with classes
• **Mask2Former**: For semantic/panoptic segmentation with categories
• **GroundingDINO + SAM**: For text-prompted segmentation
• **SAM 2**: For video segmentation tasks

Quick start

Installation

# From GitHub
pip install git+https://github.com/facebookresearch/segment-anything.git

# Optional dependencies
pip install opencv-python pycocotools matplotlib

# Or use HuggingFace transformers
pip install transformers

Download checkpoints

# ViT-H (largest, most accurate) - 2.4GB
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth

# ViT-L (medium) - 1.2GB
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_l_0b3195.pth

# ViT-B (smallest, fastest) - 375MB
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth

Basic usage with SamPredictor

import numpy as np
from segment_anything import sam_model_registry, SamPredictor

# Load model
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda")

# Create predictor
predictor = SamPredictor(sam)

# Set image (computes embeddings once)
image = cv2.imread("image.jpg")
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
predictor.set_image(image)

# Predict with point prompts
input_point = np.array([[500, 375]])  # (x, y) coordinates
input_label = np.array([1])  # 1 = foreground, 0 = background

masks, scores, logits = predictor.predict(
    point_coords=input_point,
    point_labels=input_label,
    multimask_output=True  # Returns 3 mask options
)

# Select best mask
best_mask = masks[np.argmax(scores)]

HuggingFace Transformers

import torch
from PIL import Image
from transformers import SamModel, SamProcessor

# Load model and processor
model = SamModel.from_pretrained("facebook/sam-vit-huge")
processor = SamProcessor.from_pretrained("facebook/sam-vit-huge")
model.to("cuda")

# Process image with point prompt
image = Image.open("image.jpg")
input_points = [[[450, 600]]]  # Batch of points

inputs = processor(image, input_points=input_points, return_tensors="pt")
inputs = {k: v.to("cuda") for k, v in inputs.items()}

# Generate masks
with torch.no_grad():
    outputs = model(**inputs)

# Post-process masks to original size
masks = processor.image_processor.post_process_masks(
    outputs.pred_masks.cpu(),
    inputs["original_sizes"].cpu(),
    inputs["reshaped_input_sizes"].cpu()
)

Core concepts

Model architecture

SAM Architecture:
┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│  Image Encoder  │────▶│ Prompt Encoder  │────▶│  Mask Decoder   │
│     (ViT)       │     │ (Points/Boxes)  │     │ (Transformer)   │
└─────────────────┘     └─────────────────┘     └─────────────────┘
        │                       │                       │
   Image Embeddings      Prompt Embeddings         Masks + IoU
   (computed once)       (per prompt)             predictions

Model variants

| Model | Checkpoint | Size | Speed | Accuracy | |-------|------------|------|-------|----------| | ViT-H | `vit_h` | 2.4 GB | Slowest | Best | | ViT-L | `vit_l` | 1.2 GB | Medium | Good | | ViT-B | `vit_b` | 375 MB | Fastest | Good |

Prompt types

| Prompt | Description | Use Case | |--------|-------------|----------| | Point (foreground) | Click on object | Single object selection | | Point (background) | Click outside object | Exclude regions | | Bounding box | Rectangle around object | Larger objects | | Previous mask | Low-res mask input | Iterative refinement |

Interactive segmentation

Point prompts

# Single foreground point
input_point = np.array([[500, 375]])
input_label = np.array([1])

masks, scores, logits = predictor.predict(
    point_coords=input_point,
    point_labels=input_label,
    multimask_output=True
)

# Multiple points (foreground + background)
input_points = np.array([[500, 375], [600, 400], [450, 300]])
input_labels = np.array([1, 1, 0])  # 2 foreground, 1 background

masks, scores, logits = predictor.predict(
    point_coords=input_points,
    point_labels=input_labels,
    multimask_output=False  # Single mask when prompts are clear
)

Box prompts

# Bounding box [x1, y1, x2, y2]
input_box = np.array([425, 600, 700, 875])

masks, scores, logits = predictor.predict(
    box=input_box,
    multimask_output=False
)

Combined prompts

# Box + points for precise control
masks, scores, logits = predictor.predict(