/multimodal-blip-2

name: multimodal-blip-2
description: Vision-language pre-training framework bridging frozen image encoders and LLMs. Use when you need image captioning, visual question answering, image-text retrieval, or multimodal chat with state-of-the-art zero-shot performance.
license: MIT
metadata:
  version: 1.0.0
  author: Orchestra Research
  tags: [Multimodal, Vision-Language, Image Captioning, VQA, Zero-Shot]
  dependencies: [transformers>=4.30.0, torch>=1.10.0, Pillow]

BLIP-2: Vision-Language Pre-training

Comprehensive guide to using Salesforce's BLIP-2 for vision-language tasks with frozen image encoders and large language models.

When to use BLIP-2

**Use BLIP-2 when:**

• Need high-quality image captioning with natural descriptions
• Building visual question answering (VQA) systems
• Require zero-shot image-text understanding without task-specific training
• Want to leverage LLM reasoning for visual tasks
• Building multimodal conversational AI
• Need image-text retrieval or matching

**Key features:**

• **Q-Former architecture**: Lightweight query transformer bridges vision and language
• **Frozen backbone efficiency**: No need to fine-tune large vision/language models
• **Multiple LLM backends**: OPT (2.7B, 6.7B) and FlanT5 (XL, XXL)
• **Zero-shot capabilities**: Strong performance without task-specific training
• **Efficient training**: Only trains Q-Former (~188M parameters)
• **State-of-the-art results**: Beats larger models on VQA benchmarks

**Use alternatives instead:**

• **LLaVA**: For instruction-following multimodal chat
• **InstructBLIP**: For improved instruction-following (BLIP-2 successor)
• **GPT-4V/Claude 3**: For production multimodal chat (proprietary)
• **CLIP**: For simple image-text similarity without generation
• **Flamingo**: For few-shot visual learning

Quick start

Installation

# HuggingFace Transformers (recommended)
pip install transformers accelerate torch Pillow

# Or LAVIS library (Salesforce official)
pip install salesforce-lavis

Basic image captioning

import torch
from PIL import Image
from transformers import Blip2Processor, Blip2ForConditionalGeneration

# Load model and processor
processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
model = Blip2ForConditionalGeneration.from_pretrained(
    "Salesforce/blip2-opt-2.7b",
    torch_dtype=torch.float16,
    device_map="auto"
)

# Load image
image = Image.open("photo.jpg").convert("RGB")

# Generate caption
inputs = processor(images=image, return_tensors="pt").to("cuda", torch.float16)
generated_ids = model.generate(**inputs, max_new_tokens=50)
caption = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(caption)

Visual question answering

# Ask a question about the image
question = "What color is the car in this image?"

inputs = processor(images=image, text=question, return_tensors="pt").to("cuda", torch.float16)
generated_ids = model.generate(**inputs, max_new_tokens=50)
answer = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(answer)

Using LAVIS library

import torch
from lavis.models import load_model_and_preprocess
from PIL import Image

# Load model
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model, vis_processors, txt_processors = load_model_and_preprocess(
    name="blip2_opt",
    model_type="pretrain_opt2.7b",
    is_eval=True,
    device=device
)

# Process image
image = Image.open("photo.jpg").convert("RGB")
image = vis_processors["eval"](image).unsqueeze(0).to(device)

# Caption
caption = model.generate({"image": image})
print(caption)

# VQA
question = txt_processors["eval"]("What is in this image?")
answer = model.generate({"image": image, "prompt": question})
print(answer)

Core concepts

Architecture overview

BLIP-2 Architecture:
┌─────────────────────────────────────────────────────────────┐
│                        Q-Former                              │
│  ┌─────────────────────────────────────────────────────┐    │
│  │     Learned Queries (32 queries × 768 dim)          │    │
│  └────────────────────────┬────────────────────────────┘    │
│                           │                                  │
│  ┌────────────────────────▼────────────────────────────┐    │
│  │    Cross-Attention with Image Features               │    │
│  └────────────────────────┬────────────────────────────┘    │
│                           │                                  │
│  ┌────────────────────────▼────────────────────────────┐    │
│  │    Self-Attention Layers (Transformer)               │    │
│  └────────────────────────┬────────────────────────────┘    │
└───────────────────────────┼─────────────────────────────────┘
                            │
┌───────────────────────────▼─────────────────────────────────┐
│  Frozen Vision Encoder    │      Frozen LLM                  │
│  (ViT-G/14 from EVA-CLIP) │      (OPT or FlanT5)            │
└─────────────────────────────────────────────────────────────┘

Model variants

| Model | LLM Backend | Size | Use Case | |-------|-------------|------|----------| | `blip2-opt-2.7b` | OPT-2.7B | ~4GB | General captioning, VQA | | `blip2-opt-6.7b` | OPT-6.7B | ~8GB | Better reasoning | | `blip2-flan-t5-xl` | FlanT5-XL | ~5GB | Instruction following | | `blip2-flan-t5-xxl` | FlanT5-XXL | ~13GB | Best quality |

Q-Former components

| Component | Description | Parameters | |-----------|-------------|------------| | Learned queries | Fixed set of learnable embeddings | 32 × 768 | | Image transformer | Cross-attention to vision features | ~108M | | Text transformer | Self-attention for text | ~108M | | Linear projection | Maps to LLM dimension | Varies |

Advanced usage

Batch processing

from PIL import Image
import torch

# Load multiple images
images = [Image.open(f"image_{i}.jpg").convert("RGB") for i in range(4)]
questions = [
    "What is