PaliGemma

PaliGemma architecture

PaliGemma (Google DeepMind, May 2024) is a 3B parameter vision-language model combining a SigLIP vision encoder (400M params, trained with sigmoid loss instead of contrastive loss) with a Gemma 2B language model. The vision encoder processes images at 224×224 resolution, producing 256 image tokens. These are linearly projected and prepended to the text tokens before the Gemma language model processes them. Total trainable: 3B parameters.

Unlike GPT-4V which is designed for general conversation, PaliGemma is designed as a transfer model: pre-trained on a broad mixture of vision-language tasks, then fine-tuned on specific downstream tasks. It excels at tasks with clear input-output structure: captioning, VQA, document reading, and structured prediction.

PaliGemma vs PaliGemma 2

PaliGemma (v1): 3B params, 224×224 input, strong on captioning and VQA. Widely used for fine-tuning. PaliGemma 2 (December 2024): Three sizes — 3B, 10B, 28B. Uses Gemma 2 backbone (improved quality). Supports 224×448 and 448×448 resolutions for document tasks. Significantly better on DocVQA, TextVQA, and OCR benchmarks. For new projects, prefer PaliGemma 2. For fine-tuning on a tight budget, PaliGemma v1 3B is more accessible.

Inference with HuggingFace

from transformers import PaliGemmaForConditionalGeneration, AutoProcessor
from PIL import Image
import torch, requests

model_id = "google/paligemma-3b-mix-224"
# Or PaliGemma 2: "google/paligemma2-3b-pt-224"

processor = AutoProcessor.from_pretrained(model_id)
model = PaliGemmaForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

image = Image.open("chart.png")

# PaliGemma uses task-specific prompt prefixes
prompt = "caption en"  # or "answer en What is shown in this image?"

inputs = processor(text=prompt, images=image, return_tensors="pt").to(model.device)

with torch.no_grad():
    generated_ids = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
    )

# Decode only the new tokens
generated_text = processor.decode(
    generated_ids[0][inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
)
print(generated_text)

Fine-tuning PaliGemma

from transformers import PaliGemmaForConditionalGeneration, AutoProcessor, TrainingArguments, Trainer
from datasets import Dataset
from peft import LoraConfig, get_peft_model
import torch

model_id = "google/paligemma-3b-pt-224"  # use pretrained (pt) for fine-tuning
processor = AutoProcessor.from_pretrained(model_id)
model = PaliGemmaForConditionalGeneration.from_pretrained(
    model_id, torch_dtype=torch.bfloat16
)

# LoRA config for efficient fine-tuning
lora_config = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
)
model = get_peft_model(model, lora_config)

def preprocess(examples):
    images = [Image.open(p) for p in examples["image_path"]]
    inputs = processor(
        text=examples["prompt"],
        images=images,
        suffix=examples["label"],  # target text
        return_tensors="pt",
        padding=True,
    )
    inputs["labels"] = inputs["input_ids"].clone()
    # Mask non-label tokens in loss
    inputs["labels"][inputs["token_type_ids"] == 0] = -100
    return inputs

# Dataset: {"image_path": [...], "prompt": ["answer en ..."], "label": [...]}
training_args = TrainingArguments(
    output_dir="./paligemma-finetuned",
    num_train_epochs=5,
    per_device_train_batch_size=4,
    learning_rate=2e-5,
    bf16=True,
    dataloader_num_workers=4,
)

Task-specific prompting

PaliGemma uses structured prompt prefixes that tell it what task to perform:

task_prompts = {
    "captioning": "caption en",
    "vqa": "answer en {question}",
    "ocr": "ocr",
    "detection": "detect {object}",          # returns bounding boxes
    "segmentation": "segment {object}",      # returns segmentation mask
    "document_qa": "answer en {question}",   # for DocVQA
    "reference_expression": "refer {description}",  # locate described region
}

# Examples:
prompts = [
    ("caption en", "→ A bar chart showing quarterly revenue growth"),
    ("answer en What company is shown?", "→ Acme Corp"),
    ("ocr", "→ Invoice #12345 Total: $1,234.56"),
    ("detect car", "→ "),  # location tokens
]

Benchmark performance

PaliGemma 3B zero-shot (pretrained) vs fine-tuned on specific benchmarks:

• COCO captioning (CIDEr): 144 (pretrained) → 156 (fine-tuned) — competitive with larger models
• VQAv2: 83.1% (pretrained), 86.0% (fine-tuned)
• DocVQA: 52.3% (pretrained), 84.8% (fine-tuned) — massive fine-tuning gain on document tasks
• TextVQA: 55.1% (pretrained), 73.1% (fine-tuned)

The large gap between pretrained and fine-tuned on DocVQA highlights PaliGemma's design philosophy: it's a transfer model, not a zero-shot model. For document-specific tasks, fine-tune with even 100–500 examples and see dramatic improvements.

Gotchas

• Use pretrained (pt) for fine-tuning: HuggingFace hosts both pretrained (paligemma-3b-pt-224) and fine-tuned mix (paligemma-3b-mix-224) checkpoints. For fine-tuning on a new task, start from the pretrained checkpoint — the mix checkpoint has already been fine-tuned and may interfere.
• Resolution matters for documents: At 224×224, text in dense documents is unreadable. Use PaliGemma 2 at 448×448 for document tasks, or tile the image manually before passing to the 224×224 model.
• Task prefix must match training: PaliGemma was trained with specific task prefix strings. Using the wrong prefix or omitting it produces poor results. Check the task prefix table in the paper appendix for your task.