LLaVA

LLaVA architecture

LLaVA (Visual Instruction Tuning, Liu et al. 2023) combines a frozen CLIP visual encoder with a large language model (originally Vicuna/Llama) connected by a trainable linear projection layer. The architecture is elegantly simple:

1. Visual encoder: CLIP ViT-L/14 encodes the image into visual features (a grid of 256 patch tokens)
2. Projection: A learnable linear layer maps CLIP features to the LLM's embedding space
3. Language model: Visual tokens are prepended to the text tokens and processed by the LLM

Only the projection layer (and the LLM) are trained; CLIP is frozen throughout. LLaVA-1.5 upgraded the projection to a two-layer MLP, boosting performance significantly.

Training pipeline

LLaVA uses a two-stage training process:

Stage 1 — Feature alignment: Train only the projection layer on image-caption pairs (CC3M). The goal is to map CLIP features into the LLM's embedding space. The LLM is frozen. This stage takes ~1 hour on 8 A100s.

Stage 2 — Visual instruction tuning: Fine-tune both the projection layer and the LLM (using LoRA or full fine-tuning) on multimodal instruction-following data. LLaVA's original dataset was generated by feeding COCO image captions to GPT-4 and asking it to generate instruction-following conversations — a surprisingly effective synthetic data strategy.

Running LLaVA inference

from transformers import LlavaNextProcessor, LlavaNextForConditionalGeneration
from PIL import Image
import torch, requests

model_id = "llava-hf/llava-v1.6-mistral-7b-hf"
processor = LlavaNextProcessor.from_pretrained(model_id)
model = LlavaNextForConditionalGeneration.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto"
)

# Load image (local file or URL)
image = Image.open("chart.png")
# or: image = Image.open(requests.get(url, stream=True).raw)

# Build conversation
conversation = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What does this chart show? Summarise the key trends."},
        ],
    },
]
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
inputs = processor(images=image, text=prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=512, do_sample=False)

# Decode only the new tokens
new_tokens = output[0][inputs["input_ids"].shape[1]:]
print(processor.decode(new_tokens, skip_special_tokens=True))

LLaVA-1.5 and LLaVA-NeXT

LLaVA-1.5: Upgraded projection from linear to 2-layer MLP, switched to a larger CLIP encoder, and used higher-quality instruction data. Achieved state-of-the-art among open VLMs in 2023 at just 13B total parameters. LLaVA-NeXT (v1.6): Added dynamic high-resolution processing — images are split into tiles (up to 4× the base resolution), each processed by CLIP separately, then merged. This allows reading fine text and detailed charts that lower-resolution VLMs miss. LLaVA-NeXT with Mistral-7B or LLaMA-3-8B backends is the practical open-source choice for most vision tasks today.

Building a VQA pipeline

from pathlib import Path
import json

def analyze_images(image_paths: list[str], question: str) -> list[dict]:
    results = []
    for img_path in image_paths:
        image = Image.open(img_path)
        conversation = [{"role": "user", "content": [
            {"type": "image"},
            {"type": "text", "text": question},
        ]}]
        prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
        inputs = processor(images=image, text=prompt, return_tensors="pt").to(model.device)
        with torch.no_grad():
            output = model.generate(**inputs, max_new_tokens=256, do_sample=False)
        new_tokens = output[0][inputs["input_ids"].shape[1]:]
        answer = processor.decode(new_tokens, skip_special_tokens=True)
        results.append({"image": img_path, "question": question, "answer": answer})
    return results

# Example: extract structured data from invoice images
invoices = ["invoice_001.png", "invoice_002.png"]
results = analyze_images(invoices, "Extract the total amount, vendor name, and date as JSON.")
print(json.dumps(results, indent=2))

LLaVA vs other VLMs

• Qwen-VL: Strong multilingual VLM from Alibaba; better on Chinese text recognition and document understanding.
• InternVL: Strong open-source VLM; InternVL2-8B beats LLaVA-NeXT-8B on most benchmarks as of 2024.
• PaliGemma (Google): Versatile VLM for fine-tuning on custom vision tasks; better for specialised downstream tasks.
• GPT-4V / GPT-4o: Significantly outperforms all open VLMs on complex chart understanding, medical imaging, and OCR tasks — but closed and expensive.

Gotchas

• Image resolution: The base CLIP encoder operates at 336×336 pixels. For high-resolution images (documents, charts with small text), use LLaVA-NeXT with the llava-hf/llava-v1.6-* checkpoints which support dynamic tiling.
• Multiple images: LLaVA-1.5 handles only one image per conversation. LLaVA-NeXT supports multiple images — use {"type": "image"} placeholders in the conversation for each image.
• Token count: Each 336×336 image adds 576 visual tokens to the context. For LLaVA-NeXT with high-resolution tiling, a single image can use 2000+ tokens. Factor this into your context budget.