Qwen-VL

Qwen-VL family overview

Qwen-VL (Alibaba, August 2023) was one of the first competitive open-source visual language models. Qwen2-VL (September 2024) is a major upgrade, available in 2B, 7B, and 72B sizes. Key improvements: (1) Naive Dynamic Resolution — supports any image resolution from 256×28 to 1280×1280, processing images at their native resolution without fixed-size resizing; (2) Multimodal Rotary Position Embedding (M-RoPE) — encodes both spatial and temporal positions, enabling video understanding; (3) state-of-the-art OCR on Chinese and English text in images.

Qwen2-VL architecture

Qwen2-VL uses a Vision Transformer (ViT) as the visual encoder, processing images in 14×14 pixel patches. The Naive Dynamic Resolution mechanism dynamically adjusts the number of visual tokens based on image resolution — a 224×224 image produces 256 tokens; a 1120×1120 image produces 6400 tokens. Temporal compression (2× for video) reduces video frames to manageable token counts. The visual tokens are processed by the Qwen2 language model backbone via cross-attention, with M-RoPE encoding both 2D spatial position (row, column) and temporal position (frame index) simultaneously.

Inference with HuggingFace

from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from qwen_vl_utils import process_vision_info
from PIL import Image
import torch

model_name = "Qwen/Qwen2-VL-7B-Instruct"
model = Qwen2VLForConditionalGeneration.from_pretrained(
    model_name, torch_dtype=torch.bfloat16, device_map="auto"
)
processor = AutoProcessor.from_pretrained(model_name)

# Single image VQA
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "path/to/chart.png"},
            {"type": "text", "text": "What is the highest value in this bar chart?"},
        ],
    }
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
).to(model.device)

with torch.no_grad():
    generated_ids = model.generate(**inputs, max_new_tokens=512)

output = processor.batch_decode(
    [out[len(inp):] for out, inp in zip(generated_ids, inputs.input_ids)],
    skip_special_tokens=True,
)[0]
print(output)

Document and chart understanding

Qwen2-VL excels at dense document understanding tasks:

from pathlib import Path
import base64

def analyze_document(image_path: str, question: str) -> str:
    messages = [
        {
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "image": image_path,
                    "min_pixels": 224 * 224,
                    "max_pixels": 1280 * 28 * 28,  # high res for documents
                },
                {"type": "text", "text": question},
            ],
        }
    ]
    text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    image_inputs, _ = process_vision_info(messages)
    inputs = processor(text=[text], images=image_inputs, return_tensors="pt").to(model.device)
    with torch.no_grad():
        output_ids = model.generate(**inputs, max_new_tokens=1024)
    return processor.decode(output_ids[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)

# Examples
print(analyze_document("invoice.png", "Extract vendor name, total amount, and date as JSON"))
print(analyze_document("chart.png", "Summarise the key trend shown in this chart"))
print(analyze_document("contract.pdf.png", "What is the termination clause?"))

Multi-image and video

# Multiple images in one conversation
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "before.png"},
            {"type": "image", "image": "after.png"},
            {"type": "text", "text": "What changed between these two screenshots?"},
        ],
    }
]

# Video understanding
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "video",
                "video": "demo.mp4",
                "max_pixels": 360 * 420,
                "fps": 1.0,          # sample 1 frame per second
            },
            {"type": "text", "text": "Describe what happens in this video"},
        ],
    }
]

Benchmark performance

Qwen2-VL 72B benchmarks (vs other open VLMs, as of late 2024):

• DocVQA: 96.5% — state-of-the-art among open models
• ChartQA: 88.3% — leading open model for chart understanding
• TextVQA: 85.5% — competitive with GPT-4V
• OCRBench: 877/1000 — best open model for OCR as of 2024
• MMMU: 64.5% — strong general multimodal reasoning

The 7B model achieves competitive performance on doc/chart tasks at a fraction of the compute of the 72B model — good choice for production document processing.

Gotchas

• qwen_vl_utils required: The process_vision_info function from the qwen_vl_utils package handles image/video loading and format conversion. Install it separately: pip install qwen-vl-utils. Without it, image processing fails silently.
• max_pixels matters: Set max_pixels in the image dict to control resolution. Higher resolution = more tokens = more VRAM. For typical photos 1024×1024 is sufficient; for dense documents use 1280×1280.
• Multilingual by default: Qwen2-VL responds in the language of the question. For consistent English outputs when the image may contain Chinese text, explicitly add "Reply in English" to your prompt.