Whisper

Whisper model family

Whisper is an encoder-decoder transformer trained by OpenAI on a massive 680,000-hour dataset of internet audio, spanning 96 languages. It handles a wide range of audio conditions (background noise, accents, technical vocabulary) without fine-tuning.

The model comes in five sizes: tiny (39M params), base (74M), small (244M), medium (769M), large-v3 (1.5B). Each successive size improves accuracy at the cost of inference speed and memory.

Transcription with openai-whisper

import whisper

# Load model — downloads automatically on first use
model = whisper.load_model("large-v3")

# Transcribe audio file
result = model.transcribe("audio.mp3")
print(result["text"])

# With language hint for better accuracy on non-English audio
result = model.transcribe("audio.mp3", language="fr", task="transcribe")

# Or translate to English
result = model.transcribe("audio.mp3", language="fr", task="translate")

# Get segment-level timestamps
for segment in result["segments"]:
    print(f"[{segment['start']:.1f}s – {segment['end']:.1f}s] {segment['text']}")

# CLI usage
pip install openai-whisper
whisper audio.mp3 --model large-v3 --output_dir ./transcripts
whisper audio.mp3 --model medium --language French --task translate

Faster-Whisper for production

The original openai-whisper is PyTorch-based and not optimised for throughput. faster-whisper reimplements Whisper using CTranslate2, achieving 4× faster inference with half the memory footprint.

from faster_whisper import WhisperModel

# Load model (downloads automatically)
model = WhisperModel("large-v3", device="cuda", compute_type="float16")

# Transcribe
segments, info = model.transcribe("audio.mp3", beam_size=5)
print(f"Detected language: {info.language} ({info.language_probability:.0%})")

for segment in segments:
    print(f"[{segment.start:.2f}s – {segment.end:.2f}s] {segment.text}")

# Batch transcription
audio_files = ["audio1.mp3", "audio2.mp3", "audio3.mp3"]
for f in audio_files:
    segments, _ = model.transcribe(f, language="en", beam_size=1)  # beam_size=1 fastest
    text = " ".join([s.text for s in segments])
    print(f"{f}: {text[:100]}...")

For CPU-only servers: compute_type="int8" gives further 2× speedup with minimal quality loss.

Word-level timestamps

from faster_whisper import WhisperModel

model = WhisperModel("medium", device="cuda", compute_type="float16")

# Enable word-level timestamps
segments, info = model.transcribe(
    "interview.mp3",
    word_timestamps=True,
    beam_size=5,
)

# Generate SRT subtitles with word highlighting
with open("subtitles.srt", "w") as f:
    for i, segment in enumerate(segments, 1):
        f.write(f"{i}
")
        f.write(f"{format_time(segment.start)} --> {format_time(segment.end)}
")
        f.write(segment.text.strip() + "

")

# Word-level data for karaoke / highlighting
for segment in segments:
    for word in segment.words:
        print(f"  '{word.word}': {word.start:.2f}s–{word.end:.2f}s (prob={word.probability:.2f})")

Word timestamps are useful for subtitle generation, audio search indexing, and finding specific moments in long recordings.

Multilingual transcription

Whisper's multilingual capability comes from its diverse training data. The large-v3 model performs near-human on English (2% WER), French, German, Spanish, and performs well on 90+ other languages. Performance degrades on low-resource languages with limited training data.

from faster_whisper import WhisperModel

model = WhisperModel("large-v3", device="cuda", compute_type="float16")

# Auto-detect language
segments, info = model.transcribe("unknown_language.mp3")
print(f"Detected: {info.language} ({info.language_probability:.0%})")

# Transcribe code-switched audio (multiple languages in one recording)
# Whisper handles this naturally — just set language to None for auto-detect

# Translate non-English to English
from whisper import load_model
model = load_model("large-v3")
result = model.transcribe("spanish_meeting.mp3", task="translate")
print(result["text"])  # English translation

Deployment patterns

Three common patterns for deploying Whisper in production:

Synchronous API: FastAPI endpoint that accepts audio files and returns transcript. Good for <60s audio and low-latency requirements.

Async queue: For long recordings (>5 min), use a job queue (Celery, Redis Queue). Client submits audio, gets a job ID, polls for completion. Better for batch transcription pipelines.

Streaming (WhisperLive): For real-time transcription, WhisperLive or whisper-streaming transcribe audio as it arrives using Voice Activity Detection (VAD) to segment the stream. Latency: 1–3s typically.

from fastapi import FastAPI, UploadFile
from faster_whisper import WhisperModel

app = FastAPI()
model = WhisperModel("medium", device="cuda", compute_type="float16")

@app.post("/transcribe")
async def transcribe(file: UploadFile):
    audio_bytes = await file.read()
    # Write to temp file (faster-whisper requires file path)
    with open("/tmp/audio.wav", "wb") as f:
        f.write(audio_bytes)
    segments, info = model.transcribe("/tmp/audio.wav")
    return {"text": " ".join(s.text for s in segments), "language": info.language}

Gotchas

Audio preprocessing: Whisper expects 16kHz mono WAV. For other formats (MP3, MP4, M4A), it uses ffmpeg internally. Ensure ffmpeg is installed. For very noisy audio, pre-process with noise reduction (e.g. noisereduce library) to improve accuracy.

Long audio chunking: Whisper processes 30-second chunks. For files longer than 30 minutes, faster-whisper handles chunking automatically. The original whisper library can have issues with very long files — use faster-whisper for anything > 5 minutes.

Hallucination on silence: Whisper sometimes generates phantom text for silent or very quiet audio segments. Use VAD (Voice Activity Detection) to skip silent segments: model.transcribe(audio, vad_filter=True) in faster-whisper.

GPU memory: large-v3 needs ~10GB VRAM. For <8GB GPUs, use medium (5GB) or small (2GB). On CPU, transcription is ~15–50× slower than real-time — only practical for offline batch processing.