Prompt Versioning

Prompts are code — treat them that way

Imagine your team's most important prompt lives in a Notion doc. Someone tweaks the wording on Tuesday. Production starts giving worse answers on Wednesday. Nobody knows what changed, when, or who did it — and there's no way to roll back.

This is the prompt-as-config-file antipattern. Prompts drive model behaviour just as much as code does, yet teams routinely skip the practices (version control, code review, change logs) they'd apply to any .py file.

Prompt versioning treats every prompt as a tracked artefact: it has a history, diffs are reviewable, experiments are labelled, and you can roll back to the last known-good version in seconds.

What prompt versioning solves

Silent regressions — a changed comma or reordered instruction can drop accuracy by 10%. Version control makes the diff visible before deployment.

Attribution — six months later, "why does this prompt say X?" has a clear audit trail with author and rationale.

Parallel experiments — run v1 and v2 in A/B and track which prompt hash produced which output, making analysis clean.

Multi-environment consistency — dev, staging, and prod each get a pinned version, not "whatever's in the Notion doc right now".

Minimal Git-based approach

The simplest approach: store prompts as plain text files alongside your code and let Git do the work.

prompts/
  classify_sentiment/
    v1.txt          ← original
    v2.txt          ← refined
    current -> v2.txt  ← symlink (or just use git tags)
  summarise_doc/
    v1.txt

# Load the pinned version at runtime
from pathlib import Path

PROMPT_DIR = Path(__file__).parent / "prompts"

def load_prompt(name: str, version: str = "current") -> str:
    '''Load a versioned prompt template.'''
    path = PROMPT_DIR / name / f"{version}.txt"
    return path.read_text()

system_prompt = load_prompt("classify_sentiment", version="v2")

Commit every meaningful change with a message like prompt: classify_sentiment v2 — add hedge detection. Git blame then gives you instant attribution.

Dedicated prompt registries

For larger teams, a dedicated registry adds metadata (tags, eval scores, author) that Git alone can't express cleanly.

from langsmith import Client  # or use Langfuse, PromptLayer, etc.

client = Client()

# Push a new version
client.push_prompt(
    "classify-sentiment",
    object=ChatPromptTemplate.from_messages([
        ("system", "You are a sentiment classifier. Return only: positive, negative, or neutral."),
        ("human", "{review}")
    ]),
    tags=["v2", "production"],
    description="Added handling for sarcasm and hedging language"
)

# Pull a pinned version in production
prompt = client.pull_prompt("classify-sentiment:v2")

Key registry features to look for: diff viewer, eval score history per version, environment pinning (dev/staging/prod get different versions), and webhook triggers for CI/CD.

Semantic versioning for prompts

Borrow SemVer (MAJOR.MINOR.PATCH) adapted for prompt semantics:

Store version in the file metadata or as a comment at the top of the prompt file: # version: 1.2.0 | 2024-03-15 | author: deepak

Change management workflow

'''
Recommended workflow for prompt changes
'''

# 1. Create a branch: git checkout -b prompt/sentiment-v3
# 2. Edit the prompt file
# 3. Run your eval suite against the new version:
import anthropic, json

client = anthropic.Anthropic()

def run_eval(prompt_version: str, test_cases: list[dict]) -> float:
    prompt = load_prompt("classify_sentiment", version=prompt_version)
    correct = 0
    for case in test_cases:
        resp = client.messages.create(
            model="claude-3-5-haiku-20241022",
            max_tokens=10,
            system=prompt,
            messages=[{"role": "user", "content": case["text"]}]
        )
        pred = resp.content[0].text.strip().lower()
        if pred == case["label"]:
            correct += 1
    return correct / len(test_cases)

score_v2 = run_eval("v2", test_cases)
score_v3 = run_eval("v3", test_cases)
print(f"v2: {score_v2:.1%}  v3: {score_v3:.1%}")

# 4. If v3 > v2: open PR with diff + eval numbers in description
# 5. Merge → tag the commit: git tag prompt/sentiment-v3
# 6. Deploy: update env var SENTIMENT_PROMPT_VERSION=v3

Gotchas

Don't version-control secrets inside prompts. If your prompt contains an API key, internal codename, or PII example, it will be committed to history forever. Use placeholder variables and inject at runtime.

Model upgrades are silent breaking changes. When you move from claude-3-haiku to claude-3-5-haiku, your prompt might behave differently even though the prompt text hasn't changed. Treat model upgrades as a MAJOR version bump and run your eval suite.

Branching strategy matters. Use feature branches for prompt experiments — committing directly to main makes bisecting a regression painful.

Eval coverage is load-bearing. Version control without an eval suite just gives you a well-organised history of regressions. Write evals before you need to debug a regression.