DSPy

The hand-crafting trap

Imagine you're tuning a radio. You can turn the dial by hand, inching toward the clearest signal — but every time the station drifts (new model version, different task, new data) you have to start over. That's hand-crafted prompting.

DSPy flips the model: you describe what you want (a signature), wire together how data flows (a program), and let an optimiser search for the best prompt wording and few-shot examples automatically — like an auto-tuner for your radio.

The payoff: prompts that generalise better, improve as your dataset grows, and don't silently break when you swap model providers.

What DSPy actually is

DSPy (Declarative Self-improving Python) treats an LLM pipeline as a parameterised program — prompts and few-shot examples are learnable weights, not hardcoded strings. You write Python; DSPy compiles it into optimised instructions for whatever backend you choose.

import dspy

# Configure the LM backend (Anthropic, OpenAI, local — swappable)
lm = dspy.LM("anthropic/claude-3-5-haiku-20241022", max_tokens=1024)
dspy.configure(lm=lm)

Three-line setup. Everything else flows from that.

Core abstractions

Signature — declares inputs and outputs with optional docstring describing the task:

class Sentiment(dspy.Signature):
    '''Classify the sentiment of a product review.'''
    review: str = dspy.InputField()
    sentiment: str = dspy.OutputField(desc="positive, negative, or neutral")

Module — a composable unit wrapping a signature with a strategy (Predict, ChainOfThought, ReAct…):

classify = dspy.Predict(Sentiment)
result = classify(review="The keyboard feels mushy and the trackpad lags.")
print(result.sentiment)  # → negative

Program — a Python class subclassing dspy.Module that chains modules:

class RAGPipeline(dspy.Module):
    def __init__(self):
        self.retrieve = dspy.Retrieve(k=3)
        self.generate = dspy.ChainOfThought("context, question -> answer")

    def forward(self, question):
        context = self.retrieve(question).passages
        return self.generate(context=context, question=question)

Optimiser (Teleprompter) — tunes the program on a labelled dataset:

from dspy.teleprompt import BootstrapFewShot

optimizer = BootstrapFewShot(metric=my_metric, max_bootstrapped_demos=4)
optimised = optimizer.compile(RAGPipeline(), trainset=train_data)

Your first DSPy pipeline end-to-end

import dspy

lm = dspy.LM("anthropic/claude-3-5-haiku-20241022")
dspy.configure(lm=lm)

# 1. Define a signature
class QA(dspy.Signature):
    '''Answer a question with a single concise sentence.'''
    question: str = dspy.InputField()
    answer:   str = dspy.OutputField()

# 2. Build a program
class SimpleQA(dspy.Module):
    def __init__(self):
        self.cot = dspy.ChainOfThought(QA)
    def forward(self, question):
        return self.cot(question=question)

# 3. Prepare labelled data (just a few examples to start)
trainset = [
    dspy.Example(question="What year was Python created?", answer="1991").with_inputs("question"),
    dspy.Example(question="Who wrote Hamlet?", answer="William Shakespeare").with_inputs("question"),
    dspy.Example(question="What is the boiling point of water in Celsius?", answer="100°C").with_inputs("question"),
]

# 4. Define a simple metric
def exact_match(example, pred, trace=None):
    return example.answer.lower() in pred.answer.lower()

# 5. Optimise
from dspy.teleprompt import BootstrapFewShot
optimizer = BootstrapFewShot(metric=exact_match, max_bootstrapped_demos=2)
optimised_qa = optimizer.compile(SimpleQA(), trainset=trainset)

# 6. Use it
print(optimised_qa(question="What language is TensorFlow written in?").answer)

After compile(), optimised_qa's internal prompt now includes automatically chosen few-shot examples that maximise your metric.

Optimisers explained

DSPy ships several teleprompters (optimisers), each with a different tradeoff:

Rule of thumb: start with BootstrapFewShot; graduate to MIPRO when accuracy plateaus.

When DSPy shines vs. overkill

Use DSPy when:

• You have (or can generate) a labelled evaluation set — optimisers need a metric to minimise.
• Your pipeline has multiple chained LLM calls and you're spending hours hand-tweaking each prompt.
• You expect to swap models (e.g., prototype on Haiku, deploy on Sonnet) — signatures are backend-agnostic.
• Prompt quality needs to stay high as your training data grows.

Skip DSPy when:

• You have a single, one-off prompt that works fine — the optimiser overhead isn't worth it.
• You have no labelled data and no way to write a deterministic metric.
• Latency is everything and you can't afford the compilation step.

Gotchas and limits

Compilation cost: BootstrapFewShot calls the LLM many times (train set × candidate programs). Budget accordingly — run it overnight if your train set is large.

Metric design is hard: A weak metric (e.g., substring match for open-ended answers) produces badly optimised prompts. Invest time in your metric before investing in optimisation.

Module stacking amplifies failures: In a 4-module pipeline, if module 1 gets the context slightly wrong, modules 2–4 compound the error. Add assertions (dspy.Assert) to catch bad intermediate outputs early.

class SafeRAG(dspy.Module):
    def forward(self, question):
        ctx = self.retrieve(question).passages
        dspy.Assert(len(ctx) > 0, "Retrieval returned nothing — check your corpus.")
        return self.generate(context=ctx, question=question)

Version your compiled programs: Save optimised program state with program.save("my_program.json") and reload with program.load("my_program.json"). Treat these like model checkpoints.