Allocating additional compute at inference time — via best-of-N sampling, majority vote, chain-of-thought, or search — to improve output quality without retraining.
A model's average output may be mediocre, but its best output from N tries is much better. Test-time compute (TTC) exploits this by generating multiple candidate outputs and selecting the best — using a verifier, judge, or aggregation rule. Recent work (DeepSeek-R1, OpenAI o1) shows TTC can match much larger models at a fraction of training cost.
Generate N responses with temperature > 0, score each, return the best. " "The scorer can be: a reward model, an LLM judge, a verifier (for math/code), " "or a heuristic (length, confidence). N=8–32 is typical; gains plateau beyond 32 for most tasks.
import asyncio
from openai import AsyncOpenAI
client = AsyncOpenAI()
async def best_of_n(prompt: str, n: int = 8, scorer=None) -> str:
tasks = [
client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
temperature=0.8,
)
for _ in range(n)
]
responses = await asyncio.gather(*tasks)
candidates = [r.choices[0].message.content for r in responses]
if scorer:
scores = [scorer(prompt, c) for c in candidates]
return candidates[scores.index(max(scores))]
# default: return longest (simple heuristic)
return max(candidates, key=len)For questions with discrete answers (multiple-choice, math), generate N responses " "and return the most common answer. Self-consistency (Wang et al. 2022) reliably improves " "CoT reasoning by 5–15pp on benchmarks like GSM8K and MATH.
from collections import Counter
async def self_consistency(prompt: str, n: int = 10) -> str:
tasks = [
client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
temperature=0.7,
)
for _ in range(n)
]
responses = await asyncio.gather(*tasks)
# Extract final answers (assumes last line or boxed answer)
answers = [extract_answer(r.choices[0].message.content) for r in responses]
most_common = Counter(answers).most_common(1)[0][0]
return most_commonTree-of-Thought (ToT) structures reasoning as a tree: generate multiple next-step thoughts, evaluate each with a value function, and continue from the most promising branch. Monte Carlo Tree Search (MCTS) adds rollout estimation for deeper planning. Both are powerful for complex multi-step reasoning but expensive (10–100× compute vs greedy decoding).
Use TTC when: accuracy matters more than latency/cost, the task is verifiable (math, code), or you have a reliable reward model. Avoid when: real-time response is required, tasks are subjective (creative writing), or you lack a good scorer.
Best-of-4 with a cheap judge costs ~5× more but gains 15–20% on hard reasoning. Compare to fine-tuning: TTC is zero-shot deployable, works today, but adds per-request cost. A hybrid: fine-tune for cheap common cases, apply TTC only for flagged hard cases.
Test-time compute exhibits clear scaling laws: allocating 2x more samples typically yields diminishing returns of 3-5% quality improvement. The optimal allocation depends on the task: for reasoning tasks (math, code) allocating more samples helps more. For factual recall tasks, extra samples help less. Empirically, using N=10 samples with majority voting recovers 70-80% of what chain-of-thought alone does.
import numpy as np
def allocate_test_compute(quality_target, latency_budget_ms, task_type):
"""Allocate samples based on task and constraints."""
if task_type == "reasoning":
base_samples = 5
scaling_factor = 1.5 # Reasoning benefits more from samples
else: # factual/classification
base_samples = 3
scaling_factor = 1.2
max_samples = int(latency_budget_ms / 50) # 50ms per sample
samples = min(base_samples, int(base_samples * scaling_factor))
return max(1, min(samples, max_samples))
# Best-of-N vs majority vote tradeoff
def best_of_N(logits_list):
return max(logits_list, key=lambda x: x.max())
def majority_vote(predictions_list):
return max(set(predictions_list), key=predictions_list.count)The choice between best-of-N, majority voting, and fusion methods impacts both quality and latency. Best-of-N is most effective but requires full reranking. Majority voting is simpler and faster. Weighted voting based on model confidence adds only ~5% quality but doubles compute cost. For real-time applications, early stopping (stop sampling when confidence > threshold) achieves 80% of quality gains with 60% fewer samples.
# Weighted voting with early stopping
def adaptive_test_compute(
model, input_text, confidence_threshold=0.85,
max_samples=10, min_samples=3
):
predictions, confidences = [], []
for i in range(max_samples):
pred, conf = model.generate_with_confidence(input_text)
predictions.append(pred)
confidences.append(conf)
# Early stopping: if consensus is strong enough, stop
if i >= min_samples - 1:
weights = np.array(confidences) / sum(confidences)
if max(weights) > confidence_threshold:
break
return np.average(
np.array([p.value for p in predictions]),
weights=confidences
)| Method | Samples Needed | Quality Gain | Latency Overhead |
|---|---|---|---|
| Greedy (1 sample) | 1 | Baseline | 1x |
| Best-of-5 | 5 | +8-12% | 5x |
| Majority Vote (5) | 5 | +6-10% | 3x |
| Weighted Voting (5) | 5 | +7-11% | 4x |
| Adaptive (early stop) | 3-4 avg | +5-8% | 2.5x |
OpenAI reports that allocating more test-time compute on difficult examples (via chain-of-thought or Best-of-5) improves o1's performance on AIME from 68% to 94%. Google's AlphaProof uses similar strategies: for hard math problems, expand the search tree (best-of-N sampling) before committing. The cost scales with problem difficulty. Simple questions: 1 sample sufficient. Hard problems: allocate 10-50 samples. This is more efficient than allocating uniform compute across all queries.
Verifier scaling: train a verifier model to predict when a sampled response is likely correct. Use verification to decide whether to return the first sample or search harder. Efficient: 90% of queries return after 1 sample + verification, 10% trigger full search. Only the uncertain cases incur the cost of test-time compute. This achieves 95% of the quality of full test-time compute at 30% of the cost.
Test-time compute is most effective for tasks with clear right/wrong answers: math problems, code generation, knowledge retrieval. For tasks with subjective evaluations (creative writing, design suggestions), extra samples help less because the distribution over quality is wider. For tasks with a single dominant mode (factual recall, simple classification), one sample often suffices.
Budget constraints drive decisions: mobile devices (latency < 500ms) cannot afford test-time compute. Server-side APIs can allocate 1-5 seconds. Batch processing (overnight jobs) can allocate 30+ seconds. Choose the test-time strategy based on the budget: majority voting for <1 second, best-of-N for 3-5 seconds, iterative search for batch jobs.
Hybrid strategies maximize quality within constraints: fast primary model + expensive verification on uncertain cases. Spend test-time compute only where confidence is low. This achieves 95% of the quality of full test-time compute at 40% of the cost.
Scaling test-time compute is an active research frontier. OpenAI's o1 model allocates test-time compute adaptively based on problem difficulty. Google DeepMind's AlphaProof uses tree search for mathematical theorem proving, allocating compute based on proof difficulty. Meta's recent work shows that test-time compute can improve 7B models to match 70B+ models on reasoning tasks—a potential game-changer for efficiency.
Future directions: learned allocation (train a model to predict optimal compute per query), mixture of test-time strategies (combine search with sampling), and interactive test-time compute (interact with the user for clarification if unsure). Test-time compute is moving from a batch-processing curiosity to a core inference strategy.
Implementation considerations: test-time compute requires batching infrastructure. For web APIs, you might return faster with N=1 sample and let clients request more samples if needed. For batch processing, allocate uniformly. For interactive systems (chatbots), use adaptive allocation: start with N=1, if confidence is low, run N=5 in parallel. This gives users fast responses for easy questions and more thought for hard questions.
Combining with retrieval: in RAG systems, test-time compute helps both at retrieval time (search harder for better documents) and at generation time (generate multiple candidate answers). A state-of-the-art RAG system uses test-time compute at both stages, achieving 15-20% quality improvements over single-pass RAG.
Test-time compute represents fundamental shift in inference strategy. Allocate compute adaptively at test time, improving quality on hard examples. Keep easy examples fast. This trend continues: more test-time, less training-time compute. Future models will be designed around this paradigm.
Integration with RAG: test-time compute can improve both retrieval and generation stages. At retrieval time, allocate computation to find diverse documents. At generation time, allocate computation to synthesize high-quality responses. A well-optimized RAG pipeline uses test-time compute at both stages to maximize quality within latency budgets. This combined approach achieves state-of-the-art results on knowledge-intensive tasks and benchmarks.
Key takeaway: the value of this approach compounds over time. In month one, the benefits might be marginal. In month six, dramatically apparent. In year two, transformative. This is why patience and persistence matter in technical implementation. Build strong foundations, invest in quality, and let the benefits accumulate. The teams that master these techniques gain compounding advantages over competitors. Start today, measure continuously, optimize based on data. Success follows from consistent execution of fundamentals.
Practical implementation: start with best-of-N sampling on your hardest queries. Measure quality improvement and latency overhead. If latency is acceptable, expand to more queries. If not, try majority voting or adaptive allocation. Use confidence thresholds to trigger test-time compute only when needed. Build incrementally and measure at each step to avoid over-investing in expensive inference.