Advanced Reasoning in LLMs

Scratchpad Training and Process Reward Models

Instead of prompting for CoT, train the model to produce reasoning as part of its learned output distribution. OpenAI's o1 approach: train on long CoT traces, then hide the reasoning scratchpad from users at inference time.

This shifts reasoning from a prompting trick to a learned behavior. The model learns when to reason, how deep to reason, and how to validate its own work.

Two reward modeling approaches exist: Outcome Reward Model (ORM) scores only the final answer — correct or incorrect. Simple to label. But it can't distinguish between wrong-answer steps that luckily arrive at the right conclusion.

Process Reward Model (PRM) annotates every reasoning step, scoring each independently. Much more expensive to label (each solution requires dozens of step annotations). But dramatically better for hard math — it teaches the model which intermediate reasoning paths are sound, not just whether the final answer is right.

PRMs represent the frontier of reasoning supervision. DeepSeek's o1 successor and OpenAI's research papers hint at using PRMs to unlock significantly higher accuracy on competition-level mathematics and formal proofs.

Tree Search at Inference Time

Instead of one left-to-right reasoning chain, explore a tree of possible reasoning paths. Different intermediate steps lead to different continuations. Some paths are correct; some are dead ends.

Best-of-N (BoN): Generate N independent solutions, run each through a verifier (e.g., "does the code run?", "does the math check out?"), pick the highest-scored. Simple, embarrassingly parallel, but expensive — N=16 means 16× compute.

Beam search over reasoning steps: Maintain a fixed set of k "active" partial chains. At each reasoning step, expand each chain, score all branches, prune to top k. More efficient than BoN but harder to parallelize within a single request.

Monte Carlo Tree Search (MCTS): Sample reasoning paths, backpropagate outcome rewards through the tree structure. Focuses compute on promising branches. Used in AlphaCode 2, rStar-Math, and other competition systems. Highest quality but requires 50–200× compute.

DeepSeek-R1 and GRPO

DeepSeek-R1 demonstrated a profound finding: you can train reasoning capability via pure reinforcement learning, without supervised CoT traces. Start from a base model. Apply a simple RL algorithm. Reasoning emerges.

GRPO (Group Relative Policy Optimization): Sample a group of G responses. Score each with a rule-based reward (e.g., 1 if final answer correct). Compute advantage for each response relative to the group mean, not global baseline. Update policy to increase likelihood of high-advantage responses.

GRPO avoids reward model collapse seen in PPO — where the reward model starts overfitting and giving inflated scores. It's simpler: no separate reward model, no reference model, just group-relative advantage. And empirically, it works: DeepSeek-R1 shows reasoning ability without ever seeing annotated CoT data during training.

A key emergent phenomenon: the model develops an "aha moment" behavior — spontaneously re-examining its work mid-chain when it notices a mistake, then correcting course. This self-correction emerges purely from the RL signal, not from explicit training.

from openai import OpenAI
from pydantic import BaseModel

client = OpenAI()

class ReasoningTrace(BaseModel):
    thinking: str
    answer: str
    confidence: float

COT_SYSTEM = """You are a careful reasoner. For every question:
1. Think step-by-step in the 'thinking' field
2. Only then give your final 'answer'  
3. Rate your 'confidence' from 0.0 to 1.0

Show your work explicitly — do not skip steps."""

def cot_solve(problem: str) -> ReasoningTrace:
    result = client.beta.chat.completions.parse(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": COT_SYSTEM},
            {"role": "user", "content": problem}
        ],
        response_format=ReasoningTrace,
        temperature=0.0
    )
    return result.choices[0].message.parsed

# Example: multi-step math problem
trace = cot_solve(
    "A train leaves city A at 60mph. Another leaves city B "
    "(300 miles away) at 40mph heading toward A. When do they meet?"
)
print(f"Thinking: {trace.thinking[:200]}")
print(f"Answer: {trace.answer}")
print(f"Confidence: {trace.confidence:.0%}")

# Process Reward Model pattern: score each reasoning step
def score_reasoning_steps(steps: list[str], question: str) -> list[float]:
    """PRM: score each intermediate reasoning step for correctness."""
    scores = []
    for i, step in enumerate(steps):
        prompt = (f"Q: {question}
Reasoning so far:
"
                  + "
".join(steps[:i+1])
                  + "
Is this step correct and helpful? Score 0.0-1.0:")
        resp = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
            max_tokens=5, temperature=0.0
        ).choices[0].message.content.strip()
        try: scores.append(float(resp))
        except: scores.append(0.5)
    return scores