Conversation Buffer

Why agents forget

LLMs are stateless. Each API call is independent — the model has no inherent memory of what you said last turn. Every message you've exchanged must be re-sent in the messages array for the model to "remember" it. This is short-term memory: it lives in the prompt, and it's bounded by the context window.

For a simple chatbot, you just append every turn to a list and send the whole list each time. This works until the conversation grows too long and you exceed the context window (or costs become prohibitive). The conversation buffer pattern manages this problem — deciding which past messages to include and how to compress or discard the rest.

The sliding window approach

The simplest solution: keep only the last N turns. Old messages are dropped entirely. This is fast, cheap, and requires no extra LLM calls.

from collections import deque

class SlidingWindowMemory:
    def __init__(self, max_turns: int = 10):
        self.history = deque(maxlen=max_turns * 2)  # * 2 for user+assistant pairs
        self.system_prompt = ""

    def add(self, role: str, content: str):
        self.history.append({"role": role, "content": content})

    def get_messages(self) -> list[dict]:
        return list(self.history)

memory = SlidingWindowMemory(max_turns=5)

def chat(user_message: str) -> str:
    import anthropic
    memory.add("user", user_message)
    response = anthropic.Anthropic().messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=512,
        system="You are a helpful assistant.",
        messages=memory.get_messages()
    )
    reply = response.content[0].text
    memory.add("assistant", reply)
    return reply

# Test
chat("My name is Deepak and I'm building an AI mindmap.")
chat("What was the project I mentioned?")  # Model still knows

The downside: context before the window is completely lost. If the user mentioned their name 15 turns ago and you only keep 10 turns, the model won't know their name.

Summary compression

When the conversation exceeds a threshold, use an LLM to compress old turns into a summary paragraph. The summary is prepended as a "system-level" context, and the window continues from there:

Full history (20 turns):
[Turn 1-10] → "Earlier in this conversation: user introduced themselves
  as Deepak, a developer building a GenAI mindmap. They asked about RAG
  architecture and we discussed chunking strategies and embedding models."
[Turn 11-20] → kept verbatim in messages array

This preserves key facts from old turns while keeping the prompt manageable. The summary is a lossy compression — fine details are lost, but the narrative thread is maintained.

Implementing both strategies

import anthropic

client = anthropic.Anthropic()

class SummarisedMemory:
    def __init__(self, window_size: int = 8, compress_after: int = 16):
        self.recent: list[dict] = []
        self.summary: str = ""
        self.window_size = window_size
        self.compress_after = compress_after

    def add(self, role: str, content: str):
        self.recent.append({"role": role, "content": content})
        if len(self.recent) > self.compress_after:
            self._compress()

    def _compress(self):
        # Summarise the oldest half of the window
        to_compress = self.recent[:self.compress_after // 2]
        self.recent = self.recent[self.compress_after // 2:]

        turns_text = "
".join(
            f"{m['role'].title()}: {m['content']}" for m in to_compress
        )
        if self.summary:
            turns_text = f"Previous summary: {self.summary}

{turns_text}"

        resp = client.messages.create(
            model="claude-haiku-4-5-20251001",
            max_tokens=256,
            messages=[{
                "role": "user",
                "content": f"Summarise this conversation in 2-3 sentences, "
                           f"preserving key facts and names:

{turns_text}"
            }]
        )
        self.summary = resp.content[0].text

    def get_messages(self) -> list[dict]:
        return self.recent[-self.window_size * 2:]

    def get_system(self, base_system: str) -> str:
        if self.summary:
            return f"{base_system}

Conversation so far: {self.summary}"
        return base_system

LangChain memory classes

LangChain provides pre-built memory implementations so you don't have to write this yourself:

from langchain.memory import (
    ConversationBufferMemory,         # keep all turns
    ConversationBufferWindowMemory,   # sliding window
    ConversationSummaryMemory,        # full summary compression
    ConversationSummaryBufferMemory,  # hybrid: summary + recent window
)
from langchain_anthropic import ChatAnthropic
from langchain.chains import ConversationChain

llm = ChatAnthropic(model="claude-haiku-4-5-20251001")

# Hybrid memory: summarise turns beyond token_limit, keep recent ones verbatim
memory = ConversationSummaryBufferMemory(
    llm=llm,
    max_token_limit=1000,
)

chain = ConversationChain(llm=llm, memory=memory, verbose=True)
chain.predict(input="Hi, I'm Deepak and I'm building a GenAI mindmap.")
chain.predict(input="What project did I mention?")  # still knows

ConversationSummaryBufferMemory is the most practical for production: it keeps recent turns verbatim (fast, faithful) and summarises older ones (compact). Set max_token_limit to roughly 20-30% of your model's context window.

When to upgrade to long-term memory

Conversation buffer (short-term memory) is the right choice when: you're building a chatbot or assistant within a single session, the relevant context fits within a few thousand tokens, and you don't need to remember facts across separate conversations.

You need long-term memory (vector store + retrieval) when: users return across multiple sessions, you need to remember facts from weeks or months ago, or the volume of information is too large for even summary compression (e.g., an agent that processes hundreds of documents over time).

A practical upgrade path: start with ConversationSummaryBufferMemory for per-session context, and add a vector store lookup at the start of each new session to retrieve relevant facts from past conversations. The two systems are complementary, not competing.

Gotchas

Token counting matters. You can't just count messages — count tokens. A single long user message might be worth more than five short turns. Use anthropic.count_tokens() or tiktoken to track actual token usage before building your memory window.

Compression is lossy — and the model can't tell. After summarisation, the model doesn't know what was compressed. If a user references a detail from a compressed turn ("like I said about the edge case in my first question"), the model will have lost that detail. Consider including a note: "earlier turns summarised — ask me if you need specifics".

System messages are not magic memory. Prepending summaries to the system prompt works but can "feel" lower quality to the model than actual conversation history in the messages array. For anthropic models, the conversation history format (user/assistant alternation) is the most natural way to provide context.