Every LLM application that runs longer than one turn hits the same wall: the model has no memory. It isn’t storing anything between calls. It sees exactly what you send it — nothing more — and the moment a conversation grows past a handful of turns, that constraint stops being theoretical and starts being a bug report.
The instinct is to reach for a bigger context window. That instinct is wrong, and it’s worth understanding precisely why before you design anything.
The constraint: a fixed-size box, not an elastic one
The context window is the maximum amount of text an LLM can process in a single call. Everything has to fit inside it — system prompt, conversation history, retrieved documents, the current message. Think of it as a whiteboard with a fixed number of sticky notes: once it’s full, the oldest notes fall off to make room for new ones, and the model never tells you that happened. It just behaves as if those earlier turns never existed.
“Just use a bigger window” fails for two separate reasons, and interviewers ask about both:
Cost. Every extra token in the window costs money on every single call. A 200K-token context costs roughly 40x more per call than a 4K one when it’s full. If your history keeps growing and you keep resending all of it, cost grows with it — unboundedly.
Attention, not just capacity. Even when everything technically fits, LLMs attend more reliably to the beginning and end of a long context than to the middle. This is the “lost in the middle” problem, and it gets worse — not better — as the window grows. A bigger box doesn’t fix a model’s tendency to lose track of what’s in the middle of that box; it just gives you more room to lose things in.
Growing the container was never the fix. Managing what goes into it is.
Three strategies, and the order you actually reach for them
1. In-context buffer. The simplest approach: append the full conversation history to every call. Zero setup cost, perfect recall, and it works fine for short sessions or stateless prototypes. The problem shows up as the conversation grows — turn 50 resends 49 prior turns as input, which is roughly 25x the token cost of turn 2. That’s not a rounding error at scale; it’s the difference between a sustainable per-conversation cost and one that quietly blows your unit economics.
2. Summarization memory. Once history crosses a token threshold, compress the older turns into a compact summary and drop the raw text. Think meeting minutes instead of a transcript — you keep the decisions, not the play-by-play. This is where most production chat applications should land once sessions run past a few thousand tokens.
The part teams get wrong here: a summary that only compresses chronologically will lose exactly the facts that “lost in the middle” already put at risk. A summary that survives contact with a real conversation has to explicitly carry four things forward, every time it’s regenerated — the user’s stated goal, key decisions made, outstanding items still open, and any persistent facts about the user (name, plan tier, stated preferences). Skip any one of those and the model will re-ask a question it already had the answer to, three turns after you thought the problem was solved.
3. External retrieval. For sessions that span hours, days, or multiple visits, store history in a vector database and retrieve only what’s relevant to the current turn — the same retrieval pattern behind RAG, covered here, applied to a conversation instead of a document set. This buys unlimited effective history at the cost of retrieval latency and a genuinely tricky problem: a follow-up like “show me another one” carries almost no retrieval signal on its own. A working implementation has to build a composite query — the rolling summary plus the last two or three verbatim turns — because the latest message alone is usually just pronouns.
Start with the buffer. Add summarization once cost or coherence starts to break down. Reach for external retrieval only when the session genuinely needs to survive across days or logins — not by default, and not because it sounds more sophisticated.
Memory design is a cost decision before it’s a UX decision
It’s tempting to treat memory as a product-feel problem — does the assistant seem coherent, does it remember what the user said. It is that, but the trigger for when you add summarization or retrieval should be a number, not a vibe: track token cost per call, and set your summarization threshold before the cost of sending full history on every turn exceeds the cost of one extra summarization call. Skip that step and you’ll find out the hard way, on an invoice, weeks after a “small” feature quietly stopped being small.
Episodic memory and semantic memory aren’t the same thing
Most chatbots implement one kind of memory: episodic — what happened in this conversation, gone the moment the session ends. Fewer implement semantic memory — persistent facts about a specific user (preferred language, account tier, stated preferences) that survive across sessions and get injected as context the next time that user shows up. The assistants that feel unusually sharp almost always have both. The ones that feel forgettable are missing the second one entirely, not the first.
What this looks like when you build it yourself
Reading the three strategies above is different from having built the trigger logic, watched a naive summarizer silently drop a fact on its second compression cycle, and fixed it by making the summarizer fold its own prior output back in instead of overwriting it. That failure mode — a summary that only reads the current window and forgets what it already knew — is exactly the “lost in the middle” problem showing up in your own code instead of in a paper about attention.
Session 4 of the AI Engineering interview-prep course walks through building that trigger logic end to end — token counting, the summarization threshold, and the four fields a summary has to preserve — on top of the retrieval system from the session before it, so the memory layer has an actual RAG assistant to sit on top of instead of a toy example.