"How do you manage memory in an LLM application when conversations get long?"

Deliver all five beats below in under 90 seconds. Missing one signals you know the concept but haven't designed a system around it — an interviewer will probe exactly the beat you skipped.

Miss one and they probe.

1. The constraint — the context window is fixed size; history grows unbounded if unmanaged
2. Three strategies — in-context buffer → summarisation (~4K tokens) → external retrieval (hours/multi-session)
3. Lost in the middle — design summaries to preserve key decisions explicitly, not just compress chronologically
4. Decision matrix — match the strategy to session length, not the fanciest option by default
5. Cost — every extra token is money; memory design is a cost decision, not just UX

"The context window is a fixed size I can't grow — everything I send must fit, and history grows unbounded if I don't manage it. I start with an in-context buffer for short sessions — zero setup. At around 4K tokens I add summarisation — one extra LLM call compresses old turns while I keep the summary verbatim. For sessions spanning hours or multiple days, I use external retrieval — vector-stored history, retrieve what's relevant per turn. Because models attend poorly to the middle of long contexts, I design summaries to preserve key decisions explicitly, not just compress chronologically. And because every extra token costs money on every call, I monitor context size and set a summarisation trigger before cost per turn crosses my unit economics threshold."

Your version doesn't have to match word-for-word — all five beats must be present.

• "How do you manage memory in an LLM application?" — core system-design question
• "What's the trade-off between summarisation and retrieval-based memory?" — near-guaranteed follow-up
• "What happens when a context window fills?" — tests whether you've actually hit this in production
• Naming the cost angle (not just the mechanism) separates senior from junior answers

• Chatbot "forgets" a user mid-session → in-context buffer overflowed, oldest turns silently truncated
• Spiralling API costs on long sessions → no summarisation trigger, turn 50 pays for 49 prior turns
• Support bot re-asks already-answered questions → truncation, not a model bug
• A product that never feels "smart" across visits → missing semantic memory entirely

Every LLM call sees only what's in that call's context window — nothing persists automatically between calls. If you don't deliberately manage conversation history, it either grows unbounded (cost explosion) or silently truncates (the model "forgets" and the user never gets told why).

This is the single biggest interview trap in this session. A 200K context costs roughly 40× more per call than a 4K context when full, and "lost in the middle" gets worse, not better, as context grows. Growing the container doesn't fix retrieval quality inside it — that's why memory strategy, not window size, is the real engineering decision.

The in-context buffer is the simplest strategy: append full history to every call. Zero setup, perfect recall — until the window fills. Truncation is what happens then: the oldest messages are silently dropped; the model never announces it, it just acts as if those turns never existed.

Like a whiteboard that holds a fixed number of sticky notes — when it fills, the oldest notes fall off the bottom, unannounced.

Even when everything fits in the context window, models attend more reliably to the beginning and end of a long context than to the middle — content in the middle can be effectively ignored.

Like skimming a long email: you remember the opening ask and the closing sign-off clearly, but the third paragraph blurs.

Compresses old turns into a compact summary once history crosses a token threshold (typically ~4K tokens), then replaces those turns with the summary. A good summary preserves: the user's goal, key decisions, outstanding items, and persistent user facts.

Like meeting minutes: you keep the decisions made, not the full transcript of who said what.

A failure mode where a harmful or incorrect instruction entered early in a conversation gets preserved in summaries and propagates through every subsequent turn. Prevented by keeping system rules in the system-message layer only — never letting raw user-turn content reach it.

Like a typo in the founding document of an organization that keeps getting copied into every later policy — the fix is separating the charter from the meeting notes.

Stores conversation history in a vector database (the RAG pattern from Session 03) and retrieves only the most relevant turns per new message. Enables effectively unlimited conversation length; adds retrieval latency per turn.

Like an archive room instead of a desk drawer — you can store years of history, but every lookup takes a trip to fetch the right folder.

Episodic memory: what happened in the current conversation — resets at session end. Semantic memory: persistent facts about the user (preferred language, plan tier) that survive across sessions, injected as system context at session start.

Most chatbots only implement episodic memory — the ones that feel genuinely "smart" across visits have semantic memory too.

The compounding cost problem of a raw in-context buffer: turn 50 sends all 49 previous turns as input, costing roughly 25× what turn 2 costs. Summarisation or retrieval is required before sessions reach this scale in production.

Like a group email chain where everyone keeps hitting "reply all" with full quote history — by message 50 you're re-sending the whole conversation every time.

A retrieval query for external memory built from the rolling summary plus the last 2–3 turns, not just the latest message. Needed because vague follow-ups ("show me something different") carry no retrieval signal on their own.

Like asking a colleague "the other one" mid-conversation — meaningless without the shared context of what you were just discussing.

Self-assessment, not graded. Answer before checking your notes.

1. A chatbot that handles billing support suddenly starts asking users to describe their issue — even though the user described it in detail 10 messages earlier. What is the most likely root cause?

• The model has a bug that causes random memory lapses
• The in-context buffer overflowed — the first messages were truncated when the context window filled
• The API connection was reset between turns, clearing history
• Temperature was set too high, causing the model to ignore previous context

2. "Lost in the middle" refers to which specific behaviour of LLMs?

• The model's inability to recall names mentioned earlier in a conversation
• LLMs attend poorly to content in the middle of a long context — recalling the beginning and end more reliably than the middle
• Context window overflow that loses the most recent messages
• A summarisation error that drops every other turn from the history

Self-assessment, not graded.

3. You are building a customer support chatbot. Sessions typically run 15–25 turns. Which memory strategy is the best starting point?

• External retrieval (vector database) — to handle the longest possible sessions
• No memory — use a fresh context on every turn to keep cost low
• In-context buffer — zero setup cost, works well for sessions of this length
• Summarisation memory triggered immediately on turn 2

4. What is context poisoning?

• A vulnerability where malicious documents injected via RAG contain false information
• A failure mode where a harmful or incorrect instruction entered early in a conversation propagates through all subsequent turns, including summaries
• When the context window is filled with redundant tokens, reducing effective attention on important content
• A prompt injection attack that overwrites the system prompt

Self-assessment, not graded.

5. What is the primary advantage of summarisation memory over an in-context buffer?

• It improves model accuracy by giving the model a cleaner, more structured history
• It prevents the context window from filling by compressing old turns — reducing token cost per call and extending effective conversation length
• It eliminates the possibility of "lost in the middle" by removing the middle of the conversation
• It allows the model to access information from previous sessions

6. A software architect says: "If my chatbot needs a longer memory, I'll just use a model with a 200K context window." What is the strongest objection to this answer?

• 200K context models don't exist yet
• Larger context windows improve the "lost in the middle" problem, making this a good solution
• 200K context costs roughly 40× more per call when full, and "lost in the middle" worsens with longer contexts — not improves
• The context window limit is per-turn, not per-session, so window size doesn't affect memory

Self-assessment, not graded.

7. What is the difference between episodic memory and semantic memory in LLM applications?

• Episodic memory stores factual knowledge; semantic memory stores conversation transcripts
• Episodic memory is what happened in the current conversation; semantic memory is persistent facts about the user that survive across sessions
• Episodic memory is implemented with a vector database; semantic memory uses a summarisation approach
• They are equivalent terms — "episodic" and "semantic" both refer to conversation history

8. Interview-style: a user just sent "Actually, can you show me something different?" with no other context. What's the best retrieval-query strategy for external memory, and why does the raw latest message fail here? Answer in 2 sentences.

≈45 min · the core mechanic of this session

Goal: extend your Session 03 RAG system so it remembers what it has already shown you, without re-sending the full history every turn.

Add a shown_patterns tracker and a summarisation trigger (e.g. every 4 turns): summarise the conversation so far in under 150 tokens, covering the user's goal, what's been shown, stated preferences, and open questions — then replace older turns with that summary. Run a 6-message test sequence ending in "Show me another one" — it should not repeat a pattern already shown, and "What have we covered so far?" should draw from the summary, not full history.

Write down: whether the assistant successfully avoided repeating a shown pattern, and what your summary included.

≈20 min · makes the cost decision visible, not theoretical

Goal: see the token-cost blowout and the summarisation savings directly, instead of taking the concept on faith.

Add token counting to your chat function so every turn logs input/output tokens, and whether summarisation is currently active. Re-run the same 6+ turn test sequence from Assignment 1 and watch the count grow, then drop sharply once summarisation triggers.

Write down: the token count right before and right after summarisation kicked in, and at what turn you'd set the trigger in a real production system with your own cost tolerance.