--- title: "Context Window Manager" description: "Interviewers ask 'how do you manage memory in an LLM application when conversations get long?' by default for any AI Engineer role — this project is where you..." source: "https://confidentprep.com/paths/ai-engineering-fundamentals/context-window-manager/" path: "AI Engineering Fundamentals" repository: "https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/context-window-manager" token_estimate: 5486 --- # Context Window Manager ## Overview Every LLM has no memory of its own — it sees only what you send it on this call, and nothing else. The moment a conversation runs past a handful of turns, you have to decide how to manage that history yourself, or the assistant either forgets things it was just told or the cost of every call quietly balloons. This project is where you design and build that decision, not just recite it in an interview. ### What you're building `context-window-manager` extends a retrieval-backed assistant (Project 3's territory) with a memory layer that decides, on every turn, whether to keep appending to a simple in-context buffer or to compress older turns into a structured summary — one that explicitly tracks the user's goal, key decisions, open items, and stated preferences, not just a narrative recap. The same system also draws the line between memory that's scoped to this one conversation (episodic) and facts that should persist across sessions (semantic) — most assistants only ever build the first kind, which is exactly why the second kind is what makes an assistant feel like it actually knows the user. ### How a turn is handled ![Context-window-manager memory strategy flow: a new turn either appends to the in-context buffer or triggers summarization of older turns, then the system context is built from the summary and any retrieved data, the model is called, and any fact worth persisting across sessions is written to semantic memory](/diagrams/context-window-manager-memory-strategy-flow.svg) *The two decision points — "over the summarization threshold?" and "should this fact persist across sessions?" — are exactly the two places a naive chat loop either quietly forgets something or quietly gets expensive.* ### Core concepts, three levels deep #### 1. Context window and the in-context buffer - **Definition:** the context window is the fixed per-call token budget — everything you send (system prompt, history, retrieved data, current message) has to fit inside it, on every single request. The in-context buffer is the simplest possible memory strategy: append the full conversation history to every call, with zero extra setup. - **In this project:** the buffer is your starting point and it works fine for short sessions — the failure mode only shows up once history grows past what fits, and it fails silently: the oldest turns get dropped with no signal to you or the model. - **Practical consequence:** never let the buffer grow unbounded and unmonitored. Decide a threshold before you ship, not after a user notices the assistant "forgot" something. #### 2. Summarisation memory and "lost in the middle" - **Definition:** summarisation memory compresses old turns into a compact structured summary once a threshold is crossed, then replaces those turns with the summary. "Lost in the middle" is a separate, related fact about attention: models recall the beginning and end of a long context more reliably than content buried in the middle — even when everything technically fits. - **In this project:** these two facts together are why the summary can't just be "a shorter version of the conversation." If it compresses chronologically instead of preserving specific decisions, the important fact can still get buried and effectively lost, the same way it would in an un-summarized but overly long context. - **Practical consequence:** design your summary format explicitly — user's goal, decisions made, open items, stated preferences — instead of asking the model for a generic recap. A summary that only captures "what was discussed" is not doing the job. #### 3. Context poisoning - **Definition:** a failure mode where a harmful or incorrect instruction that enters early in a conversation — through user input, not the system prompt — gets folded into a summary and then keeps propagating through every later turn, because nothing in the pipeline distinguishes it from legitimate conversation content. - **In this project:** your summarization call reads raw user-turn text and produces something that gets injected back into the system context on every future turn — that's exactly the path poisoning travels down if you're not careful about what's allowed to influence it. - **Practical consequence:** keep system-level rules in the system message layer only, and never let summarized user content get treated as an instruction. This has to be a code-level guarantee, not a prompt-level request. #### 4. Episodic vs. semantic memory - **Definition:** episodic memory is what happened in the current conversation — it resets at the end of the session and is what your in-context buffer and summarization both implement. Semantic memory is persistent facts about the user (stated goal, experience level, preferences) that survive across sessions, stored separately and injected as system context at the start of a new one. - **In this project:** the summarization layer only ever gives you episodic memory. Semantic memory is a deliberate, separate write — extracting a durable fact and saving it somewhere that outlives this conversation object. - **Practical consequence:** most chatbots only ever implement episodic memory, which is why they reset to a blank slate every session. The ones that feel like they "know you" are the ones that deliberately built the semantic layer on top — it doesn't happen automatically. ### Decision rules | If... | Then... | |---|---| | Sessions are short (under ~20 turns) or stateless/one-off | Use a plain in-context buffer — zero setup, and you don't need anything more sophisticated yet | | History is approaching a token threshold (start around 4K tokens / a handful of turns) | Trigger summarisation — compress older turns into a structured summary, keep the most recent turns verbatim | | A follow-up message is vague or reference-heavy ("show me something different") | Build the retrieval query from the rolling summary plus the last 2-3 verbatim turns, not the bare message alone | | A fact should be true of the user in every future session, not just this one | Write it to a separate semantic memory store and inject it as system context at session start — don't rely on episodic summarization to carry it forward | ### Common mistakes - **Letting the in-context buffer grow unmonitored** until it silently truncates instead of deciding a threshold and summarizing before you hit it. - **Summarizing chronologically instead of structurally** — a summary that just compresses "what happened" can still bury the one decision that mattered, the same way an unsummarized long context can. - **Letting raw user-turn text influence system-level behavior** through an unguarded summarization pass — this is exactly how context poisoning propagates. - **Treating "use a bigger context window" as a fix** for either forgetting or cost — it doesn't address lost-in-the-middle, and it makes every call more expensive, not less. ### Key concepts at a glance | Concept | One-line definition | Why it matters for `context-window-manager` | |---|---|---| | In-context buffer | Append full history to every call, zero setup | The simplest strategy, and the one that silently fails once it overflows | | Lost in the middle | Models attend well to the start/end of a long context, poorly to the middle | Why a summary has to preserve specific decisions, not just compress narratively | | Context poisoning | An early bad instruction propagates through every later summary | Why system rules must stay isolated from summarized user content | | Semantic memory | Persistent user facts injected at session start, stored separately from episodic history | The deliberate extra step that makes an assistant feel like it remembers you across sessions | ## What to do **Path:** 1 **Position:** 4 of 12 **Difficulty:** 🟢 Local — one API key, no Docker, no cloud **Estimated time:** 3–4 hours **AWS cost:** None --- ### Agent Pickup Instructions **This spec is self-contained. Build this project without reading any other spec file.** ```bash mkdir -p path-1/p1-04-context-window-manager cd path-1/p1-04-context-window-manager # Create folder structure, paste .env.example and requirements.txt from this spec # Implement src/ modules as described below # Verify: pytest tests/ -v python src/main.py # interactive conversation python src/main.py --strategy sliding # force sliding window python src/main.py --strategy summarise # force summarisation python src/main.py --compare replay.json # load saved session, compare both strategies ``` **Done when:** - [ ] `pytest tests/ -v` → minimum 5 tests, all green - [ ] Warning printed when conversation reaches 80% of context limit - [ ] `--strategy sliding` drops oldest messages visibly (log line shows what was dropped) - [ ] `--strategy summarise` compresses old messages into a summary turn - [ ] Conversation exports to JSON with `--save session.json` - [ ] `--compare` runs same session through both strategies and shows what each loses - [ ] No file in `src/` exceeds 200 lines --- ### What this project is A multi-turn CLI chatbot that manages its own context window. Most LLM applications ignore the context window until they crash with a "context too long" error in production. This project makes the problem visible and builds two real solutions: sliding window (drop oldest messages) and summarisation compression (summarise old messages into one). The learner runs the same conversation through both strategies and sees exactly what each approach loses — the core engineering tradeoff in stateful LLM applications. --- ### What the learner achieves "I built a multi-turn chatbot that handles context window limits with two strategies — sliding window and summarisation — and I can demonstrate in an interview exactly what each strategy loses and why production systems often combine both." --- ### Folder structure ``` p1-04-context-window-manager/ ├── README.md ├── GUIDE.md ├── .env.example ├── requirements.txt ├── src/ │ ├── main.py ← CLI entry point and conversation loop │ ├── llm.py ← provider-agnostic LLM wrapper │ ├── tokenizer.py ← token counting abstraction │ └── context.py ← context management strategies ├── tests/ │ └── test_context.py └── sessions/ ← created at runtime, gitignored ``` --- ### .env.example ```bash # LLM provider: anthropic | openai | ollama LLM_PROVIDER=anthropic LLM_API_KEY= # Model LLM_MODEL= # Context window size in tokens (set to match your model) # current Anthropic model: 200000 | current OpenAI model: 128000 | llama3.2: 8192 CONTEXT_LIMIT=8192 # Warn when conversation uses this fraction of the limit WARN_THRESHOLD=0.8 # Strategy to use when limit is approached: sliding | summarise CONTEXT_STRATEGY=sliding # For summarise strategy: summarise messages older than this many turns SUMMARISE_KEEP_RECENT=6 ``` --- ### requirements.txt ``` anthropic==0.40.0 openai==1.58.0 ollama==0.4.4 tiktoken==0.8.0 python-dotenv==1.0.1 ``` --- ### src/ — what to implement #### src/llm.py **`get_completion(messages: list[dict], system: str = "") -> str`** - Takes a list of `{"role": "user"|"assistant", "content": "..."}` dicts (not a single string prompt) - Routes to correct provider - `anthropic`: `client.messages.create(model=model, max_tokens=1024, system=system, messages=messages)` - `openai`: prepend `{"role": "system", "content": system}` if non-empty, then pass messages list - `ollama`: prepend system message if non-empty - Raises `ValueError` for unknown provider #### src/tokenizer.py **`count_tokens(messages: list[dict], model: str) -> int`** - Approximate token count for a list of message dicts - For openai/anthropic models: use `tiktoken.encoding_for_model()` if available, else `tiktoken.get_encoding("cl100k_base")` - For ollama: use cl100k_base as approximation (note: actual tokenizer differs) - Add 4 tokens per message for message formatting overhead - Return total int **`count_string_tokens(text: str, model: str) -> int`** - Count tokens in a single string using same approach #### src/context.py **`ContextManager` class** Constructor: `__init__(self, context_limit: int, warn_threshold: float, strategy: str, keep_recent: int)` **`add_message(self, role: str, content: str) -> None`** - Appends `{"role": role, "content": content}` to `self.messages` **`get_token_count(self, model: str) -> int`** - Returns token count of current `self.messages` **`is_approaching_limit(self, model: str) -> bool`** - Returns True if `get_token_count() / context_limit >= warn_threshold` **`apply_sliding_window(self, model: str) -> int`** - While `get_token_count() > context_limit * 0.7`: remove `self.messages[0]` - Log each removed message: `Dropped [user]: "{content[:50]}..."` to stderr - Return number of messages removed **`apply_summarise_compression(self, model: str) -> str`** - Take `self.messages[:-keep_recent]` (all but the most recent turns) - Call LLM with: "Summarise this conversation history into 2–3 sentences, preserving key facts and decisions:\n{messages}" - Replace those old messages with a single `{"role": "assistant", "content": "[Summary]: {summary}"}` message - Return the summary text (for logging) **`compress_if_needed(self, model: str) -> bool`** - If `get_token_count() > context_limit * 0.85`: - If strategy == "sliding": call `apply_sliding_window()` - If strategy == "summarise": call `apply_summarise_compression()` - Return True (compression applied) - Return False **`export(self, path: str) -> None`** - Write `{"messages": self.messages, "strategy": self.strategy}` as JSON **`load(cls, path: str) -> ContextManager`** (classmethod) - Load messages from exported JSON #### src/main.py CLI: `python src/main.py [--strategy sliding|summarise] [--save ] [--compare ]` **Interactive mode:** 1. Print `Strategy: {strategy} | Context limit: {limit} tokens` 2. Loop: `print("You:"); user_input = input("> ")` 3. Add user message to context manager 4. If approaching limit: `print(f"⚠ Context at {pct:.0%} of limit — {strategy} compression will apply")` 5. Call `context.compress_if_needed()` (after adding user message, before calling LLM) 6. Call `llm.get_completion(context.messages)` 7. Add assistant response to context manager 8. Print `Assistant: {response}` 9. Print `📊 Tokens used: {count}/{limit} ({pct:.0%})` 10. On `quit` or Ctrl-C: if `--save` provided, call `context.export(path)` **Compare mode (`--compare `):** - Load session from JSON - Replay messages through sliding window strategy, record all responses - Replay same messages through summarise strategy, record all responses - Print side-by-side comparison table: message N | sliding response | summarise response - Note: replay skips actual LLM calls for user turns, only replays to observe context state changes --- ### tests/ — what to test **File:** `tests/test_context.py` — no LLM calls for context/tokenizer tests; mock LLM for compression tests. **Test 1 — token counting increases with messages:** Create a `ContextManager`. Add 5 messages of known content. Assert `get_token_count()` is greater than 0 and increases after each add. **Test 2 — is_approaching_limit triggers correctly:** Create manager with `context_limit=100, warn_threshold=0.8`. Add messages until token count > 80. Assert `is_approaching_limit()` returns True. **Test 3 — sliding window removes oldest messages:** Add 10 messages. Call `apply_sliding_window()` until under limit. Assert messages removed from the front (oldest), not the back. **Test 4 — sliding window preserves most recent messages:** Add messages labeled "old-1" through "old-8" and "recent-1" through "recent-2". After sliding window, assert "recent-2" is still in `context.messages`. **Test 5 — export and load round-trips messages:** Create manager, add 3 messages, export to a temp file, load into new manager. Assert loaded messages match original. **Test 6 — count_tokens is consistent:** Call `count_tokens` twice with same input. Assert same result (deterministic). --- ### README.md content ```markdown # Context Window Manager A multi-turn CLI chatbot that handles context window limits with two strategies — sliding window and summarisation compression — and shows you exactly what each approach loses. ## Setup ```bash cd p1-04-context-window-manager cp .env.example .env ## Edit .env: set LLM_PROVIDER, LLM_API_KEY, and CONTEXT_LIMIT (match your model) pip install -r requirements.txt ``` ## Run ```bash ## Interactive chat with default strategy python src/main.py ## Force a specific strategy python src/main.py --strategy summarise ## Save your session python src/main.py --save sessions/my_session.json ## Compare both strategies on a saved session python src/main.py --compare sessions/my_session.json ``` Expected output mid-conversation: ``` You: > Tell me about the new auth flow we discussed ⚠ Context at 83% of limit — sliding compression will apply Dropped [user]: "What are the main API endpoints?..." Dropped [assistant]: "The main endpoints are /auth/login, /auth..." ``` ## Tests ```bash pytest tests/ -v ``` Tests verify token counting, sliding window removal order, and session export/import. No API key required. ## What to try next - Set `CONTEXT_LIMIT=2000` and have a long conversation to see compression trigger quickly - Export a session and run `--compare` to see what sliding vs summarise each forgets - Add a third strategy: keep only user messages (drop all assistant turns) ``` --- ### GUIDE.md content ```markdown # Build guide: Context Window Manager ## What you're building and why it matters Every production LLM application that has multi-turn conversations eventually hits the context window limit. At 2am on-call, this shows up as a cryptic API error about token limits. The fix is never "just increase the limit" — even a 200k context window fills up in long sessions, and at $15 per million tokens you don't want to send the entire conversation history every turn. Context management is the unsexy infrastructure that makes conversational AI work at scale. ## The decision that matters in this build **Sliding window vs summarisation.** Sliding window is simple and deterministic: drop the oldest messages. It always works, never calls the LLM, and costs nothing. Its weakness: you permanently lose the context of early conversation. Summarisation compresses old turns into a summary. It preserves semantic content at the cost of an extra LLM call. In production, most systems combine both: summarise every N turns, then slide the window if the summary itself gets too long. Build both separately here so the tradeoff is concrete, not theoretical. ## What will break **Token counting is an approximation.** tiktoken counts tokens for OpenAI models exactly. For Claude and Ollama, you're estimating with a compatible tokenizer. Set your `CONTEXT_LIMIT` to 80% of the actual model limit to leave room for the approximation error. If you set it to the exact limit and the approximation is off, you'll get an API error. **Summarisation doubles your LLM calls.** Every time compression triggers, you make an extra LLM call to generate the summary. In a long session, this can add 5–10% to your total cost. Log when compression fires so you can see how often it is actually needed. **Sliding window can drop important context.** If the user sets up important constraints in the first two turns ("always respond in French", "my name is Alex") and those turns get dropped, the assistant forgets. Production systems handle this with a "sticky context" — a separate block of critical instructions that never gets dropped. Consider adding this as an extension. ## How to talk about this in an interview "I built a context manager with two compression strategies. Sliding window is zero-cost but loses history; summarisation preserves semantics but costs an extra LLM call. I measured that in a 20-turn session, sliding window dropped 60% of the conversation by turn 15, while summarisation preserved the key facts in a 3-sentence summary. The right production choice depends on how stateful the conversation needs to be." ``` --- ### Failure modes to handle | Failure | Where | How to handle | |---------|-------|---------------| | Unknown strategy | `context.compress_if_needed()` | `raise ValueError(f"Unknown CONTEXT_STRATEGY={strategy}. Use sliding or summarise")` | | Summarisation LLM call fails | `apply_summarise_compression()` | Log warning, fall back to sliding window for this compression cycle | | Session file not found | `ContextManager.load()` | `raise FileNotFoundError(f"Session file not found: {path}")` | | CONTEXT_LIMIT not set | `context.py` | Default to 8192 and print warning: "CONTEXT_LIMIT not set — defaulting to 8192 tokens" | | Messages list empty | `apply_sliding_window()` | Return 0 immediately (nothing to drop) | --- ### The metric this project measures **Token utilisation percentage** — printed after every assistant response. Format: `📊 Tokens used: {count}/{limit} ({pct:.0%})` **Compression events** — count how many times each strategy fires per session. Printed on exit: `Session ended: sliding window fired 3 times | summarised 0 times` ### Model source currency - Anthropic model list: https://docs.anthropic.com/en/docs/about-claude/models/overview - OpenAI model list: https://platform.openai.com/docs/models - verifiedOn: 2026-08-07 - `LLM_MODEL` is required and has no implicit default; choose a supported model for the configured provider at setup time. ## Source code A reference implementation of Context Window Manager — README, guide, source and tests — is at https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/context-window-manager.