--- title: "Full AI Assistant (Capstone)" description: "This is your portfolio centerpiece for junior-to-mid AI Engineer roles — a single deployed system you can walk an interviewer through end to end, with an eval..." source: "https://confidentprep.com/paths/ai-engineering-fundamentals/ai-assistant-capstone/" path: "AI Engineering Fundamentals" repository: "https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/ai-assistant-capstone" token_estimate: 4268 --- # Full AI Assistant (Capstone) ## Overview A deployed AI assistant that combines RAG over uploaded documents, tool use for live data, context window management, and streaming responses — with an eval suite that runs against the live endpoint, not just locally. Nothing new conceptually; the whole point is proving you can integrate everything from the 11 projects before this into one coherent, deployed system instead of 11 disconnected demos. ## What to do **Path:** 1 **Position:** 12 of 12 **Difficulty:** 🔴 Requires AWS account **Estimated time:** 4–6 hours (assembly, not new concepts) **AWS cost:** ~$3–5 for a full session. Teardown script included. --- ### Agent Pickup Instructions **This spec is self-contained. Build this project without reading any other spec file. All component implementations are described inline — do not assume prior projects exist.** ```bash mkdir -p path-1/p1-12-capstone cd path-1/p1-12-capstone # Local development docker compose up curl http://localhost:8000/health # Run eval suite against local instance python scripts/run_eval.py --base-url http://localhost:8000 # Deploy to Lambda bash scripts/deploy.sh # Run eval against live deployment python scripts/run_eval.py --base-url $API_URL # Teardown bash scripts/teardown.sh ``` **Done when:** - [ ] All 5 API endpoints respond correctly locally - [ ] Eval suite runs with minimum 10 test cases and produces a JSON report - [ ] Architecture diagram renders in `docs/architecture.md` (Mermaid) - [ ] Retrieval hit rate metric tracked in `/metrics` endpoint - [ ] Lambda deployment works (inherits from p1-11 pattern) - [ ] `bash scripts/teardown.sh` deletes all AWS resources - [ ] No file in `src/` exceeds 200 lines --- ### What this project is Wires document chat, RAG retrieval, context window management, tool calling, streaming API, and prompt evaluation into one deployed assistant. The system is more than the sum of its parts: a user can upload documents, ask questions that retrieve from those documents via RAG, and the assistant uses tools for live data. An eval suite runs against the live deployment to verify the system works end-to-end. This is the portfolio centrepiece. --- ### What the learner achieves "I built and deployed a full AI assistant that combines RAG retrieval over uploaded documents, tool use for live data, context window management, and streaming responses — with a live eval suite that measures retrieval hit rate and runs against the deployed Lambda endpoint." --- ### Folder structure ``` p1-12-capstone/ ├── README.md ├── GUIDE.md ├── docs/ │ └── architecture.md ← Mermaid diagram ├── .env.example ├── requirements.txt ├── Dockerfile ← Lambda-compatible ├── docker-compose.yml ← local dev with SQLite volume ├── src/ │ ├── main.py ← FastAPI app with all endpoints │ ├── llm.py ← streaming + non-streaming LLM wrapper │ ├── rag.py ← RAG pipeline (embed, index, retrieve) │ ├── tools.py ← tool registry (calculator, datetime, wikipedia) │ ├── context.py ← context window manager │ ├── database.py ← SQLite: requests, documents, eval runs │ └── config.py ← all env vars ├── scripts/ │ ├── deploy.sh ← Lambda deploy (same pattern as p1-11) │ ├── teardown.sh │ └── run_eval.py ← eval suite runner ├── eval/ │ └── suite.yaml ← 10+ test cases covering RAG, tools, multi-turn └── tests/ └── test_integration.py ``` --- ### .env.example ```bash LLM_PROVIDER=anthropic LLM_API_KEY= LLM_MODEL= EMBEDDING_MODEL=all-MiniLM-L6-v2 DATABASE_PATH=/data/assistant.db INDEX_PATH=/data/rag_index RATE_LIMIT_REQUESTS_PER_MINUTE=20 COST_PER_1M_INPUT_TOKENS=1.00 COST_PER_1M_OUTPUT_TOKENS=5.00 MAX_CONTEXT_TOKENS=8192 CONTEXT_STRATEGY=sliding # AWS (for deployment only) AWS_REGION=us-east-1 SECRET_NAME=capstone-assistant/llm-api-key ``` --- ### requirements.txt ``` anthropic==0.40.0 openai==1.58.0 fastapi==0.115.6 uvicorn==0.34.0 sse-starlette==2.2.1 mangum==0.19.0 sentence-transformers==3.3.1 faiss-cpu==1.9.0 tiktoken==0.8.0 pdfplumber==0.11.4 requests==2.32.3 boto3==1.35.93 pyyaml==6.0.2 python-dotenv==1.0.1 ``` --- ### API Endpoints #### `GET /health` ```json {"status": "ok", "model": "", "documents_indexed": 3, "index_size": 47} ``` #### `POST /documents` — upload a document to the RAG index Body: `multipart/form-data` with `file` field (PDF or TXT). - Save to `/tmp/uploads/`, chunk and embed, add to FAISS index - Return: `{"document_id": "uuid", "chunks_added": 12, "total_chunks": 47}` #### `POST /chat` — streaming chat with RAG + tools + context management Body: `{"message": str, "session_id": str (optional), "use_rag": bool (default true)}` - Load session context from DB (last 20 messages for this session_id) - If `use_rag`: retrieve top-3 chunks for the message, inject into system prompt - Build system prompt: base instructions + RAG context + tool definitions - Check rate limit per IP - Stream response via SSE using tool-calling loop: - If LLM calls a tool: execute, inject result, continue loop - If LLM gives final answer: stream it as SSE - Save user message and assistant response to DB for session_id - Track retrieval_hit (did RAG chunks actually get referenced in the response?) - Log request to DB: tokens, cost, latency, retrieval_hit #### `POST /eval` — run eval suite against the live system Body: `{"suite_yaml": str (YAML content), "session_id": str (optional, fresh session per run)}` - Parse YAML, run each test case against `/chat` - Judge each response with LLM-as-judge - Return: `{"suite_name": ..., "results": [...], "summary": {"passed": N, "total": N, "retrieval_hit_rate": 0.7}}` #### `GET /metrics` ```json { "total_requests": 42, "total_cost_usd": 0.031, "p50_latency_ms": 1800, "p95_latency_ms": 4200, "retrieval_hit_rate": 0.73, "tool_use_rate": 0.21, "error_rate": 0.02 } ``` --- ### src/ — what to implement #### src/llm.py - `stream_completion(messages, system, tools=None)` → `Iterator[(token, prompt_tokens, completion_tokens)]` - `get_completion(messages, system, tools=None)` → `{"content": str, "tool_call": dict|None}` - Standard provider routing: anthropic, openai (ollama not supported in Lambda deployment) #### src/rag.py - `load_or_create_index(index_path: str) -> tuple[faiss.Index, list[dict]]` - `add_document(file_path: str, index, metadata, model: SentenceTransformer) -> int` — returns chunks added - `retrieve(query: str, index, metadata, model, top_k=3, threshold=0.3) -> list[dict]` - `format_rag_context(chunks: list[dict]) -> str` — formatted for system prompt injection #### src/tools.py Three tools in registry: calculator, get_datetime, wikipedia_search. Same implementation as p1-06-tool-calling-agent.md. Include full tool registry pattern. #### src/context.py `ContextManager` class with sliding window strategy. Same implementation as p1-04-context-window-manager.md. Include full class. #### src/database.py SQLite with three tables: ```sql CREATE TABLE IF NOT EXISTS requests (id, session_id, prompt_tokens, completion_tokens, cost_usd, latency_ms, retrieval_hit INTEGER, tool_used INTEGER, status, created_at); CREATE TABLE IF NOT EXISTS documents (id, filename, chunks_count, indexed_at); CREATE TABLE IF NOT EXISTS eval_runs (id, suite_name, passed, total, retrieval_hit_rate, run_at); ``` Functions: `init_db()`, `log_request(...)`, `log_document(...)`, `log_eval_run(...)`, `get_metrics() -> dict` #### src/main.py FastAPI app wiring all components. Uses `Mangum` for Lambda. All five routes implemented. --- ### eval/suite.yaml (include in project) ```yaml suite_name: "Full Assistant Eval" description: "End-to-end tests covering RAG retrieval, tool use, and multi-turn memory" scoring_dimensions: - name: relevance description: "Answer directly addresses the question asked" passing_threshold: 3 - name: accuracy description: "Factual claims in the answer are correct" passing_threshold: 4 test_cases: - id: rag-001 description: "RAG: question answerable from uploaded document" input: "What does the uploaded document say about chunking?" context: "The user has uploaded a document about text processing." use_rag: true - id: tool-001 description: "Tool use: arithmetic" input: "What is 2 to the power of 16?" context: "" use_rag: false - id: tool-002 description: "Tool use: current date" input: "What day of the week is today?" context: "" use_rag: false - id: multi-001 description: "Multi-turn: remembers prior context" input: "What was the first question I asked in this session?" context: "Prior turns exist in session." use_rag: false # ... add 6+ more test cases covering edge cases ``` --- ### docs/architecture.md (include in project) ```markdown # Architecture: Full AI Assistant ```mermaid graph TB Client["Client (curl / browser)"] APIGW["API Gateway HTTP API"] Lambda["AWS Lambda\n(FastAPI + Mangum)"] SM["Secrets Manager\n(LLM API Key)"] S3["S3\n(FAISS index - optional)"] LLM["LLM Provider\n(Anthropic / OpenAI)"] Client -->|"POST /chat\n(SSE stream)"| APIGW Client -->|"POST /documents"| APIGW Client -->|"GET /metrics"| APIGW APIGW --> Lambda Lambda -->|"GetSecretValue"| SM Lambda -->|"Stream tokens"| LLM Lambda -->|"Tool calls: Wikipedia, calculator, datetime"| External["External APIs\n(Wikipedia REST)"] Lambda -->|"/tmp storage"| SQLite["SQLite\n(requests, sessions)"] Lambda -->|"FAISS index"| SQLite subgraph "Lambda internals" Router["FastAPI Router"] RAG["RAG Module\n(FAISS + embeddings)"] Context["Context Manager\n(sliding window)"] Tools["Tool Registry\n(calculator, datetime, wiki)"] Router --> RAG Router --> Context Router --> Tools end ``` ## Data flow for a /chat request 1. Client sends `{"message": "...", "session_id": "abc"}` 2. Lambda loads last 20 messages for session "abc" from SQLite 3. RAG retrieves top-3 chunks from FAISS index (if use_rag=true) 4. Context manager checks token count, applies sliding window if needed 5. LLM streams response; if tool_call in response → execute tool → inject result → continue 6. Final answer streamed as SSE to client 7. Request logged to SQLite (tokens, cost, latency, retrieval_hit) ``` --- ### scripts/run_eval.py ```python """Run the eval suite against a live or local assistant endpoint.""" import argparse, requests, yaml, json, sys parser = argparse.ArgumentParser() parser.add_argument("--base-url", default="http://localhost:8000") parser.add_argument("--suite", default="eval/suite.yaml") args = parser.parse_args() with open(args.suite) as f: suite = yaml.safe_load(f) print(f"Running {suite['suite_name']} against {args.base_url}") print(f"{len(suite['test_cases'])} test cases\n") # Upload a sample document first (for RAG test cases) with open("sample_docs/test.txt", "rb") as f: resp = requests.post(f"{args.base_url}/documents", files={"file": f}) print(f"Uploaded sample doc: {resp.json()}") # Run each test case by posting to /eval resp = requests.post(f"{args.base_url}/eval", json={"suite_yaml": yaml.dump(suite)}) result = resp.json() print(f"\nResults:") for r in result["results"]: status = "✓" if r["passed"] else "✗" print(f" {status} {r['id']}: {r.get('summary', '')}") summary = result["summary"] print(f"\nSummary: {summary['passed']}/{summary['total']} passed") print(f"Retrieval hit rate: {summary.get('retrieval_hit_rate', 'N/A')}") sys.exit(0 if summary["passed"] == summary["total"] else 1) ``` --- ### tests/ — what to test **File:** `tests/test_integration.py` — use FastAPI test client, mock LLM and FAISS. **Test 1 — /health returns 200:** Assert status 200 and "status": "ok". **Test 2 — /chat returns SSE response:** Mock LLM stream. Assert response headers include `content-type: text/event-stream`. **Test 3 — /metrics returns correct structure:** Assert response has "total_requests", "retrieval_hit_rate", "p50_latency_ms". **Test 4 — /documents accepts file upload:** Mock FAISS indexing. POST a small TXT file. Assert response has "chunks_added". **Test 5 — rate limiter returns 429:** Mock rate limiter to deny. POST /chat. Assert 429. --- ### README.md content ```markdown # Full AI Assistant (Capstone) A deployed AI assistant combining RAG over uploaded documents, tool use for live data, context window management, and streaming responses — with an eval suite that runs against the live endpoint. ## Setup ```bash cd p1-12-capstone cp .env.example .env pip install -r requirements.txt ``` ## Run locally ```bash docker compose up python scripts/run_eval.py --base-url http://localhost:8000 ``` ## Deploy ```bash bash scripts/deploy.sh source .deployed.env python scripts/run_eval.py --base-url $API_URL ``` ## Teardown ```bash bash scripts/teardown.sh ``` ## What to try next - Upload a PDF of your company's documentation and ask questions about it - Add a new tool (e.g., `fetch_url`) to the tool registry - Tune the RAG score threshold and see how the retrieval_hit_rate changes ``` --- ### GUIDE.md content ```markdown # Build guide: Full AI Assistant Capstone ## What you're building and why it matters Every component you've built in Projects 1–11 exists to make this one thing possible: a deployed assistant that retrieves from your documents, uses tools for live data, manages long conversations, and can be evaluated against a test suite. This is not a demo. It has SSE streaming, real metrics, a live eval endpoint, and a complete teardown script. It is the centrepiece of a portfolio and the answer to "show me something you've built." ## The decision that matters in this build **Eval against the live endpoint, not a local mock.** It is easy to write tests that mock the LLM and assert that mock was called. Those tests tell you nothing about whether the deployed system works. `run_eval.py` calls the real `/eval` endpoint which calls the real LLM. The retrieval_hit_rate metric tells you whether RAG is actually helping — not whether your code runs. ## What will break **FAISS index is rebuilt on Lambda restart.** Lambda functions are stateless and ephemeral. `/tmp` persists within a warm invocation but not across cold starts. For a production system, the FAISS index should be loaded from S3 on cold start. For this project, `/documents` must be called again after each cold start. This is a real limitation to know about and discuss. **Multi-turn context requires consistent session_id.** If the client doesn't send the same `session_id` across turns, the assistant starts fresh every turn. The eval suite must use a consistent session_id for multi-turn test cases. ## How to talk about this in an interview "I built and deployed a full AI assistant to Lambda. It combines RAG over uploaded documents with tool use for live data, all behind a streaming SSE API. I measure retrieval hit rate — the fraction of RAG-enabled responses that actually used retrieved context — which tells me whether the retrieval is helping or not. The eval suite runs against the live endpoint, not a mock, with exit code 1 on failure." ``` --- ### Cost estimate | Resource | Estimated cost | |----------|---------------| | Lambda invocations (100 test) | ~$0.01 | | API Gateway | ~$0.01 | | ECR image storage | ~$0.10 | | Secrets Manager | <$0.01 | | **Total** | **~$3–5** | --- ### Failure modes to handle | Failure | Where | How to handle | |---------|-------|---------------| | FAISS index empty on Lambda cold start | `rag.py` | Return empty chunks; `use_rag=false` effectively; log warning | | Tool call loops (agent calls same tool repeatedly) | `main.py /chat` | Max 8 iterations hard cap; return partial answer | | /eval LLM judge failure | Judge scoring | Mark test as failed with "Judge unavailable", continue suite | | Document upload too large | `/documents` | Return HTTP 413 if file > 10MB | | Lambda /tmp full | `rag.py` | Log warning, attempt to clear oldest index files | --- ### The metric this project measures **Retrieval hit rate** — fraction of RAG-enabled responses where retrieved chunks were referenced. Available at `GET /metrics` as `retrieval_hit_rate`. Target: >60% for questions that are answerable from uploaded documents. Low rate means either retrieval quality is poor or chunks are not relevant to the questions. ### Model source currency - Anthropic model list: https://docs.anthropic.com/en/docs/about-claude/models/overview - OpenAI model list: https://platform.openai.com/docs/models - verifiedOn: 2026-08-07 - `LLM_MODEL` is required and has no implicit default; choose a supported model for the configured provider at setup time. ## Source code A reference implementation of Full AI Assistant (Capstone) — README, guide, source and tests — is at https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/ai-assistant-capstone.