--- title: "Docs Copilot" description: "Section-level citations and a confidence threshold are exactly the trust-building details that separate a demo RAG bot from one a team would actually adopt..." source: "https://confidentprep.com/paths/ai-augmented-engineering/docs-copilot/" path: "AI-Augmented Engineering" repository: "https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/docs-copilot" token_estimate: 4722 --- # Docs Copilot ## Overview A metadata-aware RAG documentation assistant with section-level citations, freshness tracking on the underlying docs, and an explicit "I don't have enough information" threshold instead of a confident-sounding guess. This is RAG again — but now judged on trustworthiness, not just retrieval accuracy, which is the harder and more realistic bar. ## What to do | Field | Value | |---|---| | Path | 2 — AI-Augmented Engineering | | Position | 9 of 12 | | Difficulty | 🟡 Optional Docker | | Estimated time | 4 hours | | AWS cost | None | --- ### Agent Pickup Instructions ```bash # 1. Enter project cd projects/p2-09-docs-copilot # 2. Create virtual environment python -m venv .venv && source .venv/bin/activate # 3. Install dependencies pip install -r requirements.txt # 4. Copy env file and add your API key cp .env.example .env # 5. Index your documentation folder python src/main.py --index docs/ # 6. Ask a question python src/main.py --ask "How do I configure the database connection?" # 7. Force rebuild if docs changed python src/main.py --rebuild # 8. Run tests pytest tests/ -v ``` **Done when:** - [ ] `--index` processes Markdown and PDF files, attaches metadata to every chunk - [ ] `--ask` returns an answer with citations in `[source > section]` format - [ ] Freshness check detects a changed file and automatically re-indexes it - [ ] A low-confidence question returns "I don't know" instead of a hallucinated answer - [ ] All 4 tests pass --- ### What this project is A RAG-based documentation assistant that differs from a basic RAG system in three key ways: metadata-aware chunking (section headings preserved in every chunk), freshness tracking (file modification times checked on each run to detect stale index entries), and a confidence threshold below which the system returns "I don't know" rather than generating a potentially incorrect answer. Citations appear in `[source > section_heading]` format in every answer, linking the response to the exact document section it came from. --- ### What the learner achieves "I built a metadata-aware RAG system over team docs with section-level citations and a freshness check that re-indexes changed files automatically — and an 'I don't know' threshold to prevent hallucination when the docs don't contain the answer." --- ### Folder structure ``` p2-09-docs-copilot/ ├── src/ │ ├── main.py # CLI: --index , --ask "...", --rebuild │ ├── llm.py # Provider-agnostic LLM wrapper │ ├── chunker.py # markdown_chunk_by_section, pdf_chunk_with_heading_detection, attach_metadata │ ├── indexer.py # build_with_metadata, load, search_with_metadata, check_freshness_and_reindex │ └── copilot.py # build_answer_with_citations, handle_below_threshold ├── tests/ │ ├── fixtures/ │ │ ├── docs/ │ │ │ ├── setup_guide.md # Multi-section markdown doc │ │ │ └── api_reference.md # Another markdown doc │ │ └── sample.pdf # Small test PDF (optional) │ ├── test_chunker.py │ ├── test_indexer.py │ └── test_copilot.py ├── .index/ # Auto-created │ ├── index.faiss │ ├── metadata.json │ └── file_mtimes.json ├── .env.example ├── requirements.txt └── README.md ``` --- ### .env.example ```bash # LLM provider: anthropic | openai | ollama LLM_PROVIDER=anthropic # Model to use LLM_MODEL= # API key for Anthropic ANTHROPIC_API_KEY= # API key for OpenAI OPENAI_API_KEY= # Ollama base URL OLLAMA_BASE_URL=http://localhost:11434 # Embedding model for RAG EMBED_MODEL=all-MiniLM-L6-v2 # Index storage directory INDEX_DIR=.index # Number of chunks to retrieve per query TOP_K=5 # Confidence threshold below which "I don't know" is returned (0.0 to 1.0) # FAISS inner product score — tune based on your embedding model CONFIDENCE_THRESHOLD=0.5 # Maximum characters per chunk (for markdown sections and PDF pages) MAX_CHUNK_CHARS=2000 ``` --- ### requirements.txt ``` anthropic==0.30.0 openai==1.35.0 sentence-transformers==3.0.1 faiss-cpu==1.8.0 pdfplumber==0.11.1 numpy==1.26.4 python-dotenv==1.0.1 rich==13.7.1 pytest==8.2.2 pytest-mock==3.14.0 ``` --- ### src/ — what to implement #### `src/llm.py` **`get_completion(prompt: str, system: str = "") -> str`** - Standard interface: reads `LLM_PROVIDER` and `LLM_MODEL` from env - Routes to Anthropic, OpenAI, or Ollama - Edge cases: raise `RuntimeError` if API key is missing --- #### `src/chunker.py` **`ChunkMetadata` (dataclass)** ```python @dataclass class ChunkMetadata: doc_title: str # Filename without extension section_heading: str # Nearest heading above this chunk last_modified: float # os.path.getmtime() value source_file: str # Absolute file path chunk_index: int # Position within document char_count: int # Length of chunk text ``` **`markdown_chunk_by_section(filepath: str) -> list[tuple[str, ChunkMetadata]]`** - Inputs: path to `.md` file - Output: list of `(text, metadata)` tuples — one per section - Behavior: - Read file content - Split on heading lines: `^#{1,3} ` (H1, H2, H3) - Each chunk = heading line + content until next heading - `section_heading` = the heading text (stripped of `#` symbols) - `doc_title` = filename without extension - `last_modified` = `os.path.getmtime(filepath)` - If content exceeds `MAX_CHUNK_CHARS`, split at paragraph boundaries (blank lines) - Edge cases: file with no headings → one chunk with `section_heading = doc_title` **`pdf_chunk_with_heading_detection(filepath: str) -> list[tuple[str, ChunkMetadata]]`** - Inputs: path to `.pdf` file - Output: list of `(text, metadata)` tuples — one per detected section - Behavior: - Use `pdfplumber.open(filepath)` to extract text per page - Detect headings heuristically: lines that are ALL CAPS, or lines shorter than 60 chars followed by a blank line - Group text under each detected heading - `section_heading` = detected heading or `f"Page {page_num}"` - Edge cases: PDF with no extractable text → return `[]`; encrypted PDF → log warning and return `[]` **`attach_metadata(text: str, meta: ChunkMetadata) -> dict`** - Inputs: chunk text, metadata - Output: `{"text": str, "metadata": ChunkMetadata}` - Simple wrapper — ensures consistent structure --- #### `src/indexer.py` **`build_with_metadata(chunks: list[dict], model) -> tuple`** - Inputs: list of `{"text": str, "metadata": ChunkMetadata}` dicts, loaded sentence-transformer model - Output: `(faiss_index, embeddings, chunks)` — index built from normalized embeddings - Behavior: embed all `chunk["text"]`; normalize; build `IndexFlatIP` **`save_index(index, chunks: list[dict], file_mtimes: dict, index_dir: str) -> None`** - Inputs: FAISS index, chunks, `{filepath: mtime}` dict, directory - Behavior: write `index.faiss`, `metadata.json` (chunks without FAISS array), `file_mtimes.json` **`load_index(index_dir: str) -> tuple`** - Output: `(index, chunks, file_mtimes)` - Edge cases: any file missing → raise `FileNotFoundError("Run --index first")` **`search_with_metadata(query: str, model, index, chunks: list[dict], top_k: int = 5) -> list[dict]`** - Inputs: query string, model, index, chunks, k - Output: list of `{"text": str, "metadata": ChunkMetadata, "score": float}` sorted by score descending **`check_freshness_and_reindex(index_dir: str, docs_dir: str, model) -> bool`** - Inputs: index directory, docs directory, loaded model - Output: `True` if re-index was needed and performed, `False` if all files are current - Behavior: - Load existing `file_mtimes.json` - Walk docs directory; compute current mtime for each file - If any mtime differs from stored → full re-index of that file (or full rebuild for simplicity) - Log: `Stale index detected: changed. Re-indexing.` - Update stored mtimes after re-index --- #### `src/copilot.py` **`build_answer_with_citations(query: str, retrieved_chunks: list[dict]) -> str`** - Inputs: user query, retrieved chunks with metadata - Output: answer string with citations - Behavior: - Build context string: for each chunk, format as `[Source: {doc_title} > {section_heading}]\n{text}` - System prompt: "You are a helpful documentation assistant. Answer the question using ONLY the provided context. After each statement cite the source using format [source_name > section]. If the context doesn't contain the answer, say 'I don't know.'" - User prompt: `Context:\n{context}\n\nQuestion: {query}` - Call `get_completion` - Post-process: ensure every citation appears in format `[{doc_title} > {section_heading}]` **`handle_below_threshold(query: str, top_score: float, threshold: float) -> str | None`** - Inputs: original query, highest retrieval score, threshold - Output: "I don't know" message string if below threshold, `None` if above threshold - Behavior: if `top_score < threshold`, return `f"I don't have enough information to answer '{query}'. The documentation may not cover this topic."` - This is called before the LLM — if below threshold, skip the LLM call entirely **`format_citations(chunks: list[dict]) -> list[str]`** - Inputs: retrieved chunks - Output: list of `"{doc_title} > {section_heading}"` strings (deduplicated, preserving order) --- #### `src/main.py` CLI entry point using `argparse`. Arguments: - `--index ` — index a documentation directory - `--ask "..."` — ask a question; requires index to exist - `--rebuild` — force full re-index (ignore mtimes) Behavior for `--index`: 1. Walk directory for `.md` and `.pdf` files 2. Check freshness; skip unchanged files 3. Chunk all new/changed files 4. Embed and build/update index 5. Print `Indexed N chunks from M files` Behavior for `--ask`: 1. Load index 2. Check freshness; if stale, re-index automatically 3. Search 4. Call `handle_below_threshold` with top score 5. If below threshold: print "I don't know" message; exit 0 6. Otherwise: call `build_answer_with_citations` 7. Print answer 8. Print `\nSources:` followed by citation list --- ### tests/ — what to test #### Test 1 — Metadata attached to every chunk (`test_chunker.py`) - Call `markdown_chunk_by_section` on `tests/fixtures/docs/setup_guide.md` (has 3 headings) - Assert 3 or more chunks returned - Assert every chunk's metadata has non-empty `doc_title`, `section_heading`, `last_modified` - Assert `last_modified` is a positive float (valid mtime) #### Test 2 — Freshness check detects changed file (`test_indexer.py`) - Save a fake `file_mtimes.json` with an old mtime for `setup_guide.md` - Call `check_freshness_and_reindex` (with model mocked) - Assert it returns `True` (re-index was needed) - Assert stdout contains "Stale index detected" #### Test 3 — Below-threshold returns "I don't know" message (`test_copilot.py`) - Call `handle_below_threshold(query="What is the airspeed velocity?", top_score=0.3, threshold=0.5)` - Assert result is not `None` - Assert "don't have enough information" appears in the returned string - Assert the query text appears in the returned string #### Test 4 — Citation format is `[source > section]` in answer (`test_copilot.py`) - Mock `get_completion` to return a response containing a citation - Call `build_answer_with_citations` with 2 chunks that have known doc_title and section_heading - Assert the response contains at least one `[` and `>` combination - Assert the cited doc_title appears in the response #### Test 5 — Markdown chunker handles file with no headings (`test_chunker.py`) - Create a temp `.md` file with 3 paragraphs but no headings - Call `markdown_chunk_by_section` - Assert exactly 1 chunk is returned - Assert `section_heading` equals the doc title (filename without extension) #### Test 6 — Search returns results above threshold (`test_indexer.py`) - Build a small in-memory index from 2 chunks - Search for a query semantically matching chunk 1 - Assert top result has `score > 0.0` - Assert result has `metadata` key with `section_heading` --- ### README.md content ```markdown # Docs Copilot RAG-based documentation assistant with section-level citations, freshness tracking, and an "I don't know" threshold. ## Quick start ```bash python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt cp .env.example .env # add your API key ## Index your docs python src/main.py --index docs/ ## Ask a question python src/main.py --ask "How do I reset my password?" ## Force rebuild python src/main.py --rebuild ``` ## What makes this different from basic RAG | Feature | Basic RAG | Docs Copilot | |---|---|---| | Chunking | Fixed size | By section heading | | Metadata | None | doc_title, section_heading, mtime | | Freshness | Manual rebuild | Automatic mtime check | | Low confidence | Returns something | Returns "I don't know" | | Citations | Optional | Required: [source > section] | ## Citation format Every answer includes source citations: ``` You can reset your password from the Settings page. [setup_guide > Account Settings] ``` ## Running tests ```bash pytest tests/ -v ``` ``` --- ### GUIDE.md content ```markdown # Build Guide — Docs Copilot ## Step 1 — Metadata-aware chunking The key difference from basic RAG: preserve the section heading with every chunk. ```python def markdown_chunk_by_section(filepath): content = open(filepath).read() lines = content.splitlines() current_heading = os.path.basename(filepath).replace(".md", "") current_lines = [] chunks = [] for line in lines: if line.startswith("#"): if current_lines: text = "\n".join(current_lines).strip() meta = ChunkMetadata( doc_title=os.path.basename(filepath).replace(".md", ""), section_heading=current_heading, last_modified=os.path.getmtime(filepath), source_file=filepath, chunk_index=len(chunks), char_count=len(text), ) chunks.append((text, meta)) current_lines = [] current_heading = line.lstrip("#").strip() else: current_lines.append(line) # Don't forget the last section if current_lines: ... return chunks ``` ## Step 2 — Freshness tracking Store mtimes, not content hashes. Mtime is cheaper to read than hashing: ```python import os, json def check_freshness(index_dir, docs_dir): stored = json.load(open(f"{index_dir}/file_mtimes.json")) for filepath in walk_docs(docs_dir): current_mtime = os.path.getmtime(filepath) if stored.get(filepath) != current_mtime: return True # stale return False ``` ## Step 3 — The confidence gate Check the top retrieval score BEFORE calling the LLM: ```python results = search_with_metadata(query, model, index, chunks, top_k) top_score = results[0]["score"] if results else 0.0 if top_score < CONFIDENCE_THRESHOLD: print(handle_below_threshold(query, top_score, CONFIDENCE_THRESHOLD)) return ``` This saves an LLM call and avoids hallucination on topics not covered by the docs. ## Step 4 — Citations in the prompt Format context so the LLM knows the source of each chunk: ``` [Source: setup_guide > Database Configuration] To configure the database, set DATABASE_URL in your .env file... [Source: api_reference > Authentication] All API calls require a Bearer token in the Authorization header... ``` Instruct the LLM: "After each claim, cite the source in [source > section] format." ## Debugging tips - If citations are missing, check that the system prompt explicitly requires them - If freshness check always triggers, verify mtime comparison uses `==` not `is` - If PDF chunks are empty, use `pdfplumber` — not PyPDF2 — for better text extraction ## How to talk about this in an interview **"How is this different from the RAG you built in Path 1?"** > Three differences. First, I chunk by section heading instead of fixed size — this gives more semantically coherent chunks. Second, I attach metadata (doc title, section, mtime) to every chunk so citations are reliable. Third, I gate on a confidence threshold — if retrieval scores are below the threshold, I return 'I don't know' rather than risk hallucination. **"How does freshness tracking work?"** > I store the mtime of each indexed file alongside the index. On every `--ask`, I check current mtimes against stored ones. If any file changed, I re-index it before answering. This means the index is always current without requiring manual rebuilds. **"What threshold do you use and how did you choose it?"** > I use 0.5 on the inner product of L2-normalized vectors (equivalent to cosine similarity). I tuned it by asking 10 questions I knew the docs covered and 10 I knew they didn't, and found 0.5 gave zero false "I know" responses on the second set. ``` --- ### Failure modes to handle | Failure | Where | How to handle | |---|---|---| | PDF is encrypted or empty | `chunker.py` | Log warning with filename; return `[]`; continue | | Index missing when `--ask` called | `indexer.py` | Raise `FileNotFoundError("Run --index first")` | | All retrieval scores below threshold | `copilot.py` | Return "I don't know" message without LLM call | | Markdown file has no headings | `chunker.py` | Return single chunk with `section_heading = doc_title` | | Doc directory has no supported files | `main.py` | Print "No .md or .pdf files found in ." and exit 0 | | `file_mtimes.json` missing (first run) | `indexer.py` | Treat all files as stale; build fresh index | --- ### The metric this project measures **What is measured:** Chunk count with metadata, freshness detection accuracy, and citation presence rate in answers. **Format (stdout):** ``` Indexed 23 chunks from 4 files (2 new, 2 unchanged) Answer generated | top_score=0.74 | citations=2 | sources: [setup_guide > Database Config] ``` **Target:** Every chunk in the index must have non-null `doc_title`, `section_heading`, and `last_modified`. When a fixture file is modified, the next `--ask` must trigger re-indexing before answering. Every answer must contain at least one `[source > section]` citation. These three confirm metadata attachment, freshness tracking, and citation generation are all wired correctly. ### Model source currency - Anthropic model list: https://docs.anthropic.com/en/docs/about-claude/models/overview - OpenAI model list: https://platform.openai.com/docs/models - verifiedOn: 2026-08-07 - `LLM_MODEL` is required and has no implicit default; choose a supported model for the configured provider at setup time. ## Source code A reference implementation of Docs Copilot — README, guide, source and tests — is at https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/docs-copilot.