Docs Copilot

Field	Value
Path	2 — AI-Augmented Engineering
Position	9 of 12
Difficulty	🟡 Optional Docker
Estimated time	4 hours
AWS cost	None

---

Agent Pickup Instructions

# 1. Enter project
cd projects/p2-09-docs-copilot

# 2. Create virtual environment
python -m venv .venv && source .venv/bin/activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Copy env file and add your API key
cp .env.example .env

# 5. Index your documentation folder
python src/main.py --index docs/

# 6. Ask a question
python src/main.py --ask "How do I configure the database connection?"

# 7. Force rebuild if docs changed
python src/main.py --rebuild

# 8. Run tests
pytest tests/ -v

Done when:

• --index processes Markdown and PDF files, attaches metadata to every chunk
• --ask returns an answer with citations in [source > section] format
• Freshness check detects a changed file and automatically re-indexes it
• A low-confidence question returns "I don't know" instead of a hallucinated answer
• All 4 tests pass

---

What this project is

A RAG-based documentation assistant that differs from a basic RAG system in three key ways: metadata-aware chunking (section headings preserved in every chunk), freshness tracking (file modification times checked on each run to detect stale index entries), and a confidence threshold below which the system returns "I don't know" rather than generating a potentially incorrect answer. Citations appear in [source > section_heading] format in every answer, linking the response to the exact document section it came from.

---

What the learner achieves

"I built a metadata-aware RAG system over team docs with section-level citations and a freshness check that re-indexes changed files automatically — and an 'I don't know' threshold to prevent hallucination when the docs don't contain the answer."

---

Folder structure

p2-09-docs-copilot/
├── src/
│   ├── main.py        # CLI: --index <dir>, --ask "...", --rebuild
│   ├── llm.py         # Provider-agnostic LLM wrapper
│   ├── chunker.py     # markdown_chunk_by_section, pdf_chunk_with_heading_detection, attach_metadata
│   ├── indexer.py     # build_with_metadata, load, search_with_metadata, check_freshness_and_reindex
│   └── copilot.py     # build_answer_with_citations, handle_below_threshold
├── tests/
│   ├── fixtures/
│   │   ├── docs/
│   │   │   ├── setup_guide.md       # Multi-section markdown doc
│   │   │   └── api_reference.md     # Another markdown doc
│   │   └── sample.pdf               # Small test PDF (optional)
│   ├── test_chunker.py
│   ├── test_indexer.py
│   └── test_copilot.py
├── .index/                           # Auto-created
│   ├── index.faiss
│   ├── metadata.json
│   └── file_mtimes.json
├── .env.example
├── requirements.txt
└── README.md

---

.env.example

# LLM provider: anthropic | openai | ollama
LLM_PROVIDER=anthropic

# Model to use
LLM_MODEL=

# API key for Anthropic
ANTHROPIC_API_KEY=

# API key for OpenAI
OPENAI_API_KEY=

# Ollama base URL
OLLAMA_BASE_URL=http://localhost:11434

# Embedding model for RAG
EMBED_MODEL=all-MiniLM-L6-v2

# Index storage directory
INDEX_DIR=.index

# Number of chunks to retrieve per query
TOP_K=5

# Confidence threshold below which "I don't know" is returned (0.0 to 1.0)
# FAISS inner product score — tune based on your embedding model
CONFIDENCE_THRESHOLD=0.5

# Maximum characters per chunk (for markdown sections and PDF pages)
MAX_CHUNK_CHARS=2000

---

requirements.txt

anthropic==0.30.0
openai==1.35.0
sentence-transformers==3.0.1
faiss-cpu==1.8.0
pdfplumber==0.11.1
numpy==1.26.4
python-dotenv==1.0.1
rich==13.7.1
pytest==8.2.2
pytest-mock==3.14.0

---

src/ — what to implement

src/llm.py

get_completion(prompt: str, system: str = "") -> str

• Standard interface: reads LLM_PROVIDER and LLM_MODEL from env
• Routes to Anthropic, OpenAI, or Ollama
• Edge cases: raise RuntimeError if API key is missing

---

src/chunker.py

ChunkMetadata (dataclass)

@dataclass
class ChunkMetadata:
    doc_title: str         # Filename without extension
    section_heading: str   # Nearest heading above this chunk
    last_modified: float   # os.path.getmtime() value
    source_file: str       # Absolute file path
    chunk_index: int       # Position within document
    char_count: int        # Length of chunk text

markdown_chunk_by_section(filepath: str) -> list[tuple[str, ChunkMetadata]]

• Inputs: path to .md file
• Output: list of (text, metadata) tuples — one per section
• Behavior:
  • Read file content
  • Split on heading lines: ^#{1,3} (H1, H2, H3)
  • Each chunk = heading line + content until next heading
  • section_heading = the heading text (stripped of # symbols)
  • doc_title = filename without extension
  • last_modified = os.path.getmtime(filepath)
  • If content exceeds MAX_CHUNK_CHARS, split at paragraph boundaries (blank lines)
• Edge cases: file with no headings → one chunk with section_heading = doc_title

pdf_chunk_with_heading_detection(filepath: str) -> list[tuple[str, ChunkMetadata]]

• Inputs: path to .pdf file
• Output: list of (text, metadata) tuples — one per detected section
• Behavior:
  • Use pdfplumber.open(filepath) to extract text per page
  • Detect headings heuristically: lines that are ALL CAPS, or lines shorter than 60 chars followed by a blank line
  • Group text under each detected heading
  • section_heading = detected heading or f"Page {page_num}"
• Edge cases: PDF with no extractable text → return []; encrypted PDF → log warning and return []

attach_metadata(text: str, meta: ChunkMetadata) -> dict

• Inputs: chunk text, metadata
• Output: {"text": str, "metadata": ChunkMetadata}
• Simple wrapper — ensures consistent structure

---

src/indexer.py

build_with_metadata(chunks: list[dict], model) -> tuple

• Inputs: list of {"text": str, "metadata": ChunkMetadata} dicts, loaded sentence-transformer model
• Output: (faiss_index, embeddings, chunks) — index built from normalized embeddings
• Behavior: embed all chunk["text"]; normalize; build IndexFlatIP

save_index(index, chunks: list[dict], file_mtimes: dict, index_dir: str) -> None

• Inputs: FAISS index, chunks, {filepath: mtime} dict, directory
• Behavior: write index.faiss, metadata.json (chunks without FAISS array), file_mtimes.json

load_index(index_dir: str) -> tuple

• Output: (index, chunks, file_mtimes)
• Edge cases: any file missing → raise FileNotFoundError("Run --index first")

search_with_metadata(query: str, model, index, chunks: list[dict], top_k: int = 5) -> list[dict]

• Inputs: query string, model, index, chunks, k
• Output: list of {"text": str, "metadata": ChunkMetadata, "score": float} sorted by score descending

check_freshness_and_reindex(index_dir: str, docs_dir: str, model) -> bool

• Inputs: index directory, docs directory, loaded model
• Output: True if re-index was needed and performed, False if all files are current
• Behavior:
  • Load existing file_mtimes.json
  • Walk docs directory; compute current mtime for each file
  • If any mtime differs from stored → full re-index of that file (or full rebuild for simplicity)
  • Log: Stale index detected: <filename> changed. Re-indexing.
  • Update stored mtimes after re-index

---

src/copilot.py

build_answer_with_citations(query: str, retrieved_chunks: list[dict]) -> str

• Inputs: user query, retrieved chunks with metadata
• Output: answer string with citations
• Behavior:
  • Build context string: for each chunk, format as [Source: {doc_title} > {section_heading}]\n{text}
  • System prompt: "You are a helpful documentation assistant. Answer the question using ONLY the provided context. After each statement cite the source using format [source_name > section]. If the context doesn't contain the answer, say 'I don't know.'"
  • User prompt: Context:\n{context}\n\nQuestion: {query}
  • Call get_completion
  • Post-process: ensure every citation appears in format [{doc_title} > {section_heading}]

handle_below_threshold(query: str, top_score: float, threshold: float) -> str | None

• Inputs: original query, highest retrieval score, threshold
• Output: "I don't know" message string if below threshold, None if above threshold
• Behavior: if top_score < threshold, return f"I don't have enough information to answer '{query}'. The documentation may not cover this topic."
• This is called before the LLM — if below threshold, skip the LLM call entirely

format_citations(chunks: list[dict]) -> list[str]

• Inputs: retrieved chunks
• Output: list of "{doc_title} > {section_heading}" strings (deduplicated, preserving order)

---

src/main.py

CLI entry point using argparse.

Arguments:

• --index <dir> — index a documentation directory
• --ask "..." — ask a question; requires index to exist
• --rebuild — force full re-index (ignore mtimes)

Behavior for --index:

1. Walk directory for .md and .pdf files
2. Check freshness; skip unchanged files
3. Chunk all new/changed files
4. Embed and build/update index
5. Print Indexed N chunks from M files

Behavior for --ask:

1. Load index
2. Check freshness; if stale, re-index automatically
3. Search
4. Call handle_below_threshold with top score
5. If below threshold: print "I don't know" message; exit 0
6. Otherwise: call build_answer_with_citations
7. Print answer
8. Print \nSources: followed by citation list

---

tests/ — what to test

Test 1 — Metadata attached to every chunk (test_chunker.py)

• Call markdown_chunk_by_section on tests/fixtures/docs/setup_guide.md (has 3 headings)
• Assert 3 or more chunks returned
• Assert every chunk's metadata has non-empty doc_title, section_heading, last_modified
• Assert last_modified is a positive float (valid mtime)

Test 2 — Freshness check detects changed file (test_indexer.py)

• Save a fake file_mtimes.json with an old mtime for setup_guide.md
• Call check_freshness_and_reindex (with model mocked)
• Assert it returns True (re-index was needed)
• Assert stdout contains "Stale index detected"

Test 3 — Below-threshold returns "I don't know" message (test_copilot.py)

• Call handle_below_threshold(query="What is the airspeed velocity?", top_score=0.3, threshold=0.5)
• Assert result is not None
• Assert "don't have enough information" appears in the returned string
• Assert the query text appears in the returned string

Test 4 — Citation format is [source > section] in answer (test_copilot.py)

• Mock get_completion to return a response containing a citation
• Call build_answer_with_citations with 2 chunks that have known doc_title and section_heading
• Assert the response contains at least one [ and > combination
• Assert the cited doc_title appears in the response

Test 5 — Markdown chunker handles file with no headings (test_chunker.py)

• Create a temp .md file with 3 paragraphs but no headings
• Call markdown_chunk_by_section
• Assert exactly 1 chunk is returned
• Assert section_heading equals the doc title (filename without extension)

Test 6 — Search returns results above threshold (test_indexer.py)

• Build a small in-memory index from 2 chunks
• Search for a query semantically matching chunk 1
• Assert top result has score > 0.0
• Assert result has metadata key with section_heading

---

README.md content

# Docs Copilot

RAG-based documentation assistant with section-level citations, freshness tracking, and an "I don't know" threshold.

## Quick start

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env   # add your API key

# Index your docs
python src/main.py --index docs/

# Ask a question
python src/main.py --ask "How do I reset my password?"

# Force rebuild
python src/main.py --rebuild

What makes this different from basic RAG

Feature	Basic RAG	Docs Copilot
Chunking	Fixed size	By section heading
Metadata	None	doc_title, section_heading, mtime
Freshness	Manual rebuild	Automatic mtime check
Low confidence	Returns something	Returns "I don't know"
Citations	Optional	Required: [source > section]

Citation format

Every answer includes source citations:

You can reset your password from the Settings page. [setup_guide > Account Settings]

Running tests

pytest tests/ -v

---

## GUIDE.md content

```markdown
# Build Guide — Docs Copilot

## Step 1 — Metadata-aware chunking

The key difference from basic RAG: preserve the section heading with every chunk.

```python
def markdown_chunk_by_section(filepath):
    content = open(filepath).read()
    lines = content.splitlines()
    
    current_heading = os.path.basename(filepath).replace(".md", "")
    current_lines = []
    chunks = []
    
    for line in lines:
        if line.startswith("#"):
            if current_lines:
                text = "\n".join(current_lines).strip()
                meta = ChunkMetadata(
                    doc_title=os.path.basename(filepath).replace(".md", ""),
                    section_heading=current_heading,
                    last_modified=os.path.getmtime(filepath),
                    source_file=filepath,
                    chunk_index=len(chunks),
                    char_count=len(text),
                )
                chunks.append((text, meta))
                current_lines = []
            current_heading = line.lstrip("#").strip()
        else:
            current_lines.append(line)
    
    # Don't forget the last section
    if current_lines:
        ...
    
    return chunks

Step 2 — Freshness tracking

Store mtimes, not content hashes. Mtime is cheaper to read than hashing:

import os, json

def check_freshness(index_dir, docs_dir):
    stored = json.load(open(f"{index_dir}/file_mtimes.json"))
    for filepath in walk_docs(docs_dir):
        current_mtime = os.path.getmtime(filepath)
        if stored.get(filepath) != current_mtime:
            return True   # stale
    return False

Step 3 — The confidence gate

Check the top retrieval score BEFORE calling the LLM:

results = search_with_metadata(query, model, index, chunks, top_k)
top_score = results[0]["score"] if results else 0.0

if top_score < CONFIDENCE_THRESHOLD:
    print(handle_below_threshold(query, top_score, CONFIDENCE_THRESHOLD))
    return

This saves an LLM call and avoids hallucination on topics not covered by the docs.

Step 4 — Citations in the prompt

Format context so the LLM knows the source of each chunk:

[Source: setup_guide > Database Configuration]
To configure the database, set DATABASE_URL in your .env file...

[Source: api_reference > Authentication]
All API calls require a Bearer token in the Authorization header...

Instruct the LLM: "After each claim, cite the source in [source > section] format."

Debugging tips

• If citations are missing, check that the system prompt explicitly requires them
• If freshness check always triggers, verify mtime comparison uses == not is
• If PDF chunks are empty, use pdfplumber — not PyPDF2 — for better text extraction

How to talk about this in an interview

"How is this different from the RAG you built in Path 1?"

Three differences. First, I chunk by section heading instead of fixed size — this gives more semantically coherent chunks. Second, I attach metadata (doc title, section, mtime) to every chunk so citations are reliable. Third, I gate on a confidence threshold — if retrieval scores are below the threshold, I return 'I don't know' rather than risk hallucination.

"How does freshness tracking work?"

I store the mtime of each indexed file alongside the index. On every --ask, I check current mtimes against stored ones. If any file changed, I re-index it before answering. This means the index is always current without requiring manual rebuilds.

"What threshold do you use and how did you choose it?"

I use 0.5 on the inner product of L2-normalized vectors (equivalent to cosine similarity). I tuned it by asking 10 questions I knew the docs covered and 10 I knew they didn't, and found 0.5 gave zero false "I know" responses on the second set.

---

## Failure modes to handle

| Failure | Where | How to handle |
|---|---|---|
| PDF is encrypted or empty | `chunker.py` | Log warning with filename; return `[]`; continue |
| Index missing when `--ask` called | `indexer.py` | Raise `FileNotFoundError("Run --index first")` |
| All retrieval scores below threshold | `copilot.py` | Return "I don't know" message without LLM call |
| Markdown file has no headings | `chunker.py` | Return single chunk with `section_heading = doc_title` |
| Doc directory has no supported files | `main.py` | Print "No .md or .pdf files found in <dir>." and exit 0 |
| `file_mtimes.json` missing (first run) | `indexer.py` | Treat all files as stale; build fresh index |

---

## The metric this project measures

**What is measured:** Chunk count with metadata, freshness detection accuracy, and citation presence rate in answers.

**Format (stdout):**

Indexed 23 chunks from 4 files (2 new, 2 unchanged) Answer generated | top_score=0.74 | citations=2 | sources: [setup_guide > Database Config]

**Target:** Every chunk in the index must have non-null `doc_title`, `section_heading`, and `last_modified`. When a fixture file is modified, the next `--ask` must trigger re-indexing before answering. Every answer must contain at least one `[source > section]` citation. These three confirm metadata attachment, freshness tracking, and citation generation are all wired correctly.


## Model source currency

- Anthropic model list: https://docs.anthropic.com/en/docs/about-claude/models/overview
- OpenAI model list: https://platform.openai.com/docs/models
- verifiedOn: 2026-08-07
- `LLM_MODEL` is required and has no implicit default; choose a supported model for the configured provider at setup time.