Semantic Codebase Search

Field	Value
Path	2 — AI-Augmented Engineering
Position	3 of 12
Difficulty	🟢 Local only
Estimated time	4 hours
AWS cost	None

---

Agent Pickup Instructions

# 1. Enter project
cd projects/p2-03-semantic-codebase-search

# 2. Create virtual environment
python -m venv .venv && source .venv/bin/activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Copy env file
cp .env.example .env

# 5. Index a Python codebase
python src/main.py --index ./src

# 6. Search semantically
python src/main.py --search "database connection pooling"

# 7. Compare models on a query
python src/main.py --compare "authentication middleware"

# 8. Run tests
pytest tests/ -v

Done when:

• --index processes a directory and prints Indexed N functions from M files
• --search returns results with file path, function name, and line number
• Re-running --index on unchanged files prints Skipped N files (unchanged)
• --compare shows two scored result lists side by side
• All 4 tests pass

---

What this project is

A local semantic search engine for Python codebases that operates at function granularity. It uses AST parsing to extract function boundaries, embeds each function's signature and docstring with a sentence-transformer model, stores vectors in a FAISS index, and supports both AST-based (Python) and heuristic (non-Python) chunking. Incremental indexing via file hash comparison avoids re-embedding unchanged files. A comparison mode runs the same query against two different embedding models and displays the ranked score difference, letting engineers see concretely how code-tuned models differ from general-purpose ones.

---

What the learner achieves

"I built a function-level semantic search engine for codebases using AST chunking, FAISS vector storage, and incremental indexing with file hashing — and I can demonstrate the concrete score difference between a code-tuned and a general-purpose embedding model."

---

Folder structure

p2-03-semantic-codebase-search/
├── src/
│   ├── main.py        # CLI: --index, --search, --compare, --reindex
│   ├── llm.py         # Provider-agnostic LLM wrapper (for future extension)
│   ├── embedder.py    # load_model, embed_functions
│   ├── chunker.py     # ast_chunk_python, heuristic_chunk, file_hash
│   └── indexer.py     # build_index, load_index, search, save_index
├── tests/
│   ├── fixtures/
│   │   ├── sample_module.py      # 5 functions with docstrings
│   │   └── sample_config.yaml    # Non-Python file for heuristic test
│   ├── test_chunker.py
│   ├── test_indexer.py
│   └── test_embedder.py
├── .index/                        # Auto-created: faiss files + metadata
│   ├── index.faiss
│   ├── metadata.json
│   └── file_hashes.json
├── .env.example
├── requirements.txt
└── README.md

---

.env.example

# Primary embedding model (code-tuned)
# Options: microsoft/codebert-base | all-MiniLM-L6-v2 | sentence-transformers/all-mpnet-base-v2
EMBED_MODEL=all-MiniLM-L6-v2

# Comparison model (general-purpose) — used only with --compare flag
COMPARE_MODEL=all-MiniLM-L6-v2

# Path to store the FAISS index and metadata
INDEX_DIR=.index

# Number of results to return per search
TOP_K=5

# Minimum score threshold to display a result (0.0 to 1.0)
MIN_SCORE=0.0

# LLM provider (not used for search — included for future LLM-assisted reranking)
LLM_PROVIDER=anthropic
LLM_MODEL=
ANTHROPIC_API_KEY=
OPENAI_API_KEY=
OLLAMA_BASE_URL=http://localhost:11434

---

requirements.txt

sentence-transformers==3.0.1
faiss-cpu==1.8.0
numpy==1.26.4
python-dotenv==1.0.1
rich==13.7.1
pytest==8.2.2
pytest-mock==3.14.0
anthropic==0.30.0
openai==1.35.0

---

src/ — what to implement

src/llm.py

get_completion(prompt: str, system: str = "") -> str

• Standard interface: reads LLM_PROVIDER and LLM_MODEL from env
• Not used in search pipeline — included for future LLM-assisted reranking extension
• Same implementation pattern as p2-01

---

src/chunker.py

file_hash(filepath: str) -> str

• Inputs: absolute file path
• Output: MD5 hex digest of the file contents
• Behavior: read file in binary mode, compute hashlib.md5
• Edge cases: if file cannot be read, raise FileNotFoundError

ast_chunk_python(filepath: str) -> list[dict]

• Inputs: absolute path to a .py file
• Output: list of chunk dicts, each with keys: {"function_name": str, "file": str, "lineno": int, "end_lineno": int, "text": str}
• Behavior: parse with ast.parse; walk all FunctionDef and AsyncFunctionDef nodes; for each, extract the function name, start/end line numbers, and extract the raw source lines using linecache or inspect.getsource pattern; text = function signature (first line) + docstring (if present) — do NOT include the full body
• Edge cases: file with syntax errors logs a warning and returns []; skip nested functions (only top-level and class-level methods); skip __ dunder methods

heuristic_chunk(filepath: str, chunk_size: int = 30) -> list[dict]

• Inputs: absolute path to any file, lines per chunk
• Output: list of chunk dicts with {"file": str, "lineno": int, "end_lineno": int, "text": str, "function_name": str}
• Behavior: read file line by line; split into chunks of chunk_size lines; function_name is set to f"chunk_{start_line}"; text is the raw lines joined
• Use case: handles JavaScript, YAML, shell scripts, etc.

---

src/embedder.py

load_model(model_name: str) -> SentenceTransformer

• Inputs: model name string
• Output: loaded SentenceTransformer model
• Behavior: load from HuggingFace hub; cache locally (sentence-transformers handles this automatically); print Loading model: <name> to stdout

embed_functions(chunks: list[dict], model) -> tuple[list[dict], any]

• Inputs: list of chunk dicts, loaded model
• Output: (chunks, embeddings_array) where embeddings_array is a numpy float32 array of shape (N, D)
• Behavior: for each chunk, build embed text as f"{chunk['function_name']}: {chunk['text']}" (concatenate name + text for richer representation); call model.encode(texts, show_progress_bar=True)
• Edge cases: if chunks is empty, return ([], None)

---

src/indexer.py

build_index(chunks: list[dict], embeddings) -> faiss.Index

• Inputs: chunk list, numpy embeddings array
• Output: FAISS IndexFlatIP (inner product / cosine after normalization)
• Behavior: normalize embeddings with faiss.normalize_L2; build IndexFlatIP; add embeddings

save_index(index, chunks: list[dict], file_hashes: dict, index_dir: str) -> None

• Inputs: FAISS index, chunk metadata list, dict of {filepath: hash}, directory path
• Output: writes index.faiss, metadata.json, file_hashes.json to index_dir
• Behavior: faiss.write_index; json.dump metadata and hashes

load_index(index_dir: str) -> tuple[faiss.Index, list[dict], dict]

• Inputs: directory path
• Output: (index, chunks, file_hashes)
• Edge cases: if any file is missing, raise FileNotFoundError("Run --index first")

search(query: str, model, index, chunks: list[dict], top_k: int = 5) -> list[dict]

• Inputs: query string, loaded model, FAISS index, chunks list, k
• Output: list of result dicts {"file": str, "function_name": str, "lineno": int, "score": float, "text": str} sorted by score descending
• Behavior: embed query, normalize, call index.search; map FAISS indices back to chunk metadata

get_changed_files(directory: str, file_hashes: dict) -> tuple[list[str], list[str]]

• Inputs: directory to scan, existing hash dict
• Output: (changed_files, unchanged_files) — lists of file paths
• Behavior: walk directory; compute current hash for each file; compare to stored hash; new files counted as changed

---

src/main.py

CLI entry point using argparse.

Arguments:

• --index <dir> — index a directory; uses incremental logic by default
• --reindex — force full reindex (ignore existing hashes)
• --search "..." — search query; requires index to exist
• --compare "..." — run query against both EMBED_MODEL and COMPARE_MODEL; print side-by-side results
• --top-k N — number of results (default from env)

Behavior for --index:

1. Walk directory for .py files and other files
2. Load existing hashes (if any)
3. Skip unchanged files; log Skipped N files (unchanged)
4. Chunk changed/new files
5. Embed new chunks
6. Merge with existing index (or build fresh)
7. Save index
8. Print Indexed N functions from M files

Behavior for --search:

1. Load index
2. Embed query
3. Search
4. Print results as a table: Score | File | Function | Line

Behavior for --compare:

1. Run search with EMBED_MODEL — collect results with scores
2. Run search with COMPARE_MODEL — collect results with scores
3. Print side-by-side table showing rank differences

---

tests/ — what to test

Test 1 — AST chunker extracts function boundaries (test_chunker.py)

• Use tests/fixtures/sample_module.py which contains 5 named functions
• Call ast_chunk_python on it
• Assert exactly 5 chunks are returned
• Assert each chunk has function_name, lineno, end_lineno, text
• Assert no chunk's function_name starts with __

Test 2 — Incremental index skips unchanged files (test_indexer.py)

• Index sample_module.py once; save hashes
• Call get_changed_files with the same directory and the saved hashes
• Assert unchanged_files contains sample_module.py and changed_files is empty

Test 3 — Search returns results with file, function, and line (test_indexer.py)

• Build a small in-memory index from sample_module.py chunks
• Call search("database query")
• Assert each result dict has keys: file, function_name, lineno, score
• Assert scores are between 0 and 1

Test 4 — Heuristic chunker handles non-Python files (test_chunker.py)

• Call heuristic_chunk on tests/fixtures/sample_config.yaml
• Assert at least 1 chunk is returned
• Assert each chunk has lineno and text
• Assert function_name follows the chunk_N pattern

Test 5 — File hash changes when content changes (test_chunker.py)

• Compute hash of sample_module.py
• Write a modified version to a temp file
• Assert the two hashes are different

Test 6 — AST chunker handles syntax error gracefully (test_chunker.py)

• Create a temp .py file with invalid syntax
• Call ast_chunk_python on it
• Assert it returns [] without raising an exception

---

README.md content

# Semantic Codebase Search

Function-level semantic search over your codebase using local embeddings and FAISS.

## Quick start

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env

# Index your codebase
python src/main.py --index ./src

# Search semantically
python src/main.py --search "authentication middleware"

# Compare models on same query
python src/main.py --compare "error handling retry logic"

How it works

1. AST chunking — Python files are chunked at function boundaries; other files in 30-line blocks
2. Embedding — Each chunk's signature + docstring is embedded with sentence-transformers
3. FAISS — Inner-product index with L2-normalized vectors (equivalent to cosine similarity)
4. Incremental — File hashes detect changes; only re-embed modified files

Index structure

.index/
├── index.faiss       # FAISS binary index
├── metadata.json     # Chunk metadata (file, function, lineno)
└── file_hashes.json  # Per-file MD5 for incremental updates

Commands

Command	Description
--index <dir>	Index a directory (incremental by default)
--reindex	Force full re-index
--search "..."	Semantic search
--compare "..."	Compare two embedding models
--top-k N	Number of results (default: 5)

Running tests

pytest tests/ -v

---

## GUIDE.md content

```markdown
# Build Guide — Semantic Codebase Search

## Step 1 — AST chunking

Use Python's `ast` module to walk function definitions:

```python
import ast

tree = ast.parse(source)
for node in ast.walk(tree):
    if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
        # node.name, node.lineno, node.end_lineno
        docstring = ast.get_docstring(node) or ""
        first_line = source_lines[node.lineno - 1]
        text = f"{first_line}\n{docstring}"

The key insight: embed the signature + docstring, not the full body. The docstring describes intent; the signature describes interface. Together they're what a programmer searches for. The body is implementation detail.

Step 2 — Incremental indexing

Hash each file with MD5:

import hashlib

def file_hash(path):
    with open(path, "rb") as f:
        return hashlib.md5(f.read()).hexdigest()

Store {filepath: hash} as JSON. On next run, compare. Only chunk + embed files whose hash changed.

Step 3 — FAISS with cosine similarity

For cosine similarity with FAISS's IndexFlatIP:

import faiss
import numpy as np

embeddings = model.encode(texts)
faiss.normalize_L2(embeddings)  # convert to unit vectors
index = faiss.IndexFlatIP(embeddings.shape[1])
index.add(embeddings)

For search, normalize the query vector the same way before calling index.search.

Step 4 — Model comparison

Run the same query twice with different models. The difference in rankings reveals:

• Code-tuned models rank syntactic matches higher
• General models rank semantic/intent matches higher

For the --compare output, show both ranked lists and highlight rank position changes.

Debugging tips

• If search returns nonsense, print the chunk texts being embedded — you may be embedding empty strings
• If FAISS index fails to load, check that metadata.json and index.faiss have the same number of entries
• The heuristic chunker is intentionally simple; for production you'd use tree-sitter for JS/TS

How to talk about this in an interview

"Why function-level chunking instead of file-level?"

File-level embedding loses signal — a 500-line file has many unrelated functions. Function-level gives you precise retrieval: you get the exact function that answers the query, not the whole file you have to scan.

"How does incremental indexing work?"

I hash each file with MD5. On re-run I compare current hashes to stored hashes. Only changed or new files get re-embedded. This keeps re-indexing fast enough to run as a pre-commit hook.

"What's the score difference between models?"

Code-tuned models (like CodeBERT) are better at matching identifier names and API patterns. General models are better at matching intent described in comments and docstrings. The --compare mode makes this concrete for any query.

---

## Failure modes to handle

| Failure | Where | How to handle |
|---|---|---|
| Python file has syntax error | `chunker.py` | Log warning with filename; return `[]` for that file; continue |
| FAISS index missing on `--search` | `indexer.py` | Raise `FileNotFoundError` with message "Run --index first" |
| HuggingFace model download fails | `embedder.py` | Catch `OSError`; print "Model download failed. Check network." and exit 1 |
| Directory has no Python files | `main.py` | Print "No indexable files found in <dir>." and exit 0 |
| Chunk list is empty after filtering | `indexer.py` | Skip FAISS build; print "Nothing new to index." |
| `file_hashes.json` is corrupted | `indexer.py` | Log warning; treat as if no hashes exist; force full reindex |

---

## The metric this project measures

**What is measured:** Index size (number of functions indexed), search latency (ms), and incremental skip rate (% of files skipped on re-run).

**Format (stdout):**

Indexed 47 functions from 12 files | skipped=8 unchanged | latency=2.3s Search completed in 12ms | top score=0.847

**Target:** On a re-run with no file changes, skip rate must be 100% (zero files re-embedded). Search must complete in under 100ms for an index of up to 500 functions. These two numbers together demonstrate that the incremental and retrieval logic are both functioning correctly.


## Model source currency

- Anthropic model list: https://docs.anthropic.com/en/docs/about-claude/models/overview
- OpenAI model list: https://platform.openai.com/docs/models
- verifiedOn: 2026-08-07
- `LLM_MODEL` is required and has no implicit default; choose a supported model for the configured provider at setup time.