---
title: "Semantic Codebase Search"
description: "Demonstrates embeddings + vector search applied to a problem every engineer already has — a concrete talking point for Developer Productivity or Platform..."
source: "https://confidentprep.com/paths/ai-augmented-engineering/semantic-codebase-search/"
path: "AI-Augmented Engineering"
repository: "https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/semantic-codebase-search"
token_estimate: 4309
---

# Semantic Codebase Search

## Overview

Indexes a Python codebase at function granularity using AST chunking and FAISS, skips
unchanged files via file hashing on re-index, and compares two embedding models side by
side on the same query. `grep` finds text; this finds intent — and the re-index/hashing
logic is the same pattern real production search tools use to stay fast on large repos.

## What to do

| Field | Value |
|---|---|
| Path | 2 — AI-Augmented Engineering |
| Position | 3 of 12 |
| Difficulty | 🟢 Local only |
| Estimated time | 4 hours |
| AWS cost | None |

---

### Agent Pickup Instructions

```bash
# 1. Enter project
cd projects/p2-03-semantic-codebase-search

# 2. Create virtual environment
python -m venv .venv && source .venv/bin/activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Copy env file
cp .env.example .env

# 5. Index a Python codebase
python src/main.py --index ./src

# 6. Search semantically
python src/main.py --search "database connection pooling"

# 7. Compare models on a query
python src/main.py --compare "authentication middleware"

# 8. Run tests
pytest tests/ -v
```

**Done when:**
- [ ] `--index` processes a directory and prints `Indexed N functions from M files`
- [ ] `--search` returns results with file path, function name, and line number
- [ ] Re-running `--index` on unchanged files prints `Skipped N files (unchanged)`
- [ ] `--compare` shows two scored result lists side by side
- [ ] All 4 tests pass

---

### What this project is

A local semantic search engine for Python codebases that operates at function granularity. It uses AST parsing to extract function boundaries, embeds each function's signature and docstring with a sentence-transformer model, stores vectors in a FAISS index, and supports both AST-based (Python) and heuristic (non-Python) chunking. Incremental indexing via file hash comparison avoids re-embedding unchanged files. A comparison mode runs the same query against two different embedding models and displays the ranked score difference, letting engineers see concretely how code-tuned models differ from general-purpose ones.

---

### What the learner achieves

"I built a function-level semantic search engine for codebases using AST chunking, FAISS vector storage, and incremental indexing with file hashing — and I can demonstrate the concrete score difference between a code-tuned and a general-purpose embedding model."

---

### Folder structure

```
p2-03-semantic-codebase-search/
├── src/
│   ├── main.py        # CLI: --index, --search, --compare, --reindex
│   ├── llm.py         # Provider-agnostic LLM wrapper (for future extension)
│   ├── embedder.py    # load_model, embed_functions
│   ├── chunker.py     # ast_chunk_python, heuristic_chunk, file_hash
│   └── indexer.py     # build_index, load_index, search, save_index
├── tests/
│   ├── fixtures/
│   │   ├── sample_module.py      # 5 functions with docstrings
│   │   └── sample_config.yaml    # Non-Python file for heuristic test
│   ├── test_chunker.py
│   ├── test_indexer.py
│   └── test_embedder.py
├── .index/                        # Auto-created: faiss files + metadata
│   ├── index.faiss
│   ├── metadata.json
│   └── file_hashes.json
├── .env.example
├── requirements.txt
└── README.md
```

---

### .env.example

```bash
# Primary embedding model (code-tuned)
# Options: microsoft/codebert-base | all-MiniLM-L6-v2 | sentence-transformers/all-mpnet-base-v2
EMBED_MODEL=all-MiniLM-L6-v2

# Comparison model (general-purpose) — used only with --compare flag
COMPARE_MODEL=all-MiniLM-L6-v2

# Path to store the FAISS index and metadata
INDEX_DIR=.index

# Number of results to return per search
TOP_K=5

# Minimum score threshold to display a result (0.0 to 1.0)
MIN_SCORE=0.0

# LLM provider (not used for search — included for future LLM-assisted reranking)
LLM_PROVIDER=anthropic
LLM_MODEL=
ANTHROPIC_API_KEY=
OPENAI_API_KEY=
OLLAMA_BASE_URL=http://localhost:11434
```

---

### requirements.txt

```
sentence-transformers==3.0.1
faiss-cpu==1.8.0
numpy==1.26.4
python-dotenv==1.0.1
rich==13.7.1
pytest==8.2.2
pytest-mock==3.14.0
anthropic==0.30.0
openai==1.35.0
```

---

### src/ — what to implement

#### `src/llm.py`

**`get_completion(prompt: str, system: str = "") -> str`**
- Standard interface: reads `LLM_PROVIDER` and `LLM_MODEL` from env
- Not used in search pipeline — included for future LLM-assisted reranking extension
- Same implementation pattern as p2-01

---

#### `src/chunker.py`

**`file_hash(filepath: str) -> str`**
- Inputs: absolute file path
- Output: MD5 hex digest of the file contents
- Behavior: read file in binary mode, compute `hashlib.md5`
- Edge cases: if file cannot be read, raise `FileNotFoundError`

**`ast_chunk_python(filepath: str) -> list[dict]`**
- Inputs: absolute path to a `.py` file
- Output: list of chunk dicts, each with keys: `{"function_name": str, "file": str, "lineno": int, "end_lineno": int, "text": str}`
- Behavior: parse with `ast.parse`; walk all `FunctionDef` and `AsyncFunctionDef` nodes; for each, extract the function name, start/end line numbers, and extract the raw source lines using `linecache` or `inspect.getsource` pattern; `text` = function signature (first line) + docstring (if present) — do NOT include the full body
- Edge cases: file with syntax errors logs a warning and returns `[]`; skip nested functions (only top-level and class-level methods); skip `__` dunder methods

**`heuristic_chunk(filepath: str, chunk_size: int = 30) -> list[dict]`**
- Inputs: absolute path to any file, lines per chunk
- Output: list of chunk dicts with `{"file": str, "lineno": int, "end_lineno": int, "text": str, "function_name": str}`
- Behavior: read file line by line; split into chunks of `chunk_size` lines; `function_name` is set to `f"chunk_{start_line}"`; `text` is the raw lines joined
- Use case: handles JavaScript, YAML, shell scripts, etc.

---

#### `src/embedder.py`

**`load_model(model_name: str) -> SentenceTransformer`**
- Inputs: model name string
- Output: loaded `SentenceTransformer` model
- Behavior: load from HuggingFace hub; cache locally (sentence-transformers handles this automatically); print `Loading model: <name>` to stdout

**`embed_functions(chunks: list[dict], model) -> tuple[list[dict], any]`**
- Inputs: list of chunk dicts, loaded model
- Output: `(chunks, embeddings_array)` where `embeddings_array` is a numpy float32 array of shape `(N, D)`
- Behavior: for each chunk, build embed text as `f"{chunk['function_name']}: {chunk['text']}"` (concatenate name + text for richer representation); call `model.encode(texts, show_progress_bar=True)`
- Edge cases: if `chunks` is empty, return `([], None)`

---

#### `src/indexer.py`

**`build_index(chunks: list[dict], embeddings) -> faiss.Index`**
- Inputs: chunk list, numpy embeddings array
- Output: FAISS `IndexFlatIP` (inner product / cosine after normalization)
- Behavior: normalize embeddings with `faiss.normalize_L2`; build `IndexFlatIP`; add embeddings

**`save_index(index, chunks: list[dict], file_hashes: dict, index_dir: str) -> None`**
- Inputs: FAISS index, chunk metadata list, dict of `{filepath: hash}`, directory path
- Output: writes `index.faiss`, `metadata.json`, `file_hashes.json` to `index_dir`
- Behavior: `faiss.write_index`; `json.dump` metadata and hashes

**`load_index(index_dir: str) -> tuple[faiss.Index, list[dict], dict]`**
- Inputs: directory path
- Output: `(index, chunks, file_hashes)`
- Edge cases: if any file is missing, raise `FileNotFoundError("Run --index first")`

**`search(query: str, model, index, chunks: list[dict], top_k: int = 5) -> list[dict]`**
- Inputs: query string, loaded model, FAISS index, chunks list, k
- Output: list of result dicts `{"file": str, "function_name": str, "lineno": int, "score": float, "text": str}` sorted by score descending
- Behavior: embed query, normalize, call `index.search`; map FAISS indices back to chunk metadata

**`get_changed_files(directory: str, file_hashes: dict) -> tuple[list[str], list[str]]`**
- Inputs: directory to scan, existing hash dict
- Output: `(changed_files, unchanged_files)` — lists of file paths
- Behavior: walk directory; compute current hash for each file; compare to stored hash; new files counted as changed

---

#### `src/main.py`

CLI entry point using `argparse`.

Arguments:
- `--index <dir>` — index a directory; uses incremental logic by default
- `--reindex` — force full reindex (ignore existing hashes)
- `--search "..."` — search query; requires index to exist
- `--compare "..."` — run query against both `EMBED_MODEL` and `COMPARE_MODEL`; print side-by-side results
- `--top-k N` — number of results (default from env)

Behavior for `--index`:
1. Walk directory for `.py` files and other files
2. Load existing hashes (if any)
3. Skip unchanged files; log `Skipped N files (unchanged)`
4. Chunk changed/new files
5. Embed new chunks
6. Merge with existing index (or build fresh)
7. Save index
8. Print `Indexed N functions from M files`

Behavior for `--search`:
1. Load index
2. Embed query
3. Search
4. Print results as a table: Score | File | Function | Line

Behavior for `--compare`:
1. Run search with `EMBED_MODEL` — collect results with scores
2. Run search with `COMPARE_MODEL` — collect results with scores
3. Print side-by-side table showing rank differences

---

### tests/ — what to test

#### Test 1 — AST chunker extracts function boundaries (`test_chunker.py`)
- Use `tests/fixtures/sample_module.py` which contains 5 named functions
- Call `ast_chunk_python` on it
- Assert exactly 5 chunks are returned
- Assert each chunk has `function_name`, `lineno`, `end_lineno`, `text`
- Assert no chunk's `function_name` starts with `__`

#### Test 2 — Incremental index skips unchanged files (`test_indexer.py`)
- Index `sample_module.py` once; save hashes
- Call `get_changed_files` with the same directory and the saved hashes
- Assert `unchanged_files` contains `sample_module.py` and `changed_files` is empty

#### Test 3 — Search returns results with file, function, and line (`test_indexer.py`)
- Build a small in-memory index from `sample_module.py` chunks
- Call `search("database query")`
- Assert each result dict has keys: `file`, `function_name`, `lineno`, `score`
- Assert scores are between 0 and 1

#### Test 4 — Heuristic chunker handles non-Python files (`test_chunker.py`)
- Call `heuristic_chunk` on `tests/fixtures/sample_config.yaml`
- Assert at least 1 chunk is returned
- Assert each chunk has `lineno` and `text`
- Assert `function_name` follows the `chunk_N` pattern

#### Test 5 — File hash changes when content changes (`test_chunker.py`)
- Compute hash of `sample_module.py`
- Write a modified version to a temp file
- Assert the two hashes are different

#### Test 6 — AST chunker handles syntax error gracefully (`test_chunker.py`)
- Create a temp `.py` file with invalid syntax
- Call `ast_chunk_python` on it
- Assert it returns `[]` without raising an exception

---

### README.md content

```markdown
# Semantic Codebase Search

Function-level semantic search over your codebase using local embeddings and FAISS.

## Quick start

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env

## Index your codebase
python src/main.py --index ./src

## Search semantically
python src/main.py --search "authentication middleware"

## Compare models on same query
python src/main.py --compare "error handling retry logic"
```

## How it works

1. **AST chunking** — Python files are chunked at function boundaries; other files in 30-line blocks
2. **Embedding** — Each chunk's signature + docstring is embedded with sentence-transformers
3. **FAISS** — Inner-product index with L2-normalized vectors (equivalent to cosine similarity)
4. **Incremental** — File hashes detect changes; only re-embed modified files

## Index structure

```
.index/
├── index.faiss       # FAISS binary index
├── metadata.json     # Chunk metadata (file, function, lineno)
└── file_hashes.json  # Per-file MD5 for incremental updates
```

## Commands

| Command | Description |
|---|---|
| `--index <dir>` | Index a directory (incremental by default) |
| `--reindex` | Force full re-index |
| `--search "..."` | Semantic search |
| `--compare "..."` | Compare two embedding models |
| `--top-k N` | Number of results (default: 5) |

## Running tests

```bash
pytest tests/ -v
```
```

---

### GUIDE.md content

```markdown
# Build Guide — Semantic Codebase Search

## Step 1 — AST chunking

Use Python's `ast` module to walk function definitions:

```python
import ast

tree = ast.parse(source)
for node in ast.walk(tree):
    if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
        # node.name, node.lineno, node.end_lineno
        docstring = ast.get_docstring(node) or ""
        first_line = source_lines[node.lineno - 1]
        text = f"{first_line}\n{docstring}"
```

The key insight: embed the **signature + docstring**, not the full body. The docstring describes intent; the signature describes interface. Together they're what a programmer searches for. The body is implementation detail.

## Step 2 — Incremental indexing

Hash each file with MD5:

```python
import hashlib

def file_hash(path):
    with open(path, "rb") as f:
        return hashlib.md5(f.read()).hexdigest()
```

Store `{filepath: hash}` as JSON. On next run, compare. Only chunk + embed files whose hash changed.

## Step 3 — FAISS with cosine similarity

For cosine similarity with FAISS's `IndexFlatIP`:

```python
import faiss
import numpy as np

embeddings = model.encode(texts)
faiss.normalize_L2(embeddings)  # convert to unit vectors
index = faiss.IndexFlatIP(embeddings.shape[1])
index.add(embeddings)
```

For search, normalize the query vector the same way before calling `index.search`.

## Step 4 — Model comparison

Run the same query twice with different models. The difference in rankings reveals:
- Code-tuned models rank syntactic matches higher
- General models rank semantic/intent matches higher

For the `--compare` output, show both ranked lists and highlight rank position changes.

## Debugging tips

- If search returns nonsense, print the chunk texts being embedded — you may be embedding empty strings
- If FAISS index fails to load, check that `metadata.json` and `index.faiss` have the same number of entries
- The heuristic chunker is intentionally simple; for production you'd use tree-sitter for JS/TS

## How to talk about this in an interview

**"Why function-level chunking instead of file-level?"**
> File-level embedding loses signal — a 500-line file has many unrelated functions. Function-level gives you precise retrieval: you get the exact function that answers the query, not the whole file you have to scan.

**"How does incremental indexing work?"**
> I hash each file with MD5. On re-run I compare current hashes to stored hashes. Only changed or new files get re-embedded. This keeps re-indexing fast enough to run as a pre-commit hook.

**"What's the score difference between models?"**
> Code-tuned models (like CodeBERT) are better at matching identifier names and API patterns. General models are better at matching intent described in comments and docstrings. The `--compare` mode makes this concrete for any query.
```

---

### Failure modes to handle

| Failure | Where | How to handle |
|---|---|---|
| Python file has syntax error | `chunker.py` | Log warning with filename; return `[]` for that file; continue |
| FAISS index missing on `--search` | `indexer.py` | Raise `FileNotFoundError` with message "Run --index first" |
| HuggingFace model download fails | `embedder.py` | Catch `OSError`; print "Model download failed. Check network." and exit 1 |
| Directory has no Python files | `main.py` | Print "No indexable files found in <dir>." and exit 0 |
| Chunk list is empty after filtering | `indexer.py` | Skip FAISS build; print "Nothing new to index." |
| `file_hashes.json` is corrupted | `indexer.py` | Log warning; treat as if no hashes exist; force full reindex |

---

### The metric this project measures

**What is measured:** Index size (number of functions indexed), search latency (ms), and incremental skip rate (% of files skipped on re-run).

**Format (stdout):**
```
Indexed 47 functions from 12 files | skipped=8 unchanged | latency=2.3s
Search completed in 12ms | top score=0.847
```

**Target:** On a re-run with no file changes, skip rate must be 100% (zero files re-embedded). Search must complete in under 100ms for an index of up to 500 functions. These two numbers together demonstrate that the incremental and retrieval logic are both functioning correctly.


### Model source currency

- Anthropic model list: https://docs.anthropic.com/en/docs/about-claude/models/overview
- OpenAI model list: https://platform.openai.com/docs/models
- verifiedOn: 2026-08-07
- `LLM_MODEL` is required and has no implicit default; choose a supported model for the configured provider at setup time.

## Source code

A reference implementation of Semantic Codebase Search — README, guide, source and tests — is at https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/semantic-codebase-search.