AI-Augmented Engineering

Field	Value
Path	2 — AI-Augmented Engineering
Position	1 of 12
Difficulty	🟢 Local only
Estimated time	4 hours
AWS cost	None

---

Agent Pickup Instructions

# 1. Clone / enter project
cd projects/p2-01-code-review-bot

# 2. Create virtual environment
python -m venv .venv && source .venv/bin/activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Copy env file and fill in your API key
cp .env.example .env

# 5. Run against a sample diff
git diff HEAD~1 > /tmp/recent.diff
python src/main.py --diff /tmp/recent.diff --output table

# 6. Run tests
pytest tests/ -v

Done when:

• python src/main.py --diff tests/fixtures/hardcoded_secret.py.diff --min-severity HIGH prints at least one HIGH finding
• JSON output (--output json) is valid JSON with the schema: [{file, line_range, severity, category, finding, suggestion}]
• All 4 tests pass
• Three sample test fixtures each produce at least one finding

---

What this project is

A command-line tool that reads a git diff (from a file or stdin), splits it into per-file chunks, sends each chunk to an LLM with a structured JSON prompt, and returns a prioritized list of code-review findings with file, line range, severity (HIGH/MEDIUM/LOW), category (security/performance/correctness/style), human-readable finding, and a concrete suggestion. Large diffs are handled automatically by chunking per file so no single LLM call exceeds the context window.

---

What the learner achieves

"I built a structured code-review pipeline that parses raw git diffs, chunks them per file, calls an LLM in JSON mode to classify findings by severity and category, and outputs actionable results — which I used to catch a hardcoded secret, an N+1 query, and an off-by-one error in a controlled test suite."

---

Folder structure

p2-01-code-review-bot/
├── src/
│   ├── main.py          # CLI entry point — argparse, orchestration
│   ├── llm.py           # Provider-agnostic LLM wrapper
│   ├── reviewer.py      # chunk_diff_by_file, review_chunk, merge_results
│   └── reporter.py      # print_table, save_json
├── tests/
│   ├── fixtures/
│   │   ├── hardcoded_secret.py   # Contains: password = "hunter2"
│   │   ├── n_plus_one.py         # Contains: for user in users: db.query(...)
│   │   └── off_by_one.py         # Contains: for i in range(len(items)+1)
│   ├── test_reviewer.py
│   ├── test_reporter.py
│   └── test_integration.py
├── .env.example
├── requirements.txt
└── README.md

---

.env.example

# LLM provider: anthropic | openai | ollama
LLM_PROVIDER=anthropic

# Model to use (provider-specific)
# anthropic: configured Anthropic model
# openai:    current OpenAI model
# ollama:    codellama
LLM_MODEL=

# API key for Anthropic (leave blank if using ollama)
ANTHROPIC_API_KEY=

# API key for OpenAI (leave blank if using anthropic or ollama)
OPENAI_API_KEY=

# Ollama base URL (only needed if LLM_PROVIDER=ollama)
OLLAMA_BASE_URL=http://localhost:11434

# Max lines per diff chunk sent to LLM (increase if model supports large context)
MAX_CHUNK_LINES=300

# Minimum severity to show: HIGH | MEDIUM | LOW
MIN_SEVERITY=LOW

---

requirements.txt

anthropic==0.30.0
openai==1.35.0
python-dotenv==1.0.1
rich==13.7.1
pytest==8.2.2
pytest-mock==3.14.0
tabulate==0.9.0

---

src/ — what to implement

src/llm.py

get_completion(prompt: str, system: str = "") -> str

• Inputs: user prompt string, optional system prompt string
• Output: raw string response from the LLM
• Behavior: reads LLM_PROVIDER and LLM_MODEL from env; routes to Anthropic, OpenAI, or Ollama accordingly; always requests JSON mode / instructs model to return only valid JSON
• Edge cases: raise RuntimeError with provider name if API key is missing; if Ollama is unreachable, raise ConnectionError with the base URL

get_json_completion(prompt: str, system: str = "") -> dict | list

• Inputs: same as get_completion
• Output: parsed Python object (dict or list)
• Behavior: calls get_completion, strips markdown code fences if present, parses JSON; raises ValueError with the raw text if parsing fails

---

src/reviewer.py

chunk_diff_by_file(diff_text: str) -> list[dict]

• Inputs: raw unified diff string
• Output: list of {"filename": str, "diff": str, "lines": int} — one entry per file in the diff
• Behavior: split on diff --git headers; each chunk contains only the lines for that file; compute line count from + and - lines (not context lines)
• Edge cases: empty diff returns []; binary file entries (no @@ header) are skipped with a log message

review_chunk(chunk: dict, min_severity: str = "LOW") -> list[dict]

• Inputs: one chunk dict from chunk_diff_by_file, min severity string
• Output: list of finding dicts matching schema {file, line_range, severity, category, finding, suggestion}
• Behavior: build system prompt instructing LLM to return a JSON array of findings; inject the chunk diff as user content; call get_json_completion; filter by min_severity after receiving results
• Edge cases: if LLM returns empty array, return []; if JSON parse fails, log warning and return []; never propagate exceptions to caller

merge_results(results_per_chunk: list[list[dict]]) -> list[dict]

• Inputs: list of per-chunk finding lists
• Output: flat sorted list — HIGH first, then MEDIUM, then LOW; within same severity, alphabetical by filename
• Behavior: flatten and sort; deduplicate exact duplicates (same file + line_range + category)

---

src/reporter.py

print_table(findings: list[dict]) -> None

• Inputs: list of finding dicts
• Output: prints a formatted table to stdout using rich or tabulate
• Behavior: columns are Severity, File, Lines, Category, Finding (truncated to 60 chars), Suggestion (truncated to 60 chars); HIGH rows printed in red, MEDIUM in yellow, LOW in default color

save_json(findings: list[dict], output_path: str) -> None

• Inputs: list of finding dicts, file path
• Output: writes JSON file; also prints Saved N findings to <path> to stdout
• Edge cases: if output directory does not exist, create it with os.makedirs

count_by_severity(findings: list[dict]) -> dict

• Inputs: list of finding dicts
• Output: {"HIGH": int, "MEDIUM": int, "LOW": int}
• Behavior: count each severity; missing severity keys default to 0

---

src/main.py

CLI entry point using argparse.

Arguments:

• --diff <file> — path to diff file; if omitted, reads from stdin
• --min-severity HIGH|MEDIUM|LOW — default LOW
• --output json|table — default table; json writes to review_output.json
• --json-out <path> — explicit path for JSON output

Behavior:

1. Load .env with python-dotenv
2. Read diff text from file or stdin
3. Call chunk_diff_by_file
4. For each chunk, call review_chunk (sequential, not parallel)
5. Call merge_results
6. Print summary: Found N findings (X HIGH, Y MEDIUM, Z LOW)
7. Log this summary line to stdout before rendering output
8. Render via print_table or save_json based on --output
9. Exit code 1 if any HIGH findings found, 0 otherwise

---

tests/ — what to test

Test 1 — Structured output parsing (test_reviewer.py)

• Feed review_chunk a mock LLM response containing a JSON array with one HIGH security finding
• Assert the returned list has exactly one item with severity == "HIGH" and category == "security"
• The LLM is mocked; the test exercises the parsing and filtering logic, not the network call

Test 2 — Severity filter (test_reviewer.py)

• Mock LLM returns findings of all three severities
• Call review_chunk with min_severity="HIGH"
• Assert only the HIGH finding is returned (MEDIUM and LOW filtered out)

Test 3 — Large diff chunking (test_reviewer.py)

• Create a synthetic diff with 3 files
• Call chunk_diff_by_file on it
• Assert result has exactly 3 chunks, each with the correct filename
• Assert no chunk's diff string contains lines from another file

Test 4 — Known issue detection — hardcoded secret (test_integration.py)

• Read tests/fixtures/hardcoded_secret.py and generate a synthetic diff for it
• Run through review_chunk with a real LLM call (skip if ANTHROPIC_API_KEY not set)
• Assert at least one finding has category == "security"

Test 5 — reporter count (test_reporter.py)

• Call count_by_severity with a list containing 2 HIGH, 1 MEDIUM, 0 LOW
• Assert {"HIGH": 2, "MEDIUM": 1, "LOW": 0}

Test 6 — JSON output roundtrip (test_reporter.py)

• Call save_json with 2 findings to a temp path
• Read the file back, parse JSON
• Assert the list length is 2 and the severity field is preserved exactly

---

README.md content

# Code Review Bot

CLI that reads a git diff and returns structured code-review findings.

## Quick start

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env   # add your API key
git diff HEAD~1 | python src/main.py --output table

Usage

# Review a saved diff
python src/main.py --diff my.diff

# Only show HIGH severity findings
python src/main.py --diff my.diff --min-severity HIGH

# Output as JSON
python src/main.py --diff my.diff --output json --json-out results.json

Finding schema

Field	Values
severity	HIGH / MEDIUM / LOW
category	security / performance / correctness / style
file	relative path from diff header
line_range	e.g. "42-55"
finding	human-readable description
suggestion	concrete fix

Exit codes

• 0 — no HIGH findings
• 1 — at least one HIGH finding (use in CI to block merges)

Running tests

pytest tests/ -v

---

## GUIDE.md content

```markdown
# Build Guide — Code Review Bot

## Step 1 — Parse the diff (chunker.py logic)

A unified diff looks like:

diff --git a/auth.py b/auth.py index abc..def 100644 --- a/auth.py +++ b/auth.py @@ -10,6 +10,8 @@ def login():

• password = "hunter2"

Split on `diff --git` lines. Each resulting chunk is one file review unit.

## Step 2 — Build the LLM prompt

System prompt:

You are a senior engineer performing a security and quality code review. Return ONLY a JSON array. Each element must have: file, line_range, severity (HIGH|MEDIUM|LOW), category (security|performance|correctness|style), finding, suggestion. If there are no issues, return [].

User content: the raw diff chunk for one file.

## Step 3 — Parse and filter

- Strip any ```json ... ``` fences before parsing
- Filter by min_severity after receiving results
- If JSON parse fails: log a warning, return [] (never crash)

## Step 4 — Merge and display

- Flatten all per-chunk results
- Sort: HIGH > MEDIUM > LOW
- Deduplicate on (file, line_range, category)

## Step 5 — Wire the CLI

Use argparse. Reading from stdin (`sys.stdin.read()`) lets users pipe:
```bash
git diff HEAD~1 | python src/main.py

Debugging tips

• Print the raw LLM response before parsing to diagnose JSON errors
• Use --min-severity HIGH when developing to reduce noise
• Start with a tiny 5-line diff to confirm JSON mode is working

How to talk about this in an interview

"What problem does this solve?"

Manual code review is inconsistent and slow. This bot gives a first-pass review in seconds, catches classes of issues humans miss when fatigued (hardcoded secrets, N+1 patterns), and integrates into CI via exit codes.

"How did you handle large diffs?"

I chunk per file. Each LLM call sees only one file's changes, so context window limits never apply to the whole PR.

"How do you ensure the output is structured?"

System prompt enforces a JSON schema. After receiving the response, I validate each required key exists and the severity value is one of the three allowed values before showing results.

"How would you make this production-grade?"

Add a schema validation step (jsonschema), cache results per file hash to avoid re-reviewing unchanged files, and publish findings to a GitHub PR comment via PyGithub.

---

## Failure modes to handle

| Failure | Where | How to handle |
|---|---|---|
| LLM returns prose instead of JSON | `reviewer.py` | Strip markdown fences, attempt parse; if still fails log warning and return `[]` |
| Diff chunk exceeds model context window | `reviewer.py` | Split oversized chunk by `@@` hunk boundaries before retrying |
| Binary file in diff (no `@@` header) | `reviewer.py` / `chunker` | Detect by absence of `@@`; skip with log `Skipping binary file: <name>` |
| API key missing | `llm.py` | Raise `RuntimeError("ANTHROPIC_API_KEY not set")` immediately |
| Empty diff passed | `main.py` | Print `No changes found in diff.` and exit 0 |
| JSON has wrong severity value | `reviewer.py` | Normalize to uppercase; if not in `{HIGH,MEDIUM,LOW}` set to MEDIUM |

---

## The metric this project measures

**What is measured:** Number of findings at each severity level per diff reviewed.

**Format (stdout):**

Found 3 findings (1 HIGH, 1 MEDIUM, 1 LOW)

**Target:** Against the three fixture files (hardcoded_secret.py, n_plus_one.py, off_by_one.py), the bot must produce at least 3 findings total with at least 1 HIGH (the secret) and at least 1 MEDIUM or higher (the N+1 or off-by-one). This proves the structured output pipeline is functional and the LLM is actually analyzing code semantics.


## Model source currency

- Anthropic model list: https://docs.anthropic.com/en/docs/about-claude/models/overview
- OpenAI model list: https://platform.openai.com/docs/models
- verifiedOn: 2026-08-07
- `LLM_MODEL` is required and has no implicit default; choose a supported model for the configured provider at setup time.