---
title: "PR Diff Summarizer"
description: "A small, concrete example of AI-augmented developer productivity tooling — exactly the kind of internal tool that gets a senior engineer noticed for AI..."
source: "https://confidentprep.com/paths/ai-augmented-engineering/pr-diff-summarizer/"
path: "AI-Augmented Engineering"
repository: "https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/pr-diff-summarizer"
token_estimate: 3666
---

# PR Diff Summarizer

## Overview

Turns a git diff into a plain-English summary, an architecture impact paragraph, and a
test coverage flag — with zero raw code lines in the output, forcing the model to
actually explain rather than just paraphrase. Same input as the last project (a diff),
different output shape — proof that one well-understood input can power multiple tools.

## What to do

| Field | Value |
|---|---|
| Path | 2 — AI-Augmented Engineering |
| Position | 2 of 12 |
| Difficulty | 🟢 Local only |
| Estimated time | 3 hours |
| AWS cost | None |

---

### Agent Pickup Instructions

```bash
# 1. Enter project
cd projects/p2-02-pr-diff-summarizer

# 2. Create virtual environment
python -m venv .venv && source .venv/bin/activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Copy env file and fill in your API key
cp .env.example .env

# 5. Run against a sample diff
git diff HEAD~1 > /tmp/recent.diff
python src/main.py --diff /tmp/recent.diff --title "Add OAuth login flow"

# 6. Run tests
pytest tests/ -v
```

**Done when:**
- [ ] `python src/main.py --diff <file>` outputs three clearly labeled sections: plain-English summary, architecture impact, and test coverage flag
- [ ] A diff that modifies only `src/` files (no `test_` files) triggers `test_coverage_flag: true`
- [ ] All 4 tests pass
- [ ] Output contains no raw code snippets (no `+` / `-` diff lines in the summary text)

---

### What this project is

A command-line tool that takes a git diff plus optional PR title and reviewer comments and produces three audience-aware outputs: a plain-English summary written for non-technical stakeholders, an architecture impact paragraph describing what changes in the system design, and a test coverage flag that signals whether the diff adds or modifies tests. Unlike a raw code review, the emphasis is on communication — the output should be readable by a product manager or engineering manager without any coding knowledge.

---

### What the learner achieves

"I built an audience-aware diff summarization pipeline that separates plain-English communication from technical analysis, detects test coverage gaps without an LLM, and produces a structured report a product manager can actually read."

---

### Folder structure

```
p2-02-pr-diff-summarizer/
├── src/
│   ├── main.py          # CLI entry point — argparse, orchestration
│   ├── llm.py           # Provider-agnostic LLM wrapper
│   └── summarizer.py    # parse_diff_stats, detect_test_changes,
│                        # generate_summary, generate_arch_impact
├── tests/
│   ├── fixtures/
│   │   ├── with_tests.diff       # Diff that includes test_ files
│   │   ├── no_tests.diff         # Diff with no test_ files
│   │   └── large_refactor.diff   # Multi-file refactor diff
│   ├── test_summarizer.py
│   └── test_integration.py
├── .env.example
├── requirements.txt
└── README.md
```

---

### .env.example

```bash
# LLM provider: anthropic | openai | ollama
LLM_PROVIDER=anthropic

# Model to use (provider-specific)
LLM_MODEL=

# API key for Anthropic (leave blank if using ollama)
ANTHROPIC_API_KEY=

# API key for OpenAI (leave blank if using anthropic or ollama)
OPENAI_API_KEY=

# Ollama base URL (only needed if LLM_PROVIDER=ollama)
OLLAMA_BASE_URL=http://localhost:11434

# Maximum length of summary in words (enforced in prompt)
SUMMARY_MAX_WORDS=150

# Architecture paragraph max words
ARCH_MAX_WORDS=100
```

---

### requirements.txt

```
anthropic==0.30.0
openai==1.35.0
python-dotenv==1.0.1
rich==13.7.1
pytest==8.2.2
pytest-mock==3.14.0
```

---

### src/ — what to implement

#### `src/llm.py`

**`get_completion(prompt: str, system: str = "") -> str`**
- Inputs: user prompt string, optional system prompt string
- Output: raw string response from the LLM
- Behavior: reads `LLM_PROVIDER` and `LLM_MODEL` from env; routes to Anthropic, OpenAI, or Ollama; returns the assistant message text
- Edge cases: raise `RuntimeError` with provider name if API key is missing

---

#### `src/summarizer.py`

**`parse_diff_stats(diff_text: str) -> dict`**
- Inputs: raw unified diff string
- Output: `{"files_changed": int, "lines_added": int, "lines_removed": int, "filenames": list[str]}`
- Behavior: count lines starting with `+` (not `+++`) for added; `-` (not `---`) for removed; extract filenames from `diff --git a/<path>` headers
- Edge cases: empty diff returns all zeros with empty list; binary files counted in `files_changed` but add 0 lines

**`detect_test_changes(diff_text: str) -> bool`**
- Inputs: raw unified diff string
- Output: `True` if no test files are modified or added; `False` if at least one test file is present
- Behavior: a test file is any filename matching `test_*.py`, `*_test.py`, or paths containing `/tests/`; scan filenames extracted from diff headers
- Note: returns `True` (coverage flag raised) when tests are ABSENT — this flags missing coverage

**`generate_summary(diff_text: str, title: str = "", stats: dict = None) -> str`**
- Inputs: diff text, optional PR title, optional pre-computed stats dict
- Output: plain-English summary paragraph (max `SUMMARY_MAX_WORDS` words from env, default 150)
- Behavior: build system prompt instructing LLM to write for a non-technical audience; include stats and title in user content; instruct LLM to never include code snippets; call `get_completion`
- Edge cases: if LLM response contains lines starting with `+` or `-`, strip them (post-process)

**`generate_arch_impact(diff_text: str, stats: dict = None) -> str`**
- Inputs: diff text, optional stats dict
- Output: architecture impact paragraph (max `ARCH_MAX_WORDS` words from env, default 100)
- Behavior: system prompt asks for 1-paragraph architecture impact — what modules are affected, what system-level properties change (latency, scalability, security boundary); call `get_completion`
- Edge cases: pure documentation-only diffs (only `.md` files) should return "No architectural changes."

**`build_report(summary: str, arch_impact: str, test_flag: bool, stats: dict, comments_text: str = "") -> str`**
- Inputs: all generated sections
- Output: formatted markdown string with four sections: Summary, Architecture Impact, Test Coverage Flag, Diff Stats
- Behavior: test coverage flag renders as `⚠️ No test changes detected` or `✅ Tests modified`; stats render as a small table

---

#### `src/main.py`

CLI entry point using `argparse`.

Arguments:
- `--diff <file>` — path to diff file; if omitted, reads stdin
- `--title "..."` — optional PR title passed into summary prompt
- `--comments <file>` — optional file of reviewer comments (plain text); appended to summary prompt

Behavior:
1. Load `.env`
2. Read diff text
3. Call `parse_diff_stats` — log stats to stdout before calling LLM
4. Call `detect_test_changes`
5. Call `generate_summary` and `generate_arch_impact` (sequential)
6. Call `build_report`
7. Print report to stdout
8. Also log: `Summary generated | files_changed=N | lines_added=N | lines_removed=N | test_flag=<bool>`

---

### tests/ — what to test

#### Test 1 — Test coverage detection — missing tests (`test_summarizer.py`)
- Use fixture `no_tests.diff` (contains only `src/auth.py` changes)
- Call `detect_test_changes`
- Assert result is `True` (flag raised — no tests present)

#### Test 2 — Test coverage detection — tests present (`test_summarizer.py`)
- Use fixture `with_tests.diff` (contains both `src/auth.py` and `tests/test_auth.py`)
- Call `detect_test_changes`
- Assert result is `False` (flag not raised — tests are present)

#### Test 3 — Diff stats parsing (`test_summarizer.py`)
- Create a minimal inline diff string with exactly 2 added lines, 1 removed line, 2 files
- Call `parse_diff_stats`
- Assert `files_changed == 2`, `lines_added == 2`, `lines_removed == 1`

#### Test 4 — Summary contains no raw diff lines (`test_summarizer.py`)
- Mock LLM to return a response that accidentally includes a line starting with `+`
- Call `generate_summary` and let it post-process
- Assert no line in the result starts with `+` or `-`

#### Test 5 — Arch impact for docs-only diff (`test_summarizer.py`)
- Mock LLM to return a long response
- Pass a diff that only touches `.md` files
- Assert the function returns "No architectural changes." without calling the LLM
- (This exercises the early-return path before the LLM call)

#### Test 6 — Report format (`test_summarizer.py`)
- Call `build_report` with known strings
- Assert the returned markdown contains all four section headings
- Assert the test flag section matches the expected warning or success string

---

### README.md content

```markdown
# PR Diff Summarizer

Turns a git diff into three audience-aware outputs:
- A plain-English summary for non-technical readers
- An architecture impact paragraph for engineering managers
- A test coverage flag when tests are absent from the diff

## Quick start

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env   # add your API key
git diff HEAD~1 | python src/main.py --title "My PR title"
```

## Usage

```bash
## Summarize a saved diff
python src/main.py --diff my.diff

## Include PR title and reviewer comments
python src/main.py --diff my.diff --title "Refactor auth module" --comments comments.txt

## Read from stdin
git diff main...feature | python src/main.py
```

## Output sections

1. **Summary** — written for a product manager or engineering manager
2. **Architecture Impact** — module-level system design changes
3. **Test Coverage Flag** — warns when no test files appear in the diff
4. **Diff Stats** — files changed, lines added/removed

## Running tests

```bash
pytest tests/ -v
```
```

---

### GUIDE.md content

```markdown
# Build Guide — PR Diff Summarizer

## Step 1 — Stats first (no LLM)

Parse diff stats before touching the LLM. This gives you:
- Context to inject into prompts ("This PR changes 12 files and adds 400 lines")
- A cheap test coverage signal (regex, no API call)
- A log line you can emit before any async work

## Step 2 — Two separate prompts, two separate system personas

Do not mix the two LLM calls:

**Summary system prompt persona:**
```
You are a technical writer translating a code change for a non-technical audience.
Write in plain English. Do not include code, variable names, or file paths.
Keep the summary under {SUMMARY_MAX_WORDS} words.
```

**Architecture impact persona:**
```
You are a software architect reviewing a pull request for system design impact.
Describe module boundaries, data flow changes, and any new dependencies introduced.
One paragraph, under {ARCH_MAX_WORDS} words.
```

Using separate calls gives better results than asking for both in one prompt.

## Step 3 — Test coverage without LLM

```python
TEST_PATTERNS = ["test_", "_test.py", "/tests/"]

def detect_test_changes(diff_text: str) -> bool:
    filenames = extract_filenames(diff_text)
    has_tests = any(
        any(p in f for p in TEST_PATTERNS)
        for f in filenames
    )
    return not has_tests  # True = flag raised = missing tests
```

## Step 4 — Post-process for no raw code

After each LLM call, strip lines that start with `+` or `-` (they're raw diff noise the model sometimes echoes back). This is a cheap guard that makes output always safe to show to non-engineers.

## Step 5 — Compose the report

Write `build_report` to accept strings and return a formatted markdown string. This makes it trivially testable — no mocking needed, just string assertions.

## Debugging tips

- If summaries are too technical, strengthen the system prompt: add "Do not mention function names, variable names, or file paths."
- For arch impact, if the model is vague, inject the stats dict as context: "The diff changes 8 files including auth.py and database.py."
- Diff fixture files in `tests/fixtures/` should be real diffs — generate them with `git diff` on a real commit.

## How to talk about this in an interview

**"Why two separate LLM calls?"**
> Audience-aware generation. The non-technical summary needs one persona (writer, avoid jargon), the architecture paragraph needs another (architect, use technical terms). Mixing them produces mediocre output for both. Keeping them separate also lets me tune each prompt independently.

**"How did you handle test coverage detection?"**
> I do it without an LLM — regex on the filenames extracted from the diff headers. It's faster, cheaper, and more reliable than asking the model to infer it. The LLM is used only for the tasks it does uniquely well: generating natural language.

**"What would you add to make this production-ready?"**
> Post it directly as a GitHub PR comment via the PyGithub API. Add a Slack webhook integration. Cache by diff hash so re-running on the same diff is free.
```

---

### Failure modes to handle

| Failure | Where | How to handle |
|---|---|---|
| LLM echoes raw diff lines in summary | `summarizer.py` | Post-process: strip lines starting with `+` or `-` from response |
| Diff is empty (no changes) | `main.py` | Print `No changes detected in diff.` and exit 0 |
| Diff contains only binary files | `summarizer.py` | `parse_diff_stats` returns 0 lines; inject "binary changes only" note into prompt |
| Comments file does not exist | `main.py` | Print warning and continue without comments — do not crash |
| LLM response exceeds word limit | `summarizer.py` | Truncate to word limit with a trailing `[...]` |
| API key missing | `llm.py` | Raise `RuntimeError("LLM_PROVIDER key not set")` with provider name |

---

### The metric this project measures

**What is measured:** Three signals per diff: summary word count, test coverage flag (boolean), and whether the output contains any raw code lines.

**Format (stdout log line):**
```
Summary generated | files_changed=8 | lines_added=243 | lines_removed=91 | test_flag=True
```

**Target:** For a diff that touches only `src/` files, `test_flag` must be `True`. The summary must be under `SUMMARY_MAX_WORDS` words and contain zero lines starting with `+` or `-`. These three checks together confirm audience-aware generation, not just raw LLM pass-through.


### Model source currency

- Anthropic model list: https://docs.anthropic.com/en/docs/about-claude/models/overview
- OpenAI model list: https://platform.openai.com/docs/models
- verifiedOn: 2026-08-07
- `LLM_MODEL` is required and has no implicit default; choose a supported model for the configured provider at setup time.

## Source code

A reference implementation of PR Diff Summarizer — README, guide, source and tests — is at https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/pr-diff-summarizer.