AI PR Reviewer (Capstone)

Field	Value
Path	2 — AI-Augmented Engineering
Position	12 of 12
Difficulty	🔴 Requires GitHub account (GitHub Actions)
Estimated time	4 hours
AWS cost	None (GitHub Actions free tier)

---

Agent Pickup Instructions

# 1. Enter project
cd projects/ai-pr-reviewer

# 2. Create virtual environment
python -m venv .venv && source .venv/bin/activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Copy env file and add your API key
cp .env.example .env

# 5. Test locally with act (install act first: https://github.com/nektos/act)
bash scripts/local_test.sh

# 6. Push to GitHub and open a test PR to trigger the Action

# 7. Run unit tests
pytest tests/ -v

Done when:

• GitHub Action triggers on PR open/update
• Code review findings posted as PR comment in markdown table
• Test generation results posted as a second PR comment
• HIGH severity finding causes the Action to exit with code 1 (PR check fails)
• OTEL span created per PR with model, cost, latency, and trace_id attributes
• All 4 tests pass
• act runs the workflow locally without errors

---

What this project is

A GitHub Action that runs two AI tools — the code review bot (p2-01 pattern) and the test generator (p2-05 pattern) — on every pull request. It posts findings as PR comments using the GitHub API, enforces a severity gate (HIGH finding → Action exits 1 → PR check fails), and instruments every run with an OpenTelemetry span tracking model, cost, latency, and trace ID. A config file in the repository root controls which tools are active and what severity gate to enforce. Local testing is supported via act without pushing to GitHub.

---

What the learner achieves

"I packaged two AI engineering tools into a GitHub Action that runs on every PR, posts structured findings as comments, enforces a severity gate that blocks merges on HIGH findings, and instruments every run with OTEL spans for cost and latency tracking."

---

Folder structure

ai-pr-reviewer/
├── .github/
│   └── workflows/
│       └── ai-review.yml       # GitHub Actions workflow definition
├── src/
│   ├── action_runner.py        # Main entry point for the Action
│   ├── llm.py                  # Provider-agnostic LLM wrapper
│   ├── pr_commenter.py         # post_review_comment, post_test_results
│   └── cost_tracker.py         # log_pr_cost (OTEL instrumented)
├── tests/
│   ├── fixtures/
│   │   └── sample.diff         # Test diff for local runs
│   ├── test_action_runner.py
│   ├── test_pr_commenter.py
│   └── test_cost_tracker.py
├── scripts/
│   └── local_test.sh           # act-based local test runner
├── .aiworkflow.yml             # Tool configuration (committed to repo)
├── .env.example
├── requirements.txt
└── README.md

---

.github/workflows/ai-review.yml

name: AI Code Review

on:
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  ai-review:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
      contents: read

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Get PR diff
        id: diff
        run: |
          git diff origin/${{ github.base_ref }}...HEAD > /tmp/pr.diff
          echo "diff_path=/tmp/pr.diff" >> $GITHUB_OUTPUT

      - name: Run AI Review Action
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          PR_NUMBER: ${{ github.event.pull_request.number }}
          REPO_NAME: ${{ github.repository }}
          BASE_REF: ${{ github.base_ref }}
        run: |
          python src/action_runner.py --diff ${{ steps.diff.outputs.diff_path }}

---

.aiworkflow.yml (committed to repo)

ai_review:
  code_review:
    enabled: true
    min_severity: LOW
    severity_gate: HIGH    # Action fails if any finding at this level

  test_generation:
    enabled: true
    max_retries: 2

  observability:
    enabled: true
    otel_exporter: console  # Use 'jaeger' if you have a collector

settings:
  post_comments: true         # Set false for dry-run (no PR comments)
  comment_on_pass: false      # Don't post comment if no issues found

---

.env.example

# LLM provider: anthropic | openai | ollama
LLM_PROVIDER=anthropic

# Model to use
LLM_MODEL=

# API key for Anthropic (also set as GitHub Secret ANTHROPIC_API_KEY)
ANTHROPIC_API_KEY=

# API key for OpenAI
OPENAI_API_KEY=

# Ollama base URL
OLLAMA_BASE_URL=http://localhost:11434

# GitHub token (set automatically in GitHub Actions; needed for local testing)
GITHUB_TOKEN=

# PR number (set automatically in GitHub Actions)
PR_NUMBER=

# Repository name e.g. "owner/repo" (set automatically in GitHub Actions)
REPO_NAME=

# OTEL exporter: console | jaeger
OTEL_EXPORTER=console

# Jaeger endpoint (optional)
JAEGER_ENDPOINT=http://localhost:4317

# Cost per 1K tokens (for OTEL cost tracking)
COST_PER_1K_INPUT_TOKENS=0.00025
COST_PER_1K_OUTPUT_TOKENS=0.00125

---

requirements.txt

anthropic==0.30.0
openai==1.35.0
PyGithub==2.5.0
opentelemetry-sdk==1.25.0
opentelemetry-api==1.25.0
opentelemetry-exporter-otlp-proto-grpc==1.25.0
python-dotenv==1.0.1
rich==13.7.1
pytest==8.2.2
pytest-mock==3.14.0
pyyaml==6.0.1
tabulate==0.9.0

---

src/ — what to implement

src/llm.py

get_completion(prompt: str, system: str = "") -> tuple[str, dict]

• Returns (response_text, usage_dict) where usage_dict = {"input_tokens": int, "output_tokens": int}
• Standard routing to Anthropic, OpenAI, or Ollama
• This version returns usage data so cost_tracker.py can record it

---

src/cost_tracker.py

PRCostTracker (class)

__init__(self, pr_number: str, repo_name: str)

• Sets up OTEL TracerProvider (console or Jaeger based on OTEL_EXPORTER env var)
• Stores PR metadata
• Initializes accumulated stats: {"input_tokens": 0, "output_tokens": 0, "cost_usd": 0.0, "latency_ms": 0, "calls": 0}

record_llm_call(self, usage: dict, latency_ms: int) -> None

• Inputs: usage dict from llm.py, latency in ms
• Behavior: accumulate stats; estimate cost from token counts using env vars

log_pr_cost(self) -> str

• Output: trace ID string
• Behavior:
  • Create an OTEL span named ai_review.pr
  • Set attributes: pr.number, pr.repo, llm.model, llm.provider, llm.total_input_tokens, llm.total_output_tokens, llm.total_cost_usd, llm.total_latency_ms, llm.calls
  • Generate and set trace_id as uuid4().hex
  • End span
  • Print PR Review Cost | pr={pr_number} | calls={calls} | tokens={total} | cost=${cost:.6f} | trace_id={id}
  • Return trace_id

---

src/pr_commenter.py

format_review_comment(findings: list[dict], trace_id: str) -> str

• Inputs: list of finding dicts (same schema as p2-01), trace ID
• Output: markdown string for GitHub PR comment
• Format:

  ## AI Code Review
  
  | Severity | File | Lines | Category | Finding | Suggestion |
  |---|---|---|---|---|---|
  | 🔴 HIGH | auth.py | 12-15 | security | Hardcoded password | Use env var |
  
  **Summary:** 1 HIGH, 0 MEDIUM, 2 LOW findings
  
  _Trace ID: a1b2c3d4 | Model: <configured LLM_MODEL>_
  

• Severity emoji: HIGH = 🔴, MEDIUM = 🟡, LOW = 🟢

format_test_results_comment(results: dict, trace_id: str) -> str

• Inputs: {"functions_tested": int, "tests_generated": int, "retries": int, "coverage_pct": float | None}, trace ID
• Output: markdown string for second PR comment
• Format:

  ## AI Test Generation Results
  
  - Functions analyzed: N
  - Tests generated: N
  - Retries needed: N
  - Coverage: N%
  
  _Trace ID: a1b2c3d4_
  

post_comment(body: str, pr_number: int, repo_name: str, github_token: str) -> str

• Inputs: markdown body, PR number, repo name ("owner/repo"), GitHub token
• Output: URL of the created comment
• Behavior: use PyGithub: Github(token).get_repo(repo_name).get_pull(pr_number).create_issue_comment(body)
• Edge cases: if post_comments=False in config, log the body to stdout and return "dry-run" instead of posting

post_review_comment(findings: list[dict], trace_id: str, config: dict) -> None

• Orchestrates: format + post (or dry-run)

post_test_results(results: dict, trace_id: str, config: dict) -> None

• Orchestrates: format + post (or dry-run)

---

src/action_runner.py

Main entry point called by the GitHub Action.

Arguments (argparse):

• --diff <path> — path to the PR diff file (generated by git diff in the workflow)
• --dry-run — skip posting comments (for local testing)

run_code_review(diff_text: str, config: dict, tracker: PRCostTracker) -> tuple[list[dict], int]

• Inline the p2-01 chunking + review logic
• Returns (findings, retry_count)
• Records each LLM call with tracker.record_llm_call

run_test_generation(diff_text: str, config: dict, tracker: PRCostTracker) -> dict

• Inline the p2-05 test generation logic (simplified: generate for changed functions only)
• Returns results dict for format_test_results_comment

check_severity_gate(findings: list[dict], gate_level: str) -> bool

• Inputs: findings list, gate level from config (default "HIGH")
• Output: True if gate is triggered (should fail), False if clear
• Behavior: return True if any finding's severity equals or exceeds gate_level
• Severity order: HIGH > MEDIUM > LOW

main() -> None

• Behavior:
  1. Load .env, load .aiworkflow.yml config
  2. Read diff file
  3. Create PRCostTracker
  4. If code_review.enabled: run run_code_review; call post_review_comment
  5. If test_generation.enabled: run run_test_generation; call post_test_results
  6. Call tracker.log_pr_cost → get trace_id
  7. Call check_severity_gate: if triggered, print Severity gate triggered: HIGH finding found. and sys.exit(1)
  8. Otherwise exit 0

---

scripts/local_test.sh

#!/bin/bash
# Local test using act (https://github.com/nektos/act)
# Install act: brew install act (Mac) or see https://github.com/nektos/act

set -e

echo "Running AI Review Action locally with act..."

# Export secrets from .env
export $(cat .env | grep -v '#' | xargs)

# Create a sample diff for testing
git diff HEAD~1 > /tmp/local_test.diff 2>/dev/null || cp tests/fixtures/sample.diff /tmp/local_test.diff

# Run with act (uses docker by default)
act pull_request \
    --secret ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY" \
    --secret GITHUB_TOKEN="$GITHUB_TOKEN" \
    --env PR_NUMBER=1 \
    --env REPO_NAME="owner/test-repo" \
    --env BASE_REF=main \
    -j ai-review \
    --dry-run

echo "Local test complete."

---

tests/ — what to test

Test 1 — Severity gate logic — HIGH triggers exit (test_action_runner.py)

• Call check_severity_gate with findings containing one HIGH-severity item
• Assert result is True (gate triggered)
• Call check_severity_gate with only MEDIUM findings and gate set to HIGH
• Assert result is False (gate not triggered)

Test 2 — PR comment formatter produces valid markdown (test_pr_commenter.py)

• Call format_review_comment with 2 sample findings (one HIGH, one LOW)
• Assert result starts with ## AI Code Review
• Assert the HIGH finding row contains 🔴
• Assert the trace ID appears in the output
• Assert the summary line shows the correct counts

Test 3 — Cost tracker records span attributes (test_cost_tracker.py)

• Create PRCostTracker with OTEL_EXPORTER=console
• Call record_llm_call twice with known usage values
• Call log_pr_cost
• Capture span (use InMemorySpanExporter)
• Assert span has attributes: pr.number, llm.total_cost_usd, llm.calls, llm.model, trace_id
• Assert llm.calls == 2

Test 4 — Config respects tool enable/disable flags (test_action_runner.py)

• Write a temp .aiworkflow.yml with code_review: enabled: false
• Mock run_code_review and run_test_generation
• Call main() with the temp config path
• Assert run_code_review was NOT called
• Assert run_test_generation WAS called (it's still enabled)

Test 5 — Dry-run skips posting but logs body (test_pr_commenter.py)

• Call post_comment with a config where post_comments=False
• Assert the function returns "dry-run" without calling PyGithub
• Capture stdout and assert the markdown body was printed

Test 6 — Test results comment format (test_pr_commenter.py)

• Call format_test_results_comment with known values
• Assert markdown contains "Functions analyzed", "Tests generated", "Retries"
• Assert trace ID appears in the output

---

README.md content

# AI-Powered GitHub Action (Capstone)

GitHub Action that runs AI code review and test generation on every PR.
Posts findings as comments. HIGH finding → PR check fails.

## Setup

1. Add `ANTHROPIC_API_KEY` to your repository's **Settings → Secrets and variables → Actions**
2. Commit `.github/workflows/ai-review.yml` to your repository
3. Open a PR — the Action runs automatically

## What it does

On every PR:
1. **Code Review** — scans the diff for security, performance, correctness, and style issues
2. **Test Generation** — generates pytest tests for changed Python functions
3. **Severity Gate** — exits with code 1 if any HIGH finding is found (blocks merge)
4. **OTEL Tracking** — records model, tokens, cost, and latency per PR review

## PR comment format

**Code Review:**

| Severity | File | Lines | Category | Finding |
|---|---|---|---|---|
| 🔴 HIGH | auth.py | 12-15 | security | Hardcoded password |

**Test Generation:**

- Functions analyzed: 3
- Tests generated: 8
- Coverage: 72%

## Local testing

```bash
# Install act: https://github.com/nektos/act
bash scripts/local_test.sh

Configuration

Edit .aiworkflow.yml:

ai_review:
  code_review:
    severity_gate: HIGH   # Change to MEDIUM to be stricter
  test_generation:
    enabled: false        # Disable test gen for large PRs

Running tests

pytest tests/ -v

---

## GUIDE.md content

```markdown
# Build Guide — AI-Powered GitHub Action

## Step 1 — Understand the GitHub Actions environment

When a PR is opened, the Action:
1. Checks out the code
2. Generates a diff with `git diff origin/${{ github.base_ref }}...HEAD`
3. Passes it to `python src/action_runner.py --diff /tmp/pr.diff`
4. The Python script has access to env vars: `GITHUB_TOKEN`, `PR_NUMBER`, `REPO_NAME`

The Python process's exit code controls the PR check:
- Exit 0 → check passes → merge allowed
- Exit 1 → check fails → merge blocked

## Step 2 — Posting PR comments

Use PyGithub:

```python
from github import Github

def post_comment(body, pr_number, repo_name, github_token):
    g = Github(github_token)
    repo = g.get_repo(repo_name)
    pr = repo.get_pull(int(pr_number))
    comment = pr.create_issue_comment(body)
    return comment.html_url

The GITHUB_TOKEN is automatically provided by GitHub Actions — you don't need a personal access token for commenting on the same repo.

Step 3 — The severity gate

SEVERITY_ORDER = {"LOW": 0, "MEDIUM": 1, "HIGH": 2}

def check_severity_gate(findings, gate_level):
    gate_value = SEVERITY_ORDER.get(gate_level, 2)
    return any(
        SEVERITY_ORDER.get(f["severity"], 0) >= gate_value
        for f in findings
    )

After all tools run, check the gate. If triggered:

if check_severity_gate(findings, config["code_review"]["severity_gate"]):
    print("Severity gate triggered: HIGH finding found.")
    sys.exit(1)

Step 4 — OTEL instrumentation

Record cost and latency per PR, not per call:

with tracer.start_as_current_span("ai_review.pr") as span:
    span.set_attribute("pr.number", pr_number)
    span.set_attribute("llm.total_cost_usd", total_cost)
    span.set_attribute("llm.calls", call_count)
    span.set_attribute("trace_id", trace_id)

This gives you one span per PR to track aggregate cost and performance over time.

Step 5 — Local testing with act

act runs GitHub Actions locally using Docker. Install it, then:

act pull_request \
    --secret ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY" \
    --secret GITHUB_TOKEN="$GITHUB_TOKEN" \
    --env PR_NUMBER=1 \
    --env REPO_NAME="owner/repo" \
    -j ai-review

The --dry-run flag skips Docker and just validates the workflow YAML.

Step 6 — Dry-run mode for development

Set post_comments: false in .aiworkflow.yml during development. This prints the comment body to stdout instead of calling the GitHub API — essential for local testing without a real PR.

Debugging tips

• If the Action fails with "git diff returned no output", the fetch-depth: 0 option in the workflow YAML is required — add it to actions/checkout
• If GITHUB_TOKEN lacks permission to post comments, check that permissions: pull-requests: write is in the workflow
• If act can't find Docker, use act --platform ubuntu-latest=catthehacker/ubuntu:act-latest

How to talk about this in an interview

"Why package AI tools as a GitHub Action instead of just a CLI?"

Integration. The value of a code review tool is zero if engineers don't use it. By running on every PR automatically and posting results as comments, it's part of the workflow without requiring any behavior change from developers.

"What does the severity gate buy you?"

It's the difference between a suggestion and a guardrail. Findings without consequence get ignored. A HIGH finding that blocks the PR merge forces a decision: fix it or dismiss it explicitly. Either way, the team has a record.

"How do you prevent the Action from becoming too slow?"

Two levers. First, review only the changed files — don't re-review unchanged code. Second, the test_generation.enabled flag in config lets teams disable test gen on large PRs where it would be slow. Cost and latency are logged per run so you can see where time is going.

"How do you track cost over time?"

Each PR review creates an OTEL span with total cost and tokens. If you export to a time-series database, you can plot cost per PR, cost per author, and cost trend as the codebase grows. I log the trace ID in the PR comment so you can look up the full span for any PR.

---

## Failure modes to handle

| Failure | Where | How to handle |
|---|---|---|
| `GITHUB_TOKEN` missing | `pr_commenter.py` | If `post_comments=True` but token is absent, raise `EnvironmentError`; suggest adding to secrets |
| GitHub API rate limit | `pr_commenter.py` | Catch `github.GithubException`; log warning; continue (don't fail the Action over a comment) |
| Diff is empty (no changes) | `action_runner.py` | Print "Empty diff — nothing to review." and exit 0 |
| OTEL export fails | `cost_tracker.py` | Catch connection errors; log warning; continue (observability failure must not fail the PR check) |
| HIGH finding with `post_comments=False` | `action_runner.py` | Still check severity gate and exit 1 — dry-run skips comments but not the gate |
| `.aiworkflow.yml` missing from repo | `action_runner.py` | Use defaults (all tools enabled, severity gate = HIGH); log "Config not found, using defaults" |

---

## The metric this project measures

**What is measured:** Per-PR: review findings count by severity, test generation coverage percentage, and total LLM cost in USD — all logged via OTEL span.

**Format (stdout at Action end):**

PR Review Cost | pr=42 | calls=5 | tokens=3,241 | cost=$0.000810 | trace_id=a1b2c3d4e5f6... Severity gate: CLEAR (0 HIGH findings)

**Or on gate trigger:**

PR Review Cost | pr=42 | calls=5 | tokens=3,241 | cost=$0.000810 | trace_id=a1b2c3d4e5f6... Severity gate triggered: 1 HIGH finding found. Exit code: 1

**Target:** The Action must post at least one PR comment with the review table, exit 0 on a clean diff, and exit 1 on a diff that contains a hardcoded secret. Cost must be logged with the correct trace ID in both cases. This confirms end-to-end integration: LLM call → structured output → PR comment → severity gate → OTEL instrumentation.


## Model source currency

- Anthropic model list: https://docs.anthropic.com/en/docs/about-claude/models/overview
- OpenAI model list: https://platform.openai.com/docs/models
- verifiedOn: 2026-08-07
- `LLM_MODEL` is required and has no implicit default; choose a supported model for the configured provider at setup time.