--- title: "AI PR Reviewer (Capstone)" description: "A GitHub Action that runs in CI and can block a merge is the strongest possible proof point for 'AI-Augmented Senior Engineer' — it's infrastructure your team..." source: "https://confidentprep.com/paths/ai-augmented-engineering/ai-pr-reviewer/" path: "AI-Augmented Engineering" repository: "https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/ai-pr-reviewer" token_estimate: 5336 --- # AI PR Reviewer (Capstone) ## Overview A GitHub Action that runs code review and test generation on every PR, posts findings as comments, enforces a severity gate that can fail the check, and tracks cost/latency via OpenTelemetry. This is projects 1, 5, and 6 from this path — review, test generation, observability — combined into infrastructure that runs automatically on every PR, with no human remembering to invoke it. ## What to do | Field | Value | |---|---| | Path | 2 — AI-Augmented Engineering | | Position | 12 of 12 | | Difficulty | 🔴 Requires GitHub account (GitHub Actions) | | Estimated time | 4 hours | | AWS cost | None (GitHub Actions free tier) | --- ### Agent Pickup Instructions ```bash # 1. Enter project cd projects/ai-pr-reviewer # 2. Create virtual environment python -m venv .venv && source .venv/bin/activate # 3. Install dependencies pip install -r requirements.txt # 4. Copy env file and add your API key cp .env.example .env # 5. Test locally with act (install act first: https://github.com/nektos/act) bash scripts/local_test.sh # 6. Push to GitHub and open a test PR to trigger the Action # 7. Run unit tests pytest tests/ -v ``` **Done when:** - [ ] GitHub Action triggers on PR open/update - [ ] Code review findings posted as PR comment in markdown table - [ ] Test generation results posted as a second PR comment - [ ] HIGH severity finding causes the Action to exit with code 1 (PR check fails) - [ ] OTEL span created per PR with model, cost, latency, and trace_id attributes - [ ] All 4 tests pass - [ ] `act` runs the workflow locally without errors --- ### What this project is A GitHub Action that runs two AI tools — the code review bot (p2-01 pattern) and the test generator (p2-05 pattern) — on every pull request. It posts findings as PR comments using the GitHub API, enforces a severity gate (HIGH finding → Action exits 1 → PR check fails), and instruments every run with an OpenTelemetry span tracking model, cost, latency, and trace ID. A config file in the repository root controls which tools are active and what severity gate to enforce. Local testing is supported via `act` without pushing to GitHub. --- ### What the learner achieves "I packaged two AI engineering tools into a GitHub Action that runs on every PR, posts structured findings as comments, enforces a severity gate that blocks merges on HIGH findings, and instruments every run with OTEL spans for cost and latency tracking." --- ### Folder structure ``` ai-pr-reviewer/ ├── .github/ │ └── workflows/ │ └── ai-review.yml # GitHub Actions workflow definition ├── src/ │ ├── action_runner.py # Main entry point for the Action │ ├── llm.py # Provider-agnostic LLM wrapper │ ├── pr_commenter.py # post_review_comment, post_test_results │ └── cost_tracker.py # log_pr_cost (OTEL instrumented) ├── tests/ │ ├── fixtures/ │ │ └── sample.diff # Test diff for local runs │ ├── test_action_runner.py │ ├── test_pr_commenter.py │ └── test_cost_tracker.py ├── scripts/ │ └── local_test.sh # act-based local test runner ├── .aiworkflow.yml # Tool configuration (committed to repo) ├── .env.example ├── requirements.txt └── README.md ``` --- ### .github/workflows/ai-review.yml ```yaml name: AI Code Review on: pull_request: types: [opened, synchronize, reopened] jobs: ai-review: runs-on: ubuntu-latest permissions: pull-requests: write contents: read steps: - name: Checkout repository uses: actions/checkout@v4 with: fetch-depth: 0 - name: Set up Python uses: actions/setup-python@v5 with: python-version: "3.11" - name: Install dependencies run: pip install -r requirements.txt - name: Get PR diff id: diff run: | git diff origin/${{ github.base_ref }}...HEAD > /tmp/pr.diff echo "diff_path=/tmp/pr.diff" >> $GITHUB_OUTPUT - name: Run AI Review Action env: ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} PR_NUMBER: ${{ github.event.pull_request.number }} REPO_NAME: ${{ github.repository }} BASE_REF: ${{ github.base_ref }} run: | python src/action_runner.py --diff ${{ steps.diff.outputs.diff_path }} ``` --- ### .aiworkflow.yml (committed to repo) ```yaml ai_review: code_review: enabled: true min_severity: LOW severity_gate: HIGH # Action fails if any finding at this level test_generation: enabled: true max_retries: 2 observability: enabled: true otel_exporter: console # Use 'jaeger' if you have a collector settings: post_comments: true # Set false for dry-run (no PR comments) comment_on_pass: false # Don't post comment if no issues found ``` --- ### .env.example ```bash # LLM provider: anthropic | openai | ollama LLM_PROVIDER=anthropic # Model to use LLM_MODEL= # API key for Anthropic (also set as GitHub Secret ANTHROPIC_API_KEY) ANTHROPIC_API_KEY= # API key for OpenAI OPENAI_API_KEY= # Ollama base URL OLLAMA_BASE_URL=http://localhost:11434 # GitHub token (set automatically in GitHub Actions; needed for local testing) GITHUB_TOKEN= # PR number (set automatically in GitHub Actions) PR_NUMBER= # Repository name e.g. "owner/repo" (set automatically in GitHub Actions) REPO_NAME= # OTEL exporter: console | jaeger OTEL_EXPORTER=console # Jaeger endpoint (optional) JAEGER_ENDPOINT=http://localhost:4317 # Cost per 1K tokens (for OTEL cost tracking) COST_PER_1K_INPUT_TOKENS=0.00025 COST_PER_1K_OUTPUT_TOKENS=0.00125 ``` --- ### requirements.txt ``` anthropic==0.30.0 openai==1.35.0 PyGithub==2.5.0 opentelemetry-sdk==1.25.0 opentelemetry-api==1.25.0 opentelemetry-exporter-otlp-proto-grpc==1.25.0 python-dotenv==1.0.1 rich==13.7.1 pytest==8.2.2 pytest-mock==3.14.0 pyyaml==6.0.1 tabulate==0.9.0 ``` --- ### src/ — what to implement #### `src/llm.py` **`get_completion(prompt: str, system: str = "") -> tuple[str, dict]`** - Returns `(response_text, usage_dict)` where `usage_dict = {"input_tokens": int, "output_tokens": int}` - Standard routing to Anthropic, OpenAI, or Ollama - This version returns usage data so `cost_tracker.py` can record it --- #### `src/cost_tracker.py` **`PRCostTracker` (class)** `__init__(self, pr_number: str, repo_name: str)` - Sets up OTEL `TracerProvider` (console or Jaeger based on `OTEL_EXPORTER` env var) - Stores PR metadata - Initializes accumulated stats: `{"input_tokens": 0, "output_tokens": 0, "cost_usd": 0.0, "latency_ms": 0, "calls": 0}` `record_llm_call(self, usage: dict, latency_ms: int) -> None` - Inputs: usage dict from `llm.py`, latency in ms - Behavior: accumulate stats; estimate cost from token counts using env vars `log_pr_cost(self) -> str` - Output: trace ID string - Behavior: - Create an OTEL span named `ai_review.pr` - Set attributes: `pr.number`, `pr.repo`, `llm.model`, `llm.provider`, `llm.total_input_tokens`, `llm.total_output_tokens`, `llm.total_cost_usd`, `llm.total_latency_ms`, `llm.calls` - Generate and set `trace_id` as `uuid4().hex` - End span - Print `PR Review Cost | pr={pr_number} | calls={calls} | tokens={total} | cost=${cost:.6f} | trace_id={id}` - Return trace_id --- #### `src/pr_commenter.py` **`format_review_comment(findings: list[dict], trace_id: str) -> str`** - Inputs: list of finding dicts (same schema as p2-01), trace ID - Output: markdown string for GitHub PR comment - Format: ```markdown ## AI Code Review | Severity | File | Lines | Category | Finding | Suggestion | |---|---|---|---|---|---| | 🔴 HIGH | auth.py | 12-15 | security | Hardcoded password | Use env var | **Summary:** 1 HIGH, 0 MEDIUM, 2 LOW findings _Trace ID: a1b2c3d4 | Model: <configured LLM_MODEL>_ ``` - Severity emoji: HIGH = 🔴, MEDIUM = 🟡, LOW = 🟢 **`format_test_results_comment(results: dict, trace_id: str) -> str`** - Inputs: `{"functions_tested": int, "tests_generated": int, "retries": int, "coverage_pct": float | None}`, trace ID - Output: markdown string for second PR comment - Format: ```markdown ## AI Test Generation Results - Functions analyzed: N - Tests generated: N - Retries needed: N - Coverage: N% _Trace ID: a1b2c3d4_ ``` **`post_comment(body: str, pr_number: int, repo_name: str, github_token: str) -> str`** - Inputs: markdown body, PR number, repo name ("owner/repo"), GitHub token - Output: URL of the created comment - Behavior: use `PyGithub`: `Github(token).get_repo(repo_name).get_pull(pr_number).create_issue_comment(body)` - Edge cases: if `post_comments=False` in config, log the body to stdout and return "dry-run" instead of posting **`post_review_comment(findings: list[dict], trace_id: str, config: dict) -> None`** - Orchestrates: format + post (or dry-run) **`post_test_results(results: dict, trace_id: str, config: dict) -> None`** - Orchestrates: format + post (or dry-run) --- #### `src/action_runner.py` Main entry point called by the GitHub Action. Arguments (argparse): - `--diff ` — path to the PR diff file (generated by `git diff` in the workflow) - `--dry-run` — skip posting comments (for local testing) **`run_code_review(diff_text: str, config: dict, tracker: PRCostTracker) -> tuple[list[dict], int]`** - Inline the p2-01 chunking + review logic - Returns `(findings, retry_count)` - Records each LLM call with `tracker.record_llm_call` **`run_test_generation(diff_text: str, config: dict, tracker: PRCostTracker) -> dict`** - Inline the p2-05 test generation logic (simplified: generate for changed functions only) - Returns results dict for `format_test_results_comment` **`check_severity_gate(findings: list[dict], gate_level: str) -> bool`** - Inputs: findings list, gate level from config (default "HIGH") - Output: `True` if gate is triggered (should fail), `False` if clear - Behavior: return `True` if any finding's `severity` equals or exceeds `gate_level` - Severity order: HIGH > MEDIUM > LOW **`main() -> None`** - Behavior: 1. Load `.env`, load `.aiworkflow.yml` config 2. Read diff file 3. Create `PRCostTracker` 4. If `code_review.enabled`: run `run_code_review`; call `post_review_comment` 5. If `test_generation.enabled`: run `run_test_generation`; call `post_test_results` 6. Call `tracker.log_pr_cost` → get trace_id 7. Call `check_severity_gate`: if triggered, print `Severity gate triggered: HIGH finding found.` and `sys.exit(1)` 8. Otherwise exit 0 --- ### scripts/local_test.sh ```bash #!/bin/bash # Local test using act (https://github.com/nektos/act) # Install act: brew install act (Mac) or see https://github.com/nektos/act set -e echo "Running AI Review Action locally with act..." # Export secrets from .env export $(cat .env | grep -v '#' | xargs) # Create a sample diff for testing git diff HEAD~1 > /tmp/local_test.diff 2>/dev/null || cp tests/fixtures/sample.diff /tmp/local_test.diff # Run with act (uses docker by default) act pull_request \ --secret ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY" \ --secret GITHUB_TOKEN="$GITHUB_TOKEN" \ --env PR_NUMBER=1 \ --env REPO_NAME="owner/test-repo" \ --env BASE_REF=main \ -j ai-review \ --dry-run echo "Local test complete." ``` --- ### tests/ — what to test #### Test 1 — Severity gate logic — HIGH triggers exit (`test_action_runner.py`) - Call `check_severity_gate` with findings containing one HIGH-severity item - Assert result is `True` (gate triggered) - Call `check_severity_gate` with only MEDIUM findings and gate set to HIGH - Assert result is `False` (gate not triggered) #### Test 2 — PR comment formatter produces valid markdown (`test_pr_commenter.py`) - Call `format_review_comment` with 2 sample findings (one HIGH, one LOW) - Assert result starts with `## AI Code Review` - Assert the HIGH finding row contains `🔴` - Assert the trace ID appears in the output - Assert the summary line shows the correct counts #### Test 3 — Cost tracker records span attributes (`test_cost_tracker.py`) - Create `PRCostTracker` with `OTEL_EXPORTER=console` - Call `record_llm_call` twice with known usage values - Call `log_pr_cost` - Capture span (use `InMemorySpanExporter`) - Assert span has attributes: `pr.number`, `llm.total_cost_usd`, `llm.calls`, `llm.model`, `trace_id` - Assert `llm.calls == 2` #### Test 4 — Config respects tool enable/disable flags (`test_action_runner.py`) - Write a temp `.aiworkflow.yml` with `code_review: enabled: false` - Mock `run_code_review` and `run_test_generation` - Call `main()` with the temp config path - Assert `run_code_review` was NOT called - Assert `run_test_generation` WAS called (it's still enabled) #### Test 5 — Dry-run skips posting but logs body (`test_pr_commenter.py`) - Call `post_comment` with a config where `post_comments=False` - Assert the function returns `"dry-run"` without calling PyGithub - Capture stdout and assert the markdown body was printed #### Test 6 — Test results comment format (`test_pr_commenter.py`) - Call `format_test_results_comment` with known values - Assert markdown contains "Functions analyzed", "Tests generated", "Retries" - Assert trace ID appears in the output --- ### README.md content ```markdown # AI-Powered GitHub Action (Capstone) GitHub Action that runs AI code review and test generation on every PR. Posts findings as comments. HIGH finding → PR check fails. ## Setup 1. Add `ANTHROPIC_API_KEY` to your repository's **Settings → Secrets and variables → Actions** 2. Commit `.github/workflows/ai-review.yml` to your repository 3. Open a PR — the Action runs automatically ## What it does On every PR: 1. **Code Review** — scans the diff for security, performance, correctness, and style issues 2. **Test Generation** — generates pytest tests for changed Python functions 3. **Severity Gate** — exits with code 1 if any HIGH finding is found (blocks merge) 4. **OTEL Tracking** — records model, tokens, cost, and latency per PR review ## PR comment format **Code Review:** | Severity | File | Lines | Category | Finding | |---|---|---|---|---| | 🔴 HIGH | auth.py | 12-15 | security | Hardcoded password | **Test Generation:** - Functions analyzed: 3 - Tests generated: 8 - Coverage: 72% ## Local testing ```bash ## Install act: https://github.com/nektos/act bash scripts/local_test.sh ``` ## Configuration Edit `.aiworkflow.yml`: ```yaml ai_review: code_review: severity_gate: HIGH # Change to MEDIUM to be stricter test_generation: enabled: false # Disable test gen for large PRs ``` ## Running tests ```bash pytest tests/ -v ``` ``` --- ### GUIDE.md content ```markdown # Build Guide — AI-Powered GitHub Action ## Step 1 — Understand the GitHub Actions environment When a PR is opened, the Action: 1. Checks out the code 2. Generates a diff with `git diff origin/${{ github.base_ref }}...HEAD` 3. Passes it to `python src/action_runner.py --diff /tmp/pr.diff` 4. The Python script has access to env vars: `GITHUB_TOKEN`, `PR_NUMBER`, `REPO_NAME` The Python process's exit code controls the PR check: - Exit 0 → check passes → merge allowed - Exit 1 → check fails → merge blocked ## Step 2 — Posting PR comments Use PyGithub: ```python from github import Github def post_comment(body, pr_number, repo_name, github_token): g = Github(github_token) repo = g.get_repo(repo_name) pr = repo.get_pull(int(pr_number)) comment = pr.create_issue_comment(body) return comment.html_url ``` The `GITHUB_TOKEN` is automatically provided by GitHub Actions — you don't need a personal access token for commenting on the same repo. ## Step 3 — The severity gate ```python SEVERITY_ORDER = {"LOW": 0, "MEDIUM": 1, "HIGH": 2} def check_severity_gate(findings, gate_level): gate_value = SEVERITY_ORDER.get(gate_level, 2) return any( SEVERITY_ORDER.get(f["severity"], 0) >= gate_value for f in findings ) ``` After all tools run, check the gate. If triggered: ```python if check_severity_gate(findings, config["code_review"]["severity_gate"]): print("Severity gate triggered: HIGH finding found.") sys.exit(1) ``` ## Step 4 — OTEL instrumentation Record cost and latency per PR, not per call: ```python with tracer.start_as_current_span("ai_review.pr") as span: span.set_attribute("pr.number", pr_number) span.set_attribute("llm.total_cost_usd", total_cost) span.set_attribute("llm.calls", call_count) span.set_attribute("trace_id", trace_id) ``` This gives you one span per PR to track aggregate cost and performance over time. ## Step 5 — Local testing with act `act` runs GitHub Actions locally using Docker. Install it, then: ```bash act pull_request \ --secret ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY" \ --secret GITHUB_TOKEN="$GITHUB_TOKEN" \ --env PR_NUMBER=1 \ --env REPO_NAME="owner/repo" \ -j ai-review ``` The `--dry-run` flag skips Docker and just validates the workflow YAML. ## Step 6 — Dry-run mode for development Set `post_comments: false` in `.aiworkflow.yml` during development. This prints the comment body to stdout instead of calling the GitHub API — essential for local testing without a real PR. ## Debugging tips - If the Action fails with "git diff returned no output", the `fetch-depth: 0` option in the workflow YAML is required — add it to `actions/checkout` - If GITHUB_TOKEN lacks permission to post comments, check that `permissions: pull-requests: write` is in the workflow - If `act` can't find Docker, use `act --platform ubuntu-latest=catthehacker/ubuntu:act-latest` ## How to talk about this in an interview **"Why package AI tools as a GitHub Action instead of just a CLI?"** > Integration. The value of a code review tool is zero if engineers don't use it. By running on every PR automatically and posting results as comments, it's part of the workflow without requiring any behavior change from developers. **"What does the severity gate buy you?"** > It's the difference between a suggestion and a guardrail. Findings without consequence get ignored. A HIGH finding that blocks the PR merge forces a decision: fix it or dismiss it explicitly. Either way, the team has a record. **"How do you prevent the Action from becoming too slow?"** > Two levers. First, review only the changed files — don't re-review unchanged code. Second, the `test_generation.enabled` flag in config lets teams disable test gen on large PRs where it would be slow. Cost and latency are logged per run so you can see where time is going. **"How do you track cost over time?"** > Each PR review creates an OTEL span with total cost and tokens. If you export to a time-series database, you can plot cost per PR, cost per author, and cost trend as the codebase grows. I log the trace ID in the PR comment so you can look up the full span for any PR. ``` --- ### Failure modes to handle | Failure | Where | How to handle | |---|---|---| | `GITHUB_TOKEN` missing | `pr_commenter.py` | If `post_comments=True` but token is absent, raise `EnvironmentError`; suggest adding to secrets | | GitHub API rate limit | `pr_commenter.py` | Catch `github.GithubException`; log warning; continue (don't fail the Action over a comment) | | Diff is empty (no changes) | `action_runner.py` | Print "Empty diff — nothing to review." and exit 0 | | OTEL export fails | `cost_tracker.py` | Catch connection errors; log warning; continue (observability failure must not fail the PR check) | | HIGH finding with `post_comments=False` | `action_runner.py` | Still check severity gate and exit 1 — dry-run skips comments but not the gate | | `.aiworkflow.yml` missing from repo | `action_runner.py` | Use defaults (all tools enabled, severity gate = HIGH); log "Config not found, using defaults" | --- ### The metric this project measures **What is measured:** Per-PR: review findings count by severity, test generation coverage percentage, and total LLM cost in USD — all logged via OTEL span. **Format (stdout at Action end):** ``` PR Review Cost | pr=42 | calls=5 | tokens=3,241 | cost=$0.000810 | trace_id=a1b2c3d4e5f6... Severity gate: CLEAR (0 HIGH findings) ``` **Or on gate trigger:** ``` PR Review Cost | pr=42 | calls=5 | tokens=3,241 | cost=$0.000810 | trace_id=a1b2c3d4e5f6... Severity gate triggered: 1 HIGH finding found. Exit code: 1 ``` **Target:** The Action must post at least one PR comment with the review table, exit 0 on a clean diff, and exit 1 on a diff that contains a hardcoded secret. Cost must be logged with the correct trace ID in both cases. This confirms end-to-end integration: LLM call → structured output → PR comment → severity gate → OTEL instrumentation. ### Model source currency - Anthropic model list: https://docs.anthropic.com/en/docs/about-claude/models/overview - OpenAI model list: https://platform.openai.com/docs/models - verifiedOn: 2026-08-07 - `LLM_MODEL` is required and has no implicit default; choose a supported model for the configured provider at setup time. ## Source code A reference implementation of AI PR Reviewer (Capstone) — README, guide, source and tests — is at https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/ai-pr-reviewer.