Prompt Evaluation Framework

Path: 1
Position: 9 of 12
Difficulty: 🟢 Local — one API key, no Docker, no cloud
Estimated time: 3–4 hours
AWS cost: None

---

Agent Pickup Instructions

This spec is self-contained. Build this project without reading any other spec file.

mkdir -p path-1/p1-09-prompt-eval
cd path-1/p1-09-prompt-eval

pytest tests/ -v
python src/main.py run --suite test_suites/qa_suite.yaml
python src/main.py diff --suite test_suites/qa_suite.yaml --prompt-a prompts/v1.txt --prompt-b prompts/v2.txt
echo $?   # should be 1 if any test failed

Done when:

• pytest tests/ -v → minimum 4 tests, all green
• run command produces JSON report and prints pass/fail per test case
• diff command produces a comparison table of two prompts
• Exit code is 1 when any test fails (for CI integration)
• Judge reasoning is included in the JSON report per dimension
• No file in src/ exceeds 200 lines

---

What this project is

A test harness for prompts. Defines test cases in YAML with expected inputs and scoring dimensions, calls an LLM-as-judge to score responses on each dimension, and produces a JSON report with pass/fail per case. A diff mode compares two prompt versions side by side. The exit code contract (1 on failure) makes this usable in a CI pipeline — drop it in a GitHub Action and your prompts have a regression test.

---

What the learner achieves

"I built a prompt evaluation framework with YAML test cases, LLM-as-judge scoring across multiple dimensions, and exit code 1 on failure — I can drop it in a GitHub Action and get notified when a prompt change degrades quality."

---

Folder structure

p1-09-prompt-eval/
├── README.md
├── GUIDE.md
├── .env.example
├── requirements.txt
├── test_suites/
│   └── qa_suite.yaml     ← sample test suite (include in project)
├── prompts/
│   ├── v1.txt            ← sample prompt version 1 (include in project)
│   └── v2.txt            ← sample prompt version 2 (include in project)
├── src/
│   ├── main.py           ← CLI (run / diff subcommands)
│   ├── llm.py            ← provider-agnostic LLM wrapper
│   ├── runner.py         ← test suite runner
│   ├── judge.py          ← LLM-as-judge scorer
│   └── reporter.py       ← JSON report + table printer
└── tests/
    └── test_eval.py

---

.env.example

LLM_PROVIDER=anthropic
LLM_API_KEY=
LLM_MODEL=

# Separate judge model (can be same or different from generation model)
JUDGE_MODEL=
JUDGE_PROVIDER=anthropic
JUDGE_API_KEY=

---

requirements.txt

anthropic==0.40.0
openai==1.58.0
ollama==0.4.4
pyyaml==6.0.2
python-dotenv==1.0.1

---

test_suites/qa_suite.yaml (include this file in the project)

suite_name: "Q&A Quality Suite"
description: "Tests for a customer support assistant prompt"

scoring_dimensions:
  - name: precision
    description: "Answer addresses exactly what was asked, no more"
    passing_threshold: 3      # score must be >= 3 (out of 5) to pass
  - name: completeness
    description: "Answer covers all key points needed to resolve the question"
    passing_threshold: 3
  - name: tone
    description: "Answer is professional and empathetic"
    passing_threshold: 4

test_cases:
  - id: tc-001
    description: "Refund policy question"
    input: "Can I get a refund after 30 days?"
    context: "Our refund policy is 30 days from purchase. After 30 days, store credit only."
  - id: tc-002
    description: "Shipping timeline question"
    input: "When will my order arrive?"
    context: "Standard shipping is 5–7 business days. Express is 2 business days."
  - id: tc-003
    description: "Account locked question"
    input: "I can't log in, my account seems locked."
    context: "Accounts lock after 5 failed login attempts. Reset via email link."

---

src/ — what to implement

src/llm.py

get_completion(prompt: str, system: str = "", model_override: str = None, provider_override: str = None) -> str

• Standard router. If model_override/provider_override provided, use those instead of env vars.
• Used for both generation and judging (with different models potentially).

src/runner.py

load_suite(yaml_path: str) -> dict — load and return parsed YAML as dict

run_test_case(test_case: dict, prompt_template: str, dimensions: list[dict]) -> dict

• Fill prompt: replace {input} and {context} in template with test case values
• Call LLM to generate response
• Score with judge (one call per dimension via judge.score_response())
• Return: {"id": tc_id, "input": ..., "response": ..., "scores": {"precision": {"score": 4, "passed": True, "reasoning": "..."}}, "passed": bool, "latency_ms": N}

run_suite(suite: dict, prompt_template: str) -> dict

• Run all test cases in suite
• Return: {"suite_name": ..., "results": [...], "summary": {"total": N, "passed": N, "failed": N, "pass_rate": 0.75}}

src/judge.py

score_response(response: str, test_input: str, context: str, dimension: dict) -> dict

• Returns {"score": 1-5, "passed": bool, "reasoning": str}
• Judge prompt:

  You are an objective evaluator. Score the following AI response on one dimension.
  
  Dimension: {dimension["name"]}
  Description: {dimension["description"]}
  
  User question: {test_input}
  Context provided to the AI: {context}
  AI Response: {response}
  
  Score 1-5 where:
  1 = completely fails this dimension
  3 = acceptable, meets minimum bar
  5 = excellent, exceeds expectations
  
  Return JSON: {"score": N, "reasoning": "one sentence"}
  

• Parse JSON from response (strip fences)
• passed = score >= dimension["passing_threshold"]
• On parse failure: return {"score": 0, "passed": False, "reasoning": "Judge returned invalid JSON"}

src/reporter.py

print_results_table(results: list[dict], dimensions: list[dict]) -> None

• Print a table: rows = test cases, columns = dimensions + overall pass/fail
• Mark pass with ✓, fail with ✗

save_json_report(suite_result: dict, output_path: str) -> None — json.dump with indent=2

print_diff_table(prompt_a_results: dict, prompt_b_results: dict) -> None

• Side-by-side: test case | prompt A scores | prompt B scores | winner

src/main.py

CLI with two subcommands:

python src/main.py run --suite <yaml> [--prompt <template.txt>] [--output <report.json>]

• Default prompt template: "Answer this question using the context provided.\n\nContext: {context}\n\nQuestion: {input}"
• Run suite, print table, save JSON if --output provided
• Exit with code 1 if any test case failed

python src/main.py diff --suite <yaml> --prompt-a <a.txt> --prompt-b <b.txt> [--output <report.json>]

• Run suite with prompt A, then with prompt B
• Print diff table
• Print: Prompt A: X/N passed | Prompt B: Y/N passed

---

tests/ — what to test

File: tests/test_eval.py

Test 1 — load_suite parses YAML correctly: Load test_suites/qa_suite.yaml. Assert suite["suite_name"] is a string and len(suite["test_cases"]) >= 3.

Test 2 — judge returns valid structure: Mock llm.get_completion to return '{"score": 4, "reasoning": "Good answer"}'. Call score_response(...). Assert result has "score", "passed", "reasoning" keys.

Test 3 — judge handles malformed JSON: Mock get_completion to return "not json". Call score_response(...). Assert result["score"] == 0 and result["passed"] == False.

Test 4 — pass/fail threshold applied correctly: Mock judge to return score=3. Set passing_threshold=4. Assert passed == False. Set passing_threshold=3. Assert passed == True.

Test 5 — reporter saves valid JSON: Create a sample result dict. Call save_json_report(result, tmp_path). Assert file exists and parses as JSON.

---

README.md content

# Prompt Evaluation Framework

A test harness for LLM prompts. Define test cases in YAML, score responses with
an LLM judge across multiple dimensions, and get exit code 1 on failure — so you
can run it in CI.

## Setup

```bash
cd p1-09-prompt-eval
cp .env.example .env
pip install -r requirements.txt

Run

# Run a test suite
python src/main.py run --suite test_suites/qa_suite.yaml

# Compare two prompt versions
python src/main.py diff --suite test_suites/qa_suite.yaml \
    --prompt-a prompts/v1.txt --prompt-b prompts/v2.txt

# Save report
python src/main.py run --suite test_suites/qa_suite.yaml --output report.json

Expected output:

Running qa_suite (3 test cases)...

Test Case        precision  completeness  tone   PASS
─────────────────────────────────────────────────────
tc-001 Refund    4/5 ✓      3/5 ✓         5/5 ✓   ✓
tc-002 Shipping  2/5 ✗      4/5 ✓         4/5 ✓   ✗
tc-003 Locked    3/5 ✓      3/5 ✓         4/5 ✓   ✓

Summary: 2/3 passed (67%)

Tests

pytest tests/ -v

What to try next

• Add this to a GitHub Action: on PR, run your prompt suite and fail the build if any test fails
• Add a new dimension: "citation" — does the answer cite a source?
• Write a test suite for the RAG pipeline from p1-03

---

## GUIDE.md content

```markdown
# Build guide: Prompt Evaluation Framework

## What you're building and why it matters

"This prompt feels better" is not engineering. Prompt evaluation is. Real teams
maintain test suites for their prompts for the same reason they maintain test suites
for their code: to detect regressions before deployment. When you change a system
prompt and 20% of your test cases fail, you catch it before your users do.
LLM-as-judge scoring — using one LLM to evaluate another's output — is the current
best practice for open-ended evaluation. It is not perfect (the judge has biases)
but it is far better than manual review at scale.

## The decision that matters in this build

**One LLM call per dimension vs one call for all dimensions.** You could ask the judge
to score all dimensions in one call (cheaper, fewer calls) or one dimension at a time
(more accurate, more expensive). Use one call per dimension. When you bundle dimensions,
the judge anchors on the first score and adjusts the others relatively — a well-known
bias. Separate calls give independent scores.

## What will break

**The judge is not objective.** LLM judges have a length bias (longer answers score higher)
and a self-preference bias (Claude judges favour Claude responses). Mitigate by making
the scoring rubric very explicit in the judge prompt. "Score 3 means acceptable, not good"
prevents the judge from giving 4s to everything out of politeness.

**Exit code 1 requires explicit `sys.exit(1)`**, not just a non-zero return from a function.
Test this explicitly — it is the whole point of the CI integration.

## How to talk about this in an interview

"I built a prompt evaluation framework where each test case is scored on independent
dimensions by an LLM judge. The key engineering decision was one judge call per
dimension to prevent anchor bias. I can drop this in a GitHub Action — exit code 1
on any failure means the PR is blocked if prompt quality degrades."

---

Failure modes to handle

Failure	Where	How to handle
YAML parse error	runner.load_suite()	Print clear error with filename and line, exit code 1
Judge returns invalid JSON	judge.score_response()	Return score=0, passed=False, note in reasoning
LLM error during test run	runner.run_test_case()	Mark test case as failed with reason, continue suite
Prompt template missing {input}	runner.run_test_case()	Raise ValueError with template problem
Output file write fails	reporter.save_json_report()	Print warning, don't fail the run

---

The metric this project measures

Pass rate per suite — fraction of test cases that pass all dimensions. Printed as: Summary: X/N passed (Y%) Per-dimension pass rate — in the JSON report, breakdown by dimension. These numbers are what you compare before and after changing a prompt.

Model source currency

• Anthropic model list: https://docs.anthropic.com/en/docs/about-claude/models/overview
• OpenAI model list: https://platform.openai.com/docs/models
• verifiedOn: 2026-08-07
• LLM_MODEL is required and has no implicit default; choose a supported model for the configured provider at setup time.