--- title: "Prompt Evaluation Framework" description: "Eval engineering is an explicitly named skill in senior AI Engineer job descriptions — this project lets you say you've built a CI-integrated eval harness, not..." source: "https://confidentprep.com/paths/ai-engineering-fundamentals/prompt-eval/" path: "AI Engineering Fundamentals" repository: "https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/prompt-eval" token_estimate: 3253 --- # Prompt Evaluation Framework ## Overview A test harness for LLM prompts: define test cases in YAML, score responses with an LLM judge across multiple dimensions, and get a nonzero exit code on failure so it runs in CI. Every project before this one, you tested by eyeballing the output. This is where you learn to test AI systems the way you'd test any other code — repeatably, and without a human in the loop every time. ## What to do **Path:** 1 **Position:** 9 of 12 **Difficulty:** 🟢 Local — one API key, no Docker, no cloud **Estimated time:** 3–4 hours **AWS cost:** None --- ### Agent Pickup Instructions **This spec is self-contained. Build this project without reading any other spec file.** ```bash mkdir -p path-1/p1-09-prompt-eval cd path-1/p1-09-prompt-eval pytest tests/ -v python src/main.py run --suite test_suites/qa_suite.yaml python src/main.py diff --suite test_suites/qa_suite.yaml --prompt-a prompts/v1.txt --prompt-b prompts/v2.txt echo $? # should be 1 if any test failed ``` **Done when:** - [ ] `pytest tests/ -v` → minimum 4 tests, all green - [ ] `run` command produces JSON report and prints pass/fail per test case - [ ] `diff` command produces a comparison table of two prompts - [ ] Exit code is 1 when any test fails (for CI integration) - [ ] Judge reasoning is included in the JSON report per dimension - [ ] No file in `src/` exceeds 200 lines --- ### What this project is A test harness for prompts. Defines test cases in YAML with expected inputs and scoring dimensions, calls an LLM-as-judge to score responses on each dimension, and produces a JSON report with pass/fail per case. A diff mode compares two prompt versions side by side. The exit code contract (1 on failure) makes this usable in a CI pipeline — drop it in a GitHub Action and your prompts have a regression test. --- ### What the learner achieves "I built a prompt evaluation framework with YAML test cases, LLM-as-judge scoring across multiple dimensions, and exit code 1 on failure — I can drop it in a GitHub Action and get notified when a prompt change degrades quality." --- ### Folder structure ``` p1-09-prompt-eval/ ├── README.md ├── GUIDE.md ├── .env.example ├── requirements.txt ├── test_suites/ │ └── qa_suite.yaml ← sample test suite (include in project) ├── prompts/ │ ├── v1.txt ← sample prompt version 1 (include in project) │ └── v2.txt ← sample prompt version 2 (include in project) ├── src/ │ ├── main.py ← CLI (run / diff subcommands) │ ├── llm.py ← provider-agnostic LLM wrapper │ ├── runner.py ← test suite runner │ ├── judge.py ← LLM-as-judge scorer │ └── reporter.py ← JSON report + table printer └── tests/ └── test_eval.py ``` --- ### .env.example ```bash LLM_PROVIDER=anthropic LLM_API_KEY= LLM_MODEL= # Separate judge model (can be same or different from generation model) JUDGE_MODEL= JUDGE_PROVIDER=anthropic JUDGE_API_KEY= ``` --- ### requirements.txt ``` anthropic==0.40.0 openai==1.58.0 ollama==0.4.4 pyyaml==6.0.2 python-dotenv==1.0.1 ``` --- ### test_suites/qa_suite.yaml (include this file in the project) ```yaml suite_name: "Q&A Quality Suite" description: "Tests for a customer support assistant prompt" scoring_dimensions: - name: precision description: "Answer addresses exactly what was asked, no more" passing_threshold: 3 # score must be >= 3 (out of 5) to pass - name: completeness description: "Answer covers all key points needed to resolve the question" passing_threshold: 3 - name: tone description: "Answer is professional and empathetic" passing_threshold: 4 test_cases: - id: tc-001 description: "Refund policy question" input: "Can I get a refund after 30 days?" context: "Our refund policy is 30 days from purchase. After 30 days, store credit only." - id: tc-002 description: "Shipping timeline question" input: "When will my order arrive?" context: "Standard shipping is 5–7 business days. Express is 2 business days." - id: tc-003 description: "Account locked question" input: "I can't log in, my account seems locked." context: "Accounts lock after 5 failed login attempts. Reset via email link." ``` --- ### src/ — what to implement #### src/llm.py **`get_completion(prompt: str, system: str = "", model_override: str = None, provider_override: str = None) -> str`** - Standard router. If `model_override`/`provider_override` provided, use those instead of env vars. - Used for both generation and judging (with different models potentially). #### src/runner.py **`load_suite(yaml_path: str) -> dict`** — load and return parsed YAML as dict **`run_test_case(test_case: dict, prompt_template: str, dimensions: list[dict]) -> dict`** - Fill prompt: replace `{input}` and `{context}` in template with test case values - Call LLM to generate response - Score with judge (one call per dimension via `judge.score_response()`) - Return: `{"id": tc_id, "input": ..., "response": ..., "scores": {"precision": {"score": 4, "passed": True, "reasoning": "..."}}, "passed": bool, "latency_ms": N}` **`run_suite(suite: dict, prompt_template: str) -> dict`** - Run all test cases in suite - Return: `{"suite_name": ..., "results": [...], "summary": {"total": N, "passed": N, "failed": N, "pass_rate": 0.75}}` #### src/judge.py **`score_response(response: str, test_input: str, context: str, dimension: dict) -> dict`** - Returns `{"score": 1-5, "passed": bool, "reasoning": str}` - Judge prompt: ``` You are an objective evaluator. Score the following AI response on one dimension. Dimension: {dimension["name"]} Description: {dimension["description"]} User question: {test_input} Context provided to the AI: {context} AI Response: {response} Score 1-5 where: 1 = completely fails this dimension 3 = acceptable, meets minimum bar 5 = excellent, exceeds expectations Return JSON: {"score": N, "reasoning": "one sentence"} ``` - Parse JSON from response (strip fences) - `passed = score >= dimension["passing_threshold"]` - On parse failure: return `{"score": 0, "passed": False, "reasoning": "Judge returned invalid JSON"}` #### src/reporter.py **`print_results_table(results: list[dict], dimensions: list[dict]) -> None`** - Print a table: rows = test cases, columns = dimensions + overall pass/fail - Mark pass with ✓, fail with ✗ **`save_json_report(suite_result: dict, output_path: str) -> None`** — `json.dump` with indent=2 **`print_diff_table(prompt_a_results: dict, prompt_b_results: dict) -> None`** - Side-by-side: test case | prompt A scores | prompt B scores | winner #### src/main.py CLI with two subcommands: `python src/main.py run --suite [--prompt ] [--output ]` - Default prompt template: `"Answer this question using the context provided.\n\nContext: {context}\n\nQuestion: {input}"` - Run suite, print table, save JSON if `--output` provided - Exit with code `1` if any test case failed `python src/main.py diff --suite --prompt-a --prompt-b [--output ]` - Run suite with prompt A, then with prompt B - Print diff table - Print: `Prompt A: X/N passed | Prompt B: Y/N passed` --- ### tests/ — what to test **File:** `tests/test_eval.py` **Test 1 — load_suite parses YAML correctly:** Load `test_suites/qa_suite.yaml`. Assert `suite["suite_name"]` is a string and `len(suite["test_cases"]) >= 3`. **Test 2 — judge returns valid structure:** Mock `llm.get_completion` to return `'{"score": 4, "reasoning": "Good answer"}'`. Call `score_response(...)`. Assert result has "score", "passed", "reasoning" keys. **Test 3 — judge handles malformed JSON:** Mock `get_completion` to return `"not json"`. Call `score_response(...)`. Assert `result["score"] == 0` and `result["passed"] == False`. **Test 4 — pass/fail threshold applied correctly:** Mock judge to return score=3. Set `passing_threshold=4`. Assert `passed == False`. Set `passing_threshold=3`. Assert `passed == True`. **Test 5 — reporter saves valid JSON:** Create a sample result dict. Call `save_json_report(result, tmp_path)`. Assert file exists and parses as JSON. --- ### README.md content ```markdown # Prompt Evaluation Framework A test harness for LLM prompts. Define test cases in YAML, score responses with an LLM judge across multiple dimensions, and get exit code 1 on failure — so you can run it in CI. ## Setup ```bash cd p1-09-prompt-eval cp .env.example .env pip install -r requirements.txt ``` ## Run ```bash ## Run a test suite python src/main.py run --suite test_suites/qa_suite.yaml ## Compare two prompt versions python src/main.py diff --suite test_suites/qa_suite.yaml \ --prompt-a prompts/v1.txt --prompt-b prompts/v2.txt ## Save report python src/main.py run --suite test_suites/qa_suite.yaml --output report.json ``` Expected output: ``` Running qa_suite (3 test cases)... Test Case precision completeness tone PASS ───────────────────────────────────────────────────── tc-001 Refund 4/5 ✓ 3/5 ✓ 5/5 ✓ ✓ tc-002 Shipping 2/5 ✗ 4/5 ✓ 4/5 ✓ ✗ tc-003 Locked 3/5 ✓ 3/5 ✓ 4/5 ✓ ✓ Summary: 2/3 passed (67%) ``` ## Tests ```bash pytest tests/ -v ``` ## What to try next - Add this to a GitHub Action: on PR, run your prompt suite and fail the build if any test fails - Add a new dimension: "citation" — does the answer cite a source? - Write a test suite for the RAG pipeline from p1-03 ``` --- ### GUIDE.md content ```markdown # Build guide: Prompt Evaluation Framework ## What you're building and why it matters "This prompt feels better" is not engineering. Prompt evaluation is. Real teams maintain test suites for their prompts for the same reason they maintain test suites for their code: to detect regressions before deployment. When you change a system prompt and 20% of your test cases fail, you catch it before your users do. LLM-as-judge scoring — using one LLM to evaluate another's output — is the current best practice for open-ended evaluation. It is not perfect (the judge has biases) but it is far better than manual review at scale. ## The decision that matters in this build **One LLM call per dimension vs one call for all dimensions.** You could ask the judge to score all dimensions in one call (cheaper, fewer calls) or one dimension at a time (more accurate, more expensive). Use one call per dimension. When you bundle dimensions, the judge anchors on the first score and adjusts the others relatively — a well-known bias. Separate calls give independent scores. ## What will break **The judge is not objective.** LLM judges have a length bias (longer answers score higher) and a self-preference bias (Claude judges favour Claude responses). Mitigate by making the scoring rubric very explicit in the judge prompt. "Score 3 means acceptable, not good" prevents the judge from giving 4s to everything out of politeness. **Exit code 1 requires explicit `sys.exit(1)`**, not just a non-zero return from a function. Test this explicitly — it is the whole point of the CI integration. ## How to talk about this in an interview "I built a prompt evaluation framework where each test case is scored on independent dimensions by an LLM judge. The key engineering decision was one judge call per dimension to prevent anchor bias. I can drop this in a GitHub Action — exit code 1 on any failure means the PR is blocked if prompt quality degrades." ``` --- ### Failure modes to handle | Failure | Where | How to handle | |---------|-------|---------------| | YAML parse error | `runner.load_suite()` | Print clear error with filename and line, exit code 1 | | Judge returns invalid JSON | `judge.score_response()` | Return score=0, passed=False, note in reasoning | | LLM error during test run | `runner.run_test_case()` | Mark test case as failed with reason, continue suite | | Prompt template missing `{input}` | `runner.run_test_case()` | Raise `ValueError` with template problem | | Output file write fails | `reporter.save_json_report()` | Print warning, don't fail the run | --- ### The metric this project measures **Pass rate per suite** — fraction of test cases that pass all dimensions. Printed as: `Summary: X/N passed (Y%)` **Per-dimension pass rate** — in the JSON report, breakdown by dimension. These numbers are what you compare before and after changing a prompt. ### Model source currency - Anthropic model list: https://docs.anthropic.com/en/docs/about-claude/models/overview - OpenAI model list: https://platform.openai.com/docs/models - verifiedOn: 2026-08-07 - `LLM_MODEL` is required and has no implicit default; choose a supported model for the configured provider at setup time. ## Source code A reference implementation of Prompt Evaluation Framework — README, guide, source and tests — is at https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/prompt-eval.