--- title: "LLM Test Generator" description: "Test generation that actually runs and retries is a meaningfully different (and more hireable) claim than 'I used Copilot to write tests' — shows you can..." source: "https://confidentprep.com/paths/ai-augmented-engineering/llm-test-generator/" path: "AI-Augmented Engineering" repository: "https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/llm-test-generator" token_estimate: 4442 --- # LLM Test Generator ## Overview Generates pytest test cases for Python functions using an LLM, then validates them by actually running pytest — retrying with the failure context fed back in when a generated test breaks. The retry-with-error-context loop here is the same shape you'll see again in this path's SQL agent — once you've built it once, you recognize it everywhere. ## What to do | Field | Value | |---|---| | Path | 2 — AI-Augmented Engineering | | Position | 5 of 12 | | Difficulty | 🟢 Local only | | Estimated time | 4 hours | | AWS cost | None | --- ### Agent Pickup Instructions ```bash # 1. Enter project cd projects/p2-05-llm-test-generator # 2. Create virtual environment python -m venv .venv && source .venv/bin/activate # 3. Install dependencies pip install -r requirements.txt # 4. Copy env file and add your API key cp .env.example .env # 5. Generate tests for a module python src/main.py src/main.py # 6. Generate with coverage report python src/main.py src/main.py --coverage # 7. Run tests pytest tests/ -v ``` **Done when:** - [ ] `python src/main.py ` produces a `test_.py` that runs without syntax errors - [ ] Generated file contains only tests that actually pass (failing tests removed) - [ ] `--coverage` flag runs `pytest --cov` and prints a branch count - [ ] Retry loop kicks in on failure and shows `Retry N/3: injecting error into prompt` - [ ] All 4 tests pass --- ### What this project is An LLM-powered test generator that uses AST parsing to extract Python function metadata, sends structured function information to an LLM with a pytest-aware prompt, validates the generated tests by actually running them with pytest, and retries up to 3 times on syntax errors or test failures — injecting the error message into the next prompt so the model can self-correct. The final output file contains only tests that actually pass, not the raw LLM output. --- ### What the learner achieves "I built a test-generation pipeline with a self-correcting retry loop: the LLM generates pytest tests, I run them, and if they fail I inject the error back into the next prompt. The final file contains only tests that pass — I don't ship broken tests." --- ### Folder structure ``` p2-05-llm-test-generator/ ├── src/ │ ├── main.py # CLI: [--coverage] [--max-retries N] │ ├── llm.py # Provider-agnostic LLM wrapper │ ├── parser.py # extract_function_metadata -> list[FunctionMeta] │ ├── generator.py # build_generation_prompt, generate_tests_for_function │ ├── validator.py # run_pytest_on_generated, parse_pytest_output, retry_with_error │ └── reporter.py # print_coverage_summary ├── tests/ │ ├── fixtures/ │ │ ├── simple_math.py # Simple functions: add, subtract, divide (raises ZeroDivisionError) │ │ └── string_utils.py # String functions: truncate, slugify, count_words │ ├── test_parser.py │ ├── test_validator.py │ └── test_generator.py ├── .env.example ├── requirements.txt └── README.md ``` --- ### .env.example ```bash # LLM provider: anthropic | openai | ollama LLM_PROVIDER=anthropic # Model to use LLM_MODEL= # API key for Anthropic ANTHROPIC_API_KEY= # API key for OpenAI (leave blank if using anthropic or ollama) OPENAI_API_KEY= # Ollama base URL OLLAMA_BASE_URL=http://localhost:11434 # Max retry attempts on failing tests (1-5) MAX_RETRIES=3 # Output directory for generated test files OUTPUT_DIR=. # Pytest coverage minimum threshold (0-100, used with --coverage) COVERAGE_MIN=0 ``` --- ### requirements.txt ``` anthropic==0.30.0 openai==1.35.0 python-dotenv==1.0.1 rich==13.7.1 pytest==8.2.2 pytest-cov==5.0.0 pytest-mock==3.14.0 ``` --- ### src/ — what to implement #### `src/llm.py` **`get_completion(prompt: str, system: str = "") -> str`** - Standard interface: reads `LLM_PROVIDER` and `LLM_MODEL` from env - Routes to Anthropic, OpenAI, or Ollama - Edge cases: raise `RuntimeError` if API key is missing --- #### `src/parser.py` **`FunctionMeta` (dataclass)** ```python @dataclass class FunctionMeta: name: str params: list[dict] # [{"name": str, "type": str | None, "default": str | None}] return_type: str | None docstring: str | None source_lines: list[str] # Raw source lines of the function body lineno: int raises: list[str] # Exception names found in raise statements ``` **`extract_function_metadata(filepath: str) -> list[FunctionMeta]`** - Inputs: path to a `.py` file - Output: list of `FunctionMeta` for all public functions - Behavior: - Parse with `ast.parse` - For each `FunctionDef` / `AsyncFunctionDef` where `not name.startswith("_")` - Extract params, return type, docstring using `ast` - Extract source lines using `linecache.getlines` - Walk AST subtree to find `Raise` nodes; extract exception names - Edge cases: syntax error → raise `SyntaxError` with filename; return `[]` for files with no public functions --- #### `src/generator.py` **`build_generation_prompt(func: FunctionMeta, prior_error: str = "") -> str`** - Inputs: function metadata, optional error from previous attempt - Output: formatted prompt string - Behavior: - Build a user prompt with: function name, params with types, return type, docstring, exceptions raised - If `prior_error` is non-empty, prepend: "The previous attempt produced this error. Fix it:\n\n{prior_error}\n\n" - Include the raw function source lines so the model can see the actual implementation - Instruct: "Generate pytest test cases. Use only stdlib. Import the function from its module. Return only valid Python code, no markdown fences." **`generate_tests_for_function(func: FunctionMeta, source_module: str, max_retries: int = 3) -> str | None`** - Inputs: function metadata, Python module path (e.g., `simple_math`), max retry count - Output: a Python string containing valid passing pytest test code, or `None` if all retries fail - Behavior: - Build system prompt: "You are an expert at writing pytest tests. Return only valid Python code, no explanation, no markdown." - Call `build_generation_prompt` with no prior error - Call `get_completion` - Call `validate_and_retry` loop - This function does NOT write files — it returns a string **`validate_and_retry(func_name: str, test_code: str, source_file: str, max_retries: int) -> str | None`** - Inputs: function name, generated test code string, path to the source module, max retries - Output: validated test code string (passing), or `None` - Behavior: call `run_pytest_on_generated`; if it returns failure, build new prompt with error injected; retry; log `Retry N/{max_retries}: injecting error into prompt` --- #### `src/validator.py` **`run_pytest_on_generated(test_code: str, source_file: str, func_name: str) -> dict`** - Inputs: generated test code string, path to source Python file, function name - Output: `{"passed": bool, "error": str, "output": str}` - Behavior: - Write test code to a temp file: `tmp_test_{func_name}_{uuid}.py` - Run `pytest --tb=short -q` via `subprocess.run` - Capture stdout + stderr - Detect syntax error: if `SyntaxError` appears in output → `passed=False, error=` - Detect test failure: non-zero exit code → `passed=False, error=` - Always delete temp file in a `finally` block - Edge cases: if `subprocess.run` itself raises (pytest not installed), raise `RuntimeError` **`parse_pytest_output(output: str) -> dict`** - Inputs: raw pytest stdout/stderr string - Output: `{"total": int, "passed": int, "failed": int, "errors": int}` - Behavior: parse the summary line `N passed, M failed` using regex; handle `no tests ran` as `{"total": 0, "passed": 0, "failed": 0, "errors": 1}` **`filter_passing_tests(all_test_code: str, source_file: str) -> str`** - Inputs: combined test code for all functions, path to source file - Output: test code with only the passing test functions - Behavior: split on `def test_` boundaries; run each test function individually; keep only passing ones; reassemble with imports at top --- #### `src/reporter.py` **`print_coverage_summary(source_file: str, test_file: str) -> None`** - Inputs: path to source file, path to generated test file - Output: prints coverage summary to stdout - Behavior: run `pytest {test_file} --cov={source_module} --cov-report=term-missing -q`; capture and print output; parse for branch count from `TOTAL` line --- #### `src/main.py` CLI entry point using `argparse`. Arguments: - Positional: `source_file` — path to Python file to generate tests for - `--coverage` — after generating tests, run pytest-cov and print summary - `--max-retries N` — override `MAX_RETRIES` env var Behavior: 1. Load `.env` 2. Call `extract_function_metadata`; print `Found N public functions` 3. For each function: call `generate_tests_for_function`; collect results 4. Filter: keep only functions where generation succeeded 5. Assemble final test file: shared imports at top + all passing test blocks 6. Write to `test_.py` in `OUTPUT_DIR` 7. Print `Generated test_.py | N functions tested | M retries used` 8. If `--coverage`: call `print_coverage_summary` --- ### tests/ — what to test #### Test 1 — pytest validator detects SyntaxError (`test_validator.py`) - Call `run_pytest_on_generated` with intentionally broken Python: `def test_foo(:\n pass` - Assert `passed == False` - Assert `"SyntaxError"` appears in the `error` field #### Test 2 — Retry injects error message into next prompt (`test_generator.py`) - Mock `get_completion` to track call arguments - Set up `validate_and_retry` to fail on the first attempt (mock `run_pytest_on_generated` to return `passed=False, error="AssertionError: ..."`) - Assert `get_completion` is called twice - Assert the second call's prompt contains the error string from the first failure #### Test 3 — Only passing tests in final file (`test_validator.py`) - Create a test code string with one passing test and one failing test - Call `filter_passing_tests` - Assert the returned string contains the passing test function name - Assert the returned string does not contain the failing test function name #### Test 4 — Coverage summary shows branch count (`test_validator.py`) - Run `print_coverage_summary` on `tests/fixtures/simple_math.py` with a real test file - Capture stdout - Assert output contains a number followed by `%` - (Skip if pytest-cov not installed — use `pytest.importorskip("pytest_cov")`) #### Test 5 — Parser extracts exceptions from raise statements (`test_parser.py`) - `simple_math.py` has `def divide(a, b): if b == 0: raise ZeroDivisionError(...)` - Call `extract_function_metadata` - Assert the `divide` function's `raises` list contains `"ZeroDivisionError"` #### Test 6 — Generation prompt includes prior error when provided (`test_generator.py`) - Call `build_generation_prompt` with `prior_error="AssertionError: expected 4, got 5"` - Assert the returned prompt string contains the error text - Assert the returned prompt contains the word "Fix" --- ### README.md content ```markdown # LLM Test Generator Generates pytest test cases for Python functions using LLM, then validates them by actually running pytest — retrying with error context on failure. ## Quick start ```bash python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt cp .env.example .env # add your API key ## Generate tests python src/main.py src/my_module.py ## Generate with coverage report python src/main.py src/my_module.py --coverage ## Allow more retries python src/main.py src/my_module.py --max-retries 5 ``` ## How the retry loop works ``` Generate tests → Run pytest → Pass? → Write to file ↓ Fail Inject error into next prompt → Retry (max 3) ↓ All retries fail Skip function, log warning ``` ## Output - `test_.py` — contains only tests that actually pass - Coverage report (with `--coverage`) — branch count for each function ## Running tests ```bash pytest tests/ -v ``` ``` --- ### GUIDE.md content ```markdown # Build Guide — LLM Test Generator ## Step 1 — Extract function metadata with AST Beyond p2-04, also extract `raises` (exceptions in raise statements): ```python def get_raises(func_node): raises = [] for node in ast.walk(func_node): if isinstance(node, ast.Raise) and node.exc: if isinstance(node.exc, ast.Call): raises.append(ast.unparse(node.exc.func)) elif isinstance(node.exc, ast.Name): raises.append(node.exc.id) return raises ``` This tells the LLM "this function raises ZeroDivisionError" so it generates a test for the error case. ## Step 2 — The generation system prompt ``` You are an expert Python test engineer. Generate pytest test functions for the given Python function. Rules: - Use only stdlib and pytest - Import the function from its module at the top - Test the happy path, edge cases, and any documented exceptions - Return ONLY valid Python code — no markdown, no explanation - Each test function must start with def test_ ``` ## Step 3 — The retry loop ```python def validate_and_retry(func_name, test_code, source_file, max_retries): prior_error = "" for attempt in range(max_retries): if attempt > 0: print(f"Retry {attempt}/{max_retries}: injecting error into prompt") new_prompt = build_generation_prompt(func, prior_error=prior_error) test_code = get_completion(new_prompt, system=SYSTEM_PROMPT) result = run_pytest_on_generated(test_code, source_file, func_name) if result["passed"]: return test_code prior_error = result["error"] return None ``` ## Step 4 — Running pytest programmatically ```python import subprocess, tempfile, os def run_pytest_on_generated(test_code, source_file, func_name): tmp_path = f"/tmp/tmp_test_{func_name}_{uuid4().hex}.py" try: with open(tmp_path, "w") as f: f.write(test_code) result = subprocess.run( ["pytest", tmp_path, "--tb=short", "-q"], capture_output=True, text=True ) passed = result.returncode == 0 error = result.stdout + result.stderr return {"passed": passed, "error": error if not passed else ""} finally: os.unlink(tmp_path) ``` ## Step 5 — Filter passing tests After all functions are generated, run each test function individually to build the final file. This is safer than trusting the whole file to pass after generation. ## Debugging tips - If the LLM keeps returning markdown fences, add "DO NOT include markdown code fences" to the system prompt - If pytest can't find the source module, the import path is wrong — check `sys.path` manipulation in the generated file - Start with a simple fixture like `add(a, b): return a + b` before testing complex functions ## How to talk about this in an interview **"Why validate by actually running pytest?"** > Because the LLM doesn't run code — it guesses. The only ground truth is whether the test actually passes. Running pytest is the correctness check the LLM can't provide. **"What does the retry loop buy you?"** > Self-correction. The first attempt fails about 30% of the time on complex functions. With error context, the second attempt succeeds most of the time. Three retries handles ~95% of cases in my testing. **"How do you ensure the final test file only has passing tests?"** > I filter: run each test function individually, keep only the passing ones, then assemble. This means the output file is a guarantee, not just output — it always runs clean. ``` --- ### Failure modes to handle | Failure | Where | How to handle | |---|---|---| | LLM returns markdown fences wrapping code | `generator.py` | Strip ` ```python ` and ` ``` ` before writing to temp file | | Temp pytest file has wrong import path | `validator.py` | Inject `sys.path.insert(0, os.path.dirname(source_file))` at top of generated test | | All retries fail for a function | `generator.py` | Log warning `Could not generate passing tests for {func_name}`; skip | | Source file has no public functions | `main.py` | Print `No public functions found in .` and exit 0 | | pytest is not installed | `validator.py` | Raise `RuntimeError("pytest not found — run pip install pytest")` | | Generated test imports non-existent module | `validator.py` | Caught by pytest as `ImportError`; appears in `error` field; retry handles it | --- ### The metric this project measures **What is measured:** Number of functions tested, retry count, and pass rate (tests that passed / tests attempted). **Format (stdout):** ``` Found 3 public functions Generated test_simple_math.py | 3 functions tested | 2 retries used | pass_rate=100% ``` **Target:** For `tests/fixtures/simple_math.py` (3 simple functions), the final test file must contain at least 3 test functions that all pass with `pytest`. Retry count must be printed accurately. This proves the generate-validate-retry loop is functional end to end. ### Model source currency - Anthropic model list: https://docs.anthropic.com/en/docs/about-claude/models/overview - OpenAI model list: https://platform.openai.com/docs/models - verifiedOn: 2026-08-07 - `LLM_MODEL` is required and has no implicit default; choose a supported model for the configured provider at setup time. ## Source code A reference implementation of LLM Test Generator — README, guide, source and tests — is at https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/llm-test-generator.