LLM Test Generator

Field	Value
Path	2 — AI-Augmented Engineering
Position	5 of 12
Difficulty	🟢 Local only
Estimated time	4 hours
AWS cost	None

---

Agent Pickup Instructions

# 1. Enter project
cd projects/p2-05-llm-test-generator

# 2. Create virtual environment
python -m venv .venv && source .venv/bin/activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Copy env file and add your API key
cp .env.example .env

# 5. Generate tests for a module
python src/main.py src/main.py

# 6. Generate with coverage report
python src/main.py src/main.py --coverage

# 7. Run tests
pytest tests/ -v

Done when:

• python src/main.py <source_file> produces a test_<filename>.py that runs without syntax errors
• Generated file contains only tests that actually pass (failing tests removed)
• --coverage flag runs pytest --cov and prints a branch count
• Retry loop kicks in on failure and shows Retry N/3: injecting error into prompt
• All 4 tests pass

---

What this project is

An LLM-powered test generator that uses AST parsing to extract Python function metadata, sends structured function information to an LLM with a pytest-aware prompt, validates the generated tests by actually running them with pytest, and retries up to 3 times on syntax errors or test failures — injecting the error message into the next prompt so the model can self-correct. The final output file contains only tests that actually pass, not the raw LLM output.

---

What the learner achieves

"I built a test-generation pipeline with a self-correcting retry loop: the LLM generates pytest tests, I run them, and if they fail I inject the error back into the next prompt. The final file contains only tests that pass — I don't ship broken tests."

---

Folder structure

p2-05-llm-test-generator/
├── src/
│   ├── main.py        # CLI: <source_file> [--coverage] [--max-retries N]
│   ├── llm.py         # Provider-agnostic LLM wrapper
│   ├── parser.py      # extract_function_metadata -> list[FunctionMeta]
│   ├── generator.py   # build_generation_prompt, generate_tests_for_function
│   ├── validator.py   # run_pytest_on_generated, parse_pytest_output, retry_with_error
│   └── reporter.py    # print_coverage_summary
├── tests/
│   ├── fixtures/
│   │   ├── simple_math.py     # Simple functions: add, subtract, divide (raises ZeroDivisionError)
│   │   └── string_utils.py    # String functions: truncate, slugify, count_words
│   ├── test_parser.py
│   ├── test_validator.py
│   └── test_generator.py
├── .env.example
├── requirements.txt
└── README.md

---

.env.example

# LLM provider: anthropic | openai | ollama
LLM_PROVIDER=anthropic

# Model to use
LLM_MODEL=

# API key for Anthropic
ANTHROPIC_API_KEY=

# API key for OpenAI (leave blank if using anthropic or ollama)
OPENAI_API_KEY=

# Ollama base URL
OLLAMA_BASE_URL=http://localhost:11434

# Max retry attempts on failing tests (1-5)
MAX_RETRIES=3

# Output directory for generated test files
OUTPUT_DIR=.

# Pytest coverage minimum threshold (0-100, used with --coverage)
COVERAGE_MIN=0

---

requirements.txt

anthropic==0.30.0
openai==1.35.0
python-dotenv==1.0.1
rich==13.7.1
pytest==8.2.2
pytest-cov==5.0.0
pytest-mock==3.14.0

---

src/ — what to implement

src/llm.py

get_completion(prompt: str, system: str = "") -> str

• Standard interface: reads LLM_PROVIDER and LLM_MODEL from env
• Routes to Anthropic, OpenAI, or Ollama
• Edge cases: raise RuntimeError if API key is missing

---

src/parser.py

FunctionMeta (dataclass)

@dataclass
class FunctionMeta:
    name: str
    params: list[dict]       # [{"name": str, "type": str | None, "default": str | None}]
    return_type: str | None
    docstring: str | None
    source_lines: list[str]  # Raw source lines of the function body
    lineno: int
    raises: list[str]        # Exception names found in raise statements

extract_function_metadata(filepath: str) -> list[FunctionMeta]

• Inputs: path to a .py file
• Output: list of FunctionMeta for all public functions
• Behavior:
  • Parse with ast.parse
  • For each FunctionDef / AsyncFunctionDef where not name.startswith("_")
  • Extract params, return type, docstring using ast
  • Extract source lines using linecache.getlines
  • Walk AST subtree to find Raise nodes; extract exception names
• Edge cases: syntax error → raise SyntaxError with filename; return [] for files with no public functions

---

src/generator.py

build_generation_prompt(func: FunctionMeta, prior_error: str = "") -> str

• Inputs: function metadata, optional error from previous attempt
• Output: formatted prompt string
• Behavior:
  • Build a user prompt with: function name, params with types, return type, docstring, exceptions raised
  • If prior_error is non-empty, prepend: "The previous attempt produced this error. Fix it:\n\n{prior_error}\n\n"
  • Include the raw function source lines so the model can see the actual implementation
  • Instruct: "Generate pytest test cases. Use only stdlib. Import the function from its module. Return only valid Python code, no markdown fences."

generate_tests_for_function(func: FunctionMeta, source_module: str, max_retries: int = 3) -> str | None

• Inputs: function metadata, Python module path (e.g., simple_math), max retry count
• Output: a Python string containing valid passing pytest test code, or None if all retries fail
• Behavior:
  • Build system prompt: "You are an expert at writing pytest tests. Return only valid Python code, no explanation, no markdown."
  • Call build_generation_prompt with no prior error
  • Call get_completion
  • Call validate_and_retry loop
• This function does NOT write files — it returns a string

validate_and_retry(func_name: str, test_code: str, source_file: str, max_retries: int) -> str | None

• Inputs: function name, generated test code string, path to the source module, max retries
• Output: validated test code string (passing), or None
• Behavior: call run_pytest_on_generated; if it returns failure, build new prompt with error injected; retry; log Retry N/{max_retries}: injecting error into prompt

---

src/validator.py

run_pytest_on_generated(test_code: str, source_file: str, func_name: str) -> dict

• Inputs: generated test code string, path to source Python file, function name
• Output: {"passed": bool, "error": str, "output": str}
• Behavior:
  • Write test code to a temp file: tmp_test_{func_name}_{uuid}.py
  • Run pytest <temp_file> --tb=short -q via subprocess.run
  • Capture stdout + stderr
  • Detect syntax error: if SyntaxError appears in output → passed=False, error=<syntax error line>
  • Detect test failure: non-zero exit code → passed=False, error=<last N lines of output>
  • Always delete temp file in a finally block
• Edge cases: if subprocess.run itself raises (pytest not installed), raise RuntimeError

parse_pytest_output(output: str) -> dict

• Inputs: raw pytest stdout/stderr string
• Output: {"total": int, "passed": int, "failed": int, "errors": int}
• Behavior: parse the summary line N passed, M failed using regex; handle no tests ran as {"total": 0, "passed": 0, "failed": 0, "errors": 1}

filter_passing_tests(all_test_code: str, source_file: str) -> str

• Inputs: combined test code for all functions, path to source file
• Output: test code with only the passing test functions
• Behavior: split on def test_ boundaries; run each test function individually; keep only passing ones; reassemble with imports at top

---

src/reporter.py

print_coverage_summary(source_file: str, test_file: str) -> None

• Inputs: path to source file, path to generated test file
• Output: prints coverage summary to stdout
• Behavior: run pytest {test_file} --cov={source_module} --cov-report=term-missing -q; capture and print output; parse for branch count from TOTAL line

---

src/main.py

CLI entry point using argparse.

Arguments:

• Positional: source_file — path to Python file to generate tests for
• --coverage — after generating tests, run pytest-cov and print summary
• --max-retries N — override MAX_RETRIES env var

Behavior:

1. Load .env
2. Call extract_function_metadata; print Found N public functions
3. For each function: call generate_tests_for_function; collect results
4. Filter: keep only functions where generation succeeded
5. Assemble final test file: shared imports at top + all passing test blocks
6. Write to test_<source_filename>.py in OUTPUT_DIR
7. Print Generated test_<name>.py | N functions tested | M retries used
8. If --coverage: call print_coverage_summary

---

tests/ — what to test

Test 1 — pytest validator detects SyntaxError (test_validator.py)

• Call run_pytest_on_generated with intentionally broken Python: def test_foo(:\n pass
• Assert passed == False
• Assert "SyntaxError" appears in the error field

Test 2 — Retry injects error message into next prompt (test_generator.py)

• Mock get_completion to track call arguments
• Set up validate_and_retry to fail on the first attempt (mock run_pytest_on_generated to return passed=False, error="AssertionError: ...")
• Assert get_completion is called twice
• Assert the second call's prompt contains the error string from the first failure

Test 3 — Only passing tests in final file (test_validator.py)

• Create a test code string with one passing test and one failing test
• Call filter_passing_tests
• Assert the returned string contains the passing test function name
• Assert the returned string does not contain the failing test function name

Test 4 — Coverage summary shows branch count (test_validator.py)

• Run print_coverage_summary on tests/fixtures/simple_math.py with a real test file
• Capture stdout
• Assert output contains a number followed by %
• (Skip if pytest-cov not installed — use pytest.importorskip("pytest_cov"))

Test 5 — Parser extracts exceptions from raise statements (test_parser.py)

• simple_math.py has def divide(a, b): if b == 0: raise ZeroDivisionError(...)
• Call extract_function_metadata
• Assert the divide function's raises list contains "ZeroDivisionError"

Test 6 — Generation prompt includes prior error when provided (test_generator.py)

• Call build_generation_prompt with prior_error="AssertionError: expected 4, got 5"
• Assert the returned prompt string contains the error text
• Assert the returned prompt contains the word "Fix"

---

README.md content

# LLM Test Generator

Generates pytest test cases for Python functions using LLM, then validates them by actually running pytest — retrying with error context on failure.

## Quick start

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env   # add your API key

# Generate tests
python src/main.py src/my_module.py

# Generate with coverage report
python src/main.py src/my_module.py --coverage

# Allow more retries
python src/main.py src/my_module.py --max-retries 5

How the retry loop works

Generate tests → Run pytest → Pass? → Write to file
                                 ↓ Fail
                          Inject error into next prompt → Retry (max 3)
                                 ↓ All retries fail
                          Skip function, log warning

Output

• test_<source_name>.py — contains only tests that actually pass
• Coverage report (with --coverage) — branch count for each function

Running tests

pytest tests/ -v

---

## GUIDE.md content

```markdown
# Build Guide — LLM Test Generator

## Step 1 — Extract function metadata with AST

Beyond p2-04, also extract `raises` (exceptions in raise statements):

```python
def get_raises(func_node):
    raises = []
    for node in ast.walk(func_node):
        if isinstance(node, ast.Raise) and node.exc:
            if isinstance(node.exc, ast.Call):
                raises.append(ast.unparse(node.exc.func))
            elif isinstance(node.exc, ast.Name):
                raises.append(node.exc.id)
    return raises

This tells the LLM "this function raises ZeroDivisionError" so it generates a test for the error case.

Step 2 — The generation system prompt

You are an expert Python test engineer.
Generate pytest test functions for the given Python function.
Rules:
- Use only stdlib and pytest
- Import the function from its module at the top
- Test the happy path, edge cases, and any documented exceptions
- Return ONLY valid Python code — no markdown, no explanation
- Each test function must start with def test_

Step 3 — The retry loop

def validate_and_retry(func_name, test_code, source_file, max_retries):
    prior_error = ""
    for attempt in range(max_retries):
        if attempt > 0:
            print(f"Retry {attempt}/{max_retries}: injecting error into prompt")
            new_prompt = build_generation_prompt(func, prior_error=prior_error)
            test_code = get_completion(new_prompt, system=SYSTEM_PROMPT)
        result = run_pytest_on_generated(test_code, source_file, func_name)
        if result["passed"]:
            return test_code
        prior_error = result["error"]
    return None

Step 4 — Running pytest programmatically

import subprocess, tempfile, os

def run_pytest_on_generated(test_code, source_file, func_name):
    tmp_path = f"/tmp/tmp_test_{func_name}_{uuid4().hex}.py"
    try:
        with open(tmp_path, "w") as f:
            f.write(test_code)
        result = subprocess.run(
            ["pytest", tmp_path, "--tb=short", "-q"],
            capture_output=True, text=True
        )
        passed = result.returncode == 0
        error = result.stdout + result.stderr
        return {"passed": passed, "error": error if not passed else ""}
    finally:
        os.unlink(tmp_path)

Step 5 — Filter passing tests

After all functions are generated, run each test function individually to build the final file. This is safer than trusting the whole file to pass after generation.

Debugging tips

• If the LLM keeps returning markdown fences, add "DO NOT include markdown code fences" to the system prompt
• If pytest can't find the source module, the import path is wrong — check sys.path manipulation in the generated file
• Start with a simple fixture like add(a, b): return a + b before testing complex functions

How to talk about this in an interview

"Why validate by actually running pytest?"

Because the LLM doesn't run code — it guesses. The only ground truth is whether the test actually passes. Running pytest is the correctness check the LLM can't provide.

"What does the retry loop buy you?"

Self-correction. The first attempt fails about 30% of the time on complex functions. With error context, the second attempt succeeds most of the time. Three retries handles ~95% of cases in my testing.

"How do you ensure the final test file only has passing tests?"

I filter: run each test function individually, keep only the passing ones, then assemble. This means the output file is a guarantee, not just output — it always runs clean.

---

## Failure modes to handle

| Failure | Where | How to handle |
|---|---|---|
| LLM returns markdown fences wrapping code | `generator.py` | Strip ` ```python ` and ` ``` ` before writing to temp file |
| Temp pytest file has wrong import path | `validator.py` | Inject `sys.path.insert(0, os.path.dirname(source_file))` at top of generated test |
| All retries fail for a function | `generator.py` | Log warning `Could not generate passing tests for {func_name}`; skip |
| Source file has no public functions | `main.py` | Print `No public functions found in <file>.` and exit 0 |
| pytest is not installed | `validator.py` | Raise `RuntimeError("pytest not found — run pip install pytest")` |
| Generated test imports non-existent module | `validator.py` | Caught by pytest as `ImportError`; appears in `error` field; retry handles it |

---

## The metric this project measures

**What is measured:** Number of functions tested, retry count, and pass rate (tests that passed / tests attempted).

**Format (stdout):**

Found 3 public functions Generated test_simple_math.py | 3 functions tested | 2 retries used | pass_rate=100%

**Target:** For `tests/fixtures/simple_math.py` (3 simple functions), the final test file must contain at least 3 test functions that all pass with `pytest`. Retry count must be printed accurately. This proves the generate-validate-retry loop is functional end to end.


## Model source currency

- Anthropic model list: https://docs.anthropic.com/en/docs/about-claude/models/overview
- OpenAI model list: https://platform.openai.com/docs/models
- verifiedOn: 2026-08-07
- `LLM_MODEL` is required and has no implicit default; choose a supported model for the configured provider at setup time.