AI CLI Toolkit

Field	Value
Path	2 — AI-Augmented Engineering
Position	11 of 12
Difficulty	🟡 Optional Docker
Estimated time	3 hours
AWS cost	None

---

Agent Pickup Instructions

# 1. Enter project
cd projects/p2-11-ai-cli-toolkit

# 2. Create virtual environment
python -m venv .venv && source .venv/bin/activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Copy env file and add your API key
cp .env.example .env

# 5. Install the CLI tool
pip install -e .

# 6. Use the toolkit
ai review --diff my.diff
ai explain --diff my.diff
ai search --query "authentication middleware"
ai query --question "show all customers"

# 7. Run tests
pytest tests/ -v

Done when:

• ai review calls p2-01 pattern and shows structured findings
• ai explain calls p2-02 pattern and shows three summary sections
• ai search calls p2-03 pattern (requires a pre-built index)
• ai query calls p2-10 pattern (requires a SQLite db)
• Disabling a tool in .aiworkflow.yml makes it unavailable and excluded from help
• All 4 tests pass

---

What this project is

A unified ai command-line interface that dispatches to the AI engineering tools built in prior Path 2 projects. It uses a plugin architecture where each tool is a single Python file in tools/, a YAML config file (.aiworkflow.yml) controls which tools are active, and the dispatcher dynamically discovers and loads tool files at startup. Adding a new tool requires one file in tools/ and one entry in the config — no changes to the dispatcher needed.

---

What the learner achieves

"I built a plugin-based CLI toolkit that unified four AI engineering tools into one ai command — with a config file that controls which tools are active and a plugin loader that discovers tools dynamically, so adding a new tool is a one-file operation."

---

Folder structure

p2-11-ai-cli-toolkit/
├── src/
│   ├── main.py          # `ai <subcommand> [args]` dispatcher
│   ├── llm.py           # Provider-agnostic LLM wrapper (shared by all tools)
│   ├── config.py        # load_config: reads .aiworkflow.yml
│   └── plugin_loader.py # discover_tools, load_tool_plugin
├── tools/
│   ├── review.py        # Thin wrapper around p2-01 code review logic
│   ├── explain.py       # Thin wrapper around p2-02 diff summarizer logic
│   ├── search.py        # Thin wrapper around p2-03 semantic search logic
│   └── query.py         # Thin wrapper around p2-10 SQL agent logic
├── tests/
│   ├── test_plugin_loader.py
│   ├── test_config.py
│   └── test_dispatcher.py
├── .aiworkflow.yml      # Tool configuration
├── .env.example
├── pyproject.toml       # Entry point: ai = src.main:main
├── requirements.txt
└── README.md

---

.aiworkflow.yml (default content)

tools:
  review:
    enabled: true
    description: "Code review — finds security, performance, correctness, and style issues in a git diff"
    min_severity: LOW

  explain:
    enabled: true
    description: "PR summarizer — plain-English summary, architecture impact, and test coverage flag"

  search:
    enabled: true
    description: "Semantic codebase search — find functions by meaning, not just keyword"
    index_dir: .index

  query:
    enabled: true
    description: "Natural language to SQL — ask questions about your database in plain English"
    db_path: ecommerce.db

settings:
  default_output: table
  max_retries: 3

---

.env.example

# LLM provider: anthropic | openai | ollama
LLM_PROVIDER=anthropic

# Model to use
LLM_MODEL=

# API key for Anthropic
ANTHROPIC_API_KEY=

# API key for OpenAI
OPENAI_API_KEY=

# Ollama base URL
OLLAMA_BASE_URL=http://localhost:11434

# Path to the workflow config file
CONFIG_PATH=.aiworkflow.yml

---

requirements.txt

anthropic==0.30.0
openai==1.35.0
pyyaml==6.0.1
python-dotenv==1.0.1
rich==13.7.1
pytest==8.2.2
pytest-mock==3.14.0
sentence-transformers==3.0.1
faiss-cpu==1.8.0
sqlglot==23.3.0
tabulate==0.9.0

---

pyproject.toml

[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.backends.legacy:build"

[project]
name = "ai-cli-toolkit"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = []

[project.scripts]
ai = "src.main:main"

[tool.setuptools.packages.find]
where = ["."]
include = ["src*", "tools*"]

---

src/ — what to implement

src/llm.py

get_completion(prompt: str, system: str = "") -> str

• Standard interface: reads LLM_PROVIDER and LLM_MODEL from env
• Shared by all tools in the toolkit

---

src/config.py

load_config(config_path: str = None) -> dict

• Inputs: optional path to .aiworkflow.yml (defaults to CONFIG_PATH env var, then .aiworkflow.yml)
• Output: parsed YAML as dict
• Behavior: load with yaml.safe_load; validate that tools key exists; return full dict
• Edge cases: if file not found, return a default config with all 4 tools enabled; if YAML is invalid, raise ValueError with file path

get_active_tools(config: dict) -> list[str]

• Inputs: full config dict
• Output: list of tool names where enabled == True
• Behavior: filter config["tools"] by enabled field

get_tool_config(config: dict, tool_name: str) -> dict

• Inputs: full config, tool name
• Output: the tool's config sub-dict (e.g., {"enabled": True, "min_severity": "LOW"})
• Edge cases: if tool not in config, return {"enabled": False}

---

src/plugin_loader.py

discover_tools(tools_dir: str = "tools") -> list[str]

• Inputs: directory path to scan
• Output: list of tool names (filenames without .py extension)
• Behavior: os.listdir(tools_dir); filter for .py files; exclude __init__.py and files starting with _; return sorted list of names

load_tool_plugin(tool_name: str, tools_dir: str = "tools") -> module

• Inputs: tool name (e.g., "review"), tools directory
• Output: loaded Python module
• Behavior: use importlib.util.spec_from_file_location and importlib.util.module_from_spec to load the file dynamically
• Edge cases: if file not found, raise ImportError(f"Tool '{tool_name}' not found in {tools_dir}/")

get_tool_interface(module) -> dict

• Inputs: loaded tool module
• Output: {"name": str, "description": str, "run": callable, "add_arguments": callable}
• Behavior: every tool module must have:
  • TOOL_NAME: str — the subcommand name
  • TOOL_DESCRIPTION: str — one-line description for help
  • def run(args: argparse.Namespace, config: dict) -> None — entry point
  • def add_arguments(parser: argparse.ArgumentParser) -> None — registers tool-specific args
• If module is missing any of these, raise AttributeError with descriptive message

---

src/main.py

main() -> None

• Entry point for the ai console script
• Behavior:
  1. Load .env and config
  2. Call discover_tools("tools")
  3. Filter by get_active_tools(config)
  4. For each active tool: load_tool_plugin → get_tool_interface
  5. Create a top-level argparse.ArgumentParser with subparsers
  6. For each tool: create a subparser named tool["name"]; call tool["add_arguments"](subparser)
  7. Parse args: if no subcommand, print help and exit 0
  8. Dispatch: call the matching tool's run(args, tool_config)
• Help format when called with no subcommand:

  usage: ai <command> [args]
  
  Available tools:
    review   Code review — finds issues in a git diff
    explain  PR summarizer — plain-English diff summary
    search   Semantic codebase search
    query    Natural language to SQL
  
  Run `ai <command> --help` for tool-specific options.
  

---

tools/review.py

Tool wrapper around the p2-01 code review logic. Implements the plugin interface.

TOOL_NAME = "review"
TOOL_DESCRIPTION = "Code review — finds security, performance, correctness, and style issues in a git diff"

def add_arguments(parser):
    parser.add_argument("--diff", help="Path to diff file (reads stdin if omitted)")
    parser.add_argument("--min-severity", choices=["HIGH", "MEDIUM", "LOW"], default="LOW")
    parser.add_argument("--output", choices=["table", "json"], default="table")

def run(args, config):
    # Read diff from file or stdin
    # Call chunk_diff_by_file, review_chunk, merge_results, print_table/save_json
    # (inline the core logic — don't import from another project)
    ...

Key constraint: the tool files are self-contained wrappers. They implement the same logic as the referenced projects but inline it (do not sys.path hack to import from other project directories). The tools are thin but complete.

---

tools/explain.py

Tool wrapper around the p2-02 PR summarizer logic.

TOOL_NAME = "explain"
TOOL_DESCRIPTION = "PR summarizer — plain-English summary, architecture impact, and test coverage flag"

def add_arguments(parser):
    parser.add_argument("--diff", help="Path to diff file")
    parser.add_argument("--title", help="PR title (optional)")
    parser.add_argument("--comments", help="Path to reviewer comments file (optional)")

def run(args, config):
    # Inline the p2-02 core logic
    ...

---

tools/search.py

Tool wrapper around the p2-03 semantic search logic.

TOOL_NAME = "search"
TOOL_DESCRIPTION = "Semantic codebase search — find functions by meaning, not keyword"

def add_arguments(parser):
    parser.add_argument("--query", required=True, help="Search query")
    parser.add_argument("--index-dir", default=None, help="Override index directory from config")
    parser.add_argument("--top-k", type=int, default=5)

def run(args, config):
    # Load index from config["index_dir"] or args.index_dir
    # Run semantic search; print results
    ...

---

tools/query.py

Tool wrapper around the p2-10 SQL agent logic.

TOOL_NAME = "query"
TOOL_DESCRIPTION = "Natural language to SQL — ask questions about a SQLite database"

def add_arguments(parser):
    parser.add_argument("--question", required=True, help="Natural language question")
    parser.add_argument("--db", default=None, help="SQLite database path (override config)")

def run(args, config):
    # Load db_path from config or args
    # Introspect schema; call nl_to_sql; execute_safe; print results
    ...

---

tests/ — what to test

Test 1 — Plugin loader discovers all .py files in tools/ (test_plugin_loader.py)

• Call discover_tools("tools")
• Assert result contains ["explain", "query", "review", "search"] (sorted)
• Assert __init__ and _private files are not in results if present

Test 2 — Unknown subcommand prints help with available tools (test_dispatcher.py)

• Call main() with sys.argv = ["ai", "unknown-tool"]
• Capture stdout
• Assert "Available tools" or "usage" appears in output
• Assert exit code is non-zero (use pytest.raises(SystemExit))

Test 3 — Config file disabling a tool makes it unavailable (test_config.py)

• Write a temp .aiworkflow.yml with review: enabled: false
• Call get_active_tools on loaded config
• Assert "review" is not in the active tools list
• Assert "explain", "search", "query" are still present

Test 4 — Help output lists all active tools (test_dispatcher.py)

• Call main() with sys.argv = ["ai"] (no subcommand)
• Capture stdout
• Assert all 4 tool names appear in the output
• Assert tool descriptions appear alongside tool names

Test 5 — Plugin interface validation raises on missing attribute (test_plugin_loader.py)

• Create a temp Python file in a temp tools dir with only TOOL_NAME defined (missing run)
• Call load_tool_plugin then get_tool_interface
• Assert AttributeError is raised

Test 6 — Tool config returned correctly for active tool (test_config.py)

• Load default .aiworkflow.yml
• Call get_tool_config(config, "review")
• Assert result has "enabled" key equal to True
• Assert result has "min_severity" key

---

README.md content

# AI CLI Toolkit

A unified `ai` command that dispatches to AI engineering tools. Plugin architecture — add a new tool by adding one file to `tools/`.

## Quick start

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
pip install -e .
cp .env.example .env   # add your API key

ai review --diff my.diff
ai explain --diff my.diff --title "My PR"
ai search --query "error handling"
ai query --question "show all customers from US"

Available tools

Command	Description
ai review	Code review — security, performance, correctness, style
ai explain	PR summarizer — plain English + architecture impact
ai search	Semantic codebase search
ai query	Natural language to SQL

Configuration

Edit .aiworkflow.yml to enable/disable tools and set tool-specific defaults:

tools:
  review:
    enabled: true
    min_severity: HIGH  # only show HIGH findings
  search:
    enabled: false      # disable search if no index built

Adding a new tool

1. Create tools/my_tool.py with TOOL_NAME, TOOL_DESCRIPTION, run(args, config), add_arguments(parser)
2. Add an entry to .aiworkflow.yml under tools:
3. Run ai my_tool --help — it's available immediately

Running tests

pytest tests/ -v

---

## GUIDE.md content

```markdown
# Build Guide — AI CLI Toolkit

## Step 1 — The plugin interface contract

Every file in `tools/` must implement four things:

```python
TOOL_NAME = "review"           # The subcommand name
TOOL_DESCRIPTION = "..."       # One line for help output

def add_arguments(parser):
    # Add argparse arguments specific to this tool
    parser.add_argument("--diff", help="Path to diff file")

def run(args, config):
    # Entry point — args is the parsed Namespace, config is the tool's YAML config
    ...

That's the full contract. The dispatcher discovers files, loads them, calls add_arguments to register args, and calls run on dispatch.

Step 2 — Dynamic plugin loading

import importlib.util, os

def load_tool_plugin(tool_name, tools_dir="tools"):
    path = os.path.join(tools_dir, f"{tool_name}.py")
    if not os.path.exists(path):
        raise ImportError(f"Tool '{tool_name}' not found in {tools_dir}/")
    
    spec = importlib.util.spec_from_file_location(tool_name, path)
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    return module

Step 3 — The dispatcher

def main():
    config = load_config()
    active = get_active_tools(config)
    
    parser = argparse.ArgumentParser(prog="ai", add_help=False)
    subparsers = parser.add_subparsers(dest="command")
    
    tools = {}
    for name in active:
        module = load_tool_plugin(name)
        iface = get_tool_interface(module)
        sub = subparsers.add_parser(name, help=iface["description"])
        iface["add_arguments"](sub)
        tools[name] = iface
    
    args = parser.parse_args()
    
    if not args.command:
        print_help(tools)
        sys.exit(0)
    
    tool_config = get_tool_config(config, args.command)
    tools[args.command]["run"](args, tool_config)

Step 4 — Tool wrappers (thin but complete)

Each tool file inlines the core logic from the referenced project. This makes the toolkit self-contained:

• tools/review.py — copy the chunking + LLM call + output logic from p2-01
• tools/explain.py — copy the stats parsing + two LLM calls from p2-02
• etc.

Don't import across projects. The tools directory is the canonical place for the simplified, toolkit-integrated version of each capability.

Step 5 — Config-driven behavior

When run(args, config) receives config, it can use it for defaults:

def run(args, config):
    min_severity = args.min_severity or config.get("min_severity", "LOW")
    # ...

This lets users set project-wide defaults in .aiworkflow.yml that can be overridden per invocation.

Debugging tips

• If ai is not found after pip install -e ., check pyproject.toml entry points
• If a tool loads but run is not called, check subcommand name matches TOOL_NAME exactly
• Test discover_tools first — if it returns an empty list, check the tools directory path

How to talk about this in an interview

"Why a plugin architecture instead of one big CLI?"

Each tool has independent dependencies and configuration. A plugin system means I can add, disable, or replace tools without touching the dispatcher. The config file (aiworkflow.yml) is the API — not the code.

"How does a new engineer add a tool?"

One file in tools/, one entry in the YAML. The discovery mechanism picks it up automatically. No changes to the dispatcher, no registration step. The interface contract is four names — if your file has them, it works.

"What's the tradeoff of inlining logic vs importing from other projects?"

Inlining avoids cross-project import path hacking and makes each tool standalone. The tradeoff is duplication. In production, you'd package the core logic as a library and import it from both the individual project and the toolkit.

---

## Failure modes to handle

| Failure | Where | How to handle |
|---|---|---|
| `.aiworkflow.yml` not found | `config.py` | Return default config with all 4 tools enabled; log "No config file found, using defaults" |
| Tool file has syntax error | `plugin_loader.py` | Catch `SyntaxError`; log warning; skip this tool; continue loading others |
| Tool missing required interface attribute | `plugin_loader.py` | Raise `AttributeError` with message naming the missing attribute |
| Unknown subcommand | `main.py` | Print help text with available tools; exit with code 1 |
| Search tool called without index | `tools/search.py` | Print "No index found. Run: ai index --dir <path>" and exit 1 |
| Query tool called without db file | `tools/query.py` | Print "Database not found: {db_path}. Check .aiworkflow.yml db_path." and exit 1 |

---

## The metric this project measures

**What is measured:** Tool discovery count, config compliance (disabled tools unavailable), and dispatch success rate.

**Format (stdout on `ai` with no args):**

Available tools (4 active): review Code review — finds issues in a git diff explain PR summarizer — plain-English diff summary search Semantic codebase search query Natural language to SQL

**Target:** All 4 tool files are discovered automatically. Disabling one in `.aiworkflow.yml` reduces the count to 3 and removes it from help output. Each `ai <tool>` call successfully dispatches to the correct `run()` function. These three confirm plugin discovery, config gating, and dispatch are all working.


## Model source currency

- Anthropic model list: https://docs.anthropic.com/en/docs/about-claude/models/overview
- OpenAI model list: https://platform.openai.com/docs/models
- verifiedOn: 2026-08-07
- `LLM_MODEL` is required and has no implicit default; choose a supported model for the configured provider at setup time.