---
title: "AI CLI Toolkit"
description: "Plugin architecture + config-gated tools is a system-design skill, not just an AI skill — shows you can productize a set of scripts into something a whole team..."
source: "https://confidentprep.com/paths/ai-augmented-engineering/ai-cli-toolkit/"
path: "AI-Augmented Engineering"
repository: "https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/ai-cli-toolkit"
token_estimate: 4933
---

# AI CLI Toolkit

## Overview

A plugin-based `ai` command that unifies code review, PR summarizing, semantic search,
and NL-to-SQL into one dynamically-loaded toolkit, gated by a YAML config. Four of this
path's earlier projects, reused as plugins rather than rebuilt — the point isn't new AI
capability, it's the engineering discipline of turning four standalone scripts into one
coherent, configurable tool.

## What to do

| Field | Value |
|---|---|
| Path | 2 — AI-Augmented Engineering |
| Position | 11 of 12 |
| Difficulty | 🟡 Optional Docker |
| Estimated time | 3 hours |
| AWS cost | None |

---

### Agent Pickup Instructions

```bash
# 1. Enter project
cd projects/p2-11-ai-cli-toolkit

# 2. Create virtual environment
python -m venv .venv && source .venv/bin/activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Copy env file and add your API key
cp .env.example .env

# 5. Install the CLI tool
pip install -e .

# 6. Use the toolkit
ai review --diff my.diff
ai explain --diff my.diff
ai search --query "authentication middleware"
ai query --question "show all customers"

# 7. Run tests
pytest tests/ -v
```

**Done when:**
- [ ] `ai review` calls p2-01 pattern and shows structured findings
- [ ] `ai explain` calls p2-02 pattern and shows three summary sections
- [ ] `ai search` calls p2-03 pattern (requires a pre-built index)
- [ ] `ai query` calls p2-10 pattern (requires a SQLite db)
- [ ] Disabling a tool in `.aiworkflow.yml` makes it unavailable and excluded from help
- [ ] All 4 tests pass

---

### What this project is

A unified `ai` command-line interface that dispatches to the AI engineering tools built in prior Path 2 projects. It uses a plugin architecture where each tool is a single Python file in `tools/`, a YAML config file (`.aiworkflow.yml`) controls which tools are active, and the dispatcher dynamically discovers and loads tool files at startup. Adding a new tool requires one file in `tools/` and one entry in the config — no changes to the dispatcher needed.

---

### What the learner achieves

"I built a plugin-based CLI toolkit that unified four AI engineering tools into one `ai` command — with a config file that controls which tools are active and a plugin loader that discovers tools dynamically, so adding a new tool is a one-file operation."

---

### Folder structure

```
p2-11-ai-cli-toolkit/
├── src/
│   ├── main.py          # `ai <subcommand> [args]` dispatcher
│   ├── llm.py           # Provider-agnostic LLM wrapper (shared by all tools)
│   ├── config.py        # load_config: reads .aiworkflow.yml
│   └── plugin_loader.py # discover_tools, load_tool_plugin
├── tools/
│   ├── review.py        # Thin wrapper around p2-01 code review logic
│   ├── explain.py       # Thin wrapper around p2-02 diff summarizer logic
│   ├── search.py        # Thin wrapper around p2-03 semantic search logic
│   └── query.py         # Thin wrapper around p2-10 SQL agent logic
├── tests/
│   ├── test_plugin_loader.py
│   ├── test_config.py
│   └── test_dispatcher.py
├── .aiworkflow.yml      # Tool configuration
├── .env.example
├── pyproject.toml       # Entry point: ai = src.main:main
├── requirements.txt
└── README.md
```

---

### .aiworkflow.yml (default content)

```yaml
tools:
  review:
    enabled: true
    description: "Code review — finds security, performance, correctness, and style issues in a git diff"
    min_severity: LOW

  explain:
    enabled: true
    description: "PR summarizer — plain-English summary, architecture impact, and test coverage flag"

  search:
    enabled: true
    description: "Semantic codebase search — find functions by meaning, not just keyword"
    index_dir: .index

  query:
    enabled: true
    description: "Natural language to SQL — ask questions about your database in plain English"
    db_path: ecommerce.db

settings:
  default_output: table
  max_retries: 3
```

---

### .env.example

```bash
# LLM provider: anthropic | openai | ollama
LLM_PROVIDER=anthropic

# Model to use
LLM_MODEL=

# API key for Anthropic
ANTHROPIC_API_KEY=

# API key for OpenAI
OPENAI_API_KEY=

# Ollama base URL
OLLAMA_BASE_URL=http://localhost:11434

# Path to the workflow config file
CONFIG_PATH=.aiworkflow.yml
```

---

### requirements.txt

```
anthropic==0.30.0
openai==1.35.0
pyyaml==6.0.1
python-dotenv==1.0.1
rich==13.7.1
pytest==8.2.2
pytest-mock==3.14.0
sentence-transformers==3.0.1
faiss-cpu==1.8.0
sqlglot==23.3.0
tabulate==0.9.0
```

---

### pyproject.toml

```toml
[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.backends.legacy:build"

[project]
name = "ai-cli-toolkit"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = []

[project.scripts]
ai = "src.main:main"

[tool.setuptools.packages.find]
where = ["."]
include = ["src*", "tools*"]
```

---

### src/ — what to implement

#### `src/llm.py`

**`get_completion(prompt: str, system: str = "") -> str`**
- Standard interface: reads `LLM_PROVIDER` and `LLM_MODEL` from env
- Shared by all tools in the toolkit

---

#### `src/config.py`

**`load_config(config_path: str = None) -> dict`**
- Inputs: optional path to `.aiworkflow.yml` (defaults to `CONFIG_PATH` env var, then `.aiworkflow.yml`)
- Output: parsed YAML as dict
- Behavior: load with `yaml.safe_load`; validate that `tools` key exists; return full dict
- Edge cases: if file not found, return a default config with all 4 tools enabled; if YAML is invalid, raise `ValueError` with file path

**`get_active_tools(config: dict) -> list[str]`**
- Inputs: full config dict
- Output: list of tool names where `enabled == True`
- Behavior: filter `config["tools"]` by `enabled` field

**`get_tool_config(config: dict, tool_name: str) -> dict`**
- Inputs: full config, tool name
- Output: the tool's config sub-dict (e.g., `{"enabled": True, "min_severity": "LOW"}`)
- Edge cases: if tool not in config, return `{"enabled": False}`

---

#### `src/plugin_loader.py`

**`discover_tools(tools_dir: str = "tools") -> list[str]`**
- Inputs: directory path to scan
- Output: list of tool names (filenames without `.py` extension)
- Behavior: `os.listdir(tools_dir)`; filter for `.py` files; exclude `__init__.py` and files starting with `_`; return sorted list of names

**`load_tool_plugin(tool_name: str, tools_dir: str = "tools") -> module`**
- Inputs: tool name (e.g., `"review"`), tools directory
- Output: loaded Python module
- Behavior: use `importlib.util.spec_from_file_location` and `importlib.util.module_from_spec` to load the file dynamically
- Edge cases: if file not found, raise `ImportError(f"Tool '{tool_name}' not found in {tools_dir}/")`

**`get_tool_interface(module) -> dict`**
- Inputs: loaded tool module
- Output: `{"name": str, "description": str, "run": callable, "add_arguments": callable}`
- Behavior: every tool module must have:
  - `TOOL_NAME: str` — the subcommand name
  - `TOOL_DESCRIPTION: str` — one-line description for help
  - `def run(args: argparse.Namespace, config: dict) -> None` — entry point
  - `def add_arguments(parser: argparse.ArgumentParser) -> None` — registers tool-specific args
- If module is missing any of these, raise `AttributeError` with descriptive message

---

#### `src/main.py`

**`main() -> None`**
- Entry point for the `ai` console script
- Behavior:
  1. Load `.env` and config
  2. Call `discover_tools("tools")`
  3. Filter by `get_active_tools(config)`
  4. For each active tool: `load_tool_plugin` → `get_tool_interface`
  5. Create a top-level `argparse.ArgumentParser` with subparsers
  6. For each tool: create a subparser named `tool["name"]`; call `tool["add_arguments"](subparser)`
  7. Parse args: if no subcommand, print help and exit 0
  8. Dispatch: call the matching tool's `run(args, tool_config)`
- Help format when called with no subcommand:
  ```
  usage: ai <command> [args]
  
  Available tools:
    review   Code review — finds issues in a git diff
    explain  PR summarizer — plain-English diff summary
    search   Semantic codebase search
    query    Natural language to SQL
  
  Run `ai <command> --help` for tool-specific options.
  ```

---

#### `tools/review.py`

Tool wrapper around the p2-01 code review logic. Implements the plugin interface.

```python
TOOL_NAME = "review"
TOOL_DESCRIPTION = "Code review — finds security, performance, correctness, and style issues in a git diff"

def add_arguments(parser):
    parser.add_argument("--diff", help="Path to diff file (reads stdin if omitted)")
    parser.add_argument("--min-severity", choices=["HIGH", "MEDIUM", "LOW"], default="LOW")
    parser.add_argument("--output", choices=["table", "json"], default="table")

def run(args, config):
    # Read diff from file or stdin
    # Call chunk_diff_by_file, review_chunk, merge_results, print_table/save_json
    # (inline the core logic — don't import from another project)
    ...
```

Key constraint: the tool files are self-contained wrappers. They implement the same logic as the referenced projects but inline it (do not `sys.path` hack to import from other project directories). The tools are thin but complete.

---

#### `tools/explain.py`

Tool wrapper around the p2-02 PR summarizer logic.

```python
TOOL_NAME = "explain"
TOOL_DESCRIPTION = "PR summarizer — plain-English summary, architecture impact, and test coverage flag"

def add_arguments(parser):
    parser.add_argument("--diff", help="Path to diff file")
    parser.add_argument("--title", help="PR title (optional)")
    parser.add_argument("--comments", help="Path to reviewer comments file (optional)")

def run(args, config):
    # Inline the p2-02 core logic
    ...
```

---

#### `tools/search.py`

Tool wrapper around the p2-03 semantic search logic.

```python
TOOL_NAME = "search"
TOOL_DESCRIPTION = "Semantic codebase search — find functions by meaning, not keyword"

def add_arguments(parser):
    parser.add_argument("--query", required=True, help="Search query")
    parser.add_argument("--index-dir", default=None, help="Override index directory from config")
    parser.add_argument("--top-k", type=int, default=5)

def run(args, config):
    # Load index from config["index_dir"] or args.index_dir
    # Run semantic search; print results
    ...
```

---

#### `tools/query.py`

Tool wrapper around the p2-10 SQL agent logic.

```python
TOOL_NAME = "query"
TOOL_DESCRIPTION = "Natural language to SQL — ask questions about a SQLite database"

def add_arguments(parser):
    parser.add_argument("--question", required=True, help="Natural language question")
    parser.add_argument("--db", default=None, help="SQLite database path (override config)")

def run(args, config):
    # Load db_path from config or args
    # Introspect schema; call nl_to_sql; execute_safe; print results
    ...
```

---

### tests/ — what to test

#### Test 1 — Plugin loader discovers all `.py` files in tools/ (`test_plugin_loader.py`)
- Call `discover_tools("tools")`
- Assert result contains `["explain", "query", "review", "search"]` (sorted)
- Assert `__init__` and `_private` files are not in results if present

#### Test 2 — Unknown subcommand prints help with available tools (`test_dispatcher.py`)
- Call `main()` with `sys.argv = ["ai", "unknown-tool"]`
- Capture stdout
- Assert "Available tools" or "usage" appears in output
- Assert exit code is non-zero (use `pytest.raises(SystemExit)`)

#### Test 3 — Config file disabling a tool makes it unavailable (`test_config.py`)
- Write a temp `.aiworkflow.yml` with `review: enabled: false`
- Call `get_active_tools` on loaded config
- Assert `"review"` is not in the active tools list
- Assert `"explain"`, `"search"`, `"query"` are still present

#### Test 4 — Help output lists all active tools (`test_dispatcher.py`)
- Call `main()` with `sys.argv = ["ai"]` (no subcommand)
- Capture stdout
- Assert all 4 tool names appear in the output
- Assert tool descriptions appear alongside tool names

#### Test 5 — Plugin interface validation raises on missing attribute (`test_plugin_loader.py`)
- Create a temp Python file in a temp tools dir with only `TOOL_NAME` defined (missing `run`)
- Call `load_tool_plugin` then `get_tool_interface`
- Assert `AttributeError` is raised

#### Test 6 — Tool config returned correctly for active tool (`test_config.py`)
- Load default `.aiworkflow.yml`
- Call `get_tool_config(config, "review")`
- Assert result has `"enabled"` key equal to `True`
- Assert result has `"min_severity"` key

---

### README.md content

```markdown
# AI CLI Toolkit

A unified `ai` command that dispatches to AI engineering tools. Plugin architecture — add a new tool by adding one file to `tools/`.

## Quick start

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
pip install -e .
cp .env.example .env   # add your API key

ai review --diff my.diff
ai explain --diff my.diff --title "My PR"
ai search --query "error handling"
ai query --question "show all customers from US"
```

## Available tools

| Command | Description |
|---|---|
| `ai review` | Code review — security, performance, correctness, style |
| `ai explain` | PR summarizer — plain English + architecture impact |
| `ai search` | Semantic codebase search |
| `ai query` | Natural language to SQL |

## Configuration

Edit `.aiworkflow.yml` to enable/disable tools and set tool-specific defaults:

```yaml
tools:
  review:
    enabled: true
    min_severity: HIGH  # only show HIGH findings
  search:
    enabled: false      # disable search if no index built
```

## Adding a new tool

1. Create `tools/my_tool.py` with `TOOL_NAME`, `TOOL_DESCRIPTION`, `run(args, config)`, `add_arguments(parser)`
2. Add an entry to `.aiworkflow.yml` under `tools:`
3. Run `ai my_tool --help` — it's available immediately

## Running tests

```bash
pytest tests/ -v
```
```

---

### GUIDE.md content

```markdown
# Build Guide — AI CLI Toolkit

## Step 1 — The plugin interface contract

Every file in `tools/` must implement four things:

```python
TOOL_NAME = "review"           # The subcommand name
TOOL_DESCRIPTION = "..."       # One line for help output

def add_arguments(parser):
    # Add argparse arguments specific to this tool
    parser.add_argument("--diff", help="Path to diff file")

def run(args, config):
    # Entry point — args is the parsed Namespace, config is the tool's YAML config
    ...
```

That's the full contract. The dispatcher discovers files, loads them, calls `add_arguments` to register args, and calls `run` on dispatch.

## Step 2 — Dynamic plugin loading

```python
import importlib.util, os

def load_tool_plugin(tool_name, tools_dir="tools"):
    path = os.path.join(tools_dir, f"{tool_name}.py")
    if not os.path.exists(path):
        raise ImportError(f"Tool '{tool_name}' not found in {tools_dir}/")
    
    spec = importlib.util.spec_from_file_location(tool_name, path)
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    return module
```

## Step 3 — The dispatcher

```python
def main():
    config = load_config()
    active = get_active_tools(config)
    
    parser = argparse.ArgumentParser(prog="ai", add_help=False)
    subparsers = parser.add_subparsers(dest="command")
    
    tools = {}
    for name in active:
        module = load_tool_plugin(name)
        iface = get_tool_interface(module)
        sub = subparsers.add_parser(name, help=iface["description"])
        iface["add_arguments"](sub)
        tools[name] = iface
    
    args = parser.parse_args()
    
    if not args.command:
        print_help(tools)
        sys.exit(0)
    
    tool_config = get_tool_config(config, args.command)
    tools[args.command]["run"](args, tool_config)
```

## Step 4 — Tool wrappers (thin but complete)

Each tool file inlines the core logic from the referenced project. This makes the toolkit self-contained:
- `tools/review.py` — copy the chunking + LLM call + output logic from p2-01
- `tools/explain.py` — copy the stats parsing + two LLM calls from p2-02
- etc.

Don't import across projects. The tools directory is the canonical place for the simplified, toolkit-integrated version of each capability.

## Step 5 — Config-driven behavior

When `run(args, config)` receives `config`, it can use it for defaults:
```python
def run(args, config):
    min_severity = args.min_severity or config.get("min_severity", "LOW")
    # ...
```

This lets users set project-wide defaults in `.aiworkflow.yml` that can be overridden per invocation.

## Debugging tips

- If `ai` is not found after `pip install -e .`, check `pyproject.toml` entry points
- If a tool loads but `run` is not called, check subcommand name matches `TOOL_NAME` exactly
- Test `discover_tools` first — if it returns an empty list, check the tools directory path

## How to talk about this in an interview

**"Why a plugin architecture instead of one big CLI?"**
> Each tool has independent dependencies and configuration. A plugin system means I can add, disable, or replace tools without touching the dispatcher. The config file (`aiworkflow.yml`) is the API — not the code.

**"How does a new engineer add a tool?"**
> One file in `tools/`, one entry in the YAML. The discovery mechanism picks it up automatically. No changes to the dispatcher, no registration step. The interface contract is four names — if your file has them, it works.

**"What's the tradeoff of inlining logic vs importing from other projects?"**
> Inlining avoids cross-project import path hacking and makes each tool standalone. The tradeoff is duplication. In production, you'd package the core logic as a library and import it from both the individual project and the toolkit.
```

---

### Failure modes to handle

| Failure | Where | How to handle |
|---|---|---|
| `.aiworkflow.yml` not found | `config.py` | Return default config with all 4 tools enabled; log "No config file found, using defaults" |
| Tool file has syntax error | `plugin_loader.py` | Catch `SyntaxError`; log warning; skip this tool; continue loading others |
| Tool missing required interface attribute | `plugin_loader.py` | Raise `AttributeError` with message naming the missing attribute |
| Unknown subcommand | `main.py` | Print help text with available tools; exit with code 1 |
| Search tool called without index | `tools/search.py` | Print "No index found. Run: ai index --dir <path>" and exit 1 |
| Query tool called without db file | `tools/query.py` | Print "Database not found: {db_path}. Check .aiworkflow.yml db_path." and exit 1 |

---

### The metric this project measures

**What is measured:** Tool discovery count, config compliance (disabled tools unavailable), and dispatch success rate.

**Format (stdout on `ai` with no args):**
```
Available tools (4 active):
  review   Code review — finds issues in a git diff
  explain  PR summarizer — plain-English diff summary
  search   Semantic codebase search
  query    Natural language to SQL
```

**Target:** All 4 tool files are discovered automatically. Disabling one in `.aiworkflow.yml` reduces the count to 3 and removes it from help output. Each `ai <tool>` call successfully dispatches to the correct `run()` function. These three confirm plugin discovery, config gating, and dispatch are all working.


### Model source currency

- Anthropic model list: https://docs.anthropic.com/en/docs/about-claude/models/overview
- OpenAI model list: https://platform.openai.com/docs/models
- verifiedOn: 2026-08-07
- `LLM_MODEL` is required and has no implicit default; choose a supported model for the configured provider at setup time.

## Source code

A reference implementation of AI CLI Toolkit — README, guide, source and tests — is at https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/ai-cli-toolkit.