--- title: "Team Knowledge Extractor" description: "Internal knowledge tooling is a recurring 'AI champion' project at mid-size engineering orgs — a strong, specific example for a Developer Productivity Engineer..." source: "https://confidentprep.com/paths/ai-augmented-engineering/team-knowledge-extractor/" path: "AI-Augmented Engineering" repository: "https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/team-knowledge-extractor" token_estimate: 4459 --- # Team Knowledge Extractor ## Overview Extracts structured decisions from GitHub issues and ADR markdown using an LLM, then ranks search results with keyword + recency scoring instead of embeddings. The deliberate choice not to reach for embeddings here is itself the lesson — semantic search (project 3) isn't always the right tool, and knowing when a simpler ranking approach wins is a real engineering judgment call. ## What to do | Field | Value | |---|---| | Path | 2 — AI-Augmented Engineering | | Position | 8 of 12 | | Difficulty | 🟢 Local only | | Estimated time | 3 hours | | AWS cost | None | --- ### Agent Pickup Instructions ```bash # 1. Enter project cd projects/p2-08-team-knowledge-extractor # 2. Create virtual environment python -m venv .venv && source .venv/bin/activate # 3. Install dependencies pip install -r requirements.txt # 4. Copy env file and add your API key cp .env.example .env # 5. Extract decisions from a folder of issue exports python src/main.py --extract tests/fixtures/issues/ # 6. Search for decisions python src/main.py --search "authentication" # 7. Show all stored decisions python src/main.py --show # 8. Run tests pytest tests/ -v ``` **Done when:** - [ ] `--extract ` processes all `.json` and `.md` files and stores structured decisions - [ ] `--search "query"` returns ranked results with date, decision, and author - [ ] `--show` lists all decisions sorted by recency - [ ] All 4 tests pass - [ ] No embeddings are used (keyword + recency only) --- ### What this project is A structured knowledge extraction tool that processes GitHub issue exports (JSON format) and Architecture Decision Records (ADR markdown files) using an LLM to pull out the key decision, rationale, author, date, and tags from each document. Extracted decisions are stored in a local JSON database. Search is implemented as keyword scoring plus recency ranking — deliberately avoiding embeddings (that's the next project) to demonstrate that simpler retrieval is often sufficient for structured, well-labeled data. --- ### What the learner achieves "I built a team knowledge extraction pipeline that uses an LLM to parse unstructured GitHub issues and ADR docs into structured decision records, then provides keyword + recency search — demonstrating that you don't always need embeddings when your data is structured and well-labeled." --- ### Folder structure ``` p2-08-team-knowledge-extractor/ ├── src/ │ ├── main.py # CLI: --extract , --search "query", --show │ ├── llm.py # Provider-agnostic LLM wrapper │ ├── extractor.py # load_github_issue_json, load_adr_markdown, extract_decision │ ├── store.py # KnowledgeStore: save, search_keyword_recency, list_all │ └── searcher.py # score_keyword_match, rank_by_recency ├── tests/ │ ├── fixtures/ │ │ ├── issues/ │ │ │ ├── issue_42.json # GitHub issue JSON format │ │ │ └── issue_99.json # Another GitHub issue │ │ └── adrs/ │ │ ├── 0001-use-postgres.md # ADR markdown │ │ └── 0002-auth-strategy.md # ADR markdown │ ├── test_extractor.py │ ├── test_store.py │ └── test_searcher.py ├── .knowledge_store.json # Auto-created local decision database ├── .env.example ├── requirements.txt └── README.md ``` --- ### .env.example ```bash # LLM provider: anthropic | openai | ollama LLM_PROVIDER=anthropic # Model to use LLM_MODEL= # API key for Anthropic ANTHROPIC_API_KEY= # API key for OpenAI OPENAI_API_KEY= # Ollama base URL OLLAMA_BASE_URL=http://localhost:11434 # Path to local knowledge store JSON file STORE_PATH=.knowledge_store.json # Number of search results to show SEARCH_TOP_K=5 # Recency weight in ranking (0.0 = keyword only, 1.0 = recency only) RECENCY_WEIGHT=0.3 ``` --- ### requirements.txt ``` anthropic==0.30.0 openai==1.35.0 python-dotenv==1.0.1 rich==13.7.1 pytest==8.2.2 pytest-mock==3.14.0 ``` --- ### src/ — what to implement #### `src/llm.py` **`get_completion(prompt: str, system: str = "") -> str`** - Standard interface: reads `LLM_PROVIDER` and `LLM_MODEL` from env **`get_json_completion(prompt: str, system: str = "") -> dict | list`** - Calls `get_completion`, strips markdown fences, parses JSON - Raises `ValueError` with raw text if parsing fails --- #### `src/extractor.py` **`Decision` (dataclass)** ```python @dataclass class Decision: id: str # UUID hex date: str # ISO date string e.g. "2024-01-15" decision: str # One-sentence summary of the decision made rationale: str # Why this decision was made author: str # GitHub username or ADR author tags: list[str] # Relevant topics e.g. ["authentication", "database"] source_file: str # Filename it was extracted from source_type: str # "github_issue" or "adr" raw_title: str # Original title for display ``` **`load_github_issue_json(filepath: str) -> dict`** - Inputs: path to a JSON file - Output: raw dict with keys: `number`, `title`, `body`, `comments`, `closed_at` - Input format: ```json { "number": 42, "title": "Switch to PostgreSQL for better JSONB support", "body": "After evaluating MySQL and PostgreSQL...", "comments": [{"body": "Agreed, we should use PostgreSQL"}], "closed_at": "2024-01-15T10:00:00Z" } ``` - Edge cases: missing keys get default empty values; invalid JSON → raise `ValueError` with filepath **`load_adr_markdown(filepath: str) -> dict`** - Inputs: path to an ADR `.md` file - Output: `{"title": str, "date": str | None, "content": str}` - Behavior: read full markdown; extract `# Title` from first heading; extract date from a `Date: YYYY-MM-DD` line if present; content = full file text - Edge cases: no title found → use filename as title; no date found → `None` **`extract_decision(raw_content: dict, source_type: str, source_file: str) -> Decision`** - Inputs: raw content dict (from loader), source type string, filename - Output: `Decision` dataclass - Behavior: - Build user prompt combining title + body + comments (or ADR content) - System prompt: "You are an engineering knowledge curator. Extract the key architectural or process decision from this document. Return JSON with: decision (one sentence), rationale (2-3 sentences), author (if identifiable, else 'unknown'), tags (list of 3-5 topic keywords), date (ISO date if found, else null)." - Call `get_json_completion` - Combine LLM output with source metadata to construct `Decision` - Generate UUID with `uuid4().hex[:12]` as `id` - `date`: use LLM-extracted date if available; fall back to `closed_at` for issues; fall back to today's date - Edge cases: if LLM returns `null` for `decision`, set `decision = raw_content.get("title", "Unknown decision")` --- #### `src/store.py` **`KnowledgeStore` (class)** `__init__(self, store_path: str = None)` - Load existing JSON store from `STORE_PATH` env var or param - If file doesn't exist, initialize with empty list - Store data format: `{"decisions": [decision_dict, ...], "last_updated": "ISO datetime"}` `save(self, decision: Decision) -> None` - Add decision to store (convert dataclass to dict) - Check for duplicate by `source_file` — update in place rather than adding duplicate - Write to disk with `json.dump` (pretty-printed, indent=2) - Print `Saved: "{decision.raw_title}"` to stdout `search_keyword_recency(self, query: str, top_k: int = 5) -> list[Decision]` - Inputs: query string, number of results - Output: list of `Decision` ranked by combined keyword + recency score - Behavior: for each decision, compute `score_keyword_match(query, decision)` and recency score; combine with `RECENCY_WEIGHT`; return top k `list_all(self) -> list[Decision]` - Return all decisions sorted by date descending (most recent first) - Convert dicts back to `Decision` dataclasses `export_json(self, output_path: str) -> None` - Write full store to a specified output path --- #### `src/searcher.py` **`score_keyword_match(query: str, decision: Decision) -> float`** - Inputs: query string, a `Decision` - Output: float score between 0.0 and 1.0 - Behavior: - Tokenize query: `query.lower().split()` - Search fields: `decision.decision`, `decision.rationale`, `decision.tags`, `decision.raw_title` - For each query token: check if it appears in any search field (case-insensitive) - Score = `matched_tokens / total_tokens` - Tag matches get a 2x bonus: if token matches a tag exactly, count it twice **`recency_score(decision: Decision) -> float`** - Inputs: a `Decision` - Output: float 0.0 to 1.0 - Behavior: compute days since `decision.date`; score = `1.0 / (1.0 + log1p(days_old))`; returns 1.0 for today, decays logarithmically - Edge cases: if date is invalid/missing, return 0.5 (neutral) **`rank_decisions(query: str, decisions: list[Decision], recency_weight: float = 0.3, top_k: int = 5) -> list[Decision]`** - Inputs: query, decisions, recency weight (0-1), k - Output: sorted list - Behavior: `combined_score = (1 - recency_weight) * keyword_score + recency_weight * recency_score`; sort descending; return top k --- #### `src/main.py` CLI entry point using `argparse`. Arguments: - `--extract ` — process all `.json` and `.md` files in directory - `--search "query"` — search the knowledge store - `--show` — list all decisions - `--export ` — export full store to JSON file Behavior for `--extract`: 1. Find all `.json` and `.md` files in directory 2. For `.json` files: call `load_github_issue_json` → `extract_decision` 3. For `.md` files: call `load_adr_markdown` → `extract_decision` 4. For each: call `store.save` 5. Print `Extracted N decisions from M files` Behavior for `--search`: 1. Load store 2. Call `store.search_keyword_recency` 3. Print results as table: Rank | Date | Decision | Author | Tags --- ### tests/ — what to test #### Test 1 — Keyword search returns most relevant decision (`test_searcher.py`) - Create 3 `Decision` objects: one about "authentication", one about "database", one about "CI pipeline" - Call `rank_decisions("authentication oauth")` - Assert the first result is the authentication-related decision #### Test 2 — Recency ranking puts newer decisions first (`test_searcher.py`) - Create 2 identical decisions with different dates: one from 2020, one from 2024 - Call `rank_decisions("")` with `recency_weight=1.0` (pure recency) - Assert 2024 decision ranks first #### Test 3 — Extraction returns Decision with all required fields (`test_extractor.py`) - Mock LLM to return a valid JSON extraction result - Call `extract_decision` with a sample GitHub issue dict - Assert result has non-empty `decision`, `rationale`, `author`, `tags`, `date`, `source_file` - Assert `id` matches regex `^[0-9a-f]{12}$` #### Test 4 — Empty search query returns all results (`test_store.py`) - Save 3 decisions to a temp store - Call `search_keyword_recency("")` - Assert all 3 decisions are returned (empty query = no keyword filter) #### Test 5 — Duplicate source file updates in place (`test_store.py`) - Save a decision with `source_file="issue_42.json"` - Save another decision with the same `source_file="issue_42.json"` - Assert store has exactly 1 decision (not 2) #### Test 6 — ADR loader extracts title and date (`test_extractor.py`) - Use `tests/fixtures/adrs/0001-use-postgres.md` which starts with `# Use PostgreSQL\nDate: 2024-03-10` - Call `load_adr_markdown` - Assert `title == "Use PostgreSQL"` and `date == "2024-03-10"` --- ### README.md content ```markdown # Team Knowledge Extractor Extracts structured decisions from GitHub issues and ADR markdown files. Stores them in a searchable local JSON database with keyword + recency ranking. ## Quick start ```bash python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt cp .env.example .env # add your API key ## Extract from a folder python src/main.py --extract docs/decisions/ ## Search python src/main.py --search "database migration" ## List all python src/main.py --show ``` ## Input formats **GitHub Issue JSON:** ```json { "number": 42, "title": "Switch to PostgreSQL", "body": "We evaluated...", "comments": [{"body": "Agreed"}], "closed_at": "2024-01-15T10:00:00Z" } ``` **ADR Markdown:** ```markdown ## Use PostgreSQL for data storage Date: 2024-01-15 ### Decision We chose PostgreSQL over MySQL because... ``` ## Decision schema | Field | Description | |---|---| | `decision` | One-sentence summary | | `rationale` | Why this decision was made | | `author` | GitHub username or ADR author | | `tags` | Topic keywords (3-5) | | `date` | ISO date | | `source_file` | Origin file | ## Running tests ```bash pytest tests/ -v ``` ``` --- ### GUIDE.md content ```markdown # Build Guide — Team Knowledge Extractor ## Step 1 — Two input formats, one output schema Both input formats (GitHub JSON and ADR markdown) map to the same `Decision` dataclass. Write separate loaders but a single extractor that works on normalized content: ```python def extract_decision(raw_content, source_type, source_file): # raw_content has: title, body/content, date, author # These fields come from either loader prompt = build_extraction_prompt(raw_content) result = get_json_completion(prompt) return Decision( id=uuid4().hex[:12], decision=result["decision"], ... ) ``` ## Step 2 — The extraction prompt System prompt: ``` You are an engineering knowledge curator. Extract the architectural decision from this document. Return ONLY JSON with: { "decision": "one sentence describing the decision made", "rationale": "2-3 sentence explanation of why", "author": "person responsible or 'unknown'", "tags": ["topic1", "topic2", "topic3"], "date": "YYYY-MM-DD or null" } ``` Key constraint: `decision` must be one sentence. This forces the LLM to distill, not summarize. ## Step 3 — Keyword + recency ranking (no embeddings) ```python def score_keyword_match(query, decision): tokens = query.lower().split() if not tokens: return 1.0 # empty query matches everything text = f"{decision.decision} {decision.rationale} {' '.join(decision.tags)}".lower() matched = sum(1 for t in tokens if t in text) # bonus for tag matches tag_bonus = sum(1 for t in tokens if t in [tag.lower() for tag in decision.tags]) return min(1.0, (matched + tag_bonus) / len(tokens)) ``` ## Step 4 — The store Keep it simple: one JSON file, load on startup, save after every write. For hundreds of decisions this is fast enough. The key feature is duplicate detection by `source_file` — re-running `--extract` is idempotent. ## Debugging tips - If extraction produces empty decisions, print the raw LLM response before JSON parsing - If search returns irrelevant results, check that tags are being indexed (they get 2x weight) - Test with `--search ""` first — this returns all results and confirms the store is loaded ## How to talk about this in an interview **"Why no embeddings for search?"** > This is a deliberate design choice for structured data. Decisions have explicit tags, dates, and short summaries. Keyword + tag matching is interpretable, fast, and correct for this use case. I use embeddings in the next project for unstructured documentation — that's where they earn their complexity. **"How do you handle duplicate processing?"** > Each decision is keyed by source file. Re-running `--extract` on the same folder updates existing decisions in place rather than creating duplicates. This makes it safe to run as a cron job. **"What's the LLM doing in this project?"** > Structured extraction: converting free-form GitHub issues and markdown ADRs into the same schema. That's a task where even a small, fast model does well because the output is constrained and the instructions are precise. ``` --- ### Failure modes to handle | Failure | Where | How to handle | |---|---|---| | GitHub issue JSON missing required keys | `extractor.py` | Default to empty string for missing fields; use `title` as fallback for `decision` | | LLM returns null for `decision` field | `extractor.py` | Fall back to `raw_content["title"]` | | ADR file has no `# Title` heading | `extractor.py` | Use basename of filename as title | | Knowledge store file is corrupted JSON | `store.py` | Log warning; start fresh store; print "Corrupted store — starting fresh" | | `--search` called before any `--extract` | `main.py` | If store is empty, print "No decisions found. Run --extract first." | | Tags field is not a list in LLM response | `extractor.py` | If tags is a string, split on commas; if null, set to `[]` | --- ### The metric this project measures **What is measured:** Number of decisions extracted, keyword match rate in search results, and recency score spread across stored decisions. **Format (stdout):** ``` Extracted 4 decisions from 4 files Search: "authentication" → 3 results (top score: 0.85) ``` **Target:** Against the 4 fixture files (2 issues + 2 ADRs), all 4 decisions must be extracted with non-empty `decision` and `rationale` fields. Search for "authentication" against the auth-related fixture must return that decision in the top result. This confirms both extraction and search are functioning. ### Model source currency - Anthropic model list: https://docs.anthropic.com/en/docs/about-claude/models/overview - OpenAI model list: https://platform.openai.com/docs/models - verifiedOn: 2026-08-07 - `LLM_MODEL` is required and has no implicit default; choose a supported model for the configured provider at setup time. ## Source code A reference implementation of Team Knowledge Extractor — README, guide, source and tests — is at https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/team-knowledge-extractor.