Team Knowledge Extractor

Field	Value
Path	2 — AI-Augmented Engineering
Position	8 of 12
Difficulty	🟢 Local only
Estimated time	3 hours
AWS cost	None

---

Agent Pickup Instructions

# 1. Enter project
cd projects/p2-08-team-knowledge-extractor

# 2. Create virtual environment
python -m venv .venv && source .venv/bin/activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Copy env file and add your API key
cp .env.example .env

# 5. Extract decisions from a folder of issue exports
python src/main.py --extract tests/fixtures/issues/

# 6. Search for decisions
python src/main.py --search "authentication"

# 7. Show all stored decisions
python src/main.py --show

# 8. Run tests
pytest tests/ -v

Done when:

• --extract <dir> processes all .json and .md files and stores structured decisions
• --search "query" returns ranked results with date, decision, and author
• --show lists all decisions sorted by recency
• All 4 tests pass
• No embeddings are used (keyword + recency only)

---

What this project is

A structured knowledge extraction tool that processes GitHub issue exports (JSON format) and Architecture Decision Records (ADR markdown files) using an LLM to pull out the key decision, rationale, author, date, and tags from each document. Extracted decisions are stored in a local JSON database. Search is implemented as keyword scoring plus recency ranking — deliberately avoiding embeddings (that's the next project) to demonstrate that simpler retrieval is often sufficient for structured, well-labeled data.

---

What the learner achieves

"I built a team knowledge extraction pipeline that uses an LLM to parse unstructured GitHub issues and ADR docs into structured decision records, then provides keyword + recency search — demonstrating that you don't always need embeddings when your data is structured and well-labeled."

---

Folder structure

p2-08-team-knowledge-extractor/
├── src/
│   ├── main.py        # CLI: --extract <dir>, --search "query", --show
│   ├── llm.py         # Provider-agnostic LLM wrapper
│   ├── extractor.py   # load_github_issue_json, load_adr_markdown, extract_decision
│   ├── store.py       # KnowledgeStore: save, search_keyword_recency, list_all
│   └── searcher.py    # score_keyword_match, rank_by_recency
├── tests/
│   ├── fixtures/
│   │   ├── issues/
│   │   │   ├── issue_42.json   # GitHub issue JSON format
│   │   │   └── issue_99.json   # Another GitHub issue
│   │   └── adrs/
│   │       ├── 0001-use-postgres.md    # ADR markdown
│   │       └── 0002-auth-strategy.md  # ADR markdown
│   ├── test_extractor.py
│   ├── test_store.py
│   └── test_searcher.py
├── .knowledge_store.json   # Auto-created local decision database
├── .env.example
├── requirements.txt
└── README.md

---

.env.example

# LLM provider: anthropic | openai | ollama
LLM_PROVIDER=anthropic

# Model to use
LLM_MODEL=

# API key for Anthropic
ANTHROPIC_API_KEY=

# API key for OpenAI
OPENAI_API_KEY=

# Ollama base URL
OLLAMA_BASE_URL=http://localhost:11434

# Path to local knowledge store JSON file
STORE_PATH=.knowledge_store.json

# Number of search results to show
SEARCH_TOP_K=5

# Recency weight in ranking (0.0 = keyword only, 1.0 = recency only)
RECENCY_WEIGHT=0.3

---

requirements.txt

anthropic==0.30.0
openai==1.35.0
python-dotenv==1.0.1
rich==13.7.1
pytest==8.2.2
pytest-mock==3.14.0

---

src/ — what to implement

src/llm.py

get_completion(prompt: str, system: str = "") -> str

• Standard interface: reads LLM_PROVIDER and LLM_MODEL from env

get_json_completion(prompt: str, system: str = "") -> dict | list

• Calls get_completion, strips markdown fences, parses JSON
• Raises ValueError with raw text if parsing fails

---

src/extractor.py

Decision (dataclass)

@dataclass
class Decision:
    id: str              # UUID hex
    date: str            # ISO date string e.g. "2024-01-15"
    decision: str        # One-sentence summary of the decision made
    rationale: str       # Why this decision was made
    author: str          # GitHub username or ADR author
    tags: list[str]      # Relevant topics e.g. ["authentication", "database"]
    source_file: str     # Filename it was extracted from
    source_type: str     # "github_issue" or "adr"
    raw_title: str       # Original title for display

load_github_issue_json(filepath: str) -> dict

• Inputs: path to a JSON file
• Output: raw dict with keys: number, title, body, comments, closed_at
• Input format:

  {
    "number": 42,
    "title": "Switch to PostgreSQL for better JSONB support",
    "body": "After evaluating MySQL and PostgreSQL...",
    "comments": [{"body": "Agreed, we should use PostgreSQL"}],
    "closed_at": "2024-01-15T10:00:00Z"
  }
  

• Edge cases: missing keys get default empty values; invalid JSON → raise ValueError with filepath

load_adr_markdown(filepath: str) -> dict

• Inputs: path to an ADR .md file
• Output: {"title": str, "date": str | None, "content": str}
• Behavior: read full markdown; extract # Title from first heading; extract date from a Date: YYYY-MM-DD line if present; content = full file text
• Edge cases: no title found → use filename as title; no date found → None

extract_decision(raw_content: dict, source_type: str, source_file: str) -> Decision

• Inputs: raw content dict (from loader), source type string, filename
• Output: Decision dataclass
• Behavior:
  • Build user prompt combining title + body + comments (or ADR content)
  • System prompt: "You are an engineering knowledge curator. Extract the key architectural or process decision from this document. Return JSON with: decision (one sentence), rationale (2-3 sentences), author (if identifiable, else 'unknown'), tags (list of 3-5 topic keywords), date (ISO date if found, else null)."
  • Call get_json_completion
  • Combine LLM output with source metadata to construct Decision
  • Generate UUID with uuid4().hex[:12] as id
  • date: use LLM-extracted date if available; fall back to closed_at for issues; fall back to today's date
• Edge cases: if LLM returns null for decision, set decision = raw_content.get("title", "Unknown decision")

---

src/store.py

KnowledgeStore (class)

__init__(self, store_path: str = None)

• Load existing JSON store from STORE_PATH env var or param
• If file doesn't exist, initialize with empty list
• Store data format: {"decisions": [decision_dict, ...], "last_updated": "ISO datetime"}

save(self, decision: Decision) -> None

• Add decision to store (convert dataclass to dict)
• Check for duplicate by source_file — update in place rather than adding duplicate
• Write to disk with json.dump (pretty-printed, indent=2)
• Print Saved: "{decision.raw_title}" to stdout

search_keyword_recency(self, query: str, top_k: int = 5) -> list[Decision]

• Inputs: query string, number of results
• Output: list of Decision ranked by combined keyword + recency score
• Behavior: for each decision, compute score_keyword_match(query, decision) and recency score; combine with RECENCY_WEIGHT; return top k

list_all(self) -> list[Decision]

• Return all decisions sorted by date descending (most recent first)
• Convert dicts back to Decision dataclasses

export_json(self, output_path: str) -> None

• Write full store to a specified output path

---

src/searcher.py

score_keyword_match(query: str, decision: Decision) -> float

• Inputs: query string, a Decision
• Output: float score between 0.0 and 1.0
• Behavior:
  • Tokenize query: query.lower().split()
  • Search fields: decision.decision, decision.rationale, decision.tags, decision.raw_title
  • For each query token: check if it appears in any search field (case-insensitive)
  • Score = matched_tokens / total_tokens
  • Tag matches get a 2x bonus: if token matches a tag exactly, count it twice

recency_score(decision: Decision) -> float

• Inputs: a Decision
• Output: float 0.0 to 1.0
• Behavior: compute days since decision.date; score = 1.0 / (1.0 + log1p(days_old)); returns 1.0 for today, decays logarithmically
• Edge cases: if date is invalid/missing, return 0.5 (neutral)

rank_decisions(query: str, decisions: list[Decision], recency_weight: float = 0.3, top_k: int = 5) -> list[Decision]

• Inputs: query, decisions, recency weight (0-1), k
• Output: sorted list
• Behavior: combined_score = (1 - recency_weight) * keyword_score + recency_weight * recency_score; sort descending; return top k

---

src/main.py

CLI entry point using argparse.

Arguments:

• --extract <dir> — process all .json and .md files in directory
• --search "query" — search the knowledge store
• --show — list all decisions
• --export <path> — export full store to JSON file

Behavior for --extract:

1. Find all .json and .md files in directory
2. For .json files: call load_github_issue_json → extract_decision
3. For .md files: call load_adr_markdown → extract_decision
4. For each: call store.save
5. Print Extracted N decisions from M files

Behavior for --search:

1. Load store
2. Call store.search_keyword_recency
3. Print results as table: Rank | Date | Decision | Author | Tags

---

tests/ — what to test

Test 1 — Keyword search returns most relevant decision (test_searcher.py)

• Create 3 Decision objects: one about "authentication", one about "database", one about "CI pipeline"
• Call rank_decisions("authentication oauth")
• Assert the first result is the authentication-related decision

Test 2 — Recency ranking puts newer decisions first (test_searcher.py)

• Create 2 identical decisions with different dates: one from 2020, one from 2024
• Call rank_decisions("") with recency_weight=1.0 (pure recency)
• Assert 2024 decision ranks first

Test 3 — Extraction returns Decision with all required fields (test_extractor.py)

• Mock LLM to return a valid JSON extraction result
• Call extract_decision with a sample GitHub issue dict
• Assert result has non-empty decision, rationale, author, tags, date, source_file
• Assert id matches regex ^[0-9a-f]{12}$

Test 4 — Empty search query returns all results (test_store.py)

• Save 3 decisions to a temp store
• Call search_keyword_recency("")
• Assert all 3 decisions are returned (empty query = no keyword filter)

Test 5 — Duplicate source file updates in place (test_store.py)

• Save a decision with source_file="issue_42.json"
• Save another decision with the same source_file="issue_42.json"
• Assert store has exactly 1 decision (not 2)

Test 6 — ADR loader extracts title and date (test_extractor.py)

• Use tests/fixtures/adrs/0001-use-postgres.md which starts with # Use PostgreSQL\nDate: 2024-03-10
• Call load_adr_markdown
• Assert title == "Use PostgreSQL" and date == "2024-03-10"

---

README.md content

# Team Knowledge Extractor

Extracts structured decisions from GitHub issues and ADR markdown files. Stores them in a searchable local JSON database with keyword + recency ranking.

## Quick start

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env   # add your API key

# Extract from a folder
python src/main.py --extract docs/decisions/

# Search
python src/main.py --search "database migration"

# List all
python src/main.py --show

Input formats

GitHub Issue JSON:

{
  "number": 42,
  "title": "Switch to PostgreSQL",
  "body": "We evaluated...",
  "comments": [{"body": "Agreed"}],
  "closed_at": "2024-01-15T10:00:00Z"
}

ADR Markdown:

# Use PostgreSQL for data storage
Date: 2024-01-15

## Decision
We chose PostgreSQL over MySQL because...

Decision schema

Field	Description
decision	One-sentence summary
rationale	Why this decision was made
author	GitHub username or ADR author
tags	Topic keywords (3-5)
date	ISO date
source_file	Origin file

Running tests

pytest tests/ -v

---

## GUIDE.md content

```markdown
# Build Guide — Team Knowledge Extractor

## Step 1 — Two input formats, one output schema

Both input formats (GitHub JSON and ADR markdown) map to the same `Decision` dataclass. Write separate loaders but a single extractor that works on normalized content:

```python
def extract_decision(raw_content, source_type, source_file):
    # raw_content has: title, body/content, date, author
    # These fields come from either loader
    prompt = build_extraction_prompt(raw_content)
    result = get_json_completion(prompt)
    return Decision(
        id=uuid4().hex[:12],
        decision=result["decision"],
        ...
    )

Step 2 — The extraction prompt

System prompt:

You are an engineering knowledge curator.
Extract the architectural decision from this document.
Return ONLY JSON with:
{
  "decision": "one sentence describing the decision made",
  "rationale": "2-3 sentence explanation of why",
  "author": "person responsible or 'unknown'",
  "tags": ["topic1", "topic2", "topic3"],
  "date": "YYYY-MM-DD or null"
}

Key constraint: decision must be one sentence. This forces the LLM to distill, not summarize.

Step 3 — Keyword + recency ranking (no embeddings)

def score_keyword_match(query, decision):
    tokens = query.lower().split()
    if not tokens:
        return 1.0  # empty query matches everything
    
    text = f"{decision.decision} {decision.rationale} {' '.join(decision.tags)}".lower()
    matched = sum(1 for t in tokens if t in text)
    # bonus for tag matches
    tag_bonus = sum(1 for t in tokens if t in [tag.lower() for tag in decision.tags])
    return min(1.0, (matched + tag_bonus) / len(tokens))

Step 4 — The store

Keep it simple: one JSON file, load on startup, save after every write. For hundreds of decisions this is fast enough. The key feature is duplicate detection by source_file — re-running --extract is idempotent.

Debugging tips

• If extraction produces empty decisions, print the raw LLM response before JSON parsing
• If search returns irrelevant results, check that tags are being indexed (they get 2x weight)
• Test with --search "" first — this returns all results and confirms the store is loaded

How to talk about this in an interview

"Why no embeddings for search?"

This is a deliberate design choice for structured data. Decisions have explicit tags, dates, and short summaries. Keyword + tag matching is interpretable, fast, and correct for this use case. I use embeddings in the next project for unstructured documentation — that's where they earn their complexity.

"How do you handle duplicate processing?"

Each decision is keyed by source file. Re-running --extract on the same folder updates existing decisions in place rather than creating duplicates. This makes it safe to run as a cron job.

"What's the LLM doing in this project?"

Structured extraction: converting free-form GitHub issues and markdown ADRs into the same schema. That's a task where even a small, fast model does well because the output is constrained and the instructions are precise.

---

## Failure modes to handle

| Failure | Where | How to handle |
|---|---|---|
| GitHub issue JSON missing required keys | `extractor.py` | Default to empty string for missing fields; use `title` as fallback for `decision` |
| LLM returns null for `decision` field | `extractor.py` | Fall back to `raw_content["title"]` |
| ADR file has no `# Title` heading | `extractor.py` | Use basename of filename as title |
| Knowledge store file is corrupted JSON | `store.py` | Log warning; start fresh store; print "Corrupted store — starting fresh" |
| `--search` called before any `--extract` | `main.py` | If store is empty, print "No decisions found. Run --extract first." |
| Tags field is not a list in LLM response | `extractor.py` | If tags is a string, split on commas; if null, set to `[]` |

---

## The metric this project measures

**What is measured:** Number of decisions extracted, keyword match rate in search results, and recency score spread across stored decisions.

**Format (stdout):**

Extracted 4 decisions from 4 files Search: "authentication" → 3 results (top score: 0.85)

**Target:** Against the 4 fixture files (2 issues + 2 ADRs), all 4 decisions must be extracted with non-empty `decision` and `rationale` fields. Search for "authentication" against the auth-related fixture must return that decision in the top result. This confirms both extraction and search are functioning.


## Model source currency

- Anthropic model list: https://docs.anthropic.com/en/docs/about-claude/models/overview
- OpenAI model list: https://platform.openai.com/docs/models
- verifiedOn: 2026-08-07
- `LLM_MODEL` is required and has no implicit default; choose a supported model for the configured provider at setup time.