LLM Observability

Field	Value
Path	2 — AI-Augmented Engineering
Position	6 of 12
Difficulty	🟡 Optional Docker for Jaeger
Estimated time	4 hours
AWS cost	None

---

Agent Pickup Instructions

# 1. Enter project
cd projects/p2-06-llm-observability

# 2. Create virtual environment
python -m venv .venv && source .venv/bin/activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Copy env file and add your API key
cp .env.example .env

# 5. (Optional) Start Jaeger for trace visualization
docker compose up -d

# 6. Run the demo — 20 requests with terminal dashboard
python src/main.py --requests 20

# 7. Run tests
pytest tests/ -v

Done when:

• Running python src/main.py prints a live terminal dashboard with per-call latency, tokens, and cost
• Any call exceeding LATENCY_THRESHOLD_MS prints a WARNING log with the trace ID
• All OTEL span attributes include model, prompt_tokens, completion_tokens, latency_ms, cost_usd, provider
• All 4 tests pass
• (Optional) Jaeger UI at http://localhost:16686 shows traces when Docker is running

---

What this project is

An LLM observability layer built as a Python context manager wrapper (LLMTracer) that instruments every LLM call with OpenTelemetry spans. Each span carries model name, token counts, latency, cost estimate, provider, and trace ID. A terminal dashboard accumulates statistics across calls and flags anomalies (slow calls, high cost, unusual token ratios) with WARNING log lines containing the trace ID. An optional Docker Compose file exports spans to Jaeger for visual trace exploration. The demo script instruments 20 LLM calls to demonstrate the full pipeline.

---

What the learner achieves

"I built an LLM observability wrapper using OpenTelemetry that records model, tokens, latency, and cost per call, propagates trace IDs for correlation, and flags anomalies — demonstrating production-grade instrumentation of AI systems, not just logging."

---

Folder structure

p2-06-llm-observability/
├── src/
│   ├── main.py          # Demo script + terminal dashboard
│   ├── llm.py           # Provider-agnostic LLM wrapper
│   ├── tracer.py        # LLMTracer class
│   └── config.py        # Thresholds and config from env
├── tests/
│   ├── test_tracer.py
│   └── test_config.py
├── docker-compose.yml   # Jaeger for optional trace visualization
├── .env.example
├── requirements.txt
└── README.md

---

.env.example

# LLM provider: anthropic | openai | ollama
LLM_PROVIDER=anthropic

# Model to use
LLM_MODEL=

# API key for Anthropic
ANTHROPIC_API_KEY=

# API key for OpenAI
OPENAI_API_KEY=

# Ollama base URL
OLLAMA_BASE_URL=http://localhost:11434

# OTEL export: console | jaeger
OTEL_EXPORTER=console

# Jaeger OTLP endpoint (only used if OTEL_EXPORTER=jaeger)
JAEGER_ENDPOINT=http://localhost:4317

# Service name shown in Jaeger
OTEL_SERVICE_NAME=llm-observability-demo

# Anomaly thresholds
LATENCY_THRESHOLD_MS=5000
COST_THRESHOLD_USD=0.01
TOKEN_RATIO_THRESHOLD=10.0

# Token cost per 1K tokens (used for cost estimation)
# Update these for your model
COST_PER_1K_INPUT_TOKENS=0.00025
COST_PER_1K_OUTPUT_TOKENS=0.00125

---

requirements.txt

anthropic==0.30.0
openai==1.35.0
opentelemetry-sdk==1.25.0
opentelemetry-api==1.25.0
opentelemetry-exporter-otlp-proto-grpc==1.25.0
python-dotenv==1.0.1
rich==13.7.1
pytest==8.2.2
pytest-mock==3.14.0

---

src/ — what to implement

src/config.py

load_config() -> dict

• Inputs: none (reads from env)
• Output: dict with all threshold and config values
• Keys: latency_threshold_ms, cost_threshold_usd, token_ratio_threshold, cost_per_1k_input, cost_per_1k_output, otel_exporter, jaeger_endpoint, service_name
• Behavior: os.getenv with defaults for each; convert numeric strings to float/int

estimate_cost(input_tokens: int, output_tokens: int) -> float

• Inputs: token counts
• Output: estimated cost in USD (float, rounded to 6 decimal places)
• Behavior: (input_tokens / 1000) * cost_per_1k_input + (output_tokens / 1000) * cost_per_1k_output

---

src/llm.py

get_completion(prompt: str, system: str = "") -> tuple[str, dict]

• Inputs: prompt, system prompt
• Output: (response_text, usage_dict) where usage_dict = {"input_tokens": int, "output_tokens": int}
• Behavior: routes to provider; extracts token usage from API response (Anthropic: response.usage, OpenAI: response.usage; Ollama: estimate from response length)
• This version differs from other projects: it returns usage data alongside the text

---

src/tracer.py

LLMTracer (class)

__init__(self, service_name: str = None)

• Reads config
• Sets up OTEL TracerProvider
• If OTEL_EXPORTER=jaeger: add OTLPSpanExporter pointing to JAEGER_ENDPOINT
• If OTEL_EXPORTER=console: add ConsoleSpanExporter
• Initialize dashboard stats dict: {"calls": 0, "total_tokens": 0, "total_cost_usd": 0.0, "total_latency_ms": 0, "anomalies": 0}

trace_call(self, prompt: str, system: str = "") -> contextmanager

• Context manager (use @contextmanager decorator)
• Usage: with tracer.trace_call(prompt, system) as ctx: ctx.response, ctx.usage = ...
• Actually: implement as a method that wraps the LLM call internally:

  def traced_completion(self, prompt: str, system: str = "") -> tuple[str, str]:
      # returns (response_text, trace_id)
  

• Behavior:
  1. Generate a trace ID: uuid4().hex
  2. Start OTEL span named llm.completion
  3. Record start time
  4. Call llm.get_completion(prompt, system)
  5. Record end time → compute latency_ms
  6. Compute cost_usd via estimate_cost
  7. Set span attributes: llm.model, llm.provider, llm.prompt_tokens, llm.completion_tokens, llm.latency_ms, llm.cost_usd, llm.trace_id
  8. Update dashboard stats
  9. Call flag_anomalies with the span data
  10. Return (response_text, trace_id)

flag_anomalies(self, data: dict) -> None

• Inputs: dict with latency_ms, cost_usd, prompt_tokens, completion_tokens, trace_id
• Output: logs WARNING to stdout if any threshold is exceeded
• Checks:
  • latency_ms > LATENCY_THRESHOLD_MS → WARNING | slow_call | trace_id={} | latency_ms={}
  • cost_usd > COST_THRESHOLD_USD → WARNING | high_cost | trace_id={} | cost_usd={}
  • completion_tokens / max(prompt_tokens, 1) > TOKEN_RATIO_THRESHOLD → WARNING | high_token_ratio | trace_id={}
• Increment self.stats["anomalies"] for each warning

get_dashboard_stats(self) -> dict

• Output: copy of self.stats with derived metrics: avg_latency_ms, avg_cost_usd, calls

print_dashboard(self) -> None

• Prints a formatted table to stdout using rich:

LLM Observability Dashboard
────────────────────────────────────
Calls          : 20
Total tokens   : 4,231
Avg latency    : 342 ms
Total cost     : $0.0012
Anomalies      : 1
────────────────────────────────────

---

src/main.py

Demo script.

Arguments:

• --requests N — number of demo LLM calls to make (default 20)
• --prompt "..." — custom prompt to use (default: a rotating list of 5 short prompts)

Behavior:

1. Load .env
2. Create LLMTracer
3. Define a list of 5 short demo prompts (e.g., "What is recursion?", "Explain a binary tree.", etc.)
4. For each of N requests: pick a prompt from the rotating list; call tracer.traced_completion(prompt); print [{i}/{N}] trace_id={id} | latency={ms}ms | tokens={n} | cost=${c}
5. After all calls: print dashboard

---

tests/ — what to test

Test 1 — Span attributes include all required fields (test_tracer.py)

• Create an LLMTracer with OTEL_EXPORTER=console
• Mock llm.get_completion to return ("response", {"input_tokens": 100, "output_tokens": 50})
• Call tracer.traced_completion("test prompt")
• Capture the span (use opentelemetry.sdk.trace.export.InMemorySpanExporter)
• Assert span attributes contain: llm.model, llm.provider, llm.prompt_tokens, llm.completion_tokens, llm.latency_ms, llm.cost_usd, llm.trace_id

Test 2 — Anomaly flagging triggers at threshold (test_tracer.py)

• Create LLMTracer with LATENCY_THRESHOLD_MS=100
• Call flag_anomalies with latency_ms=5000
• Assert self.stats["anomalies"] == 1
• Capture stdout and assert "WARNING" and "slow_call" appear

Test 3 — Terminal dashboard accumulates correctly (test_tracer.py)

• Create LLMTracer
• Mock traced_completion to simulate 3 calls with known latency and token values
• Call get_dashboard_stats
• Assert calls == 3
• Assert total_tokens equals sum of token counts from the 3 calls

Test 4 — Trace ID is a UUID hex format (test_tracer.py)

• Call tracer.traced_completion("hello") with mocked LLM
• Assert the returned trace_id matches regex ^[0-9a-f]{32}$ (32 hex chars, no dashes)

Test 5 — Cost estimation is accurate (test_config.py)

• Call estimate_cost(input_tokens=1000, output_tokens=500) with known per-1K rates from env
• Assert result equals (1000/1000 * rate_in) + (500/1000 * rate_out) (float comparison with tolerance)

Test 6 — Config loads thresholds from env (test_config.py)

• Set env vars LATENCY_THRESHOLD_MS=3000, COST_THRESHOLD_USD=0.05
• Call load_config()
• Assert config["latency_threshold_ms"] == 3000
• Assert config["cost_threshold_usd"] == 0.05

---

README.md content

# LLM Observability

OpenTelemetry-based observability wrapper for LLM calls. Records model, tokens, latency, and cost per call. Flags anomalies. Exports to terminal dashboard or Jaeger.

## Quick start

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env   # add your API key

# Terminal dashboard (no Docker needed)
python src/main.py --requests 20

# With Jaeger (requires Docker)
docker compose up -d
OTEL_EXPORTER=jaeger python src/main.py --requests 20
# Open http://localhost:16686

Span attributes

Every LLM call gets an OTEL span with:

Attribute	Example
llm.model	the configured LLM_MODEL value
llm.provider	anthropic
llm.prompt_tokens	128
llm.completion_tokens	64
llm.latency_ms	342
llm.cost_usd	0.000048
llm.trace_id	a1b2c3d4...

Anomaly thresholds

Set in .env:

• LATENCY_THRESHOLD_MS — warn if a call takes longer (default 5000ms)
• COST_THRESHOLD_USD — warn if a single call costs more (default $0.01)
• TOKEN_RATIO_THRESHOLD — warn if output/input ratio is unusual (default 10.0)

Running tests

pytest tests/ -v

---

## GUIDE.md content

```markdown
# Build Guide — LLM Observability

## Step 1 — OTEL setup

```python
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor

provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("llm-observability")

For Jaeger, swap ConsoleSpanExporter with OTLPSpanExporter:

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)

Step 2 — Wrapping the LLM call

import time
from uuid import uuid4

def traced_completion(self, prompt, system=""):
    trace_id = uuid4().hex
    with tracer.start_as_current_span("llm.completion") as span:
        start = time.monotonic()
        response, usage = llm.get_completion(prompt, system)
        latency_ms = int((time.monotonic() - start) * 1000)
        
        cost = estimate_cost(usage["input_tokens"], usage["output_tokens"])
        
        span.set_attribute("llm.trace_id", trace_id)
        span.set_attribute("llm.model", os.getenv("LLM_MODEL"))
        span.set_attribute("llm.latency_ms", latency_ms)
        span.set_attribute("llm.cost_usd", cost)
        span.set_attribute("llm.prompt_tokens", usage["input_tokens"])
        span.set_attribute("llm.completion_tokens", usage["output_tokens"])
        
        self.flag_anomalies({...})
        self._update_stats(latency_ms, cost, usage)
    
    return response, trace_id

Step 3 — Anomaly flagging

Use Python's logging module for WARNING output — it's structured and can be redirected:

import logging
logger = logging.getLogger("llm.anomaly")

if latency_ms > self.config["latency_threshold_ms"]:
    logger.warning(f"slow_call | trace_id={trace_id} | latency_ms={latency_ms}")
    self.stats["anomalies"] += 1

Step 4 — Terminal dashboard

Use rich.table.Table for the dashboard. Call print_dashboard() after all requests complete.

Step 5 — Jaeger Docker Compose

services:
  jaeger:
    image: jaegertracing/all-in-one:1.57
    ports:
      - "16686:16686"   # UI
      - "4317:4317"     # OTLP gRPC

How to talk about this in an interview

"Why OTEL instead of just logging?"

OTEL gives you trace context propagation — a trace ID that connects multiple spans across services. Logs are siloed per service. OTEL lets you follow a single request through a chain of LLM calls, embeddings, and database queries in one trace view.

"How do you estimate cost without a billing API?"

I use the published per-token pricing from the provider's docs, stored in env vars. It's an estimate, not a bill, but it's accurate enough for anomaly detection and capacity planning. I store it per call so I can aggregate by time period.

"What would you add for production?"

Histograms for latency distribution (P50/P95/P99), a Prometheus exporter for Grafana dashboards, and budget alerts via email/Slack when cumulative cost exceeds a daily threshold.

---

## Failure modes to handle

| Failure | Where | How to handle |
|---|---|---|
| Jaeger is down but `OTEL_EXPORTER=jaeger` | `tracer.py` | Catch connection error at startup; fall back to console exporter; log warning |
| LLM provider does not return token counts | `llm.py` | Estimate from string length: `len(text.split())` as proxy; mark usage as `estimated=True` |
| `traced_completion` called before OTEL setup | `tracer.py` | `__init__` always runs setup; this cannot happen if constructor is called first |
| Dashboard stats overflow on long runs | `tracer.py` | Use Python floats; no overflow risk for typical usage |
| Anomaly threshold set to 0 | `config.py` | Warn at startup if threshold is 0: "LATENCY_THRESHOLD_MS=0 will flag every call" |

---

## The metric this project measures

**What is measured:** Per-call latency (ms), token count (input + output), cost (USD), and anomaly rate across a batch of calls.

**Format (stdout dashboard):**

LLM Observability Dashboard Calls : 20 Total tokens : 4,231 Avg latency : 342 ms Total cost : $0.0012 Anomalies : 1

**Target:** Over 20 demo calls, the dashboard must show accurate accumulated values (verify by summing from per-call log lines). Anomaly count must match the number of WARNING lines printed during the run. This confirms that span attributes, accumulation, and anomaly detection are all wired together correctly.


## Model source currency

- Anthropic model list: https://docs.anthropic.com/en/docs/about-claude/models/overview
- OpenAI model list: https://platform.openai.com/docs/models
- verifiedOn: 2026-08-07
- `LLM_MODEL` is required and has no implicit default; choose a supported model for the configured provider at setup time.