---
title: "LLM Observability"
description: "Observability is the skill that gets a senior engineer put in charge of a team's AI spend and reliability — a direct, ops-relevant differentiator over 'I can..."
source: "https://confidentprep.com/paths/ai-augmented-engineering/llm-observability/"
path: "AI-Augmented Engineering"
repository: "https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/llm-observability"
token_estimate: 4063
---

# LLM Observability

## Overview

An OpenTelemetry-based wrapper for LLM calls that records model, tokens, latency, and
cost per call, flags anomalies, and exports to a terminal dashboard or Jaeger. Every
project before this one made LLM calls; this is the first one that tells you, with real
numbers, what those calls are costing — a question every team with AI in production
eventually has to answer.

## What to do

| Field | Value |
|---|---|
| Path | 2 — AI-Augmented Engineering |
| Position | 6 of 12 |
| Difficulty | 🟡 Optional Docker for Jaeger |
| Estimated time | 4 hours |
| AWS cost | None |

---

### Agent Pickup Instructions

```bash
# 1. Enter project
cd projects/p2-06-llm-observability

# 2. Create virtual environment
python -m venv .venv && source .venv/bin/activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Copy env file and add your API key
cp .env.example .env

# 5. (Optional) Start Jaeger for trace visualization
docker compose up -d

# 6. Run the demo — 20 requests with terminal dashboard
python src/main.py --requests 20

# 7. Run tests
pytest tests/ -v
```

**Done when:**
- [ ] Running `python src/main.py` prints a live terminal dashboard with per-call latency, tokens, and cost
- [ ] Any call exceeding `LATENCY_THRESHOLD_MS` prints a WARNING log with the trace ID
- [ ] All OTEL span attributes include model, prompt_tokens, completion_tokens, latency_ms, cost_usd, provider
- [ ] All 4 tests pass
- [ ] (Optional) Jaeger UI at `http://localhost:16686` shows traces when Docker is running

---

### What this project is

An LLM observability layer built as a Python context manager wrapper (`LLMTracer`) that instruments every LLM call with OpenTelemetry spans. Each span carries model name, token counts, latency, cost estimate, provider, and trace ID. A terminal dashboard accumulates statistics across calls and flags anomalies (slow calls, high cost, unusual token ratios) with WARNING log lines containing the trace ID. An optional Docker Compose file exports spans to Jaeger for visual trace exploration. The demo script instruments 20 LLM calls to demonstrate the full pipeline.

---

### What the learner achieves

"I built an LLM observability wrapper using OpenTelemetry that records model, tokens, latency, and cost per call, propagates trace IDs for correlation, and flags anomalies — demonstrating production-grade instrumentation of AI systems, not just logging."

---

### Folder structure

```
p2-06-llm-observability/
├── src/
│   ├── main.py          # Demo script + terminal dashboard
│   ├── llm.py           # Provider-agnostic LLM wrapper
│   ├── tracer.py        # LLMTracer class
│   └── config.py        # Thresholds and config from env
├── tests/
│   ├── test_tracer.py
│   └── test_config.py
├── docker-compose.yml   # Jaeger for optional trace visualization
├── .env.example
├── requirements.txt
└── README.md
```

---

### .env.example

```bash
# LLM provider: anthropic | openai | ollama
LLM_PROVIDER=anthropic

# Model to use
LLM_MODEL=

# API key for Anthropic
ANTHROPIC_API_KEY=

# API key for OpenAI
OPENAI_API_KEY=

# Ollama base URL
OLLAMA_BASE_URL=http://localhost:11434

# OTEL export: console | jaeger
OTEL_EXPORTER=console

# Jaeger OTLP endpoint (only used if OTEL_EXPORTER=jaeger)
JAEGER_ENDPOINT=http://localhost:4317

# Service name shown in Jaeger
OTEL_SERVICE_NAME=llm-observability-demo

# Anomaly thresholds
LATENCY_THRESHOLD_MS=5000
COST_THRESHOLD_USD=0.01
TOKEN_RATIO_THRESHOLD=10.0

# Token cost per 1K tokens (used for cost estimation)
# Update these for your model
COST_PER_1K_INPUT_TOKENS=0.00025
COST_PER_1K_OUTPUT_TOKENS=0.00125
```

---

### requirements.txt

```
anthropic==0.30.0
openai==1.35.0
opentelemetry-sdk==1.25.0
opentelemetry-api==1.25.0
opentelemetry-exporter-otlp-proto-grpc==1.25.0
python-dotenv==1.0.1
rich==13.7.1
pytest==8.2.2
pytest-mock==3.14.0
```

---

### src/ — what to implement

#### `src/config.py`

**`load_config() -> dict`**
- Inputs: none (reads from env)
- Output: dict with all threshold and config values
- Keys: `latency_threshold_ms`, `cost_threshold_usd`, `token_ratio_threshold`, `cost_per_1k_input`, `cost_per_1k_output`, `otel_exporter`, `jaeger_endpoint`, `service_name`
- Behavior: `os.getenv` with defaults for each; convert numeric strings to float/int

**`estimate_cost(input_tokens: int, output_tokens: int) -> float`**
- Inputs: token counts
- Output: estimated cost in USD (float, rounded to 6 decimal places)
- Behavior: `(input_tokens / 1000) * cost_per_1k_input + (output_tokens / 1000) * cost_per_1k_output`

---

#### `src/llm.py`

**`get_completion(prompt: str, system: str = "") -> tuple[str, dict]`**
- Inputs: prompt, system prompt
- Output: `(response_text, usage_dict)` where `usage_dict = {"input_tokens": int, "output_tokens": int}`
- Behavior: routes to provider; extracts token usage from API response (Anthropic: `response.usage`, OpenAI: `response.usage`; Ollama: estimate from response length)
- This version differs from other projects: it returns usage data alongside the text

---

#### `src/tracer.py`

**`LLMTracer` (class)**

`__init__(self, service_name: str = None)`
- Reads config
- Sets up OTEL `TracerProvider`
- If `OTEL_EXPORTER=jaeger`: add `OTLPSpanExporter` pointing to `JAEGER_ENDPOINT`
- If `OTEL_EXPORTER=console`: add `ConsoleSpanExporter`
- Initialize dashboard stats dict: `{"calls": 0, "total_tokens": 0, "total_cost_usd": 0.0, "total_latency_ms": 0, "anomalies": 0}`

`trace_call(self, prompt: str, system: str = "") -> contextmanager`
- Context manager (use `@contextmanager` decorator)
- Usage: `with tracer.trace_call(prompt, system) as ctx: ctx.response, ctx.usage = ...`
- Actually: implement as a method that wraps the LLM call internally:
  ```python
  def traced_completion(self, prompt: str, system: str = "") -> tuple[str, str]:
      # returns (response_text, trace_id)
  ```
- Behavior:
  1. Generate a trace ID: `uuid4().hex`
  2. Start OTEL span named `llm.completion`
  3. Record start time
  4. Call `llm.get_completion(prompt, system)`
  5. Record end time → compute `latency_ms`
  6. Compute `cost_usd` via `estimate_cost`
  7. Set span attributes: `llm.model`, `llm.provider`, `llm.prompt_tokens`, `llm.completion_tokens`, `llm.latency_ms`, `llm.cost_usd`, `llm.trace_id`
  8. Update dashboard stats
  9. Call `flag_anomalies` with the span data
  10. Return `(response_text, trace_id)`

`flag_anomalies(self, data: dict) -> None`
- Inputs: dict with `latency_ms`, `cost_usd`, `prompt_tokens`, `completion_tokens`, `trace_id`
- Output: logs WARNING to stdout if any threshold is exceeded
- Checks:
  - `latency_ms > LATENCY_THRESHOLD_MS` → `WARNING | slow_call | trace_id={} | latency_ms={}`
  - `cost_usd > COST_THRESHOLD_USD` → `WARNING | high_cost | trace_id={} | cost_usd={}`
  - `completion_tokens / max(prompt_tokens, 1) > TOKEN_RATIO_THRESHOLD` → `WARNING | high_token_ratio | trace_id={}`
- Increment `self.stats["anomalies"]` for each warning

`get_dashboard_stats(self) -> dict`
- Output: copy of `self.stats` with derived metrics: `avg_latency_ms`, `avg_cost_usd`, `calls`

`print_dashboard(self) -> None`
- Prints a formatted table to stdout using `rich`:

```
LLM Observability Dashboard
────────────────────────────────────
Calls          : 20
Total tokens   : 4,231
Avg latency    : 342 ms
Total cost     : $0.0012
Anomalies      : 1
────────────────────────────────────
```

---

#### `src/main.py`

Demo script.

Arguments:
- `--requests N` — number of demo LLM calls to make (default 20)
- `--prompt "..."` — custom prompt to use (default: a rotating list of 5 short prompts)

Behavior:
1. Load `.env`
2. Create `LLMTracer`
3. Define a list of 5 short demo prompts (e.g., "What is recursion?", "Explain a binary tree.", etc.)
4. For each of N requests: pick a prompt from the rotating list; call `tracer.traced_completion(prompt)`; print `[{i}/{N}] trace_id={id} | latency={ms}ms | tokens={n} | cost=${c}`
5. After all calls: print dashboard

---

### tests/ — what to test

#### Test 1 — Span attributes include all required fields (`test_tracer.py`)
- Create an `LLMTracer` with `OTEL_EXPORTER=console`
- Mock `llm.get_completion` to return `("response", {"input_tokens": 100, "output_tokens": 50})`
- Call `tracer.traced_completion("test prompt")`
- Capture the span (use `opentelemetry.sdk.trace.export.InMemorySpanExporter`)
- Assert span attributes contain: `llm.model`, `llm.provider`, `llm.prompt_tokens`, `llm.completion_tokens`, `llm.latency_ms`, `llm.cost_usd`, `llm.trace_id`

#### Test 2 — Anomaly flagging triggers at threshold (`test_tracer.py`)
- Create `LLMTracer` with `LATENCY_THRESHOLD_MS=100`
- Call `flag_anomalies` with `latency_ms=5000`
- Assert `self.stats["anomalies"] == 1`
- Capture stdout and assert "WARNING" and "slow_call" appear

#### Test 3 — Terminal dashboard accumulates correctly (`test_tracer.py`)
- Create `LLMTracer`
- Mock `traced_completion` to simulate 3 calls with known latency and token values
- Call `get_dashboard_stats`
- Assert `calls == 3`
- Assert `total_tokens` equals sum of token counts from the 3 calls

#### Test 4 — Trace ID is a UUID hex format (`test_tracer.py`)
- Call `tracer.traced_completion("hello")` with mocked LLM
- Assert the returned `trace_id` matches regex `^[0-9a-f]{32}$` (32 hex chars, no dashes)

#### Test 5 — Cost estimation is accurate (`test_config.py`)
- Call `estimate_cost(input_tokens=1000, output_tokens=500)` with known per-1K rates from env
- Assert result equals `(1000/1000 * rate_in) + (500/1000 * rate_out)` (float comparison with tolerance)

#### Test 6 — Config loads thresholds from env (`test_config.py`)
- Set env vars `LATENCY_THRESHOLD_MS=3000`, `COST_THRESHOLD_USD=0.05`
- Call `load_config()`
- Assert `config["latency_threshold_ms"] == 3000`
- Assert `config["cost_threshold_usd"] == 0.05`

---

### README.md content

```markdown
# LLM Observability

OpenTelemetry-based observability wrapper for LLM calls. Records model, tokens, latency, and cost per call. Flags anomalies. Exports to terminal dashboard or Jaeger.

## Quick start

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env   # add your API key

## Terminal dashboard (no Docker needed)
python src/main.py --requests 20

## With Jaeger (requires Docker)
docker compose up -d
OTEL_EXPORTER=jaeger python src/main.py --requests 20
## Open http://localhost:16686
```

## Span attributes

Every LLM call gets an OTEL span with:

| Attribute | Example |
|---|---|
| `llm.model` | the configured `LLM_MODEL` value |
| `llm.provider` | `anthropic` |
| `llm.prompt_tokens` | `128` |
| `llm.completion_tokens` | `64` |
| `llm.latency_ms` | `342` |
| `llm.cost_usd` | `0.000048` |
| `llm.trace_id` | `a1b2c3d4...` |

## Anomaly thresholds

Set in `.env`:
- `LATENCY_THRESHOLD_MS` — warn if a call takes longer (default 5000ms)
- `COST_THRESHOLD_USD` — warn if a single call costs more (default $0.01)
- `TOKEN_RATIO_THRESHOLD` — warn if output/input ratio is unusual (default 10.0)

## Running tests

```bash
pytest tests/ -v
```
```

---

### GUIDE.md content

```markdown
# Build Guide — LLM Observability

## Step 1 — OTEL setup

```python
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor

provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("llm-observability")
```

For Jaeger, swap `ConsoleSpanExporter` with `OTLPSpanExporter`:
```python
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
```

## Step 2 — Wrapping the LLM call

```python
import time
from uuid import uuid4

def traced_completion(self, prompt, system=""):
    trace_id = uuid4().hex
    with tracer.start_as_current_span("llm.completion") as span:
        start = time.monotonic()
        response, usage = llm.get_completion(prompt, system)
        latency_ms = int((time.monotonic() - start) * 1000)
        
        cost = estimate_cost(usage["input_tokens"], usage["output_tokens"])
        
        span.set_attribute("llm.trace_id", trace_id)
        span.set_attribute("llm.model", os.getenv("LLM_MODEL"))
        span.set_attribute("llm.latency_ms", latency_ms)
        span.set_attribute("llm.cost_usd", cost)
        span.set_attribute("llm.prompt_tokens", usage["input_tokens"])
        span.set_attribute("llm.completion_tokens", usage["output_tokens"])
        
        self.flag_anomalies({...})
        self._update_stats(latency_ms, cost, usage)
    
    return response, trace_id
```

## Step 3 — Anomaly flagging

Use Python's `logging` module for WARNING output — it's structured and can be redirected:

```python
import logging
logger = logging.getLogger("llm.anomaly")

if latency_ms > self.config["latency_threshold_ms"]:
    logger.warning(f"slow_call | trace_id={trace_id} | latency_ms={latency_ms}")
    self.stats["anomalies"] += 1
```

## Step 4 — Terminal dashboard

Use `rich.table.Table` for the dashboard. Call `print_dashboard()` after all requests complete.

## Step 5 — Jaeger Docker Compose

```yaml
services:
  jaeger:
    image: jaegertracing/all-in-one:1.57
    ports:
      - "16686:16686"   # UI
      - "4317:4317"     # OTLP gRPC
```

## How to talk about this in an interview

**"Why OTEL instead of just logging?"**
> OTEL gives you trace context propagation — a trace ID that connects multiple spans across services. Logs are siloed per service. OTEL lets you follow a single request through a chain of LLM calls, embeddings, and database queries in one trace view.

**"How do you estimate cost without a billing API?"**
> I use the published per-token pricing from the provider's docs, stored in env vars. It's an estimate, not a bill, but it's accurate enough for anomaly detection and capacity planning. I store it per call so I can aggregate by time period.

**"What would you add for production?"**
> Histograms for latency distribution (P50/P95/P99), a Prometheus exporter for Grafana dashboards, and budget alerts via email/Slack when cumulative cost exceeds a daily threshold.
```

---

### Failure modes to handle

| Failure | Where | How to handle |
|---|---|---|
| Jaeger is down but `OTEL_EXPORTER=jaeger` | `tracer.py` | Catch connection error at startup; fall back to console exporter; log warning |
| LLM provider does not return token counts | `llm.py` | Estimate from string length: `len(text.split())` as proxy; mark usage as `estimated=True` |
| `traced_completion` called before OTEL setup | `tracer.py` | `__init__` always runs setup; this cannot happen if constructor is called first |
| Dashboard stats overflow on long runs | `tracer.py` | Use Python floats; no overflow risk for typical usage |
| Anomaly threshold set to 0 | `config.py` | Warn at startup if threshold is 0: "LATENCY_THRESHOLD_MS=0 will flag every call" |

---

### The metric this project measures

**What is measured:** Per-call latency (ms), token count (input + output), cost (USD), and anomaly rate across a batch of calls.

**Format (stdout dashboard):**
```
LLM Observability Dashboard
Calls          : 20
Total tokens   : 4,231
Avg latency    : 342 ms
Total cost     : $0.0012
Anomalies      : 1
```

**Target:** Over 20 demo calls, the dashboard must show accurate accumulated values (verify by summing from per-call log lines). Anomaly count must match the number of WARNING lines printed during the run. This confirms that span attributes, accumulation, and anomaly detection are all wired together correctly.


### Model source currency

- Anthropic model list: https://docs.anthropic.com/en/docs/about-claude/models/overview
- OpenAI model list: https://platform.openai.com/docs/models
- verifiedOn: 2026-08-07
- `LLM_MODEL` is required and has no implicit default; choose a supported model for the configured provider at setup time.

## Source code

A reference implementation of LLM Observability — README, guide, source and tests — is at https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/llm-observability.