Model Monitoring

Path: 3 — ML Engineering on AWS Position: 10 of 12 Difficulty: 🔴 Time: 3-4h AWS Cost: ~$0.05/monitoring run × 2-3 runs = $0.15. Plus endpoint from p3-07 ($0.056/hr).

---

Agent Pickup Instructions

# Bootstrap
cd projects/confident-prep/p3-10-model-monitoring
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Prerequisite: p3-07 endpoint must be running
# Check: aws sagemaker describe-endpoint --endpoint-name p3-07-adult-income-endpoint

# Step 1: Enable data capture on existing endpoint
python src/enable_capture.py --endpoint-name p3-07-adult-income-endpoint

# Step 2: Compute baseline statistics from training data
python src/compute_baseline.py --data data/adult.data

# Step 3: Schedule monitoring (hourly)
python src/schedule_monitoring.py --endpoint-name p3-07-adult-income-endpoint

# Step 4: Inject synthetic drift
python src/inject_drift.py --endpoint-name p3-07-adult-income-endpoint --n-requests 100

# Step 5: Read violation report (after monitoring job runs — may take up to 1 hour)
python src/read_violations.py

# Run tests
pytest tests/ -v

# TEARDOWN
bash scripts/teardown.sh

Done when:

• python src/enable_capture.py updates endpoint to capture 100% of requests
• python src/compute_baseline.py runs suggest_baseline() and saves baseline statistics JSON to S3
• python src/schedule_monitoring.py creates a monitoring schedule (ARN saved to .schedule-arn)
• python src/inject_drift.py sends 100 requests with shifted feature values
• python src/read_violations.py parses monitoring report and prints human-readable violations
• pytest tests/ -v shows 4/4 passing
• bash scripts/teardown.sh stops schedule, deletes schedule, deletes S3 capture + baseline data

---

What this project is

You enable SageMaker Model Monitor on the endpoint from p3-07: configure data capture (logging all requests and responses to S3), compute a baseline using DefaultModelMonitor.suggest_baseline() on the training data, and schedule hourly monitoring. Then you inject synthetic drift by sending requests where numeric features are shifted to 2 standard deviations above their training mean — values the monitor has never seen. A violation reader parses the monitoring JSON report and prints plain-English explanations of what drifted and by how much.

What the learner achieves

"I can set up SageMaker Model Monitor to detect data drift, inject synthetic drift to test that the alarm fires, and read violation reports in plain English — and I know what drift detection catches and what it misses."

---

Folder structure

p3-10-model-monitoring/
├── data/
│   └── adult.data              # training data for baseline
├── fixtures/
│   └── sample_violation_report.json  # SageMaker monitoring report format for tests
├── src/
│   ├── enable_capture.py      # update endpoint to enable DataCaptureConfig (100% capture)
│   ├── compute_baseline.py    # run suggest_baseline() on training data
│   ├── schedule_monitoring.py # create hourly monitoring schedule
│   ├── inject_drift.py        # send 100 requests with features at mean + 2*std
│   └── read_violations.py     # parse monitoring report JSON, print plain-English violations
├── scripts/
│   └── teardown.sh            # stop schedule, delete schedule, delete S3 capture + baseline
├── tests/
│   ├── test_inject_drift.py   # verify shifted values are out of expected range
│   └── test_read_violations.py # parse fixture JSON, verify output format
├── .schedule-arn               # written by schedule_monitoring.py
├── .env.example
├── requirements.txt
└── README.md

---

.env.example

# AWS region
AWS_REGION=us-east-1

# S3 bucket
S3_BUCKET=my-ml-training-bucket

# IAM role ARN with SageMaker full access
SAGEMAKER_ROLE_ARN=arn:aws:iam::123456789012:role/SageMakerExecutionRole

# Endpoint name (must match p3-07)
ENDPOINT_NAME=p3-07-adult-income-endpoint

# S3 paths for capture data and baseline
S3_CAPTURE_PATH=s3://my-ml-training-bucket/p3-10/capture/
S3_BASELINE_PATH=s3://my-ml-training-bucket/p3-10/baseline/
S3_MONITORING_RESULTS_PATH=s3://my-ml-training-bucket/p3-10/monitoring-results/

# Monitoring schedule name
MONITORING_SCHEDULE_NAME=p3-10-hourly-monitor

# Drift injection: number of std deviations above mean
DRIFT_STD_MULTIPLIER=2.0

---

requirements.txt

boto3==1.35.93
sagemaker==2.236.0
pandas==2.2.3
numpy==2.2.1
python-dotenv==1.0.1
pytest==8.3.4

---

src/ — what to implement

src/enable_capture.py

CLI: python src/enable_capture.py --endpoint-name <name>

enable_data_capture(endpoint_name: str, s3_capture_path: str, capture_percentage: int = 100) -> None

Creates a new EndpointConfig with DataCaptureConfig enabled and updates the endpoint.

DataCaptureConfig structure:

{
    "EnableCapture": True,
    "InitialSamplingPercentage": capture_percentage,
    "DestinationS3Uri": s3_capture_path,
    "CaptureOptions": [
        {"CaptureMode": "Input"},
        {"CaptureMode": "Output"},
    ],
    "CaptureContentTypeHeader": {
        "CsvContentTypes": ["text/csv"],
    }
}

Steps:

1. Describe current endpoint to get current EndpointConfigName
2. Describe current config to copy its ProductionVariants
3. Create a new EndpointConfig with the same variants + DataCaptureConfig
4. Call update_endpoint(EndpointName=endpoint_name, EndpointConfigName=new_config_name)
5. Wait for endpoint to return to InService using waiter

verify_capture_enabled(endpoint_name: str) -> bool

• Describes the current endpoint config
• Returns True if DataCaptureConfig.EnableCapture == True

main()

• enable_data_capture → verify_capture_enabled
• Print: Data capture enabled. 100% of requests will be logged to {s3_capture_path}

---

src/compute_baseline.py

CLI: python src/compute_baseline.py --data <path> [--instance-type ml.m5.large]

prepare_baseline_data(data_path: str, output_s3_uri: str) -> str

• Loads training data (UCI Adult), preprocesses (same as p3-01: column names, strip, NaN drop, get_dummies)
• Saves feature-only CSV (no target column) to S3 for baseline computation
• Returns S3 URI of uploaded file

run_baseline_suggestion(input_s3_uri: str, output_s3_uri: str, role_arn: str, instance_type: str) -> str

• Creates sagemaker.model_monitor.DefaultModelMonitor(role=role_arn, instance_type=instance_type, ...)
• Calls monitor.suggest_baseline(baseline_dataset=input_s3_uri, dataset_format=DatasetFormat.csv(), output_s3_uri=output_s3_uri, wait=True)
• Returns the S3 URI where baseline statistics were saved

main()

1. prepare_baseline_data → uploads training CSV to S3_BASELINE_PATH/data/
2. run_baseline_suggestion → runs the baseline job
3. Print: Baseline computed. Statistics at: {output_s3_uri}
4. Print statistics summary: number of features monitored

---

src/schedule_monitoring.py

CLI: python src/schedule_monitoring.py --endpoint-name <name>

create_monitoring_schedule(endpoint_name: str, schedule_name: str, baseline_s3_uri: str, output_s3_uri: str, role_arn: str) -> str

Creates a monitoring schedule using sagemaker.model_monitor.DefaultModelMonitor:

monitor = DefaultModelMonitor(role=role_arn, ...)
monitor.create_monitoring_schedule(
    monitor_schedule_name=schedule_name,
    endpoint_input=endpoint_name,
    output_s3_uri=output_s3_uri,
    statistics=baseline_s3_uri + "/statistics.json",
    constraints=baseline_s3_uri + "/constraints.json",
    schedule_cron_expression="cron(0 * ? * * *)",  # hourly
)

Returns the schedule ARN from monitor.monitoring_schedule_arn.

main()

• Create schedule
• Write ARN to .schedule-arn
• Print: Monitoring schedule created: {schedule_name}
• Print: Schedule ARN: {arn}
• Print: Note: first monitoring report will be generated in up to 1 hour.

---

src/inject_drift.py

CLI: python src/inject_drift.py --endpoint-name <name> [--n-requests 100]

compute_feature_stats(data_path: str) -> dict

• Loads UCI Adult training data
• Computes mean and std for each numeric feature
• Returns {feature_name: {"mean": float, "std": float}} for numeric features only

create_drifted_row(feature_stats: dict, std_multiplier: float = 2.0) -> np.ndarray

• For each numeric feature: set value to mean + std_multiplier * std
• For categorical features: use the most common value from training data (no drift for categoricals)
• Returns a 1D numpy array in the same column order as the OHE feature space

send_drifted_requests(endpoint_name: str, drifted_row: np.ndarray, n_requests: int) -> dict

• Converts drifted_row to CSV string
• Sends n_requests invocations to endpoint
• Returns {"sent": int, "succeeded": int, "failed": int}

main()

1. compute_feature_stats
2. create_drifted_row (with DRIFT_STD_MULTIPLIER from env, default 2.0)
3. Print: Injecting drift: {n_requests} requests with features at mean + {multiplier}*std
4. Print per-feature drift values: age: expected range [17, 90], injecting 97.3
5. send_drifted_requests
6. Print: Drift injection complete. {succeeded}/{sent} requests succeeded.

---

src/read_violations.py

CLI: python src/read_violations.py [--s3-path <path>] [--local-file <path>]

fetch_latest_report(monitoring_results_s3: str) -> dict

• Lists S3 objects under monitoring_results_s3 prefix
• Finds the most recently modified JSON file
• Downloads and parses it
• Returns the parsed dict

parse_violation(violation: dict) -> str

Converts a single SageMaker monitoring violation dict to a plain-English string.

SageMaker violation format:

{
  "feature_name": "age",
  "constraint_check_type": "distribution_non_parametric_significance",
  "description": "Value at given threshold violation",
  "metric": {"type": "threshold", "threshold": 0.05, "observed_value": 0.001}
}

Plain-English output format:

Feature 'age': distribution shifted significantly.
  Test: non-parametric significance test
  Expected p-value > 0.05, observed p-value = 0.001
  Interpretation: The distribution of 'age' in recent traffic is unlikely to match training data.

parse_monitoring_report(report: dict) -> list[str]

• Extracts the violations list from the report (path: report["violations"])
• Calls parse_violation() for each
• Returns list of plain-English strings

main()

• Fetch report from S3 or --local-file
• Parse violations
• Print:

  Model Monitor Report — {n_violations} violation(s) found
  ========================================
  {violation 1}
  ----------------------------------------
  {violation 2}
  ...
  

• If no violations: print No violations found. Features within expected distribution.

---

fixtures/sample_violation_report.json

{
  "version": 0,
  "violations": [
    {
      "feature_name": "age",
      "constraint_check_type": "distribution_non_parametric_significance",
      "description": "Value at given threshold violation",
      "metric": {
        "type": "threshold",
        "threshold": 0.05,
        "observed_value": 0.0008
      }
    },
    {
      "feature_name": "hours-per-week",
      "constraint_check_type": "distribution_non_parametric_significance",
      "description": "Value at given threshold violation",
      "metric": {
        "type": "threshold",
        "threshold": 0.05,
        "observed_value": 0.0013
      }
    }
  ]
}

---

tests/ — what to test

tests/test_inject_drift.py

test_drifted_values_are_outside_expected_range

• Load UCI Adult data subset
• Compute feature stats for age: known mean ~38.6, std ~13.6
• Call create_drifted_row(feature_stats, std_multiplier=2.0)
• Extract the age feature value from the returned row
• Assert value > mean + 1.5 * std (safely above normal range)
• Assert value == pytest.approx(38.6 + 2.0 * 13.6, abs=0.5)

test_drifted_row_has_correct_feature_count

• Create drifted row using actual training data
• Assert shape matches expected number of features after OHE (same as p3-03 output width)

tests/test_read_violations.py

test_parse_violation_returns_plain_english

• Load fixtures/sample_violation_report.json
• Call parse_monitoring_report(report)
• Assert len(result) == 2
• Assert "age" in result[0]
• Assert "0.0008" in result[0] (observed p-value appears in output)

test_parse_violation_mentions_expected_threshold

• Same fixture
• Assert "0.05" in result[0] (expected threshold appears)

test_enable_capture_config_has_correct_s3_path

• Build the DataCaptureConfig dict using enable_capture.py's logic (extract as a helper)
• Assert config["DestinationS3Uri"] starts with "s3://"
• Assert config["InitialSamplingPercentage"] == 100

test_baseline_json_structure

• Load fixtures/sample_violation_report.json (which is the monitoring output format)
• Assert top-level keys include "violations"
• Assert each violation has "feature_name" and "metric" keys

---

README.md content

# P3-10 — Model Monitoring

SageMaker Model Monitor: data capture, baseline, drift injection, violation reports.

## Prerequisites

- Python 3.11+
- AWS credentials configured
- p3-07 endpoint running (check: `aws sagemaker describe-endpoint --endpoint-name p3-07-adult-income-endpoint`)

## Quick start

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env  # fill in S3_BUCKET, SAGEMAKER_ROLE_ARN, ENDPOINT_NAME

# Enable data capture
python src/enable_capture.py --endpoint-name p3-07-adult-income-endpoint

# Compute baseline from training data
python src/compute_baseline.py --data data/adult.data

# Schedule hourly monitoring
python src/schedule_monitoring.py --endpoint-name p3-07-adult-income-endpoint

# Inject synthetic drift
python src/inject_drift.py --endpoint-name p3-07-adult-income-endpoint --n-requests 100

# Wait up to 1 hour for monitoring to run, then read violations
python src/read_violations.py

Run tests (no AWS needed)

pytest tests/ -v

TEARDOWN

bash scripts/teardown.sh

What drift detection catches and misses

See GUIDE.md for an honest assessment.

---

## GUIDE.md content

```markdown
# Guide — P3-10 Model Monitoring

## What SageMaker Model Monitor does

Model Monitor captures every request and response to your endpoint (via DataCaptureConfig),
runs a scheduled analysis job that compares the captured input distribution to your training
data baseline, and writes a violation report if any feature distribution has shifted significantly.

The significance test is a non-parametric test (Kolmogorov-Smirnov for numeric features).
The p-value threshold (default 0.05) determines what counts as significant drift.

## What drift detection catches

- **Numeric feature distribution shifts:** If `age` values in production suddenly skew much older
  (e.g., your app went viral with seniors), the KS test will catch this.
- **Missing value rate changes:** If a data pipeline bug starts producing nulls in a field that
  had none during training, Model Monitor flags it.
- **Categorical value frequency shifts:** If `workclass` was 60% Private during training and
  is now 20%, the chi-squared test will flag it.

## What drift detection misses

- **Label drift (concept drift):** The income distribution in the real world may change over time —
  what predicted ">50K" during a boom year may predict "<=50K" during a recession. Model Monitor
  cannot detect this because it only monitors inputs, not outcomes.
- **Subtle multivariate shifts:** A feature may have no marginal distribution change but correlate
  differently with other features. Univariate tests miss this.
- **Silent model degradation without input drift:** If the world changed but your input features
  happen to have the same distribution, no alarm fires. This is why p3-09's live accuracy evaluation
  is necessary alongside Model Monitor.
- **Sampling bias:** If DataCapture records only 10% of traffic, a small drift may not be detected
  until it becomes severe.

## The hourly schedule limitation

Model Monitor runs as a batch job at most once per hour (SageMaker's minimum cron granularity).
For a low-traffic endpoint, there may not be enough captured data in one hour to run a reliable
statistical test. SageMaker will still run the job but may produce inconclusive results.

## Reading violation p-values

A p-value of 0.0008 means: if the production distribution were the same as training, you would
observe this large a difference less than 0.08% of the time. This is strong evidence of drift.
A p-value of 0.04 means only 4% chance — still below 0.05 threshold, but weaker evidence.

## Interview framing

"I set up SageMaker Model Monitor to detect input data drift: enabled data capture at the endpoint,
computed a statistical baseline from training data, and scheduled hourly monitoring. I also know
its limitations — it monitors input distributions, not model accuracy, so it won't catch concept
drift or subtle multivariate shifts. That's why I pair it with the live accuracy evaluation from
the performance report."

---

Failure modes to handle

Failure	Where	How to handle
Endpoint update takes too long during capture enable	enable_capture.py	Waiter with 20-min timeout; if exceeded, check if a previous update is still in progress
Baseline job fails (not enough data)	compute_baseline.py	Catch exception from suggest_baseline, print: "Ensure training CSV has at least 50 rows"
Monitoring job hasn't run yet	read_violations.py	S3 list returns empty; print: "No monitoring report yet. Wait up to 1 hour after schedule creation."
Monitoring violation JSON has unexpected structure	read_violations.py	Validate "violations" key exists; if not, print raw JSON and ask user to report issue
Drifted features don't trigger violation	inject_drift.py note	Mean + 2*std is the injection target but KS test may require more samples — use --n-requests 500 if needed
update_endpoint fails (can't update while pending)	enable_capture.py	Check endpoint status before updating; wait for InService first

---

The metric this project measures

What: Number of features flagged with distribution violations by the monitoring job

Format: N violation(s) found: [age (p=0.0008), hours-per-week (p=0.0013)]

Target: After drift injection (mean + 2*std), at least 1 numeric feature (typically age, hours-per-week, capital-gain) should show a statistically significant violation (p < 0.05)

---

Cost estimate

Resource	Qty	Rate	Estimated cost
SageMaker Model Monitor job (ml.m5.large, ~5 min each)	2-3 runs	$0.115/hr	~$0.02
SageMaker Baseline suggestion job (ml.m5.large, ~10 min)	1 run	$0.115/hr	~$0.02
S3 capture data (~100 requests)	< 1 MB	$0.023/GB-month	<$0.01
Total per session			~$0.15

Teardown checklist

scripts/teardown.sh must (in order):

• Stop monitoring schedule: aws sagemaker stop-monitoring-schedule --monitoring-schedule-name ${MONITORING_SCHEDULE_NAME}
• Delete monitoring schedule: aws sagemaker delete-monitoring-schedule --monitoring-schedule-name ${MONITORING_SCHEDULE_NAME}
• Delete S3 capture data: aws s3 rm ${S3_CAPTURE_PATH} --recursive
• Delete S3 baseline data: aws s3 rm ${S3_BASELINE_PATH} --recursive
• Delete S3 monitoring results: aws s3 rm ${S3_MONITORING_RESULTS_PATH} --recursive
• Print Monitoring resources deleted.
• Note: endpoint itself is managed by p3-07's teardown