--- title: "Model Monitoring" description: "Hands-on experience with SageMaker Model Monitor — baseline, data capture, drift injection, violation reports — is a named, screenable skill on most ML Engineer..." source: "https://confidentprep.com/paths/ml-engineering-on-aws/model-monitoring/" path: "ML Engineering on AWS" repository: "https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/model-monitoring" token_estimate: 5097 --- # Model Monitoring ## Overview SageMaker Model Monitor: data capture on the live endpoint, a baseline from training data, deliberately injected drift, and the resulting violation reports. This is the project that proves the model from project 7 will not just degrade silently — and it directly feeds the retraining trigger in the next project. ## What to do **Path:** 3 — ML Engineering on AWS **Position:** 10 of 12 **Difficulty:** 🔴 **Time:** 3-4h **AWS Cost:** ~$0.05/monitoring run × 2-3 runs = ~$0.15. Plus endpoint from p3-07 (~$0.056/hr). --- ### Agent Pickup Instructions ```bash # Bootstrap cd projects/confident-prep/p3-10-model-monitoring python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt # Prerequisite: p3-07 endpoint must be running # Check: aws sagemaker describe-endpoint --endpoint-name p3-07-adult-income-endpoint # Step 1: Enable data capture on existing endpoint python src/enable_capture.py --endpoint-name p3-07-adult-income-endpoint # Step 2: Compute baseline statistics from training data python src/compute_baseline.py --data data/adult.data # Step 3: Schedule monitoring (hourly) python src/schedule_monitoring.py --endpoint-name p3-07-adult-income-endpoint # Step 4: Inject synthetic drift python src/inject_drift.py --endpoint-name p3-07-adult-income-endpoint --n-requests 100 # Step 5: Read violation report (after monitoring job runs — may take up to 1 hour) python src/read_violations.py # Run tests pytest tests/ -v # TEARDOWN bash scripts/teardown.sh ``` **Done when:** - [ ] `python src/enable_capture.py` updates endpoint to capture 100% of requests - [ ] `python src/compute_baseline.py` runs `suggest_baseline()` and saves baseline statistics JSON to S3 - [ ] `python src/schedule_monitoring.py` creates a monitoring schedule (ARN saved to `.schedule-arn`) - [ ] `python src/inject_drift.py` sends 100 requests with shifted feature values - [ ] `python src/read_violations.py` parses monitoring report and prints human-readable violations - [ ] `pytest tests/ -v` shows 4/4 passing - [ ] `bash scripts/teardown.sh` stops schedule, deletes schedule, deletes S3 capture + baseline data --- ### What this project is You enable SageMaker Model Monitor on the endpoint from p3-07: configure data capture (logging all requests and responses to S3), compute a baseline using `DefaultModelMonitor.suggest_baseline()` on the training data, and schedule hourly monitoring. Then you inject synthetic drift by sending requests where numeric features are shifted to 2 standard deviations above their training mean — values the monitor has never seen. A violation reader parses the monitoring JSON report and prints plain-English explanations of what drifted and by how much. ### What the learner achieves "I can set up SageMaker Model Monitor to detect data drift, inject synthetic drift to test that the alarm fires, and read violation reports in plain English — and I know what drift detection catches and what it misses." --- ### Folder structure ``` p3-10-model-monitoring/ ├── data/ │ └── adult.data # training data for baseline ├── fixtures/ │ └── sample_violation_report.json # SageMaker monitoring report format for tests ├── src/ │ ├── enable_capture.py # update endpoint to enable DataCaptureConfig (100% capture) │ ├── compute_baseline.py # run suggest_baseline() on training data │ ├── schedule_monitoring.py # create hourly monitoring schedule │ ├── inject_drift.py # send 100 requests with features at mean + 2*std │ └── read_violations.py # parse monitoring report JSON, print plain-English violations ├── scripts/ │ └── teardown.sh # stop schedule, delete schedule, delete S3 capture + baseline ├── tests/ │ ├── test_inject_drift.py # verify shifted values are out of expected range │ └── test_read_violations.py # parse fixture JSON, verify output format ├── .schedule-arn # written by schedule_monitoring.py ├── .env.example ├── requirements.txt └── README.md ``` --- ### .env.example ```dotenv # AWS region AWS_REGION=us-east-1 # S3 bucket S3_BUCKET=my-ml-training-bucket # IAM role ARN with SageMaker full access SAGEMAKER_ROLE_ARN=arn:aws:iam::123456789012:role/SageMakerExecutionRole # Endpoint name (must match p3-07) ENDPOINT_NAME=p3-07-adult-income-endpoint # S3 paths for capture data and baseline S3_CAPTURE_PATH=s3://my-ml-training-bucket/p3-10/capture/ S3_BASELINE_PATH=s3://my-ml-training-bucket/p3-10/baseline/ S3_MONITORING_RESULTS_PATH=s3://my-ml-training-bucket/p3-10/monitoring-results/ # Monitoring schedule name MONITORING_SCHEDULE_NAME=p3-10-hourly-monitor # Drift injection: number of std deviations above mean DRIFT_STD_MULTIPLIER=2.0 ``` --- ### requirements.txt ``` boto3==1.35.93 sagemaker==2.236.0 pandas==2.2.3 numpy==2.2.1 python-dotenv==1.0.1 pytest==8.3.4 ``` --- ### src/ — what to implement #### `src/enable_capture.py` **CLI:** `python src/enable_capture.py --endpoint-name ` **`enable_data_capture(endpoint_name: str, s3_capture_path: str, capture_percentage: int = 100) -> None`** Creates a new EndpointConfig with DataCaptureConfig enabled and updates the endpoint. DataCaptureConfig structure: ```python { "EnableCapture": True, "InitialSamplingPercentage": capture_percentage, "DestinationS3Uri": s3_capture_path, "CaptureOptions": [ {"CaptureMode": "Input"}, {"CaptureMode": "Output"}, ], "CaptureContentTypeHeader": { "CsvContentTypes": ["text/csv"], } } ``` Steps: 1. Describe current endpoint to get current `EndpointConfigName` 2. Describe current config to copy its `ProductionVariants` 3. Create a new EndpointConfig with the same variants + DataCaptureConfig 4. Call `update_endpoint(EndpointName=endpoint_name, EndpointConfigName=new_config_name)` 5. Wait for endpoint to return to `InService` using waiter **`verify_capture_enabled(endpoint_name: str) -> bool`** - Describes the current endpoint config - Returns True if `DataCaptureConfig.EnableCapture == True` **`main()`** - `enable_data_capture` → `verify_capture_enabled` - Print: `Data capture enabled. 100% of requests will be logged to {s3_capture_path}` --- #### `src/compute_baseline.py` **CLI:** `python src/compute_baseline.py --data [--instance-type ml.m5.large]` **`prepare_baseline_data(data_path: str, output_s3_uri: str) -> str`** - Loads training data (UCI Adult), preprocesses (same as p3-01: column names, strip, NaN drop, get_dummies) - Saves feature-only CSV (no target column) to S3 for baseline computation - Returns S3 URI of uploaded file **`run_baseline_suggestion(input_s3_uri: str, output_s3_uri: str, role_arn: str, instance_type: str) -> str`** - Creates `sagemaker.model_monitor.DefaultModelMonitor(role=role_arn, instance_type=instance_type, ...)` - Calls `monitor.suggest_baseline(baseline_dataset=input_s3_uri, dataset_format=DatasetFormat.csv(), output_s3_uri=output_s3_uri, wait=True)` - Returns the S3 URI where baseline statistics were saved **`main()`** 1. `prepare_baseline_data` → uploads training CSV to `S3_BASELINE_PATH/data/` 2. `run_baseline_suggestion` → runs the baseline job 3. Print: `Baseline computed. Statistics at: {output_s3_uri}` 4. Print statistics summary: number of features monitored --- #### `src/schedule_monitoring.py` **CLI:** `python src/schedule_monitoring.py --endpoint-name ` **`create_monitoring_schedule(endpoint_name: str, schedule_name: str, baseline_s3_uri: str, output_s3_uri: str, role_arn: str) -> str`** Creates a monitoring schedule using `sagemaker.model_monitor.DefaultModelMonitor`: ```python monitor = DefaultModelMonitor(role=role_arn, ...) monitor.create_monitoring_schedule( monitor_schedule_name=schedule_name, endpoint_input=endpoint_name, output_s3_uri=output_s3_uri, statistics=baseline_s3_uri + "/statistics.json", constraints=baseline_s3_uri + "/constraints.json", schedule_cron_expression="cron(0 * ? * * *)", # hourly ) ``` Returns the schedule ARN from `monitor.monitoring_schedule_arn`. **`main()`** - Create schedule - Write ARN to `.schedule-arn` - Print: `Monitoring schedule created: {schedule_name}` - Print: `Schedule ARN: {arn}` - Print: `Note: first monitoring report will be generated in up to 1 hour.` --- #### `src/inject_drift.py` **CLI:** `python src/inject_drift.py --endpoint-name [--n-requests 100]` **`compute_feature_stats(data_path: str) -> dict`** - Loads UCI Adult training data - Computes mean and std for each numeric feature - Returns `{feature_name: {"mean": float, "std": float}}` for numeric features only **`create_drifted_row(feature_stats: dict, std_multiplier: float = 2.0) -> np.ndarray`** - For each numeric feature: set value to `mean + std_multiplier * std` - For categorical features: use the most common value from training data (no drift for categoricals) - Returns a 1D numpy array in the same column order as the OHE feature space **`send_drifted_requests(endpoint_name: str, drifted_row: np.ndarray, n_requests: int) -> dict`** - Converts `drifted_row` to CSV string - Sends `n_requests` invocations to endpoint - Returns `{"sent": int, "succeeded": int, "failed": int}` **`main()`** 1. `compute_feature_stats` 2. `create_drifted_row` (with `DRIFT_STD_MULTIPLIER` from env, default 2.0) 3. Print: `Injecting drift: {n_requests} requests with features at mean + {multiplier}*std` 4. Print per-feature drift values: `age: expected range [17, 90], injecting 97.3` 5. `send_drifted_requests` 6. Print: `Drift injection complete. {succeeded}/{sent} requests succeeded.` --- #### `src/read_violations.py` **CLI:** `python src/read_violations.py [--s3-path ] [--local-file ]` **`fetch_latest_report(monitoring_results_s3: str) -> dict`** - Lists S3 objects under `monitoring_results_s3` prefix - Finds the most recently modified JSON file - Downloads and parses it - Returns the parsed dict **`parse_violation(violation: dict) -> str`** Converts a single SageMaker monitoring violation dict to a plain-English string. SageMaker violation format: ```json { "feature_name": "age", "constraint_check_type": "distribution_non_parametric_significance", "description": "Value at given threshold violation", "metric": {"type": "threshold", "threshold": 0.05, "observed_value": 0.001} } ``` Plain-English output format: ``` Feature 'age': distribution shifted significantly. Test: non-parametric significance test Expected p-value > 0.05, observed p-value = 0.001 Interpretation: The distribution of 'age' in recent traffic is unlikely to match training data. ``` **`parse_monitoring_report(report: dict) -> list[str]`** - Extracts the `violations` list from the report (path: `report["violations"]`) - Calls `parse_violation()` for each - Returns list of plain-English strings **`main()`** - Fetch report from S3 or `--local-file` - Parse violations - Print: ``` Model Monitor Report — {n_violations} violation(s) found ======================================== {violation 1} ---------------------------------------- {violation 2} ... ``` - If no violations: print `No violations found. Features within expected distribution.` --- ### fixtures/sample_violation_report.json ```json { "version": 0, "violations": [ { "feature_name": "age", "constraint_check_type": "distribution_non_parametric_significance", "description": "Value at given threshold violation", "metric": { "type": "threshold", "threshold": 0.05, "observed_value": 0.0008 } }, { "feature_name": "hours-per-week", "constraint_check_type": "distribution_non_parametric_significance", "description": "Value at given threshold violation", "metric": { "type": "threshold", "threshold": 0.05, "observed_value": 0.0013 } } ] } ``` --- ### tests/ — what to test #### `tests/test_inject_drift.py` **`test_drifted_values_are_outside_expected_range`** - Load UCI Adult data subset - Compute feature stats for `age`: known mean ~38.6, std ~13.6 - Call `create_drifted_row(feature_stats, std_multiplier=2.0)` - Extract the `age` feature value from the returned row - Assert value > `mean + 1.5 * std` (safely above normal range) - Assert value == pytest.approx(38.6 + 2.0 * 13.6, abs=0.5) **`test_drifted_row_has_correct_feature_count`** - Create drifted row using actual training data - Assert shape matches expected number of features after OHE (same as p3-03 output width) #### `tests/test_read_violations.py` **`test_parse_violation_returns_plain_english`** - Load `fixtures/sample_violation_report.json` - Call `parse_monitoring_report(report)` - Assert len(result) == 2 - Assert "age" in result[0] - Assert "0.0008" in result[0] (observed p-value appears in output) **`test_parse_violation_mentions_expected_threshold`** - Same fixture - Assert "0.05" in result[0] (expected threshold appears) **`test_enable_capture_config_has_correct_s3_path`** - Build the DataCaptureConfig dict using `enable_capture.py`'s logic (extract as a helper) - Assert `config["DestinationS3Uri"]` starts with `"s3://"` - Assert `config["InitialSamplingPercentage"] == 100` **`test_baseline_json_structure`** - Load `fixtures/sample_violation_report.json` (which is the monitoring output format) - Assert top-level keys include `"violations"` - Assert each violation has `"feature_name"` and `"metric"` keys --- ### README.md content ```markdown # P3-10 — Model Monitoring SageMaker Model Monitor: data capture, baseline, drift injection, violation reports. ## Prerequisites - Python 3.11+ - AWS credentials configured - p3-07 endpoint running (check: `aws sagemaker describe-endpoint --endpoint-name p3-07-adult-income-endpoint`) ## Quick start ```bash python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt cp .env.example .env # fill in S3_BUCKET, SAGEMAKER_ROLE_ARN, ENDPOINT_NAME ## Enable data capture python src/enable_capture.py --endpoint-name p3-07-adult-income-endpoint ## Compute baseline from training data python src/compute_baseline.py --data data/adult.data ## Schedule hourly monitoring python src/schedule_monitoring.py --endpoint-name p3-07-adult-income-endpoint ## Inject synthetic drift python src/inject_drift.py --endpoint-name p3-07-adult-income-endpoint --n-requests 100 ## Wait up to 1 hour for monitoring to run, then read violations python src/read_violations.py ``` ## Run tests (no AWS needed) ```bash pytest tests/ -v ``` ## TEARDOWN ```bash bash scripts/teardown.sh ``` ## What drift detection catches and misses See GUIDE.md for an honest assessment. ``` --- ### GUIDE.md content ```markdown # Guide — P3-10 Model Monitoring ## What SageMaker Model Monitor does Model Monitor captures every request and response to your endpoint (via DataCaptureConfig), runs a scheduled analysis job that compares the captured input distribution to your training data baseline, and writes a violation report if any feature distribution has shifted significantly. The significance test is a non-parametric test (Kolmogorov-Smirnov for numeric features). The p-value threshold (default 0.05) determines what counts as significant drift. ## What drift detection catches - **Numeric feature distribution shifts:** If `age` values in production suddenly skew much older (e.g., your app went viral with seniors), the KS test will catch this. - **Missing value rate changes:** If a data pipeline bug starts producing nulls in a field that had none during training, Model Monitor flags it. - **Categorical value frequency shifts:** If `workclass` was 60% Private during training and is now 20%, the chi-squared test will flag it. ## What drift detection misses - **Label drift (concept drift):** The income distribution in the real world may change over time — what predicted ">50K" during a boom year may predict "<=50K" during a recession. Model Monitor cannot detect this because it only monitors inputs, not outcomes. - **Subtle multivariate shifts:** A feature may have no marginal distribution change but correlate differently with other features. Univariate tests miss this. - **Silent model degradation without input drift:** If the world changed but your input features happen to have the same distribution, no alarm fires. This is why p3-09's live accuracy evaluation is necessary alongside Model Monitor. - **Sampling bias:** If DataCapture records only 10% of traffic, a small drift may not be detected until it becomes severe. ## The hourly schedule limitation Model Monitor runs as a batch job at most once per hour (SageMaker's minimum cron granularity). For a low-traffic endpoint, there may not be enough captured data in one hour to run a reliable statistical test. SageMaker will still run the job but may produce inconclusive results. ## Reading violation p-values A p-value of 0.0008 means: if the production distribution were the same as training, you would observe this large a difference less than 0.08% of the time. This is strong evidence of drift. A p-value of 0.04 means only 4% chance — still below 0.05 threshold, but weaker evidence. ## Interview framing "I set up SageMaker Model Monitor to detect input data drift: enabled data capture at the endpoint, computed a statistical baseline from training data, and scheduled hourly monitoring. I also know its limitations — it monitors input distributions, not model accuracy, so it won't catch concept drift or subtle multivariate shifts. That's why I pair it with the live accuracy evaluation from the performance report." ``` --- ### Failure modes to handle | Failure | Where | How to handle | |---|---|---| | Endpoint update takes too long during capture enable | `enable_capture.py` | Waiter with 20-min timeout; if exceeded, check if a previous update is still in progress | | Baseline job fails (not enough data) | `compute_baseline.py` | Catch exception from `suggest_baseline`, print: "Ensure training CSV has at least 50 rows" | | Monitoring job hasn't run yet | `read_violations.py` | S3 list returns empty; print: "No monitoring report yet. Wait up to 1 hour after schedule creation." | | Monitoring violation JSON has unexpected structure | `read_violations.py` | Validate `"violations"` key exists; if not, print raw JSON and ask user to report issue | | Drifted features don't trigger violation | `inject_drift.py` note | Mean + 2*std is the injection target but KS test may require more samples — use `--n-requests 500` if needed | | `update_endpoint` fails (can't update while pending) | `enable_capture.py` | Check endpoint status before updating; wait for InService first | --- ### The metric this project measures **What:** Number of features flagged with distribution violations by the monitoring job **Format:** `N violation(s) found: [age (p=0.0008), hours-per-week (p=0.0013)]` **Target:** After drift injection (mean + 2*std), at least 1 numeric feature (typically `age`, `hours-per-week`, `capital-gain`) should show a statistically significant violation (p < 0.05) --- ### Cost estimate | Resource | Qty | Rate | Estimated cost | |---|---|---|---| | SageMaker Model Monitor job (ml.m5.large, ~5 min each) | 2-3 runs | $0.115/hr | ~$0.02 | | SageMaker Baseline suggestion job (ml.m5.large, ~10 min) | 1 run | $0.115/hr | ~$0.02 | | S3 capture data (~100 requests) | < 1 MB | $0.023/GB-month | <$0.01 | | **Total per session** | | | **~$0.15** | ### Teardown checklist `scripts/teardown.sh` must (in order): - [ ] Stop monitoring schedule: `aws sagemaker stop-monitoring-schedule --monitoring-schedule-name ${MONITORING_SCHEDULE_NAME}` - [ ] Delete monitoring schedule: `aws sagemaker delete-monitoring-schedule --monitoring-schedule-name ${MONITORING_SCHEDULE_NAME}` - [ ] Delete S3 capture data: `aws s3 rm ${S3_CAPTURE_PATH} --recursive` - [ ] Delete S3 baseline data: `aws s3 rm ${S3_BASELINE_PATH} --recursive` - [ ] Delete S3 monitoring results: `aws s3 rm ${S3_MONITORING_RESULTS_PATH} --recursive` - [ ] Print `Monitoring resources deleted.` - [ ] Note: endpoint itself is managed by p3-07's teardown ## Source code A reference implementation of Model Monitoring — README, guide, source and tests — is at https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/model-monitoring.