--- title: "MLOps Capstone" description: "Wiring serving, A/B testing, reporting, monitoring, and retraining into one observable system is the single strongest portfolio piece for an ML Engineer or..." source: "https://confidentprep.com/paths/ml-engineering-on-aws/capstone-mlops/" path: "ML Engineering on AWS" repository: "https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/capstone-mlops" token_estimate: 6459 --- # MLOps Capstone ## Overview Wires projects 7 through 11 — serving, A/B testing, performance reporting, monitoring, and the retraining pipeline — into one complete, observable MLOps system. Running everything at once costs roughly $5-10 for a two-hour session; tear it all down immediately after with the provided script. Nothing here is new — it is everything else in this path, connected. ## What to do **Path:** 3 — ML Engineering on AWS **Position:** 12 of 12 **Difficulty:** 🔴 (full AWS — all components from p3-07 through p3-11) **Time:** 4-6h **AWS Cost:** ~$5-10 (all components running simultaneously for ~2 hours) --- ### Agent Pickup Instructions ```bash # Bootstrap cd projects/confident-prep/p3-12-capstone python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt # Prerequisites: p3-07 through p3-11 must all be deployed # Verify components first: python src/verify_components.py # Run end-to-end test python src/end_to_end_test.py # View cost dashboard python src/cost_dashboard.py # Review runbook cat docs/runbook.md # COMPLETE TEARDOWN OF ALL COMPONENTS bash scripts/teardown_all.sh ``` **Done when:** - [ ] `python src/verify_components.py` prints ARN/name of endpoint, monitoring schedule, state machine, and Lambda — no errors - [ ] `python src/end_to_end_test.py` completes: drift injected → alarm checked → pipeline triggered → execution watched → endpoint predictions verified - [ ] `python src/cost_dashboard.py` prints itemised cost table for SageMaker, Lambda, Step Functions, S3, SNS, and a total - [ ] `docs/architecture.md` exists with a Mermaid diagram showing the full loop - [ ] `docs/runbook.md` exists with at least 4 failure scenarios and resolution steps - [ ] `pytest tests/ -v` shows 4/4 passing - [ ] `bash scripts/teardown_all.sh` removes every resource and prints confirmation --- ### What this project is The capstone wires p3-07 (real-time endpoint) through p3-11 (retraining pipeline) into a single observable MLOps system. You verify all components are live, run an orchestrated end-to-end test (inject drift → check alarm → trigger retraining → watch pipeline execute → verify new endpoint predictions), view an itemised cost dashboard from AWS Cost Explorer, and prove the system works as a unit. The teardown script removes every resource created across the entire path in the correct dependency order. ### What the learner achieves "I have built and operated a complete MLOps loop on AWS: model serving, A/B testing, data capture, drift monitoring, automated retraining, and conditional deployment — and I can articulate what each component does, what it costs, and what to do when each one fails." --- ### Folder structure ``` p3-12-capstone/ ├── src/ │ ├── verify_components.py # check endpoint, monitoring, state machine, Lambda are all live │ ├── end_to_end_test.py # orchestrate full test: drift → alarm → pipeline → verify │ └── cost_dashboard.py # AWS Cost Explorer: past 24h by service ├── scripts/ │ └── teardown_all.sh # ordered teardown of every resource from p3-07 through p3-12 ├── docs/ │ ├── architecture.md # Mermaid diagram of full MLOps loop │ └── runbook.md # what to do when each component fails ├── tests/ │ ├── test_verify.py # clear error when component missing, cost zero handling │ └── test_e2e.py # endpoint predictions after pipeline, cost table format ├── .env.example ├── requirements.txt └── README.md ``` --- ### .env.example ```dotenv # AWS region AWS_REGION=us-east-1 # Cross-project resource names (must match prior projects) ENDPOINT_NAME=p3-07-adult-income-endpoint ENDPOINT_CONFIG_NAME=p3-07-adult-income-config SAGEMAKER_MODEL_NAME=p3-07-adult-income-model MONITORING_SCHEDULE_NAME=p3-10-hourly-monitor STATE_MACHINE_ARN_FILE=../p3-11-retraining-pipeline/.state-machine-arn LAMBDA_EVALUATOR_NAME=p3-11-evaluate-and-deploy SNS_TOPIC_ARN=arn:aws:sns:us-east-1:123456789012:p3-11-ml-notifications S3_BUCKET=my-ml-training-bucket MLFLOW_TRACKING_URI=http://localhost:5000 MLFLOW_MODEL_NAME=adult-income-xgboost # Cost Explorer: AWS account ID AWS_ACCOUNT_ID=123456789012 ``` --- ### requirements.txt ``` boto3==1.35.93 sagemaker==2.236.0 xgboost==2.1.3 mlflow==2.19.0 pandas==2.2.3 numpy==2.2.1 python-dotenv==1.0.1 pytest==8.3.4 ``` --- ### src/ — what to implement #### `src/verify_components.py` **CLI:** `python src/verify_components.py` **`check_endpoint(endpoint_name: str, region: str) -> dict`** - Calls `boto3.client('sagemaker').describe_endpoint(EndpointName=endpoint_name)` - Returns `{"name": str, "status": str, "ok": bool}` where `ok` is True iff status == "InService" - Catches `ClientError` with code `ValidationException` — returns `{"name": endpoint_name, "status": "NOT_FOUND", "ok": False}` **`check_monitoring_schedule(schedule_name: str, region: str) -> dict`** - Calls `describe_monitoring_schedule` - Returns `{"name": str, "status": str, "ok": bool}` where `ok` is True iff status == "Scheduled" **`check_state_machine(arn: str, region: str) -> dict`** - Calls `boto3.client('stepfunctions').describe_state_machine(stateMachineArn=arn)` - Returns `{"arn": str, "name": str, "status": str, "ok": bool}` **`check_lambda(function_name: str, region: str) -> dict`** - Calls `boto3.client('lambda').get_function(FunctionName=function_name)` - Returns `{"name": str, "runtime": str, "ok": bool}` **`verify_all(config: dict) -> dict`** - Calls all four check functions - Returns `{"endpoint": dict, "monitoring": dict, "state_machine": dict, "lambda": dict, "all_ok": bool}` - `all_ok` is True only if all four `ok` fields are True **`print_component_table(result: dict) -> None`** - Prints: ``` Component Status ============================================ Endpoint: p3-07-adult-income-endpoint [OK / MISSING] Monitoring: p3-10-hourly-monitor [OK / MISSING] State Machine: p3-11-ml-pipeline [OK / MISSING] Lambda: p3-11-evaluate-and-deploy [OK / MISSING] -------------------------------------------- Overall: ALL READY / COMPONENTS MISSING ``` **`main()`** - `verify_all` → `print_component_table` - If `all_ok == False`: print `Run the setup scripts for any MISSING components.`; exit 1 - If `all_ok == True`: print `All components live. Ready to run end_to_end_test.py`; exit 0 --- #### `src/end_to_end_test.py` **CLI:** `python src/end_to_end_test.py [--skip-drift] [--skip-alarm] [--timeout-minutes 15]` Orchestrates the following sequence, printing `[STEP N/5] description...` before each step: **`inject_drift(endpoint_name: str, data_path: str, n_requests: int = 50) -> dict`** - Reuses p3-10 inject_drift logic: send requests with features at mean + 2*std - Returns `{"sent": int, "succeeded": int}` **`check_alarm_status(endpoint_name: str, region: str) -> dict`** - Calls `boto3.client('cloudwatch').describe_alarms(AlarmNamePrefix=endpoint_name)` - Returns list of alarms with their state (`OK`, `ALARM`, `INSUFFICIENT_DATA`) - Does NOT wait for alarm to fire — just reports current status - Prints warning if state is `INSUFFICIENT_DATA`: "Monitoring may not have run yet" **`trigger_pipeline(state_machine_arn: str, input_data: dict) -> str`** - Starts a Step Functions execution with sample input - Returns execution ARN **`wait_for_execution(execution_arn: str, timeout_minutes: int = 15) -> dict`** - Polls execution status - Prints each state transition with timestamp: `[14:32:01] Preprocess → SUCCEEDED` - Returns final status dict - Times out after `timeout_minutes` with a clear message **`verify_endpoint_predictions(endpoint_name: str, data_path: str, n_samples: int = 5) -> dict`** - Sends 5 inference requests to the endpoint - Returns `{"predictions": list[float], "all_valid": bool}` where `all_valid` checks all are in [0,1] **`main()`** ``` [STEP 1/5] Verifying all components are live... [STEP 2/5] Injecting synthetic drift (50 requests, mean + 2*std)... [STEP 3/5] Checking CloudWatch alarm status... [STEP 4/5] Triggering retraining pipeline... [14:32:01] Preprocess → RUNNING [14:36:22] Preprocess → SUCCEEDED [14:36:23] Train → RUNNING [14:40:51] Train → SUCCEEDED [14:40:52] Evaluate → RUNNING [14:40:54] Evaluate → SUCCEEDED [14:40:55] ShouldDeploy → (branch: deploy) [14:40:55] NotifyDeployed → SUCCEEDED [STEP 5/5] Verifying endpoint returns valid predictions... End-to-end test PASSED. Total time: {elapsed:.1f}s ``` - Write results to `output/e2e_results.json` --- #### `src/cost_dashboard.py` **CLI:** `python src/cost_dashboard.py [--days 1]` **`fetch_cost_by_service(account_id: str, days: int, region: str) -> list[dict]`** - Calls `boto3.client('ce', region_name='us-east-1').get_cost_and_usage()` - `TimePeriod`: last `days` days - `Granularity`: DAILY - `GroupBy`: `[{"Type": "DIMENSION", "Key": "SERVICE"}]` - `Metrics`: `["UnblendedCost"]` - Returns list of `{"service": str, "cost_usd": float}` sorted by cost descending **`print_cost_table(costs: list[dict]) -> None`** - Prints: ``` AWS Cost Dashboard — Last 24 hours ==================================== Service Cost (USD) ------------------------------------ Amazon SageMaker $0.4230 Amazon S3 $0.0012 AWS Lambda $0.0000 AWS Step Functions $0.0000 Amazon SNS $0.0000 Amazon CloudWatch $0.0451 ------------------------------------ TOTAL $0.4693 ``` - Services not in the result are shown as `$0.0000` - Always shows these 6 rows even if cost is zero **Edge cases:** - If Cost Explorer API not enabled: print `Cost Explorer not enabled. Enable it in AWS Billing console.` and return a hardcoded zero table - If `days=1` and it is early in the day, results may show only partial cost — print a note --- ### docs/architecture.md content ```markdown # Architecture — Path 3 MLOps System ## Component Map ```mermaid graph LR A[UCI Adult Data\nS3 Bucket] -->|upload triggers| B[S3 Event\nNotification] B --> C[Lambda Trigger] C --> D[Step Functions\nState Machine] D --> E[SageMaker\nProcessing Job\np3-03 Pipeline] E --> F[SageMaker\nTraining Job\nXGBoost] F --> G[Lambda Evaluator\nAUC vs SSM Baseline] G -->|AUC improved > 1%| H[Deploy State\nUpdate Endpoint] G -->|AUC not improved| I[SNS: No Deploy\nNotification] H --> J[SNS: Deployed\nNotification] K[SageMaker\nEndpoint\np3-07] -->|captures requests| L[S3 Capture\nData] L --> M[Model Monitor\nScheduled Job\np3-10] M -->|violation| N[CloudWatch\nAlarm] K -->|inference| O[Users / Batch Jobs] P[MLflow Registry\np3-05] -->|production alias| K H -->|update alias| P Q[Weekly Report\np3-09] -->|AUC delta| N ``` ## Data Flow 1. **Training trigger:** New data uploaded to S3 → S3 notification → Lambda → Step Functions execution 2. **Preprocessing:** SageMaker Processing Job applies the sklearn pipeline (p3-03) 3. **Training:** SageMaker Training Job on preprocessed data → model artifact to S3 4. **Evaluation:** Lambda reads new AUC from CloudWatch, compares to SSM baseline 5. **Conditional deploy:** If AUC improves > 1%, update endpoint; else notify and skip 6. **Monitoring:** Model Monitor captures requests, computes drift vs training baseline, fires alarm 7. **Reporting:** Weekly report (p3-09) computes live accuracy vs MLflow baseline, surfaces delta ## Component Registry | Component | Project | AWS Service | Resource Name | |---|---|---|---| | Real-time endpoint | p3-07 | SageMaker Endpoint | p3-07-adult-income-endpoint | | Monitoring schedule | p3-10 | SageMaker Model Monitor | p3-10-hourly-monitor | | Retraining pipeline | p3-11 | Step Functions | p3-11-ml-pipeline | | Evaluator Lambda | p3-11 | Lambda | p3-11-evaluate-and-deploy | | Experiment registry | p3-05 | MLflow (local) | adult-income-xgboost | ``` --- ### docs/runbook.md content ```markdown # Operational Runbook — Path 3 MLOps System ## Failure Scenario 1: Endpoint is down **Symptom:** `verify_components.py` reports `Endpoint: MISSING` or `status: Failed` **Diagnosis:** ```bash aws sagemaker describe-endpoint --endpoint-name p3-07-adult-income-endpoint aws logs get-log-events --log-group-name /aws/sagemaker/Endpoints/p3-07-adult-income-endpoint \ --log-stream-name $(aws logs describe-log-streams --log-group-name \ /aws/sagemaker/Endpoints/p3-07-adult-income-endpoint --query 'logStreams[-1].logStreamName' \ --output text) ``` **Resolution:** - If status is `Failed`: check logs for container error. Most common: wrong inference.py syntax. - Re-deploy using p3-07's `deploy.py`. Do not delete prior endpoint config until new one is InService. - If endpoint was accidentally deleted: re-run `python src/deploy.py` from p3-07. **Cost impact:** While endpoint is down, no inference cost accrues. Recreating costs a new cold-start. --- ## Failure Scenario 2: Drift alarm fires but retraining pipeline fails **Symptom:** CloudWatch alarm is in ALARM state but Step Functions execution status is FAILED **Diagnosis:** ```bash ## Get most recent execution aws stepfunctions list-executions --state-machine-arn $(cat .state-machine-arn) \ --status-filter FAILED --query 'executions[0].executionArn' --output text ## Get failure details aws stepfunctions get-execution-history --execution-arn \ --query 'events[?type==`TaskFailed`]' ``` **Common causes and resolutions:** - **Preprocess state fails:** Check SageMaker Processing Job logs in CloudWatch. Usually: S3 path wrong, container image URI wrong, or IAM role missing S3 permissions. - **Train state fails:** Check Training Job failure reason: `aws sagemaker describe-training-job --training-job-name `. Usually: data format wrong, out of disk space. - **Evaluate state fails:** Lambda function error. Check Lambda logs: `aws logs tail /aws/lambda/p3-11-evaluate-and-deploy`. - **Deploy state fails:** Usually: endpoint already being updated, or IAM role missing SageMaker permissions. **Resolution:** Fix root cause, then manually start execution: ```bash python src/test_pipeline.py ``` --- ## Failure Scenario 3: Model quality degrades without drift alarm **Symptom:** p3-09's weekly report shows `Delta: -0.04 (regression — WARNING)` but Model Monitor shows no violations **Diagnosis:** This is concept drift — the relationship between features and the label has changed, but the input feature distributions look the same. Model Monitor cannot detect this. **Resolution:** 1. Check the weekly report's live accuracy trend over past 4 weeks 2. If accuracy has been declining for 2+ weeks: trigger manual retraining 3. Collect new labeled data if possible — concept drift usually means the world changed 4. Temporarily lower the AUC improvement threshold in SSM to 0.005 to allow easier deployment of retrained models ```bash aws ssm put-parameter --name /p3-11/baseline/val_auc --value 0.860 --overwrite python src/test_pipeline.py ``` --- ## Failure Scenario 4: AWS costs spike unexpectedly **Symptom:** `cost_dashboard.py` shows SageMaker cost much higher than expected **Diagnosis:** ```bash python src/cost_dashboard.py --days 7 ``` Most likely causes: - Endpoint was not torn down between sessions (p3-07 through p3-10 endpoints accumulate hourly) - Multiple training jobs running simultaneously (Step Functions executing multiple pipelines) - Monitoring schedule running more frequently than expected **Resolution — immediate:** ```bash bash scripts/teardown_all.sh ``` **Resolution — preventive:** - Always run teardown scripts at the end of each session - Set a CloudWatch billing alarm: `aws cloudwatch put-metric-alarm --alarm-name "ML-Cost-Alert" --metric-name EstimatedCharges ...` - Use `aws ce get-cost-and-usage` daily during active development **Expected costs when actively using the system:** - Endpoint idle: ~$0.056/hr (ml.t2.medium) - Training job: ~$0.01 per run (ml.m5.large, 5 min) - Monitoring job: ~$0.01 per run - Step Functions: < $0.01 per execution ``` --- ### tests/ — what to test #### `tests/test_verify.py` **`test_verify_raises_clear_error_if_endpoint_missing`** - Mock `boto3.client('sagemaker').describe_endpoint` to raise `ClientError` with `ValidationException` - Call `check_endpoint("nonexistent-endpoint", "us-east-1")` - Assert `result["ok"] == False` - Assert `result["status"] == "NOT_FOUND"` - Assert no exception is raised (graceful handling) **`test_verify_all_ok_is_false_when_any_component_missing`** - Mock: endpoint OK, monitoring OK, state machine OK, Lambda NOT_FOUND - Call `verify_all(config)` - Assert `result["all_ok"] == False` #### `tests/test_e2e.py` **`test_cost_dashboard_handles_zero_cost_services`** - Mock `get_cost_and_usage` to return data for only 2 of the 6 services (others absent) - Call `print_cost_table(costs)` - Assert output contains all 6 service names - Assert missing services show `$0.0000` **`test_endpoint_predictions_are_valid_probabilities`** - Mock `invoke_endpoint` to return `"0.75"` body - Call `verify_endpoint_predictions("mock-endpoint", "data/adult.data", n_samples=3)` with mocked runtime - Assert `result["all_valid"] == True` - Assert all predictions are in `[0.0, 1.0]` **`test_runbook_has_at_least_four_failure_scenarios`** - Open `docs/runbook.md` - Count occurrences of `## Failure Scenario` - Assert count >= 4 **`test_architecture_md_has_mermaid_diagram`** - Open `docs/architecture.md` - Assert `"```mermaid"` in content - Assert `"graph"` in content --- ### README.md content ```markdown # P3-12 — Capstone: Full MLOps Loop Wire p3-07 through p3-11 into a complete, observable MLOps system. ## WARNING This project runs ALL components simultaneously. Combined cost: ~$5-10 for a 2-hour session. Run `bash scripts/teardown_all.sh` immediately when done. ## Prerequisites All of these must be running: - p3-07: SageMaker endpoint (InService) - p3-10: Model Monitor schedule (Scheduled) - p3-11: Step Functions state machine + Lambda (active) - p3-05: MLflow server with "production" alias Check with: `python src/verify_components.py` ## Quick start ```bash python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt cp .env.example .env # verify all names match prior projects ## Verify everything is live python src/verify_components.py ## Run end-to-end test python src/end_to_end_test.py ## View costs python src/cost_dashboard.py ``` ## Architecture See `docs/architecture.md` for the Mermaid diagram of the full loop. ## Runbook See `docs/runbook.md` for what to do when things break. ## COMPLETE TEARDOWN ```bash bash scripts/teardown_all.sh ``` Deletes every resource from p3-07 through p3-12 in dependency order. ## Tests ```bash pytest tests/ -v ``` ``` --- ### GUIDE.md content ```markdown # Guide — P3-12 Capstone ## What you have built You now have a complete MLOps system: **Data path:** New training data → S3 → trigger → preprocess → train → evaluate → deploy **Serving path:** Users → endpoint → predictions → data capture → drift monitoring → alarm **Reporting path:** Weekly report → live AUC + CloudWatch health → baseline comparison This is the feedback loop that makes a deployed ML model maintainable at scale. ## What is missing (production concerns not covered) **Authentication:** This system has no authentication on the endpoint. In production you would use API Gateway + IAM authentication in front of the SageMaker endpoint. **Multi-region:** Everything runs in one region. A production system may need regional endpoints for latency and redundancy. **Rollback:** If the newly deployed model turns out to be worse in production, there is no automated rollback. You would need to keep the previous `EndpointConfig` and switch back. **Data labeling:** The performance report assumes you have labeled test data. In production, you may not get labels for weeks (e.g., churn prediction — you find out if a user churned 30 days later). This requires a separate labeling pipeline. **Feature store:** All preprocessing is re-done at training time and inference time independently. A feature store (SageMaker Feature Store) would centralize this. ## The cost structure you should internalize | Cost driver | Control lever | |---|---| | Endpoint idle time | Delete when not in use; use auto-scaling down to 0 (not available on SageMaker — use Lambda instead for low-traffic) | | Training job instance size | Use ml.m5.large for small datasets; only scale up when training time is the bottleneck | | Monitoring job frequency | Hourly is the SageMaker minimum; daily is usually sufficient for slow-moving production data | | Data storage | S3 capture data accumulates; set a lifecycle policy to expire after 90 days | ## The interview narrative You now have a story that covers: - Local training (p3-01) → managed training (p3-01 SageMaker) - EDA (p3-02) → feature engineering (p3-03) → experiment tracking (p3-05) - Batch vs real-time inference (p3-06 vs p3-07) - A/B testing (p3-08) → performance reporting (p3-09) → drift monitoring (p3-10) - Automated retraining (p3-11) → integrated system (p3-12) The narrative: "I've built an end-to-end ML system on AWS, from data preprocessing through automated retraining. I understand the trade-offs at each layer — when to use batch vs real-time inference, how to detect and respond to drift, and how to implement safe model rollouts with A/B testing and automated deployment gates." ``` --- ### Failure modes to handle | Failure | Where | How to handle | |---|---|---| | Any component not live | `verify_components.py` | Exit 1 with clear per-component status; print which setup script to run | | End-to-end test timeout | `end_to_end_test.py: wait_for_execution` | Print last known state; suggest running `python src/verify_components.py` to check status | | Cost Explorer not enabled | `cost_dashboard.py` | Print enable instructions; return zero-cost table so script still exits 0 | | Teardown order wrong (delete model before endpoint) | `scripts/teardown_all.sh` | Strict ordering: endpoint first, then configs, then models, then Lambda, then state machine | | `docs/runbook.md` missing | `tests/test_e2e.py` | Test will catch this; create the file | | Multiple Step Functions executions running simultaneously | `end_to_end_test.py` | Wait for all running executions to complete before starting the test execution | --- ### The metric this project measures **What:** End-to-end pipeline execution time — from drift injection to verified endpoint predictions **Format:** ``` [STEP 1/5] Verifying components... OK (2.1s) [STEP 2/5] Injecting drift (50 requests)... OK (18.3s) [STEP 3/5] Checking alarm status... INSUFFICIENT_DATA (no recent runs) [STEP 4/5] Pipeline execution... SUCCEEDED (22m 14s) [STEP 5/5] Endpoint predictions... 5/5 valid (1.8s) End-to-end test PASSED. Total time: 22m 36s ``` **Target:** Total pipeline execution time < 30 minutes (dominated by SageMaker Processing + Training, ~5 min each) --- ### Cost estimate | Resource | Qty | Rate | Estimated cost | |---|---|---|---| | SageMaker endpoint 2h (ml.t2.medium) | 1 | $0.056/hr | ~$0.11 | | SageMaker Training jobs | ~2 | $0.115/hr × 5min | ~$0.02 | | SageMaker Processing jobs | ~2 | $0.115/hr × 5min | ~$0.02 | | SageMaker Monitor jobs | ~2-3 | $0.115/hr × 5min | ~$0.02 | | Step Functions executions | ~2 | < $0.01 | ~$0.01 | | Lambda invocations | ~20 | Free tier | $0.00 | | CloudWatch + S3 + SNS | — | < $0.05 | ~$0.05 | | Accumulated from p3-07 through p3-11 | — | — | ~$5-10 | | **Total per session (all components)** | | | **~$5-10** | ### Teardown checklist `scripts/teardown_all.sh` — complete ordered teardown of all components: **Step 1: Monitoring (p3-10)** - [ ] `aws sagemaker stop-monitoring-schedule --monitoring-schedule-name p3-10-hourly-monitor` - [ ] `aws sagemaker delete-monitoring-schedule --monitoring-schedule-name p3-10-hourly-monitor` - [ ] `aws s3 rm s3://${S3_BUCKET}/p3-10/ --recursive` **Step 2: Endpoint (p3-07 and p3-08)** - [ ] `aws sagemaker delete-endpoint --endpoint-name ${ENDPOINT_NAME}` - [ ] `aws sagemaker wait endpoint-deleted --endpoint-name ${ENDPOINT_NAME}` - [ ] `aws sagemaker delete-endpoint-config --endpoint-config-name ${ENDPOINT_CONFIG_NAME}` - [ ] `aws sagemaker delete-model --model-name ${SAGEMAKER_MODEL_NAME}` - [ ] `aws s3 rm s3://${S3_BUCKET}/p3-07/ --recursive` - [ ] `aws s3 rm s3://${S3_BUCKET}/p3-08/ --recursive` **Step 3: Retraining pipeline (p3-11)** - [ ] `aws stepfunctions delete-state-machine --state-machine-arn $(cat .state-machine-arn)` - [ ] `aws lambda delete-function --function-name p3-11-evaluate-and-deploy` - [ ] `aws lambda delete-function --function-name p3-11-s3-trigger` (if separate) - [ ] `aws s3api put-bucket-notification-configuration --bucket ${S3_BUCKET} --notification-configuration '{}'` - [ ] `aws ssm delete-parameter --name /p3-11/baseline/val_auc` - [ ] `aws s3 rm s3://${S3_BUCKET}/p3-11/ --recursive` **Step 4: Batch artifacts (p3-06)** - [ ] `aws s3 rm s3://${S3_BUCKET}/p3-06/ --recursive` **Step 5: Feature engineering (p3-03)** - [ ] `aws s3 rm s3://${S3_BUCKET}/p3-03/ --recursive` **Step 6: Training artifacts (p3-01)** - [ ] `aws s3 rm s3://${S3_BUCKET}/p3-01/ --recursive` **Step 7: Verify** - [ ] `aws sagemaker list-endpoints --query 'Endpoints[?contains(EndpointName, `p3-`)]'` — should be empty - [ ] `aws stepfunctions list-state-machines --query 'stateMachines[?contains(name, `p3-`)]'` — should be empty - [ ] Print `All Path 3 resources deleted.` ## Source code A reference implementation of MLOps Capstone — README, guide, source and tests — is at https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/capstone-mlops.