MLOps Capstone

Path: 3 — ML Engineering on AWS Position: 12 of 12 Difficulty: 🔴 (full AWS — all components from p3-07 through p3-11) Time: 4-6h AWS Cost: ~$5-10 (all components running simultaneously for ~2 hours)

---

Agent Pickup Instructions

# Bootstrap
cd projects/confident-prep/p3-12-capstone
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Prerequisites: p3-07 through p3-11 must all be deployed
# Verify components first:
python src/verify_components.py

# Run end-to-end test
python src/end_to_end_test.py

# View cost dashboard
python src/cost_dashboard.py

# Review runbook
cat docs/runbook.md

# COMPLETE TEARDOWN OF ALL COMPONENTS
bash scripts/teardown_all.sh

Done when:

• python src/verify_components.py prints ARN/name of endpoint, monitoring schedule, state machine, and Lambda — no errors
• python src/end_to_end_test.py completes: drift injected → alarm checked → pipeline triggered → execution watched → endpoint predictions verified
• python src/cost_dashboard.py prints itemised cost table for SageMaker, Lambda, Step Functions, S3, SNS, and a total
• docs/architecture.md exists with a Mermaid diagram showing the full loop
• docs/runbook.md exists with at least 4 failure scenarios and resolution steps
• pytest tests/ -v shows 4/4 passing
• bash scripts/teardown_all.sh removes every resource and prints confirmation

---

What this project is

The capstone wires p3-07 (real-time endpoint) through p3-11 (retraining pipeline) into a single observable MLOps system. You verify all components are live, run an orchestrated end-to-end test (inject drift → check alarm → trigger retraining → watch pipeline execute → verify new endpoint predictions), view an itemised cost dashboard from AWS Cost Explorer, and prove the system works as a unit. The teardown script removes every resource created across the entire path in the correct dependency order.

What the learner achieves

"I have built and operated a complete MLOps loop on AWS: model serving, A/B testing, data capture, drift monitoring, automated retraining, and conditional deployment — and I can articulate what each component does, what it costs, and what to do when each one fails."

---

Folder structure

p3-12-capstone/
├── src/
│   ├── verify_components.py   # check endpoint, monitoring, state machine, Lambda are all live
│   ├── end_to_end_test.py     # orchestrate full test: drift → alarm → pipeline → verify
│   └── cost_dashboard.py      # AWS Cost Explorer: past 24h by service
├── scripts/
│   └── teardown_all.sh        # ordered teardown of every resource from p3-07 through p3-12
├── docs/
│   ├── architecture.md        # Mermaid diagram of full MLOps loop
│   └── runbook.md             # what to do when each component fails
├── tests/
│   ├── test_verify.py         # clear error when component missing, cost zero handling
│   └── test_e2e.py            # endpoint predictions after pipeline, cost table format
├── .env.example
├── requirements.txt
└── README.md

---

.env.example

# AWS region
AWS_REGION=us-east-1

# Cross-project resource names (must match prior projects)
ENDPOINT_NAME=p3-07-adult-income-endpoint
ENDPOINT_CONFIG_NAME=p3-07-adult-income-config
SAGEMAKER_MODEL_NAME=p3-07-adult-income-model

MONITORING_SCHEDULE_NAME=p3-10-hourly-monitor

STATE_MACHINE_ARN_FILE=../p3-11-retraining-pipeline/.state-machine-arn
LAMBDA_EVALUATOR_NAME=p3-11-evaluate-and-deploy

SNS_TOPIC_ARN=arn:aws:sns:us-east-1:123456789012:p3-11-ml-notifications

S3_BUCKET=my-ml-training-bucket

MLFLOW_TRACKING_URI=http://localhost:5000
MLFLOW_MODEL_NAME=adult-income-xgboost

# Cost Explorer: AWS account ID
AWS_ACCOUNT_ID=123456789012

---

requirements.txt

boto3==1.35.93
sagemaker==2.236.0
xgboost==2.1.3
mlflow==2.19.0
pandas==2.2.3
numpy==2.2.1
python-dotenv==1.0.1
pytest==8.3.4

---

src/ — what to implement

src/verify_components.py

CLI: python src/verify_components.py

check_endpoint(endpoint_name: str, region: str) -> dict

• Calls boto3.client('sagemaker').describe_endpoint(EndpointName=endpoint_name)
• Returns {"name": str, "status": str, "ok": bool} where ok is True iff status == "InService"
• Catches ClientError with code ValidationException — returns {"name": endpoint_name, "status": "NOT_FOUND", "ok": False}

check_monitoring_schedule(schedule_name: str, region: str) -> dict

• Calls describe_monitoring_schedule
• Returns {"name": str, "status": str, "ok": bool} where ok is True iff status == "Scheduled"

check_state_machine(arn: str, region: str) -> dict

• Calls boto3.client('stepfunctions').describe_state_machine(stateMachineArn=arn)
• Returns {"arn": str, "name": str, "status": str, "ok": bool}

check_lambda(function_name: str, region: str) -> dict

• Calls boto3.client('lambda').get_function(FunctionName=function_name)
• Returns {"name": str, "runtime": str, "ok": bool}

verify_all(config: dict) -> dict

• Calls all four check functions
• Returns {"endpoint": dict, "monitoring": dict, "state_machine": dict, "lambda": dict, "all_ok": bool}
• all_ok is True only if all four ok fields are True

print_component_table(result: dict) -> None

• Prints:

Component Status
============================================
Endpoint:           p3-07-adult-income-endpoint  [OK / MISSING]
Monitoring:         p3-10-hourly-monitor          [OK / MISSING]
State Machine:      p3-11-ml-pipeline             [OK / MISSING]
Lambda:             p3-11-evaluate-and-deploy      [OK / MISSING]
--------------------------------------------
Overall:            ALL READY / COMPONENTS MISSING

main()

• verify_all → print_component_table
• If all_ok == False: print Run the setup scripts for any MISSING components.; exit 1
• If all_ok == True: print All components live. Ready to run end_to_end_test.py; exit 0

---

src/end_to_end_test.py

CLI: python src/end_to_end_test.py [--skip-drift] [--skip-alarm] [--timeout-minutes 15]

Orchestrates the following sequence, printing [STEP N/5] description... before each step:

inject_drift(endpoint_name: str, data_path: str, n_requests: int = 50) -> dict

• Reuses p3-10 inject_drift logic: send requests with features at mean + 2*std
• Returns {"sent": int, "succeeded": int}

check_alarm_status(endpoint_name: str, region: str) -> dict

• Calls boto3.client('cloudwatch').describe_alarms(AlarmNamePrefix=endpoint_name)
• Returns list of alarms with their state (OK, ALARM, INSUFFICIENT_DATA)
• Does NOT wait for alarm to fire — just reports current status
• Prints warning if state is INSUFFICIENT_DATA: "Monitoring may not have run yet"

trigger_pipeline(state_machine_arn: str, input_data: dict) -> str

• Starts a Step Functions execution with sample input
• Returns execution ARN

wait_for_execution(execution_arn: str, timeout_minutes: int = 15) -> dict

• Polls execution status
• Prints each state transition with timestamp: [14:32:01] Preprocess → SUCCEEDED
• Returns final status dict
• Times out after timeout_minutes with a clear message

verify_endpoint_predictions(endpoint_name: str, data_path: str, n_samples: int = 5) -> dict

• Sends 5 inference requests to the endpoint
• Returns {"predictions": list[float], "all_valid": bool} where all_valid checks all are in [0,1]

main()

[STEP 1/5] Verifying all components are live...
[STEP 2/5] Injecting synthetic drift (50 requests, mean + 2*std)...
[STEP 3/5] Checking CloudWatch alarm status...
[STEP 4/5] Triggering retraining pipeline...
  [14:32:01] Preprocess → RUNNING
  [14:36:22] Preprocess → SUCCEEDED
  [14:36:23] Train → RUNNING
  [14:40:51] Train → SUCCEEDED
  [14:40:52] Evaluate → RUNNING
  [14:40:54] Evaluate → SUCCEEDED
  [14:40:55] ShouldDeploy → (branch: deploy)
  [14:40:55] NotifyDeployed → SUCCEEDED
[STEP 5/5] Verifying endpoint returns valid predictions...
End-to-end test PASSED. Total time: {elapsed:.1f}s

• Write results to output/e2e_results.json

---

src/cost_dashboard.py

CLI: python src/cost_dashboard.py [--days 1]

fetch_cost_by_service(account_id: str, days: int, region: str) -> list[dict]

• Calls boto3.client('ce', region_name='us-east-1').get_cost_and_usage()
  • TimePeriod: last days days
  • Granularity: DAILY
  • GroupBy: [{"Type": "DIMENSION", "Key": "SERVICE"}]
  • Metrics: ["UnblendedCost"]
• Returns list of {"service": str, "cost_usd": float} sorted by cost descending

print_cost_table(costs: list[dict]) -> None

• Prints:

AWS Cost Dashboard — Last 24 hours
====================================
Service                     Cost (USD)
------------------------------------
Amazon SageMaker            $0.4230
Amazon S3                   $0.0012
AWS Lambda                  $0.0000
AWS Step Functions          $0.0000
Amazon SNS                  $0.0000
Amazon CloudWatch           $0.0451
------------------------------------
TOTAL                       $0.4693

• Services not in the result are shown as $0.0000
• Always shows these 6 rows even if cost is zero

Edge cases:

• If Cost Explorer API not enabled: print Cost Explorer not enabled. Enable it in AWS Billing console. and return a hardcoded zero table
• If days=1 and it is early in the day, results may show only partial cost — print a note

---

docs/architecture.md content

# Architecture — Path 3 MLOps System

## Component Map

```mermaid
graph LR
    A[UCI Adult Data\nS3 Bucket] -->|upload triggers| B[S3 Event\nNotification]
    B --> C[Lambda Trigger]
    C --> D[Step Functions\nState Machine]
    
    D --> E[SageMaker\nProcessing Job\np3-03 Pipeline]
    E --> F[SageMaker\nTraining Job\nXGBoost]
    F --> G[Lambda Evaluator\nAUC vs SSM Baseline]
    
    G -->|AUC improved > 1%| H[Deploy State\nUpdate Endpoint]
    G -->|AUC not improved| I[SNS: No Deploy\nNotification]
    H --> J[SNS: Deployed\nNotification]
    
    K[SageMaker\nEndpoint\np3-07] -->|captures requests| L[S3 Capture\nData]
    L --> M[Model Monitor\nScheduled Job\np3-10]
    M -->|violation| N[CloudWatch\nAlarm]
    
    K -->|inference| O[Users / Batch Jobs]
    
    P[MLflow Registry\np3-05] -->|production alias| K
    H -->|update alias| P
    
    Q[Weekly Report\np3-09] -->|AUC delta| N

Data Flow

1. Training trigger: New data uploaded to S3 → S3 notification → Lambda → Step Functions execution
2. Preprocessing: SageMaker Processing Job applies the sklearn pipeline (p3-03)
3. Training: SageMaker Training Job on preprocessed data → model artifact to S3
4. Evaluation: Lambda reads new AUC from CloudWatch, compares to SSM baseline
5. Conditional deploy: If AUC improves > 1%, update endpoint; else notify and skip
6. Monitoring: Model Monitor captures requests, computes drift vs training baseline, fires alarm
7. Reporting: Weekly report (p3-09) computes live accuracy vs MLflow baseline, surfaces delta

Component Registry

Component	Project	AWS Service	Resource Name
Real-time endpoint	p3-07	SageMaker Endpoint	p3-07-adult-income-endpoint
Monitoring schedule	p3-10	SageMaker Model Monitor	p3-10-hourly-monitor
Retraining pipeline	p3-11	Step Functions	p3-11-ml-pipeline
Evaluator Lambda	p3-11	Lambda	p3-11-evaluate-and-deploy
Experiment registry	p3-05	MLflow (local)	adult-income-xgboost

---

## docs/runbook.md content

```markdown
# Operational Runbook — Path 3 MLOps System

## Failure Scenario 1: Endpoint is down

**Symptom:** `verify_components.py` reports `Endpoint: MISSING` or `status: Failed`

**Diagnosis:**
```bash
aws sagemaker describe-endpoint --endpoint-name p3-07-adult-income-endpoint
aws logs get-log-events --log-group-name /aws/sagemaker/Endpoints/p3-07-adult-income-endpoint \
  --log-stream-name $(aws logs describe-log-streams --log-group-name \
  /aws/sagemaker/Endpoints/p3-07-adult-income-endpoint --query 'logStreams[-1].logStreamName' \
  --output text)

Resolution:

• If status is Failed: check logs for container error. Most common: wrong inference.py syntax.
• Re-deploy using p3-07's deploy.py. Do not delete prior endpoint config until new one is InService.
• If endpoint was accidentally deleted: re-run python src/deploy.py from p3-07.

Cost impact: While endpoint is down, no inference cost accrues. Recreating costs a new cold-start.

---

Failure Scenario 2: Drift alarm fires but retraining pipeline fails

Symptom: CloudWatch alarm is in ALARM state but Step Functions execution status is FAILED

Diagnosis:

# Get most recent execution
aws stepfunctions list-executions --state-machine-arn $(cat .state-machine-arn) \
  --status-filter FAILED --query 'executions[0].executionArn' --output text

# Get failure details
aws stepfunctions get-execution-history --execution-arn <arn> \
  --query 'events[?type==`TaskFailed`]'

Common causes and resolutions:

• Preprocess state fails: Check SageMaker Processing Job logs in CloudWatch. Usually: S3 path wrong, container image URI wrong, or IAM role missing S3 permissions.
• Train state fails: Check Training Job failure reason: aws sagemaker describe-training-job --training-job-name <name>. Usually: data format wrong, out of disk space.
• Evaluate state fails: Lambda function error. Check Lambda logs: aws logs tail /aws/lambda/p3-11-evaluate-and-deploy.
• Deploy state fails: Usually: endpoint already being updated, or IAM role missing SageMaker permissions.

Resolution: Fix root cause, then manually start execution:

python src/test_pipeline.py

---

Failure Scenario 3: Model quality degrades without drift alarm

Symptom: p3-09's weekly report shows Delta: -0.04 (regression — WARNING) but Model Monitor shows no violations

Diagnosis: This is concept drift — the relationship between features and the label has changed, but the input feature distributions look the same. Model Monitor cannot detect this.

Resolution:

1. Check the weekly report's live accuracy trend over past 4 weeks
2. If accuracy has been declining for 2+ weeks: trigger manual retraining
3. Collect new labeled data if possible — concept drift usually means the world changed
4. Temporarily lower the AUC improvement threshold in SSM to 0.005 to allow easier deployment of retrained models

aws ssm put-parameter --name /p3-11/baseline/val_auc --value 0.860 --overwrite
python src/test_pipeline.py

---

Failure Scenario 4: AWS costs spike unexpectedly

Symptom: cost_dashboard.py shows SageMaker cost much higher than expected

Diagnosis:

python src/cost_dashboard.py --days 7

Most likely causes:

• Endpoint was not torn down between sessions (p3-07 through p3-10 endpoints accumulate hourly)
• Multiple training jobs running simultaneously (Step Functions executing multiple pipelines)
• Monitoring schedule running more frequently than expected

Resolution — immediate:

bash scripts/teardown_all.sh

Resolution — preventive:

• Always run teardown scripts at the end of each session
• Set a CloudWatch billing alarm: aws cloudwatch put-metric-alarm --alarm-name "ML-Cost-Alert" --metric-name EstimatedCharges ...
• Use aws ce get-cost-and-usage daily during active development

Expected costs when actively using the system:

• Endpoint idle: ~$0.056/hr (ml.t2.medium)
• Training job: ~$0.01 per run (ml.m5.large, 5 min)
• Monitoring job: ~$0.01 per run
• Step Functions: < $0.01 per execution

---

## tests/ — what to test

### `tests/test_verify.py`

**`test_verify_raises_clear_error_if_endpoint_missing`**
- Mock `boto3.client('sagemaker').describe_endpoint` to raise `ClientError` with `ValidationException`
- Call `check_endpoint("nonexistent-endpoint", "us-east-1")`
- Assert `result["ok"] == False`
- Assert `result["status"] == "NOT_FOUND"`
- Assert no exception is raised (graceful handling)

**`test_verify_all_ok_is_false_when_any_component_missing`**
- Mock: endpoint OK, monitoring OK, state machine OK, Lambda NOT_FOUND
- Call `verify_all(config)`
- Assert `result["all_ok"] == False`

### `tests/test_e2e.py`

**`test_cost_dashboard_handles_zero_cost_services`**
- Mock `get_cost_and_usage` to return data for only 2 of the 6 services (others absent)
- Call `print_cost_table(costs)`
- Assert output contains all 6 service names
- Assert missing services show `$0.0000`

**`test_endpoint_predictions_are_valid_probabilities`**
- Mock `invoke_endpoint` to return `"0.75"` body
- Call `verify_endpoint_predictions("mock-endpoint", "data/adult.data", n_samples=3)` with mocked runtime
- Assert `result["all_valid"] == True`
- Assert all predictions are in `[0.0, 1.0]`

**`test_runbook_has_at_least_four_failure_scenarios`**
- Open `docs/runbook.md`
- Count occurrences of `## Failure Scenario`
- Assert count >= 4

**`test_architecture_md_has_mermaid_diagram`**
- Open `docs/architecture.md`
- Assert `"```mermaid"` in content
- Assert `"graph"` in content

---

## README.md content

```markdown
# P3-12 — Capstone: Full MLOps Loop

Wire p3-07 through p3-11 into a complete, observable MLOps system.

## WARNING

This project runs ALL components simultaneously. Combined cost: ~$5-10 for a 2-hour session.
Run `bash scripts/teardown_all.sh` immediately when done.

## Prerequisites

All of these must be running:
- p3-07: SageMaker endpoint (InService)
- p3-10: Model Monitor schedule (Scheduled)
- p3-11: Step Functions state machine + Lambda (active)
- p3-05: MLflow server with "production" alias

Check with: `python src/verify_components.py`

## Quick start

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env  # verify all names match prior projects

# Verify everything is live
python src/verify_components.py

# Run end-to-end test
python src/end_to_end_test.py

# View costs
python src/cost_dashboard.py

Architecture

See docs/architecture.md for the Mermaid diagram of the full loop.

Runbook

See docs/runbook.md for what to do when things break.

COMPLETE TEARDOWN

bash scripts/teardown_all.sh

Deletes every resource from p3-07 through p3-12 in dependency order.

Tests

pytest tests/ -v

---

## GUIDE.md content

```markdown
# Guide — P3-12 Capstone

## What you have built

You now have a complete MLOps system:

**Data path:** New training data → S3 → trigger → preprocess → train → evaluate → deploy
**Serving path:** Users → endpoint → predictions → data capture → drift monitoring → alarm
**Reporting path:** Weekly report → live AUC + CloudWatch health → baseline comparison

This is the feedback loop that makes a deployed ML model maintainable at scale.

## What is missing (production concerns not covered)

**Authentication:** This system has no authentication on the endpoint. In production you would
use API Gateway + IAM authentication in front of the SageMaker endpoint.

**Multi-region:** Everything runs in one region. A production system may need regional endpoints
for latency and redundancy.

**Rollback:** If the newly deployed model turns out to be worse in production, there is no
automated rollback. You would need to keep the previous `EndpointConfig` and switch back.

**Data labeling:** The performance report assumes you have labeled test data. In production,
you may not get labels for weeks (e.g., churn prediction — you find out if a user churned
30 days later). This requires a separate labeling pipeline.

**Feature store:** All preprocessing is re-done at training time and inference time independently.
A feature store (SageMaker Feature Store) would centralize this.

## The cost structure you should internalize

| Cost driver | Control lever |
|---|---|
| Endpoint idle time | Delete when not in use; use auto-scaling down to 0 (not available on SageMaker — use Lambda instead for low-traffic) |
| Training job instance size | Use ml.m5.large for small datasets; only scale up when training time is the bottleneck |
| Monitoring job frequency | Hourly is the SageMaker minimum; daily is usually sufficient for slow-moving production data |
| Data storage | S3 capture data accumulates; set a lifecycle policy to expire after 90 days |

## The interview narrative

You now have a story that covers:
- Local training (p3-01) → managed training (p3-01 SageMaker)
- EDA (p3-02) → feature engineering (p3-03) → experiment tracking (p3-05)
- Batch vs real-time inference (p3-06 vs p3-07)
- A/B testing (p3-08) → performance reporting (p3-09) → drift monitoring (p3-10)
- Automated retraining (p3-11) → integrated system (p3-12)

The narrative: "I've built an end-to-end ML system on AWS, from data preprocessing through
automated retraining. I understand the trade-offs at each layer — when to use batch vs
real-time inference, how to detect and respond to drift, and how to implement safe model
rollouts with A/B testing and automated deployment gates."

---

Failure modes to handle

Failure	Where	How to handle
Any component not live	verify_components.py	Exit 1 with clear per-component status; print which setup script to run
End-to-end test timeout	end_to_end_test.py: wait_for_execution	Print last known state; suggest running python src/verify_components.py to check status
Cost Explorer not enabled	cost_dashboard.py	Print enable instructions; return zero-cost table so script still exits 0
Teardown order wrong (delete model before endpoint)	scripts/teardown_all.sh	Strict ordering: endpoint first, then configs, then models, then Lambda, then state machine
docs/runbook.md missing	tests/test_e2e.py	Test will catch this; create the file
Multiple Step Functions executions running simultaneously	end_to_end_test.py	Wait for all running executions to complete before starting the test execution

---

The metric this project measures

What: End-to-end pipeline execution time — from drift injection to verified endpoint predictions

Format:

[STEP 1/5] Verifying components... OK (2.1s)
[STEP 2/5] Injecting drift (50 requests)... OK (18.3s)
[STEP 3/5] Checking alarm status... INSUFFICIENT_DATA (no recent runs)
[STEP 4/5] Pipeline execution... SUCCEEDED (22m 14s)
[STEP 5/5] Endpoint predictions... 5/5 valid (1.8s)
End-to-end test PASSED. Total time: 22m 36s

Target: Total pipeline execution time < 30 minutes (dominated by SageMaker Processing + Training, ~5 min each)

---

Cost estimate

Resource	Qty	Rate	Estimated cost
SageMaker endpoint 2h (ml.t2.medium)	1	$0.056/hr	~$0.11
SageMaker Training jobs	~2	$0.115/hr × 5min	~$0.02
SageMaker Processing jobs	~2	$0.115/hr × 5min	~$0.02
SageMaker Monitor jobs	~2-3	$0.115/hr × 5min	~$0.02
Step Functions executions	~2	< $0.01	~$0.01
Lambda invocations	~20	Free tier	$0.00
CloudWatch + S3 + SNS	—	< $0.05	~$0.05
Accumulated from p3-07 through p3-11	—	—	~$5-10
Total per session (all components)			~$5-10

Teardown checklist

scripts/teardown_all.sh — complete ordered teardown of all components:

Step 1: Monitoring (p3-10)

• aws sagemaker stop-monitoring-schedule --monitoring-schedule-name p3-10-hourly-monitor
• aws sagemaker delete-monitoring-schedule --monitoring-schedule-name p3-10-hourly-monitor
• aws s3 rm s3://${S3_BUCKET}/p3-10/ --recursive

Step 2: Endpoint (p3-07 and p3-08)

• aws sagemaker delete-endpoint --endpoint-name ${ENDPOINT_NAME}
• aws sagemaker wait endpoint-deleted --endpoint-name ${ENDPOINT_NAME}
• aws sagemaker delete-endpoint-config --endpoint-config-name ${ENDPOINT_CONFIG_NAME}
• aws sagemaker delete-model --model-name ${SAGEMAKER_MODEL_NAME}
• aws s3 rm s3://${S3_BUCKET}/p3-07/ --recursive
• aws s3 rm s3://${S3_BUCKET}/p3-08/ --recursive

Step 3: Retraining pipeline (p3-11)

• aws stepfunctions delete-state-machine --state-machine-arn $(cat .state-machine-arn)
• aws lambda delete-function --function-name p3-11-evaluate-and-deploy
• aws lambda delete-function --function-name p3-11-s3-trigger (if separate)
• aws s3api put-bucket-notification-configuration --bucket ${S3_BUCKET} --notification-configuration '{}'
• aws ssm delete-parameter --name /p3-11/baseline/val_auc
• aws s3 rm s3://${S3_BUCKET}/p3-11/ --recursive

Step 4: Batch artifacts (p3-06)

• aws s3 rm s3://${S3_BUCKET}/p3-06/ --recursive

Step 5: Feature engineering (p3-03)

• aws s3 rm s3://${S3_BUCKET}/p3-03/ --recursive

Step 6: Training artifacts (p3-01)

• aws s3 rm s3://${S3_BUCKET}/p3-01/ --recursive

Step 7: Verify

• aws sagemaker list-endpoints --query 'Endpoints[?contains(EndpointName, p3-)]' — should be empty
• aws stepfunctions list-state-machines --query 'stateMachines[?contains(name, p3-)]' — should be empty
• Print All Path 3 resources deleted.