ML Engineering on AWS

Path: 3 — ML Engineering on AWS Position: 1 of 12 Difficulty: 🟡 (local training works without AWS; SageMaker is the "scale up") Time: 3-4h local, 5-6h with SageMaker AWS Cost: ~$0.10–0.50 per SageMaker training job (ml.m5.large)

---

Agent Pickup Instructions

# Bootstrap
cd projects/confident-prep/p3-01-local-to-sagemaker
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Download dataset
mkdir -p data
curl -o data/adult.data \
  "https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data"

# Run locally (no AWS)
python src/local_train.py --data data/adult.data

# Submit to SageMaker (requires AWS env vars)
python src/sagemaker_launch.py --instance-type ml.m5.large

# Run tests
pytest tests/ -v

Done when:

• python src/local_train.py prints accuracy > 0.80 and saves models/model.xgb
• python src/sagemaker_launch.py submits a training job that completes successfully
• python src/cost_estimator.py --instance ml.m5.large --minutes 4 prints a dollar value
• python src/metrics_fetcher.py --job-name <job> prints accuracy from CloudWatch logs
• pytest tests/ -v shows 4/4 passing
• bash scripts/teardown.sh removes all S3 objects created

---

What this project is

You train an XGBoost binary classifier on the UCI Adult Income dataset (predict whether income exceeds $50K) first on your local machine, then submit the exact same training script to SageMaker as a managed training job — SageMaker pulls a container, mounts your data from S3, runs your script, and saves the model artifact back to S3. You compare training time and cost between ml.m5.large and ml.m5.xlarge, and you fetch the accuracy metric from CloudWatch after the job finishes.

What the learner achieves

"I trained a model locally and then moved it to SageMaker without changing the training script — I understand what the managed training abstraction is actually doing and can estimate its cost."

---

Folder structure

p3-01-local-to-sagemaker/
├── data/
│   └── adult.data              # raw UCI download (no header row)
├── models/                     # local training outputs
│   └── model.xgb              # saved after local_train.py
├── src/
│   ├── train.py               # main training script — runs locally AND on SageMaker
│   ├── local_train.py         # wrapper to invoke train.py locally
│   ├── sagemaker_launch.py    # upload data, submit SageMaker job, wait, download artifact
│   ├── cost_estimator.py      # estimate job cost from instance type + duration
│   └── metrics_fetcher.py     # fetch accuracy from CloudWatch logs for a finished job
├── scripts/
│   └── teardown.sh            # delete all S3 objects and SageMaker artifacts created
├── tests/
│   ├── test_train.py          # verify train.py accuracy, file output, entrypoint identity
│   └── test_cost_estimator.py # verify cost formula for known inputs
├── .env.example
├── requirements.txt
└── README.md

---

.env.example

# AWS region where your SageMaker jobs will run
AWS_REGION=us-east-1

# S3 bucket for SageMaker training data and model artifacts
# Must already exist; the scripts do NOT create the bucket
S3_BUCKET=my-ml-training-bucket

# IAM role ARN that SageMaker will assume during training
# Must have AmazonSageMakerFullAccess and S3 read/write on S3_BUCKET
SAGEMAKER_ROLE_ARN=arn:aws:iam::123456789012:role/SageMakerExecutionRole

# Local directories (no AWS required)
LOCAL_DATA_DIR=data/
LOCAL_MODEL_DIR=models/

---

requirements.txt

xgboost==2.1.3
scikit-learn==1.6.0
pandas==2.2.3
boto3==1.35.93
sagemaker==2.236.0
python-dotenv==1.0.1
pytest==8.3.4

---

src/ — what to implement

src/train.py

Single training script that runs identically in both environments.

CLI arguments (argparse):

• --data-dir (str, default /opt/ml/input/data/train) — directory containing adult.data
• --model-dir (str, default /opt/ml/model) — where to save the output model
• --max-depth (int, default 5) — XGBoost max_depth
• --learning-rate (float, default 0.1) — XGBoost learning_rate
• --n-estimators (int, default 100) — number of boosting rounds

load_data(data_dir: str) -> tuple[pd.DataFrame, pd.Series]

• Reads adult.data from data_dir (CSV, no header row)
• Assigns column names: age, workclass, fnlwgt, education, education-num, marital-status, occupation, relationship, race, sex, capital-gain, capital-loss, hours-per-week, native-country, income
• Strips whitespace from string columns
• Replaces '?' with NaN
• Drops rows with any NaN
• Encodes categorical columns with pd.get_dummies
• Returns X, y where y is 1 if income contains >50K else 0

train_model(X_train, y_train, params: dict) -> xgb.XGBClassifier

• Fits an XGBClassifier with use_label_encoder=False, eval_metric='logloss'
• Returns fitted model

evaluate_model(model, X_test, y_test) -> dict

• Returns {"accuracy": float, "n_test": int}
• Prints ACCURACY: {accuracy:.4f} — SageMaker scrapes this line from stdout for CloudWatch

save_model(model, model_dir: str) -> str

• Saves model to {model_dir}/model.xgb using model.save_model()
• Returns the saved path

main()

• Calls load_data → train_test_split (80/20, random_state=42) → train_model → evaluate_model → save_model
• Exits with code 0 on success

Edge cases:

• If data_dir contains no .data or .csv file, raise FileNotFoundError with a message listing what was found
• If accuracy < 0.70, print a warning but do not fail — SageMaker should not retry on low accuracy

---

src/local_train.py

Thin wrapper for local execution.

CLI: python src/local_train.py --data data/adult.data [--max-depth N] [--learning-rate X] [--n-estimators N]

run_local(data_path: str, model_dir: str, hyperparams: dict) -> dict

• Copies data_path to a temp directory
• Calls train.main() via subprocess with --data-dir, --model-dir, and hyperparam flags
• Captures stdout, extracts ACCURACY: line, returns {"accuracy": float, "model_path": str}

Behavior: prints Training locally..., then the accuracy line, then Model saved to {path}.

---

src/sagemaker_launch.py

CLI: python src/sagemaker_launch.py [--instance-type ml.m5.large] [--max-depth N] [--learning-rate X]

upload_data_to_s3(local_path: str, bucket: str, prefix: str) -> str

• Uploads adult.data to s3://{bucket}/{prefix}/adult.data
• Returns the S3 URI

launch_training_job(s3_data_uri: str, instance_type: str, hyperparams: dict) -> str

• Creates a sagemaker.estimator.Estimator using the built-in XGBoost container (framework xgboost, version 1.7-1)
• Sets entry_point='src/train.py', source_dir='.'
• Calls .fit({"train": s3_data_uri}) with wait=True
• Returns the training job name

download_model_artifact(job_name: str, local_dir: str) -> str

• Calls sagemaker.Session().download_data() to pull model.tar.gz from S3 output
• Extracts and saves model.xgb to local_dir
• Returns local path

compare_instance_types(s3_data_uri: str, hyperparams: dict) -> None

• Launches two sequential training jobs: ml.m5.large and ml.m5.xlarge
• Prints: ml.m5.large: {minutes:.1f} min, ${cost:.4f} for each

Edge cases:

• If SAGEMAKER_ROLE_ARN is not set, print a clear message and exit — do not throw a raw KeyError
• If the training job fails (status Failed), print the failure reason from describe_training_job

---

src/cost_estimator.py

INSTANCE_PRICES: dict[str, float]

• Hardcoded: {"ml.m5.large": 0.115, "ml.m5.xlarge": 0.23, "ml.m5.2xlarge": 0.46}
• Units: USD per hour

estimate_cost(instance_type: str, duration_minutes: float) -> float

• Returns INSTANCE_PRICES[instance_type] * (duration_minutes / 60)
• Raises ValueError if instance_type not in INSTANCE_PRICES
• Rounds to 6 decimal places

format_cost_report(instance_type: str, duration_minutes: float) -> str

• Returns: "Instance: {instance_type} | Duration: {duration_minutes:.1f} min | Cost: ${cost:.4f}"

CLI: python src/cost_estimator.py --instance ml.m5.large --minutes 4.5 — prints the formatted report.

---

src/metrics_fetcher.py

fetch_accuracy_from_logs(job_name: str, region: str) -> float | None

• Uses boto3.client('logs') to query log group /aws/sagemaker/TrainingJobs
• Filters log stream for job_name, searches for line matching ACCURACY: (\d+\.\d+)
• Returns the float, or None if not found (job may not have output it yet)

wait_for_metric(job_name: str, region: str, timeout_seconds: int = 120) -> float

• Polls every 10 seconds until fetch_accuracy_from_logs returns non-None or timeout
• Raises TimeoutError on timeout

CLI: python src/metrics_fetcher.py --job-name <job-name> — prints Accuracy: {value}

---

tests/ — what to test

tests/test_train.py

test_train_produces_accuracy_above_threshold

• Run train.main() with actual adult.data in a temp dir
• Assert returned accuracy > 0.80
• Assert model.xgb exists in --model-dir

test_train_saves_model_file

• Call train.save_model(mock_model, tmp_path) with a tiny fitted XGBClassifier
• Assert tmp_path / "model.xgb" exists and is non-empty

test_local_and_sagemaker_use_same_entrypoint

• Read src/train.py path
• Assert the file exists and contains def main() — same entrypoint sagemaker_launch.py will use

test_load_data_handles_question_mark

• Create a small CSV with ? in workclass column
• Call train.load_data() with that directory
• Assert no ? values appear in the returned DataFrame

---

tests/test_cost_estimator.py

test_estimate_cost_known_value

• estimate_cost("ml.m5.large", 60) should return 0.115 (exactly 1 hour)
• estimate_cost("ml.m5.xlarge", 30) should return 0.115 (half hour at double rate)

test_estimate_cost_unknown_instance_raises

• estimate_cost("ml.p4d.24xlarge", 10) raises ValueError

---

README.md content

# P3-01 — Local to SageMaker

Train an XGBoost model locally, then submit the same script to SageMaker.

## Prerequisites

- Python 3.11+
- AWS credentials configured (`aws configure`) — only needed for SageMaker step
- IAM role with SageMaker and S3 permissions — see `.env.example`

## Quick start (local only — no AWS)

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
mkdir -p data && curl -o data/adult.data \
  "https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data"
python src/local_train.py --data data/adult.data

Submit to SageMaker

cp .env.example .env
# Fill in AWS_REGION, S3_BUCKET, SAGEMAKER_ROLE_ARN
python src/sagemaker_launch.py --instance-type ml.m5.large

Compare instance types

python src/sagemaker_launch.py --compare-instances

Estimate cost

python src/cost_estimator.py --instance ml.m5.large --minutes 5

Tests

pytest tests/ -v

Teardown

bash scripts/teardown.sh

Deletes: S3 training data, S3 model artifact. Does NOT delete the S3 bucket itself.

---

## GUIDE.md content

```markdown
# Guide — P3-01 Local to SageMaker

## What SageMaker training actually does

When you call `.fit()`:
1. SageMaker pulls a Docker container with XGBoost pre-installed
2. It mounts your S3 data at `/opt/ml/input/data/train/`
3. It runs your `train.py` script inside the container
4. It copies everything your script writes to `/opt/ml/model/` back to S3 as `model.tar.gz`
5. It terminates the instance

Your script does not know it is in SageMaker. That is the point — the same code runs locally and at scale.

## Why the same script works in both places

`train.py` uses `--data-dir` and `--model-dir` arguments. Locally you pass your own paths. SageMaker injects `/opt/ml/input/data/train` and `/opt/ml/model` automatically. The script never imports boto3 or sagemaker — it is pure ML code.

## What to look at in the AWS console

1. **SageMaker → Training jobs** — see the job status, instance type, duration
2. **CloudWatch → Log groups → /aws/sagemaker/TrainingJobs** — see your script's stdout including the `ACCURACY:` line
3. **S3 → your bucket** — see the uploaded data and the output `model.tar.gz`

## Cost breakdown

| Component | Price |
|---|---|
| ml.m5.large per hour | $0.115 |
| ml.m5.xlarge per hour | $0.23 |
| S3 storage (GB/month) | $0.023 |
| S3 PUT request | ~$0.000005 |

A 5-minute training job on ml.m5.large costs about $0.01.

## What to try next

- Change `--n-estimators` to 500. Does it improve accuracy? How much longer does it take?
- Try submitting to ml.m5.xlarge and compare the wall-clock time.
- Open the CloudWatch logs while the job is running — you will see your print statements in near real time.

## Interview framing

"I understand what SageMaker managed training abstracts away — it handles container orchestration, data mounting, and artifact storage — and I know how to write training code that works identically locally and in the cloud by using path arguments instead of hardcoded paths."

---

Failure modes to handle

Failure	Where	How to handle
adult.data has no header row	train.py: load_data	Hardcode column names list; never rely on CSV header
? values appear as strings not NaN	train.py: load_data	Explicitly replace '?' with np.nan after reading
SageMaker role ARN not set	sagemaker_launch.py	Check env var, print clear message with link to IAM setup, sys.exit(1)
Training job status is Failed	sagemaker_launch.py	Call describe_training_job, print FailureReason, raise exception
CloudWatch logs not yet available	metrics_fetcher.py	Retry with exponential backoff up to timeout_seconds
Instance type not in price table	cost_estimator.py	Raise ValueError with list of known types
model.xgb missing after training	local_train.py	Assert file exists after subprocess call; print error if missing

---

The metric this project measures

What: Accuracy of the XGBoost classifier on the UCI Adult Income test set (20% holdout, random_state=42)

Format: Float between 0 and 1, printed as ACCURACY: 0.8712 to stdout

Target: > 0.80 (UCI Adult baseline with basic preprocessing)

Secondary metric: SageMaker job duration in minutes, fetched from describe_training_job and printed alongside estimated cost

---

Cost estimate

Resource	Qty	Rate	Estimated cost
ml.m5.large training job	1 run × ~5 min	$0.115/hr	~$0.01
ml.m5.xlarge training job	1 run × ~4 min	$0.23/hr	~$0.015
S3 storage (data + artifact)	~50 MB	$0.023/GB-month	<$0.01
S3 PUT requests	~10	$0.005/1000	<$0.01
Total per session			~$0.10–0.50

Teardown checklist

scripts/teardown.sh must:

• Delete s3://${S3_BUCKET}/p3-01/data/adult.data
• Delete s3://${S3_BUCKET}/p3-01/output/ prefix (model artifact)
• Verify deletion with aws s3 ls — print "No objects found" to confirm
• Print Teardown complete. Estimated charges: < $0.50

The script does NOT delete the S3 bucket itself (it may be shared with other projects).