Model Serving

Path: 3 — ML Engineering on AWS Position: 7 of 12 Difficulty: 🔴 Time: 3-4h AWS Cost: ~$0.05/hour × 2 hours = ~$0.10 per session. DELETE ENDPOINT IMMEDIATELY after testing.

---

Agent Pickup Instructions

# Bootstrap
cd projects/confident-prep/p3-07-model-serving
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Ensure p3-05 MLflow registry has "production" alias

# Deploy endpoint (starts costing money immediately)
python src/deploy.py

# Benchmark
python src/benchmark.py --endpoint-name $(cat .endpoint-name)

# Cold start test
python src/cold_start.py --endpoint-name $(cat .endpoint-name)

# RUN TEARDOWN WHEN DONE — endpoint costs money while running
bash scripts/teardown.sh

Done when:

• python src/deploy.py creates endpoint and writes name to .endpoint-name
• Endpoint status is InService in SageMaker console
• python src/benchmark.py prints p50/p95/p99 latency from 100 invocations
• python src/cold_start.py prints first-invocation latency after endpoint recreation
• python src/cost_logger.py prints total cost since endpoint creation
• pytest tests/ -v shows 4/4 passing
• bash scripts/teardown.sh removes endpoint, endpoint config, and SageMaker model
• Verify teardown: aws sagemaker list-endpoints --name-contains p3-07 returns empty

---

What this project is

You deploy the best model from p3-05's MLflow registry as a real-time SageMaker endpoint with a custom inference script. You benchmark it with 100 synchronous invocations and compute p50/p95/p99 latency. You measure cold start: delete the endpoint, recreate it, measure time from creation call to first successful prediction. You track cost from creation to teardown. The endpoint runs on ml.t2.medium — the smallest inference instance — to minimize cost during learning.

What the learner achieves

"I can deploy a model to a SageMaker real-time endpoint, benchmark its latency distribution, and measure cold start time — and I know what these numbers mean for a production SLA."

---

Folder structure

p3-07-model-serving/
├── src/
│   ├── deploy.py              # package model, create endpoint, write name to .endpoint-name
│   ├── inference.py           # SageMaker handlers: model_fn, input_fn, predict_fn, output_fn
│   ├── benchmark.py           # 100 synchronous invocations, compute p50/p95/p99
│   ├── cold_start.py          # delete endpoint, recreate, measure first-invocation latency
│   └── cost_logger.py         # compute cost from endpoint creation to now
├── scripts/
│   └── teardown.sh            # delete endpoint + config + model; verify deletion
├── tests/
│   ├── test_inference.py      # predict_fn returns probabilities, output_fn CSV format
│   └── test_benchmark.py      # 100 measurements, p95 < p99 always, cost calculation
├── .endpoint-name             # written by deploy.py; read by other scripts
├── .env.example
├── requirements.txt
└── README.md

---

.env.example

# AWS region
AWS_REGION=us-east-1

# S3 bucket for model artifacts
S3_BUCKET=my-ml-training-bucket

# IAM role ARN with SageMaker full access
SAGEMAKER_ROLE_ARN=arn:aws:iam::123456789012:role/SageMakerExecutionRole

# MLflow tracking URI (to load "production" model)
MLFLOW_TRACKING_URI=http://localhost:5000

# MLflow model name
MLFLOW_MODEL_NAME=adult-income-xgboost

# SageMaker resource names (must be globally unique within your account/region)
SAGEMAKER_MODEL_NAME=p3-07-adult-income-model
SAGEMAKER_ENDPOINT_CONFIG_NAME=p3-07-adult-income-config
SAGEMAKER_ENDPOINT_NAME=p3-07-adult-income-endpoint

# Inference instance type (ml.t2.medium = cheapest option)
INFERENCE_INSTANCE_TYPE=ml.t2.medium

---

requirements.txt

boto3==1.35.93
sagemaker==2.236.0
xgboost==2.1.3
mlflow==2.19.0
pandas==2.2.3
numpy==2.2.1
python-dotenv==1.0.1
pytest==8.3.4

---

src/ — what to implement

src/deploy.py

CLI: python src/deploy.py [--instance-type ml.t2.medium]

load_and_package_model(mlflow_model_name: str, alias: str, s3_bucket: str, s3_prefix: str) -> str

• Loads native XGBClassifier from MLflow using mlflow.xgboost.load_model(f"models:/{name}@{alias}")
• Saves as model.xgb in a temp dir
• Copies src/inference.py to the same temp dir
• Creates model.tar.gz with both files at root
• Uploads to s3://{s3_bucket}/{s3_prefix}/model.tar.gz
• Returns the S3 URI

create_sagemaker_model(model_s3_uri: str, role_arn: str, model_name: str, region: str) -> None

• Calls boto3.client('sagemaker').create_model()
• Container: sagemaker.image_uris.retrieve("xgboost", region, version="1.7-1")
• Env: {"SAGEMAKER_PROGRAM": "inference.py"}

create_endpoint_config(model_name: str, config_name: str, instance_type: str) -> None

• Calls create_endpoint_config() with a single ProductionVariant:
  • VariantName: "primary", InitialInstanceCount: 1, InitialVariantWeight: 1

create_endpoint(config_name: str, endpoint_name: str) -> None

• Calls create_endpoint()
• Waits for InService status using waiter = sm.get_waiter('endpoint_in_service')
• Waiter timeout: 15 minutes

main()

1. Load and package model → upload to S3
2. Create SageMaker model resource
3. Create endpoint config
4. Create endpoint (wait for InService)
5. Write endpoint name to .endpoint-name file: open(".endpoint-name", "w").write(endpoint_name)
6. Record creation timestamp to .endpoint-created-at file (ISO format)
7. Print: Endpoint {endpoint_name} is InService. Ready for inference.

---

src/inference.py

model_fn(model_dir: str) -> xgb.XGBClassifier

• Loads {model_dir}/model.xgb
• Returns fitted model

input_fn(request_body: str | bytes, content_type: str) -> np.ndarray

• Accepts "text/csv"
• Parses CSV (single row or multiple rows)
• Returns 2D numpy array
• Raises ValueError for unsupported content types

predict_fn(input_data: np.ndarray, model: xgb.XGBClassifier) -> list[float]

• Calls model.predict_proba(input_data)[:, 1].tolist()
• Returns a list of floats (probabilities)
• Edge case: if input_data.ndim == 1, reshape to (1, -1) before predicting

output_fn(prediction: list[float], accept: str) -> tuple[str, str]

• For "text/csv": returns (",".join(str(p) for p in prediction), "text/csv")
• For "application/json": returns (json.dumps({"predictions": prediction}), "application/json")
• Default: CSV

---

src/benchmark.py

CLI: python src/benchmark.py --endpoint-name <name> [--n-invocations 100] [--data data/adult.data]

create_sample_payload(data_path: str, n_rows: int = 1) -> str

• Loads n_rows rows from adult.data, preprocesses (column names, dummies)
• Returns CSV string of the feature row(s)

invoke_once(runtime_client, endpoint_name: str, payload: str) -> tuple[float, str]

• Records time before and after runtime_client.invoke_endpoint()
• Returns (latency_ms, response_body)

benchmark_endpoint(endpoint_name: str, data_path: str, n_invocations: int = 100) -> dict

• Runs n_invocations sequential invocations using same payload (single row CSV)
• Collects list of latency_ms values
• Computes: p50, p95, p99 using statistics.quantiles(latencies, n=100) indices [49, 94, 98]
• Returns {"p50_ms": float, "p95_ms": float, "p99_ms": float, "n": int, "latencies": list[float]}

print_benchmark_report(results: dict) -> None

• Prints:

Benchmark Results (n={n} invocations)
p50 latency: {p50:.1f} ms
p95 latency: {p95:.1f} ms
p99 latency: {p99:.1f} ms

main()

• Parse args → benchmark_endpoint → print_benchmark_report
• Save raw latency list to output/latencies.json

---

src/cold_start.py

CLI: python src/cold_start.py --endpoint-name <name>

delete_endpoint(sm_client, endpoint_name: str) -> None

• Calls sm_client.delete_endpoint(EndpointName=endpoint_name)
• Waits until endpoint no longer exists using sm_client.get_waiter('endpoint_deleted')

time_to_first_prediction(endpoint_name: str, config_name: str, payload: str) -> dict

• Records t_start = time.monotonic()
• Calls sm_client.create_endpoint(EndpointName=endpoint_name, EndpointConfigName=config_name)
• Polls with sm_client.describe_endpoint until status == InService
• Immediately invokes runtime_client.invoke_endpoint()
• Records t_end = time.monotonic()
• Returns {"time_to_inservice_seconds": float, "time_to_first_prediction_seconds": float}

main()

• Read endpoint name and config name from .endpoint-name and env
• Create sample payload
• delete_endpoint → wait
• time_to_first_prediction
• Print:

  Cold start measurement:
    Time to InService: {t_inservice:.1f}s
    Time to first prediction: {t_first:.1f}s
  

• Write .endpoint-name again (endpoint was re-created)

---

src/cost_logger.py

CLI: python src/cost_logger.py [--instance-type ml.t2.medium]

INSTANCE_PRICES: dict[str, float] — same as prior projects plus "ml.t2.medium": 0.056

compute_cost(instance_type: str, created_at_iso: str) -> dict

• Reads creation time from arg or .endpoint-created-at file
• Computes elapsed hours since creation
• Returns {"elapsed_hours": float, "cost_usd": float, "instance_type": str}

main()

• Read created_at from .endpoint-created-at
• Compute and print:

  Endpoint running for {elapsed_hours:.2f} hours
  Instance: {instance_type} (${rate}/hr)
  Total cost: ${cost:.4f}
  

---

tests/ — what to test

tests/test_inference.py

test_predict_fn_returns_probabilities_as_list

• Create a tiny XGBClassifier (fit on 10 rows)
• Call inference.predict_fn(np.random.rand(5, 10), model)
• Assert result is a list
• Assert len == 5
• Assert all values in [0.0, 1.0]

test_predict_fn_reshapes_1d_input

• Create 1D array with shape (10,)
• Call inference.predict_fn(arr, model) — should not raise
• Assert result has length 1

test_output_fn_csv_format

• Call inference.output_fn([0.82, 0.31], "text/csv")
• Assert result[0] == "0.82,0.31"
• Assert result[1] == "text/csv"

tests/test_benchmark.py

test_benchmark_produces_exactly_n_measurements

• Mock invoke_once to return (50.0, "0.5") every call
• Call benchmark_endpoint(endpoint_name="mock", data_path="mock", n_invocations=100) with the mock
• Assert results["n"] == 100
• Assert len(results["latencies"]) == 100

test_p95_always_less_than_or_equal_to_p99

• Generate 100 random latency values
• Manually compute p95 and p99 using same quantile method
• Assert p95 <= p99 always

test_cost_logger_calculates_correctly

• Call cost_logger.compute_cost("ml.t2.medium", "2025-06-16T12:00:00Z") with known elapsed time
• Assert cost matches manual calculation: elapsed_hours * 0.056

---

README.md content

# P3-07 — Model Serving

Deploy a real-time SageMaker endpoint, benchmark latency (p50/p95/p99), and measure cold start.

## WARNING

This project creates a SageMaker endpoint that costs ~$0.056/hour (ml.t2.medium).
Run `bash scripts/teardown.sh` immediately after you finish testing.

## Prerequisites

- Python 3.11+
- AWS credentials configured
- p3-05 completed: MLflow registry has "production" alias
- MLflow server running at http://localhost:5000

## Quick start

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env  # fill in S3_BUCKET, SAGEMAKER_ROLE_ARN

python src/deploy.py
python src/benchmark.py --endpoint-name $(cat .endpoint-name)
python src/cost_logger.py

Cold start test

python src/cold_start.py --endpoint-name $(cat .endpoint-name)

TEARDOWN (run this before closing your session)

bash scripts/teardown.sh

Tests

pytest tests/ -v

Tests do not make AWS API calls (mocked).

---

## GUIDE.md content

```markdown
# Guide — P3-07 Model Serving

## What you will observe

**p50 latency** (median): most requests see this. For a simple XGBoost model on ml.t2.medium,
expect 10-30ms.

**p95 latency:** 5% of requests take longer than this. This is typically the SLA target for
production ML services.

**p99 latency:** 1% of requests take longer. This is where you see GC pauses, CPU contention,
and cold inference paths.

**Cold start:** Creating a fresh endpoint takes 3-5 minutes for SageMaker to provision the
instance and load your container. The first inference after creation runs the container cold —
model loading happens on the first `model_fn` call, adding 1-2 seconds.

## Latency vs throughput

`benchmark.py` sends invocations sequentially (one at a time). This measures latency, not
throughput. If you want throughput, you send parallel requests — but that requires multiple
threads and a larger instance. Sequential benchmarking is the right method for SLA validation.

## Why ml.t2.medium for learning

ml.t2.medium ($0.056/hr) is the cheapest SageMaker inference instance. A simple XGBoost
model fits easily in 4GB RAM. For production, you would use ml.c5.large or larger, or
enable auto-scaling. For this project, t2.medium lets you learn without running up a bill.

## The cost of always-on

An endpoint running 24/7 on ml.t2.medium costs $0.056 × 24 × 30 = $40.32/month.
For 10,000 predictions/day at 100 RPM, you would need the endpoint running 10,000/100/60 = 1.7 hours/day
= $0.095/day = $2.85/month. Batch inference for the same volume costs ~$0.02/day. The tradeoff:
real-time latency vs batch economics.

## Interview framing

"I can deploy a SageMaker endpoint, benchmark its p50/p95/p99 latency distribution, and
measure cold start time. I know that p95 is the typical SLA target, that sequential
benchmarking measures latency not throughput, and that endpoint cost is always-on — so the
batch vs real-time decision is driven by latency requirements, not just volume."

---

Failure modes to handle

Failure	Where	How to handle
Endpoint creation times out (>15 min)	deploy.py: create_endpoint	Waiter has 15-min timeout; if exceeded, check CloudWatch logs for container errors
invoke_endpoint returns non-200 status	benchmark.py: invoke_once	Check response["ResponseMetadata"]["HTTPStatusCode"]; log failures separately from latency list
.endpoint-created-at missing	cost_logger.py	Fall back to current time (cost = 0); warn that creation time was not recorded
Endpoint already deleted when teardown runs	scripts/teardown.sh	Use --query ... --output text and `
OHE column count mismatch at inference	inference.py: input_fn	Document: inference.py must use same feature columns as training; mismatches produce silent wrong predictions
cold_start.py fails to wait for deletion	cold_start.py: delete_endpoint	Waiter for endpoint_deleted with 10-min timeout

---

The metric this project measures

What: Endpoint invocation latency (p50, p95, p99) over 100 sequential requests

Format:

p50 latency: 18.3 ms
p95 latency: 42.1 ms
p99 latency: 71.8 ms

Target: p50 < 50ms, p95 < 100ms for a simple XGBoost model on ml.t2.medium with no pre-processing in the inference script

Secondary metric: Cold start time to first prediction, format {seconds:.1f}s, expected 180-300s for SageMaker endpoint provisioning

---

Cost estimate

Resource	Qty	Rate	Estimated cost
ml.t2.medium endpoint (2 hours)	1 endpoint	$0.056/hr	~$0.11
S3 model artifact (< 10 MB)	1 object	$0.023/GB-month	<$0.01
SageMaker API calls	~50 calls	Free tier	$0.00
Total per session			~$0.10

Teardown checklist

scripts/teardown.sh must (in this order):

• aws sagemaker delete-endpoint --endpoint-name ${SAGEMAKER_ENDPOINT_NAME}
• Wait for deletion: aws sagemaker wait endpoint-deleted --endpoint-name ${SAGEMAKER_ENDPOINT_NAME}
• aws sagemaker delete-endpoint-config --endpoint-config-name ${SAGEMAKER_ENDPOINT_CONFIG_NAME}
• aws sagemaker delete-model --model-name ${SAGEMAKER_MODEL_NAME}
• aws s3 rm s3://${S3_BUCKET}/p3-07/ --recursive
• Print Endpoint deleted. All resources cleaned up.
• Verify: aws sagemaker describe-endpoint --endpoint-name ${SAGEMAKER_ENDPOINT_NAME} should fail