--- title: "Model Serving" description: "Benchmarked p50/p95/p99 latency on a real SageMaker endpoint, including cold start, is the exact metric a hiring manager checks for when screening for..." source: "https://confidentprep.com/paths/ml-engineering-on-aws/model-serving/" path: "ML Engineering on AWS" repository: "https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/model-serving" token_estimate: 4245 --- # Model Serving ## Overview Deploy a real-time SageMaker endpoint, benchmark latency at p50/p95/p99, and measure cold start. This project costs roughly $0.056/hour while the endpoint is up — it ships with a teardown script; run it the moment you are done testing. This is the live endpoint the next two projects (A/B testing, monitoring) build directly on top of. ## What to do **Path:** 3 — ML Engineering on AWS **Position:** 7 of 12 **Difficulty:** 🔴 **Time:** 3-4h **AWS Cost:** ~$0.05/hour × 2 hours = ~$0.10 per session. DELETE ENDPOINT IMMEDIATELY after testing. --- ### Agent Pickup Instructions ```bash # Bootstrap cd projects/confident-prep/p3-07-model-serving python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt # Ensure p3-05 MLflow registry has "production" alias # Deploy endpoint (starts costing money immediately) python src/deploy.py # Benchmark python src/benchmark.py --endpoint-name $(cat .endpoint-name) # Cold start test python src/cold_start.py --endpoint-name $(cat .endpoint-name) # RUN TEARDOWN WHEN DONE — endpoint costs money while running bash scripts/teardown.sh ``` **Done when:** - [ ] `python src/deploy.py` creates endpoint and writes name to `.endpoint-name` - [ ] Endpoint status is `InService` in SageMaker console - [ ] `python src/benchmark.py` prints p50/p95/p99 latency from 100 invocations - [ ] `python src/cold_start.py` prints first-invocation latency after endpoint recreation - [ ] `python src/cost_logger.py` prints total cost since endpoint creation - [ ] `pytest tests/ -v` shows 4/4 passing - [ ] `bash scripts/teardown.sh` removes endpoint, endpoint config, and SageMaker model - [ ] Verify teardown: `aws sagemaker list-endpoints --name-contains p3-07` returns empty --- ### What this project is You deploy the best model from p3-05's MLflow registry as a real-time SageMaker endpoint with a custom inference script. You benchmark it with 100 synchronous invocations and compute p50/p95/p99 latency. You measure cold start: delete the endpoint, recreate it, measure time from creation call to first successful prediction. You track cost from creation to teardown. The endpoint runs on ml.t2.medium — the smallest inference instance — to minimize cost during learning. ### What the learner achieves "I can deploy a model to a SageMaker real-time endpoint, benchmark its latency distribution, and measure cold start time — and I know what these numbers mean for a production SLA." --- ### Folder structure ``` p3-07-model-serving/ ├── src/ │ ├── deploy.py # package model, create endpoint, write name to .endpoint-name │ ├── inference.py # SageMaker handlers: model_fn, input_fn, predict_fn, output_fn │ ├── benchmark.py # 100 synchronous invocations, compute p50/p95/p99 │ ├── cold_start.py # delete endpoint, recreate, measure first-invocation latency │ └── cost_logger.py # compute cost from endpoint creation to now ├── scripts/ │ └── teardown.sh # delete endpoint + config + model; verify deletion ├── tests/ │ ├── test_inference.py # predict_fn returns probabilities, output_fn CSV format │ └── test_benchmark.py # 100 measurements, p95 < p99 always, cost calculation ├── .endpoint-name # written by deploy.py; read by other scripts ├── .env.example ├── requirements.txt └── README.md ``` --- ### .env.example ```dotenv # AWS region AWS_REGION=us-east-1 # S3 bucket for model artifacts S3_BUCKET=my-ml-training-bucket # IAM role ARN with SageMaker full access SAGEMAKER_ROLE_ARN=arn:aws:iam::123456789012:role/SageMakerExecutionRole # MLflow tracking URI (to load "production" model) MLFLOW_TRACKING_URI=http://localhost:5000 # MLflow model name MLFLOW_MODEL_NAME=adult-income-xgboost # SageMaker resource names (must be globally unique within your account/region) SAGEMAKER_MODEL_NAME=p3-07-adult-income-model SAGEMAKER_ENDPOINT_CONFIG_NAME=p3-07-adult-income-config SAGEMAKER_ENDPOINT_NAME=p3-07-adult-income-endpoint # Inference instance type (ml.t2.medium = cheapest option) INFERENCE_INSTANCE_TYPE=ml.t2.medium ``` --- ### requirements.txt ``` boto3==1.35.93 sagemaker==2.236.0 xgboost==2.1.3 mlflow==2.19.0 pandas==2.2.3 numpy==2.2.1 python-dotenv==1.0.1 pytest==8.3.4 ``` --- ### src/ — what to implement #### `src/deploy.py` **CLI:** `python src/deploy.py [--instance-type ml.t2.medium]` **`load_and_package_model(mlflow_model_name: str, alias: str, s3_bucket: str, s3_prefix: str) -> str`** - Loads native XGBClassifier from MLflow using `mlflow.xgboost.load_model(f"models:/{name}@{alias}")` - Saves as `model.xgb` in a temp dir - Copies `src/inference.py` to the same temp dir - Creates `model.tar.gz` with both files at root - Uploads to `s3://{s3_bucket}/{s3_prefix}/model.tar.gz` - Returns the S3 URI **`create_sagemaker_model(model_s3_uri: str, role_arn: str, model_name: str, region: str) -> None`** - Calls `boto3.client('sagemaker').create_model()` - Container: `sagemaker.image_uris.retrieve("xgboost", region, version="1.7-1")` - Env: `{"SAGEMAKER_PROGRAM": "inference.py"}` **`create_endpoint_config(model_name: str, config_name: str, instance_type: str) -> None`** - Calls `create_endpoint_config()` with a single `ProductionVariant`: - `VariantName: "primary"`, `InitialInstanceCount: 1`, `InitialVariantWeight: 1` **`create_endpoint(config_name: str, endpoint_name: str) -> None`** - Calls `create_endpoint()` - Waits for `InService` status using `waiter = sm.get_waiter('endpoint_in_service')` - Waiter timeout: 15 minutes **`main()`** 1. Load and package model → upload to S3 2. Create SageMaker model resource 3. Create endpoint config 4. Create endpoint (wait for InService) 5. Write endpoint name to `.endpoint-name` file: `open(".endpoint-name", "w").write(endpoint_name)` 6. Record creation timestamp to `.endpoint-created-at` file (ISO format) 7. Print: `Endpoint {endpoint_name} is InService. Ready for inference.` --- #### `src/inference.py` **`model_fn(model_dir: str) -> xgb.XGBClassifier`** - Loads `{model_dir}/model.xgb` - Returns fitted model **`input_fn(request_body: str | bytes, content_type: str) -> np.ndarray`** - Accepts `"text/csv"` - Parses CSV (single row or multiple rows) - Returns 2D numpy array - Raises `ValueError` for unsupported content types **`predict_fn(input_data: np.ndarray, model: xgb.XGBClassifier) -> list[float]`** - Calls `model.predict_proba(input_data)[:, 1].tolist()` - Returns a list of floats (probabilities) - Edge case: if `input_data.ndim == 1`, reshape to `(1, -1)` before predicting **`output_fn(prediction: list[float], accept: str) -> tuple[str, str]`** - For `"text/csv"`: returns `(",".join(str(p) for p in prediction), "text/csv")` - For `"application/json"`: returns `(json.dumps({"predictions": prediction}), "application/json")` - Default: CSV --- #### `src/benchmark.py` **CLI:** `python src/benchmark.py --endpoint-name [--n-invocations 100] [--data data/adult.data]` **`create_sample_payload(data_path: str, n_rows: int = 1) -> str`** - Loads `n_rows` rows from `adult.data`, preprocesses (column names, dummies) - Returns CSV string of the feature row(s) **`invoke_once(runtime_client, endpoint_name: str, payload: str) -> tuple[float, str]`** - Records time before and after `runtime_client.invoke_endpoint()` - Returns `(latency_ms, response_body)` **`benchmark_endpoint(endpoint_name: str, data_path: str, n_invocations: int = 100) -> dict`** - Runs `n_invocations` sequential invocations using same payload (single row CSV) - Collects list of latency_ms values - Computes: p50, p95, p99 using `statistics.quantiles(latencies, n=100)` indices [49, 94, 98] - Returns `{"p50_ms": float, "p95_ms": float, "p99_ms": float, "n": int, "latencies": list[float]}` **`print_benchmark_report(results: dict) -> None`** - Prints: ``` Benchmark Results (n={n} invocations) p50 latency: {p50:.1f} ms p95 latency: {p95:.1f} ms p99 latency: {p99:.1f} ms ``` **`main()`** - Parse args → `benchmark_endpoint` → `print_benchmark_report` - Save raw latency list to `output/latencies.json` --- #### `src/cold_start.py` **CLI:** `python src/cold_start.py --endpoint-name ` **`delete_endpoint(sm_client, endpoint_name: str) -> None`** - Calls `sm_client.delete_endpoint(EndpointName=endpoint_name)` - Waits until endpoint no longer exists using `sm_client.get_waiter('endpoint_deleted')` **`time_to_first_prediction(endpoint_name: str, config_name: str, payload: str) -> dict`** - Records `t_start = time.monotonic()` - Calls `sm_client.create_endpoint(EndpointName=endpoint_name, EndpointConfigName=config_name)` - Polls with `sm_client.describe_endpoint` until status == `InService` - Immediately invokes `runtime_client.invoke_endpoint()` - Records `t_end = time.monotonic()` - Returns `{"time_to_inservice_seconds": float, "time_to_first_prediction_seconds": float}` **`main()`** - Read endpoint name and config name from `.endpoint-name` and env - Create sample payload - `delete_endpoint` → wait - `time_to_first_prediction` - Print: ``` Cold start measurement: Time to InService: {t_inservice:.1f}s Time to first prediction: {t_first:.1f}s ``` - Write `.endpoint-name` again (endpoint was re-created) --- #### `src/cost_logger.py` **CLI:** `python src/cost_logger.py [--instance-type ml.t2.medium]` **`INSTANCE_PRICES: dict[str, float]`** — same as prior projects plus `"ml.t2.medium": 0.056` **`compute_cost(instance_type: str, created_at_iso: str) -> dict`** - Reads creation time from arg or `.endpoint-created-at` file - Computes elapsed hours since creation - Returns `{"elapsed_hours": float, "cost_usd": float, "instance_type": str}` **`main()`** - Read `created_at` from `.endpoint-created-at` - Compute and print: ``` Endpoint running for {elapsed_hours:.2f} hours Instance: {instance_type} (${rate}/hr) Total cost: ${cost:.4f} ``` --- ### tests/ — what to test #### `tests/test_inference.py` **`test_predict_fn_returns_probabilities_as_list`** - Create a tiny XGBClassifier (fit on 10 rows) - Call `inference.predict_fn(np.random.rand(5, 10), model)` - Assert result is a list - Assert len == 5 - Assert all values in `[0.0, 1.0]` **`test_predict_fn_reshapes_1d_input`** - Create 1D array with shape `(10,)` - Call `inference.predict_fn(arr, model)` — should not raise - Assert result has length 1 **`test_output_fn_csv_format`** - Call `inference.output_fn([0.82, 0.31], "text/csv")` - Assert result[0] == `"0.82,0.31"` - Assert result[1] == `"text/csv"` #### `tests/test_benchmark.py` **`test_benchmark_produces_exactly_n_measurements`** - Mock `invoke_once` to return `(50.0, "0.5")` every call - Call `benchmark_endpoint(endpoint_name="mock", data_path="mock", n_invocations=100)` with the mock - Assert `results["n"] == 100` - Assert `len(results["latencies"]) == 100` **`test_p95_always_less_than_or_equal_to_p99`** - Generate 100 random latency values - Manually compute p95 and p99 using same quantile method - Assert `p95 <= p99` always **`test_cost_logger_calculates_correctly`** - Call `cost_logger.compute_cost("ml.t2.medium", "2025-06-16T12:00:00Z")` with known elapsed time - Assert cost matches manual calculation: `elapsed_hours * 0.056` --- ### README.md content ```markdown # P3-07 — Model Serving Deploy a real-time SageMaker endpoint, benchmark latency (p50/p95/p99), and measure cold start. ## WARNING This project creates a SageMaker endpoint that costs ~$0.056/hour (ml.t2.medium). Run `bash scripts/teardown.sh` immediately after you finish testing. ## Prerequisites - Python 3.11+ - AWS credentials configured - p3-05 completed: MLflow registry has "production" alias - MLflow server running at http://localhost:5000 ## Quick start ```bash python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt cp .env.example .env # fill in S3_BUCKET, SAGEMAKER_ROLE_ARN python src/deploy.py python src/benchmark.py --endpoint-name $(cat .endpoint-name) python src/cost_logger.py ``` ## Cold start test ```bash python src/cold_start.py --endpoint-name $(cat .endpoint-name) ``` ## TEARDOWN (run this before closing your session) ```bash bash scripts/teardown.sh ``` ## Tests ```bash pytest tests/ -v ``` Tests do not make AWS API calls (mocked). ``` --- ### GUIDE.md content ```markdown # Guide — P3-07 Model Serving ## What you will observe **p50 latency** (median): most requests see this. For a simple XGBoost model on ml.t2.medium, expect 10-30ms. **p95 latency:** 5% of requests take longer than this. This is typically the SLA target for production ML services. **p99 latency:** 1% of requests take longer. This is where you see GC pauses, CPU contention, and cold inference paths. **Cold start:** Creating a fresh endpoint takes 3-5 minutes for SageMaker to provision the instance and load your container. The first inference after creation runs the container cold — model loading happens on the first `model_fn` call, adding 1-2 seconds. ## Latency vs throughput `benchmark.py` sends invocations sequentially (one at a time). This measures latency, not throughput. If you want throughput, you send parallel requests — but that requires multiple threads and a larger instance. Sequential benchmarking is the right method for SLA validation. ## Why ml.t2.medium for learning ml.t2.medium ($0.056/hr) is the cheapest SageMaker inference instance. A simple XGBoost model fits easily in 4GB RAM. For production, you would use ml.c5.large or larger, or enable auto-scaling. For this project, t2.medium lets you learn without running up a bill. ## The cost of always-on An endpoint running 24/7 on ml.t2.medium costs $0.056 × 24 × 30 = $40.32/month. For 10,000 predictions/day at 100 RPM, you would need the endpoint running 10,000/100/60 = 1.7 hours/day = $0.095/day = $2.85/month. Batch inference for the same volume costs ~$0.02/day. The tradeoff: real-time latency vs batch economics. ## Interview framing "I can deploy a SageMaker endpoint, benchmark its p50/p95/p99 latency distribution, and measure cold start time. I know that p95 is the typical SLA target, that sequential benchmarking measures latency not throughput, and that endpoint cost is always-on — so the batch vs real-time decision is driven by latency requirements, not just volume." ``` --- ### Failure modes to handle | Failure | Where | How to handle | |---|---|---| | Endpoint creation times out (>15 min) | `deploy.py: create_endpoint` | Waiter has 15-min timeout; if exceeded, check CloudWatch logs for container errors | | `invoke_endpoint` returns non-200 status | `benchmark.py: invoke_once` | Check `response["ResponseMetadata"]["HTTPStatusCode"]`; log failures separately from latency list | | `.endpoint-created-at` missing | `cost_logger.py` | Fall back to current time (cost = 0); warn that creation time was not recorded | | Endpoint already deleted when teardown runs | `scripts/teardown.sh` | Use `--query ... --output text` and `|| true` to ignore not-found errors | | OHE column count mismatch at inference | `inference.py: input_fn` | Document: inference.py must use same feature columns as training; mismatches produce silent wrong predictions | | `cold_start.py` fails to wait for deletion | `cold_start.py: delete_endpoint` | Waiter for `endpoint_deleted` with 10-min timeout | --- ### The metric this project measures **What:** Endpoint invocation latency (p50, p95, p99) over 100 sequential requests **Format:** ``` p50 latency: 18.3 ms p95 latency: 42.1 ms p99 latency: 71.8 ms ``` **Target:** p50 < 50ms, p95 < 100ms for a simple XGBoost model on ml.t2.medium with no pre-processing in the inference script **Secondary metric:** Cold start time to first prediction, format `{seconds:.1f}s`, expected 180-300s for SageMaker endpoint provisioning --- ### Cost estimate | Resource | Qty | Rate | Estimated cost | |---|---|---|---| | ml.t2.medium endpoint (2 hours) | 1 endpoint | $0.056/hr | ~$0.11 | | S3 model artifact (< 10 MB) | 1 object | $0.023/GB-month | <$0.01 | | SageMaker API calls | ~50 calls | Free tier | $0.00 | | **Total per session** | | | **~$0.10** | ### Teardown checklist `scripts/teardown.sh` must (in this order): - [ ] `aws sagemaker delete-endpoint --endpoint-name ${SAGEMAKER_ENDPOINT_NAME}` - [ ] Wait for deletion: `aws sagemaker wait endpoint-deleted --endpoint-name ${SAGEMAKER_ENDPOINT_NAME}` - [ ] `aws sagemaker delete-endpoint-config --endpoint-config-name ${SAGEMAKER_ENDPOINT_CONFIG_NAME}` - [ ] `aws sagemaker delete-model --model-name ${SAGEMAKER_MODEL_NAME}` - [ ] `aws s3 rm s3://${S3_BUCKET}/p3-07/ --recursive` - [ ] Print `Endpoint deleted. All resources cleaned up.` - [ ] Verify: `aws sagemaker describe-endpoint --endpoint-name ${SAGEMAKER_ENDPOINT_NAME}` should fail ## Source code A reference implementation of Model Serving — README, guide, source and tests — is at https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/model-serving.