Manual Versioning

Path: 3 — ML Engineering on AWS Position: 4 of 12 Difficulty: 🟢 (pure local, no cloud) Time: 2-3h AWS Cost: None

---

Agent Pickup Instructions

# Bootstrap
cd projects/confident-prep/p3-04-manual-versioning
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Download dataset (same as prior projects)
mkdir -p data
curl -o data/adult.data \
  "https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data"

# Train all 3 variants
python src/train_all.py

# Compare results
python src/compare.py

# Train a single variant manually
python src/train_variant.py --max-depth 5 --learning-rate 0.1 --data data/adult.data

# Run tests
pytest tests/ -v

Done when:

• python src/train_all.py creates 3 subdirectories under models/, each with model.xgb and metadata.json
• python src/compare.py prints a table showing all 3 runs with their metrics
• Each metadata.json has all required keys: run_id, hyperparams, metrics, model_path, created_at
• pytest tests/ -v shows 4/4 passing
• No two run_id values are identical

---

What this project is

You train three XGBoost model variants with different hyperparameters (max_depth and learning_rate). Each variant saves its model file and a metadata.json to a timestamped directory under models/. A comparison script scans all metadata files and prints a ranked table. You feel the friction of this approach — hunting for the right JSON file, no diff between runs, no central server — which is exactly the motivation for MLflow in p3-05.

What the learner achieves

"I know what manual experiment tracking looks like and why it breaks down — and I can articulate exactly what MLflow solves compared to a folder of JSON files."

---

Folder structure

p3-04-manual-versioning/
├── data/
│   └── adult.data              # raw UCI dataset
├── models/                     # created by training scripts
│   ├── run_20250616_143022/
│   │   ├── model.xgb
│   │   └── metadata.json
│   ├── run_20250616_143156/
│   │   ├── model.xgb
│   │   └── metadata.json
│   └── run_20250616_143341/
│       ├── model.xgb
│       └── metadata.json
├── src/
│   ├── train_variant.py        # CLI: train one variant, save to timestamped dir
│   ├── compare.py              # scan models/, print comparison table
│   └── train_all.py           # run 3 hardcoded variants sequentially
├── tests/
│   ├── test_metadata.py        # metadata.json schema, uniqueness, compare.py correctness
│   └── test_train_variant.py   # model file saved, AUC reasonable
├── .env.example
├── requirements.txt
└── README.md

---

.env.example

# No AWS credentials needed for this project.

# Local data path
LOCAL_DATA_DIR=data/

# Models output directory
MODELS_DIR=models/

---

requirements.txt

xgboost==2.1.3
scikit-learn==1.6.0
pandas==2.2.3
numpy==2.2.1
pytest==8.3.4
python-dotenv==1.0.1

---

src/ — what to implement

src/train_variant.py

CLI arguments:

• --max-depth (int, required)
• --learning-rate (float, required)
• --n-estimators (int, default 100)
• --data (str, default data/adult.data)
• --models-dir (str, default models/)

generate_run_id() -> str

• Returns "run_{timestamp}" where timestamp is datetime.now().strftime("%Y%m%d_%H%M%S")
• Called once per training run

load_and_prepare(data_path: str) -> tuple

• Reads UCI Adult data (same loading logic as p3-01: assign column names, strip whitespace, replace ? with NaN, drop NaN rows, pd.get_dummies, target encode)
• Returns (X_train, X_test, y_train, y_test) with 80/20 split, random_state=42

train_and_evaluate(X_train, y_train, X_test, y_test, hyperparams: dict) -> dict

• Fits XGBClassifier(use_label_encoder=False, eval_metric='logloss', **hyperparams)
• Measures wall-clock training time with time.perf_counter()
• Computes roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])
• Returns {"model": fitted_model, "val_auc": float, "train_time_seconds": float}

save_run(model, run_id: str, hyperparams: dict, metrics: dict, models_dir: str) -> dict

• Creates directory {models_dir}/{run_id}/
• Saves model to {models_dir}/{run_id}/model.xgb
• Builds metadata dict (see schema below)
• Writes metadata dict as JSON to {models_dir}/{run_id}/metadata.json
• Returns the metadata dict

metadata.json schema (must match exactly):

{
  "run_id": "run_20250616_143022",
  "hyperparams": {
    "max_depth": 3,
    "learning_rate": 0.1,
    "n_estimators": 100
  },
  "metrics": {
    "val_auc": 0.8762,
    "train_time_seconds": 4.21
  },
  "model_path": "models/run_20250616_143022/model.xgb",
  "created_at": "2025-06-16T14:30:22Z"
}

• created_at format: datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ")
• val_auc rounded to 4 decimal places
• train_time_seconds rounded to 2 decimal places

main()

• Parse args, run load_and_prepare → train_and_evaluate → save_run
• Print: Run {run_id} complete. val_auc={val_auc:.4f}, train_time={train_time:.2f}s
• Print: Saved to {model_path}

---

src/compare.py

CLI: python src/compare.py [--models-dir models/] [--sort-by val_auc]

find_all_runs(models_dir: str) -> list[dict]

• Scans models_dir for all metadata.json files (one level deep: models/*/metadata.json)
• Loads each JSON
• Returns list of metadata dicts, sorted by run_id (ascending, i.e., chronological)

format_comparison_table(runs: list[dict]) -> str

• Builds a plain-text table:

run_id                  | max_depth | learning_rate | val_auc | train_time_s
------------------------|-----------|---------------|---------|-------------
run_20250616_143022     |         3 |         0.100 |  0.8762 |         4.21
run_20250616_143156     |         5 |         0.050 |  0.8891 |         6.43
run_20250616_143341     |         7 |         0.010 |  0.8623 |         9.87

• Mark the best row by val_auc with * prefix on run_id
• Add a summary line: Best run: {run_id} (val_auc={val_auc:.4f})

identify_best_run(runs: list[dict], metric: str = "val_auc") -> dict

• Returns the metadata dict with the highest value of metrics[metric]
• Raises ValueError if runs is empty

main()

• Call find_all_runs, format and print table, print best run summary

---

src/train_all.py

VARIANTS: list[dict]

VARIANTS = [
    {"max_depth": 3, "learning_rate": 0.1, "n_estimators": 100},
    {"max_depth": 5, "learning_rate": 0.05, "n_estimators": 100},
    {"max_depth": 7, "learning_rate": 0.01, "n_estimators": 200},
]

main()

• Iterates through VARIANTS
• For each: calls train_variant.main() via subprocess (so run_ids get different timestamps)
• Wait 2 seconds between runs to ensure unique timestamps: time.sleep(2)
• After all 3 complete, calls compare.main() to print the summary table
• Total prints: 3 training summaries + 1 comparison table

---

tests/ — what to test

tests/test_metadata.py

test_metadata_json_has_all_required_keys

• Run train_variant.save_run(mock_model, "run_test", hyperparams, metrics, tmp_path)
• Load {tmp_path}/run_test/metadata.json
• Assert all keys present: run_id, hyperparams, metrics, model_path, created_at
• Assert metrics contains val_auc and train_time_seconds
• Assert hyperparams contains max_depth, learning_rate, n_estimators

test_compare_finds_all_three_runs_after_train_all

• Run train_all.main() with a temp --models-dir
• Call compare.find_all_runs(models_dir)
• Assert len(runs) == 3

test_best_run_identified_correctly

• Construct 3 mock metadata dicts with val_auc: 0.85, 0.90, 0.87
• Call compare.identify_best_run(runs)
• Assert returned run has val_auc == 0.90

test_run_id_is_unique_per_run

• Run generate_run_id() twice with 1-second sleep between
• Assert the two run_ids are different strings

tests/test_train_variant.py

test_model_file_is_saved

• Call train_variant.main() with args pointing to data/adult.data and a temp models_dir
• Assert model.xgb file exists in the created run directory

test_val_auc_is_reasonable

• Same setup
• Load metadata.json, check metrics.val_auc > 0.80

---

README.md content

# P3-04 — Manual Versioning

Train three XGBoost variants, track results with timestamped folders and JSON metadata.

## Prerequisites

- Python 3.11+
- No AWS credentials needed

## Quick start

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
mkdir -p data && curl -o data/adult.data \
  "https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data"
python src/train_all.py

Train a single variant

python src/train_variant.py --max-depth 5 --learning-rate 0.05 --data data/adult.data

Compare all runs

python src/compare.py

Output

Each run creates models/run_{timestamp}/model.xgb and models/run_{timestamp}/metadata.json.

compare.py prints a table and marks the best run by val_AUC.

Tests

pytest tests/ -v

---

## GUIDE.md content

```markdown
# Guide — P3-04 Manual Versioning

## The problem you are about to feel

After running `train_all.py`, open `models/`. You have 3 directories with timestamps you
need to decode. If you want to know which used max_depth=5, you open 3 JSON files. If you
ran an extra experiment by hand, it is in a fourth directory. If a colleague ran experiments
on their machine, their folder is not here at all.

This friction is intentional. The next project (p3-05) installs MLflow and this problem
disappears. You will appreciate MLflow more for having done it the hard way first.

## What metadata.json gives you

Every run is self-describing. You can always reconstruct what happened by reading the JSON,
even if you lost the training logs. The `model_path` field lets `compare.py` find and load
the model for inference without searching.

## What metadata.json does NOT give you

- Git SHA of the code that produced the run (did you change train_variant.py between runs?)
- Input data hash (was `adult.data` the same file?)
- System metrics (CPU usage, memory peak)
- Artifacts other than the model (preprocessor, feature names)
- Search by arbitrary metadata fields

MLflow tracks all of these out of the box.

## The timestamp collision risk

`generate_run_id()` uses seconds precision. If two runs start in the same second (e.g., in
a parallel grid search), their directories would collide. The `train_all.py` script waits
2 seconds between runs to avoid this. In production, use a UUID or MLflow's run_id.

## Interview framing

"I've done manual experiment tracking with JSON metadata files and know exactly why it breaks
down: no central search, no input data versioning, no code versioning, collision risk in parallel
runs. That's why I use MLflow — not because it's trendy, but because I've felt what it replaces."

---

Failure modes to handle

Failure	Where	How to handle
Two runs start in the same second → same run_id	train_variant.py: generate_run_id	train_all.py waits 2 seconds between runs; document the risk
models/ directory does not exist	train_variant.py: save_run	os.makedirs(run_dir, exist_ok=True)
metadata.json has wrong val_auc type (string instead of float)	compare.py: identify_best_run	float(run["metrics"]["val_auc"]) to ensure numeric comparison
models/ contains a directory without metadata.json (partial run)	compare.py: find_all_runs	Skip directories where metadata.json does not exist; print warning
--data path does not exist	train_variant.py: load_and_prepare	FileNotFoundError with the path
models/ is empty when compare.py runs	compare.py: identify_best_run	Raise ValueError("No runs found in models/. Run train_all.py first.")

---

The metric this project measures

What: Validation AUC (ROC) per training run, plus comparison delta between best and worst variant

Format: Printed per-run as val_auc=0.8762 and in comparison table; delta printed as AUC spread: {max_auc - min_auc:.4f}

Target: All 3 variants should achieve val_AUC > 0.80. The max_depth=5 variant typically performs best on UCI Adult with these hyperparameters.