---
title: "Manual Versioning"
description: "Doing version tracking by hand first means you can explain why MLflow/registry tools exist, not just that you used one — a stronger interview answer than tool..."
source: "https://confidentprep.com/paths/ml-engineering-on-aws/manual-versioning/"
path: "ML Engineering on AWS"
repository: "https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/manual-versioning"
token_estimate: 3236
---

# Manual Versioning

## Overview

Train three XGBoost variants and track results yourself with timestamped folders and JSON
metadata — no framework. It is deliberately tedious, on purpose: the next project
introduces MLflow to solve exactly the problem this one makes you feel by hand.

## What to do

**Path:** 3 — ML Engineering on AWS
**Position:** 4 of 12
**Difficulty:** 🟢 (pure local, no cloud)
**Time:** 2-3h
**AWS Cost:** None

---

### Agent Pickup Instructions

```bash
# Bootstrap
cd projects/confident-prep/p3-04-manual-versioning
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Download dataset (same as prior projects)
mkdir -p data
curl -o data/adult.data \
  "https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data"

# Train all 3 variants
python src/train_all.py

# Compare results
python src/compare.py

# Train a single variant manually
python src/train_variant.py --max-depth 5 --learning-rate 0.1 --data data/adult.data

# Run tests
pytest tests/ -v
```

**Done when:**
- [ ] `python src/train_all.py` creates 3 subdirectories under `models/`, each with `model.xgb` and `metadata.json`
- [ ] `python src/compare.py` prints a table showing all 3 runs with their metrics
- [ ] Each `metadata.json` has all required keys: `run_id`, `hyperparams`, `metrics`, `model_path`, `created_at`
- [ ] `pytest tests/ -v` shows 4/4 passing
- [ ] No two `run_id` values are identical

---

### What this project is

You train three XGBoost model variants with different hyperparameters (max_depth and learning_rate). Each variant saves its model file and a `metadata.json` to a timestamped directory under `models/`. A comparison script scans all metadata files and prints a ranked table. You feel the friction of this approach — hunting for the right JSON file, no diff between runs, no central server — which is exactly the motivation for MLflow in p3-05.

### What the learner achieves

"I know what manual experiment tracking looks like and why it breaks down — and I can articulate exactly what MLflow solves compared to a folder of JSON files."

---

### Folder structure

```
p3-04-manual-versioning/
├── data/
│   └── adult.data              # raw UCI dataset
├── models/                     # created by training scripts
│   ├── run_20250616_143022/
│   │   ├── model.xgb
│   │   └── metadata.json
│   ├── run_20250616_143156/
│   │   ├── model.xgb
│   │   └── metadata.json
│   └── run_20250616_143341/
│       ├── model.xgb
│       └── metadata.json
├── src/
│   ├── train_variant.py        # CLI: train one variant, save to timestamped dir
│   ├── compare.py              # scan models/, print comparison table
│   └── train_all.py           # run 3 hardcoded variants sequentially
├── tests/
│   ├── test_metadata.py        # metadata.json schema, uniqueness, compare.py correctness
│   └── test_train_variant.py   # model file saved, AUC reasonable
├── .env.example
├── requirements.txt
└── README.md
```

---

### .env.example

```dotenv
# No AWS credentials needed for this project.

# Local data path
LOCAL_DATA_DIR=data/

# Models output directory
MODELS_DIR=models/
```

---

### requirements.txt

```
xgboost==2.1.3
scikit-learn==1.6.0
pandas==2.2.3
numpy==2.2.1
pytest==8.3.4
python-dotenv==1.0.1
```

---

### src/ — what to implement

#### `src/train_variant.py`

**CLI arguments:**
- `--max-depth` (int, required)
- `--learning-rate` (float, required)
- `--n-estimators` (int, default 100)
- `--data` (str, default `data/adult.data`)
- `--models-dir` (str, default `models/`)

**`generate_run_id() -> str`**
- Returns `"run_{timestamp}"` where timestamp is `datetime.now().strftime("%Y%m%d_%H%M%S")`
- Called once per training run

**`load_and_prepare(data_path: str) -> tuple`**
- Reads UCI Adult data (same loading logic as p3-01: assign column names, strip whitespace, replace `?` with NaN, drop NaN rows, pd.get_dummies, target encode)
- Returns `(X_train, X_test, y_train, y_test)` with 80/20 split, `random_state=42`

**`train_and_evaluate(X_train, y_train, X_test, y_test, hyperparams: dict) -> dict`**
- Fits `XGBClassifier(use_label_encoder=False, eval_metric='logloss', **hyperparams)`
- Measures wall-clock training time with `time.perf_counter()`
- Computes `roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])`
- Returns `{"model": fitted_model, "val_auc": float, "train_time_seconds": float}`

**`save_run(model, run_id: str, hyperparams: dict, metrics: dict, models_dir: str) -> dict`**
- Creates directory `{models_dir}/{run_id}/`
- Saves model to `{models_dir}/{run_id}/model.xgb`
- Builds metadata dict (see schema below)
- Writes metadata dict as JSON to `{models_dir}/{run_id}/metadata.json`
- Returns the metadata dict

**`metadata.json schema`** (must match exactly):
```json
{
  "run_id": "run_20250616_143022",
  "hyperparams": {
    "max_depth": 3,
    "learning_rate": 0.1,
    "n_estimators": 100
  },
  "metrics": {
    "val_auc": 0.8762,
    "train_time_seconds": 4.21
  },
  "model_path": "models/run_20250616_143022/model.xgb",
  "created_at": "2025-06-16T14:30:22Z"
}
```

- `created_at` format: `datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ")`
- `val_auc` rounded to 4 decimal places
- `train_time_seconds` rounded to 2 decimal places

**`main()`**
- Parse args, run `load_and_prepare → train_and_evaluate → save_run`
- Print: `Run {run_id} complete. val_auc={val_auc:.4f}, train_time={train_time:.2f}s`
- Print: `Saved to {model_path}`

---

#### `src/compare.py`

**CLI:** `python src/compare.py [--models-dir models/] [--sort-by val_auc]`

**`find_all_runs(models_dir: str) -> list[dict]`**
- Scans `models_dir` for all `metadata.json` files (one level deep: `models/*/metadata.json`)
- Loads each JSON
- Returns list of metadata dicts, sorted by `run_id` (ascending, i.e., chronological)

**`format_comparison_table(runs: list[dict]) -> str`**
- Builds a plain-text table:
```
run_id                  | max_depth | learning_rate | val_auc | train_time_s
------------------------|-----------|---------------|---------|-------------
run_20250616_143022     |         3 |         0.100 |  0.8762 |         4.21
run_20250616_143156     |         5 |         0.050 |  0.8891 |         6.43
run_20250616_143341     |         7 |         0.010 |  0.8623 |         9.87
```
- Mark the best row by val_auc with `*` prefix on run_id
- Add a summary line: `Best run: {run_id} (val_auc={val_auc:.4f})`

**`identify_best_run(runs: list[dict], metric: str = "val_auc") -> dict`**
- Returns the metadata dict with the highest value of `metrics[metric]`
- Raises `ValueError` if `runs` is empty

**`main()`**
- Call `find_all_runs`, format and print table, print best run summary

---

#### `src/train_all.py`

**`VARIANTS: list[dict]`**
```python
VARIANTS = [
    {"max_depth": 3, "learning_rate": 0.1, "n_estimators": 100},
    {"max_depth": 5, "learning_rate": 0.05, "n_estimators": 100},
    {"max_depth": 7, "learning_rate": 0.01, "n_estimators": 200},
]
```

**`main()`**
- Iterates through `VARIANTS`
- For each: calls `train_variant.main()` via subprocess (so run_ids get different timestamps)
- Wait 2 seconds between runs to ensure unique timestamps: `time.sleep(2)`
- After all 3 complete, calls `compare.main()` to print the summary table
- Total prints: 3 training summaries + 1 comparison table

---

### tests/ — what to test

#### `tests/test_metadata.py`

**`test_metadata_json_has_all_required_keys`**
- Run `train_variant.save_run(mock_model, "run_test", hyperparams, metrics, tmp_path)`
- Load `{tmp_path}/run_test/metadata.json`
- Assert all keys present: `run_id`, `hyperparams`, `metrics`, `model_path`, `created_at`
- Assert `metrics` contains `val_auc` and `train_time_seconds`
- Assert `hyperparams` contains `max_depth`, `learning_rate`, `n_estimators`

**`test_compare_finds_all_three_runs_after_train_all`**
- Run `train_all.main()` with a temp `--models-dir`
- Call `compare.find_all_runs(models_dir)`
- Assert `len(runs) == 3`

**`test_best_run_identified_correctly`**
- Construct 3 mock metadata dicts with val_auc: 0.85, 0.90, 0.87
- Call `compare.identify_best_run(runs)`
- Assert returned run has `val_auc == 0.90`

**`test_run_id_is_unique_per_run`**
- Run `generate_run_id()` twice with 1-second sleep between
- Assert the two run_ids are different strings

#### `tests/test_train_variant.py`

**`test_model_file_is_saved`**
- Call `train_variant.main()` with args pointing to `data/adult.data` and a temp `models_dir`
- Assert `model.xgb` file exists in the created run directory

**`test_val_auc_is_reasonable`**
- Same setup
- Load `metadata.json`, check `metrics.val_auc > 0.80`

---

### README.md content

```markdown
# P3-04 — Manual Versioning

Train three XGBoost variants, track results with timestamped folders and JSON metadata.

## Prerequisites

- Python 3.11+
- No AWS credentials needed

## Quick start

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
mkdir -p data && curl -o data/adult.data \
  "https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data"
python src/train_all.py
```

## Train a single variant

```bash
python src/train_variant.py --max-depth 5 --learning-rate 0.05 --data data/adult.data
```

## Compare all runs

```bash
python src/compare.py
```

## Output

Each run creates `models/run_{timestamp}/model.xgb` and `models/run_{timestamp}/metadata.json`.

`compare.py` prints a table and marks the best run by val_AUC.

## Tests

```bash
pytest tests/ -v
```
```

---

### GUIDE.md content

```markdown
# Guide — P3-04 Manual Versioning

## The problem you are about to feel

After running `train_all.py`, open `models/`. You have 3 directories with timestamps you
need to decode. If you want to know which used max_depth=5, you open 3 JSON files. If you
ran an extra experiment by hand, it is in a fourth directory. If a colleague ran experiments
on their machine, their folder is not here at all.

This friction is intentional. The next project (p3-05) installs MLflow and this problem
disappears. You will appreciate MLflow more for having done it the hard way first.

## What metadata.json gives you

Every run is self-describing. You can always reconstruct what happened by reading the JSON,
even if you lost the training logs. The `model_path` field lets `compare.py` find and load
the model for inference without searching.

## What metadata.json does NOT give you

- Git SHA of the code that produced the run (did you change train_variant.py between runs?)
- Input data hash (was `adult.data` the same file?)
- System metrics (CPU usage, memory peak)
- Artifacts other than the model (preprocessor, feature names)
- Search by arbitrary metadata fields

MLflow tracks all of these out of the box.

## The timestamp collision risk

`generate_run_id()` uses seconds precision. If two runs start in the same second (e.g., in
a parallel grid search), their directories would collide. The `train_all.py` script waits
2 seconds between runs to avoid this. In production, use a UUID or MLflow's run_id.

## Interview framing

"I've done manual experiment tracking with JSON metadata files and know exactly why it breaks
down: no central search, no input data versioning, no code versioning, collision risk in parallel
runs. That's why I use MLflow — not because it's trendy, but because I've felt what it replaces."
```

---

### Failure modes to handle

| Failure | Where | How to handle |
|---|---|---|
| Two runs start in the same second → same run_id | `train_variant.py: generate_run_id` | `train_all.py` waits 2 seconds between runs; document the risk |
| `models/` directory does not exist | `train_variant.py: save_run` | `os.makedirs(run_dir, exist_ok=True)` |
| `metadata.json` has wrong val_auc type (string instead of float) | `compare.py: identify_best_run` | `float(run["metrics"]["val_auc"])` to ensure numeric comparison |
| `models/` contains a directory without `metadata.json` (partial run) | `compare.py: find_all_runs` | Skip directories where `metadata.json` does not exist; print warning |
| `--data` path does not exist | `train_variant.py: load_and_prepare` | `FileNotFoundError` with the path |
| `models/` is empty when `compare.py` runs | `compare.py: identify_best_run` | Raise `ValueError("No runs found in models/. Run train_all.py first.")` |

---

### The metric this project measures

**What:** Validation AUC (ROC) per training run, plus comparison delta between best and worst variant

**Format:** Printed per-run as `val_auc=0.8762` and in comparison table; delta printed as `AUC spread: {max_auc - min_auc:.4f}`

**Target:** All 3 variants should achieve val_AUC > 0.80. The max_depth=5 variant typically performs best on UCI Adult with these hyperparameters.

## Source code

A reference implementation of Manual Versioning — README, guide, source and tests — is at https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/manual-versioning.