---
title: "Data Exploration"
description: "A documented, repeatable EDA process is the kind of unglamorous rigor that separates 'I trained a model' from 'I understand what the model is learning from' in..."
source: "https://confidentprep.com/paths/ml-engineering-on-aws/data-exploration/"
path: "ML Engineering on AWS"
repository: "https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/data-exploration"
token_estimate: 3556
---

# Data Exploration

## Overview

An EDA pipeline for the UCI Adult Income dataset that outputs a markdown data quality
report instead of just notebook cells. Everything downstream in this path — features,
training, monitoring for drift — only means something if you can point back to what
"normal" looked like in the data to begin with. This project is that baseline.

## What to do

**Path:** 3 — ML Engineering on AWS
**Position:** 2 of 12
**Difficulty:** 🟢 (pure Python/pandas, no cloud)
**Time:** 2-3h
**AWS Cost:** None

---

### Agent Pickup Instructions

```bash
# Bootstrap
cd projects/confident-prep/p3-02-data-exploration
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Download dataset (same as p3-01)
mkdir -p data
curl -o data/adult.data \
  "https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data"

# Run EDA pipeline
python src/main.py --data data/adult.data --output report.md

# Run tests
pytest tests/ -v
```

**Done when:**
- [ ] `python src/main.py --data data/adult.data --output report.md` exits 0 and writes `report.md`
- [ ] `report.md` contains sections: Dataset Overview, Class Balance, Feature Statistics, Missing Values, Correlation, Data Quality Flags
- [ ] `pytest tests/ -v` shows 4/4 passing
- [ ] No AWS credentials needed at any point

---

### What this project is

An EDA pipeline that ingests the UCI Adult Income dataset, computes class balance, per-feature statistics (mean/std/min/max for numeric, value_counts for categorical), missing value counts, numeric feature correlations, and a list of columns needing imputation. Everything is written to a structured Markdown report. This is deliberate groundwork — the decisions made here (which columns to impute, which to encode) feed directly into the feature engineering pipeline in p3-03.

### What the learner achieves

"I can write a systematic EDA pipeline that produces a data quality report usable as input to feature engineering decisions, and I know what questions to answer before touching a model."

---

### Folder structure

```
p3-02-data-exploration/
├── data/
│   └── adult.data              # raw UCI download (no header row)
├── src/
│   ├── main.py                # CLI entry point: --data, --output
│   ├── loader.py              # load_adult_dataset — handles no-header CSV, NaN replacement
│   ├── analyzer.py            # compute_class_balance, compute_feature_stats,
│   │                          # compute_missing_counts, compute_correlations, flag_high_missing
│   └── reporter.py            # generate_markdown_report — writes structured .md file
├── tests/
│   ├── test_loader.py         # question-mark handling, column names, whitespace stripping
│   └── test_analyzer.py       # class balance sums to 1.0, missing flag threshold, correlation shape
├── .env.example
├── requirements.txt
└── README.md
```

---

### .env.example

```dotenv
# No cloud credentials needed for this project.
# Kept for consistency with other Path 3 projects.

# Local data directory
LOCAL_DATA_DIR=data/

# High-missing threshold: columns with fraction missing above this get flagged
HIGH_MISSING_THRESHOLD=0.2
```

---

### requirements.txt

```
pandas==2.2.3
numpy==2.2.1
python-dotenv==1.0.1
pytest==8.3.4
```

---

### src/ — what to implement

#### `src/loader.py`

**`COLUMN_NAMES: list[str]`**
```python
COLUMN_NAMES = [
    "age", "workclass", "fnlwgt", "education", "education-num",
    "marital-status", "occupation", "relationship", "race", "sex",
    "capital-gain", "capital-loss", "hours-per-week", "native-country", "income"
]
```

**`NUMERIC_COLUMNS: list[str]`**
```python
NUMERIC_COLUMNS = ["age", "fnlwgt", "education-num", "capital-gain",
                   "capital-loss", "hours-per-week"]
```

**`CATEGORICAL_COLUMNS: list[str]`**
```python
CATEGORICAL_COLUMNS = ["workclass", "education", "marital-status", "occupation",
                       "relationship", "race", "sex", "native-country", "income"]
```

**`load_adult_dataset(path: str) -> pd.DataFrame`**
- Reads CSV with `header=None`, assigns `COLUMN_NAMES`
- Strips leading/trailing whitespace from all string-valued cells using `.str.strip()` on each object-dtype column
- Replaces the string `'?'` with `np.nan` in all string columns (after stripping — `' ?'` becomes `'?'` then `NaN`)
- Does NOT drop NaN rows — the analyzer needs to count them
- Returns the DataFrame with original dtypes (numeric columns parsed as int/float by pandas)

**Edge cases:**
- If `path` does not exist, raise `FileNotFoundError` with the path in the message
- If the CSV has fewer than 15 columns, raise `ValueError("Expected 15 columns, got {n}")`

---

#### `src/analyzer.py`

**`compute_class_balance(df: pd.DataFrame) -> dict`**
- Counts values in `income` column: `>50K` and `<=50K`
- Returns `{"<=50K": float, ">50K": float}` where values are proportions (fractions, not counts)
- Values must sum to 1.0 (use `value_counts(normalize=True)`)
- Edge case: if `income` column missing, raise `KeyError("income column not found")`

**`compute_feature_stats(df: pd.DataFrame) -> dict`**
- For each numeric column: compute `{"mean": float, "std": float, "min": float, "max": float, "median": float}`
- For each categorical column: compute `{"top_values": dict}` — value_counts as a dict, top 5 values only
- Returns `{"numeric": {col: stats_dict}, "categorical": {col: stats_dict}}`

**`compute_missing_counts(df: pd.DataFrame) -> dict`**
- For each column, count `NaN` values and compute fraction of total rows
- Returns `{col: {"count": int, "fraction": float}}` for ALL columns (even those with 0 missing)

**`compute_correlations(df: pd.DataFrame) -> pd.DataFrame`**
- Computes Pearson correlation matrix on numeric columns only
- Returns a `pd.DataFrame` with shape `(n_numeric, n_numeric)`
- Edge case: if fewer than 2 numeric columns, return empty DataFrame

**`flag_high_missing(missing_counts: dict, threshold: float = 0.2) -> list[str]`**
- Returns list of column names where `fraction > threshold`
- Sorted alphabetically
- Returns empty list if no columns exceed threshold

**`find_most_correlated_pair(corr_df: pd.DataFrame) -> tuple[str, str, float]`**
- Finds the highest absolute correlation between two distinct columns (exclude diagonal)
- Returns `(col_a, col_b, correlation_value)` where correlation_value is signed
- Edge case: if corr_df is empty, return `("", "", 0.0)`

---

#### `src/reporter.py`

**`generate_markdown_report(df: pd.DataFrame, stats: dict, output_path: str) -> None`**

Writes a Markdown file at `output_path` with these sections in order:

```
# Data Quality Report — UCI Adult Income

## Dataset Overview
- Rows: {n_rows}
- Columns: {n_cols}
- Numeric columns: {list}
- Categorical columns: {list}

## Class Balance
| Label | Count | Fraction |
|---|---|---|
| <=50K | {n} | {frac:.3f} |
| >50K  | {n} | {frac:.3f} |

**Imbalance ratio:** {majority/minority:.2f}:1

## Feature Statistics

### Numeric Features
| Column | Mean | Std | Min | Median | Max |
|---|---|---|---|---|---|
...

### Categorical Features
| Column | Top Value | Top Count | Unique Values |
|---|---|---|---|
...

## Missing Values
| Column | Missing Count | Missing Fraction |
|---|---|---|
...

## Correlation (Numeric Features)
Most correlated pair: {col_a} ↔ {col_b} (r = {value:.3f})

[correlation matrix as markdown table]

## Data Quality Flags
Columns flagged for imputation strategy (>{threshold*100:.0f}% missing):
{bulleted list or "None — all columns below threshold"}
```

**Edge cases:**
- Create parent directories of `output_path` if they do not exist
- If correlation matrix is empty, write `Correlation: insufficient numeric columns.` in that section

---

#### `src/main.py`

**CLI:** `python src/main.py --data <path> --output <path> [--missing-threshold 0.2]`

**`main()`**
- Parse args
- Load dataset via `loader.load_adult_dataset`
- Run all analyzer functions
- Assemble stats dict
- Call `reporter.generate_markdown_report`
- Print: `Report written to {output_path} ({n_rows} rows, {n_cols} columns)`
- Print data quality flags inline: `Flagged for imputation: {columns or 'none'}`
- Exit 0

---

### tests/ — what to test

#### `tests/test_loader.py`

**`test_load_handles_question_mark_as_nan`**
- Create a small 15-column CSV with `' ?'` in workclass column
- Call `load_adult_dataset(path)`
- Assert `df['workclass'].isna().any()` is True
- Assert no cell contains the string `'?'`

**`test_load_assigns_correct_column_names`**
- Load any valid adult.data sample
- Assert `list(df.columns) == COLUMN_NAMES`

**`test_load_strips_whitespace`**
- Create CSV row where `income` value is `' >50K '`
- Load and assert the value equals `'>50K'` not `' >50K '`

#### `tests/test_analyzer.py`

**`test_class_balance_sums_to_one`**
- Load actual dataset (or create a 10-row fixture with known counts)
- Call `compute_class_balance(df)`
- Assert `abs(sum(result.values()) - 1.0) < 1e-9`
- Assert result has exactly 2 keys

**`test_flag_high_missing_threshold`**
- Construct a mock `missing_counts` dict with one column at 0.25 fraction and one at 0.10
- Call `flag_high_missing(missing_counts, threshold=0.2)`
- Assert only the 0.25-fraction column is in the result

**`test_flag_high_missing_returns_empty_when_none_exceed`**
- All fractions < 0.2
- Assert result is `[]`

**`test_report_file_written_and_non_empty`**
- Run `main.main()` with real data and a temp output path
- Assert the file exists
- Assert file size > 500 bytes
- Assert the string `## Class Balance` appears in file content

---

### README.md content

```markdown
# P3-02 — Data Exploration

EDA pipeline for the UCI Adult Income dataset. Outputs a Markdown data quality report.

## Prerequisites

- Python 3.11+
- No AWS credentials needed

## Quick start

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
mkdir -p data && curl -o data/adult.data \
  "https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data"
python src/main.py --data data/adult.data --output report.md
```

## What you get

`report.md` with:
- Class balance (and imbalance ratio)
- Per-feature statistics (numeric: mean/std/min/median/max; categorical: top values)
- Missing value counts per column
- Pearson correlation matrix (numeric features)
- List of columns flagged for imputation strategy (>20% missing)

## Change the missing threshold

```bash
python src/main.py --data data/adult.data --output report.md --missing-threshold 0.05
```

## Tests

```bash
pytest tests/ -v
```
```

---

### GUIDE.md content

```markdown
# Guide — P3-02 Data Exploration

## Why EDA before modeling

Skipping EDA is the fastest way to build a model that silently fails. The UCI Adult dataset
has `?` in three columns — if you read those as strings, your encoder will create a `?` category
that poisons predictions. The missing fraction matters: if 30% of `occupation` is missing,
median imputation is a different choice than if 1% is missing.

## What to look for in the output

**Class balance:** UCI Adult is roughly 76%/24% (<=50K / >50K). You need to decide whether to
use `class_weight='balanced'` in your model or upsample the minority class. Neither is always right.

**Correlation:** `education-num` and `fnlwgt` — check if they are highly correlated. If so,
dropping one may simplify the model without hurting accuracy.

**Missing values:** In UCI Adult, `workclass`, `occupation`, and `native-country` have `?` replaced
with NaN. These are the columns that will need imputation in p3-03.

## How to read the correlation table

Pearson correlation measures linear dependence. A value of +1 means perfect positive correlation,
-1 perfect negative. Values above |0.7| between features are worth investigating — one may be
derivable from the other (redundancy).

## The imputation decision

The report flags columns above the threshold. For `native-country` (which is 98% United States),
missing values likely represent the same group. For `occupation`, missing is likely not at random
(unemployed people may not report occupation). These decisions belong in your feature engineering
spec, not in train.py.

## Interview framing

"Before any feature engineering or training, I run a systematic EDA that quantifies class imbalance,
flags columns with high missing rates, and checks for correlated features — so every preprocessing
decision has a documented reason rather than being arbitrary."
```

---

### Failure modes to handle

| Failure | Where | How to handle |
|---|---|---|
| `?` appears with surrounding whitespace as `' ?'` | `loader.py: load_adult_dataset` | Strip whitespace before replacing `'?'` with NaN |
| CSV has no header — pandas reads first row as data | `loader.py: load_adult_dataset` | Always read with `header=None` and assign `COLUMN_NAMES` |
| `income` column has `' >50K'` vs `'>50K'` variants | `analyzer.py: compute_class_balance` | Strip whitespace before grouping |
| Fewer than 2 numeric columns prevents correlation | `analyzer.py: compute_correlations` | Return empty DataFrame; reporter handles it gracefully |
| Output directory does not exist | `reporter.py: generate_markdown_report` | `os.makedirs(parent, exist_ok=True)` before writing |
| Dataset path does not exist | `loader.py: load_adult_dataset` | Raise `FileNotFoundError` with the path |

---

### The metric this project measures

**What:** Number of data quality flags — columns identified as needing imputation strategy (fraction missing above threshold)

**Format:** Integer, printed as `Flagged for imputation: N columns [col1, col2, ...]`

**Target:** For UCI Adult Income at default threshold (0.20): 0 flagged columns (no column exceeds 20% missing — the `?` values are about 5-7% of rows in affected columns, which is below the default threshold). Using threshold=0.05 should flag `workclass`, `occupation`, `native-country`.

**Secondary metric:** Imbalance ratio (majority class count / minority class count), format `{ratio:.2f}:1`, expected ~3.2:1 for UCI Adult.

## Source code

A reference implementation of Data Exploration — README, guide, source and tests — is at https://github.com/confidentprep14-web/ai-engineering-projects/tree/main/data-exploration.