Data Exploration

Path: 3 — ML Engineering on AWS Position: 2 of 12 Difficulty: 🟢 (pure Python/pandas, no cloud) Time: 2-3h AWS Cost: None

---

Agent Pickup Instructions

# Bootstrap
cd projects/confident-prep/p3-02-data-exploration
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Download dataset (same as p3-01)
mkdir -p data
curl -o data/adult.data \
  "https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data"

# Run EDA pipeline
python src/main.py --data data/adult.data --output report.md

# Run tests
pytest tests/ -v

Done when:

• python src/main.py --data data/adult.data --output report.md exits 0 and writes report.md
• report.md contains sections: Dataset Overview, Class Balance, Feature Statistics, Missing Values, Correlation, Data Quality Flags
• pytest tests/ -v shows 4/4 passing
• No AWS credentials needed at any point

---

What this project is

An EDA pipeline that ingests the UCI Adult Income dataset, computes class balance, per-feature statistics (mean/std/min/max for numeric, value_counts for categorical), missing value counts, numeric feature correlations, and a list of columns needing imputation. Everything is written to a structured Markdown report. This is deliberate groundwork — the decisions made here (which columns to impute, which to encode) feed directly into the feature engineering pipeline in p3-03.

What the learner achieves

"I can write a systematic EDA pipeline that produces a data quality report usable as input to feature engineering decisions, and I know what questions to answer before touching a model."

---

Folder structure

p3-02-data-exploration/
├── data/
│   └── adult.data              # raw UCI download (no header row)
├── src/
│   ├── main.py                # CLI entry point: --data, --output
│   ├── loader.py              # load_adult_dataset — handles no-header CSV, NaN replacement
│   ├── analyzer.py            # compute_class_balance, compute_feature_stats,
│   │                          # compute_missing_counts, compute_correlations, flag_high_missing
│   └── reporter.py            # generate_markdown_report — writes structured .md file
├── tests/
│   ├── test_loader.py         # question-mark handling, column names, whitespace stripping
│   └── test_analyzer.py       # class balance sums to 1.0, missing flag threshold, correlation shape
├── .env.example
├── requirements.txt
└── README.md

---

.env.example

# No cloud credentials needed for this project.
# Kept for consistency with other Path 3 projects.

# Local data directory
LOCAL_DATA_DIR=data/

# High-missing threshold: columns with fraction missing above this get flagged
HIGH_MISSING_THRESHOLD=0.2

---

requirements.txt

pandas==2.2.3
numpy==2.2.1
python-dotenv==1.0.1
pytest==8.3.4

---

src/ — what to implement

src/loader.py

COLUMN_NAMES: list[str]

COLUMN_NAMES = [
    "age", "workclass", "fnlwgt", "education", "education-num",
    "marital-status", "occupation", "relationship", "race", "sex",
    "capital-gain", "capital-loss", "hours-per-week", "native-country", "income"
]

NUMERIC_COLUMNS: list[str]

NUMERIC_COLUMNS = ["age", "fnlwgt", "education-num", "capital-gain",
                   "capital-loss", "hours-per-week"]

CATEGORICAL_COLUMNS: list[str]

CATEGORICAL_COLUMNS = ["workclass", "education", "marital-status", "occupation",
                       "relationship", "race", "sex", "native-country", "income"]

load_adult_dataset(path: str) -> pd.DataFrame

• Reads CSV with header=None, assigns COLUMN_NAMES
• Strips leading/trailing whitespace from all string-valued cells using .str.strip() on each object-dtype column
• Replaces the string '?' with np.nan in all string columns (after stripping — ' ?' becomes '?' then NaN)
• Does NOT drop NaN rows — the analyzer needs to count them
• Returns the DataFrame with original dtypes (numeric columns parsed as int/float by pandas)

Edge cases:

• If path does not exist, raise FileNotFoundError with the path in the message
• If the CSV has fewer than 15 columns, raise ValueError("Expected 15 columns, got {n}")

---

src/analyzer.py

compute_class_balance(df: pd.DataFrame) -> dict

• Counts values in income column: >50K and <=50K
• Returns {"<=50K": float, ">50K": float} where values are proportions (fractions, not counts)
• Values must sum to 1.0 (use value_counts(normalize=True))
• Edge case: if income column missing, raise KeyError("income column not found")

compute_feature_stats(df: pd.DataFrame) -> dict

• For each numeric column: compute {"mean": float, "std": float, "min": float, "max": float, "median": float}
• For each categorical column: compute {"top_values": dict} — value_counts as a dict, top 5 values only
• Returns {"numeric": {col: stats_dict}, "categorical": {col: stats_dict}}

compute_missing_counts(df: pd.DataFrame) -> dict

• For each column, count NaN values and compute fraction of total rows
• Returns {col: {"count": int, "fraction": float}} for ALL columns (even those with 0 missing)

compute_correlations(df: pd.DataFrame) -> pd.DataFrame

• Computes Pearson correlation matrix on numeric columns only
• Returns a pd.DataFrame with shape (n_numeric, n_numeric)
• Edge case: if fewer than 2 numeric columns, return empty DataFrame

flag_high_missing(missing_counts: dict, threshold: float = 0.2) -> list[str]

• Returns list of column names where fraction > threshold
• Sorted alphabetically
• Returns empty list if no columns exceed threshold

find_most_correlated_pair(corr_df: pd.DataFrame) -> tuple[str, str, float]

• Finds the highest absolute correlation between two distinct columns (exclude diagonal)
• Returns (col_a, col_b, correlation_value) where correlation_value is signed
• Edge case: if corr_df is empty, return ("", "", 0.0)

---

src/reporter.py

generate_markdown_report(df: pd.DataFrame, stats: dict, output_path: str) -> None

Writes a Markdown file at output_path with these sections in order:

# Data Quality Report — UCI Adult Income

## Dataset Overview
- Rows: {n_rows}
- Columns: {n_cols}
- Numeric columns: {list}
- Categorical columns: {list}

## Class Balance
| Label | Count | Fraction |
|---|---|---|
| <=50K | {n} | {frac:.3f} |
| >50K  | {n} | {frac:.3f} |

**Imbalance ratio:** {majority/minority:.2f}:1

## Feature Statistics

### Numeric Features
| Column | Mean | Std | Min | Median | Max |
|---|---|---|---|---|---|
...

### Categorical Features
| Column | Top Value | Top Count | Unique Values |
|---|---|---|---|
...

## Missing Values
| Column | Missing Count | Missing Fraction |
|---|---|---|
...

## Correlation (Numeric Features)
Most correlated pair: {col_a} ↔ {col_b} (r = {value:.3f})

[correlation matrix as markdown table]

## Data Quality Flags
Columns flagged for imputation strategy (>{threshold*100:.0f}% missing):
{bulleted list or "None — all columns below threshold"}

Edge cases:

• Create parent directories of output_path if they do not exist
• If correlation matrix is empty, write Correlation: insufficient numeric columns. in that section

---

src/main.py

CLI: python src/main.py --data <path> --output <path> [--missing-threshold 0.2]

main()

• Parse args
• Load dataset via loader.load_adult_dataset
• Run all analyzer functions
• Assemble stats dict
• Call reporter.generate_markdown_report
• Print: Report written to {output_path} ({n_rows} rows, {n_cols} columns)
• Print data quality flags inline: Flagged for imputation: {columns or 'none'}
• Exit 0

---

tests/ — what to test

tests/test_loader.py

test_load_handles_question_mark_as_nan

• Create a small 15-column CSV with ' ?' in workclass column
• Call load_adult_dataset(path)
• Assert df['workclass'].isna().any() is True
• Assert no cell contains the string '?'

test_load_assigns_correct_column_names

• Load any valid adult.data sample
• Assert list(df.columns) == COLUMN_NAMES

test_load_strips_whitespace

• Create CSV row where income value is ' >50K '
• Load and assert the value equals '>50K' not ' >50K '

tests/test_analyzer.py

test_class_balance_sums_to_one

• Load actual dataset (or create a 10-row fixture with known counts)
• Call compute_class_balance(df)
• Assert abs(sum(result.values()) - 1.0) < 1e-9
• Assert result has exactly 2 keys

test_flag_high_missing_threshold

• Construct a mock missing_counts dict with one column at 0.25 fraction and one at 0.10
• Call flag_high_missing(missing_counts, threshold=0.2)
• Assert only the 0.25-fraction column is in the result

test_flag_high_missing_returns_empty_when_none_exceed

• All fractions < 0.2
• Assert result is []

test_report_file_written_and_non_empty

• Run main.main() with real data and a temp output path
• Assert the file exists
• Assert file size > 500 bytes
• Assert the string ## Class Balance appears in file content

---

README.md content

# P3-02 — Data Exploration

EDA pipeline for the UCI Adult Income dataset. Outputs a Markdown data quality report.

## Prerequisites

- Python 3.11+
- No AWS credentials needed

## Quick start

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
mkdir -p data && curl -o data/adult.data \
  "https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data"
python src/main.py --data data/adult.data --output report.md

What you get

report.md with:

• Class balance (and imbalance ratio)
• Per-feature statistics (numeric: mean/std/min/median/max; categorical: top values)
• Missing value counts per column
• Pearson correlation matrix (numeric features)
• List of columns flagged for imputation strategy (>20% missing)

Change the missing threshold

python src/main.py --data data/adult.data --output report.md --missing-threshold 0.05

Tests

pytest tests/ -v

---

## GUIDE.md content

```markdown
# Guide — P3-02 Data Exploration

## Why EDA before modeling

Skipping EDA is the fastest way to build a model that silently fails. The UCI Adult dataset
has `?` in three columns — if you read those as strings, your encoder will create a `?` category
that poisons predictions. The missing fraction matters: if 30% of `occupation` is missing,
median imputation is a different choice than if 1% is missing.

## What to look for in the output

**Class balance:** UCI Adult is roughly 76%/24% (<=50K / >50K). You need to decide whether to
use `class_weight='balanced'` in your model or upsample the minority class. Neither is always right.

**Correlation:** `education-num` and `fnlwgt` — check if they are highly correlated. If so,
dropping one may simplify the model without hurting accuracy.

**Missing values:** In UCI Adult, `workclass`, `occupation`, and `native-country` have `?` replaced
with NaN. These are the columns that will need imputation in p3-03.

## How to read the correlation table

Pearson correlation measures linear dependence. A value of +1 means perfect positive correlation,
-1 perfect negative. Values above |0.7| between features are worth investigating — one may be
derivable from the other (redundancy).

## The imputation decision

The report flags columns above the threshold. For `native-country` (which is 98% United States),
missing values likely represent the same group. For `occupation`, missing is likely not at random
(unemployed people may not report occupation). These decisions belong in your feature engineering
spec, not in train.py.

## Interview framing

"Before any feature engineering or training, I run a systematic EDA that quantifies class imbalance,
flags columns with high missing rates, and checks for correlated features — so every preprocessing
decision has a documented reason rather than being arbitrary."

---

Failure modes to handle

Failure	Where	How to handle
? appears with surrounding whitespace as ' ?'	loader.py: load_adult_dataset	Strip whitespace before replacing '?' with NaN
CSV has no header — pandas reads first row as data	loader.py: load_adult_dataset	Always read with header=None and assign COLUMN_NAMES
income column has ' >50K' vs '>50K' variants	analyzer.py: compute_class_balance	Strip whitespace before grouping
Fewer than 2 numeric columns prevents correlation	analyzer.py: compute_correlations	Return empty DataFrame; reporter handles it gracefully
Output directory does not exist	reporter.py: generate_markdown_report	os.makedirs(parent, exist_ok=True) before writing
Dataset path does not exist	loader.py: load_adult_dataset	Raise FileNotFoundError with the path

---

The metric this project measures

What: Number of data quality flags — columns identified as needing imputation strategy (fraction missing above threshold)

Format: Integer, printed as Flagged for imputation: N columns [col1, col2, ...]

Target: For UCI Adult Income at default threshold (0.20): 0 flagged columns (no column exceeds 20% missing — the ? values are about 5-7% of rows in affected columns, which is below the default threshold). Using threshold=0.05 should flag workclass, occupation, native-country.

Secondary metric: Imbalance ratio (majority class count / minority class count), format {ratio:.2f}:1, expected ~3.2:1 for UCI Adult.