diff --git a/README.md b/README.md index cab3161..860cc67 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,9 @@ Small busy-hour classifier (baked for the "train new data" demo): The classifier predicts `busy` = hourly footfall above the sensor's p75 using hour, weekday, weekend, temperature, rain, and holiday. It trains in under a second on the challenge CSV and is retrained from an uploaded CSV through -`POST /train`. +`POST /train`. Full definition, confusion matrix, per-sensor table, and honest +limits: [`docs/small-model.md`](docs/small-model.md). Paper tables and figures: +[`docs/metrics.md`](docs/metrics.md). ## Quick start @@ -47,6 +49,8 @@ Requirements: Python 3.12 and `uv` (the repo also works with a plain venv). uv sync make data # needs the raw CSV, first run also fetches Open-Meteo weather make train # writes artifacts/model.joblib, metrics.json, forecasts.json +make small # prints the busy-hour classifier metrics + confusion matrix +make figures # regenerates docs/figures/*.svg from metrics.json make test make check # CI gate: sync --frozen, ruff, compileall, pytest make serve # http://localhost:8000 @@ -69,7 +73,8 @@ committed so the API and Docker image work without the raw file or network. | `artifacts/sensors.json` | 8 sensors: id, name, lat/lon, mean, p95, battery | | `artifacts/series.json` + `series/.json` | Hourly series per sensor | | `artifacts/forecasts.json` | 48h + 7d forecasts for 4 scenarios | -| `artifacts/metrics.json` | Versions, coverage, headline metrics, ablation, limits | +| `artifacts/metrics.json` | Versions, coverage, headline metrics, ablation, per-sensor, small summary, limits | +| `artifacts/small_metrics.json` | Busy classifier: metrics, confusion matrix, per-sensor, thresholds | | `artifacts/context.json` | Current + 7-day weather, BW holidays | | `artifacts/data_audit.json` | Row/sensor checks against `mission/data.md` | | `artifacts/model.joblib` | Baked HistGB forecast bundle | @@ -118,6 +123,29 @@ and persists to `PULSECAST_MODELS` (default `/tmp/pulsecast/models`). A restart falls back to the baked model. `model=small` is the fast demo; `model=forecast` retrains the global HistGB (slower, still background). +## End-to-end check + +`scripts/e2e_train.py` starts a real uvicorn process on port 18000 with a +control token, then checks the snapshot, every `/predict` combination, every +`/simulate` scenario, auth and error paths, one-job-at-a-time, and both real +training jobs (small + forecast) including hot-swap and `/predict` afterward: + +```bash +uv run python scripts/e2e_train.py --port 18000 --report /tmp/e2e.json +``` + +The unit and contract tests are `make test`; `make check` is the CI gate. + +## Docs + +| File | Contents | +|---|---| +| `docs/metrics.md` | Paper tables/figures: protocol, headline, ablation, per sensor, uncertainty, small model, claim limits | +| `docs/small-model.md` | Busy-hour target, thresholds, split, metrics, confusion matrix, per sensor, limits | +| `docs/train-e2e.md` | Exact live training-job evidence (small + forecast, hot-swap, errors) | +| `docs/figures/*.svg` | Ablation MAE, per-sensor MAE, small-model confusion matrix (generated) | +| `model_card.md` | Model card: features, split, baseline, ablation, leakage, limitations | + ## Docker ```bash @@ -153,6 +181,8 @@ src/pulsecast/ small.py busy classifier + flexible CSV parser retrain.py runtime forecast retraining from upload server.py FastAPI app and training job manager +scripts/ train_small.py, plot_metrics.py, e2e_train.py +docs/ metrics.md, small-model.md, generated figures tests/ schema, bootstrap, health, prediction, small-training job model_card.md features, split, baseline, ablation, limits, leakage ``` diff --git a/docs/figures/ablation_mae.svg b/docs/figures/ablation_mae.svg new file mode 100644 index 0000000..6e5a8d8 --- /dev/null +++ b/docs/figures/ablation_mae.svg @@ -0,0 +1,36 @@ + + + +Ablation: holdout MAE by feature set +Holdout 2026-10-10T23:00:00Z .. 2026-11-07T23:00:00Z, n=5350 + +0.0 + +1.5 + +2.9 + +4.4 + +5.8 +seasonal naive + +5.40 +day no weather + +3.96 +day holiday + +3.84 +day weather + +3.36 +week no weather + +4.08 +week weather + +3.32 +Source: artifacts/metrics.json · PulseCast 28-day time-based holdout + + diff --git a/docs/figures/per_sensor_mae.svg b/docs/figures/per_sensor_mae.svg new file mode 100644 index 0000000..3e08a5d --- /dev/null +++ b/docs/figures/per_sensor_mae.svg @@ -0,0 +1,60 @@ + + + +Per-sensor holdout MAE: production model vs seasonal naive +Holdout 2026-10-10T23:00:00Z .. 2026-11-07T23:00:00Z, n=5350 + +0.0 + +2.8 + +5.6 + +8.4 + +11.2 +seasonal naive +weather-7d +Density 01 + +3.58 + +2.26 +Götzenturm + +3.84 + +2.40 +Hagenbuchersee 1 + +4.01 + +2.47 +Hagenbuchersee 2 + +6.49 + +4.12 +Marahaus + +4.57 + +2.67 +Neckarufer + +5.89 + +3.63 +Soleo + +4.40 + +3.19 +Stadtgalerie + +10.39 + +5.83 +Source: artifacts/metrics.json · PulseCast 28-day time-based holdout + + diff --git a/docs/figures/small_model_confusion.svg b/docs/figures/small_model_confusion.svg new file mode 100644 index 0000000..56b65c2 --- /dev/null +++ b/docs/figures/small_model_confusion.svg @@ -0,0 +1,24 @@ + + + +Small busy-hour classifier: holdout confusion matrix +n=5,710 · accuracy 0.708 · precision 0.360 · recall 0.574 · F1 0.443 · ROC-AUC 0.726 +Predicted not busy +Predicted busy +Actually not busy +Actually busy + +3,377 +TN · 74.1% of row + +1,178 +FP · 25.9% of row + +492 +FN · 42.6% of row + +663 +TP · 57.4% of row +Source: artifacts/small_metrics.json · chronological 80/20 split + + diff --git a/docs/metrics.md b/docs/metrics.md new file mode 100644 index 0000000..5c891ce --- /dev/null +++ b/docs/metrics.md @@ -0,0 +1,187 @@ +# Metrics — paper support + +All values in this file are generated by `make train` and stored in +`artifacts/metrics.json` (forecast + ablation + per-sensor + small model) and +`artifacts/small_metrics.json` (small-model detail). No number here is typed by +hand: to refresh a table, rerun `make train && make figures` and copy the new +JSON, or quote the JSON directly. + +## Run provenance + +| Field | Value | +|---|---| +| Model version | `histgb-2026-11-14` | +| Trained at | 2026-11-14T05:27:21Z | +| Data version | 2026-11-07T23:51:00Z | +| Context (weather) version | 2026-11-14T03:47:09Z | +| Train rows | 18,602 hourly sensor rows | +| Holdout | 2026-10-10T23:00:00Z → 2026-11-07T23:00:00Z | +| Evaluation rows | 5,350 (8 sensors, hours with a seasonal-naive baseline) | +| Sensors | 8 (one merged `density-01`, seven `hn-*`) | +| Raw input | 192,464 rows, 96,232 density rows, 96,232 battery rows | +| Weather | Open-Meteo historical archive (training), forecast (scenarios) | +| Calendar | Baden-Württemberg holidays 2025–2026 | + +## Protocol + +- One global `HistGradientBoostingRegressor` over all eight sensors. Target is + the mean of 15-minute `densityData` readings in a UTC hour. +- Time-based 28-day holdout measured back from the last observed hour. No + shuffling, no random split. +- Seasonal naive baseline: the exact 168-row lag (same observed hour one week + earlier), evaluated on the same rows. +- The production bundle is refit on all observed rows **after** the holdout is + evaluated; the holdout is never in the production fit. +- WAPE = `sum(|error|) / sum(|actual|)`. R² is computed on the same rows. +- The production 7-day model uses only features at least 168 hours old, so a + 168-hour forecast needs no recursion. +- Weather in the holdout is observed, not forecast. Live accuracy will be lower + than the table because the dashboard uses forecast weather. + +## Headline + +| Model | MAE | WAPE | R² | +|---|---:|---:|---:| +| Seasonal naive (exact 168-row lag) | 5.40 | 44.7% | 0.462 | +| **Weather-aware 7-day (production)** | **3.32** | **27.5%** | **0.777** | + +- MAE improvement over seasonal naive: **38.5%**. +- WAPE improvement: 17.2 percentage points. +- The mission reference numbers for seasonal naive (5.40 / 44.7% / 0.462) are + reproduced exactly. +- The production row is slightly better than the reference 7-day model + (4.37 / 36.2% / 0.672) because `lag_week` uses the previous *observed* same + weekday/hour across sensor gaps instead of recursion. + +## Ablation + +| Model | Description | MAE | WAPE | R² | n | +|---|---|---|---:|---:|---:|---:| +| `seasonal_naive` | exact 168-row lag (same observed hour last week) | 5.40 | 44.7% | 0.462 | 5,350 | +| `day_no_weather` | calendar + lags 24/48/168 + 24h/168h rolling means | 3.96 | 32.8% | 0.706 | 5,350 | +| `day_holiday` | day-ahead + Baden-Württemberg holiday flag | 3.84 | 31.8% | 0.732 | 5,350 | +| `day_weather` | day-ahead + temperature + rain (observed weather) | 3.36 | 27.8% | 0.763 | 5,350 | +| `week_no_weather` | 7-day-safe calendar + previous same weekday/hour values | 4.08 | 33.8% | 0.697 | 5,350 | +| **`week_weather`** | **7-day-safe + temperature + rain (production)** | **3.32** | **27.5%** | **0.777** | **5,350** | + +Derived effects: + +- Weather at day-ahead: 3.96 → 3.36 MAE, **−15.2%**. +- Weather at 7-day-safe: 4.08 → 3.32 MAE, **−18.6%**. +- Holiday flag at day-ahead: 3.96 → 3.84 MAE, **−0.12 MAE**. Holidays are + context, not a headline accuracy claim. +- The mission reference day-ahead weather row (3.36 / 27.8%) is reproduced. + +![Ablation MAE by feature set](figures/ablation_mae.svg) + +*Figure 1. Holdout MAE for each feature set; teal is the production model. +Weather is the largest single gain; the holiday flag is a small addition.* + +## Per sensor + +Production model and seasonal naive on the same holdout rows. + +| Sensor | Name | n | MAE | WAPE | R² | Baseline MAE | Baseline WAPE | Baseline R² | MAE improvement | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:| +| `density-01` | Density 01 | 669 | 2.26 | 30.8% | 0.722 | 3.58 | 48.8% | 0.238 | 36.8% | +| `hn-goetzentuem-01` | Götzenturm | 669 | 2.40 | 26.9% | 0.718 | 3.84 | 43.0% | 0.281 | 37.5% | +| `hn-hagenbuchersee-01` | Hagenbuchersee 1 | 669 | 2.47 | 38.6% | 0.602 | 4.01 | 62.8% | −0.212 | 38.5% | +| `hn-hagenbuchersee-02` | Hagenbuchersee 2 | 669 | 4.12 | 31.1% | 0.731 | 6.49 | 48.9% | 0.337 | 36.5% | +| `hn-marahaus-01` | Marahaus | 669 | 2.67 | 29.8% | 0.671 | 4.57 | 51.0% | −0.045 | 41.7% | +| `hn-neckarufer-01` | Neckarufer | 668 | 3.63 | 28.2% | 0.695 | 5.89 | 45.8% | 0.110 | 38.4% | +| `hn-soleo-01` | Soleo | 669 | 3.19 | 20.2% | 0.548 | 4.40 | 28.0% | 0.163 | 27.7% | +| `hn-stadtgalerie-01` | Stadtgalerie | 668 | 5.83 | 25.3% | 0.765 | 10.39 | 45.1% | 0.480 | 43.9% | + +- Every sensor improves over its own seasonal naive baseline (27.7%–43.9%). +- Stadtgalerie has the largest absolute error (MAE 5.83) and the largest + volume; Soleo has the smallest relative gain and the lowest R² (0.548). +- Two baselines have negative R² (`hagenbuchersee-01`, `marahaus-01`), which is + expected for a naive lag on a short, variable holdout; the model is positive + on all eight. +- Sensor totals: 669 × 6 + 668 × 2 = 5,350, matching the headline. + +![Per-sensor MAE](figures/per_sensor_mae.svg) + +*Figure 2. Per-sensor holdout MAE, production model (blue) vs seasonal naive +(grey). Every sensor improves; the model is not carried by one large sensor.* + +## Uncertainty band + +The `lo`/`hi` band is the holdout residual p10/p90 added to the point forecast, +clipped at zero: + +| p10 | p50 | p90 | +|---:|---:|---:| +| −4.83 | −0.59 | +4.88 | + +This is an empirical 80% band, not a calibrated guarantee, and the median +residual being −0.59 shows a small optimistic bias on the holdout. + +## Small busy-hour classifier + +Full definition and limitations: [`docs/small-model.md`](small-model.md). +Summary from the same baked run (`logit-20261114-052723`): + +| Accuracy | Precision | Recall | F1 | ROC-AUC | Train rows | Test rows | Base rate | +|---:|---:|---:|---:|---:|---:|---:|---:| +| 0.708 | 0.360 | 0.574 | 0.443 | 0.726 | 18,242 | 5,710 | 0.202 | + +| | Predicted not busy | Predicted busy | +|---|---:|---:| +| **Actually not busy** | 3,377 (TN) | 1,178 (FP) | +| **Actually busy** | 492 (FN) | 663 (TP) | + +![Small model confusion matrix](figures/small_model_confusion.svg) + +*Figure 3. Holdout confusion matrix for the small busy-hour classifier. The +balanced weights trade precision for recall; accuracy 0.708 is below the 0.798 +majority-class baseline, so report recall/F1 and ROC-AUC instead.* + +Per-sensor small-model holdout: + +| Sensor | n | Positive rate | Accuracy | Precision | Recall | F1 | +|---|---:|---:|---:|---:|---:|---:| +| `density-01` | 714 | 0.111 | 0.761 | 0.300 | 0.873 | 0.447 | +| `hn-goetzentuem-01` | 714 | 0.197 | 0.732 | 0.391 | 0.638 | 0.485 | +| `hn-hagenbuchersee-01` | 714 | 0.049 | 0.724 | 0.148 | 0.971 | 0.257 | +| `hn-hagenbuchersee-02` | 714 | 0.123 | 0.783 | 0.355 | 0.932 | 0.514 | +| `hn-marahaus-01` | 714 | 0.055 | 0.710 | 0.135 | 0.795 | 0.230 | +| `hn-neckarufer-01` | 713 | 0.090 | 0.764 | 0.274 | 0.984 | 0.429 | +| `hn-soleo-01` | 714 | 0.704 | 0.581 | 0.943 | 0.431 | 0.592 | +| `hn-stadtgalerie-01` | 713 | 0.289 | 0.604 | 0.335 | 0.374 | 0.353 | + +## Claims the data supports + +- A global, weather-aware HistGB forecasts hourly footfall for all eight + inner-city sensors with MAE 3.32 / WAPE 27.5% / R² 0.777 on a 28-day + time-based holdout. +- Weather features are the main measurable gain: −18.6% MAE versus the 7-day + no-weather model, −38.5% versus seasonal naive. +- Every sensor improves over its own seasonal naive baseline (27.7%–43.9%). +- A tiny logistic model can flag busy hours with ROC-AUC 0.726 and 57.4% + recall at 36.0% precision, retrained from a CSV in under 5 seconds. + +## Claims the data does not support + +- People counts. Values are an index of estimated devices; penetration is + unknown. +- Annual seasonality, city-wide surfaces, or store-location recommendations + from five months at eight sensors in ~580 m. +- Causal weather effects. The ablation is predictive, not a treatment effect. +- Live accuracy equal to the holdout. The holdout uses observed weather; the + demo uses forecast weather and `predict` adds scenario overlays. +- Holidays as a large driver. The flag moves day-ahead MAE by only −0.12. +- The small classifier as the production forecast. It is a demo artifact with + six features and a per-window p75 target. +- Tuned hyperparameters. The model configuration is fixed; the 28-day holdout + is the honest signal, not a search result. + +## Reproduce + +```bash +make data # parse + weather + holidays + JSON artifacts +make train # holdout, ablation, per-sensor, small model, forecasts +make small # print the small-model metrics and confusion matrix +make figures # regenerate docs/figures/*.svg from artifacts/metrics.json +make check # sync --frozen, ruff, compileall, pytest +``` diff --git a/docs/small-model.md b/docs/small-model.md new file mode 100644 index 0000000..a57465a --- /dev/null +++ b/docs/small-model.md @@ -0,0 +1,187 @@ +# Small model — busy-hour classifier + +This is the small, fast, explainable model that powers the "train new data" +demo and the "I coded an AI" artifact. It is deliberately separate from the +production 7-day forecaster: six cheap features, a linear model, a time-based +split, and under a second of fitting. It is not the model behind the dashboard +forecast. + +Every number below comes from the baked artifact +`artifacts/small_metrics.json` (version `logit-20261114-052723`, trained +2026-11-14T05:27:23Z) and can be reproduced with `make small`. + +## Target + +`busy` is a binary label for one sensor-hour: + +```text +busy = 1 if hourly_mean_density > that sensor's 75th percentile +busy = 0 otherwise +``` + +- The p75 is computed **per sensor on the training window only**, then applied + unchanged to the holdout. Computing it on the full dataset would leak the + holdout distribution into the label. +- A p75 threshold makes "busy" mean "busier than three of four hours at this + location", not an absolute footfall claim. The same absolute value can be + busy at Soleo and quiet at Stadtgalerie. +- Hours above the threshold are exactly 25% of the training hours by + construction, before missing hours are dropped. + +| Sensor | Training-window p75 | +|---|---:| +| `density-01` | 15.23 | +| `hn-goetzentuem-01` | 15.40 | +| `hn-hagenbuchersee-01` | 18.20 | +| `hn-hagenbuchersee-02` | 29.17 | +| `hn-marahaus-01` | 22.23 | +| `hn-neckarufer-01` | 28.35 | +| `hn-soleo-01` | 11.38 | +| `hn-stadtgalerie-01` | 38.41 | + +## Data and split + +- Input: the challenge CSV, `densityData` rows only, aggregated to the mean of + the 15-minute readings in each UTC hour. +- Window: 2026-06-12T00:00:00Z → 2026-11-07T23:00:00Z. +- 23,952 hourly sensor rows across all 8 usable sensors. +- Split is chronological, not random: the first 80% of **timestamps** train, + the last 20% test. + - Train: 2026-06-12T00:00:00Z → 2026-10-09T01:00:00Z, 18,242 rows. + - Test: 2026-10-09T02:00:00Z → 2026-11-07T23:00:00Z, 5,710 rows. +- Excluded devices: `maker-fair`, `niklas-01`, `hn-bollwerksturm-01` (test + installations / insufficient coverage). See `artifacts/data_audit.json`. + +## Features + +| Feature | Source | Note | +|---|---|---| +| `hour` | local Europe/Berlin hour | 0–23, captures the daily curve | +| `weekday` | local Europe/Berlin weekday | 0 = Monday | +| `is_weekend` | weekday ≥ 5 | | +| `is_holiday` | Baden-Württemberg calendar | context, not a headline claim | +| `temp_c` | observed Open-Meteo archive | joined to the nearest hour (±3 h) | +| `precip_mm` | observed Open-Meteo archive | same join | + +No lags, no target history, no sensor identifier. That is intentional: the +classifier must retrain from a small user upload that may not contain a long +history, and it must not depend on future rows. Sensor identity is still +implicit in the per-sensor threshold. + +## Model + +```text +StandardScaler -> LogisticRegression( + C=1.0, max_iter=1000, class_weight="balanced", + solver="lbfgs", random_state=42) +``` + +`class_weight="balanced"` compensates for the 25/75 class split caused by the +p75 threshold. It buys recall at the cost of precision, which is the right +trade for a "flag the busy hours" demo. + +Fit time: **0.30 s** on the full challenge CSV in this run (contract: < 5 s, +including the API job's CSV parsing). + +## Metrics + +Time-based holdout, 5,710 rows, base rate 20.2% positive. + +| Accuracy | Precision | Recall | F1 | ROC-AUC | +|---:|---:|---:|---:|---:| +| 0.708 | 0.360 | 0.574 | 0.443 | 0.726 | + +Confusion matrix: + +| | Predicted not busy | Predicted busy | +|---|---:|---:| +| **Actually not busy** | 3,377 (TN) | 1,178 (FP) | +| **Actually busy** | 492 (FN) | 663 (TP) | + +Derived: + +- Specificity (true-negative rate): 0.741 +- Negative predictive value: 0.873 +- False-positive rate: 0.259 +- False-negative rate: 0.426 +- Precision lift over the 20.2% base rate: 1.78× +- Accuracy of the majority-class baseline ("always not busy"): 0.798 + +The model is **less accurate than always saying "not busy"**, but it finds 663 +of 1,155 busy hours (57.4% recall) that the majority baseline never finds. The +balanced weights push it toward over-flagging: 1,178 false positives for 1,155 +actual busy hours. That is why precision is 0.36 and accuracy is not the right +headline. + +Per-sensor holdout: + +| Sensor | n | Pos. rate | Accuracy | Precision | Recall | F1 | +|---|---:|---:|---:|---:|---:|---:| +| `density-01` | 714 | 0.111 | 0.761 | 0.300 | 0.873 | 0.447 | +| `hn-goetzentuem-01` | 714 | 0.197 | 0.732 | 0.391 | 0.638 | 0.485 | +| `hn-hagenbuchersee-01` | 714 | 0.049 | 0.724 | 0.148 | 0.971 | 0.257 | +| `hn-hagenbuchersee-02` | 714 | 0.123 | 0.783 | 0.355 | 0.932 | 0.514 | +| `hn-marahaus-01` | 714 | 0.055 | 0.710 | 0.135 | 0.795 | 0.230 | +| `hn-neckarufer-01` | 713 | 0.090 | 0.764 | 0.274 | 0.984 | 0.429 | +| `hn-soleo-01` | 714 | 0.704 | 0.581 | 0.943 | 0.431 | 0.592 | +| `hn-stadtgalerie-01` | 713 | 0.289 | 0.604 | 0.335 | 0.374 | 0.353 | + +The pooled model has the lowest accuracy on `hn-soleo-01` (0.581, an unusually +high base rate of 70.4%) and the lowest recall on `hn-stadtgalerie-01` (0.374). +The high recall on low-base-rate sensors is mostly the balanced weights firing; +it is not evidence of a strong signal there. + +## Reproduce + +```bash +# Full challenge CSV, prints every number above. +make small + +# Explicit CSV, all rows, save the baked artifact. +uv run python scripts/train_small.py --csv new-data.csv --all --save + +# Machine-readable payload. +uv run python scripts/train_small.py --json +``` + +The runtime path is the same function (`pulsecast.small.train_small_from_csv`): + +```bash +curl -s -X POST localhost:18000/train \ + -H "Authorization: Bearer $CONTROL_TOKEN" \ + -F model=small -F file=@new-data.csv +curl -s localhost:18000/train/ \ + -H "Authorization: Bearer $CONTROL_TOKEN" +``` + +The job returns `accuracy`, `precision`, `recall`, `f1`, `roc_auc`, `n_train`, +`n_test`, `positive_rate`, and `sensors`; the full confusion matrix and +per-sensor table are stored in `artifacts/small_metrics.json`. On success the +bundle is hot-swapped and persisted to `PULSECAST_MODELS/small_latest.joblib`. +A restart falls back to the baked artifact. + +## Honest limits + +- **It is a demo, not the production forecast.** The dashboard uses the global + HistGB 7-day model, not this classifier. +- **Threshold convention.** "Busy" is the sensor's own p75, recomputed from the + upload's training window. A different upload can move the thresholds and + change every metric. Do not compare runs trained on different windows. +- **No lags or sensor id.** The model sees only calendar and weather. It cannot + know that the previous hour was packed, and it pools all locations while + evaluating per sensor. +- **Weather fallback.** If an uploaded CSV does not overlap the baked weather + archive, missing weather is filled with 14 °C / 0 mm. That keeps the job + running but weakens the weather features; the returned metrics show it. +- **One chronological split.** Five months, one holdout. No cross-validation, + no annual seasonality claim, no per-weekday calibration. +- **Threshold metrics only.** This page reports hard 0.5-threshold metrics, not + a calibrated probability. The probability output is not a business forecast. +- **Counters are devices.** Footfall density is an index of estimated devices, + not a count of people. A "busy" flag inherits that uncertainty. +- **Small uploads are unstable.** A CSV with one sensor, only one class, or + fewer than four hourly timestamps fails with a clear error. A CSV with a + short window may train but produce meaningless metrics. +- **No causal claims.** Weather and holiday coefficients are associations in + this sample, not causal effects, and the classifier is not used for scenario + uplift (that overlay is explicit and separate). diff --git a/docs/train-e2e.md b/docs/train-e2e.md new file mode 100644 index 0000000..ada7c0c --- /dev/null +++ b/docs/train-e2e.md @@ -0,0 +1,118 @@ +# Training end-to-end evidence + +Reproduce with: + +```bash +uv run python scripts/e2e_train.py --port 18000 --report /tmp/e2e.json +``` + +The script starts a real `uvicorn pulsecast.server:app` process with +`CONTROL_TOKEN`, waits for startup, and drives the HTTP API only. It exits +non-zero on the first failed check. Unit/contract tests (`make check`) are +separate. + +## Run + +| Field | Value | +|---|---| +| Date | 2026-11-14, 05:33 UTC | +| Branch | `feat/ml-final` | +| Port | **18000** | +| Checks | **38 passed, 0 failed** | +| Total wall time | 10.4 s | +| Forecast version before | `histgb-2026-11-14` | +| Data version | `2026-11-07T23:51:00Z` | +| Context version | `2026-11-14T03:47:09Z` | + +## Snapshot and robustness (20 checks) + +- `/healthz` → `{"ok":true}`; `/readyz` → 200 `model=true`. +- `/meta` covers 8 sensors and the baked model beats the seasonal-naive + baseline on the holdout. +- `/bootstrap` → 8 sensors, all 8 hourly series (23,952 points), 168 forecast + weather hours, 24 holidays, versions equal to `/meta`. +- `/predict`: **128/128** combinations (8 sensors × 4 horizons × 4 scenarios) + returned the contract point counts (`15m` 1, `1h` 4, `48h` 48, `7d` 168), + monotonic UTC timestamps, non-negative values, and `lo ≤ v ≤ hi`. +- `/simulate`: all 4 scenarios × 8 sensors × 4 horizons returned valid points. +- Weather override changes the forecast (`-8 °C / 6 mm` vs `30 °C / 0 mm`). +- Unknown sensor → 404; missing token → 401 `unauthorized`; wrong token → 401; + unknown job → 404 `not_found`; invalid model → 400 `bad_request`; missing + JSON path → 400. + +## Small training job (real 60-day CSV subset) + +Input: 36,846 density rows, 8 sensors, 2026-09-08T23:53Z → 2026-11-07T23:51Z. + +| Field | Value | +|---|---| +| Job ID | `0b35347963974cbbbe584b77ecd4b846` | +| Status | `succeeded` | +| Model version | `logit-20261114-053344` | +| Wall time (submit → terminal) | **0.172 s** (contract < 5 s) | +| n_train / n_test | 8,151 / 2,301 | +| Base rate | 0.180 | +| Accuracy | 0.824 | +| Precision | 0.542 | +| Recall | 0.157 | +| F1 | 0.243 | +| ROC-AUC | 0.813 | + +This is a different window from the baked full-window model in +`docs/small-model.md`, so the numbers are not directly comparable: the p75 +target and the 80/20 split are recomputed inside the 60-day upload. The +low recall illustrates the honest limit noted in the small-model doc — a short, +recent window with a different threshold is a different classification problem. + +Post-conditions verified: + +- `small_latest.joblib` and `-small.joblib` exist under + `PULSECAST_MODELS`. +- `/meta.small_model.model_version` equals the job version and its metrics + match the job exactly (in-memory hot-swap). +- `/predict` still returns 168 valid points after the swap. + +## Forecast training job (full challenge CSV) and 409 + +Input: full `schwarz-pax.csv`, 19,383,568 bytes. + +| Field | Value | +|---|---| +| Job ID | `235209a0d00f449ba8be717eb76386d6` | +| Status | `succeeded` | +| Model version | `histgb-20261114-053349` | +| Wall time | **4.380 s** | +| MAE | 3.32 | +| WAPE | 27.5% | +| R² | 0.777 | +| Baseline MAE / WAPE / R² | 5.40 / 44.7% / 0.462 | +| n_eval | 5,350 | + +The ablation in the job payload matches `artifacts/metrics.json` row for row +(3.96, 3.84, 3.36, 4.08, 3.32). While this job was in `running`, a second +`POST /train` returned **409** `conflict`, proving one job at a time. + +Post-conditions verified: + +- `forecast_latest.joblib` and `-forecast.joblib` exist under + `PULSECAST_MODELS`. +- `/meta.model_version` equals the retrained version and + `/meta.model_source` is `retrained`. +- `/predict` reports the retrained version; `/readyz` stays 200. +- `data_version` after the retrain is `2026-11-07T23:00:00Z` (the hourly input + maximum) rather than the raw-file floor `23:51:00Z`; both describe the same + observed window. + +## Failure isolation + +A malformed CSV (`not,a,pulsecast,file`) was accepted as a job (202), then +failed with a clear error: + +```text +CSV needs a time column (time, timestamp, datetime, date, hour) and a value +column (value, density, footfall, count, pax, visitors); got ['not', 'a', +'pulsecast', 'file'] +``` + +`/healthz` still returned 200 afterward: a bad upload cannot take down the +service or block `/predict`. diff --git a/model_card.md b/model_card.md index 5f684cb..97ed55a 100644 --- a/model_card.md +++ b/model_card.md @@ -33,7 +33,11 @@ Day-ahead ablation models additionally use `lag_day`, `lag_2d`, `roll_24`, ## Target and data - Target: mean of 15-minute `densityData` readings in a UTC hour, per sensor. -- Training window: 2026-06-12 → 2026-10-10; holdout: 2026-10-10 → 2026-11-07. +- Training rows: 2026-06-12T00:00:00Z → 2026-10-10T22:00:00Z, 18,602 rows. +- Holdout: 2026-10-10T23:00:00Z → 2026-11-07T23:00:00Z, 5,350 + evaluation rows (hours with a seasonal-naive baseline). +- The production bundle is refit on all observed rows (through + 2026-11-07T23:00:00Z) only after the holdout is evaluated. - Eight usable sensors after excluding `maker-fair` (73 rows, Hannover), `niklas-01` (8 rows), `hn-bollwerksturm-01` (5 rows). - `density-01` merges two physical devices (`863f75b0`, `f8c19730`) into one @@ -57,8 +61,8 @@ Time-based 28-day holdout, no shuffling. Seasonal naive is the baseline. WAPE is | 7-day-safe, no weather | 4.08 | 33.8% | 0.697 | | **7-day-safe, + temperature/rain (production)** | **3.32** | **27.5%** | **0.777** | -Weather is the useful signal: adding temperature and rain cuts MAE by ~19% -versus the 7-day no-weather model and ~38% versus seasonal naive. The holiday +Weather is the useful signal: adding temperature and rain cuts MAE by 18.6% +versus the 7-day no-weather model and 38.5% versus seasonal naive. The holiday flag alone changes day-ahead MAE by −0.12 in this run; holidays are context, not a headline accuracy claim. @@ -69,23 +73,63 @@ the previous *observed* same weekday/hour across sensor gaps, while the baseline uses the exact 168-row lag. R² differs slightly from the reference 0.779 (0.777 here) because of that feature choice. +Per-sensor holdout MAE improves over each sensor's own seasonal naive by +27.7%–43.9%; the full table, figures, and provenance are in +[`docs/metrics.md`](docs/metrics.md). The largest absolute error is +Stadtgalerie (MAE 5.83, its baseline 10.39); Soleo has the smallest relative +gain (27.7%) and lowest R² (0.548). Every sensor has a positive R² under the +model; two seasonal-naive baselines are negative. + ## Uncertainty band The `lo`/`hi` band is the 10th/90th percentile of holdout residuals added to the -point forecast, clipped at zero. It is an empirical 80% band, not a calibrated -guarantee. +point forecast, clipped at zero: p10 −4.83, p50 −0.59, p90 +4.88. It is an +empirical 80% band, not a calibrated guarantee; the negative median residual +means the point forecast is slightly optimistic on the holdout. -## Small classifier +## Small classifier ("I coded an AI" artifact) + +This model is separate from the production forecaster and powers the +`model=small` retraining demo. Full definition, thresholds, confusion matrix +interpretation, and limits: [`docs/small-model.md`](docs/small-model.md). - Target: `busy` = hourly footfall strictly above that sensor's p75, where the - p75 is computed on the training window only. -- Features: `hour`, `weekday`, `is_weekend`, `temp_c`, `precip_mm`, `is_holiday`. -- Model: standardized logistic regression, `class_weight="balanced"`. -- Split: last 20% of hourly timestamps, time-ordered. -- Result: accuracy 0.708, precision 0.360, recall 0.574, F1 0.443, ROC-AUC 0.726. -- Trains in ~0.25 s on the challenge CSV and under 1 s on a small upload. -- The low precision reflects a hard p75 threshold and a deliberately tiny model; - it is a demo artifact, not the production forecast. + p75 is computed on the training window only and then frozen for the holdout. +- Features: `hour`, `weekday`, `is_weekend`, `temp_c`, `precip_mm`, `is_holiday` + (no lags, no sensor identifier, no target history). +- Model: standardized logistic regression, `class_weight="balanced"`, `C=1.0`, + `max_iter=1000`, `random_state=42`. +- Split: last 20% of hourly timestamps, time-ordered. Train + 2026-06-12T00:00Z → 2026-10-09T01:00Z (18,242 rows); test + 2026-10-09T02:00Z → 2026-11-07T23:00Z (5,710 rows). +- Result (`logit-20261114-052723`): accuracy 0.708, precision 0.360, recall + 0.574, F1 0.443, ROC-AUC 0.726, base rate 0.202. +- Confusion matrix: TN 3,377, FP 1,178, FN 492, TP 663. Specificity 0.741, + negative predictive value 0.873. +- Fit time: 0.30 s on the full challenge CSV; the API job (parse + fit) must + stay under the 5 s contract. The live e2e run trained a 60-day upload subset + in 0.17 s wall time. +- Per-sensor F1 ranges from 0.230 (Marahaus, 5.5% positive) to 0.592 (Soleo, + 70.4% positive). The balanced weights over-flag low-base-rate sensors; accuracy + is lowest on Soleo (0.581) and recall is lowest on Stadtgalerie (0.374). +- Accuracy is below the 0.798 majority-class baseline. Do not use accuracy as + the headline: the model finds 663 of 1,155 busy hours, which the majority + baseline never finds. The low precision reflects a hard p75 threshold, a + deliberately tiny model, and balanced class weights. +- This is a demo artifact, not the production forecast. Thresholds move with + the upload window, and a CSV without weather overlap falls back to + 14 °C / 0 mm. + +## Artifacts and reproduction + +- `artifacts/metrics.json` — versions, headline, ablation, per-sensor, small + model summary. +- `artifacts/small_metrics.json` — full small-model payload including the + confusion matrix, per-sensor table, and thresholds. +- `make train` regenerates models and metrics; `make small` prints the + small-model numbers; `make figures` regenerates `docs/figures/*.svg`. +- `docs/metrics.md` is the paper support sheet; it quotes the JSON and does not + invent numbers. ## No-leakage rules