Files
ML-py/docs/metrics.md
T
Noel Sabia c7ea524b5b docs: small-model artifact, paper metrics, train e2e evidence
- docs/small-model.md: target, features, split, metrics, full confusion
  matrix, per-sensor table, honest limits
- docs/metrics.md: provenance, protocol, headline, ablation, per-sensor,
  uncertainty, small model, supported/unsupported claims
- docs/figures: generated ablation, per-sensor, confusion-matrix SVGs
- docs/train-e2e.md: exact 38-check live run, small + forecast job results
- model_card.md and README updated with final metrics and links
2026-11-14 06:35:22 +01:00

188 lines
8.7 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Metrics — paper support
All values in this file are generated by `make train` and stored in
`artifacts/metrics.json` (forecast + ablation + per-sensor + small model) and
`artifacts/small_metrics.json` (small-model detail). No number here is typed by
hand: to refresh a table, rerun `make train && make figures` and copy the new
JSON, or quote the JSON directly.
## Run provenance
| Field | Value |
|---|---|
| Model version | `histgb-2026-11-14` |
| Trained at | 2026-11-14T05:27:21Z |
| Data version | 2026-11-07T23:51:00Z |
| Context (weather) version | 2026-11-14T03:47:09Z |
| Train rows | 18,602 hourly sensor rows |
| Holdout | 2026-10-10T23:00:00Z → 2026-11-07T23:00:00Z |
| Evaluation rows | 5,350 (8 sensors, hours with a seasonal-naive baseline) |
| Sensors | 8 (one merged `density-01`, seven `hn-*`) |
| Raw input | 192,464 rows, 96,232 density rows, 96,232 battery rows |
| Weather | Open-Meteo historical archive (training), forecast (scenarios) |
| Calendar | Baden-Württemberg holidays 2025–2026 |
## Protocol
- One global `HistGradientBoostingRegressor` over all eight sensors. Target is
the mean of 15-minute `densityData` readings in a UTC hour.
- Time-based 28-day holdout measured back from the last observed hour. No
shuffling, no random split.
- Seasonal naive baseline: the exact 168-row lag (same observed hour one week
earlier), evaluated on the same rows.
- The production bundle is refit on all observed rows **after** the holdout is
evaluated; the holdout is never in the production fit.
- WAPE = `sum(|error|) / sum(|actual|)`. R² is computed on the same rows.
- The production 7-day model uses only features at least 168 hours old, so a
168-hour forecast needs no recursion.
- Weather in the holdout is observed, not forecast. Live accuracy will be lower
than the table because the dashboard uses forecast weather.
## Headline
| Model | MAE | WAPE | R² |
|---|---:|---:|---:|
| Seasonal naive (exact 168-row lag) | 5.40 | 44.7% | 0.462 |
| **Weather-aware 7-day (production)** | **3.32** | **27.5%** | **0.777** |
- MAE improvement over seasonal naive: **38.5%**.
- WAPE improvement: 17.2 percentage points.
- The mission reference numbers for seasonal naive (5.40 / 44.7% / 0.462) are
reproduced exactly.
- The production row is slightly better than the reference 7-day model
(4.37 / 36.2% / 0.672) because `lag_week` uses the previous *observed* same
weekday/hour across sensor gaps instead of recursion.
## Ablation
| Model | Description | MAE | WAPE | R² | n |
|---|---|---|---:|---:|---:|---:|
| `seasonal_naive` | exact 168-row lag (same observed hour last week) | 5.40 | 44.7% | 0.462 | 5,350 |
| `day_no_weather` | calendar + lags 24/48/168 + 24h/168h rolling means | 3.96 | 32.8% | 0.706 | 5,350 |
| `day_holiday` | day-ahead + Baden-Württemberg holiday flag | 3.84 | 31.8% | 0.732 | 5,350 |
| `day_weather` | day-ahead + temperature + rain (observed weather) | 3.36 | 27.8% | 0.763 | 5,350 |
| `week_no_weather` | 7-day-safe calendar + previous same weekday/hour values | 4.08 | 33.8% | 0.697 | 5,350 |
| **`week_weather`** | **7-day-safe + temperature + rain (production)** | **3.32** | **27.5%** | **0.777** | **5,350** |
Derived effects:
- Weather at day-ahead: 3.96 → 3.36 MAE, **−15.2%**.
- Weather at 7-day-safe: 4.08 → 3.32 MAE, **−18.6%**.
- Holiday flag at day-ahead: 3.96 → 3.84 MAE, **−0.12 MAE**. Holidays are
context, not a headline accuracy claim.
- The mission reference day-ahead weather row (3.36 / 27.8%) is reproduced.
![Ablation MAE by feature set](figures/ablation_mae.svg)
*Figure 1. Holdout MAE for each feature set; teal is the production model.
Weather is the largest single gain; the holiday flag is a small addition.*
## Per sensor
Production model and seasonal naive on the same holdout rows.
| Sensor | Name | n | MAE | WAPE | R² | Baseline MAE | Baseline WAPE | Baseline R² | MAE improvement |
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|
| `density-01` | Density 01 | 669 | 2.26 | 30.8% | 0.722 | 3.58 | 48.8% | 0.238 | 36.8% |
| `hn-goetzentuem-01` | Götzenturm | 669 | 2.40 | 26.9% | 0.718 | 3.84 | 43.0% | 0.281 | 37.5% |
| `hn-hagenbuchersee-01` | Hagenbuchersee 1 | 669 | 2.47 | 38.6% | 0.602 | 4.01 | 62.8% | −0.212 | 38.5% |
| `hn-hagenbuchersee-02` | Hagenbuchersee 2 | 669 | 4.12 | 31.1% | 0.731 | 6.49 | 48.9% | 0.337 | 36.5% |
| `hn-marahaus-01` | Marahaus | 669 | 2.67 | 29.8% | 0.671 | 4.57 | 51.0% | −0.045 | 41.7% |
| `hn-neckarufer-01` | Neckarufer | 668 | 3.63 | 28.2% | 0.695 | 5.89 | 45.8% | 0.110 | 38.4% |
| `hn-soleo-01` | Soleo | 669 | 3.19 | 20.2% | 0.548 | 4.40 | 28.0% | 0.163 | 27.7% |
| `hn-stadtgalerie-01` | Stadtgalerie | 668 | 5.83 | 25.3% | 0.765 | 10.39 | 45.1% | 0.480 | 43.9% |
- Every sensor improves over its own seasonal naive baseline (27.7%–43.9%).
- Stadtgalerie has the largest absolute error (MAE 5.83) and the largest
volume; Soleo has the smallest relative gain and the lowest R² (0.548).
- Two baselines have negative R² (`hagenbuchersee-01`, `marahaus-01`), which is
expected for a naive lag on a short, variable holdout; the model is positive
on all eight.
- Sensor totals: 669 × 6 + 668 × 2 = 5,350, matching the headline.
![Per-sensor MAE](figures/per_sensor_mae.svg)
*Figure 2. Per-sensor holdout MAE, production model (blue) vs seasonal naive
(grey). Every sensor improves; the model is not carried by one large sensor.*
## Uncertainty band
The `lo`/`hi` band is the holdout residual p10/p90 added to the point forecast,
clipped at zero:
| p10 | p50 | p90 |
|---:|---:|---:|
| −4.83 | −0.59 | +4.88 |
This is an empirical 80% band, not a calibrated guarantee, and the median
residual being −0.59 shows a small optimistic bias on the holdout.
## Small busy-hour classifier
Full definition and limitations: [`docs/small-model.md`](small-model.md).
Summary from the same baked run (`logit-20261114-052723`):
| Accuracy | Precision | Recall | F1 | ROC-AUC | Train rows | Test rows | Base rate |
|---:|---:|---:|---:|---:|---:|---:|---:|
| 0.708 | 0.360 | 0.574 | 0.443 | 0.726 | 18,242 | 5,710 | 0.202 |
| | Predicted not busy | Predicted busy |
|---|---:|---:|
| **Actually not busy** | 3,377 (TN) | 1,178 (FP) |
| **Actually busy** | 492 (FN) | 663 (TP) |
![Small model confusion matrix](figures/small_model_confusion.svg)
*Figure 3. Holdout confusion matrix for the small busy-hour classifier. The
balanced weights trade precision for recall; accuracy 0.708 is below the 0.798
majority-class baseline, so report recall/F1 and ROC-AUC instead.*
Per-sensor small-model holdout:
| Sensor | n | Positive rate | Accuracy | Precision | Recall | F1 |
|---|---:|---:|---:|---:|---:|---:|
| `density-01` | 714 | 0.111 | 0.761 | 0.300 | 0.873 | 0.447 |
| `hn-goetzentuem-01` | 714 | 0.197 | 0.732 | 0.391 | 0.638 | 0.485 |
| `hn-hagenbuchersee-01` | 714 | 0.049 | 0.724 | 0.148 | 0.971 | 0.257 |
| `hn-hagenbuchersee-02` | 714 | 0.123 | 0.783 | 0.355 | 0.932 | 0.514 |
| `hn-marahaus-01` | 714 | 0.055 | 0.710 | 0.135 | 0.795 | 0.230 |
| `hn-neckarufer-01` | 713 | 0.090 | 0.764 | 0.274 | 0.984 | 0.429 |
| `hn-soleo-01` | 714 | 0.704 | 0.581 | 0.943 | 0.431 | 0.592 |
| `hn-stadtgalerie-01` | 713 | 0.289 | 0.604 | 0.335 | 0.374 | 0.353 |
## Claims the data supports
- A global, weather-aware HistGB forecasts hourly footfall for all eight
inner-city sensors with MAE 3.32 / WAPE 27.5% / R² 0.777 on a 28-day
time-based holdout.
- Weather features are the main measurable gain: −18.6% MAE versus the 7-day
no-weather model, −38.5% versus seasonal naive.
- Every sensor improves over its own seasonal naive baseline (27.7%–43.9%).
- A tiny logistic model can flag busy hours with ROC-AUC 0.726 and 57.4%
recall at 36.0% precision, retrained from a CSV in under 5 seconds.
## Claims the data does not support
- People counts. Values are an index of estimated devices; penetration is
unknown.
- Annual seasonality, city-wide surfaces, or store-location recommendations
from five months at eight sensors in ~580 m.
- Causal weather effects. The ablation is predictive, not a treatment effect.
- Live accuracy equal to the holdout. The holdout uses observed weather; the
demo uses forecast weather and `predict` adds scenario overlays.
- Holidays as a large driver. The flag moves day-ahead MAE by only −0.12.
- The small classifier as the production forecast. It is a demo artifact with
six features and a per-window p75 target.
- Tuned hyperparameters. The model configuration is fixed; the 28-day holdout
is the honest signal, not a search result.
## Reproduce
```bash
make data # parse + weather + holidays + JSON artifacts
make train # holdout, ablation, per-sensor, small model, forecasts
make small # print the small-model metrics and confusion matrix
make figures # regenerate docs/figures/*.svg from artifacts/metrics.json
make check # sync --frozen, ruff, compileall, pytest
```