docs: small-model artifact, paper metrics, train e2e evidence

- docs/small-model.md: target, features, split, metrics, full confusion
  matrix, per-sensor table, honest limits
- docs/metrics.md: provenance, protocol, headline, ablation, per-sensor,
  uncertainty, small model, supported/unsupported claims
- docs/figures: generated ablation, per-sensor, confusion-matrix SVGs
- docs/train-e2e.md: exact 38-check live run, small + forecast job results
- model_card.md and README updated with final metrics and links
This commit is contained in:
Noel Sabia committed 2026-11-14 06:35:22 +01:00
1 parent 2bcf66c550
commit c7ea524b5b
8 files changed
+702 -16

No files matched your search

+187
View File
@@ -0,0 +1,187 @@
# Metrics — paper support
All values in this file are generated by `make train` and stored in
`artifacts/metrics.json` (forecast + ablation + per-sensor + small model) and
`artifacts/small_metrics.json` (small-model detail). No number here is typed by
hand: to refresh a table, rerun `make train && make figures` and copy the new
JSON, or quote the JSON directly.
## Run provenance
| Field | Value |
|---|---|
| Model version | `histgb-2026-11-14` |
| Trained at | 2026-11-14T05:27:21Z |
| Data version | 2026-11-07T23:51:00Z |
| Context (weather) version | 2026-11-14T03:47:09Z |
| Train rows | 18,602 hourly sensor rows |
| Holdout | 2026-10-10T23:00:00Z → 2026-11-07T23:00:00Z |
| Evaluation rows | 5,350 (8 sensors, hours with a seasonal-naive baseline) |
| Sensors | 8 (one merged `density-01`, seven `hn-*`) |
| Raw input | 192,464 rows, 96,232 density rows, 96,232 battery rows |
| Weather | Open-Meteo historical archive (training), forecast (scenarios) |
| Calendar | Baden-Württemberg holidays 2025–2026 |
## Protocol
- One global `HistGradientBoostingRegressor` over all eight sensors. Target is
the mean of 15-minute `densityData` readings in a UTC hour.
- Time-based 28-day holdout measured back from the last observed hour. No
shuffling, no random split.
- Seasonal naive baseline: the exact 168-row lag (same observed hour one week
earlier), evaluated on the same rows.
- The production bundle is refit on all observed rows **after** the holdout is
evaluated; the holdout is never in the production fit.
- WAPE = `sum(|error|) / sum(|actual|)`. R² is computed on the same rows.
- The production 7-day model uses only features at least 168 hours old, so a
168-hour forecast needs no recursion.
- Weather in the holdout is observed, not forecast. Live accuracy will be lower
than the table because the dashboard uses forecast weather.
## Headline
| Model | MAE | WAPE | R² |
|---|---:|---:|---:|
| Seasonal naive (exact 168-row lag) | 5.40 | 44.7% | 0.462 |
| **Weather-aware 7-day (production)** | **3.32** | **27.5%** | **0.777** |
- MAE improvement over seasonal naive: **38.5%**.
- WAPE improvement: 17.2 percentage points.
- The mission reference numbers for seasonal naive (5.40 / 44.7% / 0.462) are
reproduced exactly.
- The production row is slightly better than the reference 7-day model
(4.37 / 36.2% / 0.672) because `lag_week` uses the previous *observed* same
weekday/hour across sensor gaps instead of recursion.
## Ablation
| Model | Description | MAE | WAPE | R² | n |
|---|---|---|---:|---:|---:|---:|
| `seasonal_naive` | exact 168-row lag (same observed hour last week) | 5.40 | 44.7% | 0.462 | 5,350 |
| `day_no_weather` | calendar + lags 24/48/168 + 24h/168h rolling means | 3.96 | 32.8% | 0.706 | 5,350 |
| `day_holiday` | day-ahead + Baden-Württemberg holiday flag | 3.84 | 31.8% | 0.732 | 5,350 |
| `day_weather` | day-ahead + temperature + rain (observed weather) | 3.36 | 27.8% | 0.763 | 5,350 |
| `week_no_weather` | 7-day-safe calendar + previous same weekday/hour values | 4.08 | 33.8% | 0.697 | 5,350 |
| **`week_weather`** | **7-day-safe + temperature + rain (production)** | **3.32** | **27.5%** | **0.777** | **5,350** |
Derived effects:
- Weather at day-ahead: 3.96 → 3.36 MAE, **−15.2%**.
- Weather at 7-day-safe: 4.08 → 3.32 MAE, **−18.6%**.
- Holiday flag at day-ahead: 3.96 → 3.84 MAE, **−0.12 MAE**. Holidays are
context, not a headline accuracy claim.
- The mission reference day-ahead weather row (3.36 / 27.8%) is reproduced.
![Ablation MAE by feature set](figures/ablation_mae.svg)
*Figure 1. Holdout MAE for each feature set; teal is the production model.
Weather is the largest single gain; the holiday flag is a small addition.*
## Per sensor
Production model and seasonal naive on the same holdout rows.
| Sensor | Name | n | MAE | WAPE | R² | Baseline MAE | Baseline WAPE | Baseline R² | MAE improvement |
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|
| `density-01` | Density 01 | 669 | 2.26 | 30.8% | 0.722 | 3.58 | 48.8% | 0.238 | 36.8% |
| `hn-goetzentuem-01` | Götzenturm | 669 | 2.40 | 26.9% | 0.718 | 3.84 | 43.0% | 0.281 | 37.5% |
| `hn-hagenbuchersee-01` | Hagenbuchersee 1 | 669 | 2.47 | 38.6% | 0.602 | 4.01 | 62.8% | −0.212 | 38.5% |
| `hn-hagenbuchersee-02` | Hagenbuchersee 2 | 669 | 4.12 | 31.1% | 0.731 | 6.49 | 48.9% | 0.337 | 36.5% |
| `hn-marahaus-01` | Marahaus | 669 | 2.67 | 29.8% | 0.671 | 4.57 | 51.0% | −0.045 | 41.7% |
| `hn-neckarufer-01` | Neckarufer | 668 | 3.63 | 28.2% | 0.695 | 5.89 | 45.8% | 0.110 | 38.4% |
| `hn-soleo-01` | Soleo | 669 | 3.19 | 20.2% | 0.548 | 4.40 | 28.0% | 0.163 | 27.7% |
| `hn-stadtgalerie-01` | Stadtgalerie | 668 | 5.83 | 25.3% | 0.765 | 10.39 | 45.1% | 0.480 | 43.9% |
- Every sensor improves over its own seasonal naive baseline (27.7%–43.9%).
- Stadtgalerie has the largest absolute error (MAE 5.83) and the largest
volume; Soleo has the smallest relative gain and the lowest R² (0.548).
- Two baselines have negative R² (`hagenbuchersee-01`, `marahaus-01`), which is
expected for a naive lag on a short, variable holdout; the model is positive
on all eight.
- Sensor totals: 669 × 6 + 668 × 2 = 5,350, matching the headline.
![Per-sensor MAE](figures/per_sensor_mae.svg)
*Figure 2. Per-sensor holdout MAE, production model (blue) vs seasonal naive
(grey). Every sensor improves; the model is not carried by one large sensor.*
## Uncertainty band
The `lo`/`hi` band is the holdout residual p10/p90 added to the point forecast,
clipped at zero:
| p10 | p50 | p90 |
|---:|---:|---:|
| −4.83 | −0.59 | +4.88 |
This is an empirical 80% band, not a calibrated guarantee, and the median
residual being −0.59 shows a small optimistic bias on the holdout.
## Small busy-hour classifier
Full definition and limitations: [`docs/small-model.md`](small-model.md).
Summary from the same baked run (`logit-20261114-052723`):
| Accuracy | Precision | Recall | F1 | ROC-AUC | Train rows | Test rows | Base rate |
|---:|---:|---:|---:|---:|---:|---:|---:|
| 0.708 | 0.360 | 0.574 | 0.443 | 0.726 | 18,242 | 5,710 | 0.202 |
| | Predicted not busy | Predicted busy |
|---|---:|---:|
| **Actually not busy** | 3,377 (TN) | 1,178 (FP) |
| **Actually busy** | 492 (FN) | 663 (TP) |
![Small model confusion matrix](figures/small_model_confusion.svg)
*Figure 3. Holdout confusion matrix for the small busy-hour classifier. The
balanced weights trade precision for recall; accuracy 0.708 is below the 0.798
majority-class baseline, so report recall/F1 and ROC-AUC instead.*
Per-sensor small-model holdout:
| Sensor | n | Positive rate | Accuracy | Precision | Recall | F1 |
|---|---:|---:|---:|---:|---:|---:|
| `density-01` | 714 | 0.111 | 0.761 | 0.300 | 0.873 | 0.447 |
| `hn-goetzentuem-01` | 714 | 0.197 | 0.732 | 0.391 | 0.638 | 0.485 |
| `hn-hagenbuchersee-01` | 714 | 0.049 | 0.724 | 0.148 | 0.971 | 0.257 |
| `hn-hagenbuchersee-02` | 714 | 0.123 | 0.783 | 0.355 | 0.932 | 0.514 |
| `hn-marahaus-01` | 714 | 0.055 | 0.710 | 0.135 | 0.795 | 0.230 |
| `hn-neckarufer-01` | 713 | 0.090 | 0.764 | 0.274 | 0.984 | 0.429 |
| `hn-soleo-01` | 714 | 0.704 | 0.581 | 0.943 | 0.431 | 0.592 |
| `hn-stadtgalerie-01` | 713 | 0.289 | 0.604 | 0.335 | 0.374 | 0.353 |
## Claims the data supports
- A global, weather-aware HistGB forecasts hourly footfall for all eight
inner-city sensors with MAE 3.32 / WAPE 27.5% / R² 0.777 on a 28-day
time-based holdout.
- Weather features are the main measurable gain: −18.6% MAE versus the 7-day
no-weather model, −38.5% versus seasonal naive.
- Every sensor improves over its own seasonal naive baseline (27.7%–43.9%).
- A tiny logistic model can flag busy hours with ROC-AUC 0.726 and 57.4%
recall at 36.0% precision, retrained from a CSV in under 5 seconds.
## Claims the data does not support
- People counts. Values are an index of estimated devices; penetration is
unknown.
- Annual seasonality, city-wide surfaces, or store-location recommendations
from five months at eight sensors in ~580 m.
- Causal weather effects. The ablation is predictive, not a treatment effect.
- Live accuracy equal to the holdout. The holdout uses observed weather; the
demo uses forecast weather and `predict` adds scenario overlays.
- Holidays as a large driver. The flag moves day-ahead MAE by only −0.12.
- The small classifier as the production forecast. It is a demo artifact with
six features and a per-window p75 target.
- Tuned hyperparameters. The model configuration is fixed; the 28-day holdout
is the honest signal, not a search result.
## Reproduce
```bash
make data # parse + weather + holidays + JSON artifacts
make train # holdout, ablation, per-sensor, small model, forecasts
make small # print the small-model metrics and confusion matrix
make figures # regenerate docs/figures/*.svg from artifacts/metrics.json
make check # sync --frozen, ruff, compileall, pytest
```