- docs/small-model.md: target, features, split, metrics, full confusion matrix, per-sensor table, honest limits - docs/metrics.md: provenance, protocol, headline, ablation, per-sensor, uncertainty, small model, supported/unsupported claims - docs/figures: generated ablation, per-sensor, confusion-matrix SVGs - docs/train-e2e.md: exact 38-check live run, small + forecast job results - model_card.md and README updated with final metrics and links
8.7 KiB
Metrics — paper support
All values in this file are generated by make train and stored in
artifacts/metrics.json (forecast + ablation + per-sensor + small model) and
artifacts/small_metrics.json (small-model detail). No number here is typed by
hand: to refresh a table, rerun make train && make figures and copy the new
JSON, or quote the JSON directly.
Run provenance
| Field | Value |
|---|---|
| Model version | histgb-2026-11-14 |
| Trained at | 2026-11-14T05:27:21Z |
| Data version | 2026-11-07T23:51:00Z |
| Context (weather) version | 2026-11-14T03:47:09Z |
| Train rows | 18,602 hourly sensor rows |
| Holdout | 2026-10-10T23:00:00Z → 2026-11-07T23:00:00Z |
| Evaluation rows | 5,350 (8 sensors, hours with a seasonal-naive baseline) |
| Sensors | 8 (one merged density-01, seven hn-*) |
| Raw input | 192,464 rows, 96,232 density rows, 96,232 battery rows |
| Weather | Open-Meteo historical archive (training), forecast (scenarios) |
| Calendar | Baden-Württemberg holidays 2025–2026 |
Protocol
- One global
HistGradientBoostingRegressorover all eight sensors. Target is the mean of 15-minutedensityDatareadings in a UTC hour. - Time-based 28-day holdout measured back from the last observed hour. No shuffling, no random split.
- Seasonal naive baseline: the exact 168-row lag (same observed hour one week earlier), evaluated on the same rows.
- The production bundle is refit on all observed rows after the holdout is evaluated; the holdout is never in the production fit.
- WAPE =
sum(|error|) / sum(|actual|). R² is computed on the same rows. - The production 7-day model uses only features at least 168 hours old, so a 168-hour forecast needs no recursion.
- Weather in the holdout is observed, not forecast. Live accuracy will be lower than the table because the dashboard uses forecast weather.
Headline
| Model | MAE | WAPE | R² |
|---|---|---|---|
| Seasonal naive (exact 168-row lag) | 5.40 | 44.7% | 0.462 |
| Weather-aware 7-day (production) | 3.32 | 27.5% | 0.777 |
- MAE improvement over seasonal naive: 38.5%.
- WAPE improvement: 17.2 percentage points.
- The mission reference numbers for seasonal naive (5.40 / 44.7% / 0.462) are reproduced exactly.
- The production row is slightly better than the reference 7-day model
(4.37 / 36.2% / 0.672) because
lag_weekuses the previous observed same weekday/hour across sensor gaps instead of recursion.
Ablation
| Model | Description | MAE | WAPE | R² | n | |
|---|---|---|---|---|---|---|
seasonal_naive |
exact 168-row lag (same observed hour last week) | 5.40 | 44.7% | 0.462 | 5,350 | |
day_no_weather |
calendar + lags 24/48/168 + 24h/168h rolling means | 3.96 | 32.8% | 0.706 | 5,350 | |
day_holiday |
day-ahead + Baden-Württemberg holiday flag | 3.84 | 31.8% | 0.732 | 5,350 | |
day_weather |
day-ahead + temperature + rain (observed weather) | 3.36 | 27.8% | 0.763 | 5,350 | |
week_no_weather |
7-day-safe calendar + previous same weekday/hour values | 4.08 | 33.8% | 0.697 | 5,350 | |
week_weather |
7-day-safe + temperature + rain (production) | 3.32 | 27.5% | 0.777 | 5,350 |
Derived effects:
- Weather at day-ahead: 3.96 → 3.36 MAE, −15.2%.
- Weather at 7-day-safe: 4.08 → 3.32 MAE, −18.6%.
- Holiday flag at day-ahead: 3.96 → 3.84 MAE, −0.12 MAE. Holidays are context, not a headline accuracy claim.
- The mission reference day-ahead weather row (3.36 / 27.8%) is reproduced.
Figure 1. Holdout MAE for each feature set; teal is the production model. Weather is the largest single gain; the holiday flag is a small addition.
Per sensor
Production model and seasonal naive on the same holdout rows.
| Sensor | Name | n | MAE | WAPE | R² | Baseline MAE | Baseline WAPE | Baseline R² | MAE improvement |
|---|---|---|---|---|---|---|---|---|---|
density-01 |
Density 01 | 669 | 2.26 | 30.8% | 0.722 | 3.58 | 48.8% | 0.238 | 36.8% |
hn-goetzentuem-01 |
Götzenturm | 669 | 2.40 | 26.9% | 0.718 | 3.84 | 43.0% | 0.281 | 37.5% |
hn-hagenbuchersee-01 |
Hagenbuchersee 1 | 669 | 2.47 | 38.6% | 0.602 | 4.01 | 62.8% | −0.212 | 38.5% |
hn-hagenbuchersee-02 |
Hagenbuchersee 2 | 669 | 4.12 | 31.1% | 0.731 | 6.49 | 48.9% | 0.337 | 36.5% |
hn-marahaus-01 |
Marahaus | 669 | 2.67 | 29.8% | 0.671 | 4.57 | 51.0% | −0.045 | 41.7% |
hn-neckarufer-01 |
Neckarufer | 668 | 3.63 | 28.2% | 0.695 | 5.89 | 45.8% | 0.110 | 38.4% |
hn-soleo-01 |
Soleo | 669 | 3.19 | 20.2% | 0.548 | 4.40 | 28.0% | 0.163 | 27.7% |
hn-stadtgalerie-01 |
Stadtgalerie | 668 | 5.83 | 25.3% | 0.765 | 10.39 | 45.1% | 0.480 | 43.9% |
- Every sensor improves over its own seasonal naive baseline (27.7%–43.9%).
- Stadtgalerie has the largest absolute error (MAE 5.83) and the largest volume; Soleo has the smallest relative gain and the lowest R² (0.548).
- Two baselines have negative R² (
hagenbuchersee-01,marahaus-01), which is expected for a naive lag on a short, variable holdout; the model is positive on all eight. - Sensor totals: 669 × 6 + 668 × 2 = 5,350, matching the headline.
Figure 2. Per-sensor holdout MAE, production model (blue) vs seasonal naive (grey). Every sensor improves; the model is not carried by one large sensor.
Uncertainty band
The lo/hi band is the holdout residual p10/p90 added to the point forecast,
clipped at zero:
| p10 | p50 | p90 |
|---|---|---|
| −4.83 | −0.59 | +4.88 |
This is an empirical 80% band, not a calibrated guarantee, and the median residual being −0.59 shows a small optimistic bias on the holdout.
Small busy-hour classifier
Full definition and limitations: docs/small-model.md.
Summary from the same baked run (logit-20261114-052723):
| Accuracy | Precision | Recall | F1 | ROC-AUC | Train rows | Test rows | Base rate |
|---|---|---|---|---|---|---|---|
| 0.708 | 0.360 | 0.574 | 0.443 | 0.726 | 18,242 | 5,710 | 0.202 |
| Predicted not busy | Predicted busy | |
|---|---|---|
| Actually not busy | 3,377 (TN) | 1,178 (FP) |
| Actually busy | 492 (FN) | 663 (TP) |
Figure 3. Holdout confusion matrix for the small busy-hour classifier. The balanced weights trade precision for recall; accuracy 0.708 is below the 0.798 majority-class baseline, so report recall/F1 and ROC-AUC instead.
Per-sensor small-model holdout:
| Sensor | n | Positive rate | Accuracy | Precision | Recall | F1 |
|---|---|---|---|---|---|---|
density-01 |
714 | 0.111 | 0.761 | 0.300 | 0.873 | 0.447 |
hn-goetzentuem-01 |
714 | 0.197 | 0.732 | 0.391 | 0.638 | 0.485 |
hn-hagenbuchersee-01 |
714 | 0.049 | 0.724 | 0.148 | 0.971 | 0.257 |
hn-hagenbuchersee-02 |
714 | 0.123 | 0.783 | 0.355 | 0.932 | 0.514 |
hn-marahaus-01 |
714 | 0.055 | 0.710 | 0.135 | 0.795 | 0.230 |
hn-neckarufer-01 |
713 | 0.090 | 0.764 | 0.274 | 0.984 | 0.429 |
hn-soleo-01 |
714 | 0.704 | 0.581 | 0.943 | 0.431 | 0.592 |
hn-stadtgalerie-01 |
713 | 0.289 | 0.604 | 0.335 | 0.374 | 0.353 |
Claims the data supports
- A global, weather-aware HistGB forecasts hourly footfall for all eight inner-city sensors with MAE 3.32 / WAPE 27.5% / R² 0.777 on a 28-day time-based holdout.
- Weather features are the main measurable gain: −18.6% MAE versus the 7-day no-weather model, −38.5% versus seasonal naive.
- Every sensor improves over its own seasonal naive baseline (27.7%–43.9%).
- A tiny logistic model can flag busy hours with ROC-AUC 0.726 and 57.4% recall at 36.0% precision, retrained from a CSV in under 5 seconds.
Claims the data does not support
- People counts. Values are an index of estimated devices; penetration is unknown.
- Annual seasonality, city-wide surfaces, or store-location recommendations from five months at eight sensors in ~580 m.
- Causal weather effects. The ablation is predictive, not a treatment effect.
- Live accuracy equal to the holdout. The holdout uses observed weather; the
demo uses forecast weather and
predictadds scenario overlays. - Holidays as a large driver. The flag moves day-ahead MAE by only −0.12.
- The small classifier as the production forecast. It is a demo artifact with six features and a per-window p75 target.
- Tuned hyperparameters. The model configuration is fixed; the 28-day holdout is the honest signal, not a search result.
Reproduce
make data # parse + weather + holidays + JSON artifacts
make train # holdout, ablation, per-sensor, small model, forecasts
make small # print the small-model metrics and confusion matrix
make figures # regenerate docs/figures/*.svg from artifacts/metrics.json
make check # sync --frozen, ruff, compileall, pytest