Files
ML-py/docs/metrics.md
Noel Sabia 070ecd2a78
ci / check (push) Successful in 1m12s
ci: fix Gitea runner setup and tighten docs
Replace setup-python with setup-uv: the self-hosted Gitea runner lacks the
toolcache and lsb_release dependency, so the job failed before any project
step. uv installs its own Python 3.12; the checks are unchanged.

Align the README and docs with the committed artifacts: quick start serves
from baked artifacts, rebuild steps are separate, and the reference-model
comparison no longer overstates the result. Drop stale external-doc
references and comments that restated the code.
2026-11-14 09:10:47 +01:00

8.8 KiB
Raw Permalink Blame History

Metrics — paper support

Every value in this file comes from the committed artifacts: artifacts/metrics.json (forecast + ablation + per-sensor + small model), artifacts/small_metrics.json (small-model detail), and artifacts/data_audit.json (raw-input counts). To refresh a table, rerun make train && make figures and quote the new JSON.

Run provenance

Field Value
Model version histgb-2026-11-14
Trained at 2026-11-14T05:27:21Z
Data version 2026-11-07T23:51:00Z
Context (weather) version 2026-11-14T03:47:09Z
Train rows 18,602 hourly sensor rows
Holdout 2026-10-10T23:00:00Z → 2026-11-07T23:00:00Z
Evaluation rows 5,350 (8 sensors, hours with a seasonal-naive baseline)
Sensors 8 (one merged density-01, seven hn-*)
Raw input 192,464 rows, 96,232 density rows, 96,232 battery rows
Weather Open-Meteo historical archive (training), forecast (scenarios)
Calendar Baden-Württemberg holidays 2025–2026

Protocol

  • One global HistGradientBoostingRegressor over all eight sensors. Target is the mean of 15-minute densityData readings in a UTC hour.
  • Time-based 28-day holdout measured back from the last observed hour. No shuffling, no random split.
  • Seasonal naive baseline: the exact 168-row lag (same observed hour one week earlier), evaluated on the same rows.
  • The production bundle is refit on all observed rows after the holdout is evaluated; the holdout is never in the production fit.
  • WAPE = sum(|error|) / sum(|actual|). R² is computed on the same rows.
  • The production 7-day model uses only features at least 168 hours old, so a 168-hour forecast needs no recursion.
  • Weather in the holdout is observed, not forecast. Live accuracy will be lower than the table because the dashboard uses forecast weather.

Headline

Model MAE WAPE R²
Seasonal naive (exact 168-row lag) 5.40 44.7% 0.462
Weather-aware 7-day (production) 3.32 27.5% 0.777
  • MAE improvement over seasonal naive: 38.5%.
  • WAPE improvement: 17.2 percentage points.
  • The mission reference numbers for seasonal naive (5.40 / 44.7% / 0.462) are reproduced exactly.
  • The production row also beats the mission reference 7-day no-weather model (4.37 / 36.2% / 0.672): weather supplies most of the gain, and lag_week uses the previous observed same weekday/hour across sensor gaps instead of recursion.

Ablation

Model Description MAE WAPE R² n
seasonal_naive exact 168-row lag (same observed hour last week) 5.40 44.7% 0.462 5,350
day_no_weather calendar + lags 24/48/168 + 24h/168h rolling means 3.96 32.8% 0.706 5,350
day_holiday day-ahead + Baden-Württemberg holiday flag 3.84 31.8% 0.732 5,350
day_weather day-ahead + temperature + rain (observed weather) 3.36 27.8% 0.763 5,350
week_no_weather 7-day-safe calendar + previous same weekday/hour values 4.08 33.8% 0.697 5,350
week_weather 7-day-safe + temperature + rain (production) 3.32 27.5% 0.777 5,350

Derived effects:

  • Weather at day-ahead: 3.96 → 3.36 MAE, −15.2%.
  • Weather at 7-day-safe: 4.08 → 3.32 MAE, −18.6%.
  • Holiday flag at day-ahead: 3.96 → 3.84 MAE, −0.12 MAE. Holidays are context, not a headline accuracy claim.
  • The mission reference day-ahead weather row (3.36 / 27.8%) is reproduced.

Ablation MAE by feature set

Figure 1. Holdout MAE for each feature set; teal is the production model. Weather is the largest single gain; the holiday flag is a small addition.

Per sensor

Production model and seasonal naive on the same holdout rows.

Sensor Name n MAE WAPE R² Baseline MAE Baseline WAPE Baseline R² MAE improvement
density-01 Density 01 669 2.26 30.8% 0.722 3.58 48.8% 0.238 36.8%
hn-goetzentuem-01 Götzenturm 669 2.40 26.9% 0.718 3.84 43.0% 0.281 37.5%
hn-hagenbuchersee-01 Hagenbuchersee 1 669 2.47 38.6% 0.602 4.01 62.8% −0.212 38.5%
hn-hagenbuchersee-02 Hagenbuchersee 2 669 4.12 31.1% 0.731 6.49 48.9% 0.337 36.5%
hn-marahaus-01 Marahaus 669 2.67 29.8% 0.671 4.57 51.0% −0.045 41.7%
hn-neckarufer-01 Neckarufer 668 3.63 28.2% 0.695 5.89 45.8% 0.110 38.4%
hn-soleo-01 Soleo 669 3.19 20.2% 0.548 4.40 28.0% 0.163 27.7%
hn-stadtgalerie-01 Stadtgalerie 668 5.83 25.3% 0.765 10.39 45.1% 0.480 43.9%
  • Every sensor improves over its own seasonal naive baseline (27.7%–43.9%).
  • Stadtgalerie has the largest absolute error (MAE 5.83) and the largest volume; Soleo has the smallest relative gain and the lowest R² (0.548).
  • Two baselines have negative R² (hagenbuchersee-01, marahaus-01), which is expected for a naive lag on a short, variable holdout; the model is positive on all eight.
  • Sensor totals: 669 × 6 + 668 × 2 = 5,350, matching the headline.

Per-sensor MAE

Figure 2. Per-sensor holdout MAE, production model (blue) vs seasonal naive (grey). Every sensor improves; the model is not carried by one large sensor.

Uncertainty band

The lo/hi band is the holdout residual p10/p90 added to the point forecast, clipped at zero:

p10 p50 p90
−4.83 −0.59 +4.88

This is an empirical 80% band, not a calibrated guarantee, and the median residual being −0.59 shows a small optimistic bias on the holdout.

Small busy-hour classifier

Full definition and limitations: docs/small-model.md. Summary from the same baked run (logit-20261114-052723):

Accuracy Precision Recall F1 ROC-AUC Train rows Test rows Base rate
0.708 0.360 0.574 0.443 0.726 18,242 5,710 0.202
Predicted not busy Predicted busy
Actually not busy 3,377 (TN) 1,178 (FP)
Actually busy 492 (FN) 663 (TP)

Small model confusion matrix

Figure 3. Holdout confusion matrix for the small busy-hour classifier. The balanced weights trade precision for recall; accuracy 0.708 is below the 0.798 majority-class baseline, so report recall/F1 and ROC-AUC instead.

Per-sensor small-model holdout:

Sensor n Positive rate Accuracy Precision Recall F1
density-01 714 0.111 0.761 0.300 0.873 0.447
hn-goetzentuem-01 714 0.197 0.732 0.391 0.638 0.485
hn-hagenbuchersee-01 714 0.049 0.724 0.148 0.971 0.257
hn-hagenbuchersee-02 714 0.123 0.783 0.355 0.932 0.514
hn-marahaus-01 714 0.055 0.710 0.135 0.795 0.230
hn-neckarufer-01 713 0.090 0.764 0.274 0.984 0.429
hn-soleo-01 714 0.704 0.581 0.943 0.431 0.592
hn-stadtgalerie-01 713 0.289 0.604 0.335 0.374 0.353

Claims the data supports

  • A global, weather-aware HistGB forecasts hourly footfall for all eight inner-city sensors with MAE 3.32 / WAPE 27.5% / R² 0.777 on a 28-day time-based holdout.
  • Weather features are the main measurable gain: −18.6% MAE versus the 7-day no-weather model, −38.5% versus seasonal naive.
  • Every sensor improves over its own seasonal naive baseline (27.7%–43.9%).
  • A tiny logistic model can flag busy hours with ROC-AUC 0.726 and 57.4% recall at 36.0% precision, retrained from a CSV in under 5 seconds.

Claims the data does not support

  • People counts. Values are an index of estimated devices; penetration is unknown.
  • Annual seasonality, city-wide surfaces, or store-location recommendations from five months at eight sensors in ~580 m.
  • Causal weather effects. The ablation is predictive, not a treatment effect.
  • Live accuracy equal to the holdout. The holdout uses observed weather; the demo uses forecast weather and predict adds scenario overlays.
  • Holidays as a large driver. The flag moves day-ahead MAE by only −0.12.
  • The small classifier as the production forecast. It is a demo artifact with six features and a per-window p75 target.
  • Tuned hyperparameters. The model configuration is fixed; the 28-day holdout is the held-out result, not a search result.

Reproduce

make data      # parse + weather + holidays + JSON artifacts
make train     # holdout, ablation, per-sensor, small model, forecasts
make small     # print the small-model metrics and confusion matrix
make figures   # regenerate docs/figures/*.svg from artifacts/metrics.json
make check     # sync --frozen, ruff, compileall, pytest