# Metrics — paper support Every value in this file comes from the committed artifacts: `artifacts/metrics.json` (forecast + ablation + per-sensor + small model), `artifacts/small_metrics.json` (small-model detail), and `artifacts/data_audit.json` (raw-input counts). To refresh a table, rerun `make train && make figures` and quote the new JSON. ## Run provenance | Field | Value | |---|---| | Model version | `histgb-2026-11-14` | | Trained at | 2026-11-14T05:27:21Z | | Data version | 2026-11-07T23:51:00Z | | Context (weather) version | 2026-11-14T03:47:09Z | | Train rows | 18,602 hourly sensor rows | | Holdout | 2026-10-10T23:00:00Z → 2026-11-07T23:00:00Z | | Evaluation rows | 5,350 (8 sensors, hours with a seasonal-naive baseline) | | Sensors | 8 (one merged `density-01`, seven `hn-*`) | | Raw input | 192,464 rows, 96,232 density rows, 96,232 battery rows | | Weather | Open-Meteo historical archive (training), forecast (scenarios) | | Calendar | Baden-Württemberg holidays 2025–2026 | ## Protocol - One global `HistGradientBoostingRegressor` over all eight sensors. Target is the mean of 15-minute `densityData` readings in a UTC hour. - Time-based 28-day holdout measured back from the last observed hour. No shuffling, no random split. - Seasonal naive baseline: the exact 168-row lag (same observed hour one week earlier), evaluated on the same rows. - The production bundle is refit on all observed rows **after** the holdout is evaluated; the holdout is never in the production fit. - WAPE = `sum(|error|) / sum(|actual|)`. R² is computed on the same rows. - The production 7-day model uses only features at least 168 hours old, so a 168-hour forecast needs no recursion. - Weather in the holdout is observed, not forecast. Live accuracy will be lower than the table because the dashboard uses forecast weather. ## Headline | Model | MAE | WAPE | R² | |---|---:|---:|---:| | Seasonal naive (exact 168-row lag) | 5.40 | 44.7% | 0.462 | | **Weather-aware 7-day (production)** | **3.32** | **27.5%** | **0.777** | - MAE improvement over seasonal naive: **38.5%**. - WAPE improvement: 17.2 percentage points. - The mission reference numbers for seasonal naive (5.40 / 44.7% / 0.462) are reproduced exactly. - The production row also beats the mission reference 7-day no-weather model (4.37 / 36.2% / 0.672): weather supplies most of the gain, and `lag_week` uses the previous *observed* same weekday/hour across sensor gaps instead of recursion. ## Ablation | Model | Description | MAE | WAPE | R² | n | |---|---|---|---:|---:|---:|---:| | `seasonal_naive` | exact 168-row lag (same observed hour last week) | 5.40 | 44.7% | 0.462 | 5,350 | | `day_no_weather` | calendar + lags 24/48/168 + 24h/168h rolling means | 3.96 | 32.8% | 0.706 | 5,350 | | `day_holiday` | day-ahead + Baden-Württemberg holiday flag | 3.84 | 31.8% | 0.732 | 5,350 | | `day_weather` | day-ahead + temperature + rain (observed weather) | 3.36 | 27.8% | 0.763 | 5,350 | | `week_no_weather` | 7-day-safe calendar + previous same weekday/hour values | 4.08 | 33.8% | 0.697 | 5,350 | | **`week_weather`** | **7-day-safe + temperature + rain (production)** | **3.32** | **27.5%** | **0.777** | **5,350** | Derived effects: - Weather at day-ahead: 3.96 → 3.36 MAE, **−15.2%**. - Weather at 7-day-safe: 4.08 → 3.32 MAE, **−18.6%**. - Holiday flag at day-ahead: 3.96 → 3.84 MAE, **−0.12 MAE**. Holidays are context, not a headline accuracy claim. - The mission reference day-ahead weather row (3.36 / 27.8%) is reproduced. ![Ablation MAE by feature set](figures/ablation_mae.svg) *Figure 1. Holdout MAE for each feature set; teal is the production model. Weather is the largest single gain; the holiday flag is a small addition.* ## Per sensor Production model and seasonal naive on the same holdout rows. | Sensor | Name | n | MAE | WAPE | R² | Baseline MAE | Baseline WAPE | Baseline R² | MAE improvement | |---|---|---:|---:|---:|---:|---:|---:|---:|---:| | `density-01` | Density 01 | 669 | 2.26 | 30.8% | 0.722 | 3.58 | 48.8% | 0.238 | 36.8% | | `hn-goetzentuem-01` | Götzenturm | 669 | 2.40 | 26.9% | 0.718 | 3.84 | 43.0% | 0.281 | 37.5% | | `hn-hagenbuchersee-01` | Hagenbuchersee 1 | 669 | 2.47 | 38.6% | 0.602 | 4.01 | 62.8% | −0.212 | 38.5% | | `hn-hagenbuchersee-02` | Hagenbuchersee 2 | 669 | 4.12 | 31.1% | 0.731 | 6.49 | 48.9% | 0.337 | 36.5% | | `hn-marahaus-01` | Marahaus | 669 | 2.67 | 29.8% | 0.671 | 4.57 | 51.0% | −0.045 | 41.7% | | `hn-neckarufer-01` | Neckarufer | 668 | 3.63 | 28.2% | 0.695 | 5.89 | 45.8% | 0.110 | 38.4% | | `hn-soleo-01` | Soleo | 669 | 3.19 | 20.2% | 0.548 | 4.40 | 28.0% | 0.163 | 27.7% | | `hn-stadtgalerie-01` | Stadtgalerie | 668 | 5.83 | 25.3% | 0.765 | 10.39 | 45.1% | 0.480 | 43.9% | - Every sensor improves over its own seasonal naive baseline (27.7%–43.9%). - Stadtgalerie has the largest absolute error (MAE 5.83) and the largest volume; Soleo has the smallest relative gain and the lowest R² (0.548). - Two baselines have negative R² (`hagenbuchersee-01`, `marahaus-01`), which is expected for a naive lag on a short, variable holdout; the model is positive on all eight. - Sensor totals: 669 × 6 + 668 × 2 = 5,350, matching the headline. ![Per-sensor MAE](figures/per_sensor_mae.svg) *Figure 2. Per-sensor holdout MAE, production model (blue) vs seasonal naive (grey). Every sensor improves; the model is not carried by one large sensor.* ## Uncertainty band The `lo`/`hi` band is the holdout residual p10/p90 added to the point forecast, clipped at zero: | p10 | p50 | p90 | |---:|---:|---:| | −4.83 | −0.59 | +4.88 | This is an empirical 80% band, not a calibrated guarantee, and the median residual being −0.59 shows a small optimistic bias on the holdout. ## Small busy-hour classifier Full definition and limitations: [`docs/small-model.md`](small-model.md). Summary from the same baked run (`logit-20261114-052723`): | Accuracy | Precision | Recall | F1 | ROC-AUC | Train rows | Test rows | Base rate | |---:|---:|---:|---:|---:|---:|---:|---:| | 0.708 | 0.360 | 0.574 | 0.443 | 0.726 | 18,242 | 5,710 | 0.202 | | | Predicted not busy | Predicted busy | |---|---:|---:| | **Actually not busy** | 3,377 (TN) | 1,178 (FP) | | **Actually busy** | 492 (FN) | 663 (TP) | ![Small model confusion matrix](figures/small_model_confusion.svg) *Figure 3. Holdout confusion matrix for the small busy-hour classifier. The balanced weights trade precision for recall; accuracy 0.708 is below the 0.798 majority-class baseline, so report recall/F1 and ROC-AUC instead.* Per-sensor small-model holdout: | Sensor | n | Positive rate | Accuracy | Precision | Recall | F1 | |---|---:|---:|---:|---:|---:|---:| | `density-01` | 714 | 0.111 | 0.761 | 0.300 | 0.873 | 0.447 | | `hn-goetzentuem-01` | 714 | 0.197 | 0.732 | 0.391 | 0.638 | 0.485 | | `hn-hagenbuchersee-01` | 714 | 0.049 | 0.724 | 0.148 | 0.971 | 0.257 | | `hn-hagenbuchersee-02` | 714 | 0.123 | 0.783 | 0.355 | 0.932 | 0.514 | | `hn-marahaus-01` | 714 | 0.055 | 0.710 | 0.135 | 0.795 | 0.230 | | `hn-neckarufer-01` | 713 | 0.090 | 0.764 | 0.274 | 0.984 | 0.429 | | `hn-soleo-01` | 714 | 0.704 | 0.581 | 0.943 | 0.431 | 0.592 | | `hn-stadtgalerie-01` | 713 | 0.289 | 0.604 | 0.335 | 0.374 | 0.353 | ## Claims the data supports - A global, weather-aware HistGB forecasts hourly footfall for all eight inner-city sensors with MAE 3.32 / WAPE 27.5% / R² 0.777 on a 28-day time-based holdout. - Weather features are the main measurable gain: −18.6% MAE versus the 7-day no-weather model, −38.5% versus seasonal naive. - Every sensor improves over its own seasonal naive baseline (27.7%–43.9%). - A tiny logistic model can flag busy hours with ROC-AUC 0.726 and 57.4% recall at 36.0% precision, retrained from a CSV in under 5 seconds. ## Claims the data does not support - People counts. Values are an index of estimated devices; penetration is unknown. - Annual seasonality, city-wide surfaces, or store-location recommendations from five months at eight sensors in ~580 m. - Causal weather effects. The ablation is predictive, not a treatment effect. - Live accuracy equal to the holdout. The holdout uses observed weather; the demo uses forecast weather and `predict` adds scenario overlays. - Holidays as a large driver. The flag moves day-ahead MAE by only −0.12. - The small classifier as the production forecast. It is a demo artifact with six features and a per-window p75 target. - Tuned hyperparameters. The model configuration is fixed; the 28-day holdout is the held-out result, not a search result. ## Reproduce ```bash make data # parse + weather + holidays + JSON artifacts make train # holdout, ablation, per-sensor, small model, forecasts make small # print the small-model metrics and confusion matrix make figures # regenerate docs/figures/*.svg from artifacts/metrics.json make check # sync --frozen, ruff, compileall, pytest ```