ci / check (push) Successful in 1m12s
Replace setup-python with setup-uv: the self-hosted Gitea runner lacks the toolcache and lsb_release dependency, so the job failed before any project step. uv installs its own Python 3.12; the checks are unchanged. Align the README and docs with the committed artifacts: quick start serves from baked artifacts, rebuild steps are separate, and the reference-model comparison no longer overstates the result. Drop stale external-doc references and comments that restated the code.
189 lines
8.8 KiB
Markdown
189 lines
8.8 KiB
Markdown
# Metrics — paper support
|
||
|
||
Every value in this file comes from the committed artifacts:
|
||
`artifacts/metrics.json` (forecast + ablation + per-sensor + small model),
|
||
`artifacts/small_metrics.json` (small-model detail), and
|
||
`artifacts/data_audit.json` (raw-input counts). To refresh a table, rerun
|
||
`make train && make figures` and quote the new JSON.
|
||
|
||
## Run provenance
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Model version | `histgb-2026-11-14` |
|
||
| Trained at | 2026-11-14T05:27:21Z |
|
||
| Data version | 2026-11-07T23:51:00Z |
|
||
| Context (weather) version | 2026-11-14T03:47:09Z |
|
||
| Train rows | 18,602 hourly sensor rows |
|
||
| Holdout | 2026-10-10T23:00:00Z → 2026-11-07T23:00:00Z |
|
||
| Evaluation rows | 5,350 (8 sensors, hours with a seasonal-naive baseline) |
|
||
| Sensors | 8 (one merged `density-01`, seven `hn-*`) |
|
||
| Raw input | 192,464 rows, 96,232 density rows, 96,232 battery rows |
|
||
| Weather | Open-Meteo historical archive (training), forecast (scenarios) |
|
||
| Calendar | Baden-Württemberg holidays 2025–2026 |
|
||
|
||
## Protocol
|
||
|
||
- One global `HistGradientBoostingRegressor` over all eight sensors. Target is
|
||
the mean of 15-minute `densityData` readings in a UTC hour.
|
||
- Time-based 28-day holdout measured back from the last observed hour. No
|
||
shuffling, no random split.
|
||
- Seasonal naive baseline: the exact 168-row lag (same observed hour one week
|
||
earlier), evaluated on the same rows.
|
||
- The production bundle is refit on all observed rows **after** the holdout is
|
||
evaluated; the holdout is never in the production fit.
|
||
- WAPE = `sum(|error|) / sum(|actual|)`. R² is computed on the same rows.
|
||
- The production 7-day model uses only features at least 168 hours old, so a
|
||
168-hour forecast needs no recursion.
|
||
- Weather in the holdout is observed, not forecast. Live accuracy will be lower
|
||
than the table because the dashboard uses forecast weather.
|
||
|
||
## Headline
|
||
|
||
| Model | MAE | WAPE | R² |
|
||
|---|---:|---:|---:|
|
||
| Seasonal naive (exact 168-row lag) | 5.40 | 44.7% | 0.462 |
|
||
| **Weather-aware 7-day (production)** | **3.32** | **27.5%** | **0.777** |
|
||
|
||
- MAE improvement over seasonal naive: **38.5%**.
|
||
- WAPE improvement: 17.2 percentage points.
|
||
- The mission reference numbers for seasonal naive (5.40 / 44.7% / 0.462) are
|
||
reproduced exactly.
|
||
- The production row also beats the mission reference 7-day no-weather model
|
||
(4.37 / 36.2% / 0.672): weather supplies most of the gain, and `lag_week`
|
||
uses the previous *observed* same weekday/hour across sensor gaps instead of
|
||
recursion.
|
||
|
||
## Ablation
|
||
|
||
| Model | Description | MAE | WAPE | R² | n |
|
||
|---|---|---|---:|---:|---:|---:|
|
||
| `seasonal_naive` | exact 168-row lag (same observed hour last week) | 5.40 | 44.7% | 0.462 | 5,350 |
|
||
| `day_no_weather` | calendar + lags 24/48/168 + 24h/168h rolling means | 3.96 | 32.8% | 0.706 | 5,350 |
|
||
| `day_holiday` | day-ahead + Baden-Württemberg holiday flag | 3.84 | 31.8% | 0.732 | 5,350 |
|
||
| `day_weather` | day-ahead + temperature + rain (observed weather) | 3.36 | 27.8% | 0.763 | 5,350 |
|
||
| `week_no_weather` | 7-day-safe calendar + previous same weekday/hour values | 4.08 | 33.8% | 0.697 | 5,350 |
|
||
| **`week_weather`** | **7-day-safe + temperature + rain (production)** | **3.32** | **27.5%** | **0.777** | **5,350** |
|
||
|
||
Derived effects:
|
||
|
||
- Weather at day-ahead: 3.96 → 3.36 MAE, **−15.2%**.
|
||
- Weather at 7-day-safe: 4.08 → 3.32 MAE, **−18.6%**.
|
||
- Holiday flag at day-ahead: 3.96 → 3.84 MAE, **−0.12 MAE**. Holidays are
|
||
context, not a headline accuracy claim.
|
||
- The mission reference day-ahead weather row (3.36 / 27.8%) is reproduced.
|
||
|
||

|
||
|
||
*Figure 1. Holdout MAE for each feature set; teal is the production model.
|
||
Weather is the largest single gain; the holiday flag is a small addition.*
|
||
|
||
## Per sensor
|
||
|
||
Production model and seasonal naive on the same holdout rows.
|
||
|
||
| Sensor | Name | n | MAE | WAPE | R² | Baseline MAE | Baseline WAPE | Baseline R² | MAE improvement |
|
||
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|
|
||
| `density-01` | Density 01 | 669 | 2.26 | 30.8% | 0.722 | 3.58 | 48.8% | 0.238 | 36.8% |
|
||
| `hn-goetzentuem-01` | Götzenturm | 669 | 2.40 | 26.9% | 0.718 | 3.84 | 43.0% | 0.281 | 37.5% |
|
||
| `hn-hagenbuchersee-01` | Hagenbuchersee 1 | 669 | 2.47 | 38.6% | 0.602 | 4.01 | 62.8% | −0.212 | 38.5% |
|
||
| `hn-hagenbuchersee-02` | Hagenbuchersee 2 | 669 | 4.12 | 31.1% | 0.731 | 6.49 | 48.9% | 0.337 | 36.5% |
|
||
| `hn-marahaus-01` | Marahaus | 669 | 2.67 | 29.8% | 0.671 | 4.57 | 51.0% | −0.045 | 41.7% |
|
||
| `hn-neckarufer-01` | Neckarufer | 668 | 3.63 | 28.2% | 0.695 | 5.89 | 45.8% | 0.110 | 38.4% |
|
||
| `hn-soleo-01` | Soleo | 669 | 3.19 | 20.2% | 0.548 | 4.40 | 28.0% | 0.163 | 27.7% |
|
||
| `hn-stadtgalerie-01` | Stadtgalerie | 668 | 5.83 | 25.3% | 0.765 | 10.39 | 45.1% | 0.480 | 43.9% |
|
||
|
||
- Every sensor improves over its own seasonal naive baseline (27.7%–43.9%).
|
||
- Stadtgalerie has the largest absolute error (MAE 5.83) and the largest
|
||
volume; Soleo has the smallest relative gain and the lowest R² (0.548).
|
||
- Two baselines have negative R² (`hagenbuchersee-01`, `marahaus-01`), which is
|
||
expected for a naive lag on a short, variable holdout; the model is positive
|
||
on all eight.
|
||
- Sensor totals: 669 × 6 + 668 × 2 = 5,350, matching the headline.
|
||
|
||

|
||
|
||
*Figure 2. Per-sensor holdout MAE, production model (blue) vs seasonal naive
|
||
(grey). Every sensor improves; the model is not carried by one large sensor.*
|
||
|
||
## Uncertainty band
|
||
|
||
The `lo`/`hi` band is the holdout residual p10/p90 added to the point forecast,
|
||
clipped at zero:
|
||
|
||
| p10 | p50 | p90 |
|
||
|---:|---:|---:|
|
||
| −4.83 | −0.59 | +4.88 |
|
||
|
||
This is an empirical 80% band, not a calibrated guarantee, and the median
|
||
residual being −0.59 shows a small optimistic bias on the holdout.
|
||
|
||
## Small busy-hour classifier
|
||
|
||
Full definition and limitations: [`docs/small-model.md`](small-model.md).
|
||
Summary from the same baked run (`logit-20261114-052723`):
|
||
|
||
| Accuracy | Precision | Recall | F1 | ROC-AUC | Train rows | Test rows | Base rate |
|
||
|---:|---:|---:|---:|---:|---:|---:|---:|
|
||
| 0.708 | 0.360 | 0.574 | 0.443 | 0.726 | 18,242 | 5,710 | 0.202 |
|
||
|
||
| | Predicted not busy | Predicted busy |
|
||
|---|---:|---:|
|
||
| **Actually not busy** | 3,377 (TN) | 1,178 (FP) |
|
||
| **Actually busy** | 492 (FN) | 663 (TP) |
|
||
|
||

|
||
|
||
*Figure 3. Holdout confusion matrix for the small busy-hour classifier. The
|
||
balanced weights trade precision for recall; accuracy 0.708 is below the 0.798
|
||
majority-class baseline, so report recall/F1 and ROC-AUC instead.*
|
||
|
||
Per-sensor small-model holdout:
|
||
|
||
| Sensor | n | Positive rate | Accuracy | Precision | Recall | F1 |
|
||
|---|---:|---:|---:|---:|---:|---:|
|
||
| `density-01` | 714 | 0.111 | 0.761 | 0.300 | 0.873 | 0.447 |
|
||
| `hn-goetzentuem-01` | 714 | 0.197 | 0.732 | 0.391 | 0.638 | 0.485 |
|
||
| `hn-hagenbuchersee-01` | 714 | 0.049 | 0.724 | 0.148 | 0.971 | 0.257 |
|
||
| `hn-hagenbuchersee-02` | 714 | 0.123 | 0.783 | 0.355 | 0.932 | 0.514 |
|
||
| `hn-marahaus-01` | 714 | 0.055 | 0.710 | 0.135 | 0.795 | 0.230 |
|
||
| `hn-neckarufer-01` | 713 | 0.090 | 0.764 | 0.274 | 0.984 | 0.429 |
|
||
| `hn-soleo-01` | 714 | 0.704 | 0.581 | 0.943 | 0.431 | 0.592 |
|
||
| `hn-stadtgalerie-01` | 713 | 0.289 | 0.604 | 0.335 | 0.374 | 0.353 |
|
||
|
||
## Claims the data supports
|
||
|
||
- A global, weather-aware HistGB forecasts hourly footfall for all eight
|
||
inner-city sensors with MAE 3.32 / WAPE 27.5% / R² 0.777 on a 28-day
|
||
time-based holdout.
|
||
- Weather features are the main measurable gain: −18.6% MAE versus the 7-day
|
||
no-weather model, −38.5% versus seasonal naive.
|
||
- Every sensor improves over its own seasonal naive baseline (27.7%–43.9%).
|
||
- A tiny logistic model can flag busy hours with ROC-AUC 0.726 and 57.4%
|
||
recall at 36.0% precision, retrained from a CSV in under 5 seconds.
|
||
|
||
## Claims the data does not support
|
||
|
||
- People counts. Values are an index of estimated devices; penetration is
|
||
unknown.
|
||
- Annual seasonality, city-wide surfaces, or store-location recommendations
|
||
from five months at eight sensors in ~580 m.
|
||
- Causal weather effects. The ablation is predictive, not a treatment effect.
|
||
- Live accuracy equal to the holdout. The holdout uses observed weather; the
|
||
demo uses forecast weather and `predict` adds scenario overlays.
|
||
- Holidays as a large driver. The flag moves day-ahead MAE by only −0.12.
|
||
- The small classifier as the production forecast. It is a demo artifact with
|
||
six features and a per-window p75 target.
|
||
- Tuned hyperparameters. The model configuration is fixed; the 28-day holdout
|
||
is the held-out result, not a search result.
|
||
|
||
## Reproduce
|
||
|
||
```bash
|
||
make data # parse + weather + holidays + JSON artifacts
|
||
make train # holdout, ablation, per-sensor, small model, forecasts
|
||
make small # print the small-model metrics and confusion matrix
|
||
make figures # regenerate docs/figures/*.svg from artifacts/metrics.json
|
||
make check # sync --frozen, ruff, compileall, pytest
|
||
```
|