docs: small-model artifact, paper metrics, train e2e evidence
- docs/small-model.md: target, features, split, metrics, full confusion matrix, per-sensor table, honest limits - docs/metrics.md: provenance, protocol, headline, ablation, per-sensor, uncertainty, small model, supported/unsupported claims - docs/figures: generated ablation, per-sensor, confusion-matrix SVGs - docs/train-e2e.md: exact 38-check live run, small + forecast job results - model_card.md and README updated with final metrics and links
This commit is contained in:
1 parent
2bcf66c550
commit
c7ea524b5b
8 files changed
+702
-16
No files matched your search
+187
@@ -0,0 +1,187 @@
|
||||
# Metrics — paper support
|
||||
|
||||
All values in this file are generated by `make train` and stored in
|
||||
`artifacts/metrics.json` (forecast + ablation + per-sensor + small model) and
|
||||
`artifacts/small_metrics.json` (small-model detail). No number here is typed by
|
||||
hand: to refresh a table, rerun `make train && make figures` and copy the new
|
||||
JSON, or quote the JSON directly.
|
||||
|
||||
## Run provenance
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Model version | `histgb-2026-11-14` |
|
||||
| Trained at | 2026-11-14T05:27:21Z |
|
||||
| Data version | 2026-11-07T23:51:00Z |
|
||||
| Context (weather) version | 2026-11-14T03:47:09Z |
|
||||
| Train rows | 18,602 hourly sensor rows |
|
||||
| Holdout | 2026-10-10T23:00:00Z → 2026-11-07T23:00:00Z |
|
||||
| Evaluation rows | 5,350 (8 sensors, hours with a seasonal-naive baseline) |
|
||||
| Sensors | 8 (one merged `density-01`, seven `hn-*`) |
|
||||
| Raw input | 192,464 rows, 96,232 density rows, 96,232 battery rows |
|
||||
| Weather | Open-Meteo historical archive (training), forecast (scenarios) |
|
||||
| Calendar | Baden-Württemberg holidays 2025–2026 |
|
||||
|
||||
## Protocol
|
||||
|
||||
- One global `HistGradientBoostingRegressor` over all eight sensors. Target is
|
||||
the mean of 15-minute `densityData` readings in a UTC hour.
|
||||
- Time-based 28-day holdout measured back from the last observed hour. No
|
||||
shuffling, no random split.
|
||||
- Seasonal naive baseline: the exact 168-row lag (same observed hour one week
|
||||
earlier), evaluated on the same rows.
|
||||
- The production bundle is refit on all observed rows **after** the holdout is
|
||||
evaluated; the holdout is never in the production fit.
|
||||
- WAPE = `sum(|error|) / sum(|actual|)`. R² is computed on the same rows.
|
||||
- The production 7-day model uses only features at least 168 hours old, so a
|
||||
168-hour forecast needs no recursion.
|
||||
- Weather in the holdout is observed, not forecast. Live accuracy will be lower
|
||||
than the table because the dashboard uses forecast weather.
|
||||
|
||||
## Headline
|
||||
|
||||
| Model | MAE | WAPE | R² |
|
||||
|---|---:|---:|---:|
|
||||
| Seasonal naive (exact 168-row lag) | 5.40 | 44.7% | 0.462 |
|
||||
| **Weather-aware 7-day (production)** | **3.32** | **27.5%** | **0.777** |
|
||||
|
||||
- MAE improvement over seasonal naive: **38.5%**.
|
||||
- WAPE improvement: 17.2 percentage points.
|
||||
- The mission reference numbers for seasonal naive (5.40 / 44.7% / 0.462) are
|
||||
reproduced exactly.
|
||||
- The production row is slightly better than the reference 7-day model
|
||||
(4.37 / 36.2% / 0.672) because `lag_week` uses the previous *observed* same
|
||||
weekday/hour across sensor gaps instead of recursion.
|
||||
|
||||
## Ablation
|
||||
|
||||
| Model | Description | MAE | WAPE | R² | n |
|
||||
|---|---|---|---:|---:|---:|---:|
|
||||
| `seasonal_naive` | exact 168-row lag (same observed hour last week) | 5.40 | 44.7% | 0.462 | 5,350 |
|
||||
| `day_no_weather` | calendar + lags 24/48/168 + 24h/168h rolling means | 3.96 | 32.8% | 0.706 | 5,350 |
|
||||
| `day_holiday` | day-ahead + Baden-Württemberg holiday flag | 3.84 | 31.8% | 0.732 | 5,350 |
|
||||
| `day_weather` | day-ahead + temperature + rain (observed weather) | 3.36 | 27.8% | 0.763 | 5,350 |
|
||||
| `week_no_weather` | 7-day-safe calendar + previous same weekday/hour values | 4.08 | 33.8% | 0.697 | 5,350 |
|
||||
| **`week_weather`** | **7-day-safe + temperature + rain (production)** | **3.32** | **27.5%** | **0.777** | **5,350** |
|
||||
|
||||
Derived effects:
|
||||
|
||||
- Weather at day-ahead: 3.96 → 3.36 MAE, **−15.2%**.
|
||||
- Weather at 7-day-safe: 4.08 → 3.32 MAE, **−18.6%**.
|
||||
- Holiday flag at day-ahead: 3.96 → 3.84 MAE, **−0.12 MAE**. Holidays are
|
||||
context, not a headline accuracy claim.
|
||||
- The mission reference day-ahead weather row (3.36 / 27.8%) is reproduced.
|
||||
|
||||

|
||||
|
||||
*Figure 1. Holdout MAE for each feature set; teal is the production model.
|
||||
Weather is the largest single gain; the holiday flag is a small addition.*
|
||||
|
||||
## Per sensor
|
||||
|
||||
Production model and seasonal naive on the same holdout rows.
|
||||
|
||||
| Sensor | Name | n | MAE | WAPE | R² | Baseline MAE | Baseline WAPE | Baseline R² | MAE improvement |
|
||||
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| `density-01` | Density 01 | 669 | 2.26 | 30.8% | 0.722 | 3.58 | 48.8% | 0.238 | 36.8% |
|
||||
| `hn-goetzentuem-01` | Götzenturm | 669 | 2.40 | 26.9% | 0.718 | 3.84 | 43.0% | 0.281 | 37.5% |
|
||||
| `hn-hagenbuchersee-01` | Hagenbuchersee 1 | 669 | 2.47 | 38.6% | 0.602 | 4.01 | 62.8% | −0.212 | 38.5% |
|
||||
| `hn-hagenbuchersee-02` | Hagenbuchersee 2 | 669 | 4.12 | 31.1% | 0.731 | 6.49 | 48.9% | 0.337 | 36.5% |
|
||||
| `hn-marahaus-01` | Marahaus | 669 | 2.67 | 29.8% | 0.671 | 4.57 | 51.0% | −0.045 | 41.7% |
|
||||
| `hn-neckarufer-01` | Neckarufer | 668 | 3.63 | 28.2% | 0.695 | 5.89 | 45.8% | 0.110 | 38.4% |
|
||||
| `hn-soleo-01` | Soleo | 669 | 3.19 | 20.2% | 0.548 | 4.40 | 28.0% | 0.163 | 27.7% |
|
||||
| `hn-stadtgalerie-01` | Stadtgalerie | 668 | 5.83 | 25.3% | 0.765 | 10.39 | 45.1% | 0.480 | 43.9% |
|
||||
|
||||
- Every sensor improves over its own seasonal naive baseline (27.7%–43.9%).
|
||||
- Stadtgalerie has the largest absolute error (MAE 5.83) and the largest
|
||||
volume; Soleo has the smallest relative gain and the lowest R² (0.548).
|
||||
- Two baselines have negative R² (`hagenbuchersee-01`, `marahaus-01`), which is
|
||||
expected for a naive lag on a short, variable holdout; the model is positive
|
||||
on all eight.
|
||||
- Sensor totals: 669 × 6 + 668 × 2 = 5,350, matching the headline.
|
||||
|
||||

|
||||
|
||||
*Figure 2. Per-sensor holdout MAE, production model (blue) vs seasonal naive
|
||||
(grey). Every sensor improves; the model is not carried by one large sensor.*
|
||||
|
||||
## Uncertainty band
|
||||
|
||||
The `lo`/`hi` band is the holdout residual p10/p90 added to the point forecast,
|
||||
clipped at zero:
|
||||
|
||||
| p10 | p50 | p90 |
|
||||
|---:|---:|---:|
|
||||
| −4.83 | −0.59 | +4.88 |
|
||||
|
||||
This is an empirical 80% band, not a calibrated guarantee, and the median
|
||||
residual being −0.59 shows a small optimistic bias on the holdout.
|
||||
|
||||
## Small busy-hour classifier
|
||||
|
||||
Full definition and limitations: [`docs/small-model.md`](small-model.md).
|
||||
Summary from the same baked run (`logit-20261114-052723`):
|
||||
|
||||
| Accuracy | Precision | Recall | F1 | ROC-AUC | Train rows | Test rows | Base rate |
|
||||
|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| 0.708 | 0.360 | 0.574 | 0.443 | 0.726 | 18,242 | 5,710 | 0.202 |
|
||||
|
||||
| | Predicted not busy | Predicted busy |
|
||||
|---|---:|---:|
|
||||
| **Actually not busy** | 3,377 (TN) | 1,178 (FP) |
|
||||
| **Actually busy** | 492 (FN) | 663 (TP) |
|
||||
|
||||

|
||||
|
||||
*Figure 3. Holdout confusion matrix for the small busy-hour classifier. The
|
||||
balanced weights trade precision for recall; accuracy 0.708 is below the 0.798
|
||||
majority-class baseline, so report recall/F1 and ROC-AUC instead.*
|
||||
|
||||
Per-sensor small-model holdout:
|
||||
|
||||
| Sensor | n | Positive rate | Accuracy | Precision | Recall | F1 |
|
||||
|---|---:|---:|---:|---:|---:|---:|
|
||||
| `density-01` | 714 | 0.111 | 0.761 | 0.300 | 0.873 | 0.447 |
|
||||
| `hn-goetzentuem-01` | 714 | 0.197 | 0.732 | 0.391 | 0.638 | 0.485 |
|
||||
| `hn-hagenbuchersee-01` | 714 | 0.049 | 0.724 | 0.148 | 0.971 | 0.257 |
|
||||
| `hn-hagenbuchersee-02` | 714 | 0.123 | 0.783 | 0.355 | 0.932 | 0.514 |
|
||||
| `hn-marahaus-01` | 714 | 0.055 | 0.710 | 0.135 | 0.795 | 0.230 |
|
||||
| `hn-neckarufer-01` | 713 | 0.090 | 0.764 | 0.274 | 0.984 | 0.429 |
|
||||
| `hn-soleo-01` | 714 | 0.704 | 0.581 | 0.943 | 0.431 | 0.592 |
|
||||
| `hn-stadtgalerie-01` | 713 | 0.289 | 0.604 | 0.335 | 0.374 | 0.353 |
|
||||
|
||||
## Claims the data supports
|
||||
|
||||
- A global, weather-aware HistGB forecasts hourly footfall for all eight
|
||||
inner-city sensors with MAE 3.32 / WAPE 27.5% / R² 0.777 on a 28-day
|
||||
time-based holdout.
|
||||
- Weather features are the main measurable gain: −18.6% MAE versus the 7-day
|
||||
no-weather model, −38.5% versus seasonal naive.
|
||||
- Every sensor improves over its own seasonal naive baseline (27.7%–43.9%).
|
||||
- A tiny logistic model can flag busy hours with ROC-AUC 0.726 and 57.4%
|
||||
recall at 36.0% precision, retrained from a CSV in under 5 seconds.
|
||||
|
||||
## Claims the data does not support
|
||||
|
||||
- People counts. Values are an index of estimated devices; penetration is
|
||||
unknown.
|
||||
- Annual seasonality, city-wide surfaces, or store-location recommendations
|
||||
from five months at eight sensors in ~580 m.
|
||||
- Causal weather effects. The ablation is predictive, not a treatment effect.
|
||||
- Live accuracy equal to the holdout. The holdout uses observed weather; the
|
||||
demo uses forecast weather and `predict` adds scenario overlays.
|
||||
- Holidays as a large driver. The flag moves day-ahead MAE by only −0.12.
|
||||
- The small classifier as the production forecast. It is a demo artifact with
|
||||
six features and a per-window p75 target.
|
||||
- Tuned hyperparameters. The model configuration is fixed; the 28-day holdout
|
||||
is the honest signal, not a search result.
|
||||
|
||||
## Reproduce
|
||||
|
||||
```bash
|
||||
make data # parse + weather + holidays + JSON artifacts
|
||||
make train # holdout, ablation, per-sensor, small model, forecasts
|
||||
make small # print the small-model metrics and confusion matrix
|
||||
make figures # regenerate docs/figures/*.svg from artifacts/metrics.json
|
||||
make check # sync --frozen, ruff, compileall, pytest
|
||||
```
|
||||
Reference in new issue
Block a user