docs: small-model artifact, paper metrics, train e2e evidence

- docs/small-model.md: target, features, split, metrics, full confusion
  matrix, per-sensor table, honest limits
- docs/metrics.md: provenance, protocol, headline, ablation, per-sensor,
  uncertainty, small model, supported/unsupported claims
- docs/figures: generated ablation, per-sensor, confusion-matrix SVGs
- docs/train-e2e.md: exact 38-check live run, small + forecast job results
- model_card.md and README updated with final metrics and links
This commit is contained in:
Noel Sabia committed 2026-11-14 06:35:22 +01:00
1 parent 2bcf66c550
commit c7ea524b5b
8 files changed
+702 -16

No files matched your search

+58 -14
View File
@@ -33,7 +33,11 @@ Day-ahead ablation models additionally use `lag_day`, `lag_2d`, `roll_24`,
## Target and data
- Target: mean of 15-minute `densityData` readings in a UTC hour, per sensor.
- Training window: 2026-06-12 → 2026-10-10; holdout: 2026-10-10 → 2026-11-07.
- Training rows: 2026-06-12T00:00:00Z → 2026-10-10T22:00:00Z, 18,602 rows.
- Holdout: 2026-10-10T23:00:00Z → 2026-11-07T23:00:00Z, 5,350
evaluation rows (hours with a seasonal-naive baseline).
- The production bundle is refit on all observed rows (through
2026-11-07T23:00:00Z) only after the holdout is evaluated.
- Eight usable sensors after excluding `maker-fair` (73 rows, Hannover),
`niklas-01` (8 rows), `hn-bollwerksturm-01` (5 rows).
- `density-01` merges two physical devices (`863f75b0`, `f8c19730`) into one
@@ -57,8 +61,8 @@ Time-based 28-day holdout, no shuffling. Seasonal naive is the baseline. WAPE is
| 7-day-safe, no weather | 4.08 | 33.8% | 0.697 |
| **7-day-safe, + temperature/rain (production)** | **3.32** | **27.5%** | **0.777** |
Weather is the useful signal: adding temperature and rain cuts MAE by ~19%
versus the 7-day no-weather model and ~38% versus seasonal naive. The holiday
Weather is the useful signal: adding temperature and rain cuts MAE by 18.6%
versus the 7-day no-weather model and 38.5% versus seasonal naive. The holiday
flag alone changes day-ahead MAE by −0.12 in this run; holidays are context, not
a headline accuracy claim.
@@ -69,23 +73,63 @@ the previous *observed* same weekday/hour across sensor gaps, while the baseline
uses the exact 168-row lag. R² differs slightly from the reference 0.779
(0.777 here) because of that feature choice.
Per-sensor holdout MAE improves over each sensor's own seasonal naive by
27.7%–43.9%; the full table, figures, and provenance are in
[`docs/metrics.md`](docs/metrics.md). The largest absolute error is
Stadtgalerie (MAE 5.83, its baseline 10.39); Soleo has the smallest relative
gain (27.7%) and lowest R² (0.548). Every sensor has a positive R² under the
model; two seasonal-naive baselines are negative.
## Uncertainty band
The `lo`/`hi` band is the 10th/90th percentile of holdout residuals added to the
point forecast, clipped at zero. It is an empirical 80% band, not a calibrated
guarantee.
point forecast, clipped at zero: p10 −4.83, p50 −0.59, p90 +4.88. It is an
empirical 80% band, not a calibrated guarantee; the negative median residual
means the point forecast is slightly optimistic on the holdout.
## Small classifier
## Small classifier ("I coded an AI" artifact)
This model is separate from the production forecaster and powers the
`model=small` retraining demo. Full definition, thresholds, confusion matrix
interpretation, and limits: [`docs/small-model.md`](docs/small-model.md).
- Target: `busy` = hourly footfall strictly above that sensor's p75, where the
p75 is computed on the training window only.
- Features: `hour`, `weekday`, `is_weekend`, `temp_c`, `precip_mm`, `is_holiday`.
- Model: standardized logistic regression, `class_weight="balanced"`.
- Split: last 20% of hourly timestamps, time-ordered.
- Result: accuracy 0.708, precision 0.360, recall 0.574, F1 0.443, ROC-AUC 0.726.
- Trains in ~0.25 s on the challenge CSV and under 1 s on a small upload.
- The low precision reflects a hard p75 threshold and a deliberately tiny model;
it is a demo artifact, not the production forecast.
p75 is computed on the training window only and then frozen for the holdout.
- Features: `hour`, `weekday`, `is_weekend`, `temp_c`, `precip_mm`, `is_holiday`
(no lags, no sensor identifier, no target history).
- Model: standardized logistic regression, `class_weight="balanced"`, `C=1.0`,
`max_iter=1000`, `random_state=42`.
- Split: last 20% of hourly timestamps, time-ordered. Train
2026-06-12T00:00Z → 2026-10-09T01:00Z (18,242 rows); test
2026-10-09T02:00Z → 2026-11-07T23:00Z (5,710 rows).
- Result (`logit-20261114-052723`): accuracy 0.708, precision 0.360, recall
0.574, F1 0.443, ROC-AUC 0.726, base rate 0.202.
- Confusion matrix: TN 3,377, FP 1,178, FN 492, TP 663. Specificity 0.741,
negative predictive value 0.873.
- Fit time: 0.30 s on the full challenge CSV; the API job (parse + fit) must
stay under the 5 s contract. The live e2e run trained a 60-day upload subset
in 0.17 s wall time.
- Per-sensor F1 ranges from 0.230 (Marahaus, 5.5% positive) to 0.592 (Soleo,
70.4% positive). The balanced weights over-flag low-base-rate sensors; accuracy
is lowest on Soleo (0.581) and recall is lowest on Stadtgalerie (0.374).
- Accuracy is below the 0.798 majority-class baseline. Do not use accuracy as
the headline: the model finds 663 of 1,155 busy hours, which the majority
baseline never finds. The low precision reflects a hard p75 threshold, a
deliberately tiny model, and balanced class weights.
- This is a demo artifact, not the production forecast. Thresholds move with
the upload window, and a CSV without weather overlap falls back to
14 °C / 0 mm.
## Artifacts and reproduction
- `artifacts/metrics.json` — versions, headline, ablation, per-sensor, small
model summary.
- `artifacts/small_metrics.json` — full small-model payload including the
confusion matrix, per-sensor table, and thresholds.
- `make train` regenerates models and metrics; `make small` prints the
small-model numbers; `make figures` regenerates `docs/figures/*.svg`.
- `docs/metrics.md` is the paper support sheet; it quotes the JSON and does not
invent numbers.
## No-leakage rules