docs: small-model artifact, paper metrics, train e2e evidence

- docs/small-model.md: target, features, split, metrics, full confusion
  matrix, per-sensor table, honest limits
- docs/metrics.md: provenance, protocol, headline, ablation, per-sensor,
  uncertainty, small model, supported/unsupported claims
- docs/figures: generated ablation, per-sensor, confusion-matrix SVGs
- docs/train-e2e.md: exact 38-check live run, small + forecast job results
- model_card.md and README updated with final metrics and links
This commit is contained in:
Noel Sabia committed 2026-11-14 06:35:22 +01:00
1 parent 2bcf66c550
commit c7ea524b5b
8 files changed
+702 -16

No files matched your search

+36
View File
@@ -0,0 +1,36 @@
<svg xmlns="http://www.w3.org/2000/svg" width="960" height="370" viewBox="0 0 960 370" role="img" aria-label="Ablation: holdout MAE by feature set">
<rect width="100%" height="100%" fill="#ffffff"/>
<g font-family="-apple-system, Segoe UI, Helvetica, Arial, sans-serif">
<text x="24" y="34" font-size="20" font-weight="600" fill="#0f172a">Ablation: holdout MAE by feature set</text>
<text x="24" y="56" font-size="13" fill="#475569">Holdout 2026-10-10T23:00:00Z .. 2026-11-07T23:00:00Z, n=5350</text>
<line x1="230.0" y1="68" x2="230.0" y2="314" stroke="#e2e8f0" stroke-width="1"/>
<text x="230.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">0.0</text>
<line x1="388.5" y1="68" x2="388.5" y2="314" stroke="#e2e8f0" stroke-width="1"/>
<text x="388.5" y="62" font-size="11" fill="#64748b" text-anchor="middle">1.5</text>
<line x1="547.0" y1="68" x2="547.0" y2="314" stroke="#e2e8f0" stroke-width="1"/>
<text x="547.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">2.9</text>
<line x1="705.5" y1="68" x2="705.5" y2="314" stroke="#e2e8f0" stroke-width="1"/>
<text x="705.5" y="62" font-size="11" fill="#64748b" text-anchor="middle">4.4</text>
<line x1="864.0" y1="68" x2="864.0" y2="314" stroke="#e2e8f0" stroke-width="1"/>
<text x="864.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">5.8</text>
<text x="218" y="106" font-size="12" fill="#0f172a" text-anchor="end">seasonal naive</text>
<rect x="230" y="92" width="587.0" height="22" fill="#2563eb"/>
<text x="823.0" y="107" font-size="12" fill="#0f172a">5.40</text>
<text x="218" y="144" font-size="12" fill="#0f172a" text-anchor="end">day no weather</text>
<rect x="230" y="130" width="430.5" height="22" fill="#2563eb"/>
<text x="666.5" y="145" font-size="12" fill="#0f172a">3.96</text>
<text x="218" y="182" font-size="12" fill="#0f172a" text-anchor="end">day holiday</text>
<rect x="230" y="168" width="417.4" height="22" fill="#2563eb"/>
<text x="653.4" y="183" font-size="12" fill="#0f172a">3.84</text>
<text x="218" y="220" font-size="12" fill="#0f172a" text-anchor="end">day weather</text>
<rect x="230" y="206" width="365.3" height="22" fill="#2563eb"/>
<text x="601.3" y="221" font-size="12" fill="#0f172a">3.36</text>
<text x="218" y="258" font-size="12" fill="#0f172a" text-anchor="end">week no weather</text>
<rect x="230" y="244" width="443.5" height="22" fill="#2563eb"/>
<text x="679.5" y="259" font-size="12" fill="#0f172a">4.08</text>
<text x="218" y="296" font-size="12" fill="#0f172a" text-anchor="end">week weather</text>
<rect x="230" y="282" width="360.9" height="22" fill="#0f766e"/>
<text x="596.9" y="297" font-size="12" fill="#0f172a">3.32</text>
<text x="24" y="358" font-size="11" fill="#64748b">Source: artifacts/metrics.json · PulseCast 28-day time-based holdout</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 2.7 KiB

+60
View File
@@ -0,0 +1,60 @@
<svg xmlns="http://www.w3.org/2000/svg" width="960" height="446" viewBox="0 0 960 446" role="img" aria-label="Per-sensor holdout MAE: production model vs seasonal naive">
<rect width="100%" height="100%" fill="#ffffff"/>
<g font-family="-apple-system, Segoe UI, Helvetica, Arial, sans-serif">
<text x="24" y="34" font-size="20" font-weight="600" fill="#0f172a">Per-sensor holdout MAE: production model vs seasonal naive</text>
<text x="24" y="56" font-size="13" fill="#475569">Holdout 2026-10-10T23:00:00Z .. 2026-11-07T23:00:00Z, n=5350</text>
<line x1="230.0" y1="68" x2="230.0" y2="390" stroke="#e2e8f0" stroke-width="1"/>
<text x="230.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">0.0</text>
<line x1="388.5" y1="68" x2="388.5" y2="390" stroke="#e2e8f0" stroke-width="1"/>
<text x="388.5" y="62" font-size="11" fill="#64748b" text-anchor="middle">2.8</text>
<line x1="547.0" y1="68" x2="547.0" y2="390" stroke="#e2e8f0" stroke-width="1"/>
<text x="547.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">5.6</text>
<line x1="705.5" y1="68" x2="705.5" y2="390" stroke="#e2e8f0" stroke-width="1"/>
<text x="705.5" y="62" font-size="11" fill="#64748b" text-anchor="middle">8.4</text>
<line x1="864.0" y1="68" x2="864.0" y2="390" stroke="#e2e8f0" stroke-width="1"/>
<text x="864.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">11.2</text>
<rect x="230" y="52" width="10" height="10" fill="#94a3b8"/><text x="246" y="61" font-size="12" fill="#475569">seasonal naive</text>
<rect x="370" y="52" width="10" height="10" fill="#2563eb"/><text x="386" y="61" font-size="12" fill="#475569">weather-7d</text>
<text x="218" y="106" font-size="12" fill="#0f172a" text-anchor="end">Density 01</text>
<rect x="230" y="89" width="202.3" height="12" fill="#94a3b8"/>
<text x="438.3" y="99" font-size="11" fill="#64748b">3.58</text>
<rect x="230" y="104" width="127.7" height="12" fill="#2563eb"/>
<text x="363.7" y="114" font-size="11" fill="#0f172a">2.26</text>
<text x="218" y="144" font-size="12" fill="#0f172a" text-anchor="end">Götzenturm</text>
<rect x="230" y="127" width="217.0" height="12" fill="#94a3b8"/>
<text x="453.0" y="137" font-size="11" fill="#64748b">3.84</text>
<rect x="230" y="142" width="135.6" height="12" fill="#2563eb"/>
<text x="371.6" y="152" font-size="11" fill="#0f172a">2.40</text>
<text x="218" y="182" font-size="12" fill="#0f172a" text-anchor="end">Hagenbuchersee 1</text>
<rect x="230" y="165" width="226.6" height="12" fill="#94a3b8"/>
<text x="462.6" y="175" font-size="11" fill="#64748b">4.01</text>
<rect x="230" y="180" width="139.6" height="12" fill="#2563eb"/>
<text x="375.6" y="190" font-size="11" fill="#0f172a">2.47</text>
<text x="218" y="220" font-size="12" fill="#0f172a" text-anchor="end">Hagenbuchersee 2</text>
<rect x="230" y="203" width="366.7" height="12" fill="#94a3b8"/>
<text x="602.7" y="213" font-size="11" fill="#64748b">6.49</text>
<rect x="230" y="218" width="232.8" height="12" fill="#2563eb"/>
<text x="468.8" y="228" font-size="11" fill="#0f172a">4.12</text>
<text x="218" y="258" font-size="12" fill="#0f172a" text-anchor="end">Marahaus</text>
<rect x="230" y="241" width="258.2" height="12" fill="#94a3b8"/>
<text x="494.2" y="251" font-size="11" fill="#64748b">4.57</text>
<rect x="230" y="256" width="150.9" height="12" fill="#2563eb"/>
<text x="386.9" y="266" font-size="11" fill="#0f172a">2.67</text>
<text x="218" y="296" font-size="12" fill="#0f172a" text-anchor="end">Neckarufer</text>
<rect x="230" y="279" width="332.8" height="12" fill="#94a3b8"/>
<text x="568.8" y="289" font-size="11" fill="#64748b">5.89</text>
<rect x="230" y="294" width="205.1" height="12" fill="#2563eb"/>
<text x="441.1" y="304" font-size="11" fill="#0f172a">3.63</text>
<text x="218" y="334" font-size="12" fill="#0f172a" text-anchor="end">Soleo</text>
<rect x="230" y="317" width="248.6" height="12" fill="#94a3b8"/>
<text x="484.6" y="327" font-size="11" fill="#64748b">4.40</text>
<rect x="230" y="332" width="180.2" height="12" fill="#2563eb"/>
<text x="416.2" y="342" font-size="11" fill="#0f172a">3.19</text>
<text x="218" y="372" font-size="12" fill="#0f172a" text-anchor="end">Stadtgalerie</text>
<rect x="230" y="355" width="587.0" height="12" fill="#94a3b8"/>
<text x="823.0" y="365" font-size="11" fill="#64748b">10.39</text>
<rect x="230" y="370" width="329.4" height="12" fill="#2563eb"/>
<text x="565.4" y="380" font-size="11" fill="#0f172a">5.83</text>
<text x="24" y="434" font-size="11" fill="#64748b">Source: artifacts/metrics.json · PulseCast 28-day time-based holdout</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 4.5 KiB

+24
View File
@@ -0,0 +1,24 @@
<svg xmlns="http://www.w3.org/2000/svg" width="760" height="520" viewBox="0 0 760 520" role="img" aria-label="Small busy-hour classifier: holdout confusion matrix">
<rect width="100%" height="100%" fill="#ffffff"/>
<g font-family="-apple-system, Segoe UI, Helvetica, Arial, sans-serif">
<text x="24" y="34" font-size="20" font-weight="600" fill="#0f172a">Small busy-hour classifier: holdout confusion matrix</text>
<text x="24" y="56" font-size="13" fill="#475569">n=5,710 · accuracy 0.708 · precision 0.360 · recall 0.574 · F1 0.443 · ROC-AUC 0.726</text>
<text x="295.0" y="114" font-size="13" fill="#475569" text-anchor="middle">Predicted not busy</text>
<text x="485.0" y="114" font-size="13" fill="#475569" text-anchor="middle">Predicted busy</text>
<text x="182" y="225.0" font-size="13" fill="#475569" text-anchor="end">Actually not busy</text>
<text x="182" y="415.0" font-size="13" fill="#475569" text-anchor="end">Actually busy</text>
<rect x="200" y="130" width="190" height="190" fill="#1d4ed8" stroke="#ffffff" stroke-width="2"/>
<text x="295.0" y="217.0" font-size="30" font-weight="600" fill="#ffffff" text-anchor="middle">3,377</text>
<text x="295.0" y="245.0" font-size="13" fill="#ffffff" text-anchor="middle">TN · 74.1% of row</text>
<rect x="390" y="130" width="190" height="190" fill="#8aa7ee" stroke="#ffffff" stroke-width="2"/>
<text x="485.0" y="217.0" font-size="30" font-weight="600" fill="#0f172a" text-anchor="middle">1,178</text>
<text x="485.0" y="245.0" font-size="13" fill="#0f172a" text-anchor="middle">FP · 25.9% of row</text>
<rect x="200" y="320" width="190" height="190" fill="#acc3f5" stroke="#ffffff" stroke-width="2"/>
<text x="295.0" y="407.0" font-size="30" font-weight="600" fill="#0f172a" text-anchor="middle">492</text>
<text x="295.0" y="435.0" font-size="13" fill="#0f172a" text-anchor="middle">FN · 42.6% of row</text>
<rect x="390" y="320" width="190" height="190" fill="#a3bcf3" stroke="#ffffff" stroke-width="2"/>
<text x="485.0" y="407.0" font-size="30" font-weight="600" fill="#0f172a" text-anchor="middle">663</text>
<text x="485.0" y="435.0" font-size="13" fill="#0f172a" text-anchor="middle">TP · 57.4% of row</text>
<text x="24" y="508" font-size="11" fill="#64748b">Source: artifacts/small_metrics.json · chronological 80/20 split</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 2.3 KiB

+187
View File
@@ -0,0 +1,187 @@
# Metrics — paper support
All values in this file are generated by `make train` and stored in
`artifacts/metrics.json` (forecast + ablation + per-sensor + small model) and
`artifacts/small_metrics.json` (small-model detail). No number here is typed by
hand: to refresh a table, rerun `make train && make figures` and copy the new
JSON, or quote the JSON directly.
## Run provenance
| Field | Value |
|---|---|
| Model version | `histgb-2026-11-14` |
| Trained at | 2026-11-14T05:27:21Z |
| Data version | 2026-11-07T23:51:00Z |
| Context (weather) version | 2026-11-14T03:47:09Z |
| Train rows | 18,602 hourly sensor rows |
| Holdout | 2026-10-10T23:00:00Z → 2026-11-07T23:00:00Z |
| Evaluation rows | 5,350 (8 sensors, hours with a seasonal-naive baseline) |
| Sensors | 8 (one merged `density-01`, seven `hn-*`) |
| Raw input | 192,464 rows, 96,232 density rows, 96,232 battery rows |
| Weather | Open-Meteo historical archive (training), forecast (scenarios) |
| Calendar | Baden-Württemberg holidays 2025–2026 |
## Protocol
- One global `HistGradientBoostingRegressor` over all eight sensors. Target is
the mean of 15-minute `densityData` readings in a UTC hour.
- Time-based 28-day holdout measured back from the last observed hour. No
shuffling, no random split.
- Seasonal naive baseline: the exact 168-row lag (same observed hour one week
earlier), evaluated on the same rows.
- The production bundle is refit on all observed rows **after** the holdout is
evaluated; the holdout is never in the production fit.
- WAPE = `sum(|error|) / sum(|actual|)`. R² is computed on the same rows.
- The production 7-day model uses only features at least 168 hours old, so a
168-hour forecast needs no recursion.
- Weather in the holdout is observed, not forecast. Live accuracy will be lower
than the table because the dashboard uses forecast weather.
## Headline
| Model | MAE | WAPE | R² |
|---|---:|---:|---:|
| Seasonal naive (exact 168-row lag) | 5.40 | 44.7% | 0.462 |
| **Weather-aware 7-day (production)** | **3.32** | **27.5%** | **0.777** |
- MAE improvement over seasonal naive: **38.5%**.
- WAPE improvement: 17.2 percentage points.
- The mission reference numbers for seasonal naive (5.40 / 44.7% / 0.462) are
reproduced exactly.
- The production row is slightly better than the reference 7-day model
(4.37 / 36.2% / 0.672) because `lag_week` uses the previous *observed* same
weekday/hour across sensor gaps instead of recursion.
## Ablation
| Model | Description | MAE | WAPE | R² | n |
|---|---|---|---:|---:|---:|---:|
| `seasonal_naive` | exact 168-row lag (same observed hour last week) | 5.40 | 44.7% | 0.462 | 5,350 |
| `day_no_weather` | calendar + lags 24/48/168 + 24h/168h rolling means | 3.96 | 32.8% | 0.706 | 5,350 |
| `day_holiday` | day-ahead + Baden-Württemberg holiday flag | 3.84 | 31.8% | 0.732 | 5,350 |
| `day_weather` | day-ahead + temperature + rain (observed weather) | 3.36 | 27.8% | 0.763 | 5,350 |
| `week_no_weather` | 7-day-safe calendar + previous same weekday/hour values | 4.08 | 33.8% | 0.697 | 5,350 |
| **`week_weather`** | **7-day-safe + temperature + rain (production)** | **3.32** | **27.5%** | **0.777** | **5,350** |
Derived effects:
- Weather at day-ahead: 3.96 → 3.36 MAE, **−15.2%**.
- Weather at 7-day-safe: 4.08 → 3.32 MAE, **−18.6%**.
- Holiday flag at day-ahead: 3.96 → 3.84 MAE, **−0.12 MAE**. Holidays are
context, not a headline accuracy claim.
- The mission reference day-ahead weather row (3.36 / 27.8%) is reproduced.
![Ablation MAE by feature set](figures/ablation_mae.svg)
*Figure 1. Holdout MAE for each feature set; teal is the production model.
Weather is the largest single gain; the holiday flag is a small addition.*
## Per sensor
Production model and seasonal naive on the same holdout rows.
| Sensor | Name | n | MAE | WAPE | R² | Baseline MAE | Baseline WAPE | Baseline R² | MAE improvement |
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|
| `density-01` | Density 01 | 669 | 2.26 | 30.8% | 0.722 | 3.58 | 48.8% | 0.238 | 36.8% |
| `hn-goetzentuem-01` | Götzenturm | 669 | 2.40 | 26.9% | 0.718 | 3.84 | 43.0% | 0.281 | 37.5% |
| `hn-hagenbuchersee-01` | Hagenbuchersee 1 | 669 | 2.47 | 38.6% | 0.602 | 4.01 | 62.8% | −0.212 | 38.5% |
| `hn-hagenbuchersee-02` | Hagenbuchersee 2 | 669 | 4.12 | 31.1% | 0.731 | 6.49 | 48.9% | 0.337 | 36.5% |
| `hn-marahaus-01` | Marahaus | 669 | 2.67 | 29.8% | 0.671 | 4.57 | 51.0% | −0.045 | 41.7% |
| `hn-neckarufer-01` | Neckarufer | 668 | 3.63 | 28.2% | 0.695 | 5.89 | 45.8% | 0.110 | 38.4% |
| `hn-soleo-01` | Soleo | 669 | 3.19 | 20.2% | 0.548 | 4.40 | 28.0% | 0.163 | 27.7% |
| `hn-stadtgalerie-01` | Stadtgalerie | 668 | 5.83 | 25.3% | 0.765 | 10.39 | 45.1% | 0.480 | 43.9% |
- Every sensor improves over its own seasonal naive baseline (27.7%–43.9%).
- Stadtgalerie has the largest absolute error (MAE 5.83) and the largest
volume; Soleo has the smallest relative gain and the lowest R² (0.548).
- Two baselines have negative R² (`hagenbuchersee-01`, `marahaus-01`), which is
expected for a naive lag on a short, variable holdout; the model is positive
on all eight.
- Sensor totals: 669 × 6 + 668 × 2 = 5,350, matching the headline.
![Per-sensor MAE](figures/per_sensor_mae.svg)
*Figure 2. Per-sensor holdout MAE, production model (blue) vs seasonal naive
(grey). Every sensor improves; the model is not carried by one large sensor.*
## Uncertainty band
The `lo`/`hi` band is the holdout residual p10/p90 added to the point forecast,
clipped at zero:
| p10 | p50 | p90 |
|---:|---:|---:|
| −4.83 | −0.59 | +4.88 |
This is an empirical 80% band, not a calibrated guarantee, and the median
residual being −0.59 shows a small optimistic bias on the holdout.
## Small busy-hour classifier
Full definition and limitations: [`docs/small-model.md`](small-model.md).
Summary from the same baked run (`logit-20261114-052723`):
| Accuracy | Precision | Recall | F1 | ROC-AUC | Train rows | Test rows | Base rate |
|---:|---:|---:|---:|---:|---:|---:|---:|
| 0.708 | 0.360 | 0.574 | 0.443 | 0.726 | 18,242 | 5,710 | 0.202 |
| | Predicted not busy | Predicted busy |
|---|---:|---:|
| **Actually not busy** | 3,377 (TN) | 1,178 (FP) |
| **Actually busy** | 492 (FN) | 663 (TP) |
![Small model confusion matrix](figures/small_model_confusion.svg)
*Figure 3. Holdout confusion matrix for the small busy-hour classifier. The
balanced weights trade precision for recall; accuracy 0.708 is below the 0.798
majority-class baseline, so report recall/F1 and ROC-AUC instead.*
Per-sensor small-model holdout:
| Sensor | n | Positive rate | Accuracy | Precision | Recall | F1 |
|---|---:|---:|---:|---:|---:|---:|
| `density-01` | 714 | 0.111 | 0.761 | 0.300 | 0.873 | 0.447 |
| `hn-goetzentuem-01` | 714 | 0.197 | 0.732 | 0.391 | 0.638 | 0.485 |
| `hn-hagenbuchersee-01` | 714 | 0.049 | 0.724 | 0.148 | 0.971 | 0.257 |
| `hn-hagenbuchersee-02` | 714 | 0.123 | 0.783 | 0.355 | 0.932 | 0.514 |
| `hn-marahaus-01` | 714 | 0.055 | 0.710 | 0.135 | 0.795 | 0.230 |
| `hn-neckarufer-01` | 713 | 0.090 | 0.764 | 0.274 | 0.984 | 0.429 |
| `hn-soleo-01` | 714 | 0.704 | 0.581 | 0.943 | 0.431 | 0.592 |
| `hn-stadtgalerie-01` | 713 | 0.289 | 0.604 | 0.335 | 0.374 | 0.353 |
## Claims the data supports
- A global, weather-aware HistGB forecasts hourly footfall for all eight
inner-city sensors with MAE 3.32 / WAPE 27.5% / R² 0.777 on a 28-day
time-based holdout.
- Weather features are the main measurable gain: −18.6% MAE versus the 7-day
no-weather model, −38.5% versus seasonal naive.
- Every sensor improves over its own seasonal naive baseline (27.7%–43.9%).
- A tiny logistic model can flag busy hours with ROC-AUC 0.726 and 57.4%
recall at 36.0% precision, retrained from a CSV in under 5 seconds.
## Claims the data does not support
- People counts. Values are an index of estimated devices; penetration is
unknown.
- Annual seasonality, city-wide surfaces, or store-location recommendations
from five months at eight sensors in ~580 m.
- Causal weather effects. The ablation is predictive, not a treatment effect.
- Live accuracy equal to the holdout. The holdout uses observed weather; the
demo uses forecast weather and `predict` adds scenario overlays.
- Holidays as a large driver. The flag moves day-ahead MAE by only −0.12.
- The small classifier as the production forecast. It is a demo artifact with
six features and a per-window p75 target.
- Tuned hyperparameters. The model configuration is fixed; the 28-day holdout
is the honest signal, not a search result.
## Reproduce
```bash
make data # parse + weather + holidays + JSON artifacts
make train # holdout, ablation, per-sensor, small model, forecasts
make small # print the small-model metrics and confusion matrix
make figures # regenerate docs/figures/*.svg from artifacts/metrics.json
make check # sync --frozen, ruff, compileall, pytest
```
+187
View File
@@ -0,0 +1,187 @@
# Small model — busy-hour classifier
This is the small, fast, explainable model that powers the "train new data"
demo and the "I coded an AI" artifact. It is deliberately separate from the
production 7-day forecaster: six cheap features, a linear model, a time-based
split, and under a second of fitting. It is not the model behind the dashboard
forecast.
Every number below comes from the baked artifact
`artifacts/small_metrics.json` (version `logit-20261114-052723`, trained
2026-11-14T05:27:23Z) and can be reproduced with `make small`.
## Target
`busy` is a binary label for one sensor-hour:
```text
busy = 1 if hourly_mean_density > that sensor's 75th percentile
busy = 0 otherwise
```
- The p75 is computed **per sensor on the training window only**, then applied
unchanged to the holdout. Computing it on the full dataset would leak the
holdout distribution into the label.
- A p75 threshold makes "busy" mean "busier than three of four hours at this
location", not an absolute footfall claim. The same absolute value can be
busy at Soleo and quiet at Stadtgalerie.
- Hours above the threshold are exactly 25% of the training hours by
construction, before missing hours are dropped.
| Sensor | Training-window p75 |
|---|---:|
| `density-01` | 15.23 |
| `hn-goetzentuem-01` | 15.40 |
| `hn-hagenbuchersee-01` | 18.20 |
| `hn-hagenbuchersee-02` | 29.17 |
| `hn-marahaus-01` | 22.23 |
| `hn-neckarufer-01` | 28.35 |
| `hn-soleo-01` | 11.38 |
| `hn-stadtgalerie-01` | 38.41 |
## Data and split
- Input: the challenge CSV, `densityData` rows only, aggregated to the mean of
the 15-minute readings in each UTC hour.
- Window: 2026-06-12T00:00:00Z → 2026-11-07T23:00:00Z.
- 23,952 hourly sensor rows across all 8 usable sensors.
- Split is chronological, not random: the first 80% of **timestamps** train,
the last 20% test.
- Train: 2026-06-12T00:00:00Z → 2026-10-09T01:00:00Z, 18,242 rows.
- Test: 2026-10-09T02:00:00Z → 2026-11-07T23:00:00Z, 5,710 rows.
- Excluded devices: `maker-fair`, `niklas-01`, `hn-bollwerksturm-01` (test
installations / insufficient coverage). See `artifacts/data_audit.json`.
## Features
| Feature | Source | Note |
|---|---|---|
| `hour` | local Europe/Berlin hour | 0–23, captures the daily curve |
| `weekday` | local Europe/Berlin weekday | 0 = Monday |
| `is_weekend` | weekday ≥ 5 | |
| `is_holiday` | Baden-Württemberg calendar | context, not a headline claim |
| `temp_c` | observed Open-Meteo archive | joined to the nearest hour (±3 h) |
| `precip_mm` | observed Open-Meteo archive | same join |
No lags, no target history, no sensor identifier. That is intentional: the
classifier must retrain from a small user upload that may not contain a long
history, and it must not depend on future rows. Sensor identity is still
implicit in the per-sensor threshold.
## Model
```text
StandardScaler -> LogisticRegression(
C=1.0, max_iter=1000, class_weight="balanced",
solver="lbfgs", random_state=42)
```
`class_weight="balanced"` compensates for the 25/75 class split caused by the
p75 threshold. It buys recall at the cost of precision, which is the right
trade for a "flag the busy hours" demo.
Fit time: **0.30 s** on the full challenge CSV in this run (contract: < 5 s,
including the API job's CSV parsing).
## Metrics
Time-based holdout, 5,710 rows, base rate 20.2% positive.
| Accuracy | Precision | Recall | F1 | ROC-AUC |
|---:|---:|---:|---:|---:|
| 0.708 | 0.360 | 0.574 | 0.443 | 0.726 |
Confusion matrix:
| | Predicted not busy | Predicted busy |
|---|---:|---:|
| **Actually not busy** | 3,377 (TN) | 1,178 (FP) |
| **Actually busy** | 492 (FN) | 663 (TP) |
Derived:
- Specificity (true-negative rate): 0.741
- Negative predictive value: 0.873
- False-positive rate: 0.259
- False-negative rate: 0.426
- Precision lift over the 20.2% base rate: 1.78×
- Accuracy of the majority-class baseline ("always not busy"): 0.798
The model is **less accurate than always saying "not busy"**, but it finds 663
of 1,155 busy hours (57.4% recall) that the majority baseline never finds. The
balanced weights push it toward over-flagging: 1,178 false positives for 1,155
actual busy hours. That is why precision is 0.36 and accuracy is not the right
headline.
Per-sensor holdout:
| Sensor | n | Pos. rate | Accuracy | Precision | Recall | F1 |
|---|---:|---:|---:|---:|---:|---:|
| `density-01` | 714 | 0.111 | 0.761 | 0.300 | 0.873 | 0.447 |
| `hn-goetzentuem-01` | 714 | 0.197 | 0.732 | 0.391 | 0.638 | 0.485 |
| `hn-hagenbuchersee-01` | 714 | 0.049 | 0.724 | 0.148 | 0.971 | 0.257 |
| `hn-hagenbuchersee-02` | 714 | 0.123 | 0.783 | 0.355 | 0.932 | 0.514 |
| `hn-marahaus-01` | 714 | 0.055 | 0.710 | 0.135 | 0.795 | 0.230 |
| `hn-neckarufer-01` | 713 | 0.090 | 0.764 | 0.274 | 0.984 | 0.429 |
| `hn-soleo-01` | 714 | 0.704 | 0.581 | 0.943 | 0.431 | 0.592 |
| `hn-stadtgalerie-01` | 713 | 0.289 | 0.604 | 0.335 | 0.374 | 0.353 |
The pooled model has the lowest accuracy on `hn-soleo-01` (0.581, an unusually
high base rate of 70.4%) and the lowest recall on `hn-stadtgalerie-01` (0.374).
The high recall on low-base-rate sensors is mostly the balanced weights firing;
it is not evidence of a strong signal there.
## Reproduce
```bash
# Full challenge CSV, prints every number above.
make small
# Explicit CSV, all rows, save the baked artifact.
uv run python scripts/train_small.py --csv new-data.csv --all --save
# Machine-readable payload.
uv run python scripts/train_small.py --json
```
The runtime path is the same function (`pulsecast.small.train_small_from_csv`):
```bash
curl -s -X POST localhost:18000/train \
-H "Authorization: Bearer $CONTROL_TOKEN" \
-F model=small -F file=@new-data.csv
curl -s localhost:18000/train/<job_id> \
-H "Authorization: Bearer $CONTROL_TOKEN"
```
The job returns `accuracy`, `precision`, `recall`, `f1`, `roc_auc`, `n_train`,
`n_test`, `positive_rate`, and `sensors`; the full confusion matrix and
per-sensor table are stored in `artifacts/small_metrics.json`. On success the
bundle is hot-swapped and persisted to `PULSECAST_MODELS/small_latest.joblib`.
A restart falls back to the baked artifact.
## Honest limits
- **It is a demo, not the production forecast.** The dashboard uses the global
HistGB 7-day model, not this classifier.
- **Threshold convention.** "Busy" is the sensor's own p75, recomputed from the
upload's training window. A different upload can move the thresholds and
change every metric. Do not compare runs trained on different windows.
- **No lags or sensor id.** The model sees only calendar and weather. It cannot
know that the previous hour was packed, and it pools all locations while
evaluating per sensor.
- **Weather fallback.** If an uploaded CSV does not overlap the baked weather
archive, missing weather is filled with 14 °C / 0 mm. That keeps the job
running but weakens the weather features; the returned metrics show it.
- **One chronological split.** Five months, one holdout. No cross-validation,
no annual seasonality claim, no per-weekday calibration.
- **Threshold metrics only.** This page reports hard 0.5-threshold metrics, not
a calibrated probability. The probability output is not a business forecast.
- **Counters are devices.** Footfall density is an index of estimated devices,
not a count of people. A "busy" flag inherits that uncertainty.
- **Small uploads are unstable.** A CSV with one sensor, only one class, or
fewer than four hourly timestamps fails with a clear error. A CSV with a
short window may train but produce meaningless metrics.
- **No causal claims.** Weather and holiday coefficients are associations in
this sample, not causal effects, and the classifier is not used for scenario
uplift (that overlay is explicit and separate).
+118
View File
@@ -0,0 +1,118 @@
# Training end-to-end evidence
Reproduce with:
```bash
uv run python scripts/e2e_train.py --port 18000 --report /tmp/e2e.json
```
The script starts a real `uvicorn pulsecast.server:app` process with
`CONTROL_TOKEN`, waits for startup, and drives the HTTP API only. It exits
non-zero on the first failed check. Unit/contract tests (`make check`) are
separate.
## Run
| Field | Value |
|---|---|
| Date | 2026-11-14, 05:33 UTC |
| Branch | `feat/ml-final` |
| Port | **18000** |
| Checks | **38 passed, 0 failed** |
| Total wall time | 10.4 s |
| Forecast version before | `histgb-2026-11-14` |
| Data version | `2026-11-07T23:51:00Z` |
| Context version | `2026-11-14T03:47:09Z` |
## Snapshot and robustness (20 checks)
- `/healthz` → `{"ok":true}`; `/readyz` → 200 `model=true`.
- `/meta` covers 8 sensors and the baked model beats the seasonal-naive
baseline on the holdout.
- `/bootstrap` → 8 sensors, all 8 hourly series (23,952 points), 168 forecast
weather hours, 24 holidays, versions equal to `/meta`.
- `/predict`: **128/128** combinations (8 sensors × 4 horizons × 4 scenarios)
returned the contract point counts (`15m` 1, `1h` 4, `48h` 48, `7d` 168),
monotonic UTC timestamps, non-negative values, and `lo ≤ v ≤ hi`.
- `/simulate`: all 4 scenarios × 8 sensors × 4 horizons returned valid points.
- Weather override changes the forecast (`-8 °C / 6 mm` vs `30 °C / 0 mm`).
- Unknown sensor → 404; missing token → 401 `unauthorized`; wrong token → 401;
unknown job → 404 `not_found`; invalid model → 400 `bad_request`; missing
JSON path → 400.
## Small training job (real 60-day CSV subset)
Input: 36,846 density rows, 8 sensors, 2026-09-08T23:53Z → 2026-11-07T23:51Z.
| Field | Value |
|---|---|
| Job ID | `0b35347963974cbbbe584b77ecd4b846` |
| Status | `succeeded` |
| Model version | `logit-20261114-053344` |
| Wall time (submit → terminal) | **0.172 s** (contract < 5 s) |
| n_train / n_test | 8,151 / 2,301 |
| Base rate | 0.180 |
| Accuracy | 0.824 |
| Precision | 0.542 |
| Recall | 0.157 |
| F1 | 0.243 |
| ROC-AUC | 0.813 |
This is a different window from the baked full-window model in
`docs/small-model.md`, so the numbers are not directly comparable: the p75
target and the 80/20 split are recomputed inside the 60-day upload. The
low recall illustrates the honest limit noted in the small-model doc — a short,
recent window with a different threshold is a different classification problem.
Post-conditions verified:
- `small_latest.joblib` and `<job_id>-small.joblib` exist under
`PULSECAST_MODELS`.
- `/meta.small_model.model_version` equals the job version and its metrics
match the job exactly (in-memory hot-swap).
- `/predict` still returns 168 valid points after the swap.
## Forecast training job (full challenge CSV) and 409
Input: full `schwarz-pax.csv`, 19,383,568 bytes.
| Field | Value |
|---|---|
| Job ID | `235209a0d00f449ba8be717eb76386d6` |
| Status | `succeeded` |
| Model version | `histgb-20261114-053349` |
| Wall time | **4.380 s** |
| MAE | 3.32 |
| WAPE | 27.5% |
| R² | 0.777 |
| Baseline MAE / WAPE / R² | 5.40 / 44.7% / 0.462 |
| n_eval | 5,350 |
The ablation in the job payload matches `artifacts/metrics.json` row for row
(3.96, 3.84, 3.36, 4.08, 3.32). While this job was in `running`, a second
`POST /train` returned **409** `conflict`, proving one job at a time.
Post-conditions verified:
- `forecast_latest.joblib` and `<job_id>-forecast.joblib` exist under
`PULSECAST_MODELS`.
- `/meta.model_version` equals the retrained version and
`/meta.model_source` is `retrained`.
- `/predict` reports the retrained version; `/readyz` stays 200.
- `data_version` after the retrain is `2026-11-07T23:00:00Z` (the hourly input
maximum) rather than the raw-file floor `23:51:00Z`; both describe the same
observed window.
## Failure isolation
A malformed CSV (`not,a,pulsecast,file`) was accepted as a job (202), then
failed with a clear error:
```text
CSV needs a time column (time, timestamp, datetime, date, hour) and a value
column (value, density, footfall, count, pax, visitors); got ['not', 'a',
'pulsecast', 'file']
```
`/healthz` still returned 200 afterward: a bad upload cannot take down the
service or block `/predict`.