docs: small-model artifact, paper metrics, train e2e evidence
- docs/small-model.md: target, features, split, metrics, full confusion matrix, per-sensor table, honest limits - docs/metrics.md: provenance, protocol, headline, ablation, per-sensor, uncertainty, small model, supported/unsupported claims - docs/figures: generated ablation, per-sensor, confusion-matrix SVGs - docs/train-e2e.md: exact 38-check live run, small + forecast job results - model_card.md and README updated with final metrics and links
This commit is contained in:
1 parent
2bcf66c550
commit
c7ea524b5b
8 files changed
+702
-16
No files matched your search
@@ -0,0 +1,36 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="960" height="370" viewBox="0 0 960 370" role="img" aria-label="Ablation: holdout MAE by feature set">
|
||||
<rect width="100%" height="100%" fill="#ffffff"/>
|
||||
<g font-family="-apple-system, Segoe UI, Helvetica, Arial, sans-serif">
|
||||
<text x="24" y="34" font-size="20" font-weight="600" fill="#0f172a">Ablation: holdout MAE by feature set</text>
|
||||
<text x="24" y="56" font-size="13" fill="#475569">Holdout 2026-10-10T23:00:00Z .. 2026-11-07T23:00:00Z, n=5350</text>
|
||||
<line x1="230.0" y1="68" x2="230.0" y2="314" stroke="#e2e8f0" stroke-width="1"/>
|
||||
<text x="230.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">0.0</text>
|
||||
<line x1="388.5" y1="68" x2="388.5" y2="314" stroke="#e2e8f0" stroke-width="1"/>
|
||||
<text x="388.5" y="62" font-size="11" fill="#64748b" text-anchor="middle">1.5</text>
|
||||
<line x1="547.0" y1="68" x2="547.0" y2="314" stroke="#e2e8f0" stroke-width="1"/>
|
||||
<text x="547.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">2.9</text>
|
||||
<line x1="705.5" y1="68" x2="705.5" y2="314" stroke="#e2e8f0" stroke-width="1"/>
|
||||
<text x="705.5" y="62" font-size="11" fill="#64748b" text-anchor="middle">4.4</text>
|
||||
<line x1="864.0" y1="68" x2="864.0" y2="314" stroke="#e2e8f0" stroke-width="1"/>
|
||||
<text x="864.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">5.8</text>
|
||||
<text x="218" y="106" font-size="12" fill="#0f172a" text-anchor="end">seasonal naive</text>
|
||||
<rect x="230" y="92" width="587.0" height="22" fill="#2563eb"/>
|
||||
<text x="823.0" y="107" font-size="12" fill="#0f172a">5.40</text>
|
||||
<text x="218" y="144" font-size="12" fill="#0f172a" text-anchor="end">day no weather</text>
|
||||
<rect x="230" y="130" width="430.5" height="22" fill="#2563eb"/>
|
||||
<text x="666.5" y="145" font-size="12" fill="#0f172a">3.96</text>
|
||||
<text x="218" y="182" font-size="12" fill="#0f172a" text-anchor="end">day holiday</text>
|
||||
<rect x="230" y="168" width="417.4" height="22" fill="#2563eb"/>
|
||||
<text x="653.4" y="183" font-size="12" fill="#0f172a">3.84</text>
|
||||
<text x="218" y="220" font-size="12" fill="#0f172a" text-anchor="end">day weather</text>
|
||||
<rect x="230" y="206" width="365.3" height="22" fill="#2563eb"/>
|
||||
<text x="601.3" y="221" font-size="12" fill="#0f172a">3.36</text>
|
||||
<text x="218" y="258" font-size="12" fill="#0f172a" text-anchor="end">week no weather</text>
|
||||
<rect x="230" y="244" width="443.5" height="22" fill="#2563eb"/>
|
||||
<text x="679.5" y="259" font-size="12" fill="#0f172a">4.08</text>
|
||||
<text x="218" y="296" font-size="12" fill="#0f172a" text-anchor="end">week weather</text>
|
||||
<rect x="230" y="282" width="360.9" height="22" fill="#0f766e"/>
|
||||
<text x="596.9" y="297" font-size="12" fill="#0f172a">3.32</text>
|
||||
<text x="24" y="358" font-size="11" fill="#64748b">Source: artifacts/metrics.json · PulseCast 28-day time-based holdout</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.7 KiB |
@@ -0,0 +1,60 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="960" height="446" viewBox="0 0 960 446" role="img" aria-label="Per-sensor holdout MAE: production model vs seasonal naive">
|
||||
<rect width="100%" height="100%" fill="#ffffff"/>
|
||||
<g font-family="-apple-system, Segoe UI, Helvetica, Arial, sans-serif">
|
||||
<text x="24" y="34" font-size="20" font-weight="600" fill="#0f172a">Per-sensor holdout MAE: production model vs seasonal naive</text>
|
||||
<text x="24" y="56" font-size="13" fill="#475569">Holdout 2026-10-10T23:00:00Z .. 2026-11-07T23:00:00Z, n=5350</text>
|
||||
<line x1="230.0" y1="68" x2="230.0" y2="390" stroke="#e2e8f0" stroke-width="1"/>
|
||||
<text x="230.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">0.0</text>
|
||||
<line x1="388.5" y1="68" x2="388.5" y2="390" stroke="#e2e8f0" stroke-width="1"/>
|
||||
<text x="388.5" y="62" font-size="11" fill="#64748b" text-anchor="middle">2.8</text>
|
||||
<line x1="547.0" y1="68" x2="547.0" y2="390" stroke="#e2e8f0" stroke-width="1"/>
|
||||
<text x="547.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">5.6</text>
|
||||
<line x1="705.5" y1="68" x2="705.5" y2="390" stroke="#e2e8f0" stroke-width="1"/>
|
||||
<text x="705.5" y="62" font-size="11" fill="#64748b" text-anchor="middle">8.4</text>
|
||||
<line x1="864.0" y1="68" x2="864.0" y2="390" stroke="#e2e8f0" stroke-width="1"/>
|
||||
<text x="864.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">11.2</text>
|
||||
<rect x="230" y="52" width="10" height="10" fill="#94a3b8"/><text x="246" y="61" font-size="12" fill="#475569">seasonal naive</text>
|
||||
<rect x="370" y="52" width="10" height="10" fill="#2563eb"/><text x="386" y="61" font-size="12" fill="#475569">weather-7d</text>
|
||||
<text x="218" y="106" font-size="12" fill="#0f172a" text-anchor="end">Density 01</text>
|
||||
<rect x="230" y="89" width="202.3" height="12" fill="#94a3b8"/>
|
||||
<text x="438.3" y="99" font-size="11" fill="#64748b">3.58</text>
|
||||
<rect x="230" y="104" width="127.7" height="12" fill="#2563eb"/>
|
||||
<text x="363.7" y="114" font-size="11" fill="#0f172a">2.26</text>
|
||||
<text x="218" y="144" font-size="12" fill="#0f172a" text-anchor="end">Götzenturm</text>
|
||||
<rect x="230" y="127" width="217.0" height="12" fill="#94a3b8"/>
|
||||
<text x="453.0" y="137" font-size="11" fill="#64748b">3.84</text>
|
||||
<rect x="230" y="142" width="135.6" height="12" fill="#2563eb"/>
|
||||
<text x="371.6" y="152" font-size="11" fill="#0f172a">2.40</text>
|
||||
<text x="218" y="182" font-size="12" fill="#0f172a" text-anchor="end">Hagenbuchersee 1</text>
|
||||
<rect x="230" y="165" width="226.6" height="12" fill="#94a3b8"/>
|
||||
<text x="462.6" y="175" font-size="11" fill="#64748b">4.01</text>
|
||||
<rect x="230" y="180" width="139.6" height="12" fill="#2563eb"/>
|
||||
<text x="375.6" y="190" font-size="11" fill="#0f172a">2.47</text>
|
||||
<text x="218" y="220" font-size="12" fill="#0f172a" text-anchor="end">Hagenbuchersee 2</text>
|
||||
<rect x="230" y="203" width="366.7" height="12" fill="#94a3b8"/>
|
||||
<text x="602.7" y="213" font-size="11" fill="#64748b">6.49</text>
|
||||
<rect x="230" y="218" width="232.8" height="12" fill="#2563eb"/>
|
||||
<text x="468.8" y="228" font-size="11" fill="#0f172a">4.12</text>
|
||||
<text x="218" y="258" font-size="12" fill="#0f172a" text-anchor="end">Marahaus</text>
|
||||
<rect x="230" y="241" width="258.2" height="12" fill="#94a3b8"/>
|
||||
<text x="494.2" y="251" font-size="11" fill="#64748b">4.57</text>
|
||||
<rect x="230" y="256" width="150.9" height="12" fill="#2563eb"/>
|
||||
<text x="386.9" y="266" font-size="11" fill="#0f172a">2.67</text>
|
||||
<text x="218" y="296" font-size="12" fill="#0f172a" text-anchor="end">Neckarufer</text>
|
||||
<rect x="230" y="279" width="332.8" height="12" fill="#94a3b8"/>
|
||||
<text x="568.8" y="289" font-size="11" fill="#64748b">5.89</text>
|
||||
<rect x="230" y="294" width="205.1" height="12" fill="#2563eb"/>
|
||||
<text x="441.1" y="304" font-size="11" fill="#0f172a">3.63</text>
|
||||
<text x="218" y="334" font-size="12" fill="#0f172a" text-anchor="end">Soleo</text>
|
||||
<rect x="230" y="317" width="248.6" height="12" fill="#94a3b8"/>
|
||||
<text x="484.6" y="327" font-size="11" fill="#64748b">4.40</text>
|
||||
<rect x="230" y="332" width="180.2" height="12" fill="#2563eb"/>
|
||||
<text x="416.2" y="342" font-size="11" fill="#0f172a">3.19</text>
|
||||
<text x="218" y="372" font-size="12" fill="#0f172a" text-anchor="end">Stadtgalerie</text>
|
||||
<rect x="230" y="355" width="587.0" height="12" fill="#94a3b8"/>
|
||||
<text x="823.0" y="365" font-size="11" fill="#64748b">10.39</text>
|
||||
<rect x="230" y="370" width="329.4" height="12" fill="#2563eb"/>
|
||||
<text x="565.4" y="380" font-size="11" fill="#0f172a">5.83</text>
|
||||
<text x="24" y="434" font-size="11" fill="#64748b">Source: artifacts/metrics.json · PulseCast 28-day time-based holdout</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 4.5 KiB |
@@ -0,0 +1,24 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="760" height="520" viewBox="0 0 760 520" role="img" aria-label="Small busy-hour classifier: holdout confusion matrix">
|
||||
<rect width="100%" height="100%" fill="#ffffff"/>
|
||||
<g font-family="-apple-system, Segoe UI, Helvetica, Arial, sans-serif">
|
||||
<text x="24" y="34" font-size="20" font-weight="600" fill="#0f172a">Small busy-hour classifier: holdout confusion matrix</text>
|
||||
<text x="24" y="56" font-size="13" fill="#475569">n=5,710 · accuracy 0.708 · precision 0.360 · recall 0.574 · F1 0.443 · ROC-AUC 0.726</text>
|
||||
<text x="295.0" y="114" font-size="13" fill="#475569" text-anchor="middle">Predicted not busy</text>
|
||||
<text x="485.0" y="114" font-size="13" fill="#475569" text-anchor="middle">Predicted busy</text>
|
||||
<text x="182" y="225.0" font-size="13" fill="#475569" text-anchor="end">Actually not busy</text>
|
||||
<text x="182" y="415.0" font-size="13" fill="#475569" text-anchor="end">Actually busy</text>
|
||||
<rect x="200" y="130" width="190" height="190" fill="#1d4ed8" stroke="#ffffff" stroke-width="2"/>
|
||||
<text x="295.0" y="217.0" font-size="30" font-weight="600" fill="#ffffff" text-anchor="middle">3,377</text>
|
||||
<text x="295.0" y="245.0" font-size="13" fill="#ffffff" text-anchor="middle">TN · 74.1% of row</text>
|
||||
<rect x="390" y="130" width="190" height="190" fill="#8aa7ee" stroke="#ffffff" stroke-width="2"/>
|
||||
<text x="485.0" y="217.0" font-size="30" font-weight="600" fill="#0f172a" text-anchor="middle">1,178</text>
|
||||
<text x="485.0" y="245.0" font-size="13" fill="#0f172a" text-anchor="middle">FP · 25.9% of row</text>
|
||||
<rect x="200" y="320" width="190" height="190" fill="#acc3f5" stroke="#ffffff" stroke-width="2"/>
|
||||
<text x="295.0" y="407.0" font-size="30" font-weight="600" fill="#0f172a" text-anchor="middle">492</text>
|
||||
<text x="295.0" y="435.0" font-size="13" fill="#0f172a" text-anchor="middle">FN · 42.6% of row</text>
|
||||
<rect x="390" y="320" width="190" height="190" fill="#a3bcf3" stroke="#ffffff" stroke-width="2"/>
|
||||
<text x="485.0" y="407.0" font-size="30" font-weight="600" fill="#0f172a" text-anchor="middle">663</text>
|
||||
<text x="485.0" y="435.0" font-size="13" fill="#0f172a" text-anchor="middle">TP · 57.4% of row</text>
|
||||
<text x="24" y="508" font-size="11" fill="#64748b">Source: artifacts/small_metrics.json · chronological 80/20 split</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.3 KiB |
+187
@@ -0,0 +1,187 @@
|
||||
# Metrics — paper support
|
||||
|
||||
All values in this file are generated by `make train` and stored in
|
||||
`artifacts/metrics.json` (forecast + ablation + per-sensor + small model) and
|
||||
`artifacts/small_metrics.json` (small-model detail). No number here is typed by
|
||||
hand: to refresh a table, rerun `make train && make figures` and copy the new
|
||||
JSON, or quote the JSON directly.
|
||||
|
||||
## Run provenance
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Model version | `histgb-2026-11-14` |
|
||||
| Trained at | 2026-11-14T05:27:21Z |
|
||||
| Data version | 2026-11-07T23:51:00Z |
|
||||
| Context (weather) version | 2026-11-14T03:47:09Z |
|
||||
| Train rows | 18,602 hourly sensor rows |
|
||||
| Holdout | 2026-10-10T23:00:00Z → 2026-11-07T23:00:00Z |
|
||||
| Evaluation rows | 5,350 (8 sensors, hours with a seasonal-naive baseline) |
|
||||
| Sensors | 8 (one merged `density-01`, seven `hn-*`) |
|
||||
| Raw input | 192,464 rows, 96,232 density rows, 96,232 battery rows |
|
||||
| Weather | Open-Meteo historical archive (training), forecast (scenarios) |
|
||||
| Calendar | Baden-Württemberg holidays 2025–2026 |
|
||||
|
||||
## Protocol
|
||||
|
||||
- One global `HistGradientBoostingRegressor` over all eight sensors. Target is
|
||||
the mean of 15-minute `densityData` readings in a UTC hour.
|
||||
- Time-based 28-day holdout measured back from the last observed hour. No
|
||||
shuffling, no random split.
|
||||
- Seasonal naive baseline: the exact 168-row lag (same observed hour one week
|
||||
earlier), evaluated on the same rows.
|
||||
- The production bundle is refit on all observed rows **after** the holdout is
|
||||
evaluated; the holdout is never in the production fit.
|
||||
- WAPE = `sum(|error|) / sum(|actual|)`. R² is computed on the same rows.
|
||||
- The production 7-day model uses only features at least 168 hours old, so a
|
||||
168-hour forecast needs no recursion.
|
||||
- Weather in the holdout is observed, not forecast. Live accuracy will be lower
|
||||
than the table because the dashboard uses forecast weather.
|
||||
|
||||
## Headline
|
||||
|
||||
| Model | MAE | WAPE | R² |
|
||||
|---|---:|---:|---:|
|
||||
| Seasonal naive (exact 168-row lag) | 5.40 | 44.7% | 0.462 |
|
||||
| **Weather-aware 7-day (production)** | **3.32** | **27.5%** | **0.777** |
|
||||
|
||||
- MAE improvement over seasonal naive: **38.5%**.
|
||||
- WAPE improvement: 17.2 percentage points.
|
||||
- The mission reference numbers for seasonal naive (5.40 / 44.7% / 0.462) are
|
||||
reproduced exactly.
|
||||
- The production row is slightly better than the reference 7-day model
|
||||
(4.37 / 36.2% / 0.672) because `lag_week` uses the previous *observed* same
|
||||
weekday/hour across sensor gaps instead of recursion.
|
||||
|
||||
## Ablation
|
||||
|
||||
| Model | Description | MAE | WAPE | R² | n |
|
||||
|---|---|---|---:|---:|---:|---:|
|
||||
| `seasonal_naive` | exact 168-row lag (same observed hour last week) | 5.40 | 44.7% | 0.462 | 5,350 |
|
||||
| `day_no_weather` | calendar + lags 24/48/168 + 24h/168h rolling means | 3.96 | 32.8% | 0.706 | 5,350 |
|
||||
| `day_holiday` | day-ahead + Baden-Württemberg holiday flag | 3.84 | 31.8% | 0.732 | 5,350 |
|
||||
| `day_weather` | day-ahead + temperature + rain (observed weather) | 3.36 | 27.8% | 0.763 | 5,350 |
|
||||
| `week_no_weather` | 7-day-safe calendar + previous same weekday/hour values | 4.08 | 33.8% | 0.697 | 5,350 |
|
||||
| **`week_weather`** | **7-day-safe + temperature + rain (production)** | **3.32** | **27.5%** | **0.777** | **5,350** |
|
||||
|
||||
Derived effects:
|
||||
|
||||
- Weather at day-ahead: 3.96 → 3.36 MAE, **−15.2%**.
|
||||
- Weather at 7-day-safe: 4.08 → 3.32 MAE, **−18.6%**.
|
||||
- Holiday flag at day-ahead: 3.96 → 3.84 MAE, **−0.12 MAE**. Holidays are
|
||||
context, not a headline accuracy claim.
|
||||
- The mission reference day-ahead weather row (3.36 / 27.8%) is reproduced.
|
||||
|
||||

|
||||
|
||||
*Figure 1. Holdout MAE for each feature set; teal is the production model.
|
||||
Weather is the largest single gain; the holiday flag is a small addition.*
|
||||
|
||||
## Per sensor
|
||||
|
||||
Production model and seasonal naive on the same holdout rows.
|
||||
|
||||
| Sensor | Name | n | MAE | WAPE | R² | Baseline MAE | Baseline WAPE | Baseline R² | MAE improvement |
|
||||
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| `density-01` | Density 01 | 669 | 2.26 | 30.8% | 0.722 | 3.58 | 48.8% | 0.238 | 36.8% |
|
||||
| `hn-goetzentuem-01` | Götzenturm | 669 | 2.40 | 26.9% | 0.718 | 3.84 | 43.0% | 0.281 | 37.5% |
|
||||
| `hn-hagenbuchersee-01` | Hagenbuchersee 1 | 669 | 2.47 | 38.6% | 0.602 | 4.01 | 62.8% | −0.212 | 38.5% |
|
||||
| `hn-hagenbuchersee-02` | Hagenbuchersee 2 | 669 | 4.12 | 31.1% | 0.731 | 6.49 | 48.9% | 0.337 | 36.5% |
|
||||
| `hn-marahaus-01` | Marahaus | 669 | 2.67 | 29.8% | 0.671 | 4.57 | 51.0% | −0.045 | 41.7% |
|
||||
| `hn-neckarufer-01` | Neckarufer | 668 | 3.63 | 28.2% | 0.695 | 5.89 | 45.8% | 0.110 | 38.4% |
|
||||
| `hn-soleo-01` | Soleo | 669 | 3.19 | 20.2% | 0.548 | 4.40 | 28.0% | 0.163 | 27.7% |
|
||||
| `hn-stadtgalerie-01` | Stadtgalerie | 668 | 5.83 | 25.3% | 0.765 | 10.39 | 45.1% | 0.480 | 43.9% |
|
||||
|
||||
- Every sensor improves over its own seasonal naive baseline (27.7%–43.9%).
|
||||
- Stadtgalerie has the largest absolute error (MAE 5.83) and the largest
|
||||
volume; Soleo has the smallest relative gain and the lowest R² (0.548).
|
||||
- Two baselines have negative R² (`hagenbuchersee-01`, `marahaus-01`), which is
|
||||
expected for a naive lag on a short, variable holdout; the model is positive
|
||||
on all eight.
|
||||
- Sensor totals: 669 × 6 + 668 × 2 = 5,350, matching the headline.
|
||||
|
||||

|
||||
|
||||
*Figure 2. Per-sensor holdout MAE, production model (blue) vs seasonal naive
|
||||
(grey). Every sensor improves; the model is not carried by one large sensor.*
|
||||
|
||||
## Uncertainty band
|
||||
|
||||
The `lo`/`hi` band is the holdout residual p10/p90 added to the point forecast,
|
||||
clipped at zero:
|
||||
|
||||
| p10 | p50 | p90 |
|
||||
|---:|---:|---:|
|
||||
| −4.83 | −0.59 | +4.88 |
|
||||
|
||||
This is an empirical 80% band, not a calibrated guarantee, and the median
|
||||
residual being −0.59 shows a small optimistic bias on the holdout.
|
||||
|
||||
## Small busy-hour classifier
|
||||
|
||||
Full definition and limitations: [`docs/small-model.md`](small-model.md).
|
||||
Summary from the same baked run (`logit-20261114-052723`):
|
||||
|
||||
| Accuracy | Precision | Recall | F1 | ROC-AUC | Train rows | Test rows | Base rate |
|
||||
|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| 0.708 | 0.360 | 0.574 | 0.443 | 0.726 | 18,242 | 5,710 | 0.202 |
|
||||
|
||||
| | Predicted not busy | Predicted busy |
|
||||
|---|---:|---:|
|
||||
| **Actually not busy** | 3,377 (TN) | 1,178 (FP) |
|
||||
| **Actually busy** | 492 (FN) | 663 (TP) |
|
||||
|
||||

|
||||
|
||||
*Figure 3. Holdout confusion matrix for the small busy-hour classifier. The
|
||||
balanced weights trade precision for recall; accuracy 0.708 is below the 0.798
|
||||
majority-class baseline, so report recall/F1 and ROC-AUC instead.*
|
||||
|
||||
Per-sensor small-model holdout:
|
||||
|
||||
| Sensor | n | Positive rate | Accuracy | Precision | Recall | F1 |
|
||||
|---|---:|---:|---:|---:|---:|---:|
|
||||
| `density-01` | 714 | 0.111 | 0.761 | 0.300 | 0.873 | 0.447 |
|
||||
| `hn-goetzentuem-01` | 714 | 0.197 | 0.732 | 0.391 | 0.638 | 0.485 |
|
||||
| `hn-hagenbuchersee-01` | 714 | 0.049 | 0.724 | 0.148 | 0.971 | 0.257 |
|
||||
| `hn-hagenbuchersee-02` | 714 | 0.123 | 0.783 | 0.355 | 0.932 | 0.514 |
|
||||
| `hn-marahaus-01` | 714 | 0.055 | 0.710 | 0.135 | 0.795 | 0.230 |
|
||||
| `hn-neckarufer-01` | 713 | 0.090 | 0.764 | 0.274 | 0.984 | 0.429 |
|
||||
| `hn-soleo-01` | 714 | 0.704 | 0.581 | 0.943 | 0.431 | 0.592 |
|
||||
| `hn-stadtgalerie-01` | 713 | 0.289 | 0.604 | 0.335 | 0.374 | 0.353 |
|
||||
|
||||
## Claims the data supports
|
||||
|
||||
- A global, weather-aware HistGB forecasts hourly footfall for all eight
|
||||
inner-city sensors with MAE 3.32 / WAPE 27.5% / R² 0.777 on a 28-day
|
||||
time-based holdout.
|
||||
- Weather features are the main measurable gain: −18.6% MAE versus the 7-day
|
||||
no-weather model, −38.5% versus seasonal naive.
|
||||
- Every sensor improves over its own seasonal naive baseline (27.7%–43.9%).
|
||||
- A tiny logistic model can flag busy hours with ROC-AUC 0.726 and 57.4%
|
||||
recall at 36.0% precision, retrained from a CSV in under 5 seconds.
|
||||
|
||||
## Claims the data does not support
|
||||
|
||||
- People counts. Values are an index of estimated devices; penetration is
|
||||
unknown.
|
||||
- Annual seasonality, city-wide surfaces, or store-location recommendations
|
||||
from five months at eight sensors in ~580 m.
|
||||
- Causal weather effects. The ablation is predictive, not a treatment effect.
|
||||
- Live accuracy equal to the holdout. The holdout uses observed weather; the
|
||||
demo uses forecast weather and `predict` adds scenario overlays.
|
||||
- Holidays as a large driver. The flag moves day-ahead MAE by only −0.12.
|
||||
- The small classifier as the production forecast. It is a demo artifact with
|
||||
six features and a per-window p75 target.
|
||||
- Tuned hyperparameters. The model configuration is fixed; the 28-day holdout
|
||||
is the honest signal, not a search result.
|
||||
|
||||
## Reproduce
|
||||
|
||||
```bash
|
||||
make data # parse + weather + holidays + JSON artifacts
|
||||
make train # holdout, ablation, per-sensor, small model, forecasts
|
||||
make small # print the small-model metrics and confusion matrix
|
||||
make figures # regenerate docs/figures/*.svg from artifacts/metrics.json
|
||||
make check # sync --frozen, ruff, compileall, pytest
|
||||
```
|
||||
@@ -0,0 +1,187 @@
|
||||
# Small model — busy-hour classifier
|
||||
|
||||
This is the small, fast, explainable model that powers the "train new data"
|
||||
demo and the "I coded an AI" artifact. It is deliberately separate from the
|
||||
production 7-day forecaster: six cheap features, a linear model, a time-based
|
||||
split, and under a second of fitting. It is not the model behind the dashboard
|
||||
forecast.
|
||||
|
||||
Every number below comes from the baked artifact
|
||||
`artifacts/small_metrics.json` (version `logit-20261114-052723`, trained
|
||||
2026-11-14T05:27:23Z) and can be reproduced with `make small`.
|
||||
|
||||
## Target
|
||||
|
||||
`busy` is a binary label for one sensor-hour:
|
||||
|
||||
```text
|
||||
busy = 1 if hourly_mean_density > that sensor's 75th percentile
|
||||
busy = 0 otherwise
|
||||
```
|
||||
|
||||
- The p75 is computed **per sensor on the training window only**, then applied
|
||||
unchanged to the holdout. Computing it on the full dataset would leak the
|
||||
holdout distribution into the label.
|
||||
- A p75 threshold makes "busy" mean "busier than three of four hours at this
|
||||
location", not an absolute footfall claim. The same absolute value can be
|
||||
busy at Soleo and quiet at Stadtgalerie.
|
||||
- Hours above the threshold are exactly 25% of the training hours by
|
||||
construction, before missing hours are dropped.
|
||||
|
||||
| Sensor | Training-window p75 |
|
||||
|---|---:|
|
||||
| `density-01` | 15.23 |
|
||||
| `hn-goetzentuem-01` | 15.40 |
|
||||
| `hn-hagenbuchersee-01` | 18.20 |
|
||||
| `hn-hagenbuchersee-02` | 29.17 |
|
||||
| `hn-marahaus-01` | 22.23 |
|
||||
| `hn-neckarufer-01` | 28.35 |
|
||||
| `hn-soleo-01` | 11.38 |
|
||||
| `hn-stadtgalerie-01` | 38.41 |
|
||||
|
||||
## Data and split
|
||||
|
||||
- Input: the challenge CSV, `densityData` rows only, aggregated to the mean of
|
||||
the 15-minute readings in each UTC hour.
|
||||
- Window: 2026-06-12T00:00:00Z → 2026-11-07T23:00:00Z.
|
||||
- 23,952 hourly sensor rows across all 8 usable sensors.
|
||||
- Split is chronological, not random: the first 80% of **timestamps** train,
|
||||
the last 20% test.
|
||||
- Train: 2026-06-12T00:00:00Z → 2026-10-09T01:00:00Z, 18,242 rows.
|
||||
- Test: 2026-10-09T02:00:00Z → 2026-11-07T23:00:00Z, 5,710 rows.
|
||||
- Excluded devices: `maker-fair`, `niklas-01`, `hn-bollwerksturm-01` (test
|
||||
installations / insufficient coverage). See `artifacts/data_audit.json`.
|
||||
|
||||
## Features
|
||||
|
||||
| Feature | Source | Note |
|
||||
|---|---|---|
|
||||
| `hour` | local Europe/Berlin hour | 0–23, captures the daily curve |
|
||||
| `weekday` | local Europe/Berlin weekday | 0 = Monday |
|
||||
| `is_weekend` | weekday ≥ 5 | |
|
||||
| `is_holiday` | Baden-Württemberg calendar | context, not a headline claim |
|
||||
| `temp_c` | observed Open-Meteo archive | joined to the nearest hour (±3 h) |
|
||||
| `precip_mm` | observed Open-Meteo archive | same join |
|
||||
|
||||
No lags, no target history, no sensor identifier. That is intentional: the
|
||||
classifier must retrain from a small user upload that may not contain a long
|
||||
history, and it must not depend on future rows. Sensor identity is still
|
||||
implicit in the per-sensor threshold.
|
||||
|
||||
## Model
|
||||
|
||||
```text
|
||||
StandardScaler -> LogisticRegression(
|
||||
C=1.0, max_iter=1000, class_weight="balanced",
|
||||
solver="lbfgs", random_state=42)
|
||||
```
|
||||
|
||||
`class_weight="balanced"` compensates for the 25/75 class split caused by the
|
||||
p75 threshold. It buys recall at the cost of precision, which is the right
|
||||
trade for a "flag the busy hours" demo.
|
||||
|
||||
Fit time: **0.30 s** on the full challenge CSV in this run (contract: < 5 s,
|
||||
including the API job's CSV parsing).
|
||||
|
||||
## Metrics
|
||||
|
||||
Time-based holdout, 5,710 rows, base rate 20.2% positive.
|
||||
|
||||
| Accuracy | Precision | Recall | F1 | ROC-AUC |
|
||||
|---:|---:|---:|---:|---:|
|
||||
| 0.708 | 0.360 | 0.574 | 0.443 | 0.726 |
|
||||
|
||||
Confusion matrix:
|
||||
|
||||
| | Predicted not busy | Predicted busy |
|
||||
|---|---:|---:|
|
||||
| **Actually not busy** | 3,377 (TN) | 1,178 (FP) |
|
||||
| **Actually busy** | 492 (FN) | 663 (TP) |
|
||||
|
||||
Derived:
|
||||
|
||||
- Specificity (true-negative rate): 0.741
|
||||
- Negative predictive value: 0.873
|
||||
- False-positive rate: 0.259
|
||||
- False-negative rate: 0.426
|
||||
- Precision lift over the 20.2% base rate: 1.78×
|
||||
- Accuracy of the majority-class baseline ("always not busy"): 0.798
|
||||
|
||||
The model is **less accurate than always saying "not busy"**, but it finds 663
|
||||
of 1,155 busy hours (57.4% recall) that the majority baseline never finds. The
|
||||
balanced weights push it toward over-flagging: 1,178 false positives for 1,155
|
||||
actual busy hours. That is why precision is 0.36 and accuracy is not the right
|
||||
headline.
|
||||
|
||||
Per-sensor holdout:
|
||||
|
||||
| Sensor | n | Pos. rate | Accuracy | Precision | Recall | F1 |
|
||||
|---|---:|---:|---:|---:|---:|---:|
|
||||
| `density-01` | 714 | 0.111 | 0.761 | 0.300 | 0.873 | 0.447 |
|
||||
| `hn-goetzentuem-01` | 714 | 0.197 | 0.732 | 0.391 | 0.638 | 0.485 |
|
||||
| `hn-hagenbuchersee-01` | 714 | 0.049 | 0.724 | 0.148 | 0.971 | 0.257 |
|
||||
| `hn-hagenbuchersee-02` | 714 | 0.123 | 0.783 | 0.355 | 0.932 | 0.514 |
|
||||
| `hn-marahaus-01` | 714 | 0.055 | 0.710 | 0.135 | 0.795 | 0.230 |
|
||||
| `hn-neckarufer-01` | 713 | 0.090 | 0.764 | 0.274 | 0.984 | 0.429 |
|
||||
| `hn-soleo-01` | 714 | 0.704 | 0.581 | 0.943 | 0.431 | 0.592 |
|
||||
| `hn-stadtgalerie-01` | 713 | 0.289 | 0.604 | 0.335 | 0.374 | 0.353 |
|
||||
|
||||
The pooled model has the lowest accuracy on `hn-soleo-01` (0.581, an unusually
|
||||
high base rate of 70.4%) and the lowest recall on `hn-stadtgalerie-01` (0.374).
|
||||
The high recall on low-base-rate sensors is mostly the balanced weights firing;
|
||||
it is not evidence of a strong signal there.
|
||||
|
||||
## Reproduce
|
||||
|
||||
```bash
|
||||
# Full challenge CSV, prints every number above.
|
||||
make small
|
||||
|
||||
# Explicit CSV, all rows, save the baked artifact.
|
||||
uv run python scripts/train_small.py --csv new-data.csv --all --save
|
||||
|
||||
# Machine-readable payload.
|
||||
uv run python scripts/train_small.py --json
|
||||
```
|
||||
|
||||
The runtime path is the same function (`pulsecast.small.train_small_from_csv`):
|
||||
|
||||
```bash
|
||||
curl -s -X POST localhost:18000/train \
|
||||
-H "Authorization: Bearer $CONTROL_TOKEN" \
|
||||
-F model=small -F file=@new-data.csv
|
||||
curl -s localhost:18000/train/<job_id> \
|
||||
-H "Authorization: Bearer $CONTROL_TOKEN"
|
||||
```
|
||||
|
||||
The job returns `accuracy`, `precision`, `recall`, `f1`, `roc_auc`, `n_train`,
|
||||
`n_test`, `positive_rate`, and `sensors`; the full confusion matrix and
|
||||
per-sensor table are stored in `artifacts/small_metrics.json`. On success the
|
||||
bundle is hot-swapped and persisted to `PULSECAST_MODELS/small_latest.joblib`.
|
||||
A restart falls back to the baked artifact.
|
||||
|
||||
## Honest limits
|
||||
|
||||
- **It is a demo, not the production forecast.** The dashboard uses the global
|
||||
HistGB 7-day model, not this classifier.
|
||||
- **Threshold convention.** "Busy" is the sensor's own p75, recomputed from the
|
||||
upload's training window. A different upload can move the thresholds and
|
||||
change every metric. Do not compare runs trained on different windows.
|
||||
- **No lags or sensor id.** The model sees only calendar and weather. It cannot
|
||||
know that the previous hour was packed, and it pools all locations while
|
||||
evaluating per sensor.
|
||||
- **Weather fallback.** If an uploaded CSV does not overlap the baked weather
|
||||
archive, missing weather is filled with 14 °C / 0 mm. That keeps the job
|
||||
running but weakens the weather features; the returned metrics show it.
|
||||
- **One chronological split.** Five months, one holdout. No cross-validation,
|
||||
no annual seasonality claim, no per-weekday calibration.
|
||||
- **Threshold metrics only.** This page reports hard 0.5-threshold metrics, not
|
||||
a calibrated probability. The probability output is not a business forecast.
|
||||
- **Counters are devices.** Footfall density is an index of estimated devices,
|
||||
not a count of people. A "busy" flag inherits that uncertainty.
|
||||
- **Small uploads are unstable.** A CSV with one sensor, only one class, or
|
||||
fewer than four hourly timestamps fails with a clear error. A CSV with a
|
||||
short window may train but produce meaningless metrics.
|
||||
- **No causal claims.** Weather and holiday coefficients are associations in
|
||||
this sample, not causal effects, and the classifier is not used for scenario
|
||||
uplift (that overlay is explicit and separate).
|
||||
@@ -0,0 +1,118 @@
|
||||
# Training end-to-end evidence
|
||||
|
||||
Reproduce with:
|
||||
|
||||
```bash
|
||||
uv run python scripts/e2e_train.py --port 18000 --report /tmp/e2e.json
|
||||
```
|
||||
|
||||
The script starts a real `uvicorn pulsecast.server:app` process with
|
||||
`CONTROL_TOKEN`, waits for startup, and drives the HTTP API only. It exits
|
||||
non-zero on the first failed check. Unit/contract tests (`make check`) are
|
||||
separate.
|
||||
|
||||
## Run
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Date | 2026-11-14, 05:33 UTC |
|
||||
| Branch | `feat/ml-final` |
|
||||
| Port | **18000** |
|
||||
| Checks | **38 passed, 0 failed** |
|
||||
| Total wall time | 10.4 s |
|
||||
| Forecast version before | `histgb-2026-11-14` |
|
||||
| Data version | `2026-11-07T23:51:00Z` |
|
||||
| Context version | `2026-11-14T03:47:09Z` |
|
||||
|
||||
## Snapshot and robustness (20 checks)
|
||||
|
||||
- `/healthz` → `{"ok":true}`; `/readyz` → 200 `model=true`.
|
||||
- `/meta` covers 8 sensors and the baked model beats the seasonal-naive
|
||||
baseline on the holdout.
|
||||
- `/bootstrap` → 8 sensors, all 8 hourly series (23,952 points), 168 forecast
|
||||
weather hours, 24 holidays, versions equal to `/meta`.
|
||||
- `/predict`: **128/128** combinations (8 sensors × 4 horizons × 4 scenarios)
|
||||
returned the contract point counts (`15m` 1, `1h` 4, `48h` 48, `7d` 168),
|
||||
monotonic UTC timestamps, non-negative values, and `lo ≤ v ≤ hi`.
|
||||
- `/simulate`: all 4 scenarios × 8 sensors × 4 horizons returned valid points.
|
||||
- Weather override changes the forecast (`-8 °C / 6 mm` vs `30 °C / 0 mm`).
|
||||
- Unknown sensor → 404; missing token → 401 `unauthorized`; wrong token → 401;
|
||||
unknown job → 404 `not_found`; invalid model → 400 `bad_request`; missing
|
||||
JSON path → 400.
|
||||
|
||||
## Small training job (real 60-day CSV subset)
|
||||
|
||||
Input: 36,846 density rows, 8 sensors, 2026-09-08T23:53Z → 2026-11-07T23:51Z.
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Job ID | `0b35347963974cbbbe584b77ecd4b846` |
|
||||
| Status | `succeeded` |
|
||||
| Model version | `logit-20261114-053344` |
|
||||
| Wall time (submit → terminal) | **0.172 s** (contract < 5 s) |
|
||||
| n_train / n_test | 8,151 / 2,301 |
|
||||
| Base rate | 0.180 |
|
||||
| Accuracy | 0.824 |
|
||||
| Precision | 0.542 |
|
||||
| Recall | 0.157 |
|
||||
| F1 | 0.243 |
|
||||
| ROC-AUC | 0.813 |
|
||||
|
||||
This is a different window from the baked full-window model in
|
||||
`docs/small-model.md`, so the numbers are not directly comparable: the p75
|
||||
target and the 80/20 split are recomputed inside the 60-day upload. The
|
||||
low recall illustrates the honest limit noted in the small-model doc — a short,
|
||||
recent window with a different threshold is a different classification problem.
|
||||
|
||||
Post-conditions verified:
|
||||
|
||||
- `small_latest.joblib` and `<job_id>-small.joblib` exist under
|
||||
`PULSECAST_MODELS`.
|
||||
- `/meta.small_model.model_version` equals the job version and its metrics
|
||||
match the job exactly (in-memory hot-swap).
|
||||
- `/predict` still returns 168 valid points after the swap.
|
||||
|
||||
## Forecast training job (full challenge CSV) and 409
|
||||
|
||||
Input: full `schwarz-pax.csv`, 19,383,568 bytes.
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Job ID | `235209a0d00f449ba8be717eb76386d6` |
|
||||
| Status | `succeeded` |
|
||||
| Model version | `histgb-20261114-053349` |
|
||||
| Wall time | **4.380 s** |
|
||||
| MAE | 3.32 |
|
||||
| WAPE | 27.5% |
|
||||
| R² | 0.777 |
|
||||
| Baseline MAE / WAPE / R² | 5.40 / 44.7% / 0.462 |
|
||||
| n_eval | 5,350 |
|
||||
|
||||
The ablation in the job payload matches `artifacts/metrics.json` row for row
|
||||
(3.96, 3.84, 3.36, 4.08, 3.32). While this job was in `running`, a second
|
||||
`POST /train` returned **409** `conflict`, proving one job at a time.
|
||||
|
||||
Post-conditions verified:
|
||||
|
||||
- `forecast_latest.joblib` and `<job_id>-forecast.joblib` exist under
|
||||
`PULSECAST_MODELS`.
|
||||
- `/meta.model_version` equals the retrained version and
|
||||
`/meta.model_source` is `retrained`.
|
||||
- `/predict` reports the retrained version; `/readyz` stays 200.
|
||||
- `data_version` after the retrain is `2026-11-07T23:00:00Z` (the hourly input
|
||||
maximum) rather than the raw-file floor `23:51:00Z`; both describe the same
|
||||
observed window.
|
||||
|
||||
## Failure isolation
|
||||
|
||||
A malformed CSV (`not,a,pulsecast,file`) was accepted as a job (202), then
|
||||
failed with a clear error:
|
||||
|
||||
```text
|
||||
CSV needs a time column (time, timestamp, datetime, date, hour) and a value
|
||||
column (value, density, footfall, count, pax, visitors); got ['not', 'a',
|
||||
'pulsecast', 'file']
|
||||
```
|
||||
|
||||
`/healthz` still returned 200 afterward: a bad upload cannot take down the
|
||||
service or block `/predict`.
|
||||
Reference in new issue
Block a user