Files
ML-py/model_card.md
T
Noel Sabia 070ecd2a78
ci / check (push) Successful in 1m12s
ci: fix Gitea runner setup and tighten docs
Replace setup-python with setup-uv: the self-hosted Gitea runner lacks the
toolcache and lsb_release dependency, so the job failed before any project
step. uv installs its own Python 3.12; the checks are unchanged.

Align the README and docs with the committed artifacts: quick start serves
from baked artifacts, rebuild steps are separate, and the reference-model
comparison no longer overstates the result. Drop stale external-doc
references and comments that restated the code.
2026-11-14 09:10:47 +01:00

174 lines
8.5 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Model card — PulseCast footfall forecast
## Model
One global `HistGradientBoostingRegressor` predicts hourly footfall density for
all eight inner-city sensors. A separately trained logistic regression predicts
a binary `busy` hour for the training demo. Both are batch-trained; no model
tunes itself at inference time.
- Forecast version: `histgb-YYYY-MM-DD` (baked), `histgb-YYYYMMDD-HHMMSS` after
a runtime retrain.
- Small classifier version: `logit-YYYYMMDD-HHMMSS`.
- Framework: scikit-learn 1.9, pandas 3.1, Python 3.12.
## Intended use
Decision support for a store manager: next 7 days of footfall, an uncertainty
band, and a transparent staffing heuristic on top. The model is not a census of
people, not a city-wide heatmap, and not an automated labor optimizer.
## Features (forecast)
| Group | Features |
|---|---|
| Calendar | `hour`, `weekday`, `month`, `is_weekend`, `is_holiday`, `sensor_code` |
| Seasonal | `lag_week` (previous observed same weekday/hour), `seasonal_mean_4w` |
| Weather | `temp_c`, `precip_mm` |
The production 7-day model uses only features that are at least 168 hours old.
Day-ahead ablation models additionally use `lag_day`, `lag_2d`, `roll_24`,
`roll_168`.
## Target and data
- Target: mean of 15-minute `densityData` readings in a UTC hour, per sensor.
- Training rows: 2026-06-12T00:00:00Z → 2026-10-10T22:00:00Z, 18,602 rows.
- Holdout: 2026-10-10T23:00:00Z → 2026-11-07T23:00:00Z, 5,350
evaluation rows (hours with a seasonal-naive baseline).
- The production bundle is refit on all observed rows (through
2026-11-07T23:00:00Z) only after the holdout is evaluated.
- Eight usable sensors after excluding `maker-fair` (73 rows, Hannover),
`niklas-01` (8 rows), `hn-bollwerksturm-01` (5 rows).
- `density-01` merges two physical devices (`863f75b0`, `f8c19730`) into one
logical location. The older device history is outside the 8-sensor window and
is not used for the production model.
- Weather: Open-Meteo historical archive for training, forecast for inference
and scenarios, Heilbronn grid cell.
- Holidays: explicit Baden-Württemberg 2025–2026 calendar.
## Evaluation
Time-based 28-day holdout, no shuffling. Seasonal naive is the baseline. WAPE is
`sum(|error|) / sum(|actual|)`. R² is computed on the same 5,350 rows.
| Model | MAE | WAPE | R² |
|---|---:|---:|---:|
| Seasonal naive (exact 168-row lag) | 5.40 | 44.7% | 0.462 |
| Day-ahead, no weather | 3.96 | 32.8% | 0.706 |
| Day-ahead, + holiday flag | 3.84 | 31.8% | 0.732 |
| Day-ahead, + temperature/rain | 3.36 | 27.8% | 0.763 |
| 7-day-safe, no weather | 4.08 | 33.8% | 0.697 |
| **7-day-safe, + temperature/rain (production)** | **3.32** | **27.5%** | **0.777** |
Weather is the useful signal: adding temperature and rain cuts MAE by 18.6%
versus the 7-day no-weather model and 38.5% versus seasonal naive. The holiday
flag alone changes day-ahead MAE by −0.12 in this run; holidays are context, not
a headline accuracy claim.
The seasonal-naive row reproduces the mission reference exactly
(5.40 / 44.7% / 0.462), and the day-ahead weather row matches 3.36 / 27.8%. The
production 7-day model edges that day-ahead reference on MAE and WAPE
(3.32 / 27.5%) but is slightly lower on R² (0.777 vs 0.779). The two use
different features; `lag_week` uses the previous *observed* same weekday/hour
across sensor gaps, while the seasonal-naive baseline uses the exact 168-row
lag.
Per-sensor holdout MAE improves over each sensor's own seasonal naive by
27.7%–43.9%; the full table, figures, and provenance are in
[`docs/metrics.md`](docs/metrics.md). The largest absolute error is
Stadtgalerie (MAE 5.83, its baseline 10.39); Soleo has the smallest relative
gain (27.7%) and lowest R² (0.548). Every sensor has a positive R² under the
model; two seasonal-naive baselines are negative.
## Uncertainty band
The `lo`/`hi` band is the 10th/90th percentile of holdout residuals added to the
point forecast, clipped at zero: p10 −4.83, p50 −0.59, p90 +4.88. It is an
empirical 80% band, not a calibrated guarantee; the negative median residual
means the point forecast is slightly optimistic on the holdout.
## Small classifier ("I coded an AI" artifact)
This model is separate from the production forecaster and powers the
`model=small` retraining demo. Full definition, thresholds, confusion matrix
interpretation, and limits: [`docs/small-model.md`](docs/small-model.md).
- Target: `busy` = hourly footfall strictly above that sensor's p75, where the
p75 is computed on the training window only and then frozen for the holdout.
- Features: `hour`, `weekday`, `is_weekend`, `temp_c`, `precip_mm`, `is_holiday`
(no lags, no sensor identifier, no target history).
- Model: standardized logistic regression, `class_weight="balanced"`, `C=1.0`,
`max_iter=1000`, `random_state=42`.
- Split: last 20% of hourly timestamps, time-ordered. Train
2026-06-12T00:00Z → 2026-10-09T01:00Z (18,242 rows); test
2026-10-09T02:00Z → 2026-11-07T23:00Z (5,710 rows).
- Result (`logit-20261114-052723`): accuracy 0.708, precision 0.360, recall
0.574, F1 0.443, ROC-AUC 0.726, base rate 0.202.
- Confusion matrix: TN 3,377, FP 1,178, FN 492, TP 663. Specificity 0.741,
negative predictive value 0.873.
- Fit time: 0.30 s on the full challenge CSV; the API job (parse + fit) must
stay under the 5 s contract. The live e2e run trained a 60-day upload subset
in 0.17 s wall time.
- Per-sensor F1 ranges from 0.230 (Marahaus, 5.5% positive) to 0.592 (Soleo,
70.4% positive). The balanced weights over-flag low-base-rate sensors; accuracy
is lowest on Soleo (0.581) and recall is lowest on Stadtgalerie (0.374).
- Accuracy is below the 0.798 majority-class baseline. Do not use accuracy as
the headline: the model finds 663 of 1,155 busy hours, which the majority
baseline never finds. The low precision reflects a hard p75 threshold, a
deliberately tiny model, and balanced class weights.
- This is a demo artifact, not the production forecast. Thresholds move with
the upload window, and a CSV without weather overlap falls back to
14 °C / 0 mm.
## Artifacts and reproduction
- `artifacts/metrics.json` — versions, headline, ablation, per-sensor, small
model summary.
- `artifacts/small_metrics.json` — full small-model payload including the
confusion matrix, per-sensor table, and thresholds.
- `make train` regenerates models and metrics; `make small` prints the
small-model numbers; `make figures` regenerates `docs/figures/*.svg`.
- `docs/metrics.md` is the paper support sheet; it quotes the JSON and does not
invent numbers.
## No-leakage rules
- All splits are time-based; no random shuffling.
- The seasonal-naive baseline uses an exact 168-row lag. The model's
`lag_week` uses the previous *observed* same weekday/hour; future lags are
never filled from future targets.
- Rolling means are shifted at least 24 hours back.
- The production 7-day model uses only ≥168-hour-old information, so a 168-hour
forecast needs no recursion.
- Weather in the holdout is observed, not forecast. The demo uses forecast
weather, so live accuracy will be lower than the holdout number.
- The holdout is used for reporting the headline; the production bundle is
refit on all observed rows only after the holdout is evaluated.
## Limitations
- PAX counters detect discoverable devices; penetration is unknown. Values are
an index of estimated devices, not people.
- Eight sensors cover ~580 m; the data does not support a city-wide surface or a
store-location recommender.
- Five months of multi-sensor history; no annual seasonality claim. `density-01`
has a longer single-sensor history, but that cannot represent the network.
- Sensor gaps of 4–6 days remain. The model tolerates missing lags but cannot
recover unobserved footfall.
- No events, markets, school holidays, bridge days, or promotions are modeled.
Scenario uplifts are explicit assumptions, not causal estimates.
- Weather is observed in training and forecast in the demo; forecast error is
not included in the reported metrics.
- The staffing heuristic assumes visitors-per-staff-hour, shift length, safety
factor, and labor cost. There is no POS, queue, sales, or labor-cost data.
## Ethical and operational notes
- No PII, MAC addresses, or individual tracking is stored or exposed.
- Public API reads are open; training is token-protected and internal.
- Model artifacts and derived aggregates are committed; the raw challenge CSV is
not. No secrets are baked into the image.
- On restart the service falls back to the baked model; retrained models persist
only in `/tmp` and are not assumed to survive a pod restart.