Files
ML-py/model_card.md
T
Noel Sabia dc83825de0 fix: use exact 168-row lag for the seasonal-naive baseline
The baseline now reproduces mission/data.md exactly: MAE 5.40, WAPE 44.7%,
R2 0.462. The model's lag_week remains the previous observed same weekday/hour,
which handles sensor gaps and is documented in the model card.
2026-11-14 05:04:41 +01:00

129 lines
5.9 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Model card — PulseCast footfall forecast
## Model
One global `HistGradientBoostingRegressor` predicts hourly footfall density for
all eight inner-city sensors. A separately trained logistic regression predicts
a binary `busy` hour for the training demo. Both are batch-trained; no model
tunes itself at inference time.
- Forecast version: `histgb-YYYY-MM-DD` (baked), `histgb-YYYYMMDD-HHMMSS` after
a runtime retrain.
- Small classifier version: `logit-YYYYMMDD-HHMMSS`.
- Framework: scikit-learn 1.9, pandas 3.1, Python 3.12.
## Intended use
Decision support for a store manager: next 7 days of footfall, an uncertainty
band, and a transparent staffing heuristic on top. The model is not a census of
people, not a city-wide heatmap, and not an automated labor optimizer.
## Features (forecast)
| Group | Features |
|---|---|
| Calendar | `hour`, `weekday`, `month`, `is_weekend`, `is_holiday`, `sensor_code` |
| Seasonal | `lag_week` (previous observed same weekday/hour), `seasonal_mean_4w` |
| Weather | `temp_c`, `precip_mm` |
The production 7-day model uses only features that are at least 168 hours old.
Day-ahead ablation models additionally use `lag_day`, `lag_2d`, `roll_24`,
`roll_168`.
## Target and data
- Target: mean of 15-minute `densityData` readings in a UTC hour, per sensor.
- Training window: 2026-06-12 → 2026-10-10; holdout: 2026-10-10 → 2026-11-07.
- Eight usable sensors after excluding `maker-fair` (73 rows, Hannover),
`niklas-01` (8 rows), `hn-bollwerksturm-01` (5 rows).
- `density-01` merges two physical devices (`863f75b0`, `f8c19730`) into one
logical location. The older device history is outside the 8-sensor window and
is not used for the production model.
- Weather: Open-Meteo historical archive for training, forecast for inference
and scenarios, Heilbronn grid cell.
- Holidays: explicit Baden-Württemberg 2025–2026 calendar.
## Evaluation
Time-based 28-day holdout, no shuffling. Seasonal naive is the baseline. WAPE is
`sum(|error|) / sum(|actual|)`. R² is computed on the same 5,350 rows.
| Model | MAE | WAPE | R² |
|---|---:|---:|---:|
| Seasonal naive (exact 168-row lag) | 5.40 | 44.7% | 0.462 |
| Day-ahead, no weather | 3.96 | 32.8% | 0.706 |
| Day-ahead, + holiday flag | 3.84 | 31.8% | 0.732 |
| Day-ahead, + temperature/rain | 3.36 | 27.8% | 0.763 |
| 7-day-safe, no weather | 4.08 | 33.8% | 0.697 |
| **7-day-safe, + temperature/rain (production)** | **3.32** | **27.5%** | **0.777** |
Weather is the useful signal: adding temperature and rain cuts MAE by ~19%
versus the 7-day no-weather model and ~38% versus seasonal naive. The holiday
flag alone changes day-ahead MAE by −0.12 in this run; holidays are context, not
a headline accuracy claim.
The seasonal-naive row reproduces the mission reference exactly
(5.40 / 44.7% / 0.462), and the day-ahead weather row matches 3.36 / 27.8%. The
production model is slightly better than the reference because `lag_week` uses
the previous *observed* same weekday/hour across sensor gaps, while the baseline
uses the exact 168-row lag. R² differs slightly from the reference 0.779
(0.777 here) because of that feature choice.
## Uncertainty band
The `lo`/`hi` band is the 10th/90th percentile of holdout residuals added to the
point forecast, clipped at zero. It is an empirical 80% band, not a calibrated
guarantee.
## Small classifier
- Target: `busy` = hourly footfall strictly above that sensor's p75, where the
p75 is computed on the training window only.
- Features: `hour`, `weekday`, `is_weekend`, `temp_c`, `precip_mm`, `is_holiday`.
- Model: standardized logistic regression, `class_weight="balanced"`.
- Split: last 20% of hourly timestamps, time-ordered.
- Result: accuracy 0.708, precision 0.360, recall 0.574, F1 0.443, ROC-AUC 0.726.
- Trains in ~0.25 s on the challenge CSV and under 1 s on a small upload.
- The low precision reflects a hard p75 threshold and a deliberately tiny model;
it is a demo artifact, not the production forecast.
## No-leakage rules
- All splits are time-based; no random shuffling.
- The seasonal-naive baseline uses an exact 168-row lag. The model's
`lag_week` uses the previous *observed* same weekday/hour; future lags are
never filled from future targets.
- Rolling means are shifted at least 24 hours back.
- The production 7-day model uses only ≥168-hour-old information, so a 168-hour
forecast needs no recursion.
- Weather in the holdout is observed, not forecast. The demo uses forecast
weather, so live accuracy will be lower than the holdout number.
- The holdout is used for reporting the headline; the production bundle is
refit on all observed rows only after the holdout is evaluated.
## Limitations
- PAX counters detect discoverable devices; penetration is unknown. Values are
an index of estimated devices, not people.
- Eight sensors cover ~580 m; the data does not support a city-wide surface or a
store-location recommender.
- Five months of multi-sensor history; no annual seasonality claim. `density-01`
has a longer single-sensor history, but that cannot represent the network.
- Sensor gaps of 4–6 days remain. The model tolerates missing lags but cannot
recover unobserved footfall.
- No events, markets, school holidays, bridge days, or promotions are modeled.
Scenario uplifts are explicit assumptions, not causal estimates.
- Weather is observed in training and forecast in the demo; forecast error is
not included in the reported metrics.
- The staffing heuristic assumes visitors-per-staff-hour, shift length, safety
factor, and labor cost. There is no POS, queue, sales, or labor-cost data.
## Ethical and operational notes
- No PII, MAC addresses, or individual tracking is stored or exposed.
- Public API reads are open; training is token-protected and internal.
- Model artifacts and derived aggregates are committed; the raw challenge CSV is
not. No secrets are baked into the image.
- On restart the service falls back to the baked model; retrained models persist
only in `/tmp` and are not assumed to survive a pod restart.