The baseline now reproduces mission/data.md exactly: MAE 5.40, WAPE 44.7%, R2 0.462. The model's lag_week remains the previous observed same weekday/hour, which handles sensor gaps and is documented in the model card.
129 lines
5.9 KiB
Markdown
129 lines
5.9 KiB
Markdown
# Model card — PulseCast footfall forecast
|
||
|
||
## Model
|
||
|
||
One global `HistGradientBoostingRegressor` predicts hourly footfall density for
|
||
all eight inner-city sensors. A separately trained logistic regression predicts
|
||
a binary `busy` hour for the training demo. Both are batch-trained; no model
|
||
tunes itself at inference time.
|
||
|
||
- Forecast version: `histgb-YYYY-MM-DD` (baked), `histgb-YYYYMMDD-HHMMSS` after
|
||
a runtime retrain.
|
||
- Small classifier version: `logit-YYYYMMDD-HHMMSS`.
|
||
- Framework: scikit-learn 1.9, pandas 3.1, Python 3.12.
|
||
|
||
## Intended use
|
||
|
||
Decision support for a store manager: next 7 days of footfall, an uncertainty
|
||
band, and a transparent staffing heuristic on top. The model is not a census of
|
||
people, not a city-wide heatmap, and not an automated labor optimizer.
|
||
|
||
## Features (forecast)
|
||
|
||
| Group | Features |
|
||
|---|---|
|
||
| Calendar | `hour`, `weekday`, `month`, `is_weekend`, `is_holiday`, `sensor_code` |
|
||
| Seasonal | `lag_week` (previous observed same weekday/hour), `seasonal_mean_4w` |
|
||
| Weather | `temp_c`, `precip_mm` |
|
||
|
||
The production 7-day model uses only features that are at least 168 hours old.
|
||
Day-ahead ablation models additionally use `lag_day`, `lag_2d`, `roll_24`,
|
||
`roll_168`.
|
||
|
||
## Target and data
|
||
|
||
- Target: mean of 15-minute `densityData` readings in a UTC hour, per sensor.
|
||
- Training window: 2026-06-12 → 2026-10-10; holdout: 2026-10-10 → 2026-11-07.
|
||
- Eight usable sensors after excluding `maker-fair` (73 rows, Hannover),
|
||
`niklas-01` (8 rows), `hn-bollwerksturm-01` (5 rows).
|
||
- `density-01` merges two physical devices (`863f75b0`, `f8c19730`) into one
|
||
logical location. The older device history is outside the 8-sensor window and
|
||
is not used for the production model.
|
||
- Weather: Open-Meteo historical archive for training, forecast for inference
|
||
and scenarios, Heilbronn grid cell.
|
||
- Holidays: explicit Baden-Württemberg 2025–2026 calendar.
|
||
|
||
## Evaluation
|
||
|
||
Time-based 28-day holdout, no shuffling. Seasonal naive is the baseline. WAPE is
|
||
`sum(|error|) / sum(|actual|)`. R² is computed on the same 5,350 rows.
|
||
|
||
| Model | MAE | WAPE | R² |
|
||
|---|---:|---:|---:|
|
||
| Seasonal naive (exact 168-row lag) | 5.40 | 44.7% | 0.462 |
|
||
| Day-ahead, no weather | 3.96 | 32.8% | 0.706 |
|
||
| Day-ahead, + holiday flag | 3.84 | 31.8% | 0.732 |
|
||
| Day-ahead, + temperature/rain | 3.36 | 27.8% | 0.763 |
|
||
| 7-day-safe, no weather | 4.08 | 33.8% | 0.697 |
|
||
| **7-day-safe, + temperature/rain (production)** | **3.32** | **27.5%** | **0.777** |
|
||
|
||
Weather is the useful signal: adding temperature and rain cuts MAE by ~19%
|
||
versus the 7-day no-weather model and ~38% versus seasonal naive. The holiday
|
||
flag alone changes day-ahead MAE by −0.12 in this run; holidays are context, not
|
||
a headline accuracy claim.
|
||
|
||
The seasonal-naive row reproduces the mission reference exactly
|
||
(5.40 / 44.7% / 0.462), and the day-ahead weather row matches 3.36 / 27.8%. The
|
||
production model is slightly better than the reference because `lag_week` uses
|
||
the previous *observed* same weekday/hour across sensor gaps, while the baseline
|
||
uses the exact 168-row lag. R² differs slightly from the reference 0.779
|
||
(0.777 here) because of that feature choice.
|
||
|
||
## Uncertainty band
|
||
|
||
The `lo`/`hi` band is the 10th/90th percentile of holdout residuals added to the
|
||
point forecast, clipped at zero. It is an empirical 80% band, not a calibrated
|
||
guarantee.
|
||
|
||
## Small classifier
|
||
|
||
- Target: `busy` = hourly footfall strictly above that sensor's p75, where the
|
||
p75 is computed on the training window only.
|
||
- Features: `hour`, `weekday`, `is_weekend`, `temp_c`, `precip_mm`, `is_holiday`.
|
||
- Model: standardized logistic regression, `class_weight="balanced"`.
|
||
- Split: last 20% of hourly timestamps, time-ordered.
|
||
- Result: accuracy 0.708, precision 0.360, recall 0.574, F1 0.443, ROC-AUC 0.726.
|
||
- Trains in ~0.25 s on the challenge CSV and under 1 s on a small upload.
|
||
- The low precision reflects a hard p75 threshold and a deliberately tiny model;
|
||
it is a demo artifact, not the production forecast.
|
||
|
||
## No-leakage rules
|
||
|
||
- All splits are time-based; no random shuffling.
|
||
- The seasonal-naive baseline uses an exact 168-row lag. The model's
|
||
`lag_week` uses the previous *observed* same weekday/hour; future lags are
|
||
never filled from future targets.
|
||
- Rolling means are shifted at least 24 hours back.
|
||
- The production 7-day model uses only ≥168-hour-old information, so a 168-hour
|
||
forecast needs no recursion.
|
||
- Weather in the holdout is observed, not forecast. The demo uses forecast
|
||
weather, so live accuracy will be lower than the holdout number.
|
||
- The holdout is used for reporting the headline; the production bundle is
|
||
refit on all observed rows only after the holdout is evaluated.
|
||
|
||
## Limitations
|
||
|
||
- PAX counters detect discoverable devices; penetration is unknown. Values are
|
||
an index of estimated devices, not people.
|
||
- Eight sensors cover ~580 m; the data does not support a city-wide surface or a
|
||
store-location recommender.
|
||
- Five months of multi-sensor history; no annual seasonality claim. `density-01`
|
||
has a longer single-sensor history, but that cannot represent the network.
|
||
- Sensor gaps of 4–6 days remain. The model tolerates missing lags but cannot
|
||
recover unobserved footfall.
|
||
- No events, markets, school holidays, bridge days, or promotions are modeled.
|
||
Scenario uplifts are explicit assumptions, not causal estimates.
|
||
- Weather is observed in training and forecast in the demo; forecast error is
|
||
not included in the reported metrics.
|
||
- The staffing heuristic assumes visitors-per-staff-hour, shift length, safety
|
||
factor, and labor cost. There is no POS, queue, sales, or labor-cost data.
|
||
|
||
## Ethical and operational notes
|
||
|
||
- No PII, MAC addresses, or individual tracking is stored or exposed.
|
||
- Public API reads are open; training is token-protected and internal.
|
||
- Model artifacts and derived aggregates are committed; the raw challenge CSV is
|
||
not. No secrets are baked into the image.
|
||
- On restart the service falls back to the baked model; retrained models persist
|
||
only in `/tmp` and are not assumed to survive a pod restart.
|