Files
ML-py/model_card.md
T
Noel Sabia 939a8db482 feat: complete ML pipeline, FastAPI service, small model, tests, image
- finish the global HistGB bundle, scenario forecasts and 28-day ablation
- export sensors/series/forecasts/metrics/context plus baked weather frames
- add FastAPI health/meta/predict/simulate and background /train jobs
- add the p75 busy classifier, retraining from CSV, and hot-swap persistence
- add pytest coverage, multi-stage non-root Dockerfile, README and model card

Holdout: weather 7d MAE 3.32 / WAPE 27.5% / R2 0.777 vs seasonal naive
MAE 4.23 / WAPE 35.0% / R2 0.656 (5,350 eval rows).
2026-11-14 05:01:16 +01:00

125 lines
5.7 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Model card — PulseCast footfall forecast
## Model
One global `HistGradientBoostingRegressor` predicts hourly footfall density for
all eight inner-city sensors. A separately trained logistic regression predicts
a binary `busy` hour for the training demo. Both are batch-trained; no model
tunes itself at inference time.
- Forecast version: `histgb-YYYY-MM-DD` (baked), `histgb-YYYYMMDD-HHMMSS` after
a runtime retrain.
- Small classifier version: `logit-YYYYMMDD-HHMMSS`.
- Framework: scikit-learn 1.9, pandas 3.1, Python 3.12.
## Intended use
Decision support for a store manager: next 7 days of footfall, an uncertainty
band, and a transparent staffing heuristic on top. The model is not a census of
people, not a city-wide heatmap, and not an automated labor optimizer.
## Features (forecast)
| Group | Features |
|---|---|
| Calendar | `hour`, `weekday`, `month`, `is_weekend`, `is_holiday`, `sensor_code` |
| Seasonal | `lag_week` (previous observed same weekday/hour), `seasonal_mean_4w` |
| Weather | `temp_c`, `precip_mm` |
The production 7-day model uses only features that are at least 168 hours old.
Day-ahead ablation models additionally use `lag_day`, `lag_2d`, `roll_24`,
`roll_168`.
## Target and data
- Target: mean of 15-minute `densityData` readings in a UTC hour, per sensor.
- Training window: 2026-06-12 → 2026-10-10; holdout: 2026-10-10 → 2026-11-07.
- Eight usable sensors after excluding `maker-fair` (73 rows, Hannover),
`niklas-01` (8 rows), `hn-bollwerksturm-01` (5 rows).
- `density-01` merges two physical devices (`863f75b0`, `f8c19730`) into one
logical location. The older device history is outside the 8-sensor window and
is not used for the production model.
- Weather: Open-Meteo historical archive for training, forecast for inference
and scenarios, Heilbronn grid cell.
- Holidays: explicit Baden-Württemberg 2025–2026 calendar.
## Evaluation
Time-based 28-day holdout, no shuffling. Seasonal naive is the baseline. WAPE is
`sum(|error|) / sum(|actual|)`. R² is computed on the same 5,350 rows.
| Model | MAE | WAPE | R² |
|---|---:|---:|---:|
| Seasonal naive (same local hour last week) | 4.23 | 35.0% | 0.656 |
| Day-ahead, no weather | 3.96 | 32.8% | 0.706 |
| Day-ahead, + holiday flag | 3.84 | 31.8% | 0.732 |
| Day-ahead, + temperature/rain | 3.36 | 27.8% | 0.763 |
| 7-day-safe, no weather | 4.08 | 33.8% | 0.697 |
| **7-day-safe, + temperature/rain (production)** | **3.32** | **27.5%** | **0.777** |
Weather is the useful signal: adding temperature and rain cuts MAE by ~22%
versus the 7-day no-weather model and ~38% versus seasonal naive. The holiday
flag alone changes day-ahead MAE by −0.12 in this run; holidays are context, not
a headline accuracy claim.
The mission's quick-baseline numbers (seasonal naive 5.40 / 44.7% / 0.462) came
from a different window and feature setup. The direction and the weather gain
reproduce; the exact decimals are not copied.
## Uncertainty band
The `lo`/`hi` band is the 10th/90th percentile of holdout residuals added to the
point forecast, clipped at zero. It is an empirical 80% band, not a calibrated
guarantee.
## Small classifier
- Target: `busy` = hourly footfall strictly above that sensor's p75, where the
p75 is computed on the training window only.
- Features: `hour`, `weekday`, `is_weekend`, `temp_c`, `precip_mm`, `is_holiday`.
- Model: standardized logistic regression, `class_weight="balanced"`.
- Split: last 20% of hourly timestamps, time-ordered.
- Result: accuracy 0.708, precision 0.360, recall 0.574, F1 0.443, ROC-AUC 0.726.
- Trains in ~0.25 s on the challenge CSV and under 1 s on a small upload.
- The low precision reflects a hard p75 threshold and a deliberately tiny model;
it is a demo artifact, not the production forecast.
## No-leakage rules
- All splits are time-based; no random shuffling.
- Seasonal lags use the previous *observed* same weekday/hour; future lags are
never filled from future targets.
- Rolling means are shifted at least 24 hours back.
- The production 7-day model uses only ≥168-hour-old information, so a 168-hour
forecast needs no recursion.
- Weather in the holdout is observed, not forecast. The demo uses forecast
weather, so live accuracy will be lower than the holdout number.
- The holdout is used for reporting the headline; the production bundle is
refit on all observed rows only after the holdout is evaluated.
## Limitations
- PAX counters detect discoverable devices; penetration is unknown. Values are
an index of estimated devices, not people.
- Eight sensors cover ~580 m; the data does not support a city-wide surface or a
store-location recommender.
- Five months of multi-sensor history; no annual seasonality claim. `density-01`
has a longer single-sensor history, but that cannot represent the network.
- Sensor gaps of 4–6 days remain. The model tolerates missing lags but cannot
recover unobserved footfall.
- No events, markets, school holidays, bridge days, or promotions are modeled.
Scenario uplifts are explicit assumptions, not causal estimates.
- Weather is observed in training and forecast in the demo; forecast error is
not included in the reported metrics.
- The staffing heuristic assumes visitors-per-staff-hour, shift length, safety
factor, and labor cost. There is no POS, queue, sales, or labor-cost data.
## Ethical and operational notes
- No PII, MAC addresses, or individual tracking is stored or exposed.
- Public API reads are open; training is token-protected and internal.
- Model artifacts and derived aggregates are committed; the raw challenge CSV is
not. No secrets are baked into the image.
- On restart the service falls back to the baked model; retrained models persist
only in `/tmp` and are not assumed to survive a pod restart.