# Model card — PulseCast footfall forecast ## Model One global `HistGradientBoostingRegressor` predicts hourly footfall density for all eight inner-city sensors. A separately trained logistic regression predicts a binary `busy` hour for the training demo. Both are batch-trained; no model tunes itself at inference time. - Forecast version: `histgb-YYYY-MM-DD` (baked), `histgb-YYYYMMDD-HHMMSS` after a runtime retrain. - Small classifier version: `logit-YYYYMMDD-HHMMSS`. - Framework: scikit-learn 1.9, pandas 3.1, Python 3.12. ## Intended use Decision support for a store manager: next 7 days of footfall, an uncertainty band, and a transparent staffing heuristic on top. The model is not a census of people, not a city-wide heatmap, and not an automated labor optimizer. ## Features (forecast) | Group | Features | |---|---| | Calendar | `hour`, `weekday`, `month`, `is_weekend`, `is_holiday`, `sensor_code` | | Seasonal | `lag_week` (previous observed same weekday/hour), `seasonal_mean_4w` | | Weather | `temp_c`, `precip_mm` | The production 7-day model uses only features that are at least 168 hours old. Day-ahead ablation models additionally use `lag_day`, `lag_2d`, `roll_24`, `roll_168`. ## Target and data - Target: mean of 15-minute `densityData` readings in a UTC hour, per sensor. - Training rows: 2026-06-12T00:00:00Z → 2026-10-10T22:00:00Z, 18,602 rows. - Holdout: 2026-10-10T23:00:00Z → 2026-11-07T23:00:00Z, 5,350 evaluation rows (hours with a seasonal-naive baseline). - The production bundle is refit on all observed rows (through 2026-11-07T23:00:00Z) only after the holdout is evaluated. - Eight usable sensors after excluding `maker-fair` (73 rows, Hannover), `niklas-01` (8 rows), `hn-bollwerksturm-01` (5 rows). - `density-01` merges two physical devices (`863f75b0`, `f8c19730`) into one logical location. The older device history is outside the 8-sensor window and is not used for the production model. - Weather: Open-Meteo historical archive for training, forecast for inference and scenarios, Heilbronn grid cell. - Holidays: explicit Baden-Württemberg 2025–2026 calendar. ## Evaluation Time-based 28-day holdout, no shuffling. Seasonal naive is the baseline. WAPE is `sum(|error|) / sum(|actual|)`. R² is computed on the same 5,350 rows. | Model | MAE | WAPE | R² | |---|---:|---:|---:| | Seasonal naive (exact 168-row lag) | 5.40 | 44.7% | 0.462 | | Day-ahead, no weather | 3.96 | 32.8% | 0.706 | | Day-ahead, + holiday flag | 3.84 | 31.8% | 0.732 | | Day-ahead, + temperature/rain | 3.36 | 27.8% | 0.763 | | 7-day-safe, no weather | 4.08 | 33.8% | 0.697 | | **7-day-safe, + temperature/rain (production)** | **3.32** | **27.5%** | **0.777** | Weather is the useful signal: adding temperature and rain cuts MAE by 18.6% versus the 7-day no-weather model and 38.5% versus seasonal naive. The holiday flag alone changes day-ahead MAE by −0.12 in this run; holidays are context, not a headline accuracy claim. The seasonal-naive row reproduces the mission reference exactly (5.40 / 44.7% / 0.462), and the day-ahead weather row matches 3.36 / 27.8%. The production 7-day model edges that day-ahead reference on MAE and WAPE (3.32 / 27.5%) but is slightly lower on R² (0.777 vs 0.779). The two use different features; `lag_week` uses the previous *observed* same weekday/hour across sensor gaps, while the seasonal-naive baseline uses the exact 168-row lag. Per-sensor holdout MAE improves over each sensor's own seasonal naive by 27.7%–43.9%; the full table, figures, and provenance are in [`docs/metrics.md`](docs/metrics.md). The largest absolute error is Stadtgalerie (MAE 5.83, its baseline 10.39); Soleo has the smallest relative gain (27.7%) and lowest R² (0.548). Every sensor has a positive R² under the model; two seasonal-naive baselines are negative. ## Uncertainty band The `lo`/`hi` band is the 10th/90th percentile of holdout residuals added to the point forecast, clipped at zero: p10 −4.83, p50 −0.59, p90 +4.88. It is an empirical 80% band, not a calibrated guarantee; the negative median residual means the point forecast is slightly optimistic on the holdout. ## Small classifier ("I coded an AI" artifact) This model is separate from the production forecaster and powers the `model=small` retraining demo. Full definition, thresholds, confusion matrix interpretation, and limits: [`docs/small-model.md`](docs/small-model.md). - Target: `busy` = hourly footfall strictly above that sensor's p75, where the p75 is computed on the training window only and then frozen for the holdout. - Features: `hour`, `weekday`, `is_weekend`, `temp_c`, `precip_mm`, `is_holiday` (no lags, no sensor identifier, no target history). - Model: standardized logistic regression, `class_weight="balanced"`, `C=1.0`, `max_iter=1000`, `random_state=42`. - Split: last 20% of hourly timestamps, time-ordered. Train 2026-06-12T00:00Z → 2026-10-09T01:00Z (18,242 rows); test 2026-10-09T02:00Z → 2026-11-07T23:00Z (5,710 rows). - Result (`logit-20261114-052723`): accuracy 0.708, precision 0.360, recall 0.574, F1 0.443, ROC-AUC 0.726, base rate 0.202. - Confusion matrix: TN 3,377, FP 1,178, FN 492, TP 663. Specificity 0.741, negative predictive value 0.873. - Fit time: 0.30 s on the full challenge CSV; the API job (parse + fit) must stay under the 5 s contract. The live e2e run trained a 60-day upload subset in 0.17 s wall time. - Per-sensor F1 ranges from 0.230 (Marahaus, 5.5% positive) to 0.592 (Soleo, 70.4% positive). The balanced weights over-flag low-base-rate sensors; accuracy is lowest on Soleo (0.581) and recall is lowest on Stadtgalerie (0.374). - Accuracy is below the 0.798 majority-class baseline. Do not use accuracy as the headline: the model finds 663 of 1,155 busy hours, which the majority baseline never finds. The low precision reflects a hard p75 threshold, a deliberately tiny model, and balanced class weights. - This is a demo artifact, not the production forecast. Thresholds move with the upload window, and a CSV without weather overlap falls back to 14 °C / 0 mm. ## Artifacts and reproduction - `artifacts/metrics.json` — versions, headline, ablation, per-sensor, small model summary. - `artifacts/small_metrics.json` — full small-model payload including the confusion matrix, per-sensor table, and thresholds. - `make train` regenerates models and metrics; `make small` prints the small-model numbers; `make figures` regenerates `docs/figures/*.svg`. - `docs/metrics.md` is the paper support sheet; it quotes the JSON and does not invent numbers. ## No-leakage rules - All splits are time-based; no random shuffling. - The seasonal-naive baseline uses an exact 168-row lag. The model's `lag_week` uses the previous *observed* same weekday/hour; future lags are never filled from future targets. - Rolling means are shifted at least 24 hours back. - The production 7-day model uses only ≥168-hour-old information, so a 168-hour forecast needs no recursion. - Weather in the holdout is observed, not forecast. The demo uses forecast weather, so live accuracy will be lower than the holdout number. - The holdout is used for reporting the headline; the production bundle is refit on all observed rows only after the holdout is evaluated. ## Limitations - PAX counters detect discoverable devices; penetration is unknown. Values are an index of estimated devices, not people. - Eight sensors cover ~580 m; the data does not support a city-wide surface or a store-location recommender. - Five months of multi-sensor history; no annual seasonality claim. `density-01` has a longer single-sensor history, but that cannot represent the network. - Sensor gaps of 4–6 days remain. The model tolerates missing lags but cannot recover unobserved footfall. - No events, markets, school holidays, bridge days, or promotions are modeled. Scenario uplifts are explicit assumptions, not causal estimates. - Weather is observed in training and forecast in the demo; forecast error is not included in the reported metrics. - The staffing heuristic assumes visitors-per-staff-hour, shift length, safety factor, and labor cost. There is no POS, queue, sales, or labor-cost data. ## Ethical and operational notes - No PII, MAC addresses, or individual tracking is stored or exposed. - Public API reads are open; training is token-protected and internal. - Model artifacts and derived aggregates are committed; the raw challenge CSV is not. No secrets are baked into the image. - On restart the service falls back to the baked model; retrained models persist only in `/tmp` and are not assumed to survive a pod restart.