Replace setup-python with setup-uv: the self-hosted Gitea runner lacks the toolcache and lsb_release dependency, so the job failed before any project step. uv installs its own Python 3.12; the checks are unchanged. Align the README and docs with the committed artifacts: quick start serves from baked artifacts, rebuild steps are separate, and the reference-model comparison no longer overstates the result. Drop stale external-doc references and comments that restated the code.
8.5 KiB
Model card — PulseCast footfall forecast
Model
One global HistGradientBoostingRegressor predicts hourly footfall density for
all eight inner-city sensors. A separately trained logistic regression predicts
a binary busy hour for the training demo. Both are batch-trained; no model
tunes itself at inference time.
- Forecast version:
histgb-YYYY-MM-DD(baked),histgb-YYYYMMDD-HHMMSSafter a runtime retrain. - Small classifier version:
logit-YYYYMMDD-HHMMSS. - Framework: scikit-learn 1.9, pandas 3.1, Python 3.12.
Intended use
Decision support for a store manager: next 7 days of footfall, an uncertainty band, and a transparent staffing heuristic on top. The model is not a census of people, not a city-wide heatmap, and not an automated labor optimizer.
Features (forecast)
| Group | Features |
|---|---|
| Calendar | hour, weekday, month, is_weekend, is_holiday, sensor_code |
| Seasonal | lag_week (previous observed same weekday/hour), seasonal_mean_4w |
| Weather | temp_c, precip_mm |
The production 7-day model uses only features that are at least 168 hours old.
Day-ahead ablation models additionally use lag_day, lag_2d, roll_24,
roll_168.
Target and data
- Target: mean of 15-minute
densityDatareadings in a UTC hour, per sensor. - Training rows: 2026-06-12T00:00:00Z → 2026-10-10T22:00:00Z, 18,602 rows.
- Holdout: 2026-10-10T23:00:00Z → 2026-11-07T23:00:00Z, 5,350 evaluation rows (hours with a seasonal-naive baseline).
- The production bundle is refit on all observed rows (through 2026-11-07T23:00:00Z) only after the holdout is evaluated.
- Eight usable sensors after excluding
maker-fair(73 rows, Hannover),niklas-01(8 rows),hn-bollwerksturm-01(5 rows). density-01merges two physical devices (863f75b0,f8c19730) into one logical location. The older device history is outside the 8-sensor window and is not used for the production model.- Weather: Open-Meteo historical archive for training, forecast for inference and scenarios, Heilbronn grid cell.
- Holidays: explicit Baden-Württemberg 2025–2026 calendar.
Evaluation
Time-based 28-day holdout, no shuffling. Seasonal naive is the baseline. WAPE is
sum(|error|) / sum(|actual|). R² is computed on the same 5,350 rows.
| Model | MAE | WAPE | R² |
|---|---|---|---|
| Seasonal naive (exact 168-row lag) | 5.40 | 44.7% | 0.462 |
| Day-ahead, no weather | 3.96 | 32.8% | 0.706 |
| Day-ahead, + holiday flag | 3.84 | 31.8% | 0.732 |
| Day-ahead, + temperature/rain | 3.36 | 27.8% | 0.763 |
| 7-day-safe, no weather | 4.08 | 33.8% | 0.697 |
| 7-day-safe, + temperature/rain (production) | 3.32 | 27.5% | 0.777 |
Weather is the useful signal: adding temperature and rain cuts MAE by 18.6% versus the 7-day no-weather model and 38.5% versus seasonal naive. The holiday flag alone changes day-ahead MAE by −0.12 in this run; holidays are context, not a headline accuracy claim.
The seasonal-naive row reproduces the mission reference exactly
(5.40 / 44.7% / 0.462), and the day-ahead weather row matches 3.36 / 27.8%. The
production 7-day model edges that day-ahead reference on MAE and WAPE
(3.32 / 27.5%) but is slightly lower on R² (0.777 vs 0.779). The two use
different features; lag_week uses the previous observed same weekday/hour
across sensor gaps, while the seasonal-naive baseline uses the exact 168-row
lag.
Per-sensor holdout MAE improves over each sensor's own seasonal naive by
27.7%–43.9%; the full table, figures, and provenance are in
docs/metrics.md. The largest absolute error is
Stadtgalerie (MAE 5.83, its baseline 10.39); Soleo has the smallest relative
gain (27.7%) and lowest R² (0.548). Every sensor has a positive R² under the
model; two seasonal-naive baselines are negative.
Uncertainty band
The lo/hi band is the 10th/90th percentile of holdout residuals added to the
point forecast, clipped at zero: p10 −4.83, p50 −0.59, p90 +4.88. It is an
empirical 80% band, not a calibrated guarantee; the negative median residual
means the point forecast is slightly optimistic on the holdout.
Small classifier ("I coded an AI" artifact)
This model is separate from the production forecaster and powers the
model=small retraining demo. Full definition, thresholds, confusion matrix
interpretation, and limits: docs/small-model.md.
- Target:
busy= hourly footfall strictly above that sensor's p75, where the p75 is computed on the training window only and then frozen for the holdout. - Features:
hour,weekday,is_weekend,temp_c,precip_mm,is_holiday(no lags, no sensor identifier, no target history). - Model: standardized logistic regression,
class_weight="balanced",C=1.0,max_iter=1000,random_state=42. - Split: last 20% of hourly timestamps, time-ordered. Train 2026-06-12T00:00Z → 2026-10-09T01:00Z (18,242 rows); test 2026-10-09T02:00Z → 2026-11-07T23:00Z (5,710 rows).
- Result (
logit-20261114-052723): accuracy 0.708, precision 0.360, recall 0.574, F1 0.443, ROC-AUC 0.726, base rate 0.202. - Confusion matrix: TN 3,377, FP 1,178, FN 492, TP 663. Specificity 0.741, negative predictive value 0.873.
- Fit time: 0.30 s on the full challenge CSV; the API job (parse + fit) must stay under the 5 s contract. The live e2e run trained a 60-day upload subset in 0.17 s wall time.
- Per-sensor F1 ranges from 0.230 (Marahaus, 5.5% positive) to 0.592 (Soleo, 70.4% positive). The balanced weights over-flag low-base-rate sensors; accuracy is lowest on Soleo (0.581) and recall is lowest on Stadtgalerie (0.374).
- Accuracy is below the 0.798 majority-class baseline. Do not use accuracy as the headline: the model finds 663 of 1,155 busy hours, which the majority baseline never finds. The low precision reflects a hard p75 threshold, a deliberately tiny model, and balanced class weights.
- This is a demo artifact, not the production forecast. Thresholds move with the upload window, and a CSV without weather overlap falls back to 14 °C / 0 mm.
Artifacts and reproduction
artifacts/metrics.json— versions, headline, ablation, per-sensor, small model summary.artifacts/small_metrics.json— full small-model payload including the confusion matrix, per-sensor table, and thresholds.make trainregenerates models and metrics;make smallprints the small-model numbers;make figuresregeneratesdocs/figures/*.svg.docs/metrics.mdis the paper support sheet; it quotes the JSON and does not invent numbers.
No-leakage rules
- All splits are time-based; no random shuffling.
- The seasonal-naive baseline uses an exact 168-row lag. The model's
lag_weekuses the previous observed same weekday/hour; future lags are never filled from future targets. - Rolling means are shifted at least 24 hours back.
- The production 7-day model uses only ≥168-hour-old information, so a 168-hour forecast needs no recursion.
- Weather in the holdout is observed, not forecast. The demo uses forecast weather, so live accuracy will be lower than the holdout number.
- The holdout is used for reporting the headline; the production bundle is refit on all observed rows only after the holdout is evaluated.
Limitations
- PAX counters detect discoverable devices; penetration is unknown. Values are an index of estimated devices, not people.
- Eight sensors cover ~580 m; the data does not support a city-wide surface or a store-location recommender.
- Five months of multi-sensor history; no annual seasonality claim.
density-01has a longer single-sensor history, but that cannot represent the network. - Sensor gaps of 4–6 days remain. The model tolerates missing lags but cannot recover unobserved footfall.
- No events, markets, school holidays, bridge days, or promotions are modeled. Scenario uplifts are explicit assumptions, not causal estimates.
- Weather is observed in training and forecast in the demo; forecast error is not included in the reported metrics.
- The staffing heuristic assumes visitors-per-staff-hour, shift length, safety factor, and labor cost. There is no POS, queue, sales, or labor-cost data.
Ethical and operational notes
- No PII, MAC addresses, or individual tracking is stored or exposed.
- Public API reads are open; training is token-protected and internal.
- Model artifacts and derived aggregates are committed; the raw challenge CSV is not. No secrets are baked into the image.
- On restart the service falls back to the baked model; retrained models persist
only in
/tmpand are not assumed to survive a pod restart.