Files
ML-py/model_card.md
T
Noel Sabia 070ecd2a78
ci / check (push) Successful in 1m12s
ci: fix Gitea runner setup and tighten docs
Replace setup-python with setup-uv: the self-hosted Gitea runner lacks the
toolcache and lsb_release dependency, so the job failed before any project
step. uv installs its own Python 3.12; the checks are unchanged.

Align the README and docs with the committed artifacts: quick start serves
from baked artifacts, rebuild steps are separate, and the reference-model
comparison no longer overstates the result. Drop stale external-doc
references and comments that restated the code.
2026-11-14 09:10:47 +01:00

8.5 KiB
Raw Blame History

Model card — PulseCast footfall forecast

Model

One global HistGradientBoostingRegressor predicts hourly footfall density for all eight inner-city sensors. A separately trained logistic regression predicts a binary busy hour for the training demo. Both are batch-trained; no model tunes itself at inference time.

  • Forecast version: histgb-YYYY-MM-DD (baked), histgb-YYYYMMDD-HHMMSS after a runtime retrain.
  • Small classifier version: logit-YYYYMMDD-HHMMSS.
  • Framework: scikit-learn 1.9, pandas 3.1, Python 3.12.

Intended use

Decision support for a store manager: next 7 days of footfall, an uncertainty band, and a transparent staffing heuristic on top. The model is not a census of people, not a city-wide heatmap, and not an automated labor optimizer.

Features (forecast)

Group Features
Calendar hour, weekday, month, is_weekend, is_holiday, sensor_code
Seasonal lag_week (previous observed same weekday/hour), seasonal_mean_4w
Weather temp_c, precip_mm

The production 7-day model uses only features that are at least 168 hours old. Day-ahead ablation models additionally use lag_day, lag_2d, roll_24, roll_168.

Target and data

  • Target: mean of 15-minute densityData readings in a UTC hour, per sensor.
  • Training rows: 2026-06-12T00:00:00Z → 2026-10-10T22:00:00Z, 18,602 rows.
  • Holdout: 2026-10-10T23:00:00Z → 2026-11-07T23:00:00Z, 5,350 evaluation rows (hours with a seasonal-naive baseline).
  • The production bundle is refit on all observed rows (through 2026-11-07T23:00:00Z) only after the holdout is evaluated.
  • Eight usable sensors after excluding maker-fair (73 rows, Hannover), niklas-01 (8 rows), hn-bollwerksturm-01 (5 rows).
  • density-01 merges two physical devices (863f75b0, f8c19730) into one logical location. The older device history is outside the 8-sensor window and is not used for the production model.
  • Weather: Open-Meteo historical archive for training, forecast for inference and scenarios, Heilbronn grid cell.
  • Holidays: explicit Baden-Württemberg 2025–2026 calendar.

Evaluation

Time-based 28-day holdout, no shuffling. Seasonal naive is the baseline. WAPE is sum(|error|) / sum(|actual|). R² is computed on the same 5,350 rows.

Model MAE WAPE R²
Seasonal naive (exact 168-row lag) 5.40 44.7% 0.462
Day-ahead, no weather 3.96 32.8% 0.706
Day-ahead, + holiday flag 3.84 31.8% 0.732
Day-ahead, + temperature/rain 3.36 27.8% 0.763
7-day-safe, no weather 4.08 33.8% 0.697
7-day-safe, + temperature/rain (production) 3.32 27.5% 0.777

Weather is the useful signal: adding temperature and rain cuts MAE by 18.6% versus the 7-day no-weather model and 38.5% versus seasonal naive. The holiday flag alone changes day-ahead MAE by −0.12 in this run; holidays are context, not a headline accuracy claim.

The seasonal-naive row reproduces the mission reference exactly (5.40 / 44.7% / 0.462), and the day-ahead weather row matches 3.36 / 27.8%. The production 7-day model edges that day-ahead reference on MAE and WAPE (3.32 / 27.5%) but is slightly lower on R² (0.777 vs 0.779). The two use different features; lag_week uses the previous observed same weekday/hour across sensor gaps, while the seasonal-naive baseline uses the exact 168-row lag.

Per-sensor holdout MAE improves over each sensor's own seasonal naive by 27.7%–43.9%; the full table, figures, and provenance are in docs/metrics.md. The largest absolute error is Stadtgalerie (MAE 5.83, its baseline 10.39); Soleo has the smallest relative gain (27.7%) and lowest R² (0.548). Every sensor has a positive R² under the model; two seasonal-naive baselines are negative.

Uncertainty band

The lo/hi band is the 10th/90th percentile of holdout residuals added to the point forecast, clipped at zero: p10 −4.83, p50 −0.59, p90 +4.88. It is an empirical 80% band, not a calibrated guarantee; the negative median residual means the point forecast is slightly optimistic on the holdout.

Small classifier ("I coded an AI" artifact)

This model is separate from the production forecaster and powers the model=small retraining demo. Full definition, thresholds, confusion matrix interpretation, and limits: docs/small-model.md.

  • Target: busy = hourly footfall strictly above that sensor's p75, where the p75 is computed on the training window only and then frozen for the holdout.
  • Features: hour, weekday, is_weekend, temp_c, precip_mm, is_holiday (no lags, no sensor identifier, no target history).
  • Model: standardized logistic regression, class_weight="balanced", C=1.0, max_iter=1000, random_state=42.
  • Split: last 20% of hourly timestamps, time-ordered. Train 2026-06-12T00:00Z → 2026-10-09T01:00Z (18,242 rows); test 2026-10-09T02:00Z → 2026-11-07T23:00Z (5,710 rows).
  • Result (logit-20261114-052723): accuracy 0.708, precision 0.360, recall 0.574, F1 0.443, ROC-AUC 0.726, base rate 0.202.
  • Confusion matrix: TN 3,377, FP 1,178, FN 492, TP 663. Specificity 0.741, negative predictive value 0.873.
  • Fit time: 0.30 s on the full challenge CSV; the API job (parse + fit) must stay under the 5 s contract. The live e2e run trained a 60-day upload subset in 0.17 s wall time.
  • Per-sensor F1 ranges from 0.230 (Marahaus, 5.5% positive) to 0.592 (Soleo, 70.4% positive). The balanced weights over-flag low-base-rate sensors; accuracy is lowest on Soleo (0.581) and recall is lowest on Stadtgalerie (0.374).
  • Accuracy is below the 0.798 majority-class baseline. Do not use accuracy as the headline: the model finds 663 of 1,155 busy hours, which the majority baseline never finds. The low precision reflects a hard p75 threshold, a deliberately tiny model, and balanced class weights.
  • This is a demo artifact, not the production forecast. Thresholds move with the upload window, and a CSV without weather overlap falls back to 14 °C / 0 mm.

Artifacts and reproduction

  • artifacts/metrics.json — versions, headline, ablation, per-sensor, small model summary.
  • artifacts/small_metrics.json — full small-model payload including the confusion matrix, per-sensor table, and thresholds.
  • make train regenerates models and metrics; make small prints the small-model numbers; make figures regenerates docs/figures/*.svg.
  • docs/metrics.md is the paper support sheet; it quotes the JSON and does not invent numbers.

No-leakage rules

  • All splits are time-based; no random shuffling.
  • The seasonal-naive baseline uses an exact 168-row lag. The model's lag_week uses the previous observed same weekday/hour; future lags are never filled from future targets.
  • Rolling means are shifted at least 24 hours back.
  • The production 7-day model uses only ≥168-hour-old information, so a 168-hour forecast needs no recursion.
  • Weather in the holdout is observed, not forecast. The demo uses forecast weather, so live accuracy will be lower than the holdout number.
  • The holdout is used for reporting the headline; the production bundle is refit on all observed rows only after the holdout is evaluated.

Limitations

  • PAX counters detect discoverable devices; penetration is unknown. Values are an index of estimated devices, not people.
  • Eight sensors cover ~580 m; the data does not support a city-wide surface or a store-location recommender.
  • Five months of multi-sensor history; no annual seasonality claim. density-01 has a longer single-sensor history, but that cannot represent the network.
  • Sensor gaps of 4–6 days remain. The model tolerates missing lags but cannot recover unobserved footfall.
  • No events, markets, school holidays, bridge days, or promotions are modeled. Scenario uplifts are explicit assumptions, not causal estimates.
  • Weather is observed in training and forecast in the demo; forecast error is not included in the reported metrics.
  • The staffing heuristic assumes visitors-per-staff-hour, shift length, safety factor, and labor cost. There is no POS, queue, sales, or labor-cost data.

Ethical and operational notes

  • No PII, MAC addresses, or individual tracking is stored or exposed.
  • Public API reads are open; training is token-protected and internal.
  • Model artifacts and derived aggregates are committed; the raw challenge CSV is not. No secrets are baked into the image.
  • On restart the service falls back to the baked model; retrained models persist only in /tmp and are not assumed to survive a pod restart.