Files
OKF_Vault/mission/data.md
T
Noel Sabia 1ff3bc8290
release-please / release-please (push) Failing after 3s
docs(mission): add weather and holiday context
2026-11-13 20:27:06 +01:00

9.7 KiB
Raw Blame History

Data findings — PAX counters, Heilbronn

This is based on the actual challenge download, not the brief. Downloaded 2026-11-13 from the Schwarz Digits download link with the published password. The file is not committed to this repo; keep it in data/raw/ and add that folder to .gitignore, or reuse the cache copy.

File

Item Value
File data-1794095549194.csv
Size 18.49 MB, 192,464 rows
Columns time (UTC ISO-8601), sensor_id, name, longitude, latitude, type, value
Types densityData (96,232 rows), batteryData (96,232 rows)
Time range 2025-10-16 21:17 UTC → 2026-11-07 23:51 UTC
Sensors 12 device IDs; 8 usable locations (see below)
Cadence ~15 minutes (median 15.3 min; some sensors 30 min)

What is actually in it

  • 8 usable sensors, all in a ~580 m × 280 m cluster around Hagenbuchersee, Neckarufer, Stadtgalerie, Marahaus and Götzenturm. This is an inner-city micro-area, not a city-wide network.
  • 5 months of multi-sensor data: 2026-06-12 → 2026-11-07. The named sensors start then; the reason is not documented. The one sensor density-01 has 13 months of history, but it is a single location and is not enough for annual seasonality.
  • 3 sensors must be excluded: maker-fair (73 readings, located in Hannover), niklas-01 (8 readings, outside the inner city), hn-bollwerksturm-01 (5 readings).
  • density-01 has two device IDs (863f75b0 and f8c19730) at the same location, 2025-10-16 → 2026-? and 2026-? → 2026-11-07. Merge them as one logical sensor for analysis; keep the device ID for battery/health.
  • Battery data exists for every density reading. Useful for a sensor-health panel, not for forecasting.

Sensor profile (multi-sensor period, 2026-06-12 → 2026-11-07)

value is the mean of 15-minute density readings. Higher = more devices detected.

Sensor Readings Mean Median p95 Max Max daily gap
hn-stadtgalerie-01 8,982 21.1 8.4 67.2 137.2 6 days
hn-neckarufer-01 9,527 19.2 11.9 61.6 255.5 5 days
hn-hagenbuchersee-02 10,099 19.3 11.2 64.4 200.9 4 days
hn-marahaus-01 9,254 13.7 9.1 38.5 194.6 6 days
hn-hagenbuchersee-01 12,345 13.0 6.3 49.7 292.6 4 days
hn-goetzentuem-01 9,952 11.1 8.4 29.4 210.7 5 days
hn-soleo-01 10,327 10.6 8.4 28.0 111.3 4 days
density-01 12,670 9.9 7.7 28.0 63.0 35 days (over its longer history)

Across the multi-sensor period the weighted mean is ~14.4. Over the full file the distribution is right-skewed: median 7.7, p95 46.9, p99 78.4, max 292.6 — most readings are low, a few hours are very busy.

Patterns that matter for the product

Hourly mean (local time, all 8 sensors):

Hour 00 03 06 09 12 15 17 18 19 20 21 23
Mean 9.7 2.4 3.1 8.0 17.2 18.8 24.5 28.4 31.5 31.3 28.6 16.8
  • The inner city peaks 18:00–21:00 local, with the top at 19:00. Morning is quiet.
  • Saturday is the busiest day (mean 19.1), then Friday (16.3). Sunday is quieter (11.4) than Monday (11.9).
  • The three busiest locations are Stadtgalerie, Neckarufer and Hagenbuchersee-02. That is a retail/leisure cluster, which fits the staffing story.
  • Weather is a real signal: the quick model below improves by ~20% when temperature/rain are added.

Context features: latest weather + German holidays

The dashboard is not just footfall. It also shows the latest Heilbronn weather and the German/Baden-Württemberg public holidays from the past, and the ML microservice uses the same context as features. This is the "why" layer: it explains spikes and makes the forecast feel less like a black box.

Weather

  • Source: Open-Meteo, no API key.
  • Historical: archive-api.open-meteo.com for the training/validation window.
  • Current/forecast: api.open-meteo.com/v1/forecast for the dashboard and what-if scenarios.
  • Fields: temperature_2m, precipitation, cloud_cover, wind_speed_10m, relative_humidity_2m.
  • Timezone: Europe/Berlin; handle DST explicitly.
  • Dashboard: current conditions, a 7-day weather strip, and a "weather context" card on the forecast/plan screens.
  • Model: temperature + rain are the useful features; cloud/wind did not improve the short test. Keep them in the payload for display, but do not claim they improve accuracy.

German holidays (Baden-Württemberg)

Public holidays in the training window and around the demo:

Year Dates
2025 Jan 1, Jan 6, Apr 18, Apr 21, May 1, May 29, Jun 9, Jun 19, Oct 3, Nov 1, Dec 25, Dec 26
2026 Jan 1, Jan 6, Apr 3, Apr 6, May 1, May 14, May 25, Jun 4, Oct 3, Nov 1, Dec 25, Dec 26
  • Feature: is_holiday binary + holiday_name for display.
  • Dashboard: holiday badges on the time-series and plan; a tooltip explains "public holiday in Baden-Württemberg."
  • Honest effect: in the quick 28-day test, the holiday feature alone added little (MAE 4.15 vs 4.20 without weather; 3.44 vs 3.36 with weather). Weather is the main driver. Keep holidays as context and a feature, but do not claim a large accuracy lift.
  • Not included: school holidays, local events, bridge days. Those are future work.

How it flows

Open-Meteo historical ──► features ──► training/validation
Open-Meteo current/7d  ──► dashboard cards + scenario inputs
BW holiday calendar    ──► is_holiday feature + timeline badges
                                      │
                                      ▼
                         FastAPI /predict + /simulate
                                      │
                                      ▼
                         Go API cache ──► Nuxt SSR dashboard

The dashboard can show the latest weather without the ML service being available: Go caches the last context payload. The ML service uses weather/holiday inputs when it is healthy; if it is down, the cached forecast still renders.

What the data does not support

  • A city-wide heatmap. Eight points across 580 m are not a city-wide measurement. An interpolated surface is an estimate; label it as such. A "pulse map" with measured points plus a clearly marked interpolation is honest.
  • A store-location recommender. Eight sensors cannot rank commercial zones across Heilbronn. This would be a vision slide, not a feature.
  • A census of people. PAX counters detect discoverable Bluetooth devices. Device penetration is unknown. The values are an index of estimated devices, not a count of people. Show a calibration assumption (e.g. "if 40% of visitors carry a discoverable device, multiply by 2.5") rather than pretending the index is exact.
  • Annual seasonality. The 8-sensor network only covers June–November. Do not claim a full-year model. The one long sensor can illustrate a year, but not the network.

Quick forecasting baseline

A deliberately small, reproducible baseline — not a tuned model. Hourly target = mean of 15-minute readings. Train on 2026-06-12 → 2026-10-10; test on the last 28 days (2026-10-10 → 2026-11-07). Global model across sensors, HistGradientBoostingRegressor, features: hour, weekday, month, weekend, lags 24/48/168, rolling means, sensor ID. Weather from Open-Meteo (historical, Heilbronn).

Model MAE WAPE R²
Seasonal naive (same hour last week) 5.40 44.7% 0.462
Day-ahead, no weather 4.20 34.8% 0.678
Day-ahead, + temperature/rain 3.36 27.8% 0.779
7-day-ahead, no weather 4.37 36.2% 0.672

The honest read: temperature + rain cuts day-ahead error by ~20% versus the no-weather model, and ~38% versus seasonal naive. Adding cloud/wind did not improve the short test. That is a credible differentiator, not a breakthrough. The test period is 4 weeks, weather is observed rather than forecast, and events are not modeled. Do not claim more than this.

Why this matters for the product: the forecast is good enough to drive a staffing recommendation, not good enough to promise exact headcounts. Show the uncertainty band and the baseline comparison. Judges reward understanding, not decimal places.

  1. Parse CSV as UTC; convert to Europe/Berlin for features; handle DST explicitly.
  2. Merge density-01 device IDs into one logical sensor.
  3. Exclude maker-fair, niklas-01, hn-bollwerksturm-01.
  4. Aggregate to hourly means per sensor; keep 15-minute data for the drill-down view.
  5. Join historical/forecast weather from Open-Meteo; join Baden-Württemberg holidays; export a context.json for the dashboard.
  6. Train the global model; validate on a time-based holdout; export forecasts.json, metrics.json, sensors.json.
  7. Bake the model into the FastAPI image and the fallback JSON into Go. No runtime data fetch is required for the demo.
  8. Show data coverage, weather/holiday context and model metrics in the app's "About the data" panel.

Open data questions for the mentors

  1. What is the unit of densityData — devices per 15-minute interval, concurrent devices, or something else?
  2. Where exactly are the sensors mounted, and what is their catchment area?
  3. What device penetration should we assume, and is there a correction factor?
  4. Why do some sensors have 4–6 day gaps? Maintenance, power, LoRaWAN coverage?
  5. Is there an event log (markets, festivals, holidays) for the period?
  6. Are there more sensors or a longer multi-sensor history available?
  7. How often is the dataset refreshed? Is there a live API, or only this CSV?

Ask these early. The answers change the UI copy, not the build.