Files
OKF_Vault/mission/data.md
T
2026-11-13 20:14:32 +01:00

7.0 KiB
Raw Blame History

Data findings — PAX counters, Heilbronn

This is based on the actual challenge download, not the brief. Downloaded 2026-11-13 from the Schwarz Digits download link with the published password. The file is not committed to this repo; keep it in data/raw/ and add that folder to .gitignore, or reuse the cache copy.

File

Item Value
File data-1794095549194.csv
Size 18.49 MB, 192,464 rows
Columns time (UTC ISO-8601), sensor_id, name, longitude, latitude, type, value
Types densityData (96,232 rows), batteryData (96,232 rows)
Time range 2025-10-16 21:17 UTC → 2026-11-07 23:51 UTC
Sensors 12 device IDs; 8 usable locations (see below)
Cadence ~15 minutes (median 15.3 min; some sensors 30 min)

What is actually in it

  • 8 usable sensors, all in a ~580 m × 280 m cluster around Hagenbuchersee, Neckarufer, Stadtgalerie, Marahaus and Götzenturm. This is an inner-city micro-area, not a city-wide network.
  • 5 months of multi-sensor data: 2026-06-12 → 2026-11-07. The named sensors start then; the reason is not documented. The one sensor density-01 has 13 months of history, but it is a single location and is not enough for annual seasonality.
  • 3 sensors must be excluded: maker-fair (73 readings, located in Hannover), niklas-01 (8 readings, outside the inner city), hn-bollwerksturm-01 (5 readings).
  • density-01 has two device IDs (863f75b0 and f8c19730) at the same location, 2025-10-16 → 2026-? and 2026-? → 2026-11-07. Merge them as one logical sensor for analysis; keep the device ID for battery/health.
  • Battery data exists for every density reading. Useful for a sensor-health panel, not for forecasting.

Sensor profile (multi-sensor period, 2026-06-12 → 2026-11-07)

value is the mean of 15-minute density readings. Higher = more devices detected.

Sensor Readings Mean Median p95 Max Max daily gap
hn-stadtgalerie-01 8,982 21.1 8.4 67.2 137.2 6 days
hn-neckarufer-01 9,527 19.2 11.9 61.6 255.5 5 days
hn-hagenbuchersee-02 10,099 19.3 11.2 64.4 200.9 4 days
hn-marahaus-01 9,254 13.7 9.1 38.5 194.6 6 days
hn-hagenbuchersee-01 12,345 13.0 6.3 49.7 292.6 4 days
hn-goetzentuem-01 9,952 11.1 8.4 29.4 210.7 5 days
hn-soleo-01 10,327 10.6 8.4 28.0 111.3 4 days
density-01 12,670 9.9 7.7 28.0 63.0 35 days (over its longer history)

Across the multi-sensor period the weighted mean is ~14.4. Over the full file the distribution is right-skewed: median 7.7, p95 46.9, p99 78.4, max 292.6 — most readings are low, a few hours are very busy.

Patterns that matter for the product

Hourly mean (local time, all 8 sensors):

Hour 00 03 06 09 12 15 17 18 19 20 21 23
Mean 9.7 2.4 3.1 8.0 17.2 18.8 24.5 28.4 31.5 31.3 28.6 16.8
  • The inner city peaks 18:00–21:00 local, with the top at 19:00. Morning is quiet.
  • Saturday is the busiest day (mean 19.1), then Friday (16.3). Sunday is quieter (11.4) than Monday (11.9).
  • The three busiest locations are Stadtgalerie, Neckarufer and Hagenbuchersee-02. That is a retail/leisure cluster, which fits the staffing story.
  • Weather is a real signal: the quick model below improves by ~20% when temperature/rain/cloud/wind are added.

What the data does not support

  • A city-wide heatmap. Eight points across 580 m are not a city-wide measurement. An interpolated surface is an estimate; label it as such. A "pulse map" with measured points plus a clearly marked interpolation is honest.
  • A store-location recommender. Eight sensors cannot rank commercial zones across Heilbronn. This would be a vision slide, not a feature.
  • A census of people. PAX counters detect discoverable Bluetooth devices. Device penetration is unknown. The values are an index of estimated devices, not a count of people. Show a calibration assumption (e.g. "if 40% of visitors carry a discoverable device, multiply by 2.5") rather than pretending the index is exact.
  • Annual seasonality. The 8-sensor network only covers June–November. Do not claim a full-year model. The one long sensor can illustrate a year, but not the network.

Quick forecasting baseline

A deliberately small, reproducible baseline — not a tuned model. Hourly target = mean of 15-minute readings. Train on 2026-06-12 → 2026-10-10; test on the last 28 days (2026-10-10 → 2026-11-07). Global model across sensors, HistGradientBoostingRegressor, features: hour, weekday, month, weekend, lags 24/48/168, rolling means, sensor ID. Weather from Open-Meteo (historical, Heilbronn).

Model MAE WAPE R²
Seasonal naive (same hour last week) 5.40 44.7% 0.462
Day-ahead, no weather 4.20 34.8% 0.678
Day-ahead, + temperature/rain 3.36 27.8% 0.779
7-day-ahead, no weather 4.37 36.2% 0.672

The honest read: temperature + rain cuts day-ahead error by ~20% versus the no-weather model, and ~38% versus seasonal naive. Adding cloud/wind did not improve the short test. That is a credible differentiator, not a breakthrough. The test period is 4 weeks, weather is observed rather than forecast, and events are not modeled. Do not claim more than this.

Why this matters for the product: the forecast is good enough to drive a staffing recommendation, not good enough to promise exact headcounts. Show the uncertainty band and the baseline comparison. Judges reward understanding, not decimal places.

  1. Parse CSV as UTC; convert to Europe/Berlin for features; handle DST explicitly.
  2. Merge density-01 device IDs into one logical sensor.
  3. Exclude maker-fair, niklas-01, hn-bollwerksturm-01.
  4. Aggregate to hourly means per sensor; keep 15-minute data for the drill-down view.
  5. Join historical/forecast weather from Open-Meteo; join Baden-Württemberg holidays.
  6. Train the global model; validate on a time-based holdout; export forecasts.json, metrics.json, sensors.json.
  7. Bake the JSON into the Go binary (or serve from a volume). No runtime Python dependency.
  8. Show data coverage and model metrics in the app's "About the data" panel.

Open data questions for the mentors

  1. What is the unit of densityData — devices per 15-minute interval, concurrent devices, or something else?
  2. Where exactly are the sensors mounted, and what is their catchment area?
  3. What device penetration should we assume, and is there a correction factor?
  4. Why do some sensors have 4–6 day gaps? Maintenance, power, LoRaWAN coverage?
  5. Is there an event log (markets, festivals, holidays) for the period?
  6. Are there more sensors or a longer multi-sensor history available?
  7. How often is the dataset refreshed? Is there a live API, or only this CSV?

Ask these early. The answers change the UI copy, not the build.