# Data findings — PAX counters, Heilbronn This is based on the actual challenge download, not the brief. Downloaded 2026-11-13 from the Schwarz Digits download link with the published password. The file is not committed to this repo; keep it in `data/raw/` and add that folder to `.gitignore`, or reuse the cache copy. ## File | Item | Value | |---|---| | File | `data-1794095549194.csv` | | Size | 18.49 MB, 192,464 rows | | Columns | `time` (UTC ISO-8601), `sensor_id`, `name`, `longitude`, `latitude`, `type`, `value` | | Types | `densityData` (96,232 rows), `batteryData` (96,232 rows) | | Time range | 2025-10-16 21:17 UTC → 2026-11-07 23:51 UTC | | Sensors | 12 device IDs; **8 usable locations** (see below) | | Cadence | ~15 minutes (median 15.3 min; some sensors 30 min) | ## What is actually in it - **8 usable sensors**, all in a **~580 m × 280 m cluster** around Hagenbuchersee, Neckarufer, Stadtgalerie, Marahaus and Götzenturm. This is an inner-city micro-area, not a city-wide network. - **5 months of multi-sensor data**: 2026-06-12 → 2026-11-07. The named sensors start then; the reason is not documented. The one sensor `density-01` has 13 months of history, but it is a single location and is not enough for annual seasonality. - **3 sensors must be excluded:** `maker-fair` (73 readings, located in Hannover), `niklas-01` (8 readings, outside the inner city), `hn-bollwerksturm-01` (5 readings). - **`density-01` has two device IDs** (`863f75b0` and `f8c19730`) at the same location, 2025-10-16 → 2026-? and 2026-? → 2026-11-07. Merge them as one logical sensor for analysis; keep the device ID for battery/health. - **Battery data** exists for every density reading. Useful for a sensor-health panel, not for forecasting. ## Sensor profile (multi-sensor period, 2026-06-12 → 2026-11-07) `value` is the mean of 15-minute density readings. Higher = more devices detected. | Sensor | Readings | Mean | Median | p95 | Max | Max daily gap | |---|---:|---:|---:|---:|---:|---:| | `hn-stadtgalerie-01` | 8,982 | 21.1 | 8.4 | 67.2 | 137.2 | 6 days | | `hn-neckarufer-01` | 9,527 | 19.2 | 11.9 | 61.6 | 255.5 | 5 days | | `hn-hagenbuchersee-02` | 10,099 | 19.3 | 11.2 | 64.4 | 200.9 | 4 days | | `hn-marahaus-01` | 9,254 | 13.7 | 9.1 | 38.5 | 194.6 | 6 days | | `hn-hagenbuchersee-01` | 12,345 | 13.0 | 6.3 | 49.7 | 292.6 | 4 days | | `hn-goetzentuem-01` | 9,952 | 11.1 | 8.4 | 29.4 | 210.7 | 5 days | | `hn-soleo-01` | 10,327 | 10.6 | 8.4 | 28.0 | 111.3 | 4 days | | `density-01` | 12,670 | 9.9 | 7.7 | 28.0 | 63.0 | 35 days (over its longer history) | Across the multi-sensor period the weighted mean is ~14.4. Over the full file the distribution is right-skewed: median 7.7, p95 46.9, p99 78.4, max 292.6 — most readings are low, a few hours are very busy. ## Patterns that matter for the product **Hourly mean (local time, all 8 sensors):** | Hour | 00 | 03 | 06 | 09 | 12 | 15 | 17 | 18 | 19 | 20 | 21 | 23 | |---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | Mean | 9.7 | 2.4 | 3.1 | 8.0 | 17.2 | 18.8 | 24.5 | 28.4 | **31.5** | 31.3 | 28.6 | 16.8 | - The inner city peaks **18:00–21:00 local**, with the top at 19:00. Morning is quiet. - **Saturday is the busiest day** (mean 19.1), then Friday (16.3). Sunday is quieter (11.4) than Monday (11.9). - The three busiest locations are **Stadtgalerie, Neckarufer and Hagenbuchersee-02**. That is a retail/leisure cluster, which fits the staffing story. - Weather is a real signal: the quick model below improves by ~20% when temperature/rain are added. ## Context features: latest weather + German holidays The dashboard is not just footfall. It also shows the latest Heilbronn weather and the German/Baden-Württemberg public holidays from the past, and the ML microservice uses the same context as features. This is the "why" layer: it explains spikes and makes the forecast feel less like a black box. ### Weather - **Source:** Open-Meteo, no API key. - **Historical:** `archive-api.open-meteo.com` for the training/validation window. - **Current/forecast:** `api.open-meteo.com/v1/forecast` for the dashboard and what-if scenarios. - **Fields:** `temperature_2m`, `precipitation`, `cloud_cover`, `wind_speed_10m`, `relative_humidity_2m`. - **Timezone:** `Europe/Berlin`; handle DST explicitly. - **Dashboard:** current conditions, a 7-day weather strip, and a "weather context" card on the forecast/plan screens. - **Model:** temperature + rain are the useful features; cloud/wind did not improve the short test. Keep them in the payload for display, but do not claim they improve accuracy. ### German holidays (Baden-Württemberg) Public holidays in the training window and around the demo: | Year | Dates | |---|---| | 2025 | Jan 1, Jan 6, Apr 18, Apr 21, May 1, May 29, Jun 9, Jun 19, Oct 3, Nov 1, Dec 25, Dec 26 | | 2026 | Jan 1, Jan 6, Apr 3, Apr 6, May 1, May 14, May 25, Jun 4, Oct 3, Nov 1, Dec 25, Dec 26 | - **Feature:** `is_holiday` binary + `holiday_name` for display. - **Dashboard:** holiday badges on the time-series and plan; a tooltip explains "public holiday in Baden-Württemberg." - **Honest effect:** in the quick 28-day test, the holiday feature alone added little (MAE 4.15 vs 4.20 without weather; 3.44 vs 3.36 with weather). Weather is the main driver. Keep holidays as context and a feature, but do not claim a large accuracy lift. - **Not included:** school holidays, local events, bridge days. Those are future work. ### How it flows ```text Open-Meteo historical ──► features ──► training/validation Open-Meteo current/7d ──► dashboard cards + scenario inputs BW holiday calendar ──► is_holiday feature + timeline badges │ ▼ FastAPI /predict + /simulate │ ▼ Go API cache ──► Nuxt SSR dashboard ``` The dashboard can show the latest weather without the ML service being available: Go caches the last context payload. The ML service uses weather/holiday inputs when it is healthy; if it is down, the cached forecast still renders. ## What the data does **not** support - **A city-wide heatmap.** Eight points across 580 m are not a city-wide measurement. An interpolated surface is an estimate; label it as such. A "pulse map" with measured points plus a clearly marked interpolation is honest. - **A store-location recommender.** Eight sensors cannot rank commercial zones across Heilbronn. This would be a vision slide, not a feature. - **A census of people.** PAX counters detect discoverable Bluetooth devices. Device penetration is unknown. The values are an **index of estimated devices**, not a count of people. Show a calibration assumption (e.g. "if 40% of visitors carry a discoverable device, multiply by 2.5") rather than pretending the index is exact. - **Annual seasonality.** The 8-sensor network only covers June–November. Do not claim a full-year model. The one long sensor can illustrate a year, but not the network. ## Quick forecasting baseline A deliberately small, reproducible baseline — not a tuned model. Hourly target = mean of 15-minute readings. Train on 2026-06-12 → 2026-10-10; test on the last 28 days (2026-10-10 → 2026-11-07). Global model across sensors, `HistGradientBoostingRegressor`, features: hour, weekday, month, weekend, lags 24/48/168, rolling means, sensor ID. Weather from Open-Meteo (historical, Heilbronn). | Model | MAE | WAPE | R² | |---|---:|---:|---:| | Seasonal naive (same hour last week) | 5.40 | 44.7% | 0.462 | | Day-ahead, no weather | 4.20 | 34.8% | 0.678 | | Day-ahead, + temperature/rain | **3.36** | **27.8%** | **0.779** | | 7-day-ahead, no weather | 4.37 | 36.2% | 0.672 | **The honest read:** temperature + rain cuts day-ahead error by ~20% versus the no-weather model, and ~38% versus seasonal naive. Adding cloud/wind did not improve the short test. That is a credible differentiator, not a breakthrough. The test period is 4 weeks, weather is observed rather than forecast, and events are not modeled. Do not claim more than this. **Why this matters for the product:** the forecast is good enough to drive a staffing recommendation, not good enough to promise exact headcounts. Show the uncertainty band and the baseline comparison. Judges reward understanding, not decimal places. ## Recommended pipeline 1. Parse CSV as UTC; convert to `Europe/Berlin` for features; handle DST explicitly. 2. Merge `density-01` device IDs into one logical sensor. 3. Exclude `maker-fair`, `niklas-01`, `hn-bollwerksturm-01`. 4. Aggregate to hourly means per sensor; keep 15-minute data for the drill-down view. 5. Join historical/forecast weather from Open-Meteo; join Baden-Württemberg holidays; export a `context.json` for the dashboard. 6. Train the global model; validate on a time-based holdout; export `forecasts.json`, `metrics.json`, `sensors.json`. 7. Bake the model into the FastAPI image and the fallback JSON into Go. No runtime data fetch is required for the demo. 8. Show data coverage, weather/holiday context and model metrics in the app's "About the data" panel. ## Open data questions for the mentors 1. What is the unit of `densityData` — devices per 15-minute interval, concurrent devices, or something else? 2. Where exactly are the sensors mounted, and what is their catchment area? 3. What device penetration should we assume, and is there a correction factor? 4. Why do some sensors have 4–6 day gaps? Maintenance, power, LoRaWAN coverage? 5. Is there an event log (markets, festivals, holidays) for the period? 6. Are there more sensors or a longer multi-sensor history available? 7. How often is the dataset refreshed? Is there a live API, or only this CSV? Ask these early. The answers change the UI copy, not the build.