99 lines
7.0 KiB
Markdown
99 lines
7.0 KiB
Markdown
# Data findings — PAX counters, Heilbronn
|
||
|
||
This is based on the actual challenge download, not the brief. Downloaded 2026-11-13 from the Schwarz Digits download link with the published password. The file is not committed to this repo; keep it in `data/raw/` and add that folder to `.gitignore`, or reuse the cache copy.
|
||
|
||
## File
|
||
|
||
| Item | Value |
|
||
|---|---|
|
||
| File | `data-1794095549194.csv` |
|
||
| Size | 18.49 MB, 192,464 rows |
|
||
| Columns | `time` (UTC ISO-8601), `sensor_id`, `name`, `longitude`, `latitude`, `type`, `value` |
|
||
| Types | `densityData` (96,232 rows), `batteryData` (96,232 rows) |
|
||
| Time range | 2025-10-16 21:17 UTC → 2026-11-07 23:51 UTC |
|
||
| Sensors | 12 device IDs; **8 usable locations** (see below) |
|
||
| Cadence | ~15 minutes (median 15.3 min; some sensors 30 min) |
|
||
|
||
## What is actually in it
|
||
|
||
- **8 usable sensors**, all in a **~580 m × 280 m cluster** around Hagenbuchersee, Neckarufer, Stadtgalerie, Marahaus and Götzenturm. This is an inner-city micro-area, not a city-wide network.
|
||
- **5 months of multi-sensor data**: 2026-06-12 → 2026-11-07. The named sensors start then; the reason is not documented. The one sensor `density-01` has 13 months of history, but it is a single location and is not enough for annual seasonality.
|
||
- **3 sensors must be excluded:** `maker-fair` (73 readings, located in Hannover), `niklas-01` (8 readings, outside the inner city), `hn-bollwerksturm-01` (5 readings).
|
||
- **`density-01` has two device IDs** (`863f75b0` and `f8c19730`) at the same location, 2025-10-16 → 2026-? and 2026-? → 2026-11-07. Merge them as one logical sensor for analysis; keep the device ID for battery/health.
|
||
- **Battery data** exists for every density reading. Useful for a sensor-health panel, not for forecasting.
|
||
|
||
## Sensor profile (multi-sensor period, 2026-06-12 → 2026-11-07)
|
||
|
||
`value` is the mean of 15-minute density readings. Higher = more devices detected.
|
||
|
||
| Sensor | Readings | Mean | Median | p95 | Max | Max daily gap |
|
||
|---|---:|---:|---:|---:|---:|---:|
|
||
| `hn-stadtgalerie-01` | 8,982 | 21.1 | 8.4 | 67.2 | 137.2 | 6 days |
|
||
| `hn-neckarufer-01` | 9,527 | 19.2 | 11.9 | 61.6 | 255.5 | 5 days |
|
||
| `hn-hagenbuchersee-02` | 10,099 | 19.3 | 11.2 | 64.4 | 200.9 | 4 days |
|
||
| `hn-marahaus-01` | 9,254 | 13.7 | 9.1 | 38.5 | 194.6 | 6 days |
|
||
| `hn-hagenbuchersee-01` | 12,345 | 13.0 | 6.3 | 49.7 | 292.6 | 4 days |
|
||
| `hn-goetzentuem-01` | 9,952 | 11.1 | 8.4 | 29.4 | 210.7 | 5 days |
|
||
| `hn-soleo-01` | 10,327 | 10.6 | 8.4 | 28.0 | 111.3 | 4 days |
|
||
| `density-01` | 12,670 | 9.9 | 7.7 | 28.0 | 63.0 | 35 days (over its longer history) |
|
||
|
||
Across the multi-sensor period the weighted mean is ~14.4. Over the full file the distribution is right-skewed: median 7.7, p95 46.9, p99 78.4, max 292.6 — most readings are low, a few hours are very busy.
|
||
|
||
## Patterns that matter for the product
|
||
|
||
**Hourly mean (local time, all 8 sensors):**
|
||
|
||
| Hour | 00 | 03 | 06 | 09 | 12 | 15 | 17 | 18 | 19 | 20 | 21 | 23 |
|
||
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||
| Mean | 9.7 | 2.4 | 3.1 | 8.0 | 17.2 | 18.8 | 24.5 | 28.4 | **31.5** | 31.3 | 28.6 | 16.8 |
|
||
|
||
- The inner city peaks **18:00–21:00 local**, with the top at 19:00. Morning is quiet.
|
||
- **Saturday is the busiest day** (mean 19.1), then Friday (16.3). Sunday is quieter (11.4) than Monday (11.9).
|
||
- The three busiest locations are **Stadtgalerie, Neckarufer and Hagenbuchersee-02**. That is a retail/leisure cluster, which fits the staffing story.
|
||
- Weather is a real signal: the quick model below improves by ~20% when temperature/rain/cloud/wind are added.
|
||
|
||
## What the data does **not** support
|
||
|
||
- **A city-wide heatmap.** Eight points across 580 m are not a city-wide measurement. An interpolated surface is an estimate; label it as such. A "pulse map" with measured points plus a clearly marked interpolation is honest.
|
||
- **A store-location recommender.** Eight sensors cannot rank commercial zones across Heilbronn. This would be a vision slide, not a feature.
|
||
- **A census of people.** PAX counters detect discoverable Bluetooth devices. Device penetration is unknown. The values are an **index of estimated devices**, not a count of people. Show a calibration assumption (e.g. "if 40% of visitors carry a discoverable device, multiply by 2.5") rather than pretending the index is exact.
|
||
- **Annual seasonality.** The 8-sensor network only covers June–November. Do not claim a full-year model. The one long sensor can illustrate a year, but not the network.
|
||
|
||
## Quick forecasting baseline
|
||
|
||
A deliberately small, reproducible baseline — not a tuned model. Hourly target = mean of 15-minute readings. Train on 2026-06-12 → 2026-10-10; test on the last 28 days (2026-10-10 → 2026-11-07). Global model across sensors, `HistGradientBoostingRegressor`, features: hour, weekday, month, weekend, lags 24/48/168, rolling means, sensor ID. Weather from Open-Meteo (historical, Heilbronn).
|
||
|
||
| Model | MAE | WAPE | R² |
|
||
|---|---:|---:|---:|
|
||
| Seasonal naive (same hour last week) | 5.40 | 44.7% | 0.462 |
|
||
| Day-ahead, no weather | 4.20 | 34.8% | 0.678 |
|
||
| Day-ahead, + temperature/rain | **3.36** | **27.8%** | **0.779** |
|
||
| 7-day-ahead, no weather | 4.37 | 36.2% | 0.672 |
|
||
|
||
**The honest read:** temperature + rain cuts day-ahead error by ~20% versus the no-weather model, and ~38% versus seasonal naive. Adding cloud/wind did not improve the short test. That is a credible differentiator, not a breakthrough. The test period is 4 weeks, weather is observed rather than forecast, and events are not modeled. Do not claim more than this.
|
||
|
||
**Why this matters for the product:** the forecast is good enough to drive a staffing recommendation, not good enough to promise exact headcounts. Show the uncertainty band and the baseline comparison. Judges reward understanding, not decimal places.
|
||
|
||
## Recommended pipeline
|
||
|
||
1. Parse CSV as UTC; convert to `Europe/Berlin` for features; handle DST explicitly.
|
||
2. Merge `density-01` device IDs into one logical sensor.
|
||
3. Exclude `maker-fair`, `niklas-01`, `hn-bollwerksturm-01`.
|
||
4. Aggregate to hourly means per sensor; keep 15-minute data for the drill-down view.
|
||
5. Join historical/forecast weather from Open-Meteo; join Baden-Württemberg holidays.
|
||
6. Train the global model; validate on a time-based holdout; export `forecasts.json`, `metrics.json`, `sensors.json`.
|
||
7. Bake the JSON into the Go binary (or serve from a volume). No runtime Python dependency.
|
||
8. Show data coverage and model metrics in the app's "About the data" panel.
|
||
|
||
## Open data questions for the mentors
|
||
|
||
1. What is the unit of `densityData` — devices per 15-minute interval, concurrent devices, or something else?
|
||
2. Where exactly are the sensors mounted, and what is their catchment area?
|
||
3. What device penetration should we assume, and is there a correction factor?
|
||
4. Why do some sensors have 4–6 day gaps? Maintenance, power, LoRaWAN coverage?
|
||
5. Is there an event log (markets, festivals, holidays) for the period?
|
||
6. Are there more sensors or a longer multi-sensor history available?
|
||
7. How often is the dataset refreshed? Is there a live API, or only this CSV?
|
||
|
||
Ask these early. The answers change the UI copy, not the build.
|