docs(mission): add PulseCast plan, data findings and access notes
release-please / release-please (push) Failing after 3s

This commit is contained in:
Noel Sabia committed 2026-11-13 20:14:32 +01:00
1 parent 48ffb5ba08
commit 8afb767585
11 files changed
+1227

No files matched your search

+98
View File
@@ -0,0 +1,98 @@
# Data findings — PAX counters, Heilbronn
This is based on the actual challenge download, not the brief. Downloaded 2026-11-13 from the Schwarz Digits download link with the published password. The file is not committed to this repo; keep it in `data/raw/` and add that folder to `.gitignore`, or reuse the cache copy.
## File
| Item | Value |
|---|---|
| File | `data-1794095549194.csv` |
| Size | 18.49 MB, 192,464 rows |
| Columns | `time` (UTC ISO-8601), `sensor_id`, `name`, `longitude`, `latitude`, `type`, `value` |
| Types | `densityData` (96,232 rows), `batteryData` (96,232 rows) |
| Time range | 2025-10-16 21:17 UTC → 2026-11-07 23:51 UTC |
| Sensors | 12 device IDs; **8 usable locations** (see below) |
| Cadence | ~15 minutes (median 15.3 min; some sensors 30 min) |
## What is actually in it
- **8 usable sensors**, all in a **~580 m × 280 m cluster** around Hagenbuchersee, Neckarufer, Stadtgalerie, Marahaus and Götzenturm. This is an inner-city micro-area, not a city-wide network.
- **5 months of multi-sensor data**: 2026-06-12 → 2026-11-07. The named sensors start then; the reason is not documented. The one sensor `density-01` has 13 months of history, but it is a single location and is not enough for annual seasonality.
- **3 sensors must be excluded:** `maker-fair` (73 readings, located in Hannover), `niklas-01` (8 readings, outside the inner city), `hn-bollwerksturm-01` (5 readings).
- **`density-01` has two device IDs** (`863f75b0` and `f8c19730`) at the same location, 2025-10-16 → 2026-? and 2026-? → 2026-11-07. Merge them as one logical sensor for analysis; keep the device ID for battery/health.
- **Battery data** exists for every density reading. Useful for a sensor-health panel, not for forecasting.
## Sensor profile (multi-sensor period, 2026-06-12 → 2026-11-07)
`value` is the mean of 15-minute density readings. Higher = more devices detected.
| Sensor | Readings | Mean | Median | p95 | Max | Max daily gap |
|---|---:|---:|---:|---:|---:|---:|
| `hn-stadtgalerie-01` | 8,982 | 21.1 | 8.4 | 67.2 | 137.2 | 6 days |
| `hn-neckarufer-01` | 9,527 | 19.2 | 11.9 | 61.6 | 255.5 | 5 days |
| `hn-hagenbuchersee-02` | 10,099 | 19.3 | 11.2 | 64.4 | 200.9 | 4 days |
| `hn-marahaus-01` | 9,254 | 13.7 | 9.1 | 38.5 | 194.6 | 6 days |
| `hn-hagenbuchersee-01` | 12,345 | 13.0 | 6.3 | 49.7 | 292.6 | 4 days |
| `hn-goetzentuem-01` | 9,952 | 11.1 | 8.4 | 29.4 | 210.7 | 5 days |
| `hn-soleo-01` | 10,327 | 10.6 | 8.4 | 28.0 | 111.3 | 4 days |
| `density-01` | 12,670 | 9.9 | 7.7 | 28.0 | 63.0 | 35 days (over its longer history) |
Across the multi-sensor period the weighted mean is ~14.4. Over the full file the distribution is right-skewed: median 7.7, p95 46.9, p99 78.4, max 292.6 — most readings are low, a few hours are very busy.
## Patterns that matter for the product
**Hourly mean (local time, all 8 sensors):**
| Hour | 00 | 03 | 06 | 09 | 12 | 15 | 17 | 18 | 19 | 20 | 21 | 23 |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| Mean | 9.7 | 2.4 | 3.1 | 8.0 | 17.2 | 18.8 | 24.5 | 28.4 | **31.5** | 31.3 | 28.6 | 16.8 |
- The inner city peaks **18:00–21:00 local**, with the top at 19:00. Morning is quiet.
- **Saturday is the busiest day** (mean 19.1), then Friday (16.3). Sunday is quieter (11.4) than Monday (11.9).
- The three busiest locations are **Stadtgalerie, Neckarufer and Hagenbuchersee-02**. That is a retail/leisure cluster, which fits the staffing story.
- Weather is a real signal: the quick model below improves by ~20% when temperature/rain/cloud/wind are added.
## What the data does **not** support
- **A city-wide heatmap.** Eight points across 580 m are not a city-wide measurement. An interpolated surface is an estimate; label it as such. A "pulse map" with measured points plus a clearly marked interpolation is honest.
- **A store-location recommender.** Eight sensors cannot rank commercial zones across Heilbronn. This would be a vision slide, not a feature.
- **A census of people.** PAX counters detect discoverable Bluetooth devices. Device penetration is unknown. The values are an **index of estimated devices**, not a count of people. Show a calibration assumption (e.g. "if 40% of visitors carry a discoverable device, multiply by 2.5") rather than pretending the index is exact.
- **Annual seasonality.** The 8-sensor network only covers June–November. Do not claim a full-year model. The one long sensor can illustrate a year, but not the network.
## Quick forecasting baseline
A deliberately small, reproducible baseline — not a tuned model. Hourly target = mean of 15-minute readings. Train on 2026-06-12 → 2026-10-10; test on the last 28 days (2026-10-10 → 2026-11-07). Global model across sensors, `HistGradientBoostingRegressor`, features: hour, weekday, month, weekend, lags 24/48/168, rolling means, sensor ID. Weather from Open-Meteo (historical, Heilbronn).
| Model | MAE | WAPE | R² |
|---|---:|---:|---:|
| Seasonal naive (same hour last week) | 5.40 | 44.7% | 0.462 |
| Day-ahead, no weather | 4.20 | 34.8% | 0.678 |
| Day-ahead, + temperature/rain | **3.36** | **27.8%** | **0.779** |
| 7-day-ahead, no weather | 4.37 | 36.2% | 0.672 |
**The honest read:** temperature + rain cuts day-ahead error by ~20% versus the no-weather model, and ~38% versus seasonal naive. Adding cloud/wind did not improve the short test. That is a credible differentiator, not a breakthrough. The test period is 4 weeks, weather is observed rather than forecast, and events are not modeled. Do not claim more than this.
**Why this matters for the product:** the forecast is good enough to drive a staffing recommendation, not good enough to promise exact headcounts. Show the uncertainty band and the baseline comparison. Judges reward understanding, not decimal places.
## Recommended pipeline
1. Parse CSV as UTC; convert to `Europe/Berlin` for features; handle DST explicitly.
2. Merge `density-01` device IDs into one logical sensor.
3. Exclude `maker-fair`, `niklas-01`, `hn-bollwerksturm-01`.
4. Aggregate to hourly means per sensor; keep 15-minute data for the drill-down view.
5. Join historical/forecast weather from Open-Meteo; join Baden-Württemberg holidays.
6. Train the global model; validate on a time-based holdout; export `forecasts.json`, `metrics.json`, `sensors.json`.
7. Bake the JSON into the Go binary (or serve from a volume). No runtime Python dependency.
8. Show data coverage and model metrics in the app's "About the data" panel.
## Open data questions for the mentors
1. What is the unit of `densityData` — devices per 15-minute interval, concurrent devices, or something else?
2. Where exactly are the sensors mounted, and what is their catchment area?
3. What device penetration should we assume, and is there a correction factor?
4. Why do some sensors have 4–6 day gaps? Maintenance, power, LoRaWAN coverage?
5. Is there an event log (markets, festivals, holidays) for the period?
6. Are there more sensors or a longer multi-sensor history available?
7. How often is the dataset refreshed? Is there a live API, or only this CSV?
Ask these early. The answers change the UI copy, not the build.