Fill verified metrics, deployment evidence, and criteria mapping

This commit is contained in:
Noel Sabia committed 2026-11-14 08:27:31 +01:00
1 parent c691324227
commit c1e2c96a63
4 files changed
+76 -65

No files matched your search

+12 -8
View File
@@ -15,7 +15,7 @@ A store manager on Kaiserstraße knows next week’s weather forecast and the st
The Schwarz Digits *Pulse of the City* challenge asks teams to turn “privacy-preserving” pedestrian counts from Heilbronn into predictive tooling for advertising, store locations, and staffing [1]. The wording is doing a lot of work. “Pedestrian counts” implies people; “privacy-preserving” implies that the sensing layer has been designed around a privacy guarantee. Neither claim is self-evident. The counters are LoRaWAN-connected ESP32 devices that passively sniff Bluetooth Low Energy (BLE) advertisements and transmit aggregated device counts [2]. The challenge brief calls them privacy-preserving because they count devices rather than identify people [1]. That is a meaningful architectural choice, but it is not a measurement guarantee, and it is not a legal anonymisation guarantee.
This paper started as an attempt to write the technical report for a hackathon system, PulseCast, that turns the counts into a weekly staffing plan. The system works: on a 28-day holdout, a global weather-aware gradient-boosted model reaches MAE 3.32, WAPE 27.5%, and R² 0.777 against a seasonal-naive baseline of 5.40, 44.7%, and 0.462. The plan is generated in under two seconds on a three-service deployment on STACKIT SKE. But the more we looked at the sensing layer, the clearer it became that the model was predicting a **different quantity** from the one the narrative claimed. The code and the data agree: each value is a short-window count of randomized BLE addresses scaled by a fixed factor. The “15-minute” cadence is the sleep cycle, not the integration window.
This paper started as an attempt to write the technical report for a hackathon system, PulseCast, that turns the counts into a weekly staffing plan. The system works: on a 28-day holdout, a global weather-aware gradient-boosted model reaches MAE 3.32, WAPE 27.5%, and R² 0.777 against a seasonal-naive baseline of 5.40, 44.7%, and 0.462. The plan is generated in 42.8 ms locally on v1.1 (contract: under two seconds) on a three-service deployment on STACKIT SKE, and the deployed system handled 120/120 concurrent SSE clients with zero connection errors and a p95 event latency of 1.03 s. But the more we looked at the sensing layer, the clearer it became that the model was predicting a **different quantity** from the one the narrative claimed. The code and the data agree: each value is a short-window count of randomized BLE addresses scaled by a fixed factor. The “15-minute” cadence is the sleep cycle, not the integration window.
That finding changes the paper. A conventional technical report would present the model metrics and the deployment, note that the data are “estimated devices, not people,” and stop. That is not enough. If a decision support system is built on a measurement that conflates flow and dwell, cannot distinguish a phone from a watch or a car, and samples 10 seconds out of every 15 minutes, then the honest contribution is a **measurement audit** plus a **reference architecture** for the next version. The model is not the hard part. The data contract is.
@@ -90,7 +90,7 @@ The sampling design adds a second bias. A person who passes in 30 seconds has ro
### 2.4 Data forensics
The delivered CSV contains 192,464 rows: 96,232 density readings and 96,232 battery readings. There are 12 device IDs; after merging the two `density-01` IDs and excluding three test sensors, eight usable locations remain in a ~580 m × 280 m cluster. The eight-sensor panel (18 June–7 November 2026) contains 83,156 density values.
The delivered CSV contains 192,464 rows: 96,232 density readings and 96,232 battery readings. There are 12 device IDs; after merging the two `density-01` IDs and excluding three test sensors, eight usable locations remain in a ~580 m × 280 m cluster: `density-01`, `hn-goetzentuem-01`, `hn-hagenbuchersee-01`, `hn-hagenbuchersee-02`, `hn-marahaus-01`, `hn-neckarufer-01`, `hn-soleo-01`, and `hn-stadtgalerie-01`. The eight-sensor panel (18 June–7 November 2026) contains 83,156 density values.
**Quantization.** Every one of those 83,156 values is an exact multiple of 0.7. The reconstructed integer `U` has median 12, p95 71, maximum 418, and 2.7% zeros. A continuous physical series would not land on a 0.7 grid in 83,156 consecutive observations; the only plausible explanation is `value = round(0.7 × integer, 4)`. This is direct, data-internal confirmation of the firmware reading [5].
@@ -336,7 +336,9 @@ This is standard MLOps hygiene [35], but it is often skipped in hackathon system
### 5.7 What the existing results show
The PulseCast results are a useful anchor. The production 7-day model reaches 3.32/27.5/0.777; the day-ahead model 3.36/27.8/0.763; the seasonal naive 5.40/44.7/0.462. Per-sensor MAE ranges 2.26–5.83. The weather ablation is large; the holiday effect is small. These results show that a global model can learn the daily/weekly shape and the weather sensitivity of the index. They do not show that the index is people, that the model generalizes across cities, or that the staffing plan improves outcomes. The honest reading is a **relative decision-support signal**.
The PulseCast results are a useful anchor. The production 7-day model reaches 3.32/27.5/0.777; the day-ahead model 3.36/27.8/0.763; the seasonal naive 5.40/44.7/0.462. Per-sensor MAE ranges 2.26–5.83. The weather ablation is large; the holiday effect is small. The mission brief’s 3.36/27.8/0.779 was the quick baseline; the generated metrics are canonical: production 7-day 3.32/27.5/0.777, day-ahead 3.36/27.8/0.763. These results show that a global model can learn the daily/weekly shape and the weather sensitivity of the index. They do not show that the index is people, that the model generalizes across cities, or that the staffing plan improves outcomes. The honest reading is a **relative decision-support signal**.
The deployed system provides a second kind of evidence. The public app is at https://pulse.noelsabia.dev and the API at https://api-pulse.noelsabia.dev. The v1.1 images are 16.9 MB (api), 249 MB (web), and 952 MB (ml). Plan generation is 42.8 ms locally; the contract is under two seconds. In the 120-client Cloudflare k6 run, 120/120 SSE connections opened and closed with zero connection errors; p95 event latency was 1.03 s, p95 scenario latency 194 ms, presenter POST latency 82.69 ms, and 2,760 events were received over 4m53s. These numbers show that the delivery path works at the demo scale. They do not validate the measurement or the decision quality.
![Figure 4. Forecast versus actual.](figures/fig_forecast.png)
@@ -426,10 +428,12 @@ A paper that reports these tests, even if some fail, is stronger than one that a
### 7.4 System
- 120 concurrent SSE clients for 5 minutes; p95 event latency; open success rate.
- Image sizes and cold deployment time.
- Failure drills: FastAPI down, Open-Meteo down, tunnel down, pod restart.
- Observability: health/ready, data/model versions, staleness labels.
Measured on v1.1: 120/120 SSE connections opened and closed, zero connection errors, p95 event latency 1.03 s, p95 scenario latency 194 ms, presenter POST 82.69 ms, 2,760 events over 4m53s; plan generation 42.8 ms locally (contract < 2 s); images 16.9 MB (api), 249 MB (web), 952 MB (ml). Remaining checks:
- cold deployment time `[TODO]`;
- failure drills: FastAPI down, Open-Meteo down, tunnel down, pod restart;
- observability: health/ready, data/model versions, staleness labels;
- a repeat of the 120-client run after any image or cluster change.
### 7.5 Proposed experiments on the existing CSV
@@ -486,7 +490,7 @@ The honest conclusion is narrower and more useful than a city-scale claim: the d
## Data, code, and AI disclosure
The raw challenge CSV is not redistributed. Derived aggregates, the trained model artifacts, the metrics JSON, and the figure-generation scripts can be shared subject to the organizers’ permission. The code for the audit and the PulseCast system is at `[TODO: repository]`; the live demo is at `[TODO: URL]`. The research notes that support the audit are in `paper/research/` and can be deposited alongside the paper or linked from the repository.
The raw challenge CSV is not redistributed. Derived aggregates, the trained model artifacts, the metrics JSON, and the figure-generation scripts can be shared subject to the organizers’ permission. The code for the audit and the PulseCast system is at `[TODO: repository]`; the live app is at https://pulse.noelsabia.dev and the API at https://api-pulse.noelsabia.dev. The research notes that support the audit are in `paper/research/` and can be deposited alongside the paper or linked from the repository.
AI tools were used for drafting and editing. The authors verified every number, citation, and claim against the delivered data, the project metrics, or a cited source. AI tools are not authors and take no responsibility for the content. The legal analysis is research, not legal advice; a deployment should obtain a written opinion from German counsel, especially on the § 25 TDDDG question.