The small-model page said make small reproduced every number, but the threshold table and several derived values only appear in the JSON payload. Say what the commands actually print, fix the stale weather-cache docstring, and lead the e2e evidence page with what it checks.
7.6 KiB
Small model — busy-hour classifier
A logistic regression that labels an hour busy when footfall exceeds that
sensor's p75. It powers the model=small retraining demo, not the dashboard
forecast: six cheap features, a time-based split, and under a second to fit.
Every number below comes from the baked artifact
artifacts/small_metrics.json (version logit-20261114-052723, trained
2026-11-14T05:27:23Z). make small prints the headline metrics and per-sensor
table; scripts/train_small.py --json prints the full payload.
Target
busy is a binary label for one sensor-hour:
busy = 1 if hourly_mean_density > that sensor's 75th percentile
busy = 0 otherwise
- The p75 is computed per sensor on the training window only, then applied unchanged to the holdout. Computing it on the full dataset would leak the holdout distribution into the label.
- A p75 threshold makes "busy" mean "busier than three of four hours at this location", not an absolute footfall claim. The same absolute value can be busy at Soleo and quiet at Stadtgalerie.
- Hours above the threshold are exactly 25% of the training hours by construction, before missing hours are dropped.
| Sensor | Training-window p75 |
|---|---|
density-01 |
15.23 |
hn-goetzentuem-01 |
15.40 |
hn-hagenbuchersee-01 |
18.20 |
hn-hagenbuchersee-02 |
29.17 |
hn-marahaus-01 |
22.23 |
hn-neckarufer-01 |
28.35 |
hn-soleo-01 |
11.38 |
hn-stadtgalerie-01 |
38.41 |
Data and split
- Input: the challenge CSV,
densityDatarows only, aggregated to the mean of the 15-minute readings in each UTC hour. - Window: 2026-06-12T00:00:00Z → 2026-11-07T23:00:00Z.
- 23,952 hourly sensor rows across all 8 usable sensors.
- Split is chronological, not random: the first 80% of timestamps train,
the last 20% test.
- Train: 2026-06-12T00:00:00Z → 2026-10-09T01:00:00Z, 18,242 rows.
- Test: 2026-10-09T02:00:00Z → 2026-11-07T23:00:00Z, 5,710 rows.
- Excluded devices:
maker-fair,niklas-01,hn-bollwerksturm-01(test installations / insufficient coverage). Seeartifacts/data_audit.json.
Features
| Feature | Source | Note |
|---|---|---|
hour |
local Europe/Berlin hour | 0–23, captures the daily curve |
weekday |
local Europe/Berlin weekday | 0 = Monday |
is_weekend |
weekday ≥ 5 | |
is_holiday |
Baden-Württemberg calendar | context, not a headline claim |
temp_c |
observed Open-Meteo archive | joined to the nearest hour (±3 h) |
precip_mm |
observed Open-Meteo archive | same join |
No lags, no target history, no sensor identifier. That is intentional: the classifier must retrain from a small user upload that may not contain a long history, and it must not depend on future rows. Sensor identity is still implicit in the per-sensor threshold.
Model
StandardScaler -> LogisticRegression(
C=1.0, max_iter=1000, class_weight="balanced",
solver="lbfgs", random_state=42)
class_weight="balanced" compensates for the 25/75 class split caused by the
p75 threshold. It buys recall at the cost of precision, which is the right
trade for a "flag the busy hours" demo.
Fit time: 0.30 s on the full challenge CSV in this run (contract: < 5 s, including the API job's CSV parsing).
Metrics
Time-based holdout, 5,710 rows, base rate 20.2% positive.
| Accuracy | Precision | Recall | F1 | ROC-AUC |
|---|---|---|---|---|
| 0.708 | 0.360 | 0.574 | 0.443 | 0.726 |
Confusion matrix:
| Predicted not busy | Predicted busy | |
|---|---|---|
| Actually not busy | 3,377 (TN) | 1,178 (FP) |
| Actually busy | 492 (FN) | 663 (TP) |
Derived:
- Specificity (true-negative rate): 0.741
- Negative predictive value: 0.873
- False-positive rate: 0.259
- False-negative rate: 0.426
- Precision lift over the 20.2% base rate: 1.78×
- Accuracy of the majority-class baseline ("always not busy"): 0.798
The model is less accurate than always saying "not busy", but it finds 663 of 1,155 busy hours (57.4% recall) that the majority baseline never finds. The balanced weights push it toward over-flagging: 1,178 false positives for 1,155 actual busy hours. That is why precision is 0.36 and accuracy is not the right headline.
Per-sensor holdout:
| Sensor | n | Pos. rate | Accuracy | Precision | Recall | F1 |
|---|---|---|---|---|---|---|
density-01 |
714 | 0.111 | 0.761 | 0.300 | 0.873 | 0.447 |
hn-goetzentuem-01 |
714 | 0.197 | 0.732 | 0.391 | 0.638 | 0.485 |
hn-hagenbuchersee-01 |
714 | 0.049 | 0.724 | 0.148 | 0.971 | 0.257 |
hn-hagenbuchersee-02 |
714 | 0.123 | 0.783 | 0.355 | 0.932 | 0.514 |
hn-marahaus-01 |
714 | 0.055 | 0.710 | 0.135 | 0.795 | 0.230 |
hn-neckarufer-01 |
713 | 0.090 | 0.764 | 0.274 | 0.984 | 0.429 |
hn-soleo-01 |
714 | 0.704 | 0.581 | 0.943 | 0.431 | 0.592 |
hn-stadtgalerie-01 |
713 | 0.289 | 0.604 | 0.335 | 0.374 | 0.353 |
The pooled model has the lowest accuracy on hn-soleo-01 (0.581, an unusually
high base rate of 70.4%) and the lowest recall on hn-stadtgalerie-01 (0.374).
The high recall on low-base-rate sensors is mostly the balanced weights firing;
it is not evidence of a strong signal there.
Reproduce
# Full challenge CSV, prints the metrics and per-sensor table above.
make small
# Explicit CSV, all rows, save the baked artifact.
uv run python scripts/train_small.py --csv new-data.csv --all --save
# Machine-readable payload.
uv run python scripts/train_small.py --json
The runtime path is the same function (pulsecast.small.train_small_from_csv):
curl -s -X POST localhost:18000/train \
-H "Authorization: Bearer $CONTROL_TOKEN" \
-F model=small -F file=@new-data.csv
curl -s localhost:18000/train/<job_id> \
-H "Authorization: Bearer $CONTROL_TOKEN"
The job returns accuracy, precision, recall, f1, roc_auc, n_train,
n_test, positive_rate, and sensors; the full confusion matrix and
per-sensor table are stored in artifacts/small_metrics.json. On success the
bundle is hot-swapped and persisted to PULSECAST_MODELS/small_latest.joblib.
A restart falls back to the baked artifact.
Honest limits
- It is a demo, not the production forecast. The dashboard uses the global HistGB 7-day model, not this classifier.
- Threshold convention. "Busy" is the sensor's own p75, recomputed from the upload's training window. A different upload can move the thresholds and change every metric. Do not compare runs trained on different windows.
- No lags or sensor id. The model sees only calendar and weather. It cannot know that the previous hour was packed, and it pools all locations while evaluating per sensor.
- Weather fallback. If an uploaded CSV does not overlap the baked weather archive, missing weather is filled with 14 °C / 0 mm. That keeps the job running but weakens the weather features; the returned metrics show it.
- One chronological split. Five months, one holdout. No cross-validation, no annual seasonality claim, no per-weekday calibration.
- Threshold metrics only. This page reports hard 0.5-threshold metrics, not a calibrated probability. The probability output is not a business forecast.
- Counters are devices. Footfall density is an index of estimated devices, not a count of people. A "busy" flag inherits that uncertainty.
- Small uploads are unstable. A CSV with one sensor, only one class, or fewer than four hourly timestamps fails with a clear error. A CSV with a short window may train but produce meaningless metrics.
- No causal claims. Weather and holiday coefficients are associations in this sample, not causal effects, and the classifier is not used for scenario uplift (that overlay is explicit and separate).