The small-model page said make small reproduced every number, but the threshold table and several derived values only appear in the JSON payload. Say what the commands actually print, fix the stale weather-cache docstring, and lead the e2e evidence page with what it checks.
187 lines
7.6 KiB
Markdown
187 lines
7.6 KiB
Markdown
# Small model — busy-hour classifier
|
||
|
||
A logistic regression that labels an hour `busy` when footfall exceeds that
|
||
sensor's p75. It powers the `model=small` retraining demo, not the dashboard
|
||
forecast: six cheap features, a time-based split, and under a second to fit.
|
||
|
||
Every number below comes from the baked artifact
|
||
`artifacts/small_metrics.json` (version `logit-20261114-052723`, trained
|
||
2026-11-14T05:27:23Z). `make small` prints the headline metrics and per-sensor
|
||
table; `scripts/train_small.py --json` prints the full payload.
|
||
|
||
## Target
|
||
|
||
`busy` is a binary label for one sensor-hour:
|
||
|
||
```text
|
||
busy = 1 if hourly_mean_density > that sensor's 75th percentile
|
||
busy = 0 otherwise
|
||
```
|
||
|
||
- The p75 is computed **per sensor on the training window only**, then applied
|
||
unchanged to the holdout. Computing it on the full dataset would leak the
|
||
holdout distribution into the label.
|
||
- A p75 threshold makes "busy" mean "busier than three of four hours at this
|
||
location", not an absolute footfall claim. The same absolute value can be
|
||
busy at Soleo and quiet at Stadtgalerie.
|
||
- Hours above the threshold are exactly 25% of the training hours by
|
||
construction, before missing hours are dropped.
|
||
|
||
| Sensor | Training-window p75 |
|
||
|---|---:|
|
||
| `density-01` | 15.23 |
|
||
| `hn-goetzentuem-01` | 15.40 |
|
||
| `hn-hagenbuchersee-01` | 18.20 |
|
||
| `hn-hagenbuchersee-02` | 29.17 |
|
||
| `hn-marahaus-01` | 22.23 |
|
||
| `hn-neckarufer-01` | 28.35 |
|
||
| `hn-soleo-01` | 11.38 |
|
||
| `hn-stadtgalerie-01` | 38.41 |
|
||
|
||
## Data and split
|
||
|
||
- Input: the challenge CSV, `densityData` rows only, aggregated to the mean of
|
||
the 15-minute readings in each UTC hour.
|
||
- Window: 2026-06-12T00:00:00Z → 2026-11-07T23:00:00Z.
|
||
- 23,952 hourly sensor rows across all 8 usable sensors.
|
||
- Split is chronological, not random: the first 80% of **timestamps** train,
|
||
the last 20% test.
|
||
- Train: 2026-06-12T00:00:00Z → 2026-10-09T01:00:00Z, 18,242 rows.
|
||
- Test: 2026-10-09T02:00:00Z → 2026-11-07T23:00:00Z, 5,710 rows.
|
||
- Excluded devices: `maker-fair`, `niklas-01`, `hn-bollwerksturm-01` (test
|
||
installations / insufficient coverage). See `artifacts/data_audit.json`.
|
||
|
||
## Features
|
||
|
||
| Feature | Source | Note |
|
||
|---|---|---|
|
||
| `hour` | local Europe/Berlin hour | 0–23, captures the daily curve |
|
||
| `weekday` | local Europe/Berlin weekday | 0 = Monday |
|
||
| `is_weekend` | weekday ≥ 5 | |
|
||
| `is_holiday` | Baden-Württemberg calendar | context, not a headline claim |
|
||
| `temp_c` | observed Open-Meteo archive | joined to the nearest hour (±3 h) |
|
||
| `precip_mm` | observed Open-Meteo archive | same join |
|
||
|
||
No lags, no target history, no sensor identifier. That is intentional: the
|
||
classifier must retrain from a small user upload that may not contain a long
|
||
history, and it must not depend on future rows. Sensor identity is still
|
||
implicit in the per-sensor threshold.
|
||
|
||
## Model
|
||
|
||
```text
|
||
StandardScaler -> LogisticRegression(
|
||
C=1.0, max_iter=1000, class_weight="balanced",
|
||
solver="lbfgs", random_state=42)
|
||
```
|
||
|
||
`class_weight="balanced"` compensates for the 25/75 class split caused by the
|
||
p75 threshold. It buys recall at the cost of precision, which is the right
|
||
trade for a "flag the busy hours" demo.
|
||
|
||
Fit time: **0.30 s** on the full challenge CSV in this run (contract: < 5 s,
|
||
including the API job's CSV parsing).
|
||
|
||
## Metrics
|
||
|
||
Time-based holdout, 5,710 rows, base rate 20.2% positive.
|
||
|
||
| Accuracy | Precision | Recall | F1 | ROC-AUC |
|
||
|---:|---:|---:|---:|---:|
|
||
| 0.708 | 0.360 | 0.574 | 0.443 | 0.726 |
|
||
|
||
Confusion matrix:
|
||
|
||
| | Predicted not busy | Predicted busy |
|
||
|---|---:|---:|
|
||
| **Actually not busy** | 3,377 (TN) | 1,178 (FP) |
|
||
| **Actually busy** | 492 (FN) | 663 (TP) |
|
||
|
||
Derived:
|
||
|
||
- Specificity (true-negative rate): 0.741
|
||
- Negative predictive value: 0.873
|
||
- False-positive rate: 0.259
|
||
- False-negative rate: 0.426
|
||
- Precision lift over the 20.2% base rate: 1.78×
|
||
- Accuracy of the majority-class baseline ("always not busy"): 0.798
|
||
|
||
The model is **less accurate than always saying "not busy"**, but it finds 663
|
||
of 1,155 busy hours (57.4% recall) that the majority baseline never finds. The
|
||
balanced weights push it toward over-flagging: 1,178 false positives for 1,155
|
||
actual busy hours. That is why precision is 0.36 and accuracy is not the right
|
||
headline.
|
||
|
||
Per-sensor holdout:
|
||
|
||
| Sensor | n | Pos. rate | Accuracy | Precision | Recall | F1 |
|
||
|---|---:|---:|---:|---:|---:|---:|
|
||
| `density-01` | 714 | 0.111 | 0.761 | 0.300 | 0.873 | 0.447 |
|
||
| `hn-goetzentuem-01` | 714 | 0.197 | 0.732 | 0.391 | 0.638 | 0.485 |
|
||
| `hn-hagenbuchersee-01` | 714 | 0.049 | 0.724 | 0.148 | 0.971 | 0.257 |
|
||
| `hn-hagenbuchersee-02` | 714 | 0.123 | 0.783 | 0.355 | 0.932 | 0.514 |
|
||
| `hn-marahaus-01` | 714 | 0.055 | 0.710 | 0.135 | 0.795 | 0.230 |
|
||
| `hn-neckarufer-01` | 713 | 0.090 | 0.764 | 0.274 | 0.984 | 0.429 |
|
||
| `hn-soleo-01` | 714 | 0.704 | 0.581 | 0.943 | 0.431 | 0.592 |
|
||
| `hn-stadtgalerie-01` | 713 | 0.289 | 0.604 | 0.335 | 0.374 | 0.353 |
|
||
|
||
The pooled model has the lowest accuracy on `hn-soleo-01` (0.581, an unusually
|
||
high base rate of 70.4%) and the lowest recall on `hn-stadtgalerie-01` (0.374).
|
||
The high recall on low-base-rate sensors is mostly the balanced weights firing;
|
||
it is not evidence of a strong signal there.
|
||
|
||
## Reproduce
|
||
|
||
```bash
|
||
# Full challenge CSV, prints the metrics and per-sensor table above.
|
||
make small
|
||
|
||
# Explicit CSV, all rows, save the baked artifact.
|
||
uv run python scripts/train_small.py --csv new-data.csv --all --save
|
||
|
||
# Machine-readable payload.
|
||
uv run python scripts/train_small.py --json
|
||
```
|
||
|
||
The runtime path is the same function (`pulsecast.small.train_small_from_csv`):
|
||
|
||
```bash
|
||
curl -s -X POST localhost:18000/train \
|
||
-H "Authorization: Bearer $CONTROL_TOKEN" \
|
||
-F model=small -F file=@new-data.csv
|
||
curl -s localhost:18000/train/<job_id> \
|
||
-H "Authorization: Bearer $CONTROL_TOKEN"
|
||
```
|
||
|
||
The job returns `accuracy`, `precision`, `recall`, `f1`, `roc_auc`, `n_train`,
|
||
`n_test`, `positive_rate`, and `sensors`; the full confusion matrix and
|
||
per-sensor table are stored in `artifacts/small_metrics.json`. On success the
|
||
bundle is hot-swapped and persisted to `PULSECAST_MODELS/small_latest.joblib`.
|
||
A restart falls back to the baked artifact.
|
||
|
||
## Honest limits
|
||
|
||
- **It is a demo, not the production forecast.** The dashboard uses the global
|
||
HistGB 7-day model, not this classifier.
|
||
- **Threshold convention.** "Busy" is the sensor's own p75, recomputed from the
|
||
upload's training window. A different upload can move the thresholds and
|
||
change every metric. Do not compare runs trained on different windows.
|
||
- **No lags or sensor id.** The model sees only calendar and weather. It cannot
|
||
know that the previous hour was packed, and it pools all locations while
|
||
evaluating per sensor.
|
||
- **Weather fallback.** If an uploaded CSV does not overlap the baked weather
|
||
archive, missing weather is filled with 14 °C / 0 mm. That keeps the job
|
||
running but weakens the weather features; the returned metrics show it.
|
||
- **One chronological split.** Five months, one holdout. No cross-validation,
|
||
no annual seasonality claim, no per-weekday calibration.
|
||
- **Threshold metrics only.** This page reports hard 0.5-threshold metrics, not
|
||
a calibrated probability. The probability output is not a business forecast.
|
||
- **Counters are devices.** Footfall density is an index of estimated devices,
|
||
not a count of people. A "busy" flag inherits that uncertainty.
|
||
- **Small uploads are unstable.** A CSV with one sensor, only one class, or
|
||
fewer than four hourly timestamps fails with a clear error. A CSV with a
|
||
short window may train but produce meaningless metrics.
|
||
- **No causal claims.** Weather and holiday coefficients are associations in
|
||
this sample, not causal effects, and the classifier is not used for scenario
|
||
uplift (that overlay is explicit and separate).
|