Files
ML-py/docs/small-model.md
T
Noel Sabia ac0d8984bb
ci / check (push) Successful in 28s
ci / check (pull_request) Successful in 30s
docs: correct reproduction and cache claims
The small-model page said make small reproduced every number, but the
threshold table and several derived values only appear in the JSON payload.
Say what the commands actually print, fix the stale weather-cache docstring,
and lead the e2e evidence page with what it checks.
2026-11-14 09:12:29 +01:00

187 lines
7.6 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Small model — busy-hour classifier
A logistic regression that labels an hour `busy` when footfall exceeds that
sensor's p75. It powers the `model=small` retraining demo, not the dashboard
forecast: six cheap features, a time-based split, and under a second to fit.
Every number below comes from the baked artifact
`artifacts/small_metrics.json` (version `logit-20261114-052723`, trained
2026-11-14T05:27:23Z). `make small` prints the headline metrics and per-sensor
table; `scripts/train_small.py --json` prints the full payload.
## Target
`busy` is a binary label for one sensor-hour:
```text
busy = 1 if hourly_mean_density > that sensor's 75th percentile
busy = 0 otherwise
```
- The p75 is computed **per sensor on the training window only**, then applied
unchanged to the holdout. Computing it on the full dataset would leak the
holdout distribution into the label.
- A p75 threshold makes "busy" mean "busier than three of four hours at this
location", not an absolute footfall claim. The same absolute value can be
busy at Soleo and quiet at Stadtgalerie.
- Hours above the threshold are exactly 25% of the training hours by
construction, before missing hours are dropped.
| Sensor | Training-window p75 |
|---|---:|
| `density-01` | 15.23 |
| `hn-goetzentuem-01` | 15.40 |
| `hn-hagenbuchersee-01` | 18.20 |
| `hn-hagenbuchersee-02` | 29.17 |
| `hn-marahaus-01` | 22.23 |
| `hn-neckarufer-01` | 28.35 |
| `hn-soleo-01` | 11.38 |
| `hn-stadtgalerie-01` | 38.41 |
## Data and split
- Input: the challenge CSV, `densityData` rows only, aggregated to the mean of
the 15-minute readings in each UTC hour.
- Window: 2026-06-12T00:00:00Z → 2026-11-07T23:00:00Z.
- 23,952 hourly sensor rows across all 8 usable sensors.
- Split is chronological, not random: the first 80% of **timestamps** train,
the last 20% test.
- Train: 2026-06-12T00:00:00Z → 2026-10-09T01:00:00Z, 18,242 rows.
- Test: 2026-10-09T02:00:00Z → 2026-11-07T23:00:00Z, 5,710 rows.
- Excluded devices: `maker-fair`, `niklas-01`, `hn-bollwerksturm-01` (test
installations / insufficient coverage). See `artifacts/data_audit.json`.
## Features
| Feature | Source | Note |
|---|---|---|
| `hour` | local Europe/Berlin hour | 0–23, captures the daily curve |
| `weekday` | local Europe/Berlin weekday | 0 = Monday |
| `is_weekend` | weekday ≥ 5 | |
| `is_holiday` | Baden-Württemberg calendar | context, not a headline claim |
| `temp_c` | observed Open-Meteo archive | joined to the nearest hour (±3 h) |
| `precip_mm` | observed Open-Meteo archive | same join |
No lags, no target history, no sensor identifier. That is intentional: the
classifier must retrain from a small user upload that may not contain a long
history, and it must not depend on future rows. Sensor identity is still
implicit in the per-sensor threshold.
## Model
```text
StandardScaler -> LogisticRegression(
C=1.0, max_iter=1000, class_weight="balanced",
solver="lbfgs", random_state=42)
```
`class_weight="balanced"` compensates for the 25/75 class split caused by the
p75 threshold. It buys recall at the cost of precision, which is the right
trade for a "flag the busy hours" demo.
Fit time: **0.30 s** on the full challenge CSV in this run (contract: < 5 s,
including the API job's CSV parsing).
## Metrics
Time-based holdout, 5,710 rows, base rate 20.2% positive.
| Accuracy | Precision | Recall | F1 | ROC-AUC |
|---:|---:|---:|---:|---:|
| 0.708 | 0.360 | 0.574 | 0.443 | 0.726 |
Confusion matrix:
| | Predicted not busy | Predicted busy |
|---|---:|---:|
| **Actually not busy** | 3,377 (TN) | 1,178 (FP) |
| **Actually busy** | 492 (FN) | 663 (TP) |
Derived:
- Specificity (true-negative rate): 0.741
- Negative predictive value: 0.873
- False-positive rate: 0.259
- False-negative rate: 0.426
- Precision lift over the 20.2% base rate: 1.78×
- Accuracy of the majority-class baseline ("always not busy"): 0.798
The model is **less accurate than always saying "not busy"**, but it finds 663
of 1,155 busy hours (57.4% recall) that the majority baseline never finds. The
balanced weights push it toward over-flagging: 1,178 false positives for 1,155
actual busy hours. That is why precision is 0.36 and accuracy is not the right
headline.
Per-sensor holdout:
| Sensor | n | Pos. rate | Accuracy | Precision | Recall | F1 |
|---|---:|---:|---:|---:|---:|---:|
| `density-01` | 714 | 0.111 | 0.761 | 0.300 | 0.873 | 0.447 |
| `hn-goetzentuem-01` | 714 | 0.197 | 0.732 | 0.391 | 0.638 | 0.485 |
| `hn-hagenbuchersee-01` | 714 | 0.049 | 0.724 | 0.148 | 0.971 | 0.257 |
| `hn-hagenbuchersee-02` | 714 | 0.123 | 0.783 | 0.355 | 0.932 | 0.514 |
| `hn-marahaus-01` | 714 | 0.055 | 0.710 | 0.135 | 0.795 | 0.230 |
| `hn-neckarufer-01` | 713 | 0.090 | 0.764 | 0.274 | 0.984 | 0.429 |
| `hn-soleo-01` | 714 | 0.704 | 0.581 | 0.943 | 0.431 | 0.592 |
| `hn-stadtgalerie-01` | 713 | 0.289 | 0.604 | 0.335 | 0.374 | 0.353 |
The pooled model has the lowest accuracy on `hn-soleo-01` (0.581, an unusually
high base rate of 70.4%) and the lowest recall on `hn-stadtgalerie-01` (0.374).
The high recall on low-base-rate sensors is mostly the balanced weights firing;
it is not evidence of a strong signal there.
## Reproduce
```bash
# Full challenge CSV, prints the metrics and per-sensor table above.
make small
# Explicit CSV, all rows, save the baked artifact.
uv run python scripts/train_small.py --csv new-data.csv --all --save
# Machine-readable payload.
uv run python scripts/train_small.py --json
```
The runtime path is the same function (`pulsecast.small.train_small_from_csv`):
```bash
curl -s -X POST localhost:18000/train \
-H "Authorization: Bearer $CONTROL_TOKEN" \
-F model=small -F file=@new-data.csv
curl -s localhost:18000/train/<job_id> \
-H "Authorization: Bearer $CONTROL_TOKEN"
```
The job returns `accuracy`, `precision`, `recall`, `f1`, `roc_auc`, `n_train`,
`n_test`, `positive_rate`, and `sensors`; the full confusion matrix and
per-sensor table are stored in `artifacts/small_metrics.json`. On success the
bundle is hot-swapped and persisted to `PULSECAST_MODELS/small_latest.joblib`.
A restart falls back to the baked artifact.
## Honest limits
- **It is a demo, not the production forecast.** The dashboard uses the global
HistGB 7-day model, not this classifier.
- **Threshold convention.** "Busy" is the sensor's own p75, recomputed from the
upload's training window. A different upload can move the thresholds and
change every metric. Do not compare runs trained on different windows.
- **No lags or sensor id.** The model sees only calendar and weather. It cannot
know that the previous hour was packed, and it pools all locations while
evaluating per sensor.
- **Weather fallback.** If an uploaded CSV does not overlap the baked weather
archive, missing weather is filled with 14 °C / 0 mm. That keeps the job
running but weakens the weather features; the returned metrics show it.
- **One chronological split.** Five months, one holdout. No cross-validation,
no annual seasonality claim, no per-weekday calibration.
- **Threshold metrics only.** This page reports hard 0.5-threshold metrics, not
a calibrated probability. The probability output is not a business forecast.
- **Counters are devices.** Footfall density is an index of estimated devices,
not a count of people. A "busy" flag inherits that uncertainty.
- **Small uploads are unstable.** A CSV with one sensor, only one class, or
fewer than four hourly timestamps fails with a clear error. A CSV with a
short window may train but produce meaningless metrics.
- **No causal claims.** Weather and holiday coefficients are associations in
this sample, not causal effects, and the classifier is not used for scenario
uplift (that overlay is explicit and separate).