Files
ML-py/docs/small-model.md
T
Noel Sabia c7ea524b5b docs: small-model artifact, paper metrics, train e2e evidence
- docs/small-model.md: target, features, split, metrics, full confusion
  matrix, per-sensor table, honest limits
- docs/metrics.md: provenance, protocol, headline, ablation, per-sensor,
  uncertainty, small model, supported/unsupported claims
- docs/figures: generated ablation, per-sensor, confusion-matrix SVGs
- docs/train-e2e.md: exact 38-check live run, small + forecast job results
- model_card.md and README updated with final metrics and links
2026-11-14 06:35:22 +01:00

188 lines
7.6 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Small model — busy-hour classifier
This is the small, fast, explainable model that powers the "train new data"
demo and the "I coded an AI" artifact. It is deliberately separate from the
production 7-day forecaster: six cheap features, a linear model, a time-based
split, and under a second of fitting. It is not the model behind the dashboard
forecast.
Every number below comes from the baked artifact
`artifacts/small_metrics.json` (version `logit-20261114-052723`, trained
2026-11-14T05:27:23Z) and can be reproduced with `make small`.
## Target
`busy` is a binary label for one sensor-hour:
```text
busy = 1 if hourly_mean_density > that sensor's 75th percentile
busy = 0 otherwise
```
- The p75 is computed **per sensor on the training window only**, then applied
unchanged to the holdout. Computing it on the full dataset would leak the
holdout distribution into the label.
- A p75 threshold makes "busy" mean "busier than three of four hours at this
location", not an absolute footfall claim. The same absolute value can be
busy at Soleo and quiet at Stadtgalerie.
- Hours above the threshold are exactly 25% of the training hours by
construction, before missing hours are dropped.
| Sensor | Training-window p75 |
|---|---:|
| `density-01` | 15.23 |
| `hn-goetzentuem-01` | 15.40 |
| `hn-hagenbuchersee-01` | 18.20 |
| `hn-hagenbuchersee-02` | 29.17 |
| `hn-marahaus-01` | 22.23 |
| `hn-neckarufer-01` | 28.35 |
| `hn-soleo-01` | 11.38 |
| `hn-stadtgalerie-01` | 38.41 |
## Data and split
- Input: the challenge CSV, `densityData` rows only, aggregated to the mean of
the 15-minute readings in each UTC hour.
- Window: 2026-06-12T00:00:00Z → 2026-11-07T23:00:00Z.
- 23,952 hourly sensor rows across all 8 usable sensors.
- Split is chronological, not random: the first 80% of **timestamps** train,
the last 20% test.
- Train: 2026-06-12T00:00:00Z → 2026-10-09T01:00:00Z, 18,242 rows.
- Test: 2026-10-09T02:00:00Z → 2026-11-07T23:00:00Z, 5,710 rows.
- Excluded devices: `maker-fair`, `niklas-01`, `hn-bollwerksturm-01` (test
installations / insufficient coverage). See `artifacts/data_audit.json`.
## Features
| Feature | Source | Note |
|---|---|---|
| `hour` | local Europe/Berlin hour | 0–23, captures the daily curve |
| `weekday` | local Europe/Berlin weekday | 0 = Monday |
| `is_weekend` | weekday ≥ 5 | |
| `is_holiday` | Baden-Württemberg calendar | context, not a headline claim |
| `temp_c` | observed Open-Meteo archive | joined to the nearest hour (±3 h) |
| `precip_mm` | observed Open-Meteo archive | same join |
No lags, no target history, no sensor identifier. That is intentional: the
classifier must retrain from a small user upload that may not contain a long
history, and it must not depend on future rows. Sensor identity is still
implicit in the per-sensor threshold.
## Model
```text
StandardScaler -> LogisticRegression(
C=1.0, max_iter=1000, class_weight="balanced",
solver="lbfgs", random_state=42)
```
`class_weight="balanced"` compensates for the 25/75 class split caused by the
p75 threshold. It buys recall at the cost of precision, which is the right
trade for a "flag the busy hours" demo.
Fit time: **0.30 s** on the full challenge CSV in this run (contract: < 5 s,
including the API job's CSV parsing).
## Metrics
Time-based holdout, 5,710 rows, base rate 20.2% positive.
| Accuracy | Precision | Recall | F1 | ROC-AUC |
|---:|---:|---:|---:|---:|
| 0.708 | 0.360 | 0.574 | 0.443 | 0.726 |
Confusion matrix:
| | Predicted not busy | Predicted busy |
|---|---:|---:|
| **Actually not busy** | 3,377 (TN) | 1,178 (FP) |
| **Actually busy** | 492 (FN) | 663 (TP) |
Derived:
- Specificity (true-negative rate): 0.741
- Negative predictive value: 0.873
- False-positive rate: 0.259
- False-negative rate: 0.426
- Precision lift over the 20.2% base rate: 1.78×
- Accuracy of the majority-class baseline ("always not busy"): 0.798
The model is **less accurate than always saying "not busy"**, but it finds 663
of 1,155 busy hours (57.4% recall) that the majority baseline never finds. The
balanced weights push it toward over-flagging: 1,178 false positives for 1,155
actual busy hours. That is why precision is 0.36 and accuracy is not the right
headline.
Per-sensor holdout:
| Sensor | n | Pos. rate | Accuracy | Precision | Recall | F1 |
|---|---:|---:|---:|---:|---:|---:|
| `density-01` | 714 | 0.111 | 0.761 | 0.300 | 0.873 | 0.447 |
| `hn-goetzentuem-01` | 714 | 0.197 | 0.732 | 0.391 | 0.638 | 0.485 |
| `hn-hagenbuchersee-01` | 714 | 0.049 | 0.724 | 0.148 | 0.971 | 0.257 |
| `hn-hagenbuchersee-02` | 714 | 0.123 | 0.783 | 0.355 | 0.932 | 0.514 |
| `hn-marahaus-01` | 714 | 0.055 | 0.710 | 0.135 | 0.795 | 0.230 |
| `hn-neckarufer-01` | 713 | 0.090 | 0.764 | 0.274 | 0.984 | 0.429 |
| `hn-soleo-01` | 714 | 0.704 | 0.581 | 0.943 | 0.431 | 0.592 |
| `hn-stadtgalerie-01` | 713 | 0.289 | 0.604 | 0.335 | 0.374 | 0.353 |
The pooled model has the lowest accuracy on `hn-soleo-01` (0.581, an unusually
high base rate of 70.4%) and the lowest recall on `hn-stadtgalerie-01` (0.374).
The high recall on low-base-rate sensors is mostly the balanced weights firing;
it is not evidence of a strong signal there.
## Reproduce
```bash
# Full challenge CSV, prints every number above.
make small
# Explicit CSV, all rows, save the baked artifact.
uv run python scripts/train_small.py --csv new-data.csv --all --save
# Machine-readable payload.
uv run python scripts/train_small.py --json
```
The runtime path is the same function (`pulsecast.small.train_small_from_csv`):
```bash
curl -s -X POST localhost:18000/train \
-H "Authorization: Bearer $CONTROL_TOKEN" \
-F model=small -F file=@new-data.csv
curl -s localhost:18000/train/<job_id> \
-H "Authorization: Bearer $CONTROL_TOKEN"
```
The job returns `accuracy`, `precision`, `recall`, `f1`, `roc_auc`, `n_train`,
`n_test`, `positive_rate`, and `sensors`; the full confusion matrix and
per-sensor table are stored in `artifacts/small_metrics.json`. On success the
bundle is hot-swapped and persisted to `PULSECAST_MODELS/small_latest.joblib`.
A restart falls back to the baked artifact.
## Honest limits
- **It is a demo, not the production forecast.** The dashboard uses the global
HistGB 7-day model, not this classifier.
- **Threshold convention.** "Busy" is the sensor's own p75, recomputed from the
upload's training window. A different upload can move the thresholds and
change every metric. Do not compare runs trained on different windows.
- **No lags or sensor id.** The model sees only calendar and weather. It cannot
know that the previous hour was packed, and it pools all locations while
evaluating per sensor.
- **Weather fallback.** If an uploaded CSV does not overlap the baked weather
archive, missing weather is filled with 14 °C / 0 mm. That keeps the job
running but weakens the weather features; the returned metrics show it.
- **One chronological split.** Five months, one holdout. No cross-validation,
no annual seasonality claim, no per-weekday calibration.
- **Threshold metrics only.** This page reports hard 0.5-threshold metrics, not
a calibrated probability. The probability output is not a business forecast.
- **Counters are devices.** Footfall density is an index of estimated devices,
not a count of people. A "busy" flag inherits that uncertainty.
- **Small uploads are unstable.** A CSV with one sensor, only one class, or
fewer than four hourly timestamps fails with a clear error. A CSV with a
short window may train but produce meaningless metrics.
- **No causal claims.** Weather and holiday coefficients are associations in
this sample, not causal effects, and the classifier is not used for scenario
uplift (that overlay is explicit and separate).