Files
ML-py/docs/small-model.md
T
Noel Sabia c7ea524b5b docs: small-model artifact, paper metrics, train e2e evidence
- docs/small-model.md: target, features, split, metrics, full confusion
  matrix, per-sensor table, honest limits
- docs/metrics.md: provenance, protocol, headline, ablation, per-sensor,
  uncertainty, small model, supported/unsupported claims
- docs/figures: generated ablation, per-sensor, confusion-matrix SVGs
- docs/train-e2e.md: exact 38-check live run, small + forecast job results
- model_card.md and README updated with final metrics and links
2026-11-14 06:35:22 +01:00

7.6 KiB
Raw Blame History

Small model — busy-hour classifier

This is the small, fast, explainable model that powers the "train new data" demo and the "I coded an AI" artifact. It is deliberately separate from the production 7-day forecaster: six cheap features, a linear model, a time-based split, and under a second of fitting. It is not the model behind the dashboard forecast.

Every number below comes from the baked artifact artifacts/small_metrics.json (version logit-20261114-052723, trained 2026-11-14T05:27:23Z) and can be reproduced with make small.

Target

busy is a binary label for one sensor-hour:

busy = 1  if hourly_mean_density > that sensor's 75th percentile
busy = 0  otherwise
  • The p75 is computed per sensor on the training window only, then applied unchanged to the holdout. Computing it on the full dataset would leak the holdout distribution into the label.
  • A p75 threshold makes "busy" mean "busier than three of four hours at this location", not an absolute footfall claim. The same absolute value can be busy at Soleo and quiet at Stadtgalerie.
  • Hours above the threshold are exactly 25% of the training hours by construction, before missing hours are dropped.
Sensor Training-window p75
density-01 15.23
hn-goetzentuem-01 15.40
hn-hagenbuchersee-01 18.20
hn-hagenbuchersee-02 29.17
hn-marahaus-01 22.23
hn-neckarufer-01 28.35
hn-soleo-01 11.38
hn-stadtgalerie-01 38.41

Data and split

  • Input: the challenge CSV, densityData rows only, aggregated to the mean of the 15-minute readings in each UTC hour.
  • Window: 2026-06-12T00:00:00Z → 2026-11-07T23:00:00Z.
  • 23,952 hourly sensor rows across all 8 usable sensors.
  • Split is chronological, not random: the first 80% of timestamps train, the last 20% test.
    • Train: 2026-06-12T00:00:00Z → 2026-10-09T01:00:00Z, 18,242 rows.
    • Test: 2026-10-09T02:00:00Z → 2026-11-07T23:00:00Z, 5,710 rows.
  • Excluded devices: maker-fair, niklas-01, hn-bollwerksturm-01 (test installations / insufficient coverage). See artifacts/data_audit.json.

Features

Feature Source Note
hour local Europe/Berlin hour 0–23, captures the daily curve
weekday local Europe/Berlin weekday 0 = Monday
is_weekend weekday ≥ 5
is_holiday Baden-Württemberg calendar context, not a headline claim
temp_c observed Open-Meteo archive joined to the nearest hour (±3 h)
precip_mm observed Open-Meteo archive same join

No lags, no target history, no sensor identifier. That is intentional: the classifier must retrain from a small user upload that may not contain a long history, and it must not depend on future rows. Sensor identity is still implicit in the per-sensor threshold.

Model

StandardScaler -> LogisticRegression(
    C=1.0, max_iter=1000, class_weight="balanced",
    solver="lbfgs", random_state=42)

class_weight="balanced" compensates for the 25/75 class split caused by the p75 threshold. It buys recall at the cost of precision, which is the right trade for a "flag the busy hours" demo.

Fit time: 0.30 s on the full challenge CSV in this run (contract: < 5 s, including the API job's CSV parsing).

Metrics

Time-based holdout, 5,710 rows, base rate 20.2% positive.

Accuracy Precision Recall F1 ROC-AUC
0.708 0.360 0.574 0.443 0.726

Confusion matrix:

Predicted not busy Predicted busy
Actually not busy 3,377 (TN) 1,178 (FP)
Actually busy 492 (FN) 663 (TP)

Derived:

  • Specificity (true-negative rate): 0.741
  • Negative predictive value: 0.873
  • False-positive rate: 0.259
  • False-negative rate: 0.426
  • Precision lift over the 20.2% base rate: 1.78×
  • Accuracy of the majority-class baseline ("always not busy"): 0.798

The model is less accurate than always saying "not busy", but it finds 663 of 1,155 busy hours (57.4% recall) that the majority baseline never finds. The balanced weights push it toward over-flagging: 1,178 false positives for 1,155 actual busy hours. That is why precision is 0.36 and accuracy is not the right headline.

Per-sensor holdout:

Sensor n Pos. rate Accuracy Precision Recall F1
density-01 714 0.111 0.761 0.300 0.873 0.447
hn-goetzentuem-01 714 0.197 0.732 0.391 0.638 0.485
hn-hagenbuchersee-01 714 0.049 0.724 0.148 0.971 0.257
hn-hagenbuchersee-02 714 0.123 0.783 0.355 0.932 0.514
hn-marahaus-01 714 0.055 0.710 0.135 0.795 0.230
hn-neckarufer-01 713 0.090 0.764 0.274 0.984 0.429
hn-soleo-01 714 0.704 0.581 0.943 0.431 0.592
hn-stadtgalerie-01 713 0.289 0.604 0.335 0.374 0.353

The pooled model has the lowest accuracy on hn-soleo-01 (0.581, an unusually high base rate of 70.4%) and the lowest recall on hn-stadtgalerie-01 (0.374). The high recall on low-base-rate sensors is mostly the balanced weights firing; it is not evidence of a strong signal there.

Reproduce

# Full challenge CSV, prints every number above.
make small

# Explicit CSV, all rows, save the baked artifact.
uv run python scripts/train_small.py --csv new-data.csv --all --save

# Machine-readable payload.
uv run python scripts/train_small.py --json

The runtime path is the same function (pulsecast.small.train_small_from_csv):

curl -s -X POST localhost:18000/train \
  -H "Authorization: Bearer $CONTROL_TOKEN" \
  -F model=small -F file=@new-data.csv
curl -s localhost:18000/train/<job_id> \
  -H "Authorization: Bearer $CONTROL_TOKEN"

The job returns accuracy, precision, recall, f1, roc_auc, n_train, n_test, positive_rate, and sensors; the full confusion matrix and per-sensor table are stored in artifacts/small_metrics.json. On success the bundle is hot-swapped and persisted to PULSECAST_MODELS/small_latest.joblib. A restart falls back to the baked artifact.

Honest limits

  • It is a demo, not the production forecast. The dashboard uses the global HistGB 7-day model, not this classifier.
  • Threshold convention. "Busy" is the sensor's own p75, recomputed from the upload's training window. A different upload can move the thresholds and change every metric. Do not compare runs trained on different windows.
  • No lags or sensor id. The model sees only calendar and weather. It cannot know that the previous hour was packed, and it pools all locations while evaluating per sensor.
  • Weather fallback. If an uploaded CSV does not overlap the baked weather archive, missing weather is filled with 14 °C / 0 mm. That keeps the job running but weakens the weather features; the returned metrics show it.
  • One chronological split. Five months, one holdout. No cross-validation, no annual seasonality claim, no per-weekday calibration.
  • Threshold metrics only. This page reports hard 0.5-threshold metrics, not a calibrated probability. The probability output is not a business forecast.
  • Counters are devices. Footfall density is an index of estimated devices, not a count of people. A "busy" flag inherits that uncertainty.
  • Small uploads are unstable. A CSV with one sensor, only one class, or fewer than four hourly timestamps fails with a clear error. A CSV with a short window may train but produce meaningless metrics.
  • No causal claims. Weather and holiday coefficients are associations in this sample, not causal effects, and the classifier is not used for scenario uplift (that overlay is explicit and separate).