ci / check (push) Successful in 1m12s
Replace setup-python with setup-uv: the self-hosted Gitea runner lacks the toolcache and lsb_release dependency, so the job failed before any project step. uv installs its own Python 3.12; the checks are unchanged. Align the README and docs with the committed artifacts: quick start serves from baked artifacts, rebuild steps are separate, and the reference-model comparison no longer overstates the result. Drop stale external-doc references and comments that restated the code.
186 lines
7.5 KiB
Markdown
186 lines
7.5 KiB
Markdown
# Small model — busy-hour classifier
|
||
|
||
A logistic regression that labels an hour `busy` when footfall exceeds that
|
||
sensor's p75. It powers the `model=small` retraining demo, not the dashboard
|
||
forecast: six cheap features, a time-based split, and under a second to fit.
|
||
|
||
Every number below comes from the baked artifact
|
||
`artifacts/small_metrics.json` (version `logit-20261114-052723`, trained
|
||
2026-11-14T05:27:23Z) and can be reproduced with `make small`.
|
||
|
||
## Target
|
||
|
||
`busy` is a binary label for one sensor-hour:
|
||
|
||
```text
|
||
busy = 1 if hourly_mean_density > that sensor's 75th percentile
|
||
busy = 0 otherwise
|
||
```
|
||
|
||
- The p75 is computed **per sensor on the training window only**, then applied
|
||
unchanged to the holdout. Computing it on the full dataset would leak the
|
||
holdout distribution into the label.
|
||
- A p75 threshold makes "busy" mean "busier than three of four hours at this
|
||
location", not an absolute footfall claim. The same absolute value can be
|
||
busy at Soleo and quiet at Stadtgalerie.
|
||
- Hours above the threshold are exactly 25% of the training hours by
|
||
construction, before missing hours are dropped.
|
||
|
||
| Sensor | Training-window p75 |
|
||
|---|---:|
|
||
| `density-01` | 15.23 |
|
||
| `hn-goetzentuem-01` | 15.40 |
|
||
| `hn-hagenbuchersee-01` | 18.20 |
|
||
| `hn-hagenbuchersee-02` | 29.17 |
|
||
| `hn-marahaus-01` | 22.23 |
|
||
| `hn-neckarufer-01` | 28.35 |
|
||
| `hn-soleo-01` | 11.38 |
|
||
| `hn-stadtgalerie-01` | 38.41 |
|
||
|
||
## Data and split
|
||
|
||
- Input: the challenge CSV, `densityData` rows only, aggregated to the mean of
|
||
the 15-minute readings in each UTC hour.
|
||
- Window: 2026-06-12T00:00:00Z → 2026-11-07T23:00:00Z.
|
||
- 23,952 hourly sensor rows across all 8 usable sensors.
|
||
- Split is chronological, not random: the first 80% of **timestamps** train,
|
||
the last 20% test.
|
||
- Train: 2026-06-12T00:00:00Z → 2026-10-09T01:00:00Z, 18,242 rows.
|
||
- Test: 2026-10-09T02:00:00Z → 2026-11-07T23:00:00Z, 5,710 rows.
|
||
- Excluded devices: `maker-fair`, `niklas-01`, `hn-bollwerksturm-01` (test
|
||
installations / insufficient coverage). See `artifacts/data_audit.json`.
|
||
|
||
## Features
|
||
|
||
| Feature | Source | Note |
|
||
|---|---|---|
|
||
| `hour` | local Europe/Berlin hour | 0–23, captures the daily curve |
|
||
| `weekday` | local Europe/Berlin weekday | 0 = Monday |
|
||
| `is_weekend` | weekday ≥ 5 | |
|
||
| `is_holiday` | Baden-Württemberg calendar | context, not a headline claim |
|
||
| `temp_c` | observed Open-Meteo archive | joined to the nearest hour (±3 h) |
|
||
| `precip_mm` | observed Open-Meteo archive | same join |
|
||
|
||
No lags, no target history, no sensor identifier. That is intentional: the
|
||
classifier must retrain from a small user upload that may not contain a long
|
||
history, and it must not depend on future rows. Sensor identity is still
|
||
implicit in the per-sensor threshold.
|
||
|
||
## Model
|
||
|
||
```text
|
||
StandardScaler -> LogisticRegression(
|
||
C=1.0, max_iter=1000, class_weight="balanced",
|
||
solver="lbfgs", random_state=42)
|
||
```
|
||
|
||
`class_weight="balanced"` compensates for the 25/75 class split caused by the
|
||
p75 threshold. It buys recall at the cost of precision, which is the right
|
||
trade for a "flag the busy hours" demo.
|
||
|
||
Fit time: **0.30 s** on the full challenge CSV in this run (contract: < 5 s,
|
||
including the API job's CSV parsing).
|
||
|
||
## Metrics
|
||
|
||
Time-based holdout, 5,710 rows, base rate 20.2% positive.
|
||
|
||
| Accuracy | Precision | Recall | F1 | ROC-AUC |
|
||
|---:|---:|---:|---:|---:|
|
||
| 0.708 | 0.360 | 0.574 | 0.443 | 0.726 |
|
||
|
||
Confusion matrix:
|
||
|
||
| | Predicted not busy | Predicted busy |
|
||
|---|---:|---:|
|
||
| **Actually not busy** | 3,377 (TN) | 1,178 (FP) |
|
||
| **Actually busy** | 492 (FN) | 663 (TP) |
|
||
|
||
Derived:
|
||
|
||
- Specificity (true-negative rate): 0.741
|
||
- Negative predictive value: 0.873
|
||
- False-positive rate: 0.259
|
||
- False-negative rate: 0.426
|
||
- Precision lift over the 20.2% base rate: 1.78×
|
||
- Accuracy of the majority-class baseline ("always not busy"): 0.798
|
||
|
||
The model is **less accurate than always saying "not busy"**, but it finds 663
|
||
of 1,155 busy hours (57.4% recall) that the majority baseline never finds. The
|
||
balanced weights push it toward over-flagging: 1,178 false positives for 1,155
|
||
actual busy hours. That is why precision is 0.36 and accuracy is not the right
|
||
headline.
|
||
|
||
Per-sensor holdout:
|
||
|
||
| Sensor | n | Pos. rate | Accuracy | Precision | Recall | F1 |
|
||
|---|---:|---:|---:|---:|---:|---:|
|
||
| `density-01` | 714 | 0.111 | 0.761 | 0.300 | 0.873 | 0.447 |
|
||
| `hn-goetzentuem-01` | 714 | 0.197 | 0.732 | 0.391 | 0.638 | 0.485 |
|
||
| `hn-hagenbuchersee-01` | 714 | 0.049 | 0.724 | 0.148 | 0.971 | 0.257 |
|
||
| `hn-hagenbuchersee-02` | 714 | 0.123 | 0.783 | 0.355 | 0.932 | 0.514 |
|
||
| `hn-marahaus-01` | 714 | 0.055 | 0.710 | 0.135 | 0.795 | 0.230 |
|
||
| `hn-neckarufer-01` | 713 | 0.090 | 0.764 | 0.274 | 0.984 | 0.429 |
|
||
| `hn-soleo-01` | 714 | 0.704 | 0.581 | 0.943 | 0.431 | 0.592 |
|
||
| `hn-stadtgalerie-01` | 713 | 0.289 | 0.604 | 0.335 | 0.374 | 0.353 |
|
||
|
||
The pooled model has the lowest accuracy on `hn-soleo-01` (0.581, an unusually
|
||
high base rate of 70.4%) and the lowest recall on `hn-stadtgalerie-01` (0.374).
|
||
The high recall on low-base-rate sensors is mostly the balanced weights firing;
|
||
it is not evidence of a strong signal there.
|
||
|
||
## Reproduce
|
||
|
||
```bash
|
||
# Full challenge CSV, prints every number above.
|
||
make small
|
||
|
||
# Explicit CSV, all rows, save the baked artifact.
|
||
uv run python scripts/train_small.py --csv new-data.csv --all --save
|
||
|
||
# Machine-readable payload.
|
||
uv run python scripts/train_small.py --json
|
||
```
|
||
|
||
The runtime path is the same function (`pulsecast.small.train_small_from_csv`):
|
||
|
||
```bash
|
||
curl -s -X POST localhost:18000/train \
|
||
-H "Authorization: Bearer $CONTROL_TOKEN" \
|
||
-F model=small -F file=@new-data.csv
|
||
curl -s localhost:18000/train/<job_id> \
|
||
-H "Authorization: Bearer $CONTROL_TOKEN"
|
||
```
|
||
|
||
The job returns `accuracy`, `precision`, `recall`, `f1`, `roc_auc`, `n_train`,
|
||
`n_test`, `positive_rate`, and `sensors`; the full confusion matrix and
|
||
per-sensor table are stored in `artifacts/small_metrics.json`. On success the
|
||
bundle is hot-swapped and persisted to `PULSECAST_MODELS/small_latest.joblib`.
|
||
A restart falls back to the baked artifact.
|
||
|
||
## Honest limits
|
||
|
||
- **It is a demo, not the production forecast.** The dashboard uses the global
|
||
HistGB 7-day model, not this classifier.
|
||
- **Threshold convention.** "Busy" is the sensor's own p75, recomputed from the
|
||
upload's training window. A different upload can move the thresholds and
|
||
change every metric. Do not compare runs trained on different windows.
|
||
- **No lags or sensor id.** The model sees only calendar and weather. It cannot
|
||
know that the previous hour was packed, and it pools all locations while
|
||
evaluating per sensor.
|
||
- **Weather fallback.** If an uploaded CSV does not overlap the baked weather
|
||
archive, missing weather is filled with 14 °C / 0 mm. That keeps the job
|
||
running but weakens the weather features; the returned metrics show it.
|
||
- **One chronological split.** Five months, one holdout. No cross-validation,
|
||
no annual seasonality claim, no per-weekday calibration.
|
||
- **Threshold metrics only.** This page reports hard 0.5-threshold metrics, not
|
||
a calibrated probability. The probability output is not a business forecast.
|
||
- **Counters are devices.** Footfall density is an index of estimated devices,
|
||
not a count of people. A "busy" flag inherits that uncertainty.
|
||
- **Small uploads are unstable.** A CSV with one sensor, only one class, or
|
||
fewer than four hourly timestamps fails with a clear error. A CSV with a
|
||
short window may train but produce meaningless metrics.
|
||
- **No causal claims.** Weather and holiday coefficients are associations in
|
||
this sample, not causal effects, and the classifier is not used for scenario
|
||
uplift (that overlay is explicit and separate).
|