feat: ci docs #3
No files matched your search
@@ -32,10 +32,10 @@ jobs:
|
||||
run: uv sync --frozen
|
||||
|
||||
- name: Static analysis (ruff)
|
||||
run: uv run ruff check src tests main.py
|
||||
run: uv run ruff check src tests scripts main.py
|
||||
|
||||
- name: Byte-compile
|
||||
run: uv run python -m compileall src
|
||||
run: uv run python -m compileall src scripts
|
||||
|
||||
- name: Tests
|
||||
run: uv run pytest -q
|
||||
@@ -1,11 +1,13 @@
|
||||
UV ?= uv
|
||||
|
||||
.PHONY: help data train test lint check serve docker clean
|
||||
.PHONY: help data train small figures test lint check serve docker clean
|
||||
|
||||
help:
|
||||
@printf '%s\n' \
|
||||
'make data parse/clean the raw CSV, join weather + holidays, export JSON' \
|
||||
'make train fit the 28-day holdout model, export metrics/forecasts/model' \
|
||||
'make small train + print the busy-hour classifier metrics from the raw CSV' \
|
||||
'make figures regenerate docs/figures/*.svg from artifacts/metrics.json' \
|
||||
'make test run pytest' \
|
||||
'make lint run ruff static checks' \
|
||||
'make check CI gate: sync --frozen, ruff, compileall, pytest' \
|
||||
@@ -19,16 +21,22 @@ data:
|
||||
train:
|
||||
$(UV) run python -m pulsecast.train
|
||||
|
||||
small:
|
||||
$(UV) run python scripts/train_small.py
|
||||
|
||||
figures:
|
||||
$(UV) run python scripts/plot_metrics.py
|
||||
|
||||
test:
|
||||
$(UV) run pytest
|
||||
|
||||
lint:
|
||||
$(UV) run ruff check src tests main.py
|
||||
$(UV) run ruff check src tests scripts main.py
|
||||
|
||||
check:
|
||||
$(UV) sync --frozen
|
||||
$(UV) run ruff check src tests main.py
|
||||
$(UV) run python -m compileall src
|
||||
$(UV) run ruff check src tests scripts main.py
|
||||
$(UV) run python -m compileall src scripts
|
||||
$(UV) run pytest -q
|
||||
|
||||
serve:
|
||||
|
||||
@@ -37,7 +37,9 @@ Small busy-hour classifier (baked for the "train new data" demo):
|
||||
The classifier predicts `busy` = hourly footfall above the sensor's p75 using
|
||||
hour, weekday, weekend, temperature, rain, and holiday. It trains in under a
|
||||
second on the challenge CSV and is retrained from an uploaded CSV through
|
||||
`POST /train`.
|
||||
`POST /train`. Full definition, confusion matrix, per-sensor table, and honest
|
||||
limits: [`docs/small-model.md`](docs/small-model.md). Paper tables and figures:
|
||||
[`docs/metrics.md`](docs/metrics.md).
|
||||
|
||||
## Quick start
|
||||
|
||||
@@ -47,6 +49,8 @@ Requirements: Python 3.12 and `uv` (the repo also works with a plain venv).
|
||||
uv sync
|
||||
make data # needs the raw CSV, first run also fetches Open-Meteo weather
|
||||
make train # writes artifacts/model.joblib, metrics.json, forecasts.json
|
||||
make small # prints the busy-hour classifier metrics + confusion matrix
|
||||
make figures # regenerates docs/figures/*.svg from metrics.json
|
||||
make test
|
||||
make check # CI gate: sync --frozen, ruff, compileall, pytest
|
||||
make serve # http://localhost:8000
|
||||
@@ -69,7 +73,8 @@ committed so the API and Docker image work without the raw file or network.
|
||||
| `artifacts/sensors.json` | 8 sensors: id, name, lat/lon, mean, p95, battery |
|
||||
| `artifacts/series.json` + `series/<id>.json` | Hourly series per sensor |
|
||||
| `artifacts/forecasts.json` | 48h + 7d forecasts for 4 scenarios |
|
||||
| `artifacts/metrics.json` | Versions, coverage, headline metrics, ablation, limits |
|
||||
| `artifacts/metrics.json` | Versions, coverage, headline metrics, ablation, per-sensor, small summary, limits |
|
||||
| `artifacts/small_metrics.json` | Busy classifier: metrics, confusion matrix, per-sensor, thresholds |
|
||||
| `artifacts/context.json` | Current + 7-day weather, BW holidays |
|
||||
| `artifacts/data_audit.json` | Row/sensor checks against `mission/data.md` |
|
||||
| `artifacts/model.joblib` | Baked HistGB forecast bundle |
|
||||
@@ -118,6 +123,29 @@ and persists to `PULSECAST_MODELS` (default `/tmp/pulsecast/models`). A restart
|
||||
falls back to the baked model. `model=small` is the fast demo; `model=forecast`
|
||||
retrains the global HistGB (slower, still background).
|
||||
|
||||
## End-to-end check
|
||||
|
||||
`scripts/e2e_train.py` starts a real uvicorn process on port 18000 with a
|
||||
control token, then checks the snapshot, every `/predict` combination, every
|
||||
`/simulate` scenario, auth and error paths, one-job-at-a-time, and both real
|
||||
training jobs (small + forecast) including hot-swap and `/predict` afterward:
|
||||
|
||||
```bash
|
||||
uv run python scripts/e2e_train.py --port 18000 --report /tmp/e2e.json
|
||||
```
|
||||
|
||||
The unit and contract tests are `make test`; `make check` is the CI gate.
|
||||
|
||||
## Docs
|
||||
|
||||
| File | Contents |
|
||||
|---|---|
|
||||
| `docs/metrics.md` | Paper tables/figures: protocol, headline, ablation, per sensor, uncertainty, small model, claim limits |
|
||||
| `docs/small-model.md` | Busy-hour target, thresholds, split, metrics, confusion matrix, per sensor, limits |
|
||||
| `docs/train-e2e.md` | Exact live training-job evidence (small + forecast, hot-swap, errors) |
|
||||
| `docs/figures/*.svg` | Ablation MAE, per-sensor MAE, small-model confusion matrix (generated) |
|
||||
| `model_card.md` | Model card: features, split, baseline, ablation, leakage, limitations |
|
||||
|
||||
## Docker
|
||||
|
||||
```bash
|
||||
@@ -153,6 +181,8 @@ src/pulsecast/
|
||||
small.py busy classifier + flexible CSV parser
|
||||
retrain.py runtime forecast retraining from upload
|
||||
server.py FastAPI app and training job manager
|
||||
scripts/ train_small.py, plot_metrics.py, e2e_train.py
|
||||
docs/ metrics.md, small-model.md, generated figures
|
||||
tests/ schema, bootstrap, health, prediction, small-training job
|
||||
model_card.md features, split, baseline, ablation, limits, leakage
|
||||
```
|
||||
|
||||
@@ -1 +1 @@
|
||||
{"data_version": "2026-11-07T23:51:00Z", "model_version": "histgb-2026-11-14", "context_version": "2026-11-14T03:47:09Z", "generated_at": "2026-11-14T04:03:38Z", "forecast_start": "2026-11-14T05:00:00Z", "forecasts": {"density-01": {"48h": {"base": [{"t": "2026-11-14T05:00:00Z", "v": 1.17, "lo": 0.0, "hi": 6.04}, {"t": "2026-11-14T06:00:00Z", "v": 1.95, "lo": 0.0, "hi": 6.83}, {"t": "2026-11-14T07:00:00Z", "v": 2.94, "lo": 0.0, "hi": 7.81}, {"t": "2026-11-14T08:00:00Z", "v": 4.08, "lo": 0.0, "hi": 8.96}, {"t": "2026-11-14T09:00:00Z", "v": 5.68, "lo": 0.85, "hi": 10.55}, {"t": "2026-11-14T10:00:00Z", "v": 8.5, "lo": 3.67, "hi": 13.38}, {"t": "2026-11-14T11:00:00Z", "v": 9.66, "lo": 4.84, "hi": 14.54}, {"t": "2026-11-14T12:00:00Z", "v": 11.39, "lo": 6.56, "hi": 16.26}, {"t": "2026-11-14T13:00:00Z", "v": 11.32, "lo": 6.5, "hi": 16.2}, {"t": "2026-11-14T14:00:00Z", "v": 12.63, "lo": 7.81, "hi": 17.51}, {"t": "2026-11-14T15:00:00Z", "v": 14.78, "lo": 9.96, "hi": 19.66}, {"t": "2026-11-14T16:00:00Z", "v": 14.21, "lo": 9.39, "hi": 19.09}, {"t": "2026-11-14T17:00:00Z", "v": 14.52, "lo": 9.7, "hi": 19.4}, {"t": "2026-11-14T18:00:00Z", "v": 15.67, "lo": 10.84, "hi": 20.55}, {"t": "2026-11-14T19:00:00Z", "v": 14.37, "lo": 9.55, "hi": 19.25}, {"t": "2026-11-14T20:00:00Z", "v": 12.64, "lo": 7.81, "hi": 17.51}, {"t": "2026-11-14T21:00:00Z", "v": 14.52, "lo": 9.7, "hi": 19.4}, {"t": "2026-11-14T22:00:00Z", "v": 12.33, "lo": 7.5, "hi": 17.21}, {"t": "2026-11-14T23:00:00Z", "v": 9.87, "lo": 5.04, "hi": 14.74}, {"t": "2026-11-15T00:00:00Z", "v": 3.94, "lo": 0.0, "hi": 8.82}, {"t": "2026-11-15T01:00:00Z", "v": 1.34, "lo": 0.0, "hi": 6.22}, {"t": "2026-11-15T02:00:00Z", "v": 1.13, "lo": 0.0, "hi": 6.01}, {"t": "2026-11-15T03:00:00Z", "v": 0.99, "lo": 0.0, "hi": 5.86}, {"t": "2026-11-15T04:00:00Z", "v": 0.99, "lo": 0.0, "hi": 5.86}, {"t": "2026-11-15T05:00:00Z", "v": 1.01, "lo": 0.0, "hi": 5.88}, {"t": "2026-11-15T06:00:00Z", "v": 1.04, "lo": 0.0, "hi": 5.92}, {"t": "2026-11-15T07:00:00Z", "v": 1.55, "lo": 0.0, "hi": 6.43}, {"t": "2026-11-15T08:00:00Z", "v": 1.65, "lo": 0.0, "hi": 6.53}, {"t": "2026-11-15T09:00:00Z", "v": 5.61, "lo": 0.78, "hi": 10.48}, {"t": "2026-11-15T10:00:00Z", "v": 6.5, "lo": 1.68, "hi": 11.38}, {"t": "2026-11-15T11:00:00Z", "v": 7.51, "lo": 2.69, "hi": 12.39}, {"t": "2026-11-15T12:00:00Z", "v": 9.58, "lo": 4.76, "hi": 14.46}, {"t": "2026-11-15T13:00:00Z", "v": 11.7, "lo": 6.88, "hi": 16.58}, {"t": "2026-11-15T14:00:00Z", "v": 18.36, "lo": 13.54, "hi": 23.24}, {"t": "2026-11-15T15:00:00Z", "v": 17.33, "lo": 12.5, "hi": 22.21}, {"t": "2026-11-15T16:00:00Z", "v": 16.04, "lo": 11.21, "hi": 20.91}, {"t": "2026-11-15T17:00:00Z", "v": 11.34, "lo": 6.51, "hi": 16.21}, {"t": "2026-11-15T18:00:00Z", "v": 10.87, "lo": 6.05, "hi": 15.75}, {"t": "2026-11-15T19:00:00Z", "v": 8.45, "lo": 3.63, "hi": 13.33}, {"t": "2026-11-15T20:00:00Z", "v": 5.82, "lo": 0.99, "hi": 10.7}, {"t": "2026-11-15T21:00:00Z", "v": 3.71, "lo": 0.0, "hi": 8.59}, {"t": "2026-11-15T22:00:00Z", "v": 2.11, "lo": 0.0, "hi": 6.99}, {"t": "2026-11-15T23:00:00Z", "v": 1.47, "lo": 0.0, "hi": 6.34}, {"t": "2026-11-16T00:00:00Z", "v": 0.91, "lo": 0.0, "hi": 5.79}, {"t": "2026-11-16T01:00:00Z", "v": 0.88, "lo": 0.0, "hi": 5.76}, {"t": "2026-11-16T02:00:00Z", "v": 1.15, "lo": 0.0, "hi": 6.03}, {"t": "2026-11-16T03:00:00Z", "v": 1.15, "lo": 0.0, "hi": 6.03}, {"t": "2026-11-16T04:00:00Z", "v": 1.74, "lo": 0.0, "hi": 6.62}], "sunny-saturday-market": [{"t": "2026-11-14T05:00:00Z", "v": 1.17, "lo": 0.0, "hi": 6.04}, {"t": "2026-11-14T06:00:00Z", "v": 1.95, "lo": 0.0, "hi": 6.83}, {"t": "2026-11-14T07:00:00Z", "v": 2.94, "lo": 0.0, "hi": 7.81}, {"t": "2026-11-14T08:00:00Z", "v": 4.08, "lo": 0.0, "hi": 8.96}, {"t": "2026-11-14T09:00:00Z", "v": 7.63, "lo": 2.81, "hi": 12.51}, {"t": "2026-11-14T10:00:00Z", "v": 10.14, "lo": 5.32, "hi": 15.02}, {"t": "2026-11-14T11:00:00Z", "v": 11.16, "lo": 6.34, "hi": 16.04}, {"t": "2026-11-14T12:00:00Z", "v": 12.88, "lo": 8.05, "hi": 17.76}, {"t": "2026-11-14T13:00:00Z", "v": 12.02, "lo": 7.2, "hi": 16.9}, {"t": "2026-11-14T14:00:00Z", "v": 17.24, "lo": 12.42, "hi": 22.12}, {"t": "2026-11-14T15:00:00Z", "v": 15.8, "lo": 10.97, "hi": 20.67}, {"t": "2026-11-14T16:00:00Z", "v": 15.97, "lo": 11.15, "hi": 20.85}, {"t": "2026-11-14T17:00:00Z", "v": 15.77, "lo": 10.94, "hi": 20.65}, {"t": "2026-11-14T18:00:00Z", "v": 16.1, "lo": 11.27, "hi": 20.97}, {"t": "2026-11-14T19:00:00Z", "v": 17.49, "lo": 12.67, "hi": 22.37}, {"t": "2026-11-14T20:00:00Z", "v": 12.64, "lo": 7.81, "hi": 17.51}, {"t": "2026-11-14T21:00:00Z", "v": 14.52, "lo": 9.7, "hi": 19.4}, {"t": "2026-11-14T22:00:00Z", "v": 12.33, "lo": 7.5, "hi": 17.21}, {"t": "2026-11-14T23:00:00Z", "v": 9.87, "lo": 5.04, "hi": 14.74}, {"t": "2026-11-15T00:00:00Z", "v": 3.94, "lo": 0.0, "hi": 8.82}, {"t": "2026-11-15T01:00:00Z", "v": 1.34, "lo": 0.0, "hi": 6.22}, {"t": "2026-11-15T02:00:00Z", "v": 1.13, "lo": 0.0, "hi": 6.01}, {"t": "2026-11-15T03:00:00Z", "v": 0.99, "lo": 0.0, "hi": 5.86}, {"Line truncated
|
||||
{"data_version": "2026-11-07T23:51:00Z", "model_version": "histgb-2026-11-14", "context_version": "2026-11-14T03:47:09Z", "generated_at": "2026-11-14T05:27:22Z", "forecast_start": "2026-11-14T06:00:00Z", "forecasts": {"density-01": {"48h": {"base": [{"t": "2026-11-14T06:00:00Z", "v": 1.95, "lo": 0.0, "hi": 6.83}, {"t": "2026-11-14T07:00:00Z", "v": 2.94, "lo": 0.0, "hi": 7.81}, {"t": "2026-11-14T08:00:00Z", "v": 4.08, "lo": 0.0, "hi": 8.96}, {"t": "2026-11-14T09:00:00Z", "v": 5.68, "lo": 0.85, "hi": 10.55}, {"t": "2026-11-14T10:00:00Z", "v": 8.5, "lo": 3.67, "hi": 13.38}, {"t": "2026-11-14T11:00:00Z", "v": 9.66, "lo": 4.84, "hi": 14.54}, {"t": "2026-11-14T12:00:00Z", "v": 11.39, "lo": 6.56, "hi": 16.26}, {"t": "2026-11-14T13:00:00Z", "v": 11.32, "lo": 6.5, "hi": 16.2}, {"t": "2026-11-14T14:00:00Z", "v": 12.63, "lo": 7.81, "hi": 17.51}, {"t": "2026-11-14T15:00:00Z", "v": 14.78, "lo": 9.96, "hi": 19.66}, {"t": "2026-11-14T16:00:00Z", "v": 14.21, "lo": 9.39, "hi": 19.09}, {"t": "2026-11-14T17:00:00Z", "v": 14.52, "lo": 9.7, "hi": 19.4}, {"t": "2026-11-14T18:00:00Z", "v": 15.67, "lo": 10.84, "hi": 20.55}, {"t": "2026-11-14T19:00:00Z", "v": 14.37, "lo": 9.55, "hi": 19.25}, {"t": "2026-11-14T20:00:00Z", "v": 12.64, "lo": 7.81, "hi": 17.51}, {"t": "2026-11-14T21:00:00Z", "v": 14.52, "lo": 9.7, "hi": 19.4}, {"t": "2026-11-14T22:00:00Z", "v": 12.33, "lo": 7.5, "hi": 17.21}, {"t": "2026-11-14T23:00:00Z", "v": 9.87, "lo": 5.04, "hi": 14.74}, {"t": "2026-11-15T00:00:00Z", "v": 3.94, "lo": 0.0, "hi": 8.82}, {"t": "2026-11-15T01:00:00Z", "v": 1.34, "lo": 0.0, "hi": 6.22}, {"t": "2026-11-15T02:00:00Z", "v": 1.13, "lo": 0.0, "hi": 6.01}, {"t": "2026-11-15T03:00:00Z", "v": 0.99, "lo": 0.0, "hi": 5.86}, {"t": "2026-11-15T04:00:00Z", "v": 0.99, "lo": 0.0, "hi": 5.86}, {"t": "2026-11-15T05:00:00Z", "v": 1.01, "lo": 0.0, "hi": 5.88}, {"t": "2026-11-15T06:00:00Z", "v": 1.04, "lo": 0.0, "hi": 5.92}, {"t": "2026-11-15T07:00:00Z", "v": 1.55, "lo": 0.0, "hi": 6.43}, {"t": "2026-11-15T08:00:00Z", "v": 1.65, "lo": 0.0, "hi": 6.53}, {"t": "2026-11-15T09:00:00Z", "v": 5.61, "lo": 0.78, "hi": 10.48}, {"t": "2026-11-15T10:00:00Z", "v": 6.5, "lo": 1.68, "hi": 11.38}, {"t": "2026-11-15T11:00:00Z", "v": 7.51, "lo": 2.69, "hi": 12.39}, {"t": "2026-11-15T12:00:00Z", "v": 9.58, "lo": 4.76, "hi": 14.46}, {"t": "2026-11-15T13:00:00Z", "v": 11.7, "lo": 6.88, "hi": 16.58}, {"t": "2026-11-15T14:00:00Z", "v": 18.36, "lo": 13.54, "hi": 23.24}, {"t": "2026-11-15T15:00:00Z", "v": 17.33, "lo": 12.5, "hi": 22.21}, {"t": "2026-11-15T16:00:00Z", "v": 16.04, "lo": 11.21, "hi": 20.91}, {"t": "2026-11-15T17:00:00Z", "v": 11.34, "lo": 6.51, "hi": 16.21}, {"t": "2026-11-15T18:00:00Z", "v": 10.87, "lo": 6.05, "hi": 15.75}, {"t": "2026-11-15T19:00:00Z", "v": 8.45, "lo": 3.63, "hi": 13.33}, {"t": "2026-11-15T20:00:00Z", "v": 5.82, "lo": 0.99, "hi": 10.7}, {"t": "2026-11-15T21:00:00Z", "v": 3.71, "lo": 0.0, "hi": 8.59}, {"t": "2026-11-15T22:00:00Z", "v": 2.11, "lo": 0.0, "hi": 6.99}, {"t": "2026-11-15T23:00:00Z", "v": 1.47, "lo": 0.0, "hi": 6.34}, {"t": "2026-11-16T00:00:00Z", "v": 0.91, "lo": 0.0, "hi": 5.79}, {"t": "2026-11-16T01:00:00Z", "v": 0.88, "lo": 0.0, "hi": 5.76}, {"t": "2026-11-16T02:00:00Z", "v": 1.15, "lo": 0.0, "hi": 6.03}, {"t": "2026-11-16T03:00:00Z", "v": 1.15, "lo": 0.0, "hi": 6.03}, {"t": "2026-11-16T04:00:00Z", "v": 1.74, "lo": 0.0, "hi": 6.62}, {"t": "2026-11-16T05:00:00Z", "v": 1.58, "lo": 0.0, "hi": 6.46}], "sunny-saturday-market": [{"t": "2026-11-14T06:00:00Z", "v": 1.95, "lo": 0.0, "hi": 6.83}, {"t": "2026-11-14T07:00:00Z", "v": 2.94, "lo": 0.0, "hi": 7.81}, {"t": "2026-11-14T08:00:00Z", "v": 4.08, "lo": 0.0, "hi": 8.96}, {"t": "2026-11-14T09:00:00Z", "v": 7.63, "lo": 2.81, "hi": 12.51}, {"t": "2026-11-14T10:00:00Z", "v": 10.14, "lo": 5.32, "hi": 15.02}, {"t": "2026-11-14T11:00:00Z", "v": 11.16, "lo": 6.34, "hi": 16.04}, {"t": "2026-11-14T12:00:00Z", "v": 12.88, "lo": 8.05, "hi": 17.76}, {"t": "2026-11-14T13:00:00Z", "v": 12.02, "lo": 7.2, "hi": 16.9}, {"t": "2026-11-14T14:00:00Z", "v": 17.24, "lo": 12.42, "hi": 22.12}, {"t": "2026-11-14T15:00:00Z", "v": 15.8, "lo": 10.97, "hi": 20.67}, {"t": "2026-11-14T16:00:00Z", "v": 15.97, "lo": 11.15, "hi": 20.85}, {"t": "2026-11-14T17:00:00Z", "v": 15.77, "lo": 10.94, "hi": 20.65}, {"t": "2026-11-14T18:00:00Z", "v": 16.1, "lo": 11.27, "hi": 20.97}, {"t": "2026-11-14T19:00:00Z", "v": 17.49, "lo": 12.67, "hi": 22.37}, {"t": "2026-11-14T20:00:00Z", "v": 12.64, "lo": 7.81, "hi": 17.51}, {"t": "2026-11-14T21:00:00Z", "v": 14.52, "lo": 9.7, "hi": 19.4}, {"t": "2026-11-14T22:00:00Z", "v": 12.33, "lo": 7.5, "hi": 17.21}, {"t": "2026-11-14T23:00:00Z", "v": 9.87, "lo": 5.04, "hi": 14.74}, {"t": "2026-11-15T00:00:00Z", "v": 3.94, "lo": 0.0, "hi": 8.82}, {"t": "2026-11-15T01:00:00Z", "v": 1.34, "lo": 0.0, "hi": 6.22}, {"t": "2026-11-15T02:00:00Z", "v": 1.13, "lo": 0.0, "hi": 6.01}, {"t": "2026-11-15T03:00:00Z", "v": 0.99, "lo": 0.0, "hi": 5.86}, {"t": "2026-11-15T04:00:00Z", "v": 0.99, "lo": 0.0, "hi": 5.86}, {"Line truncated
|
||||
+207
-2
@@ -19,7 +19,7 @@
|
||||
"baseline_r2": 0.462,
|
||||
"n_eval": 5350
|
||||
},
|
||||
"trained_at": "2026-11-14T04:03:37Z",
|
||||
"trained_at": "2026-11-14T05:27:21Z",
|
||||
"train_from": "2026-06-12T00:00:00Z",
|
||||
"train_to": "2026-11-07T23:00:00Z",
|
||||
"test_from": "2026-10-10T23:00:00Z",
|
||||
@@ -90,11 +90,216 @@
|
||||
"n": 5350
|
||||
}
|
||||
],
|
||||
"per_sensor": [
|
||||
{
|
||||
"sensor": "density-01",
|
||||
"name": "Density 01",
|
||||
"n": 669,
|
||||
"mae": 2.26,
|
||||
"wape": 30.8,
|
||||
"r2": 0.722,
|
||||
"baseline_mae": 3.58,
|
||||
"baseline_wape": 48.8,
|
||||
"baseline_r2": 0.238,
|
||||
"mae_improvement_pct": 36.8
|
||||
},
|
||||
{
|
||||
"sensor": "hn-goetzentuem-01",
|
||||
"name": "Götzenturm",
|
||||
"n": 669,
|
||||
"mae": 2.4,
|
||||
"wape": 26.9,
|
||||
"r2": 0.718,
|
||||
"baseline_mae": 3.84,
|
||||
"baseline_wape": 43.0,
|
||||
"baseline_r2": 0.281,
|
||||
"mae_improvement_pct": 37.5
|
||||
},
|
||||
{
|
||||
"sensor": "hn-hagenbuchersee-01",
|
||||
"name": "Hagenbuchersee 1",
|
||||
"n": 669,
|
||||
"mae": 2.47,
|
||||
"wape": 38.6,
|
||||
"r2": 0.602,
|
||||
"baseline_mae": 4.01,
|
||||
"baseline_wape": 62.8,
|
||||
"baseline_r2": -0.212,
|
||||
"mae_improvement_pct": 38.5
|
||||
},
|
||||
{
|
||||
"sensor": "hn-hagenbuchersee-02",
|
||||
"name": "Hagenbuchersee 2",
|
||||
"n": 669,
|
||||
"mae": 4.12,
|
||||
"wape": 31.1,
|
||||
"r2": 0.731,
|
||||
"baseline_mae": 6.49,
|
||||
"baseline_wape": 48.9,
|
||||
"baseline_r2": 0.337,
|
||||
"mae_improvement_pct": 36.5
|
||||
},
|
||||
{
|
||||
"sensor": "hn-marahaus-01",
|
||||
"name": "Marahaus",
|
||||
"n": 669,
|
||||
"mae": 2.67,
|
||||
"wape": 29.8,
|
||||
"r2": 0.671,
|
||||
"baseline_mae": 4.57,
|
||||
"baseline_wape": 51.0,
|
||||
"baseline_r2": -0.045,
|
||||
"mae_improvement_pct": 41.7
|
||||
},
|
||||
{
|
||||
"sensor": "hn-neckarufer-01",
|
||||
"name": "Neckarufer",
|
||||
"n": 668,
|
||||
"mae": 3.63,
|
||||
"wape": 28.2,
|
||||
"r2": 0.695,
|
||||
"baseline_mae": 5.89,
|
||||
"baseline_wape": 45.8,
|
||||
"baseline_r2": 0.11,
|
||||
"mae_improvement_pct": 38.4
|
||||
},
|
||||
{
|
||||
"sensor": "hn-soleo-01",
|
||||
"name": "Soleo",
|
||||
"n": 669,
|
||||
"mae": 3.19,
|
||||
"wape": 20.2,
|
||||
"r2": 0.548,
|
||||
"baseline_mae": 4.4,
|
||||
"baseline_wape": 28.0,
|
||||
"baseline_r2": 0.163,
|
||||
"mae_improvement_pct": 27.7
|
||||
},
|
||||
{
|
||||
"sensor": "hn-stadtgalerie-01",
|
||||
"name": "Stadtgalerie",
|
||||
"n": 668,
|
||||
"mae": 5.83,
|
||||
"wape": 25.3,
|
||||
"r2": 0.765,
|
||||
"baseline_mae": 10.39,
|
||||
"baseline_wape": 45.1,
|
||||
"baseline_r2": 0.48,
|
||||
"mae_improvement_pct": 43.9
|
||||
}
|
||||
],
|
||||
"limits": [
|
||||
"estimated devices, not people",
|
||||
"8 sensors in a ~580 m inner-city cluster",
|
||||
"5 months of multi-sensor data; no annual seasonality claim",
|
||||
"observed weather in training, forecast weather in scenarios",
|
||||
"holidays are context, not a large accuracy claim"
|
||||
]
|
||||
],
|
||||
"small": {
|
||||
"model_version": "logit-20261114-052723",
|
||||
"trained_at": "2026-11-14T05:27:23Z",
|
||||
"features": [
|
||||
"hour",
|
||||
"weekday",
|
||||
"is_weekend",
|
||||
"temp_c",
|
||||
"precip_mm",
|
||||
"is_holiday"
|
||||
],
|
||||
"metrics": {
|
||||
"accuracy": 0.708,
|
||||
"precision": 0.36,
|
||||
"recall": 0.574,
|
||||
"f1": 0.443,
|
||||
"roc_auc": 0.726,
|
||||
"n_train": 18242,
|
||||
"n_test": 5710,
|
||||
"positive_rate": 0.202,
|
||||
"sensors": 8
|
||||
},
|
||||
"confusion_matrix": {
|
||||
"labels": {
|
||||
"0": "not busy",
|
||||
"1": "busy"
|
||||
},
|
||||
"tn": 3377,
|
||||
"fp": 1178,
|
||||
"fn": 492,
|
||||
"tp": 663
|
||||
},
|
||||
"per_sensor": [
|
||||
{
|
||||
"sensor": "density-01",
|
||||
"n": 714,
|
||||
"positive_rate": 0.111,
|
||||
"accuracy": 0.761,
|
||||
"precision": 0.3,
|
||||
"recall": 0.873,
|
||||
"f1": 0.447
|
||||
},
|
||||
{
|
||||
"sensor": "hn-goetzentuem-01",
|
||||
"n": 714,
|
||||
"positive_rate": 0.197,
|
||||
"accuracy": 0.732,
|
||||
"precision": 0.391,
|
||||
"recall": 0.638,
|
||||
"f1": 0.485
|
||||
},
|
||||
{
|
||||
"sensor": "hn-hagenbuchersee-01",
|
||||
"n": 714,
|
||||
"positive_rate": 0.049,
|
||||
"accuracy": 0.724,
|
||||
"precision": 0.148,
|
||||
"recall": 0.971,
|
||||
"f1": 0.257
|
||||
},
|
||||
{
|
||||
"sensor": "hn-hagenbuchersee-02",
|
||||
"n": 714,
|
||||
"positive_rate": 0.123,
|
||||
"accuracy": 0.783,
|
||||
"precision": 0.355,
|
||||
"recall": 0.932,
|
||||
"f1": 0.514
|
||||
},
|
||||
{
|
||||
"sensor": "hn-marahaus-01",
|
||||
"n": 714,
|
||||
"positive_rate": 0.055,
|
||||
"accuracy": 0.71,
|
||||
"precision": 0.135,
|
||||
"recall": 0.795,
|
||||
"f1": 0.23
|
||||
},
|
||||
{
|
||||
"sensor": "hn-neckarufer-01",
|
||||
"n": 713,
|
||||
"positive_rate": 0.09,
|
||||
"accuracy": 0.764,
|
||||
"precision": 0.274,
|
||||
"recall": 0.984,
|
||||
"f1": 0.429
|
||||
},
|
||||
{
|
||||
"sensor": "hn-soleo-01",
|
||||
"n": 714,
|
||||
"positive_rate": 0.704,
|
||||
"accuracy": 0.581,
|
||||
"precision": 0.943,
|
||||
"recall": 0.431,
|
||||
"f1": 0.592
|
||||
},
|
||||
{
|
||||
"sensor": "hn-stadtgalerie-01",
|
||||
"n": 713,
|
||||
"positive_rate": 0.289,
|
||||
"accuracy": 0.604,
|
||||
"precision": 0.335,
|
||||
"recall": 0.374,
|
||||
"f1": 0.353
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
Binary file not shown.
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"model_version": "logit-20261114-040339",
|
||||
"trained_at": "2026-11-14T04:03:39Z",
|
||||
"model_version": "logit-20261114-052723",
|
||||
"trained_at": "2026-11-14T05:27:23Z",
|
||||
"features": [
|
||||
"hour",
|
||||
"weekday",
|
||||
@@ -33,5 +33,89 @@
|
||||
"n_test": 5710,
|
||||
"positive_rate": 0.202,
|
||||
"sensors": 8
|
||||
}
|
||||
},
|
||||
"confusion_matrix": {
|
||||
"labels": {
|
||||
"0": "not busy",
|
||||
"1": "busy"
|
||||
},
|
||||
"tn": 3377,
|
||||
"fp": 1178,
|
||||
"fn": 492,
|
||||
"tp": 663
|
||||
},
|
||||
"per_sensor": [
|
||||
{
|
||||
"sensor": "density-01",
|
||||
"n": 714,
|
||||
"positive_rate": 0.111,
|
||||
"accuracy": 0.761,
|
||||
"precision": 0.3,
|
||||
"recall": 0.873,
|
||||
"f1": 0.447
|
||||
},
|
||||
{
|
||||
"sensor": "hn-goetzentuem-01",
|
||||
"n": 714,
|
||||
"positive_rate": 0.197,
|
||||
"accuracy": 0.732,
|
||||
"precision": 0.391,
|
||||
"recall": 0.638,
|
||||
"f1": 0.485
|
||||
},
|
||||
{
|
||||
"sensor": "hn-hagenbuchersee-01",
|
||||
"n": 714,
|
||||
"positive_rate": 0.049,
|
||||
"accuracy": 0.724,
|
||||
"precision": 0.148,
|
||||
"recall": 0.971,
|
||||
"f1": 0.257
|
||||
},
|
||||
{
|
||||
"sensor": "hn-hagenbuchersee-02",
|
||||
"n": 714,
|
||||
"positive_rate": 0.123,
|
||||
"accuracy": 0.783,
|
||||
"precision": 0.355,
|
||||
"recall": 0.932,
|
||||
"f1": 0.514
|
||||
},
|
||||
{
|
||||
"sensor": "hn-marahaus-01",
|
||||
"n": 714,
|
||||
"positive_rate": 0.055,
|
||||
"accuracy": 0.71,
|
||||
"precision": 0.135,
|
||||
"recall": 0.795,
|
||||
"f1": 0.23
|
||||
},
|
||||
{
|
||||
"sensor": "hn-neckarufer-01",
|
||||
"n": 713,
|
||||
"positive_rate": 0.09,
|
||||
"accuracy": 0.764,
|
||||
"precision": 0.274,
|
||||
"recall": 0.984,
|
||||
"f1": 0.429
|
||||
},
|
||||
{
|
||||
"sensor": "hn-soleo-01",
|
||||
"n": 714,
|
||||
"positive_rate": 0.704,
|
||||
"accuracy": 0.581,
|
||||
"precision": 0.943,
|
||||
"recall": 0.431,
|
||||
"f1": 0.592
|
||||
},
|
||||
{
|
||||
"sensor": "hn-stadtgalerie-01",
|
||||
"n": 713,
|
||||
"positive_rate": 0.289,
|
||||
"accuracy": 0.604,
|
||||
"precision": 0.335,
|
||||
"recall": 0.374,
|
||||
"f1": 0.353
|
||||
}
|
||||
]
|
||||
}
|
||||
Binary file not shown.
@@ -0,0 +1,36 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="960" height="370" viewBox="0 0 960 370" role="img" aria-label="Ablation: holdout MAE by feature set">
|
||||
<rect width="100%" height="100%" fill="#ffffff"/>
|
||||
<g font-family="-apple-system, Segoe UI, Helvetica, Arial, sans-serif">
|
||||
<text x="24" y="34" font-size="20" font-weight="600" fill="#0f172a">Ablation: holdout MAE by feature set</text>
|
||||
<text x="24" y="56" font-size="13" fill="#475569">Holdout 2026-10-10T23:00:00Z .. 2026-11-07T23:00:00Z, n=5350</text>
|
||||
<line x1="230.0" y1="68" x2="230.0" y2="314" stroke="#e2e8f0" stroke-width="1"/>
|
||||
<text x="230.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">0.0</text>
|
||||
<line x1="388.5" y1="68" x2="388.5" y2="314" stroke="#e2e8f0" stroke-width="1"/>
|
||||
<text x="388.5" y="62" font-size="11" fill="#64748b" text-anchor="middle">1.5</text>
|
||||
<line x1="547.0" y1="68" x2="547.0" y2="314" stroke="#e2e8f0" stroke-width="1"/>
|
||||
<text x="547.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">2.9</text>
|
||||
<line x1="705.5" y1="68" x2="705.5" y2="314" stroke="#e2e8f0" stroke-width="1"/>
|
||||
<text x="705.5" y="62" font-size="11" fill="#64748b" text-anchor="middle">4.4</text>
|
||||
<line x1="864.0" y1="68" x2="864.0" y2="314" stroke="#e2e8f0" stroke-width="1"/>
|
||||
<text x="864.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">5.8</text>
|
||||
<text x="218" y="106" font-size="12" fill="#0f172a" text-anchor="end">seasonal naive</text>
|
||||
<rect x="230" y="92" width="587.0" height="22" fill="#2563eb"/>
|
||||
<text x="823.0" y="107" font-size="12" fill="#0f172a">5.40</text>
|
||||
<text x="218" y="144" font-size="12" fill="#0f172a" text-anchor="end">day no weather</text>
|
||||
<rect x="230" y="130" width="430.5" height="22" fill="#2563eb"/>
|
||||
<text x="666.5" y="145" font-size="12" fill="#0f172a">3.96</text>
|
||||
<text x="218" y="182" font-size="12" fill="#0f172a" text-anchor="end">day holiday</text>
|
||||
<rect x="230" y="168" width="417.4" height="22" fill="#2563eb"/>
|
||||
<text x="653.4" y="183" font-size="12" fill="#0f172a">3.84</text>
|
||||
<text x="218" y="220" font-size="12" fill="#0f172a" text-anchor="end">day weather</text>
|
||||
<rect x="230" y="206" width="365.3" height="22" fill="#2563eb"/>
|
||||
<text x="601.3" y="221" font-size="12" fill="#0f172a">3.36</text>
|
||||
<text x="218" y="258" font-size="12" fill="#0f172a" text-anchor="end">week no weather</text>
|
||||
<rect x="230" y="244" width="443.5" height="22" fill="#2563eb"/>
|
||||
<text x="679.5" y="259" font-size="12" fill="#0f172a">4.08</text>
|
||||
<text x="218" y="296" font-size="12" fill="#0f172a" text-anchor="end">week weather</text>
|
||||
<rect x="230" y="282" width="360.9" height="22" fill="#0f766e"/>
|
||||
<text x="596.9" y="297" font-size="12" fill="#0f172a">3.32</text>
|
||||
<text x="24" y="358" font-size="11" fill="#64748b">Source: artifacts/metrics.json · PulseCast 28-day time-based holdout</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.7 KiB |
@@ -0,0 +1,60 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="960" height="446" viewBox="0 0 960 446" role="img" aria-label="Per-sensor holdout MAE: production model vs seasonal naive">
|
||||
<rect width="100%" height="100%" fill="#ffffff"/>
|
||||
<g font-family="-apple-system, Segoe UI, Helvetica, Arial, sans-serif">
|
||||
<text x="24" y="34" font-size="20" font-weight="600" fill="#0f172a">Per-sensor holdout MAE: production model vs seasonal naive</text>
|
||||
<text x="24" y="56" font-size="13" fill="#475569">Holdout 2026-10-10T23:00:00Z .. 2026-11-07T23:00:00Z, n=5350</text>
|
||||
<line x1="230.0" y1="68" x2="230.0" y2="390" stroke="#e2e8f0" stroke-width="1"/>
|
||||
<text x="230.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">0.0</text>
|
||||
<line x1="388.5" y1="68" x2="388.5" y2="390" stroke="#e2e8f0" stroke-width="1"/>
|
||||
<text x="388.5" y="62" font-size="11" fill="#64748b" text-anchor="middle">2.8</text>
|
||||
<line x1="547.0" y1="68" x2="547.0" y2="390" stroke="#e2e8f0" stroke-width="1"/>
|
||||
<text x="547.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">5.6</text>
|
||||
<line x1="705.5" y1="68" x2="705.5" y2="390" stroke="#e2e8f0" stroke-width="1"/>
|
||||
<text x="705.5" y="62" font-size="11" fill="#64748b" text-anchor="middle">8.4</text>
|
||||
<line x1="864.0" y1="68" x2="864.0" y2="390" stroke="#e2e8f0" stroke-width="1"/>
|
||||
<text x="864.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">11.2</text>
|
||||
<rect x="230" y="52" width="10" height="10" fill="#94a3b8"/><text x="246" y="61" font-size="12" fill="#475569">seasonal naive</text>
|
||||
<rect x="370" y="52" width="10" height="10" fill="#2563eb"/><text x="386" y="61" font-size="12" fill="#475569">weather-7d</text>
|
||||
<text x="218" y="106" font-size="12" fill="#0f172a" text-anchor="end">Density 01</text>
|
||||
<rect x="230" y="89" width="202.3" height="12" fill="#94a3b8"/>
|
||||
<text x="438.3" y="99" font-size="11" fill="#64748b">3.58</text>
|
||||
<rect x="230" y="104" width="127.7" height="12" fill="#2563eb"/>
|
||||
<text x="363.7" y="114" font-size="11" fill="#0f172a">2.26</text>
|
||||
<text x="218" y="144" font-size="12" fill="#0f172a" text-anchor="end">Götzenturm</text>
|
||||
<rect x="230" y="127" width="217.0" height="12" fill="#94a3b8"/>
|
||||
<text x="453.0" y="137" font-size="11" fill="#64748b">3.84</text>
|
||||
<rect x="230" y="142" width="135.6" height="12" fill="#2563eb"/>
|
||||
<text x="371.6" y="152" font-size="11" fill="#0f172a">2.40</text>
|
||||
<text x="218" y="182" font-size="12" fill="#0f172a" text-anchor="end">Hagenbuchersee 1</text>
|
||||
<rect x="230" y="165" width="226.6" height="12" fill="#94a3b8"/>
|
||||
<text x="462.6" y="175" font-size="11" fill="#64748b">4.01</text>
|
||||
<rect x="230" y="180" width="139.6" height="12" fill="#2563eb"/>
|
||||
<text x="375.6" y="190" font-size="11" fill="#0f172a">2.47</text>
|
||||
<text x="218" y="220" font-size="12" fill="#0f172a" text-anchor="end">Hagenbuchersee 2</text>
|
||||
<rect x="230" y="203" width="366.7" height="12" fill="#94a3b8"/>
|
||||
<text x="602.7" y="213" font-size="11" fill="#64748b">6.49</text>
|
||||
<rect x="230" y="218" width="232.8" height="12" fill="#2563eb"/>
|
||||
<text x="468.8" y="228" font-size="11" fill="#0f172a">4.12</text>
|
||||
<text x="218" y="258" font-size="12" fill="#0f172a" text-anchor="end">Marahaus</text>
|
||||
<rect x="230" y="241" width="258.2" height="12" fill="#94a3b8"/>
|
||||
<text x="494.2" y="251" font-size="11" fill="#64748b">4.57</text>
|
||||
<rect x="230" y="256" width="150.9" height="12" fill="#2563eb"/>
|
||||
<text x="386.9" y="266" font-size="11" fill="#0f172a">2.67</text>
|
||||
<text x="218" y="296" font-size="12" fill="#0f172a" text-anchor="end">Neckarufer</text>
|
||||
<rect x="230" y="279" width="332.8" height="12" fill="#94a3b8"/>
|
||||
<text x="568.8" y="289" font-size="11" fill="#64748b">5.89</text>
|
||||
<rect x="230" y="294" width="205.1" height="12" fill="#2563eb"/>
|
||||
<text x="441.1" y="304" font-size="11" fill="#0f172a">3.63</text>
|
||||
<text x="218" y="334" font-size="12" fill="#0f172a" text-anchor="end">Soleo</text>
|
||||
<rect x="230" y="317" width="248.6" height="12" fill="#94a3b8"/>
|
||||
<text x="484.6" y="327" font-size="11" fill="#64748b">4.40</text>
|
||||
<rect x="230" y="332" width="180.2" height="12" fill="#2563eb"/>
|
||||
<text x="416.2" y="342" font-size="11" fill="#0f172a">3.19</text>
|
||||
<text x="218" y="372" font-size="12" fill="#0f172a" text-anchor="end">Stadtgalerie</text>
|
||||
<rect x="230" y="355" width="587.0" height="12" fill="#94a3b8"/>
|
||||
<text x="823.0" y="365" font-size="11" fill="#64748b">10.39</text>
|
||||
<rect x="230" y="370" width="329.4" height="12" fill="#2563eb"/>
|
||||
<text x="565.4" y="380" font-size="11" fill="#0f172a">5.83</text>
|
||||
<text x="24" y="434" font-size="11" fill="#64748b">Source: artifacts/metrics.json · PulseCast 28-day time-based holdout</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 4.5 KiB |
@@ -0,0 +1,24 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="760" height="520" viewBox="0 0 760 520" role="img" aria-label="Small busy-hour classifier: holdout confusion matrix">
|
||||
<rect width="100%" height="100%" fill="#ffffff"/>
|
||||
<g font-family="-apple-system, Segoe UI, Helvetica, Arial, sans-serif">
|
||||
<text x="24" y="34" font-size="20" font-weight="600" fill="#0f172a">Small busy-hour classifier: holdout confusion matrix</text>
|
||||
<text x="24" y="56" font-size="13" fill="#475569">n=5,710 · accuracy 0.708 · precision 0.360 · recall 0.574 · F1 0.443 · ROC-AUC 0.726</text>
|
||||
<text x="295.0" y="114" font-size="13" fill="#475569" text-anchor="middle">Predicted not busy</text>
|
||||
<text x="485.0" y="114" font-size="13" fill="#475569" text-anchor="middle">Predicted busy</text>
|
||||
<text x="182" y="225.0" font-size="13" fill="#475569" text-anchor="end">Actually not busy</text>
|
||||
<text x="182" y="415.0" font-size="13" fill="#475569" text-anchor="end">Actually busy</text>
|
||||
<rect x="200" y="130" width="190" height="190" fill="#1d4ed8" stroke="#ffffff" stroke-width="2"/>
|
||||
<text x="295.0" y="217.0" font-size="30" font-weight="600" fill="#ffffff" text-anchor="middle">3,377</text>
|
||||
<text x="295.0" y="245.0" font-size="13" fill="#ffffff" text-anchor="middle">TN · 74.1% of row</text>
|
||||
<rect x="390" y="130" width="190" height="190" fill="#8aa7ee" stroke="#ffffff" stroke-width="2"/>
|
||||
<text x="485.0" y="217.0" font-size="30" font-weight="600" fill="#0f172a" text-anchor="middle">1,178</text>
|
||||
<text x="485.0" y="245.0" font-size="13" fill="#0f172a" text-anchor="middle">FP · 25.9% of row</text>
|
||||
<rect x="200" y="320" width="190" height="190" fill="#acc3f5" stroke="#ffffff" stroke-width="2"/>
|
||||
<text x="295.0" y="407.0" font-size="30" font-weight="600" fill="#0f172a" text-anchor="middle">492</text>
|
||||
<text x="295.0" y="435.0" font-size="13" fill="#0f172a" text-anchor="middle">FN · 42.6% of row</text>
|
||||
<rect x="390" y="320" width="190" height="190" fill="#a3bcf3" stroke="#ffffff" stroke-width="2"/>
|
||||
<text x="485.0" y="407.0" font-size="30" font-weight="600" fill="#0f172a" text-anchor="middle">663</text>
|
||||
<text x="485.0" y="435.0" font-size="13" fill="#0f172a" text-anchor="middle">TP · 57.4% of row</text>
|
||||
<text x="24" y="508" font-size="11" fill="#64748b">Source: artifacts/small_metrics.json · chronological 80/20 split</text>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.3 KiB |
+187
@@ -0,0 +1,187 @@
|
||||
# Metrics — paper support
|
||||
|
||||
All values in this file are generated by `make train` and stored in
|
||||
`artifacts/metrics.json` (forecast + ablation + per-sensor + small model) and
|
||||
`artifacts/small_metrics.json` (small-model detail). No number here is typed by
|
||||
hand: to refresh a table, rerun `make train && make figures` and copy the new
|
||||
JSON, or quote the JSON directly.
|
||||
|
||||
## Run provenance
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Model version | `histgb-2026-11-14` |
|
||||
| Trained at | 2026-11-14T05:27:21Z |
|
||||
| Data version | 2026-11-07T23:51:00Z |
|
||||
| Context (weather) version | 2026-11-14T03:47:09Z |
|
||||
| Train rows | 18,602 hourly sensor rows |
|
||||
| Holdout | 2026-10-10T23:00:00Z → 2026-11-07T23:00:00Z |
|
||||
| Evaluation rows | 5,350 (8 sensors, hours with a seasonal-naive baseline) |
|
||||
| Sensors | 8 (one merged `density-01`, seven `hn-*`) |
|
||||
| Raw input | 192,464 rows, 96,232 density rows, 96,232 battery rows |
|
||||
| Weather | Open-Meteo historical archive (training), forecast (scenarios) |
|
||||
| Calendar | Baden-Württemberg holidays 2025–2026 |
|
||||
|
||||
## Protocol
|
||||
|
||||
- One global `HistGradientBoostingRegressor` over all eight sensors. Target is
|
||||
the mean of 15-minute `densityData` readings in a UTC hour.
|
||||
- Time-based 28-day holdout measured back from the last observed hour. No
|
||||
shuffling, no random split.
|
||||
- Seasonal naive baseline: the exact 168-row lag (same observed hour one week
|
||||
earlier), evaluated on the same rows.
|
||||
- The production bundle is refit on all observed rows **after** the holdout is
|
||||
evaluated; the holdout is never in the production fit.
|
||||
- WAPE = `sum(|error|) / sum(|actual|)`. R² is computed on the same rows.
|
||||
- The production 7-day model uses only features at least 168 hours old, so a
|
||||
168-hour forecast needs no recursion.
|
||||
- Weather in the holdout is observed, not forecast. Live accuracy will be lower
|
||||
than the table because the dashboard uses forecast weather.
|
||||
|
||||
## Headline
|
||||
|
||||
| Model | MAE | WAPE | R² |
|
||||
|---|---:|---:|---:|
|
||||
| Seasonal naive (exact 168-row lag) | 5.40 | 44.7% | 0.462 |
|
||||
| **Weather-aware 7-day (production)** | **3.32** | **27.5%** | **0.777** |
|
||||
|
||||
- MAE improvement over seasonal naive: **38.5%**.
|
||||
- WAPE improvement: 17.2 percentage points.
|
||||
- The mission reference numbers for seasonal naive (5.40 / 44.7% / 0.462) are
|
||||
reproduced exactly.
|
||||
- The production row is slightly better than the reference 7-day model
|
||||
(4.37 / 36.2% / 0.672) because `lag_week` uses the previous *observed* same
|
||||
weekday/hour across sensor gaps instead of recursion.
|
||||
|
||||
## Ablation
|
||||
|
||||
| Model | Description | MAE | WAPE | R² | n |
|
||||
|---|---|---|---:|---:|---:|---:|
|
||||
| `seasonal_naive` | exact 168-row lag (same observed hour last week) | 5.40 | 44.7% | 0.462 | 5,350 |
|
||||
| `day_no_weather` | calendar + lags 24/48/168 + 24h/168h rolling means | 3.96 | 32.8% | 0.706 | 5,350 |
|
||||
| `day_holiday` | day-ahead + Baden-Württemberg holiday flag | 3.84 | 31.8% | 0.732 | 5,350 |
|
||||
| `day_weather` | day-ahead + temperature + rain (observed weather) | 3.36 | 27.8% | 0.763 | 5,350 |
|
||||
| `week_no_weather` | 7-day-safe calendar + previous same weekday/hour values | 4.08 | 33.8% | 0.697 | 5,350 |
|
||||
| **`week_weather`** | **7-day-safe + temperature + rain (production)** | **3.32** | **27.5%** | **0.777** | **5,350** |
|
||||
|
||||
Derived effects:
|
||||
|
||||
- Weather at day-ahead: 3.96 → 3.36 MAE, **−15.2%**.
|
||||
- Weather at 7-day-safe: 4.08 → 3.32 MAE, **−18.6%**.
|
||||
- Holiday flag at day-ahead: 3.96 → 3.84 MAE, **−0.12 MAE**. Holidays are
|
||||
context, not a headline accuracy claim.
|
||||
- The mission reference day-ahead weather row (3.36 / 27.8%) is reproduced.
|
||||
|
||||

|
||||
|
||||
*Figure 1. Holdout MAE for each feature set; teal is the production model.
|
||||
Weather is the largest single gain; the holiday flag is a small addition.*
|
||||
|
||||
## Per sensor
|
||||
|
||||
Production model and seasonal naive on the same holdout rows.
|
||||
|
||||
| Sensor | Name | n | MAE | WAPE | R² | Baseline MAE | Baseline WAPE | Baseline R² | MAE improvement |
|
||||
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| `density-01` | Density 01 | 669 | 2.26 | 30.8% | 0.722 | 3.58 | 48.8% | 0.238 | 36.8% |
|
||||
| `hn-goetzentuem-01` | Götzenturm | 669 | 2.40 | 26.9% | 0.718 | 3.84 | 43.0% | 0.281 | 37.5% |
|
||||
| `hn-hagenbuchersee-01` | Hagenbuchersee 1 | 669 | 2.47 | 38.6% | 0.602 | 4.01 | 62.8% | −0.212 | 38.5% |
|
||||
| `hn-hagenbuchersee-02` | Hagenbuchersee 2 | 669 | 4.12 | 31.1% | 0.731 | 6.49 | 48.9% | 0.337 | 36.5% |
|
||||
| `hn-marahaus-01` | Marahaus | 669 | 2.67 | 29.8% | 0.671 | 4.57 | 51.0% | −0.045 | 41.7% |
|
||||
| `hn-neckarufer-01` | Neckarufer | 668 | 3.63 | 28.2% | 0.695 | 5.89 | 45.8% | 0.110 | 38.4% |
|
||||
| `hn-soleo-01` | Soleo | 669 | 3.19 | 20.2% | 0.548 | 4.40 | 28.0% | 0.163 | 27.7% |
|
||||
| `hn-stadtgalerie-01` | Stadtgalerie | 668 | 5.83 | 25.3% | 0.765 | 10.39 | 45.1% | 0.480 | 43.9% |
|
||||
|
||||
- Every sensor improves over its own seasonal naive baseline (27.7%–43.9%).
|
||||
- Stadtgalerie has the largest absolute error (MAE 5.83) and the largest
|
||||
volume; Soleo has the smallest relative gain and the lowest R² (0.548).
|
||||
- Two baselines have negative R² (`hagenbuchersee-01`, `marahaus-01`), which is
|
||||
expected for a naive lag on a short, variable holdout; the model is positive
|
||||
on all eight.
|
||||
- Sensor totals: 669 × 6 + 668 × 2 = 5,350, matching the headline.
|
||||
|
||||

|
||||
|
||||
*Figure 2. Per-sensor holdout MAE, production model (blue) vs seasonal naive
|
||||
(grey). Every sensor improves; the model is not carried by one large sensor.*
|
||||
|
||||
## Uncertainty band
|
||||
|
||||
The `lo`/`hi` band is the holdout residual p10/p90 added to the point forecast,
|
||||
clipped at zero:
|
||||
|
||||
| p10 | p50 | p90 |
|
||||
|---:|---:|---:|
|
||||
| −4.83 | −0.59 | +4.88 |
|
||||
|
||||
This is an empirical 80% band, not a calibrated guarantee, and the median
|
||||
residual being −0.59 shows a small optimistic bias on the holdout.
|
||||
|
||||
## Small busy-hour classifier
|
||||
|
||||
Full definition and limitations: [`docs/small-model.md`](small-model.md).
|
||||
Summary from the same baked run (`logit-20261114-052723`):
|
||||
|
||||
| Accuracy | Precision | Recall | F1 | ROC-AUC | Train rows | Test rows | Base rate |
|
||||
|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| 0.708 | 0.360 | 0.574 | 0.443 | 0.726 | 18,242 | 5,710 | 0.202 |
|
||||
|
||||
| | Predicted not busy | Predicted busy |
|
||||
|---|---:|---:|
|
||||
| **Actually not busy** | 3,377 (TN) | 1,178 (FP) |
|
||||
| **Actually busy** | 492 (FN) | 663 (TP) |
|
||||
|
||||

|
||||
|
||||
*Figure 3. Holdout confusion matrix for the small busy-hour classifier. The
|
||||
balanced weights trade precision for recall; accuracy 0.708 is below the 0.798
|
||||
majority-class baseline, so report recall/F1 and ROC-AUC instead.*
|
||||
|
||||
Per-sensor small-model holdout:
|
||||
|
||||
| Sensor | n | Positive rate | Accuracy | Precision | Recall | F1 |
|
||||
|---|---:|---:|---:|---:|---:|---:|
|
||||
| `density-01` | 714 | 0.111 | 0.761 | 0.300 | 0.873 | 0.447 |
|
||||
| `hn-goetzentuem-01` | 714 | 0.197 | 0.732 | 0.391 | 0.638 | 0.485 |
|
||||
| `hn-hagenbuchersee-01` | 714 | 0.049 | 0.724 | 0.148 | 0.971 | 0.257 |
|
||||
| `hn-hagenbuchersee-02` | 714 | 0.123 | 0.783 | 0.355 | 0.932 | 0.514 |
|
||||
| `hn-marahaus-01` | 714 | 0.055 | 0.710 | 0.135 | 0.795 | 0.230 |
|
||||
| `hn-neckarufer-01` | 713 | 0.090 | 0.764 | 0.274 | 0.984 | 0.429 |
|
||||
| `hn-soleo-01` | 714 | 0.704 | 0.581 | 0.943 | 0.431 | 0.592 |
|
||||
| `hn-stadtgalerie-01` | 713 | 0.289 | 0.604 | 0.335 | 0.374 | 0.353 |
|
||||
|
||||
## Claims the data supports
|
||||
|
||||
- A global, weather-aware HistGB forecasts hourly footfall for all eight
|
||||
inner-city sensors with MAE 3.32 / WAPE 27.5% / R² 0.777 on a 28-day
|
||||
time-based holdout.
|
||||
- Weather features are the main measurable gain: −18.6% MAE versus the 7-day
|
||||
no-weather model, −38.5% versus seasonal naive.
|
||||
- Every sensor improves over its own seasonal naive baseline (27.7%–43.9%).
|
||||
- A tiny logistic model can flag busy hours with ROC-AUC 0.726 and 57.4%
|
||||
recall at 36.0% precision, retrained from a CSV in under 5 seconds.
|
||||
|
||||
## Claims the data does not support
|
||||
|
||||
- People counts. Values are an index of estimated devices; penetration is
|
||||
unknown.
|
||||
- Annual seasonality, city-wide surfaces, or store-location recommendations
|
||||
from five months at eight sensors in ~580 m.
|
||||
- Causal weather effects. The ablation is predictive, not a treatment effect.
|
||||
- Live accuracy equal to the holdout. The holdout uses observed weather; the
|
||||
demo uses forecast weather and `predict` adds scenario overlays.
|
||||
- Holidays as a large driver. The flag moves day-ahead MAE by only −0.12.
|
||||
- The small classifier as the production forecast. It is a demo artifact with
|
||||
six features and a per-window p75 target.
|
||||
- Tuned hyperparameters. The model configuration is fixed; the 28-day holdout
|
||||
is the honest signal, not a search result.
|
||||
|
||||
## Reproduce
|
||||
|
||||
```bash
|
||||
make data # parse + weather + holidays + JSON artifacts
|
||||
make train # holdout, ablation, per-sensor, small model, forecasts
|
||||
make small # print the small-model metrics and confusion matrix
|
||||
make figures # regenerate docs/figures/*.svg from artifacts/metrics.json
|
||||
make check # sync --frozen, ruff, compileall, pytest
|
||||
```
|
||||
@@ -0,0 +1,187 @@
|
||||
# Small model — busy-hour classifier
|
||||
|
||||
This is the small, fast, explainable model that powers the "train new data"
|
||||
demo and the "I coded an AI" artifact. It is deliberately separate from the
|
||||
production 7-day forecaster: six cheap features, a linear model, a time-based
|
||||
split, and under a second of fitting. It is not the model behind the dashboard
|
||||
forecast.
|
||||
|
||||
Every number below comes from the baked artifact
|
||||
`artifacts/small_metrics.json` (version `logit-20261114-052723`, trained
|
||||
2026-11-14T05:27:23Z) and can be reproduced with `make small`.
|
||||
|
||||
## Target
|
||||
|
||||
`busy` is a binary label for one sensor-hour:
|
||||
|
||||
```text
|
||||
busy = 1 if hourly_mean_density > that sensor's 75th percentile
|
||||
busy = 0 otherwise
|
||||
```
|
||||
|
||||
- The p75 is computed **per sensor on the training window only**, then applied
|
||||
unchanged to the holdout. Computing it on the full dataset would leak the
|
||||
holdout distribution into the label.
|
||||
- A p75 threshold makes "busy" mean "busier than three of four hours at this
|
||||
location", not an absolute footfall claim. The same absolute value can be
|
||||
busy at Soleo and quiet at Stadtgalerie.
|
||||
- Hours above the threshold are exactly 25% of the training hours by
|
||||
construction, before missing hours are dropped.
|
||||
|
||||
| Sensor | Training-window p75 |
|
||||
|---|---:|
|
||||
| `density-01` | 15.23 |
|
||||
| `hn-goetzentuem-01` | 15.40 |
|
||||
| `hn-hagenbuchersee-01` | 18.20 |
|
||||
| `hn-hagenbuchersee-02` | 29.17 |
|
||||
| `hn-marahaus-01` | 22.23 |
|
||||
| `hn-neckarufer-01` | 28.35 |
|
||||
| `hn-soleo-01` | 11.38 |
|
||||
| `hn-stadtgalerie-01` | 38.41 |
|
||||
|
||||
## Data and split
|
||||
|
||||
- Input: the challenge CSV, `densityData` rows only, aggregated to the mean of
|
||||
the 15-minute readings in each UTC hour.
|
||||
- Window: 2026-06-12T00:00:00Z → 2026-11-07T23:00:00Z.
|
||||
- 23,952 hourly sensor rows across all 8 usable sensors.
|
||||
- Split is chronological, not random: the first 80% of **timestamps** train,
|
||||
the last 20% test.
|
||||
- Train: 2026-06-12T00:00:00Z → 2026-10-09T01:00:00Z, 18,242 rows.
|
||||
- Test: 2026-10-09T02:00:00Z → 2026-11-07T23:00:00Z, 5,710 rows.
|
||||
- Excluded devices: `maker-fair`, `niklas-01`, `hn-bollwerksturm-01` (test
|
||||
installations / insufficient coverage). See `artifacts/data_audit.json`.
|
||||
|
||||
## Features
|
||||
|
||||
| Feature | Source | Note |
|
||||
|---|---|---|
|
||||
| `hour` | local Europe/Berlin hour | 0–23, captures the daily curve |
|
||||
| `weekday` | local Europe/Berlin weekday | 0 = Monday |
|
||||
| `is_weekend` | weekday ≥ 5 | |
|
||||
| `is_holiday` | Baden-Württemberg calendar | context, not a headline claim |
|
||||
| `temp_c` | observed Open-Meteo archive | joined to the nearest hour (±3 h) |
|
||||
| `precip_mm` | observed Open-Meteo archive | same join |
|
||||
|
||||
No lags, no target history, no sensor identifier. That is intentional: the
|
||||
classifier must retrain from a small user upload that may not contain a long
|
||||
history, and it must not depend on future rows. Sensor identity is still
|
||||
implicit in the per-sensor threshold.
|
||||
|
||||
## Model
|
||||
|
||||
```text
|
||||
StandardScaler -> LogisticRegression(
|
||||
C=1.0, max_iter=1000, class_weight="balanced",
|
||||
solver="lbfgs", random_state=42)
|
||||
```
|
||||
|
||||
`class_weight="balanced"` compensates for the 25/75 class split caused by the
|
||||
p75 threshold. It buys recall at the cost of precision, which is the right
|
||||
trade for a "flag the busy hours" demo.
|
||||
|
||||
Fit time: **0.30 s** on the full challenge CSV in this run (contract: < 5 s,
|
||||
including the API job's CSV parsing).
|
||||
|
||||
## Metrics
|
||||
|
||||
Time-based holdout, 5,710 rows, base rate 20.2% positive.
|
||||
|
||||
| Accuracy | Precision | Recall | F1 | ROC-AUC |
|
||||
|---:|---:|---:|---:|---:|
|
||||
| 0.708 | 0.360 | 0.574 | 0.443 | 0.726 |
|
||||
|
||||
Confusion matrix:
|
||||
|
||||
| | Predicted not busy | Predicted busy |
|
||||
|---|---:|---:|
|
||||
| **Actually not busy** | 3,377 (TN) | 1,178 (FP) |
|
||||
| **Actually busy** | 492 (FN) | 663 (TP) |
|
||||
|
||||
Derived:
|
||||
|
||||
- Specificity (true-negative rate): 0.741
|
||||
- Negative predictive value: 0.873
|
||||
- False-positive rate: 0.259
|
||||
- False-negative rate: 0.426
|
||||
- Precision lift over the 20.2% base rate: 1.78×
|
||||
- Accuracy of the majority-class baseline ("always not busy"): 0.798
|
||||
|
||||
The model is **less accurate than always saying "not busy"**, but it finds 663
|
||||
of 1,155 busy hours (57.4% recall) that the majority baseline never finds. The
|
||||
balanced weights push it toward over-flagging: 1,178 false positives for 1,155
|
||||
actual busy hours. That is why precision is 0.36 and accuracy is not the right
|
||||
headline.
|
||||
|
||||
Per-sensor holdout:
|
||||
|
||||
| Sensor | n | Pos. rate | Accuracy | Precision | Recall | F1 |
|
||||
|---|---:|---:|---:|---:|---:|---:|
|
||||
| `density-01` | 714 | 0.111 | 0.761 | 0.300 | 0.873 | 0.447 |
|
||||
| `hn-goetzentuem-01` | 714 | 0.197 | 0.732 | 0.391 | 0.638 | 0.485 |
|
||||
| `hn-hagenbuchersee-01` | 714 | 0.049 | 0.724 | 0.148 | 0.971 | 0.257 |
|
||||
| `hn-hagenbuchersee-02` | 714 | 0.123 | 0.783 | 0.355 | 0.932 | 0.514 |
|
||||
| `hn-marahaus-01` | 714 | 0.055 | 0.710 | 0.135 | 0.795 | 0.230 |
|
||||
| `hn-neckarufer-01` | 713 | 0.090 | 0.764 | 0.274 | 0.984 | 0.429 |
|
||||
| `hn-soleo-01` | 714 | 0.704 | 0.581 | 0.943 | 0.431 | 0.592 |
|
||||
| `hn-stadtgalerie-01` | 713 | 0.289 | 0.604 | 0.335 | 0.374 | 0.353 |
|
||||
|
||||
The pooled model has the lowest accuracy on `hn-soleo-01` (0.581, an unusually
|
||||
high base rate of 70.4%) and the lowest recall on `hn-stadtgalerie-01` (0.374).
|
||||
The high recall on low-base-rate sensors is mostly the balanced weights firing;
|
||||
it is not evidence of a strong signal there.
|
||||
|
||||
## Reproduce
|
||||
|
||||
```bash
|
||||
# Full challenge CSV, prints every number above.
|
||||
make small
|
||||
|
||||
# Explicit CSV, all rows, save the baked artifact.
|
||||
uv run python scripts/train_small.py --csv new-data.csv --all --save
|
||||
|
||||
# Machine-readable payload.
|
||||
uv run python scripts/train_small.py --json
|
||||
```
|
||||
|
||||
The runtime path is the same function (`pulsecast.small.train_small_from_csv`):
|
||||
|
||||
```bash
|
||||
curl -s -X POST localhost:18000/train \
|
||||
-H "Authorization: Bearer $CONTROL_TOKEN" \
|
||||
-F model=small -F file=@new-data.csv
|
||||
curl -s localhost:18000/train/<job_id> \
|
||||
-H "Authorization: Bearer $CONTROL_TOKEN"
|
||||
```
|
||||
|
||||
The job returns `accuracy`, `precision`, `recall`, `f1`, `roc_auc`, `n_train`,
|
||||
`n_test`, `positive_rate`, and `sensors`; the full confusion matrix and
|
||||
per-sensor table are stored in `artifacts/small_metrics.json`. On success the
|
||||
bundle is hot-swapped and persisted to `PULSECAST_MODELS/small_latest.joblib`.
|
||||
A restart falls back to the baked artifact.
|
||||
|
||||
## Honest limits
|
||||
|
||||
- **It is a demo, not the production forecast.** The dashboard uses the global
|
||||
HistGB 7-day model, not this classifier.
|
||||
- **Threshold convention.** "Busy" is the sensor's own p75, recomputed from the
|
||||
upload's training window. A different upload can move the thresholds and
|
||||
change every metric. Do not compare runs trained on different windows.
|
||||
- **No lags or sensor id.** The model sees only calendar and weather. It cannot
|
||||
know that the previous hour was packed, and it pools all locations while
|
||||
evaluating per sensor.
|
||||
- **Weather fallback.** If an uploaded CSV does not overlap the baked weather
|
||||
archive, missing weather is filled with 14 °C / 0 mm. That keeps the job
|
||||
running but weakens the weather features; the returned metrics show it.
|
||||
- **One chronological split.** Five months, one holdout. No cross-validation,
|
||||
no annual seasonality claim, no per-weekday calibration.
|
||||
- **Threshold metrics only.** This page reports hard 0.5-threshold metrics, not
|
||||
a calibrated probability. The probability output is not a business forecast.
|
||||
- **Counters are devices.** Footfall density is an index of estimated devices,
|
||||
not a count of people. A "busy" flag inherits that uncertainty.
|
||||
- **Small uploads are unstable.** A CSV with one sensor, only one class, or
|
||||
fewer than four hourly timestamps fails with a clear error. A CSV with a
|
||||
short window may train but produce meaningless metrics.
|
||||
- **No causal claims.** Weather and holiday coefficients are associations in
|
||||
this sample, not causal effects, and the classifier is not used for scenario
|
||||
uplift (that overlay is explicit and separate).
|
||||
@@ -0,0 +1,118 @@
|
||||
# Training end-to-end evidence
|
||||
|
||||
Reproduce with:
|
||||
|
||||
```bash
|
||||
uv run python scripts/e2e_train.py --port 18000 --report /tmp/e2e.json
|
||||
```
|
||||
|
||||
The script starts a real `uvicorn pulsecast.server:app` process with
|
||||
`CONTROL_TOKEN`, waits for startup, and drives the HTTP API only. It exits
|
||||
non-zero on the first failed check. Unit/contract tests (`make check`) are
|
||||
separate.
|
||||
|
||||
## Run
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Date | 2026-11-14, 05:33 UTC |
|
||||
| Branch | `feat/ml-final` |
|
||||
| Port | **18000** |
|
||||
| Checks | **38 passed, 0 failed** |
|
||||
| Total wall time | 10.4 s |
|
||||
| Forecast version before | `histgb-2026-11-14` |
|
||||
| Data version | `2026-11-07T23:51:00Z` |
|
||||
| Context version | `2026-11-14T03:47:09Z` |
|
||||
|
||||
## Snapshot and robustness (20 checks)
|
||||
|
||||
- `/healthz` → `{"ok":true}`; `/readyz` → 200 `model=true`.
|
||||
- `/meta` covers 8 sensors and the baked model beats the seasonal-naive
|
||||
baseline on the holdout.
|
||||
- `/bootstrap` → 8 sensors, all 8 hourly series (23,952 points), 168 forecast
|
||||
weather hours, 24 holidays, versions equal to `/meta`.
|
||||
- `/predict`: **128/128** combinations (8 sensors × 4 horizons × 4 scenarios)
|
||||
returned the contract point counts (`15m` 1, `1h` 4, `48h` 48, `7d` 168),
|
||||
monotonic UTC timestamps, non-negative values, and `lo ≤ v ≤ hi`.
|
||||
- `/simulate`: all 4 scenarios × 8 sensors × 4 horizons returned valid points.
|
||||
- Weather override changes the forecast (`-8 °C / 6 mm` vs `30 °C / 0 mm`).
|
||||
- Unknown sensor → 404; missing token → 401 `unauthorized`; wrong token → 401;
|
||||
unknown job → 404 `not_found`; invalid model → 400 `bad_request`; missing
|
||||
JSON path → 400.
|
||||
|
||||
## Small training job (real 60-day CSV subset)
|
||||
|
||||
Input: 36,846 density rows, 8 sensors, 2026-09-08T23:53Z → 2026-11-07T23:51Z.
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Job ID | `0b35347963974cbbbe584b77ecd4b846` |
|
||||
| Status | `succeeded` |
|
||||
| Model version | `logit-20261114-053344` |
|
||||
| Wall time (submit → terminal) | **0.172 s** (contract < 5 s) |
|
||||
| n_train / n_test | 8,151 / 2,301 |
|
||||
| Base rate | 0.180 |
|
||||
| Accuracy | 0.824 |
|
||||
| Precision | 0.542 |
|
||||
| Recall | 0.157 |
|
||||
| F1 | 0.243 |
|
||||
| ROC-AUC | 0.813 |
|
||||
|
||||
This is a different window from the baked full-window model in
|
||||
`docs/small-model.md`, so the numbers are not directly comparable: the p75
|
||||
target and the 80/20 split are recomputed inside the 60-day upload. The
|
||||
low recall illustrates the honest limit noted in the small-model doc — a short,
|
||||
recent window with a different threshold is a different classification problem.
|
||||
|
||||
Post-conditions verified:
|
||||
|
||||
- `small_latest.joblib` and `<job_id>-small.joblib` exist under
|
||||
`PULSECAST_MODELS`.
|
||||
- `/meta.small_model.model_version` equals the job version and its metrics
|
||||
match the job exactly (in-memory hot-swap).
|
||||
- `/predict` still returns 168 valid points after the swap.
|
||||
|
||||
## Forecast training job (full challenge CSV) and 409
|
||||
|
||||
Input: full `schwarz-pax.csv`, 19,383,568 bytes.
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Job ID | `235209a0d00f449ba8be717eb76386d6` |
|
||||
| Status | `succeeded` |
|
||||
| Model version | `histgb-20261114-053349` |
|
||||
| Wall time | **4.380 s** |
|
||||
| MAE | 3.32 |
|
||||
| WAPE | 27.5% |
|
||||
| R² | 0.777 |
|
||||
| Baseline MAE / WAPE / R² | 5.40 / 44.7% / 0.462 |
|
||||
| n_eval | 5,350 |
|
||||
|
||||
The ablation in the job payload matches `artifacts/metrics.json` row for row
|
||||
(3.96, 3.84, 3.36, 4.08, 3.32). While this job was in `running`, a second
|
||||
`POST /train` returned **409** `conflict`, proving one job at a time.
|
||||
|
||||
Post-conditions verified:
|
||||
|
||||
- `forecast_latest.joblib` and `<job_id>-forecast.joblib` exist under
|
||||
`PULSECAST_MODELS`.
|
||||
- `/meta.model_version` equals the retrained version and
|
||||
`/meta.model_source` is `retrained`.
|
||||
- `/predict` reports the retrained version; `/readyz` stays 200.
|
||||
- `data_version` after the retrain is `2026-11-07T23:00:00Z` (the hourly input
|
||||
maximum) rather than the raw-file floor `23:51:00Z`; both describe the same
|
||||
observed window.
|
||||
|
||||
## Failure isolation
|
||||
|
||||
A malformed CSV (`not,a,pulsecast,file`) was accepted as a job (202), then
|
||||
failed with a clear error:
|
||||
|
||||
```text
|
||||
CSV needs a time column (time, timestamp, datetime, date, hour) and a value
|
||||
column (value, density, footfall, count, pax, visitors); got ['not', 'a',
|
||||
'pulsecast', 'file']
|
||||
```
|
||||
|
||||
`/healthz` still returned 200 afterward: a bad upload cannot take down the
|
||||
service or block `/predict`.
|
||||
+58
-14
@@ -33,7 +33,11 @@ Day-ahead ablation models additionally use `lag_day`, `lag_2d`, `roll_24`,
|
||||
## Target and data
|
||||
|
||||
- Target: mean of 15-minute `densityData` readings in a UTC hour, per sensor.
|
||||
- Training window: 2026-06-12 → 2026-10-10; holdout: 2026-10-10 → 2026-11-07.
|
||||
- Training rows: 2026-06-12T00:00:00Z → 2026-10-10T22:00:00Z, 18,602 rows.
|
||||
- Holdout: 2026-10-10T23:00:00Z → 2026-11-07T23:00:00Z, 5,350
|
||||
evaluation rows (hours with a seasonal-naive baseline).
|
||||
- The production bundle is refit on all observed rows (through
|
||||
2026-11-07T23:00:00Z) only after the holdout is evaluated.
|
||||
- Eight usable sensors after excluding `maker-fair` (73 rows, Hannover),
|
||||
`niklas-01` (8 rows), `hn-bollwerksturm-01` (5 rows).
|
||||
- `density-01` merges two physical devices (`863f75b0`, `f8c19730`) into one
|
||||
@@ -57,8 +61,8 @@ Time-based 28-day holdout, no shuffling. Seasonal naive is the baseline. WAPE is
|
||||
| 7-day-safe, no weather | 4.08 | 33.8% | 0.697 |
|
||||
| **7-day-safe, + temperature/rain (production)** | **3.32** | **27.5%** | **0.777** |
|
||||
|
||||
Weather is the useful signal: adding temperature and rain cuts MAE by ~19%
|
||||
versus the 7-day no-weather model and ~38% versus seasonal naive. The holiday
|
||||
Weather is the useful signal: adding temperature and rain cuts MAE by 18.6%
|
||||
versus the 7-day no-weather model and 38.5% versus seasonal naive. The holiday
|
||||
flag alone changes day-ahead MAE by −0.12 in this run; holidays are context, not
|
||||
a headline accuracy claim.
|
||||
|
||||
@@ -69,23 +73,63 @@ the previous *observed* same weekday/hour across sensor gaps, while the baseline
|
||||
uses the exact 168-row lag. R² differs slightly from the reference 0.779
|
||||
(0.777 here) because of that feature choice.
|
||||
|
||||
Per-sensor holdout MAE improves over each sensor's own seasonal naive by
|
||||
27.7%–43.9%; the full table, figures, and provenance are in
|
||||
[`docs/metrics.md`](docs/metrics.md). The largest absolute error is
|
||||
Stadtgalerie (MAE 5.83, its baseline 10.39); Soleo has the smallest relative
|
||||
gain (27.7%) and lowest R² (0.548). Every sensor has a positive R² under the
|
||||
model; two seasonal-naive baselines are negative.
|
||||
|
||||
## Uncertainty band
|
||||
|
||||
The `lo`/`hi` band is the 10th/90th percentile of holdout residuals added to the
|
||||
point forecast, clipped at zero. It is an empirical 80% band, not a calibrated
|
||||
guarantee.
|
||||
point forecast, clipped at zero: p10 −4.83, p50 −0.59, p90 +4.88. It is an
|
||||
empirical 80% band, not a calibrated guarantee; the negative median residual
|
||||
means the point forecast is slightly optimistic on the holdout.
|
||||
|
||||
## Small classifier
|
||||
## Small classifier ("I coded an AI" artifact)
|
||||
|
||||
This model is separate from the production forecaster and powers the
|
||||
`model=small` retraining demo. Full definition, thresholds, confusion matrix
|
||||
interpretation, and limits: [`docs/small-model.md`](docs/small-model.md).
|
||||
|
||||
- Target: `busy` = hourly footfall strictly above that sensor's p75, where the
|
||||
p75 is computed on the training window only.
|
||||
- Features: `hour`, `weekday`, `is_weekend`, `temp_c`, `precip_mm`, `is_holiday`.
|
||||
- Model: standardized logistic regression, `class_weight="balanced"`.
|
||||
- Split: last 20% of hourly timestamps, time-ordered.
|
||||
- Result: accuracy 0.708, precision 0.360, recall 0.574, F1 0.443, ROC-AUC 0.726.
|
||||
- Trains in ~0.25 s on the challenge CSV and under 1 s on a small upload.
|
||||
- The low precision reflects a hard p75 threshold and a deliberately tiny model;
|
||||
it is a demo artifact, not the production forecast.
|
||||
p75 is computed on the training window only and then frozen for the holdout.
|
||||
- Features: `hour`, `weekday`, `is_weekend`, `temp_c`, `precip_mm`, `is_holiday`
|
||||
(no lags, no sensor identifier, no target history).
|
||||
- Model: standardized logistic regression, `class_weight="balanced"`, `C=1.0`,
|
||||
`max_iter=1000`, `random_state=42`.
|
||||
- Split: last 20% of hourly timestamps, time-ordered. Train
|
||||
2026-06-12T00:00Z → 2026-10-09T01:00Z (18,242 rows); test
|
||||
2026-10-09T02:00Z → 2026-11-07T23:00Z (5,710 rows).
|
||||
- Result (`logit-20261114-052723`): accuracy 0.708, precision 0.360, recall
|
||||
0.574, F1 0.443, ROC-AUC 0.726, base rate 0.202.
|
||||
- Confusion matrix: TN 3,377, FP 1,178, FN 492, TP 663. Specificity 0.741,
|
||||
negative predictive value 0.873.
|
||||
- Fit time: 0.30 s on the full challenge CSV; the API job (parse + fit) must
|
||||
stay under the 5 s contract. The live e2e run trained a 60-day upload subset
|
||||
in 0.17 s wall time.
|
||||
- Per-sensor F1 ranges from 0.230 (Marahaus, 5.5% positive) to 0.592 (Soleo,
|
||||
70.4% positive). The balanced weights over-flag low-base-rate sensors; accuracy
|
||||
is lowest on Soleo (0.581) and recall is lowest on Stadtgalerie (0.374).
|
||||
- Accuracy is below the 0.798 majority-class baseline. Do not use accuracy as
|
||||
the headline: the model finds 663 of 1,155 busy hours, which the majority
|
||||
baseline never finds. The low precision reflects a hard p75 threshold, a
|
||||
deliberately tiny model, and balanced class weights.
|
||||
- This is a demo artifact, not the production forecast. Thresholds move with
|
||||
the upload window, and a CSV without weather overlap falls back to
|
||||
14 °C / 0 mm.
|
||||
|
||||
## Artifacts and reproduction
|
||||
|
||||
- `artifacts/metrics.json` — versions, headline, ablation, per-sensor, small
|
||||
model summary.
|
||||
- `artifacts/small_metrics.json` — full small-model payload including the
|
||||
confusion matrix, per-sensor table, and thresholds.
|
||||
- `make train` regenerates models and metrics; `make small` prints the
|
||||
small-model numbers; `make figures` regenerates `docs/figures/*.svg`.
|
||||
- `docs/metrics.md` is the paper support sheet; it quotes the JSON and does not
|
||||
invent numbers.
|
||||
|
||||
## No-leakage rules
|
||||
|
||||
|
||||
@@ -0,0 +1,523 @@
|
||||
#!/usr/bin/env python3
|
||||
"""End-to-end acceptance run against a real FastAPI process on port 18000.
|
||||
|
||||
Starts uvicorn with ``CONTROL_TOKEN``, then exercises robustness and the full
|
||||
training lifecycle:
|
||||
|
||||
- /healthz, /readyz, /meta, /bootstrap (8 sensors, hourly series, context)
|
||||
- /predict for every sensor x horizon x scenario (128 calls)
|
||||
- /simulate for every scenario with all sensors
|
||||
- auth, unknown job, unknown model, malformed CSV, one-job-at-a-time
|
||||
- POST /train model=small on a real CSV subset; poll to success; hot-swap
|
||||
- POST /train model=forecast on the full CSV; poll to success; hot-swap
|
||||
- /predict still works after each swap
|
||||
|
||||
Exit code is 0 only when every check passes. Figures printed to stdout are the
|
||||
measured numbers; ``--report`` writes the same result as JSON.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import pandas as pd
|
||||
|
||||
from pulsecast import config
|
||||
|
||||
HORIZON_SIZES = {"15m": 1, "1h": 4, "48h": 48, "7d": 168}
|
||||
SCENARIOS = tuple(config.SCENARIOS)
|
||||
SENSORS = tuple(sorted(config.USABLE_SENSORS))
|
||||
|
||||
|
||||
class CheckFailed(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class Runner:
|
||||
def __init__(self) -> None:
|
||||
self.checks = 0
|
||||
self.results: dict = {}
|
||||
|
||||
def check(self, condition: bool, message: str) -> None:
|
||||
self.checks += 1
|
||||
if not condition:
|
||||
raise CheckFailed(message)
|
||||
print(f" [ok] {message}")
|
||||
|
||||
|
||||
def _utc_now() -> str:
|
||||
return datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
|
||||
def _parse_ts(value: str) -> pd.Timestamp:
|
||||
return pd.Timestamp(value)
|
||||
|
||||
|
||||
def _seconds(start: str | None, end: str | None) -> float | None:
|
||||
if not start or not end:
|
||||
return None
|
||||
return round((_parse_ts(end) - _parse_ts(start)).total_seconds(), 3)
|
||||
|
||||
|
||||
def _wait_http(client: httpx.Client, url: str, timeout: float = 30.0) -> None:
|
||||
deadline = time.monotonic() + timeout
|
||||
last: Exception | None = None
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
response = client.get(url, timeout=2.0)
|
||||
if response.status_code == 200:
|
||||
return
|
||||
except Exception as exc: # noqa: BLE001 - startup race
|
||||
last = exc
|
||||
time.sleep(0.1)
|
||||
raise CheckFailed(f"server did not answer {url} within {timeout}s: {last}")
|
||||
|
||||
|
||||
def _wait_job(
|
||||
client: httpx.Client,
|
||||
job_id: str,
|
||||
headers: dict[str, str],
|
||||
*,
|
||||
timeout: float = 180.0,
|
||||
state: str | None = None,
|
||||
) -> dict:
|
||||
deadline = time.monotonic() + timeout
|
||||
last_status = None
|
||||
while time.monotonic() < deadline:
|
||||
response = client.get(f"/train/{job_id}", headers=headers)
|
||||
if response.status_code != 200:
|
||||
raise CheckFailed(f"GET /train/{job_id} -> {response.status_code}: {response.text}")
|
||||
job = response.json()
|
||||
last_status = job["status"]
|
||||
if state is not None and last_status == state:
|
||||
return job
|
||||
if state is None and last_status in {"succeeded", "failed"}:
|
||||
return job
|
||||
time.sleep(0.05)
|
||||
raise CheckFailed(f"job {job_id} did not reach {state or 'terminal'} (last={last_status})")
|
||||
|
||||
|
||||
def _job_headers(token: str) -> dict[str, str]:
|
||||
return {"Authorization": f"Bearer {token}"}
|
||||
|
||||
|
||||
def _submit(
|
||||
client: httpx.Client, headers: dict[str, str], path: Path, model: str
|
||||
) -> httpx.Response:
|
||||
return client.post(
|
||||
"/train",
|
||||
headers=headers,
|
||||
files={"file": (path.name, path.read_bytes(), "text/csv")},
|
||||
data={"model": model},
|
||||
timeout=120.0,
|
||||
)
|
||||
|
||||
|
||||
def _assert_points(points: list[dict], expected: int, label: str) -> None:
|
||||
if len(points) != expected:
|
||||
raise CheckFailed(f"{label}: expected {expected} points, got {len(points)}")
|
||||
previous = None
|
||||
for point in points:
|
||||
if set(point) != {"t", "v", "lo", "hi"}:
|
||||
raise CheckFailed(f"{label}: unexpected point keys {sorted(point)}")
|
||||
if not point["t"].endswith("Z"):
|
||||
raise CheckFailed(f"{label}: timestamp is not UTC RFC 3339: {point['t']}")
|
||||
if point["v"] < 0 or point["lo"] > point["v"] or point["v"] > point["hi"]:
|
||||
raise CheckFailed(f"{label}: invalid band {point}")
|
||||
stamp = _parse_ts(point["t"])
|
||||
if previous is not None and stamp <= previous:
|
||||
raise CheckFailed(f"{label}: timestamps are not increasing at {point['t']}")
|
||||
previous = stamp
|
||||
|
||||
|
||||
def _versions(payload: dict) -> dict:
|
||||
return {
|
||||
key: payload.get(key)
|
||||
for key in ("data_version", "model_version", "context_version")
|
||||
}
|
||||
|
||||
|
||||
def _build_subset(raw: Path, destination: Path, days: int = 60) -> tuple[Path, dict]:
|
||||
frame = pd.read_csv(raw)
|
||||
frame = frame[frame["type"].astype(str) == "densityData"].copy()
|
||||
frame["time"] = pd.to_datetime(frame["time"], utc=True)
|
||||
end = frame["time"].max()
|
||||
cut = end - pd.Timedelta(days=days)
|
||||
frame = frame[frame["time"] >= cut][["time", "name", "value"]]
|
||||
frame["time"] = frame["time"].dt.strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
frame.to_csv(destination, index=False)
|
||||
return destination, {
|
||||
"rows": len(frame),
|
||||
"sensors": int(frame["name"].nunique()),
|
||||
"from": frame["time"].min(),
|
||||
"to": frame["time"].max(),
|
||||
"days": days,
|
||||
}
|
||||
|
||||
|
||||
def _port_free(port: int) -> bool:
|
||||
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
|
||||
sock.settimeout(0.5)
|
||||
return sock.connect_ex(("127.0.0.1", port)) != 0
|
||||
|
||||
|
||||
def run(args: argparse.Namespace, runner: Runner) -> dict:
|
||||
raw = Path(args.csv)
|
||||
if not raw.exists():
|
||||
raise CheckFailed(f"CSV not found: {raw}")
|
||||
if not _port_free(args.port):
|
||||
raise CheckFailed(f"port {args.port} is already in use")
|
||||
|
||||
workdir = Path(tempfile.mkdtemp(prefix="pulsecast-e2e-"))
|
||||
models_dir = workdir / "models"
|
||||
uploads_dir = workdir / "uploads"
|
||||
subset_path, subset_info = _build_subset(raw, workdir / "small-subset.csv", args.days)
|
||||
garbage_path = workdir / "garbage.csv"
|
||||
garbage_path.write_text("not,a,pulsecast,file\n1,2,3,4\n")
|
||||
|
||||
env = os.environ.copy()
|
||||
env.update(
|
||||
CONTROL_TOKEN=args.token,
|
||||
PULSECAST_MODELS=str(models_dir),
|
||||
PULSECAST_UPLOADS=str(uploads_dir),
|
||||
)
|
||||
log_path = workdir / "server.log"
|
||||
log_handle = log_path.open("wb")
|
||||
command = [
|
||||
sys.executable,
|
||||
"-m",
|
||||
"uvicorn",
|
||||
"pulsecast.server:app",
|
||||
"--host",
|
||||
"127.0.0.1",
|
||||
"--port",
|
||||
str(args.port),
|
||||
"--log-level",
|
||||
"warning",
|
||||
]
|
||||
process = subprocess.Popen(command, env=env, stdout=log_handle, stderr=subprocess.STDOUT)
|
||||
report: dict = {
|
||||
"started_at": _utc_now(),
|
||||
"base_url": f"http://127.0.0.1:{args.port}",
|
||||
"subset": subset_info,
|
||||
"raw_csv_bytes": raw.stat().st_size,
|
||||
}
|
||||
|
||||
try:
|
||||
with httpx.Client(base_url=f"http://127.0.0.1:{args.port}", timeout=120.0) as client:
|
||||
_wait_http(client, "/healthz")
|
||||
headers = _job_headers(args.token)
|
||||
|
||||
print("Boot and snapshot")
|
||||
health = client.get("/healthz").json()
|
||||
runner.check(health == {"ok": True}, "/healthz returns {ok:true}")
|
||||
ready = client.get("/readyz")
|
||||
runner.check(ready.status_code == 200 and ready.json()["model"] is True, "/readyz 200 model=true")
|
||||
meta_before = client.get("/meta").json()
|
||||
runner.check(meta_before["coverage"]["sensors"] == 8, "/meta covers 8 sensors")
|
||||
runner.check(
|
||||
meta_before["metrics"]["mae"] < meta_before["metrics"]["baseline_mae"],
|
||||
"baked model beats seasonal naive on the holdout",
|
||||
)
|
||||
bootstrap = client.get("/bootstrap").json()
|
||||
runner.check(len(bootstrap["sensors"]) == 8, "/bootstrap returns 8 sensors")
|
||||
runner.check(
|
||||
set(bootstrap["series"]) == set(SENSORS),
|
||||
"/bootstrap series covers all 8 sensors",
|
||||
)
|
||||
series_points = sum(len(points) for points in bootstrap["series"].values())
|
||||
runner.check(series_points > 20_000, f"/bootstrap carries {series_points} hourly points")
|
||||
context = bootstrap["context"]
|
||||
runner.check(
|
||||
len(context["weather"]["forecast"]) == 168,
|
||||
"/bootstrap context has 168 forecast weather hours",
|
||||
)
|
||||
runner.check(bool(context["holidays"]), "/bootstrap context has holidays")
|
||||
runner.check(
|
||||
_versions(bootstrap) == _versions(meta_before),
|
||||
"/bootstrap versions match /meta",
|
||||
)
|
||||
report["versions_before"] = _versions(meta_before)
|
||||
report["bootstrap"] = {"sensors": 8, "series_points": series_points}
|
||||
|
||||
print("Robustness: /predict every sensor x horizon x scenario")
|
||||
predict_calls = 0
|
||||
for sensor in SENSORS:
|
||||
for horizon, count in HORIZON_SIZES.items():
|
||||
for scenario in SCENARIOS:
|
||||
response = client.post(
|
||||
"/predict",
|
||||
json={"sensor": sensor, "horizon": horizon, "scenario": scenario},
|
||||
)
|
||||
if response.status_code != 200:
|
||||
raise CheckFailed(
|
||||
f"/predict {sensor}/{horizon}/{scenario} -> "
|
||||
f"{response.status_code}: {response.text}"
|
||||
)
|
||||
payload = response.json()
|
||||
_assert_points(
|
||||
payload["points"],
|
||||
count,
|
||||
f"/predict {sensor}/{horizon}/{scenario}",
|
||||
)
|
||||
predict_calls += 1
|
||||
runner.check(predict_calls == 128, "128 /predict combinations returned valid points")
|
||||
|
||||
print("Robustness: /simulate every scenario, all sensors")
|
||||
simulate_calls = 0
|
||||
for scenario in SCENARIOS:
|
||||
response = client.post(
|
||||
"/simulate", json={"scenario": scenario, "sensors": "all"}
|
||||
)
|
||||
if response.status_code != 200:
|
||||
raise CheckFailed(f"/simulate {scenario} -> {response.status_code}: {response.text}")
|
||||
payload = response.json()
|
||||
if set(payload["forecasts"]) != set(SENSORS):
|
||||
raise CheckFailed(f"/simulate {scenario}: sensor set mismatch")
|
||||
for sensor, by_horizon in payload["forecasts"].items():
|
||||
if set(by_horizon) != set(HORIZON_SIZES):
|
||||
raise CheckFailed(f"/simulate {scenario}/{sensor}: horizon set mismatch")
|
||||
for horizon, count in HORIZON_SIZES.items():
|
||||
_assert_points(
|
||||
by_horizon[horizon],
|
||||
count,
|
||||
f"/simulate {scenario}/{sensor}/{horizon}",
|
||||
)
|
||||
simulate_calls += 1
|
||||
runner.check(simulate_calls == 4, "4 /simulate scenarios x 8 sensors x 4 horizons")
|
||||
|
||||
cold = client.post(
|
||||
"/predict",
|
||||
json={"sensor": SENSORS[0], "horizon": "48h", "weather": {"temp_c": -8.0, "precip_mm": 6.0}},
|
||||
).json()["points"]
|
||||
warm = client.post(
|
||||
"/predict",
|
||||
json={"sensor": SENSORS[0], "horizon": "48h", "weather": {"temp_c": 30.0, "precip_mm": 0.0}},
|
||||
).json()["points"]
|
||||
runner.check(cold != warm, "weather override changes the forecast")
|
||||
|
||||
print("Error handling")
|
||||
no_token = client.post("/predict", json={"sensor": "nope"})
|
||||
runner.check(no_token.status_code == 404, "/predict unknown sensor -> 404")
|
||||
unauthorized = _submit(client, {}, subset_path, "small")
|
||||
runner.check(
|
||||
unauthorized.status_code == 401
|
||||
and unauthorized.json()["error"]["code"] == "unauthorized",
|
||||
"/train without token -> 401 unauthorized",
|
||||
)
|
||||
bad_token = _submit(client, _job_headers("wrong-token"), subset_path, "small")
|
||||
runner.check(bad_token.status_code == 401, "/train with wrong token -> 401")
|
||||
unknown_job = client.get("/train/nope", headers=headers)
|
||||
runner.check(
|
||||
unknown_job.status_code == 404
|
||||
and unknown_job.json()["error"]["code"] == "not_found",
|
||||
"/train/nope -> 404 not_found",
|
||||
)
|
||||
bad_model = _submit(client, headers, subset_path, "giant")
|
||||
runner.check(
|
||||
bad_model.status_code == 400
|
||||
and bad_model.json()["error"]["code"] == "bad_request",
|
||||
"/train invalid model -> 400 bad_request",
|
||||
)
|
||||
missing_path = client.post(
|
||||
"/train", headers=headers, json={"path": "/tmp/no-such-pulsecast.csv", "model": "small"}
|
||||
)
|
||||
runner.check(missing_path.status_code == 400, "/train missing JSON path -> 400")
|
||||
|
||||
print("Small model: real CSV subset through the API")
|
||||
started = time.monotonic()
|
||||
submitted = _submit(client, headers, subset_path, "small")
|
||||
if submitted.status_code != 202:
|
||||
raise CheckFailed(f"/train small -> {submitted.status_code}: {submitted.text}")
|
||||
small_submitted = submitted.json()
|
||||
runner.check(
|
||||
small_submitted["status"] == "queued" and small_submitted["model"] == "small",
|
||||
f"/train small queued as job {small_submitted['job_id']}",
|
||||
)
|
||||
small_job = _wait_job(client, small_submitted["job_id"], headers)
|
||||
wall = round(time.monotonic() - started, 3)
|
||||
if small_job["status"] != "succeeded":
|
||||
raise CheckFailed(f"small job failed: {small_job}")
|
||||
small_metrics = small_job["metrics"]
|
||||
for key in ("accuracy", "precision", "recall", "f1", "roc_auc", "n_train", "n_test", "positive_rate"):
|
||||
if key not in small_metrics:
|
||||
raise CheckFailed(f"small metrics missing {key}: {small_metrics}")
|
||||
train_seconds = _seconds(small_job["started_at"], small_job["finished_at"])
|
||||
runner.check(
|
||||
small_job["model_version"].startswith("logit-"),
|
||||
f"small model_version {small_job['model_version']}",
|
||||
)
|
||||
runner.check(
|
||||
wall < 5.0,
|
||||
f"small job took {wall}s wall time including upload (contract < 5 s; "
|
||||
f"server timestamps have 1 s resolution, train_seconds={train_seconds})",
|
||||
)
|
||||
runner.check(small_metrics["accuracy"] > 0.6, f"small accuracy {small_metrics['accuracy']}")
|
||||
runner.check(
|
||||
(models_dir / "small_latest.joblib").exists(),
|
||||
"small model persisted to PULSECAST_MODELS",
|
||||
)
|
||||
meta_after_small = client.get("/meta").json()
|
||||
runner.check(
|
||||
meta_after_small["small_model"]["model_version"] == small_job["model_version"],
|
||||
"small model hot-swapped into /meta",
|
||||
)
|
||||
runner.check(
|
||||
meta_after_small["small_model"]["metrics"]["accuracy"] == small_metrics["accuracy"],
|
||||
"hot-swapped small metrics match the job",
|
||||
)
|
||||
predict_after_small = client.post(
|
||||
"/predict", json={"sensor": SENSORS[0], "horizon": "7d", "scenario": "base"}
|
||||
)
|
||||
runner.check(
|
||||
predict_after_small.status_code == 200
|
||||
and len(predict_after_small.json()["points"]) == 168,
|
||||
"/predict still works after the small swap",
|
||||
)
|
||||
report["jobs"] = {
|
||||
"small": {
|
||||
"job": small_job,
|
||||
"wall_seconds": wall,
|
||||
"train_seconds": train_seconds,
|
||||
"input": subset_info,
|
||||
}
|
||||
}
|
||||
|
||||
print("Forecast model: full CSV, hot-swap, 409 while busy")
|
||||
forecast_started = time.monotonic()
|
||||
forecast_response = _submit(client, headers, raw, "forecast")
|
||||
if forecast_response.status_code != 202:
|
||||
raise CheckFailed(
|
||||
f"/train forecast -> {forecast_response.status_code}: {forecast_response.text}"
|
||||
)
|
||||
forecast_submitted = forecast_response.json()
|
||||
runner.check(
|
||||
forecast_submitted["model"] == "forecast",
|
||||
f"/train forecast queued as job {forecast_submitted['job_id']}",
|
||||
)
|
||||
_wait_job(client, forecast_submitted["job_id"], headers, state="running", timeout=30)
|
||||
busy = _submit(client, headers, subset_path, "small")
|
||||
runner.check(
|
||||
busy.status_code == 409 and busy.json()["error"]["code"] == "conflict",
|
||||
"second /train while forecast is running -> 409 conflict",
|
||||
)
|
||||
forecast_job = _wait_job(client, forecast_submitted["job_id"], headers, timeout=300)
|
||||
forecast_wall = round(time.monotonic() - forecast_started, 3)
|
||||
if forecast_job["status"] != "succeeded":
|
||||
raise CheckFailed(f"forecast job failed: {forecast_job}")
|
||||
forecast_metrics = forecast_job["metrics"]
|
||||
for key in ("mae", "wape", "r2", "baseline_mae", "n_eval"):
|
||||
if key not in forecast_metrics:
|
||||
raise CheckFailed(f"forecast metrics missing {key}: {forecast_metrics}")
|
||||
runner.check(
|
||||
forecast_job["model_version"].startswith("histgb-"),
|
||||
f"forecast model_version {forecast_job['model_version']}",
|
||||
)
|
||||
runner.check(
|
||||
forecast_metrics["mae"] < forecast_metrics["baseline_mae"],
|
||||
f"retrained forecast MAE {forecast_metrics['mae']} < baseline {forecast_metrics['baseline_mae']}",
|
||||
)
|
||||
runner.check(
|
||||
(models_dir / "forecast_latest.joblib").exists(),
|
||||
"forecast model persisted to PULSECAST_MODELS",
|
||||
)
|
||||
meta_after_forecast = client.get("/meta").json()
|
||||
runner.check(
|
||||
meta_after_forecast["model_version"] == forecast_job["model_version"]
|
||||
and meta_after_forecast["model_source"] == "retrained",
|
||||
"forecast model hot-swapped into /meta as retrained",
|
||||
)
|
||||
predict_after_forecast = client.post(
|
||||
"/predict", json={"sensor": SENSORS[-1], "horizon": "48h", "scenario": "rainy-saturday"}
|
||||
)
|
||||
runner.check(
|
||||
predict_after_forecast.status_code == 200
|
||||
and predict_after_forecast.json()["model_version"] == forecast_job["model_version"],
|
||||
"/predict uses the retrained forecast model",
|
||||
)
|
||||
runner.check(client.get("/readyz").status_code == 200, "/readyz stays 200 after hot-swap")
|
||||
report["jobs"]["forecast"] = {
|
||||
"job": forecast_job,
|
||||
"wall_seconds": forecast_wall,
|
||||
"input": {"rows": "full challenge CSV", "bytes": raw.stat().st_size},
|
||||
}
|
||||
|
||||
print("Malformed CSV fails the job (not the process)")
|
||||
bad_csv = _submit(client, headers, garbage_path, "small")
|
||||
runner.check(bad_csv.status_code == 202, "malformed CSV is accepted as a job")
|
||||
bad_job = _wait_job(client, bad_csv.json()["job_id"], headers)
|
||||
runner.check(
|
||||
bad_job["status"] == "failed" and bool(bad_job["error"]),
|
||||
f"malformed CSV job failed with {bad_job['error']!r}",
|
||||
)
|
||||
runner.check(
|
||||
client.get("/healthz").status_code == 200,
|
||||
"/healthz still 200 after a failed job",
|
||||
)
|
||||
|
||||
report["versions_after_small"] = _versions(meta_after_small)
|
||||
report["versions_after_forecast"] = _versions(meta_after_forecast)
|
||||
report["checks"] = runner.checks
|
||||
report["ok"] = True
|
||||
finally:
|
||||
process.terminate()
|
||||
try:
|
||||
process.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
process.kill()
|
||||
process.wait(timeout=5)
|
||||
log_handle.close()
|
||||
if process.returncode not in (0, -15):
|
||||
print(f"server exited with {process.returncode}; log: {log_path}", file=sys.stderr)
|
||||
print(log_path.read_text(errors="replace")[-4000:], file=sys.stderr)
|
||||
report["server_log_tail"] = log_path.read_text(errors="replace")[-1000:]
|
||||
if args.keep:
|
||||
print(f"kept workdir {workdir}")
|
||||
else:
|
||||
shutil.rmtree(workdir, ignore_errors=True)
|
||||
|
||||
report["finished_at"] = _utc_now()
|
||||
return report
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
parser.add_argument("--port", type=int, default=18000)
|
||||
parser.add_argument("--token", default=os.environ.get("CONTROL_TOKEN") or "e2e-local-token")
|
||||
parser.add_argument("--csv", default=str(config.raw_csv_path()))
|
||||
parser.add_argument("--days", type=int, default=60, help="days in the small-model CSV subset")
|
||||
parser.add_argument("--report", type=Path, help="write the JSON result here")
|
||||
parser.add_argument("--keep", action="store_true", help="keep the temporary workdir")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
runner = Runner()
|
||||
print(f"PulseCast end-to-end check on port {args.port}")
|
||||
started = time.monotonic()
|
||||
try:
|
||||
report = run(args, runner)
|
||||
except CheckFailed as exc:
|
||||
print(f"\nFAIL after {runner.checks} checks: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
elapsed = round(time.monotonic() - started, 1)
|
||||
report["elapsed_seconds"] = elapsed
|
||||
print(f"\nPASS: {runner.checks} checks in {elapsed}s")
|
||||
if args.report:
|
||||
args.report.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.report.write_text(json.dumps(report, indent=2, ensure_ascii=False) + "\n")
|
||||
print(f"report: {args.report}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,262 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Render dependency-free SVG figures from ``artifacts/metrics.json``.
|
||||
|
||||
The paper figures are generated from the measured holdout metrics, never
|
||||
hand-written. Run ``make figures`` after ``make train``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
from xml.sax.saxutils import escape
|
||||
|
||||
from pulsecast import config
|
||||
|
||||
MODEL_COLOR = "#2563eb"
|
||||
BASELINE_COLOR = "#94a3b8"
|
||||
PRODUCTION_COLOR = "#0f766e"
|
||||
TEXT_COLOR = "#0f172a"
|
||||
GRID_COLOR = "#e2e8f0"
|
||||
|
||||
|
||||
def _svg_document(
|
||||
title: str,
|
||||
subtitle: str,
|
||||
body: list[str],
|
||||
width: int,
|
||||
height: int,
|
||||
footer: str = "Source: artifacts/metrics.json · PulseCast 28-day time-based holdout",
|
||||
) -> str:
|
||||
return "\n".join(
|
||||
[
|
||||
f'<svg xmlns="http://www.w3.org/2000/svg" width="{width}" height="{height}" '
|
||||
f'viewBox="0 0 {width} {height}" role="img" aria-label="{escape(title)}">',
|
||||
'<rect width="100%" height="100%" fill="#ffffff"/>',
|
||||
'<g font-family="-apple-system, Segoe UI, Helvetica, Arial, sans-serif">',
|
||||
f'<text x="24" y="34" font-size="20" font-weight="600" fill="{TEXT_COLOR}">'
|
||||
f"{escape(title)}</text>",
|
||||
f'<text x="24" y="56" font-size="13" fill="#475569">{escape(subtitle)}</text>',
|
||||
*body,
|
||||
f'<text x="24" y="{height - 12}" font-size="11" fill="#64748b">'
|
||||
f"{escape(footer)}</text>",
|
||||
"</g>",
|
||||
"</svg>",
|
||||
"",
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def _bar_chart(
|
||||
title: str,
|
||||
subtitle: str,
|
||||
rows: list[dict],
|
||||
*,
|
||||
value_key: str = "mae",
|
||||
secondary_key: str | None = None,
|
||||
secondary_label: str | None = None,
|
||||
primary_label: str = "MAE",
|
||||
production: str | None = None,
|
||||
) -> str:
|
||||
width = 960
|
||||
left = 230
|
||||
right = 96
|
||||
top = 86
|
||||
row_height = 38
|
||||
height = top + row_height * len(rows) + 56
|
||||
plot_width = width - left - right
|
||||
values = [row[value_key] for row in rows]
|
||||
if secondary_key:
|
||||
values += [row[secondary_key] for row in rows]
|
||||
maximum = max(values) * 1.08 if values else 1.0
|
||||
|
||||
body: list[str] = []
|
||||
for tick in range(5):
|
||||
x = left + plot_width * tick / 4
|
||||
value = maximum * tick / 4
|
||||
body.append(
|
||||
f'<line x1="{x:.1f}" y1="{top - 18}" x2="{x:.1f}" y2="{top + row_height * len(rows)}" '
|
||||
f'stroke="{GRID_COLOR}" stroke-width="1"/>'
|
||||
)
|
||||
body.append(
|
||||
f'<text x="{x:.1f}" y="{top - 24}" font-size="11" fill="#64748b" '
|
||||
f'text-anchor="middle">{value:.1f}</text>'
|
||||
)
|
||||
|
||||
legend_y = top - 34
|
||||
if secondary_key:
|
||||
body.append(
|
||||
f'<rect x="{left}" y="{legend_y}" width="10" height="10" fill="{BASELINE_COLOR}"/>'
|
||||
f'<text x="{left + 16}" y="{legend_y + 9}" font-size="12" fill="#475569">'
|
||||
f"{escape(secondary_label or 'baseline')}</text>"
|
||||
)
|
||||
body.append(
|
||||
f'<rect x="{left + 140}" y="{legend_y}" width="10" height="10" fill="{MODEL_COLOR}"/>'
|
||||
f'<text x="{left + 156}" y="{legend_y + 9}" font-size="12" fill="#475569">'
|
||||
f"{escape(primary_label)}</text>"
|
||||
)
|
||||
|
||||
for index, row in enumerate(rows):
|
||||
y = top + index * row_height
|
||||
label = str(row.get("name") or row.get("sensor"))
|
||||
body.append(
|
||||
f'<text x="{left - 12}" y="{y + 20}" font-size="12" fill="{TEXT_COLOR}" '
|
||||
f'text-anchor="end">{escape(label)}</text>'
|
||||
)
|
||||
if secondary_key:
|
||||
baseline_width = plot_width * row[secondary_key] / maximum
|
||||
body.append(
|
||||
f'<rect x="{left}" y="{y + 3}" width="{baseline_width:.1f}" height="12" '
|
||||
f'fill="{BASELINE_COLOR}"/>'
|
||||
)
|
||||
body.append(
|
||||
f'<text x="{left + baseline_width + 6:.1f}" y="{y + 13}" font-size="11" '
|
||||
f'fill="#64748b">{row[secondary_key]:.2f}</text>'
|
||||
)
|
||||
model_width = plot_width * row[value_key] / maximum
|
||||
body.append(
|
||||
f'<rect x="{left}" y="{y + 18}" width="{model_width:.1f}" height="12" '
|
||||
f'fill="{MODEL_COLOR}"/>'
|
||||
)
|
||||
body.append(
|
||||
f'<text x="{left + model_width + 6:.1f}" y="{y + 28}" font-size="11" '
|
||||
f'fill="{TEXT_COLOR}">{row[value_key]:.2f}</text>'
|
||||
)
|
||||
else:
|
||||
color = PRODUCTION_COLOR if production and row.get("name") == production else MODEL_COLOR
|
||||
bar_width = plot_width * row[value_key] / maximum
|
||||
body.append(
|
||||
f'<rect x="{left}" y="{y + 6}" width="{bar_width:.1f}" height="22" fill="{color}"/>'
|
||||
)
|
||||
body.append(
|
||||
f'<text x="{left + bar_width + 6:.1f}" y="{y + 21}" font-size="12" '
|
||||
f'fill="{TEXT_COLOR}">{row[value_key]:.2f}</text>'
|
||||
)
|
||||
|
||||
return _svg_document(title, subtitle, body, width, height)
|
||||
|
||||
|
||||
def _confusion_svg(small: dict) -> str:
|
||||
matrix = small["confusion_matrix"]
|
||||
metrics = small["metrics"]
|
||||
width, height = 760, 520
|
||||
left, top, cell = 200, 130, 190
|
||||
values = {
|
||||
(0, 0): ("TN", matrix["tn"]),
|
||||
(0, 1): ("FP", matrix["fp"]),
|
||||
(1, 0): ("FN", matrix["fn"]),
|
||||
(1, 1): ("TP", matrix["tp"]),
|
||||
}
|
||||
maximum = max(value for _, value in values.values())
|
||||
body: list[str] = []
|
||||
body.append(
|
||||
f'<text x="{left + cell / 2}" y="{top - 16}" font-size="13" fill="#475569" '
|
||||
f'text-anchor="middle">Predicted not busy</text>'
|
||||
)
|
||||
body.append(
|
||||
f'<text x="{left + cell + cell / 2}" y="{top - 16}" font-size="13" fill="#475569" '
|
||||
f'text-anchor="middle">Predicted busy</text>'
|
||||
)
|
||||
row_labels = ("Actually\nnot busy", "Actually\nbusy")
|
||||
for row, label in enumerate(row_labels):
|
||||
y = top + row * cell
|
||||
body.append(
|
||||
f'<text x="{left - 18}" y="{y + cell / 2}" font-size="13" fill="#475569" '
|
||||
f'text-anchor="end">{escape(label).replace(chr(10), " ")}</text>'
|
||||
)
|
||||
for (row, column), (code, value) in values.items():
|
||||
x = left + column * cell
|
||||
y = top + row * cell
|
||||
intensity = 0.12 + 0.88 * (value / maximum)
|
||||
red = round(219 + (29 - 219) * intensity)
|
||||
green = round(234 + (78 - 234) * intensity)
|
||||
blue = round(254 + (216 - 254) * intensity)
|
||||
fill = f"#{red:02x}{green:02x}{blue:02x}"
|
||||
text_color = "#ffffff" if intensity > 0.55 else TEXT_COLOR
|
||||
row_total = matrix["tn"] + matrix["fp"] if row == 0 else matrix["fn"] + matrix["tp"]
|
||||
share = 100.0 * value / row_total if row_total else 0.0
|
||||
body.append(
|
||||
f'<rect x="{x}" y="{y}" width="{cell}" height="{cell}" fill="{fill}" '
|
||||
f'stroke="#ffffff" stroke-width="2"/>'
|
||||
)
|
||||
body.append(
|
||||
f'<text x="{x + cell / 2}" y="{y + cell / 2 - 8}" font-size="30" '
|
||||
f'font-weight="600" fill="{text_color}" text-anchor="middle">{value:,}</text>'
|
||||
)
|
||||
body.append(
|
||||
f'<text x="{x + cell / 2}" y="{y + cell / 2 + 20}" font-size="13" '
|
||||
f'fill="{text_color}" text-anchor="middle">{code} · {share:.1f}% of row</text>'
|
||||
)
|
||||
subtitle = (
|
||||
f"n={metrics['n_test']:,} · accuracy {metrics['accuracy']:.3f} · "
|
||||
f"precision {metrics['precision']:.3f} · recall {metrics['recall']:.3f} · "
|
||||
f"F1 {metrics['f1']:.3f} · ROC-AUC {metrics['roc_auc']:.3f}"
|
||||
)
|
||||
return _svg_document(
|
||||
"Small busy-hour classifier: holdout confusion matrix",
|
||||
subtitle,
|
||||
body,
|
||||
width,
|
||||
height,
|
||||
footer="Source: artifacts/small_metrics.json · chronological 80/20 split",
|
||||
)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument(
|
||||
"--artifacts",
|
||||
type=Path,
|
||||
default=config.artifacts_dir(),
|
||||
help="artifact directory containing metrics.json (default: %(default)s)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--out",
|
||||
type=Path,
|
||||
default=Path(__file__).resolve().parents[1] / "docs" / "figures",
|
||||
help="output directory for SVG files (default: %(default)s)",
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
metrics = json.loads((args.artifacts / "metrics.json").read_text())
|
||||
holdout = f"Holdout {metrics['test_from']} .. {metrics['test_to']}, n={metrics['metrics']['n_eval']}"
|
||||
args.out.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
ablation = _bar_chart(
|
||||
"Ablation: holdout MAE by feature set",
|
||||
holdout,
|
||||
[
|
||||
{
|
||||
"name": row["name"].replace("_", " "),
|
||||
"mae": row["mae"],
|
||||
}
|
||||
for row in metrics["ablation"]
|
||||
],
|
||||
production="week weather",
|
||||
)
|
||||
ablation_path = args.out / "ablation_mae.svg"
|
||||
ablation_path.write_text(ablation)
|
||||
|
||||
per_sensor = _bar_chart(
|
||||
"Per-sensor holdout MAE: production model vs seasonal naive",
|
||||
holdout,
|
||||
metrics["per_sensor"],
|
||||
secondary_key="baseline_mae",
|
||||
secondary_label="seasonal naive",
|
||||
primary_label="weather-7d",
|
||||
)
|
||||
per_sensor_path = args.out / "per_sensor_mae.svg"
|
||||
per_sensor_path.write_text(per_sensor)
|
||||
|
||||
confusion_path = args.out / "small_model_confusion.svg"
|
||||
confusion_path.write_text(_confusion_svg(metrics["small"]))
|
||||
|
||||
print(f"wrote {ablation_path}")
|
||||
print(f"wrote {per_sensor_path}")
|
||||
print(f"wrote {confusion_path}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,153 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Train and evaluate the small "busy hour" classifier from a CSV.
|
||||
|
||||
This is the repeatable artifact behind ``make small`` and the runtime
|
||||
``model=small`` training job. It prints every number reported in
|
||||
``docs/small-model.md``; ``--save`` bakes the bundle and metrics into
|
||||
``artifacts/`` exactly like ``make train`` does.
|
||||
|
||||
Examples:
|
||||
uv run python scripts/train_small.py
|
||||
uv run python scripts/train_small.py --csv new-data.csv --all
|
||||
uv run python scripts/train_small.py --save --json
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import joblib
|
||||
import pandas as pd
|
||||
|
||||
from pulsecast import config
|
||||
from pulsecast.small import load_hourly_csv, train_small_from_csv
|
||||
|
||||
|
||||
def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
parser.add_argument(
|
||||
"csv_positional",
|
||||
nargs="?",
|
||||
metavar="CSV",
|
||||
help=f"CSV to train from (default: {config.raw_csv_path()})",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--csv",
|
||||
dest="csv_option",
|
||||
help="CSV to train from; overrides the positional argument",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--start",
|
||||
default=config.MULTI_SENSOR_START,
|
||||
help=(
|
||||
"drop rows before this UTC timestamp (default: %(default)s, the "
|
||||
"8-sensor challenge window); ignored with --all"
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--all",
|
||||
action="store_true",
|
||||
help="use every row in the CSV, including density-01's older history",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--save",
|
||||
action="store_true",
|
||||
help="write artifacts/small_model.joblib and artifacts/small_metrics.json",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--json",
|
||||
action="store_true",
|
||||
help="print the full metrics payload as JSON after the summary",
|
||||
)
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def _confusion_lines(matrix: dict) -> list[str]:
|
||||
tn, fp, fn, tp = matrix["tn"], matrix["fp"], matrix["fn"], matrix["tp"]
|
||||
total = tn + fp + fn + tp
|
||||
specificity = tn / (tn + fp) if tn + fp else 0.0
|
||||
negative_predictive = tn / (tn + fn) if tn + fn else 0.0
|
||||
return [
|
||||
"Confusion matrix (rows = truth, columns = prediction)",
|
||||
f" not busy: {tn:5d} {fp:5d}",
|
||||
f" busy : {fn:5d} {tp:5d}",
|
||||
f" n={total} specificity={specificity:.3f} NPV={negative_predictive:.3f}",
|
||||
]
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = _parse_args(argv)
|
||||
path = Path(args.csv_option or args.csv_positional or config.raw_csv_path())
|
||||
if not path.exists():
|
||||
print(f"CSV not found: {path}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
start = None if args.all else pd.Timestamp(args.start)
|
||||
started = time.perf_counter()
|
||||
try:
|
||||
bundle, payload = train_small_from_csv(path, start=start)
|
||||
except (ValueError, FileNotFoundError) as exc:
|
||||
print(f"training failed: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
elapsed = time.perf_counter() - started
|
||||
|
||||
hourly = load_hourly_csv(path)
|
||||
if start is not None:
|
||||
hourly = hourly[hourly["t"] >= start]
|
||||
metrics = payload["metrics"]
|
||||
|
||||
print(f"CSV {path}")
|
||||
print(f"Rows {len(hourly):,} hourly rows, {metrics['sensors']} sensor(s)")
|
||||
print(f"Window {payload['train_from']} .. {payload['test_to']}")
|
||||
print(f"Split (time) train {payload['train_from']} .. {payload['train_to']}"
|
||||
f" ({metrics['n_train']:,} rows)")
|
||||
print(f" test {payload['test_from']} .. {payload['test_to']}"
|
||||
f" ({metrics['n_test']:,} rows)")
|
||||
print("Target busy = hourly density > sensor p75 (training window only)")
|
||||
print(f"Features {', '.join(payload['features'])}")
|
||||
print("Model standardized logistic regression (class_weight=balanced)")
|
||||
print(f"Model version {payload['model_version']}")
|
||||
print()
|
||||
print("Metrics")
|
||||
print(f" accuracy {metrics['accuracy']:.3f}")
|
||||
print(f" precision {metrics['precision']:.3f}")
|
||||
print(f" recall {metrics['recall']:.3f}")
|
||||
print(f" f1 {metrics['f1']:.3f}")
|
||||
print(f" roc_auc {metrics['roc_auc'] if metrics['roc_auc'] is not None else 'n/a'}")
|
||||
print(f" base rate {metrics['positive_rate']:.3f}")
|
||||
print()
|
||||
print("\n".join(_confusion_lines(payload["confusion_matrix"])))
|
||||
print()
|
||||
print("Per sensor (holdout)")
|
||||
print(f" {'sensor':<24} {'n':>6} {'pos':>6} {'acc':>6} {'prec':>6} {'rec':>6} {'f1':>6}")
|
||||
for row in payload.get("per_sensor", []):
|
||||
print(
|
||||
f" {row['sensor']:<24} {row['n']:>6} {row['positive_rate']:>6.3f}"
|
||||
f" {row['accuracy']:>6.3f} {row['precision']:>6.3f}"
|
||||
f" {row['recall']:>6.3f} {row['f1']:>6.3f}"
|
||||
)
|
||||
print()
|
||||
print(f"Fit time {elapsed:.2f} s (contract: < 5 s)")
|
||||
|
||||
if args.save:
|
||||
artifacts = config.artifacts_dir()
|
||||
artifacts.mkdir(parents=True, exist_ok=True)
|
||||
joblib.dump(bundle, artifacts / "small_model.joblib")
|
||||
(artifacts / "small_metrics.json").write_text(
|
||||
json.dumps(payload, indent=2, ensure_ascii=False) + "\n"
|
||||
)
|
||||
print(f"Saved {artifacts / 'small_model.joblib'}")
|
||||
print(f" {artifacts / 'small_metrics.json'}")
|
||||
|
||||
if args.json:
|
||||
print()
|
||||
print(json.dumps(payload, indent=2, ensure_ascii=False))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -17,6 +17,7 @@ import pandas as pd
|
||||
from sklearn.linear_model import LogisticRegression
|
||||
from sklearn.metrics import (
|
||||
accuracy_score,
|
||||
confusion_matrix,
|
||||
f1_score,
|
||||
precision_score,
|
||||
recall_score,
|
||||
@@ -254,6 +255,28 @@ def train_small_from_csv(
|
||||
except ValueError:
|
||||
roc_auc = None
|
||||
|
||||
tn, fp, fn, tp = confusion_matrix(truth, predicted, labels=[0, 1]).ravel()
|
||||
scored = test.assign(_predicted=predicted)
|
||||
per_sensor = []
|
||||
for sensor, group in scored.groupby("sensor", observed=True):
|
||||
sensor_truth = group["busy"].to_numpy(dtype=int)
|
||||
sensor_pred = group["_predicted"].to_numpy(dtype=int)
|
||||
per_sensor.append(
|
||||
{
|
||||
"sensor": str(sensor),
|
||||
"n": len(group),
|
||||
"positive_rate": round(float(sensor_truth.mean()), 3),
|
||||
"accuracy": round(float(accuracy_score(sensor_truth, sensor_pred)), 3),
|
||||
"precision": round(
|
||||
float(precision_score(sensor_truth, sensor_pred, zero_division=0)), 3
|
||||
),
|
||||
"recall": round(
|
||||
float(recall_score(sensor_truth, sensor_pred, zero_division=0)), 3
|
||||
),
|
||||
"f1": round(float(f1_score(sensor_truth, sensor_pred, zero_division=0)), 3),
|
||||
}
|
||||
)
|
||||
|
||||
trained_at = _utc_now()
|
||||
model_version = f"logit-{trained_at[:10].replace('-', '')}-{trained_at[11:19].replace(':', '')}"
|
||||
metrics_payload = {
|
||||
@@ -278,6 +301,14 @@ def train_small_from_csv(
|
||||
"sensors": int(hourly["sensor"].nunique()),
|
||||
}
|
||||
),
|
||||
"confusion_matrix": {
|
||||
"labels": {"0": "not busy", "1": "busy"},
|
||||
"tn": int(tn),
|
||||
"fp": int(fp),
|
||||
"fn": int(fn),
|
||||
"tp": int(tp),
|
||||
},
|
||||
"per_sensor": per_sensor,
|
||||
}
|
||||
bundle = {
|
||||
"model": model,
|
||||
|
||||
+42
-4
@@ -142,6 +142,34 @@ def train_bundle(
|
||||
baseline = by_name["seasonal_naive"]
|
||||
headline = by_name["week_weather"]
|
||||
|
||||
# Per-sensor holdout breakdown for the paper; same rows as the headline.
|
||||
scored = eval_frame.copy()
|
||||
scored["_model"] = preds["week_weather"]
|
||||
scored["_baseline"] = preds["seasonal_naive"]
|
||||
per_sensor = []
|
||||
for sensor, group in scored.groupby("sensor", observed=True):
|
||||
model_row = metrics(group["y"], group["_model"])
|
||||
baseline_row = metrics(group["y"], group["_baseline"])
|
||||
improvement = (
|
||||
100.0 * (baseline_row["mae"] - model_row["mae"]) / baseline_row["mae"]
|
||||
if baseline_row["mae"]
|
||||
else 0.0
|
||||
)
|
||||
per_sensor.append(
|
||||
{
|
||||
"sensor": str(sensor),
|
||||
"name": config.DISPLAY_NAMES.get(str(sensor), str(sensor)),
|
||||
"n": model_row["n"],
|
||||
"mae": round(model_row["mae"], 2),
|
||||
"wape": round(model_row["wape"], 1),
|
||||
"r2": round(model_row["r2"], 3),
|
||||
"baseline_mae": round(baseline_row["mae"], 2),
|
||||
"baseline_wape": round(baseline_row["wape"], 1),
|
||||
"baseline_r2": round(baseline_row["r2"], 3),
|
||||
"mae_improvement_pct": round(improvement, 1),
|
||||
}
|
||||
)
|
||||
|
||||
# Production model: refit on all observed rows after the honest holdout.
|
||||
production = fit(features, config.FEATURE_WEEK_WEATHER)
|
||||
residual = residual_quantiles(eval_frame, preds["week_weather"])
|
||||
@@ -207,6 +235,7 @@ def train_bundle(
|
||||
"features": list(config.FEATURE_WEEK_WEATHER),
|
||||
"holiday_effect": holiday_effect,
|
||||
"ablation": [_round_row(row) for row in results],
|
||||
"per_sensor": per_sensor,
|
||||
"limits": list(config.LIMITS),
|
||||
}
|
||||
return bundle, metrics_payload, results
|
||||
@@ -267,10 +296,6 @@ def run_train() -> dict:
|
||||
bundle, metrics_payload, _ = train_bundle(hourly, features, archive, meta, audit)
|
||||
joblib.dump(bundle, artifacts / "model.joblib")
|
||||
|
||||
(artifacts / "metrics.json").write_text(
|
||||
json.dumps(metrics_payload, indent=2, ensure_ascii=False) + "\n"
|
||||
)
|
||||
|
||||
start = pd.Timestamp.now(tz="UTC").ceil("h")
|
||||
forecasts = _export_forecasts(bundle, forecast, start)
|
||||
(artifacts / "forecasts.json").write_text(
|
||||
@@ -289,6 +314,19 @@ def run_train() -> dict:
|
||||
json.dumps(small_metrics, indent=2, ensure_ascii=False) + "\n"
|
||||
)
|
||||
|
||||
# One self-contained file for the paper: ablation + per-sensor + small model.
|
||||
metrics_payload["small"] = {
|
||||
"model_version": small_metrics["model_version"],
|
||||
"trained_at": small_metrics["trained_at"],
|
||||
"features": small_metrics["features"],
|
||||
"metrics": small_metrics["metrics"],
|
||||
"confusion_matrix": small_metrics["confusion_matrix"],
|
||||
"per_sensor": small_metrics["per_sensor"],
|
||||
}
|
||||
(artifacts / "metrics.json").write_text(
|
||||
json.dumps(metrics_payload, indent=2, ensure_ascii=False) + "\n"
|
||||
)
|
||||
|
||||
log.info(
|
||||
"wrote metrics.json, forecasts.json (%s..%s), model.joblib, small_model.joblib",
|
||||
start,
|
||||
|
||||
@@ -67,6 +67,24 @@ def test_metrics_and_audit():
|
||||
assert metrics["metrics"]["wape"] < metrics["metrics"]["baseline_wape"]
|
||||
assert len(metrics["features"]) == 10
|
||||
assert len(metrics["limits"]) >= 4
|
||||
assert len(metrics["per_sensor"]) == 8
|
||||
assert sum(row["n"] for row in metrics["per_sensor"]) == metrics["metrics"]["n_eval"]
|
||||
assert {row["sensor"] for row in metrics["per_sensor"]} == set(config.USABLE_SENSORS)
|
||||
for row in metrics["per_sensor"]:
|
||||
assert row["mae"] >= 0
|
||||
assert row["baseline_mae"] >= 0
|
||||
assert 0 <= row["wape"] < 100
|
||||
assert row["mae_improvement_pct"] > 0
|
||||
|
||||
small = _load("small_metrics.json")
|
||||
matrix = small["confusion_matrix"]
|
||||
total = matrix["tn"] + matrix["fp"] + matrix["fn"] + matrix["tp"]
|
||||
assert total == small["metrics"]["n_test"]
|
||||
assert sum(row["n"] for row in small["per_sensor"]) == total
|
||||
recomputed = (matrix["tn"] + matrix["tp"]) / total
|
||||
assert abs(recomputed - small["metrics"]["accuracy"]) < 0.001
|
||||
assert metrics["small"]["metrics"] == small["metrics"]
|
||||
assert set(matrix) == {"labels", "tn", "fp", "fn", "tp"}
|
||||
|
||||
audit = _load("data_audit.json")
|
||||
assert audit["all_checks_ok"] is True
|
||||
|
||||
Reference in new issue
Block a user