Merge pull request 'feat: ci docs' (#3) from feat/ci-docs into main
ci / check (push) Successful in 32s

Reviewed-on: #3
This commit was merged in pull request #3.
This commit is contained in:
Noel committed 2026-11-14 08:31:59 +00:00
commit 507f33b7ea
26 files changed
+2102 -86

No files matched your search

+8 -7
View File
@@ -18,13 +18,14 @@ jobs:
- name: Check out - name: Check out
uses: actions/checkout@v4 uses: actions/checkout@v4
- name: Set up Python # setup-python needs the runner toolcache and lsb_release; this Gitea
uses: actions/setup-python@v5 # runner has neither. uv installs its own Python 3.12.
- name: Set up uv
uses: astral-sh/setup-uv@v6
with: with:
version: "0.10.4"
python-version: "3.12" python-version: "3.12"
enable-cache: false
- name: Install uv
run: python -m pip install "uv==0.10.4"
# Same checks as local `make check`, split into steps for readable CI # Same checks as local `make check`, split into steps for readable CI
# failures. No secrets, no image push, no cluster access. # failures. No secrets, no image push, no cluster access.
@@ -32,10 +33,10 @@ jobs:
run: uv sync --frozen run: uv sync --frozen
- name: Static analysis (ruff) - name: Static analysis (ruff)
run: uv run ruff check src tests main.py run: uv run ruff check src tests scripts main.py
- name: Byte-compile - name: Byte-compile
run: uv run python -m compileall src run: uv run python -m compileall src scripts
- name: Tests - name: Tests
run: uv run pytest -q run: uv run pytest -q
+12 -4
View File
@@ -1,11 +1,13 @@
UV ?= uv UV ?= uv
.PHONY: help data train test lint check serve docker clean .PHONY: help data train small figures test lint check serve docker clean
help: help:
@printf '%s\n' \ @printf '%s\n' \
'make data parse/clean the raw CSV, join weather + holidays, export JSON' \ 'make data parse/clean the raw CSV, join weather + holidays, export JSON' \
'make train fit the 28-day holdout model, export metrics/forecasts/model' \ 'make train fit the 28-day holdout model, export metrics/forecasts/model' \
'make small train + print the busy-hour classifier metrics from the raw CSV' \
'make figures regenerate docs/figures/*.svg from artifacts/metrics.json' \
'make test run pytest' \ 'make test run pytest' \
'make lint run ruff static checks' \ 'make lint run ruff static checks' \
'make check CI gate: sync --frozen, ruff, compileall, pytest' \ 'make check CI gate: sync --frozen, ruff, compileall, pytest' \
@@ -19,16 +21,22 @@ data:
train: train:
$(UV) run python -m pulsecast.train $(UV) run python -m pulsecast.train
small:
$(UV) run python scripts/train_small.py
figures:
$(UV) run python scripts/plot_metrics.py
test: test:
$(UV) run pytest $(UV) run pytest
lint: lint:
$(UV) run ruff check src tests main.py $(UV) run ruff check src tests scripts main.py
check: check:
$(UV) sync --frozen $(UV) sync --frozen
$(UV) run ruff check src tests main.py $(UV) run ruff check src tests scripts main.py
$(UV) run python -m compileall src $(UV) run python -m compileall src scripts
$(UV) run pytest -q $(UV) run pytest -q
serve: serve:
+59 -25
View File
@@ -1,11 +1,10 @@
# PulseCast ML # PulseCast ML
Footfall pipeline, weather-aware 7-day forecast, scenario engine, and FastAPI Footfall pipeline, weather-aware 7-day forecast, scenario engine, and FastAPI
service for the PulseCast staffing copilot. service for the PulseCast staffing copilot. It turns raw 15-minute PAX counter
rows into hourly series, joins weather and Baden-Württemberg holidays, trains
The reusable asset is the pipeline: raw 15-minute PAX counters → clean hourly one global model, evaluates it on a 28-day time-based holdout, and serves the
series → observed weather + Baden-Württemberg holidays → global model → resulting JSON artifacts.
honest holdout metrics → JSON artifacts → decision-ready API.
## Results ## Results
@@ -24,9 +23,10 @@ honest holdout metrics → JSON artifacts → decision-ready API.
The seasonal-naive row reproduces the mission reference exactly The seasonal-naive row reproduces the mission reference exactly
(5.40 / 44.7% / 0.462), and the day-ahead weather row matches 3.36 / 27.8%. (5.40 / 44.7% / 0.462), and the day-ahead weather row matches 3.36 / 27.8%.
Weather cuts error by ~19% versus the no-weather 7-day model and ~38% versus Weather cuts error by ~19% versus the no-weather 7-day model and ~38% versus
seasonal naive. The production model is slightly better than the reference seasonal naive. The production 7-day model edges the reference day-ahead
because its seasonal feature uses the previous *observed* same weekday/hour weather row (3.32 vs 3.36 MAE). The two use different features; `lag_week` uses
across sensor gaps; the baseline uses the exact 168-row lag. the previous *observed* same weekday/hour across sensor gaps, while the
seasonal-naive baseline uses the exact 168-row lag.
Small busy-hour classifier (baked for the "train new data" demo): Small busy-hour classifier (baked for the "train new data" demo):
@@ -37,30 +37,35 @@ Small busy-hour classifier (baked for the "train new data" demo):
The classifier predicts `busy` = hourly footfall above the sensor's p75 using The classifier predicts `busy` = hourly footfall above the sensor's p75 using
hour, weekday, weekend, temperature, rain, and holiday. It trains in under a hour, weekday, weekend, temperature, rain, and holiday. It trains in under a
second on the challenge CSV and is retrained from an uploaded CSV through second on the challenge CSV and is retrained from an uploaded CSV through
`POST /train`. `POST /train`. Full definition, confusion matrix, per-sensor table, and limits:
[`docs/small-model.md`](docs/small-model.md). Paper tables and figures:
[`docs/metrics.md`](docs/metrics.md).
## Quick start ## Quick start
Requirements: Python 3.12 and `uv` (the repo also works with a plain venv). Requires Python 3.12 and `uv`.
```bash ```bash
uv sync uv sync
make data # needs the raw CSV, first run also fetches Open-Meteo weather make serve # uses the committed artifacts; http://localhost:8000
make check # same gate as CI: sync --frozen, ruff, compileall, pytest
```
To rebuild the artifacts from raw data:
```bash
make data # needs the raw CSV; first run also fetches Open-Meteo weather
make train # writes artifacts/model.joblib, metrics.json, forecasts.json make train # writes artifacts/model.joblib, metrics.json, forecasts.json
make small # prints the busy-hour classifier metrics + confusion matrix
make figures # regenerates docs/figures/*.svg from metrics.json
make test make test
make check # CI gate: sync --frozen, ruff, compileall, pytest
make serve # http://localhost:8000
``` ```
The raw challenge CSV is **never committed**. Put it at: The raw challenge CSV is **never committed**. The default path is
`/home/noel/Documents/Hackathon_2026/data/raw/schwarz-pax.csv`; set
```text `PULSECAST_RAW_CSV` to use another file. `data/raw/` and `data/cache/` are
/home/noel/Documents/Hackathon_2026/data/raw/schwarz-pax.csv git-ignored. The committed `artifacts/` (JSON + joblib + compact weather frames)
``` let the API and Docker image run without the raw file or network.
or point `PULSECAST_RAW_CSV` at it. `data/raw/` and `data/cache/` are git-ignored.
The generated `artifacts/` (JSON + joblib + compact weather frames) are
committed so the API and Docker image work without the raw file or network.
## Artifacts ## Artifacts
@@ -69,16 +74,18 @@ committed so the API and Docker image work without the raw file or network.
| `artifacts/sensors.json` | 8 sensors: id, name, lat/lon, mean, p95, battery | | `artifacts/sensors.json` | 8 sensors: id, name, lat/lon, mean, p95, battery |
| `artifacts/series.json` + `series/<id>.json` | Hourly series per sensor | | `artifacts/series.json` + `series/<id>.json` | Hourly series per sensor |
| `artifacts/forecasts.json` | 48h + 7d forecasts for 4 scenarios | | `artifacts/forecasts.json` | 48h + 7d forecasts for 4 scenarios |
| `artifacts/metrics.json` | Versions, coverage, headline metrics, ablation, limits | | `artifacts/metrics.json` | Versions, coverage, headline metrics, ablation, per-sensor, small summary, limits |
| `artifacts/small_metrics.json` | Busy classifier: metrics, confusion matrix, per-sensor, thresholds |
| `artifacts/context.json` | Current + 7-day weather, BW holidays | | `artifacts/context.json` | Current + 7-day weather, BW holidays |
| `artifacts/data_audit.json` | Row/sensor checks against `mission/data.md` | | `artifacts/data_audit.json` | Row/device checks for the raw CSV |
| `artifacts/model.joblib` | Baked HistGB forecast bundle | | `artifacts/model.joblib` | Baked HistGB forecast bundle |
| `artifacts/small_model.joblib` | Baked busy classifier | | `artifacts/small_model.joblib` | Baked busy classifier |
| `artifacts/weather_*.parquet` | Baked weather frames for retraining/inference | | `artifacts/weather_*.parquet` | Baked weather frames for retraining/inference |
## API ## API
Shapes follow `CONTRACT.md`. The service is internal; Go exposes it publicly. Response shapes match the Go service contract. The service is internal; Go
exposes it publicly.
| Method | Path | Notes | | Method | Path | Notes |
|---|---|---| |---|---|---|
@@ -118,6 +125,29 @@ and persists to `PULSECAST_MODELS` (default `/tmp/pulsecast/models`). A restart
falls back to the baked model. `model=small` is the fast demo; `model=forecast` falls back to the baked model. `model=small` is the fast demo; `model=forecast`
retrains the global HistGB (slower, still background). retrains the global HistGB (slower, still background).
## End-to-end check
`scripts/e2e_train.py` starts a real uvicorn process on port 18000 with a
control token, then checks the snapshot, every `/predict` combination, every
`/simulate` scenario, auth and error paths, one-job-at-a-time, and both real
training jobs (small + forecast) including hot-swap and `/predict` afterward:
```bash
uv run python scripts/e2e_train.py --port 18000 --report /tmp/e2e.json
```
The unit and contract tests are `make test`; `make check` is the CI gate.
## Docs
| File | Contents |
|---|---|
| `docs/metrics.md` | Paper tables/figures: protocol, headline, ablation, per sensor, uncertainty, small model, claim limits |
| `docs/small-model.md` | Busy-hour target, thresholds, split, metrics, confusion matrix, per sensor, limits |
| `docs/train-e2e.md` | Exact live training-job evidence (small + forecast, hot-swap, errors) |
| `docs/figures/*.svg` | Ablation MAE, per-sensor MAE, small-model confusion matrix (generated) |
| `model_card.md` | Model card: features, split, baseline, ablation, leakage, limitations |
## Docker ## Docker
```bash ```bash
@@ -132,9 +162,11 @@ artifacts baked in, writable paths only under `/tmp`.
| Variable | Default | Use | | Variable | Default | Use |
|---|---|---| |---|---|---|
| `PULSECAST_ROOT` | first parent with `pyproject.toml` | Project root |
| `PULSECAST_RAW_CSV` | challenge path | Raw CSV location | | `PULSECAST_RAW_CSV` | challenge path | Raw CSV location |
| `PULSECAST_ARTIFACTS` | `./artifacts` | Baked JSON/model output | | `PULSECAST_ARTIFACTS` | `./artifacts` | Baked JSON/model output |
| `PULSECAST_CACHE` | `./data/cache` | Parquet/weather cache | | `PULSECAST_CACHE` | `./data/cache` | Parquet/weather cache |
| `PULSECAST_WEATHER` | `PULSECAST_ARTIFACTS` | Baked weather frame location |
| `PULSECAST_MODELS` | `/tmp/pulsecast/models` | Retrained model persistence | | `PULSECAST_MODELS` | `/tmp/pulsecast/models` | Retrained model persistence |
| `PULSECAST_UPLOADS` | `/tmp/pulsecast/uploads` | Multipart upload landing | | `PULSECAST_UPLOADS` | `/tmp/pulsecast/uploads` | Multipart upload landing |
| `CONTROL_TOKEN` | unset | Required by `/train`; service returns 503 if unset | | `CONTROL_TOKEN` | unset | Required by `/train`; service returns 503 if unset |
@@ -153,6 +185,8 @@ src/pulsecast/
small.py busy classifier + flexible CSV parser small.py busy classifier + flexible CSV parser
retrain.py runtime forecast retraining from upload retrain.py runtime forecast retraining from upload
server.py FastAPI app and training job manager server.py FastAPI app and training job manager
scripts/ train_small.py, plot_metrics.py, e2e_train.py
docs/ metrics.md, small-model.md, train-e2e.md, generated figures
tests/ schema, bootstrap, health, prediction, small-training job tests/ schema, bootstrap, health, prediction, small-training job
model_card.md features, split, baseline, ablation, limits, leakage model_card.md features, split, baseline, ablation, limits, leakage
``` ```
+1 -1
View File
@@ -1 +1 @@
{"data_version": "2026-11-07T23:51:00Z", "model_version": "histgb-2026-11-14", "context_version": "2026-11-14T03:47:09Z", "generated_at": "2026-11-14T04:03:38Z", "forecast_start": "2026-11-14T05:00:00Z", "forecasts": {"density-01": {"48h": {"base": [{"t": "2026-11-14T05:00:00Z", "v": 1.17, "lo": 0.0, "hi": 6.04}, {"t": "2026-11-14T06:00:00Z", "v": 1.95, "lo": 0.0, "hi": 6.83}, {"t": "2026-11-14T07:00:00Z", "v": 2.94, "lo": 0.0, "hi": 7.81}, {"t": "2026-11-14T08:00:00Z", "v": 4.08, "lo": 0.0, "hi": 8.96}, {"t": "2026-11-14T09:00:00Z", "v": 5.68, "lo": 0.85, "hi": 10.55}, {"t": "2026-11-14T10:00:00Z", "v": 8.5, "lo": 3.67, "hi": 13.38}, {"t": "2026-11-14T11:00:00Z", "v": 9.66, "lo": 4.84, "hi": 14.54}, {"t": "2026-11-14T12:00:00Z", "v": 11.39, "lo": 6.56, "hi": 16.26}, {"t": "2026-11-14T13:00:00Z", "v": 11.32, "lo": 6.5, "hi": 16.2}, {"t": "2026-11-14T14:00:00Z", "v": 12.63, "lo": 7.81, "hi": 17.51}, {"t": "2026-11-14T15:00:00Z", "v": 14.78, "lo": 9.96, "hi": 19.66}, {"t": "2026-11-14T16:00:00Z", "v": 14.21, "lo": 9.39, "hi": 19.09}, {"t": "2026-11-14T17:00:00Z", "v": 14.52, "lo": 9.7, "hi": 19.4}, {"t": "2026-11-14T18:00:00Z", "v": 15.67, "lo": 10.84, "hi": 20.55}, {"t": "2026-11-14T19:00:00Z", "v": 14.37, "lo": 9.55, "hi": 19.25}, {"t": "2026-11-14T20:00:00Z", "v": 12.64, "lo": 7.81, "hi": 17.51}, {"t": "2026-11-14T21:00:00Z", "v": 14.52, "lo": 9.7, "hi": 19.4}, {"t": "2026-11-14T22:00:00Z", "v": 12.33, "lo": 7.5, "hi": 17.21}, {"t": "2026-11-14T23:00:00Z", "v": 9.87, "lo": 5.04, "hi": 14.74}, {"t": "2026-11-15T00:00:00Z", "v": 3.94, "lo": 0.0, "hi": 8.82}, {"t": "2026-11-15T01:00:00Z", "v": 1.34, "lo": 0.0, "hi": 6.22}, {"t": "2026-11-15T02:00:00Z", "v": 1.13, "lo": 0.0, "hi": 6.01}, {"t": "2026-11-15T03:00:00Z", "v": 0.99, "lo": 0.0, "hi": 5.86}, {"t": "2026-11-15T04:00:00Z", "v": 0.99, "lo": 0.0, "hi": 5.86}, {"t": "2026-11-15T05:00:00Z", "v": 1.01, "lo": 0.0, "hi": 5.88}, {"t": "2026-11-15T06:00:00Z", "v": 1.04, "lo": 0.0, "hi": 5.92}, {"t": "2026-11-15T07:00:00Z", "v": 1.55, "lo": 0.0, "hi": 6.43}, {"t": "2026-11-15T08:00:00Z", "v": 1.65, "lo": 0.0, "hi": 6.53}, {"t": "2026-11-15T09:00:00Z", "v": 5.61, "lo": 0.78, "hi": 10.48}, {"t": "2026-11-15T10:00:00Z", "v": 6.5, "lo": 1.68, "hi": 11.38}, {"t": "2026-11-15T11:00:00Z", "v": 7.51, "lo": 2.69, "hi": 12.39}, {"t": "2026-11-15T12:00:00Z", "v": 9.58, "lo": 4.76, "hi": 14.46}, {"t": "2026-11-15T13:00:00Z", "v": 11.7, "lo": 6.88, "hi": 16.58}, {"t": "2026-11-15T14:00:00Z", "v": 18.36, "lo": 13.54, "hi": 23.24}, {"t": "2026-11-15T15:00:00Z", "v": 17.33, "lo": 12.5, "hi": 22.21}, {"t": "2026-11-15T16:00:00Z", "v": 16.04, "lo": 11.21, "hi": 20.91}, {"t": "2026-11-15T17:00:00Z", "v": 11.34, "lo": 6.51, "hi": 16.21}, {"t": "2026-11-15T18:00:00Z", "v": 10.87, "lo": 6.05, "hi": 15.75}, {"t": "2026-11-15T19:00:00Z", "v": 8.45, "lo": 3.63, "hi": 13.33}, {"t": "2026-11-15T20:00:00Z", "v": 5.82, "lo": 0.99, "hi": 10.7}, {"t": "2026-11-15T21:00:00Z", "v": 3.71, "lo": 0.0, "hi": 8.59}, {"t": "2026-11-15T22:00:00Z", "v": 2.11, "lo": 0.0, "hi": 6.99}, {"t": "2026-11-15T23:00:00Z", "v": 1.47, "lo": 0.0, "hi": 6.34}, {"t": "2026-11-16T00:00:00Z", "v": 0.91, "lo": 0.0, "hi": 5.79}, {"t": "2026-11-16T01:00:00Z", "v": 0.88, "lo": 0.0, "hi": 5.76}, {"t": "2026-11-16T02:00:00Z", "v": 1.15, "lo": 0.0, "hi": 6.03}, {"t": "2026-11-16T03:00:00Z", "v": 1.15, "lo": 0.0, "hi": 6.03}, {"t": "2026-11-16T04:00:00Z", "v": 1.74, "lo": 0.0, "hi": 6.62}], "sunny-saturday-market": [{"t": "2026-11-14T05:00:00Z", "v": 1.17, "lo": 0.0, "hi": 6.04}, {"t": "2026-11-14T06:00:00Z", "v": 1.95, "lo": 0.0, "hi": 6.83}, {"t": "2026-11-14T07:00:00Z", "v": 2.94, "lo": 0.0, "hi": 7.81}, {"t": "2026-11-14T08:00:00Z", "v": 4.08, "lo": 0.0, "hi": 8.96}, {"t": "2026-11-14T09:00:00Z", "v": 7.63, "lo": 2.81, "hi": 12.51}, {"t": "2026-11-14T10:00:00Z", "v": 10.14, "lo": 5.32, "hi": 15.02}, {"t": "2026-11-14T11:00:00Z", "v": 11.16, "lo": 6.34, "hi": 16.04}, {"t": "2026-11-14T12:00:00Z", "v": 12.88, "lo": 8.05, "hi": 17.76}, {"t": "2026-11-14T13:00:00Z", "v": 12.02, "lo": 7.2, "hi": 16.9}, {"t": "2026-11-14T14:00:00Z", "v": 17.24, "lo": 12.42, "hi": 22.12}, {"t": "2026-11-14T15:00:00Z", "v": 15.8, "lo": 10.97, "hi": 20.67}, {"t": "2026-11-14T16:00:00Z", "v": 15.97, "lo": 11.15, "hi": 20.85}, {"t": "2026-11-14T17:00:00Z", "v": 15.77, "lo": 10.94, "hi": 20.65}, {"t": "2026-11-14T18:00:00Z", "v": 16.1, "lo": 11.27, "hi": 20.97}, {"t": "2026-11-14T19:00:00Z", "v": 17.49, "lo": 12.67, "hi": 22.37}, {"t": "2026-11-14T20:00:00Z", "v": 12.64, "lo": 7.81, "hi": 17.51}, {"t": "2026-11-14T21:00:00Z", "v": 14.52, "lo": 9.7, "hi": 19.4}, {"t": "2026-11-14T22:00:00Z", "v": 12.33, "lo": 7.5, "hi": 17.21}, {"t": "2026-11-14T23:00:00Z", "v": 9.87, "lo": 5.04, "hi": 14.74}, {"t": "2026-11-15T00:00:00Z", "v": 3.94, "lo": 0.0, "hi": 8.82}, {"t": "2026-11-15T01:00:00Z", "v": 1.34, "lo": 0.0, "hi": 6.22}, {"t": "2026-11-15T02:00:00Z", "v": 1.13, "lo": 0.0, "hi": 6.01}, {"t": "2026-11-15T03:00:00Z", "v": 0.99, "lo": 0.0, "hi": 5.86}, {"Line truncated {"data_version": "2026-11-07T23:51:00Z", "model_version": "histgb-2026-11-14", "context_version": "2026-11-14T03:47:09Z", "generated_at": "2026-11-14T05:27:22Z", "forecast_start": "2026-11-14T06:00:00Z", "forecasts": {"density-01": {"48h": {"base": [{"t": "2026-11-14T06:00:00Z", "v": 1.95, "lo": 0.0, "hi": 6.83}, {"t": "2026-11-14T07:00:00Z", "v": 2.94, "lo": 0.0, "hi": 7.81}, {"t": "2026-11-14T08:00:00Z", "v": 4.08, "lo": 0.0, "hi": 8.96}, {"t": "2026-11-14T09:00:00Z", "v": 5.68, "lo": 0.85, "hi": 10.55}, {"t": "2026-11-14T10:00:00Z", "v": 8.5, "lo": 3.67, "hi": 13.38}, {"t": "2026-11-14T11:00:00Z", "v": 9.66, "lo": 4.84, "hi": 14.54}, {"t": "2026-11-14T12:00:00Z", "v": 11.39, "lo": 6.56, "hi": 16.26}, {"t": "2026-11-14T13:00:00Z", "v": 11.32, "lo": 6.5, "hi": 16.2}, {"t": "2026-11-14T14:00:00Z", "v": 12.63, "lo": 7.81, "hi": 17.51}, {"t": "2026-11-14T15:00:00Z", "v": 14.78, "lo": 9.96, "hi": 19.66}, {"t": "2026-11-14T16:00:00Z", "v": 14.21, "lo": 9.39, "hi": 19.09}, {"t": "2026-11-14T17:00:00Z", "v": 14.52, "lo": 9.7, "hi": 19.4}, {"t": "2026-11-14T18:00:00Z", "v": 15.67, "lo": 10.84, "hi": 20.55}, {"t": "2026-11-14T19:00:00Z", "v": 14.37, "lo": 9.55, "hi": 19.25}, {"t": "2026-11-14T20:00:00Z", "v": 12.64, "lo": 7.81, "hi": 17.51}, {"t": "2026-11-14T21:00:00Z", "v": 14.52, "lo": 9.7, "hi": 19.4}, {"t": "2026-11-14T22:00:00Z", "v": 12.33, "lo": 7.5, "hi": 17.21}, {"t": "2026-11-14T23:00:00Z", "v": 9.87, "lo": 5.04, "hi": 14.74}, {"t": "2026-11-15T00:00:00Z", "v": 3.94, "lo": 0.0, "hi": 8.82}, {"t": "2026-11-15T01:00:00Z", "v": 1.34, "lo": 0.0, "hi": 6.22}, {"t": "2026-11-15T02:00:00Z", "v": 1.13, "lo": 0.0, "hi": 6.01}, {"t": "2026-11-15T03:00:00Z", "v": 0.99, "lo": 0.0, "hi": 5.86}, {"t": "2026-11-15T04:00:00Z", "v": 0.99, "lo": 0.0, "hi": 5.86}, {"t": "2026-11-15T05:00:00Z", "v": 1.01, "lo": 0.0, "hi": 5.88}, {"t": "2026-11-15T06:00:00Z", "v": 1.04, "lo": 0.0, "hi": 5.92}, {"t": "2026-11-15T07:00:00Z", "v": 1.55, "lo": 0.0, "hi": 6.43}, {"t": "2026-11-15T08:00:00Z", "v": 1.65, "lo": 0.0, "hi": 6.53}, {"t": "2026-11-15T09:00:00Z", "v": 5.61, "lo": 0.78, "hi": 10.48}, {"t": "2026-11-15T10:00:00Z", "v": 6.5, "lo": 1.68, "hi": 11.38}, {"t": "2026-11-15T11:00:00Z", "v": 7.51, "lo": 2.69, "hi": 12.39}, {"t": "2026-11-15T12:00:00Z", "v": 9.58, "lo": 4.76, "hi": 14.46}, {"t": "2026-11-15T13:00:00Z", "v": 11.7, "lo": 6.88, "hi": 16.58}, {"t": "2026-11-15T14:00:00Z", "v": 18.36, "lo": 13.54, "hi": 23.24}, {"t": "2026-11-15T15:00:00Z", "v": 17.33, "lo": 12.5, "hi": 22.21}, {"t": "2026-11-15T16:00:00Z", "v": 16.04, "lo": 11.21, "hi": 20.91}, {"t": "2026-11-15T17:00:00Z", "v": 11.34, "lo": 6.51, "hi": 16.21}, {"t": "2026-11-15T18:00:00Z", "v": 10.87, "lo": 6.05, "hi": 15.75}, {"t": "2026-11-15T19:00:00Z", "v": 8.45, "lo": 3.63, "hi": 13.33}, {"t": "2026-11-15T20:00:00Z", "v": 5.82, "lo": 0.99, "hi": 10.7}, {"t": "2026-11-15T21:00:00Z", "v": 3.71, "lo": 0.0, "hi": 8.59}, {"t": "2026-11-15T22:00:00Z", "v": 2.11, "lo": 0.0, "hi": 6.99}, {"t": "2026-11-15T23:00:00Z", "v": 1.47, "lo": 0.0, "hi": 6.34}, {"t": "2026-11-16T00:00:00Z", "v": 0.91, "lo": 0.0, "hi": 5.79}, {"t": "2026-11-16T01:00:00Z", "v": 0.88, "lo": 0.0, "hi": 5.76}, {"t": "2026-11-16T02:00:00Z", "v": 1.15, "lo": 0.0, "hi": 6.03}, {"t": "2026-11-16T03:00:00Z", "v": 1.15, "lo": 0.0, "hi": 6.03}, {"t": "2026-11-16T04:00:00Z", "v": 1.74, "lo": 0.0, "hi": 6.62}, {"t": "2026-11-16T05:00:00Z", "v": 1.58, "lo": 0.0, "hi": 6.46}], "sunny-saturday-market": [{"t": "2026-11-14T06:00:00Z", "v": 1.95, "lo": 0.0, "hi": 6.83}, {"t": "2026-11-14T07:00:00Z", "v": 2.94, "lo": 0.0, "hi": 7.81}, {"t": "2026-11-14T08:00:00Z", "v": 4.08, "lo": 0.0, "hi": 8.96}, {"t": "2026-11-14T09:00:00Z", "v": 7.63, "lo": 2.81, "hi": 12.51}, {"t": "2026-11-14T10:00:00Z", "v": 10.14, "lo": 5.32, "hi": 15.02}, {"t": "2026-11-14T11:00:00Z", "v": 11.16, "lo": 6.34, "hi": 16.04}, {"t": "2026-11-14T12:00:00Z", "v": 12.88, "lo": 8.05, "hi": 17.76}, {"t": "2026-11-14T13:00:00Z", "v": 12.02, "lo": 7.2, "hi": 16.9}, {"t": "2026-11-14T14:00:00Z", "v": 17.24, "lo": 12.42, "hi": 22.12}, {"t": "2026-11-14T15:00:00Z", "v": 15.8, "lo": 10.97, "hi": 20.67}, {"t": "2026-11-14T16:00:00Z", "v": 15.97, "lo": 11.15, "hi": 20.85}, {"t": "2026-11-14T17:00:00Z", "v": 15.77, "lo": 10.94, "hi": 20.65}, {"t": "2026-11-14T18:00:00Z", "v": 16.1, "lo": 11.27, "hi": 20.97}, {"t": "2026-11-14T19:00:00Z", "v": 17.49, "lo": 12.67, "hi": 22.37}, {"t": "2026-11-14T20:00:00Z", "v": 12.64, "lo": 7.81, "hi": 17.51}, {"t": "2026-11-14T21:00:00Z", "v": 14.52, "lo": 9.7, "hi": 19.4}, {"t": "2026-11-14T22:00:00Z", "v": 12.33, "lo": 7.5, "hi": 17.21}, {"t": "2026-11-14T23:00:00Z", "v": 9.87, "lo": 5.04, "hi": 14.74}, {"t": "2026-11-15T00:00:00Z", "v": 3.94, "lo": 0.0, "hi": 8.82}, {"t": "2026-11-15T01:00:00Z", "v": 1.34, "lo": 0.0, "hi": 6.22}, {"t": "2026-11-15T02:00:00Z", "v": 1.13, "lo": 0.0, "hi": 6.01}, {"t": "2026-11-15T03:00:00Z", "v": 0.99, "lo": 0.0, "hi": 5.86}, {"t": "2026-11-15T04:00:00Z", "v": 0.99, "lo": 0.0, "hi": 5.86}, {"Line truncated
+206 -1
View File
@@ -19,7 +19,7 @@
"baseline_r2": 0.462, "baseline_r2": 0.462,
"n_eval": 5350 "n_eval": 5350
}, },
"trained_at": "2026-11-14T04:03:37Z", "trained_at": "2026-11-14T05:27:21Z",
"train_from": "2026-06-12T00:00:00Z", "train_from": "2026-06-12T00:00:00Z",
"train_to": "2026-11-07T23:00:00Z", "train_to": "2026-11-07T23:00:00Z",
"test_from": "2026-10-10T23:00:00Z", "test_from": "2026-10-10T23:00:00Z",
@@ -90,11 +90,216 @@
"n": 5350 "n": 5350
} }
], ],
"per_sensor": [
{
"sensor": "density-01",
"name": "Density 01",
"n": 669,
"mae": 2.26,
"wape": 30.8,
"r2": 0.722,
"baseline_mae": 3.58,
"baseline_wape": 48.8,
"baseline_r2": 0.238,
"mae_improvement_pct": 36.8
},
{
"sensor": "hn-goetzentuem-01",
"name": "Götzenturm",
"n": 669,
"mae": 2.4,
"wape": 26.9,
"r2": 0.718,
"baseline_mae": 3.84,
"baseline_wape": 43.0,
"baseline_r2": 0.281,
"mae_improvement_pct": 37.5
},
{
"sensor": "hn-hagenbuchersee-01",
"name": "Hagenbuchersee 1",
"n": 669,
"mae": 2.47,
"wape": 38.6,
"r2": 0.602,
"baseline_mae": 4.01,
"baseline_wape": 62.8,
"baseline_r2": -0.212,
"mae_improvement_pct": 38.5
},
{
"sensor": "hn-hagenbuchersee-02",
"name": "Hagenbuchersee 2",
"n": 669,
"mae": 4.12,
"wape": 31.1,
"r2": 0.731,
"baseline_mae": 6.49,
"baseline_wape": 48.9,
"baseline_r2": 0.337,
"mae_improvement_pct": 36.5
},
{
"sensor": "hn-marahaus-01",
"name": "Marahaus",
"n": 669,
"mae": 2.67,
"wape": 29.8,
"r2": 0.671,
"baseline_mae": 4.57,
"baseline_wape": 51.0,
"baseline_r2": -0.045,
"mae_improvement_pct": 41.7
},
{
"sensor": "hn-neckarufer-01",
"name": "Neckarufer",
"n": 668,
"mae": 3.63,
"wape": 28.2,
"r2": 0.695,
"baseline_mae": 5.89,
"baseline_wape": 45.8,
"baseline_r2": 0.11,
"mae_improvement_pct": 38.4
},
{
"sensor": "hn-soleo-01",
"name": "Soleo",
"n": 669,
"mae": 3.19,
"wape": 20.2,
"r2": 0.548,
"baseline_mae": 4.4,
"baseline_wape": 28.0,
"baseline_r2": 0.163,
"mae_improvement_pct": 27.7
},
{
"sensor": "hn-stadtgalerie-01",
"name": "Stadtgalerie",
"n": 668,
"mae": 5.83,
"wape": 25.3,
"r2": 0.765,
"baseline_mae": 10.39,
"baseline_wape": 45.1,
"baseline_r2": 0.48,
"mae_improvement_pct": 43.9
}
],
"limits": [ "limits": [
"estimated devices, not people", "estimated devices, not people",
"8 sensors in a ~580 m inner-city cluster", "8 sensors in a ~580 m inner-city cluster",
"5 months of multi-sensor data; no annual seasonality claim", "5 months of multi-sensor data; no annual seasonality claim",
"observed weather in training, forecast weather in scenarios", "observed weather in training, forecast weather in scenarios",
"holidays are context, not a large accuracy claim" "holidays are context, not a large accuracy claim"
],
"small": {
"model_version": "logit-20261114-052723",
"trained_at": "2026-11-14T05:27:23Z",
"features": [
"hour",
"weekday",
"is_weekend",
"temp_c",
"precip_mm",
"is_holiday"
],
"metrics": {
"accuracy": 0.708,
"precision": 0.36,
"recall": 0.574,
"f1": 0.443,
"roc_auc": 0.726,
"n_train": 18242,
"n_test": 5710,
"positive_rate": 0.202,
"sensors": 8
},
"confusion_matrix": {
"labels": {
"0": "not busy",
"1": "busy"
},
"tn": 3377,
"fp": 1178,
"fn": 492,
"tp": 663
},
"per_sensor": [
{
"sensor": "density-01",
"n": 714,
"positive_rate": 0.111,
"accuracy": 0.761,
"precision": 0.3,
"recall": 0.873,
"f1": 0.447
},
{
"sensor": "hn-goetzentuem-01",
"n": 714,
"positive_rate": 0.197,
"accuracy": 0.732,
"precision": 0.391,
"recall": 0.638,
"f1": 0.485
},
{
"sensor": "hn-hagenbuchersee-01",
"n": 714,
"positive_rate": 0.049,
"accuracy": 0.724,
"precision": 0.148,
"recall": 0.971,
"f1": 0.257
},
{
"sensor": "hn-hagenbuchersee-02",
"n": 714,
"positive_rate": 0.123,
"accuracy": 0.783,
"precision": 0.355,
"recall": 0.932,
"f1": 0.514
},
{
"sensor": "hn-marahaus-01",
"n": 714,
"positive_rate": 0.055,
"accuracy": 0.71,
"precision": 0.135,
"recall": 0.795,
"f1": 0.23
},
{
"sensor": "hn-neckarufer-01",
"n": 713,
"positive_rate": 0.09,
"accuracy": 0.764,
"precision": 0.274,
"recall": 0.984,
"f1": 0.429
},
{
"sensor": "hn-soleo-01",
"n": 714,
"positive_rate": 0.704,
"accuracy": 0.581,
"precision": 0.943,
"recall": 0.431,
"f1": 0.592
},
{
"sensor": "hn-stadtgalerie-01",
"n": 713,
"positive_rate": 0.289,
"accuracy": 0.604,
"precision": 0.335,
"recall": 0.374,
"f1": 0.353
}
] ]
} }
}
Binary file not shown.
+86 -2
View File
@@ -1,6 +1,6 @@
{ {
"model_version": "logit-20261114-040339", "model_version": "logit-20261114-052723",
"trained_at": "2026-11-14T04:03:39Z", "trained_at": "2026-11-14T05:27:23Z",
"features": [ "features": [
"hour", "hour",
"weekday", "weekday",
@@ -33,5 +33,89 @@
"n_test": 5710, "n_test": 5710,
"positive_rate": 0.202, "positive_rate": 0.202,
"sensors": 8 "sensors": 8
},
"confusion_matrix": {
"labels": {
"0": "not busy",
"1": "busy"
},
"tn": 3377,
"fp": 1178,
"fn": 492,
"tp": 663
},
"per_sensor": [
{
"sensor": "density-01",
"n": 714,
"positive_rate": 0.111,
"accuracy": 0.761,
"precision": 0.3,
"recall": 0.873,
"f1": 0.447
},
{
"sensor": "hn-goetzentuem-01",
"n": 714,
"positive_rate": 0.197,
"accuracy": 0.732,
"precision": 0.391,
"recall": 0.638,
"f1": 0.485
},
{
"sensor": "hn-hagenbuchersee-01",
"n": 714,
"positive_rate": 0.049,
"accuracy": 0.724,
"precision": 0.148,
"recall": 0.971,
"f1": 0.257
},
{
"sensor": "hn-hagenbuchersee-02",
"n": 714,
"positive_rate": 0.123,
"accuracy": 0.783,
"precision": 0.355,
"recall": 0.932,
"f1": 0.514
},
{
"sensor": "hn-marahaus-01",
"n": 714,
"positive_rate": 0.055,
"accuracy": 0.71,
"precision": 0.135,
"recall": 0.795,
"f1": 0.23
},
{
"sensor": "hn-neckarufer-01",
"n": 713,
"positive_rate": 0.09,
"accuracy": 0.764,
"precision": 0.274,
"recall": 0.984,
"f1": 0.429
},
{
"sensor": "hn-soleo-01",
"n": 714,
"positive_rate": 0.704,
"accuracy": 0.581,
"precision": 0.943,
"recall": 0.431,
"f1": 0.592
},
{
"sensor": "hn-stadtgalerie-01",
"n": 713,
"positive_rate": 0.289,
"accuracy": 0.604,
"precision": 0.335,
"recall": 0.374,
"f1": 0.353
} }
]
} }
Binary file not shown.
+36
View File
@@ -0,0 +1,36 @@
<svg xmlns="http://www.w3.org/2000/svg" width="960" height="370" viewBox="0 0 960 370" role="img" aria-label="Ablation: holdout MAE by feature set">
<rect width="100%" height="100%" fill="#ffffff"/>
<g font-family="-apple-system, Segoe UI, Helvetica, Arial, sans-serif">
<text x="24" y="34" font-size="20" font-weight="600" fill="#0f172a">Ablation: holdout MAE by feature set</text>
<text x="24" y="56" font-size="13" fill="#475569">Holdout 2026-10-10T23:00:00Z .. 2026-11-07T23:00:00Z, n=5350</text>
<line x1="230.0" y1="68" x2="230.0" y2="314" stroke="#e2e8f0" stroke-width="1"/>
<text x="230.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">0.0</text>
<line x1="388.5" y1="68" x2="388.5" y2="314" stroke="#e2e8f0" stroke-width="1"/>
<text x="388.5" y="62" font-size="11" fill="#64748b" text-anchor="middle">1.5</text>
<line x1="547.0" y1="68" x2="547.0" y2="314" stroke="#e2e8f0" stroke-width="1"/>
<text x="547.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">2.9</text>
<line x1="705.5" y1="68" x2="705.5" y2="314" stroke="#e2e8f0" stroke-width="1"/>
<text x="705.5" y="62" font-size="11" fill="#64748b" text-anchor="middle">4.4</text>
<line x1="864.0" y1="68" x2="864.0" y2="314" stroke="#e2e8f0" stroke-width="1"/>
<text x="864.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">5.8</text>
<text x="218" y="106" font-size="12" fill="#0f172a" text-anchor="end">seasonal naive</text>
<rect x="230" y="92" width="587.0" height="22" fill="#2563eb"/>
<text x="823.0" y="107" font-size="12" fill="#0f172a">5.40</text>
<text x="218" y="144" font-size="12" fill="#0f172a" text-anchor="end">day no weather</text>
<rect x="230" y="130" width="430.5" height="22" fill="#2563eb"/>
<text x="666.5" y="145" font-size="12" fill="#0f172a">3.96</text>
<text x="218" y="182" font-size="12" fill="#0f172a" text-anchor="end">day holiday</text>
<rect x="230" y="168" width="417.4" height="22" fill="#2563eb"/>
<text x="653.4" y="183" font-size="12" fill="#0f172a">3.84</text>
<text x="218" y="220" font-size="12" fill="#0f172a" text-anchor="end">day weather</text>
<rect x="230" y="206" width="365.3" height="22" fill="#2563eb"/>
<text x="601.3" y="221" font-size="12" fill="#0f172a">3.36</text>
<text x="218" y="258" font-size="12" fill="#0f172a" text-anchor="end">week no weather</text>
<rect x="230" y="244" width="443.5" height="22" fill="#2563eb"/>
<text x="679.5" y="259" font-size="12" fill="#0f172a">4.08</text>
<text x="218" y="296" font-size="12" fill="#0f172a" text-anchor="end">week weather</text>
<rect x="230" y="282" width="360.9" height="22" fill="#0f766e"/>
<text x="596.9" y="297" font-size="12" fill="#0f172a">3.32</text>
<text x="24" y="358" font-size="11" fill="#64748b">Source: artifacts/metrics.json · PulseCast 28-day time-based holdout</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 2.7 KiB

+60
View File
@@ -0,0 +1,60 @@
<svg xmlns="http://www.w3.org/2000/svg" width="960" height="446" viewBox="0 0 960 446" role="img" aria-label="Per-sensor holdout MAE: production model vs seasonal naive">
<rect width="100%" height="100%" fill="#ffffff"/>
<g font-family="-apple-system, Segoe UI, Helvetica, Arial, sans-serif">
<text x="24" y="34" font-size="20" font-weight="600" fill="#0f172a">Per-sensor holdout MAE: production model vs seasonal naive</text>
<text x="24" y="56" font-size="13" fill="#475569">Holdout 2026-10-10T23:00:00Z .. 2026-11-07T23:00:00Z, n=5350</text>
<line x1="230.0" y1="68" x2="230.0" y2="390" stroke="#e2e8f0" stroke-width="1"/>
<text x="230.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">0.0</text>
<line x1="388.5" y1="68" x2="388.5" y2="390" stroke="#e2e8f0" stroke-width="1"/>
<text x="388.5" y="62" font-size="11" fill="#64748b" text-anchor="middle">2.8</text>
<line x1="547.0" y1="68" x2="547.0" y2="390" stroke="#e2e8f0" stroke-width="1"/>
<text x="547.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">5.6</text>
<line x1="705.5" y1="68" x2="705.5" y2="390" stroke="#e2e8f0" stroke-width="1"/>
<text x="705.5" y="62" font-size="11" fill="#64748b" text-anchor="middle">8.4</text>
<line x1="864.0" y1="68" x2="864.0" y2="390" stroke="#e2e8f0" stroke-width="1"/>
<text x="864.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">11.2</text>
<rect x="230" y="52" width="10" height="10" fill="#94a3b8"/><text x="246" y="61" font-size="12" fill="#475569">seasonal naive</text>
<rect x="370" y="52" width="10" height="10" fill="#2563eb"/><text x="386" y="61" font-size="12" fill="#475569">weather-7d</text>
<text x="218" y="106" font-size="12" fill="#0f172a" text-anchor="end">Density 01</text>
<rect x="230" y="89" width="202.3" height="12" fill="#94a3b8"/>
<text x="438.3" y="99" font-size="11" fill="#64748b">3.58</text>
<rect x="230" y="104" width="127.7" height="12" fill="#2563eb"/>
<text x="363.7" y="114" font-size="11" fill="#0f172a">2.26</text>
<text x="218" y="144" font-size="12" fill="#0f172a" text-anchor="end">Götzenturm</text>
<rect x="230" y="127" width="217.0" height="12" fill="#94a3b8"/>
<text x="453.0" y="137" font-size="11" fill="#64748b">3.84</text>
<rect x="230" y="142" width="135.6" height="12" fill="#2563eb"/>
<text x="371.6" y="152" font-size="11" fill="#0f172a">2.40</text>
<text x="218" y="182" font-size="12" fill="#0f172a" text-anchor="end">Hagenbuchersee 1</text>
<rect x="230" y="165" width="226.6" height="12" fill="#94a3b8"/>
<text x="462.6" y="175" font-size="11" fill="#64748b">4.01</text>
<rect x="230" y="180" width="139.6" height="12" fill="#2563eb"/>
<text x="375.6" y="190" font-size="11" fill="#0f172a">2.47</text>
<text x="218" y="220" font-size="12" fill="#0f172a" text-anchor="end">Hagenbuchersee 2</text>
<rect x="230" y="203" width="366.7" height="12" fill="#94a3b8"/>
<text x="602.7" y="213" font-size="11" fill="#64748b">6.49</text>
<rect x="230" y="218" width="232.8" height="12" fill="#2563eb"/>
<text x="468.8" y="228" font-size="11" fill="#0f172a">4.12</text>
<text x="218" y="258" font-size="12" fill="#0f172a" text-anchor="end">Marahaus</text>
<rect x="230" y="241" width="258.2" height="12" fill="#94a3b8"/>
<text x="494.2" y="251" font-size="11" fill="#64748b">4.57</text>
<rect x="230" y="256" width="150.9" height="12" fill="#2563eb"/>
<text x="386.9" y="266" font-size="11" fill="#0f172a">2.67</text>
<text x="218" y="296" font-size="12" fill="#0f172a" text-anchor="end">Neckarufer</text>
<rect x="230" y="279" width="332.8" height="12" fill="#94a3b8"/>
<text x="568.8" y="289" font-size="11" fill="#64748b">5.89</text>
<rect x="230" y="294" width="205.1" height="12" fill="#2563eb"/>
<text x="441.1" y="304" font-size="11" fill="#0f172a">3.63</text>
<text x="218" y="334" font-size="12" fill="#0f172a" text-anchor="end">Soleo</text>
<rect x="230" y="317" width="248.6" height="12" fill="#94a3b8"/>
<text x="484.6" y="327" font-size="11" fill="#64748b">4.40</text>
<rect x="230" y="332" width="180.2" height="12" fill="#2563eb"/>
<text x="416.2" y="342" font-size="11" fill="#0f172a">3.19</text>
<text x="218" y="372" font-size="12" fill="#0f172a" text-anchor="end">Stadtgalerie</text>
<rect x="230" y="355" width="587.0" height="12" fill="#94a3b8"/>
<text x="823.0" y="365" font-size="11" fill="#64748b">10.39</text>
<rect x="230" y="370" width="329.4" height="12" fill="#2563eb"/>
<text x="565.4" y="380" font-size="11" fill="#0f172a">5.83</text>
<text x="24" y="434" font-size="11" fill="#64748b">Source: artifacts/metrics.json · PulseCast 28-day time-based holdout</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 4.5 KiB

+24
View File
@@ -0,0 +1,24 @@
<svg xmlns="http://www.w3.org/2000/svg" width="760" height="520" viewBox="0 0 760 520" role="img" aria-label="Small busy-hour classifier: holdout confusion matrix">
<rect width="100%" height="100%" fill="#ffffff"/>
<g font-family="-apple-system, Segoe UI, Helvetica, Arial, sans-serif">
<text x="24" y="34" font-size="20" font-weight="600" fill="#0f172a">Small busy-hour classifier: holdout confusion matrix</text>
<text x="24" y="56" font-size="13" fill="#475569">n=5,710 · accuracy 0.708 · precision 0.360 · recall 0.574 · F1 0.443 · ROC-AUC 0.726</text>
<text x="295.0" y="114" font-size="13" fill="#475569" text-anchor="middle">Predicted not busy</text>
<text x="485.0" y="114" font-size="13" fill="#475569" text-anchor="middle">Predicted busy</text>
<text x="182" y="225.0" font-size="13" fill="#475569" text-anchor="end">Actually not busy</text>
<text x="182" y="415.0" font-size="13" fill="#475569" text-anchor="end">Actually busy</text>
<rect x="200" y="130" width="190" height="190" fill="#1d4ed8" stroke="#ffffff" stroke-width="2"/>
<text x="295.0" y="217.0" font-size="30" font-weight="600" fill="#ffffff" text-anchor="middle">3,377</text>
<text x="295.0" y="245.0" font-size="13" fill="#ffffff" text-anchor="middle">TN · 74.1% of row</text>
<rect x="390" y="130" width="190" height="190" fill="#8aa7ee" stroke="#ffffff" stroke-width="2"/>
<text x="485.0" y="217.0" font-size="30" font-weight="600" fill="#0f172a" text-anchor="middle">1,178</text>
<text x="485.0" y="245.0" font-size="13" fill="#0f172a" text-anchor="middle">FP · 25.9% of row</text>
<rect x="200" y="320" width="190" height="190" fill="#acc3f5" stroke="#ffffff" stroke-width="2"/>
<text x="295.0" y="407.0" font-size="30" font-weight="600" fill="#0f172a" text-anchor="middle">492</text>
<text x="295.0" y="435.0" font-size="13" fill="#0f172a" text-anchor="middle">FN · 42.6% of row</text>
<rect x="390" y="320" width="190" height="190" fill="#a3bcf3" stroke="#ffffff" stroke-width="2"/>
<text x="485.0" y="407.0" font-size="30" font-weight="600" fill="#0f172a" text-anchor="middle">663</text>
<text x="485.0" y="435.0" font-size="13" fill="#0f172a" text-anchor="middle">TP · 57.4% of row</text>
<text x="24" y="508" font-size="11" fill="#64748b">Source: artifacts/small_metrics.json · chronological 80/20 split</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 2.3 KiB

+188
View File
@@ -0,0 +1,188 @@
# Metrics — paper support
Every value in this file comes from the committed artifacts:
`artifacts/metrics.json` (forecast + ablation + per-sensor + small model),
`artifacts/small_metrics.json` (small-model detail), and
`artifacts/data_audit.json` (raw-input counts). To refresh a table, rerun
`make train && make figures` and quote the new JSON.
## Run provenance
| Field | Value |
|---|---|
| Model version | `histgb-2026-11-14` |
| Trained at | 2026-11-14T05:27:21Z |
| Data version | 2026-11-07T23:51:00Z |
| Context (weather) version | 2026-11-14T03:47:09Z |
| Train rows | 18,602 hourly sensor rows |
| Holdout | 2026-10-10T23:00:00Z → 2026-11-07T23:00:00Z |
| Evaluation rows | 5,350 (8 sensors, hours with a seasonal-naive baseline) |
| Sensors | 8 (one merged `density-01`, seven `hn-*`) |
| Raw input | 192,464 rows, 96,232 density rows, 96,232 battery rows |
| Weather | Open-Meteo historical archive (training), forecast (scenarios) |
| Calendar | Baden-Württemberg holidays 2025–2026 |
## Protocol
- One global `HistGradientBoostingRegressor` over all eight sensors. Target is
the mean of 15-minute `densityData` readings in a UTC hour.
- Time-based 28-day holdout measured back from the last observed hour. No
shuffling, no random split.
- Seasonal naive baseline: the exact 168-row lag (same observed hour one week
earlier), evaluated on the same rows.
- The production bundle is refit on all observed rows **after** the holdout is
evaluated; the holdout is never in the production fit.
- WAPE = `sum(|error|) / sum(|actual|)`. R² is computed on the same rows.
- The production 7-day model uses only features at least 168 hours old, so a
168-hour forecast needs no recursion.
- Weather in the holdout is observed, not forecast. Live accuracy will be lower
than the table because the dashboard uses forecast weather.
## Headline
| Model | MAE | WAPE | R² |
|---|---:|---:|---:|
| Seasonal naive (exact 168-row lag) | 5.40 | 44.7% | 0.462 |
| **Weather-aware 7-day (production)** | **3.32** | **27.5%** | **0.777** |
- MAE improvement over seasonal naive: **38.5%**.
- WAPE improvement: 17.2 percentage points.
- The mission reference numbers for seasonal naive (5.40 / 44.7% / 0.462) are
reproduced exactly.
- The production row also beats the mission reference 7-day no-weather model
(4.37 / 36.2% / 0.672): weather supplies most of the gain, and `lag_week`
uses the previous *observed* same weekday/hour across sensor gaps instead of
recursion.
## Ablation
| Model | Description | MAE | WAPE | R² | n |
|---|---|---|---:|---:|---:|---:|
| `seasonal_naive` | exact 168-row lag (same observed hour last week) | 5.40 | 44.7% | 0.462 | 5,350 |
| `day_no_weather` | calendar + lags 24/48/168 + 24h/168h rolling means | 3.96 | 32.8% | 0.706 | 5,350 |
| `day_holiday` | day-ahead + Baden-Württemberg holiday flag | 3.84 | 31.8% | 0.732 | 5,350 |
| `day_weather` | day-ahead + temperature + rain (observed weather) | 3.36 | 27.8% | 0.763 | 5,350 |
| `week_no_weather` | 7-day-safe calendar + previous same weekday/hour values | 4.08 | 33.8% | 0.697 | 5,350 |
| **`week_weather`** | **7-day-safe + temperature + rain (production)** | **3.32** | **27.5%** | **0.777** | **5,350** |
Derived effects:
- Weather at day-ahead: 3.96 → 3.36 MAE, **−15.2%**.
- Weather at 7-day-safe: 4.08 → 3.32 MAE, **−18.6%**.
- Holiday flag at day-ahead: 3.96 → 3.84 MAE, **−0.12 MAE**. Holidays are
context, not a headline accuracy claim.
- The mission reference day-ahead weather row (3.36 / 27.8%) is reproduced.
![Ablation MAE by feature set](figures/ablation_mae.svg)
*Figure 1. Holdout MAE for each feature set; teal is the production model.
Weather is the largest single gain; the holiday flag is a small addition.*
## Per sensor
Production model and seasonal naive on the same holdout rows.
| Sensor | Name | n | MAE | WAPE | R² | Baseline MAE | Baseline WAPE | Baseline R² | MAE improvement |
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|
| `density-01` | Density 01 | 669 | 2.26 | 30.8% | 0.722 | 3.58 | 48.8% | 0.238 | 36.8% |
| `hn-goetzentuem-01` | Götzenturm | 669 | 2.40 | 26.9% | 0.718 | 3.84 | 43.0% | 0.281 | 37.5% |
| `hn-hagenbuchersee-01` | Hagenbuchersee 1 | 669 | 2.47 | 38.6% | 0.602 | 4.01 | 62.8% | −0.212 | 38.5% |
| `hn-hagenbuchersee-02` | Hagenbuchersee 2 | 669 | 4.12 | 31.1% | 0.731 | 6.49 | 48.9% | 0.337 | 36.5% |
| `hn-marahaus-01` | Marahaus | 669 | 2.67 | 29.8% | 0.671 | 4.57 | 51.0% | −0.045 | 41.7% |
| `hn-neckarufer-01` | Neckarufer | 668 | 3.63 | 28.2% | 0.695 | 5.89 | 45.8% | 0.110 | 38.4% |
| `hn-soleo-01` | Soleo | 669 | 3.19 | 20.2% | 0.548 | 4.40 | 28.0% | 0.163 | 27.7% |
| `hn-stadtgalerie-01` | Stadtgalerie | 668 | 5.83 | 25.3% | 0.765 | 10.39 | 45.1% | 0.480 | 43.9% |
- Every sensor improves over its own seasonal naive baseline (27.7%–43.9%).
- Stadtgalerie has the largest absolute error (MAE 5.83) and the largest
volume; Soleo has the smallest relative gain and the lowest R² (0.548).
- Two baselines have negative R² (`hagenbuchersee-01`, `marahaus-01`), which is
expected for a naive lag on a short, variable holdout; the model is positive
on all eight.
- Sensor totals: 669 × 6 + 668 × 2 = 5,350, matching the headline.
![Per-sensor MAE](figures/per_sensor_mae.svg)
*Figure 2. Per-sensor holdout MAE, production model (blue) vs seasonal naive
(grey). Every sensor improves; the model is not carried by one large sensor.*
## Uncertainty band
The `lo`/`hi` band is the holdout residual p10/p90 added to the point forecast,
clipped at zero:
| p10 | p50 | p90 |
|---:|---:|---:|
| −4.83 | −0.59 | +4.88 |
This is an empirical 80% band, not a calibrated guarantee, and the median
residual being −0.59 shows a small optimistic bias on the holdout.
## Small busy-hour classifier
Full definition and limitations: [`docs/small-model.md`](small-model.md).
Summary from the same baked run (`logit-20261114-052723`):
| Accuracy | Precision | Recall | F1 | ROC-AUC | Train rows | Test rows | Base rate |
|---:|---:|---:|---:|---:|---:|---:|---:|
| 0.708 | 0.360 | 0.574 | 0.443 | 0.726 | 18,242 | 5,710 | 0.202 |
| | Predicted not busy | Predicted busy |
|---|---:|---:|
| **Actually not busy** | 3,377 (TN) | 1,178 (FP) |
| **Actually busy** | 492 (FN) | 663 (TP) |
![Small model confusion matrix](figures/small_model_confusion.svg)
*Figure 3. Holdout confusion matrix for the small busy-hour classifier. The
balanced weights trade precision for recall; accuracy 0.708 is below the 0.798
majority-class baseline, so report recall/F1 and ROC-AUC instead.*
Per-sensor small-model holdout:
| Sensor | n | Positive rate | Accuracy | Precision | Recall | F1 |
|---|---:|---:|---:|---:|---:|---:|
| `density-01` | 714 | 0.111 | 0.761 | 0.300 | 0.873 | 0.447 |
| `hn-goetzentuem-01` | 714 | 0.197 | 0.732 | 0.391 | 0.638 | 0.485 |
| `hn-hagenbuchersee-01` | 714 | 0.049 | 0.724 | 0.148 | 0.971 | 0.257 |
| `hn-hagenbuchersee-02` | 714 | 0.123 | 0.783 | 0.355 | 0.932 | 0.514 |
| `hn-marahaus-01` | 714 | 0.055 | 0.710 | 0.135 | 0.795 | 0.230 |
| `hn-neckarufer-01` | 713 | 0.090 | 0.764 | 0.274 | 0.984 | 0.429 |
| `hn-soleo-01` | 714 | 0.704 | 0.581 | 0.943 | 0.431 | 0.592 |
| `hn-stadtgalerie-01` | 713 | 0.289 | 0.604 | 0.335 | 0.374 | 0.353 |
## Claims the data supports
- A global, weather-aware HistGB forecasts hourly footfall for all eight
inner-city sensors with MAE 3.32 / WAPE 27.5% / R² 0.777 on a 28-day
time-based holdout.
- Weather features are the main measurable gain: −18.6% MAE versus the 7-day
no-weather model, −38.5% versus seasonal naive.
- Every sensor improves over its own seasonal naive baseline (27.7%–43.9%).
- A tiny logistic model can flag busy hours with ROC-AUC 0.726 and 57.4%
recall at 36.0% precision, retrained from a CSV in under 5 seconds.
## Claims the data does not support
- People counts. Values are an index of estimated devices; penetration is
unknown.
- Annual seasonality, city-wide surfaces, or store-location recommendations
from five months at eight sensors in ~580 m.
- Causal weather effects. The ablation is predictive, not a treatment effect.
- Live accuracy equal to the holdout. The holdout uses observed weather; the
demo uses forecast weather and `predict` adds scenario overlays.
- Holidays as a large driver. The flag moves day-ahead MAE by only −0.12.
- The small classifier as the production forecast. It is a demo artifact with
six features and a per-window p75 target.
- Tuned hyperparameters. The model configuration is fixed; the 28-day holdout
is the held-out result, not a search result.
## Reproduce
```bash
make data # parse + weather + holidays + JSON artifacts
make train # holdout, ablation, per-sensor, small model, forecasts
make small # print the small-model metrics and confusion matrix
make figures # regenerate docs/figures/*.svg from artifacts/metrics.json
make check # sync --frozen, ruff, compileall, pytest
```
+186
View File
@@ -0,0 +1,186 @@
# Small model — busy-hour classifier
A logistic regression that labels an hour `busy` when footfall exceeds that
sensor's p75. It powers the `model=small` retraining demo, not the dashboard
forecast: six cheap features, a time-based split, and under a second to fit.
Every number below comes from the baked artifact
`artifacts/small_metrics.json` (version `logit-20261114-052723`, trained
2026-11-14T05:27:23Z). `make small` prints the headline metrics and per-sensor
table; `scripts/train_small.py --json` prints the full payload.
## Target
`busy` is a binary label for one sensor-hour:
```text
busy = 1 if hourly_mean_density > that sensor's 75th percentile
busy = 0 otherwise
```
- The p75 is computed **per sensor on the training window only**, then applied
unchanged to the holdout. Computing it on the full dataset would leak the
holdout distribution into the label.
- A p75 threshold makes "busy" mean "busier than three of four hours at this
location", not an absolute footfall claim. The same absolute value can be
busy at Soleo and quiet at Stadtgalerie.
- Hours above the threshold are exactly 25% of the training hours by
construction, before missing hours are dropped.
| Sensor | Training-window p75 |
|---|---:|
| `density-01` | 15.23 |
| `hn-goetzentuem-01` | 15.40 |
| `hn-hagenbuchersee-01` | 18.20 |
| `hn-hagenbuchersee-02` | 29.17 |
| `hn-marahaus-01` | 22.23 |
| `hn-neckarufer-01` | 28.35 |
| `hn-soleo-01` | 11.38 |
| `hn-stadtgalerie-01` | 38.41 |
## Data and split
- Input: the challenge CSV, `densityData` rows only, aggregated to the mean of
the 15-minute readings in each UTC hour.
- Window: 2026-06-12T00:00:00Z → 2026-11-07T23:00:00Z.
- 23,952 hourly sensor rows across all 8 usable sensors.
- Split is chronological, not random: the first 80% of **timestamps** train,
the last 20% test.
- Train: 2026-06-12T00:00:00Z → 2026-10-09T01:00:00Z, 18,242 rows.
- Test: 2026-10-09T02:00:00Z → 2026-11-07T23:00:00Z, 5,710 rows.
- Excluded devices: `maker-fair`, `niklas-01`, `hn-bollwerksturm-01` (test
installations / insufficient coverage). See `artifacts/data_audit.json`.
## Features
| Feature | Source | Note |
|---|---|---|
| `hour` | local Europe/Berlin hour | 0–23, captures the daily curve |
| `weekday` | local Europe/Berlin weekday | 0 = Monday |
| `is_weekend` | weekday ≥ 5 | |
| `is_holiday` | Baden-Württemberg calendar | context, not a headline claim |
| `temp_c` | observed Open-Meteo archive | joined to the nearest hour (±3 h) |
| `precip_mm` | observed Open-Meteo archive | same join |
No lags, no target history, no sensor identifier. That is intentional: the
classifier must retrain from a small user upload that may not contain a long
history, and it must not depend on future rows. Sensor identity is still
implicit in the per-sensor threshold.
## Model
```text
StandardScaler -> LogisticRegression(
C=1.0, max_iter=1000, class_weight="balanced",
solver="lbfgs", random_state=42)
```
`class_weight="balanced"` compensates for the 25/75 class split caused by the
p75 threshold. It buys recall at the cost of precision, which is the right
trade for a "flag the busy hours" demo.
Fit time: **0.30 s** on the full challenge CSV in this run (contract: < 5 s,
including the API job's CSV parsing).
## Metrics
Time-based holdout, 5,710 rows, base rate 20.2% positive.
| Accuracy | Precision | Recall | F1 | ROC-AUC |
|---:|---:|---:|---:|---:|
| 0.708 | 0.360 | 0.574 | 0.443 | 0.726 |
Confusion matrix:
| | Predicted not busy | Predicted busy |
|---|---:|---:|
| **Actually not busy** | 3,377 (TN) | 1,178 (FP) |
| **Actually busy** | 492 (FN) | 663 (TP) |
Derived:
- Specificity (true-negative rate): 0.741
- Negative predictive value: 0.873
- False-positive rate: 0.259
- False-negative rate: 0.426
- Precision lift over the 20.2% base rate: 1.78×
- Accuracy of the majority-class baseline ("always not busy"): 0.798
The model is **less accurate than always saying "not busy"**, but it finds 663
of 1,155 busy hours (57.4% recall) that the majority baseline never finds. The
balanced weights push it toward over-flagging: 1,178 false positives for 1,155
actual busy hours. That is why precision is 0.36 and accuracy is not the right
headline.
Per-sensor holdout:
| Sensor | n | Pos. rate | Accuracy | Precision | Recall | F1 |
|---|---:|---:|---:|---:|---:|---:|
| `density-01` | 714 | 0.111 | 0.761 | 0.300 | 0.873 | 0.447 |
| `hn-goetzentuem-01` | 714 | 0.197 | 0.732 | 0.391 | 0.638 | 0.485 |
| `hn-hagenbuchersee-01` | 714 | 0.049 | 0.724 | 0.148 | 0.971 | 0.257 |
| `hn-hagenbuchersee-02` | 714 | 0.123 | 0.783 | 0.355 | 0.932 | 0.514 |
| `hn-marahaus-01` | 714 | 0.055 | 0.710 | 0.135 | 0.795 | 0.230 |
| `hn-neckarufer-01` | 713 | 0.090 | 0.764 | 0.274 | 0.984 | 0.429 |
| `hn-soleo-01` | 714 | 0.704 | 0.581 | 0.943 | 0.431 | 0.592 |
| `hn-stadtgalerie-01` | 713 | 0.289 | 0.604 | 0.335 | 0.374 | 0.353 |
The pooled model has the lowest accuracy on `hn-soleo-01` (0.581, an unusually
high base rate of 70.4%) and the lowest recall on `hn-stadtgalerie-01` (0.374).
The high recall on low-base-rate sensors is mostly the balanced weights firing;
it is not evidence of a strong signal there.
## Reproduce
```bash
# Full challenge CSV, prints the metrics and per-sensor table above.
make small
# Explicit CSV, all rows, save the baked artifact.
uv run python scripts/train_small.py --csv new-data.csv --all --save
# Machine-readable payload.
uv run python scripts/train_small.py --json
```
The runtime path is the same function (`pulsecast.small.train_small_from_csv`):
```bash
curl -s -X POST localhost:18000/train \
-H "Authorization: Bearer $CONTROL_TOKEN" \
-F model=small -F file=@new-data.csv
curl -s localhost:18000/train/<job_id> \
-H "Authorization: Bearer $CONTROL_TOKEN"
```
The job returns `accuracy`, `precision`, `recall`, `f1`, `roc_auc`, `n_train`,
`n_test`, `positive_rate`, and `sensors`; the full confusion matrix and
per-sensor table are stored in `artifacts/small_metrics.json`. On success the
bundle is hot-swapped and persisted to `PULSECAST_MODELS/small_latest.joblib`.
A restart falls back to the baked artifact.
## Honest limits
- **It is a demo, not the production forecast.** The dashboard uses the global
HistGB 7-day model, not this classifier.
- **Threshold convention.** "Busy" is the sensor's own p75, recomputed from the
upload's training window. A different upload can move the thresholds and
change every metric. Do not compare runs trained on different windows.
- **No lags or sensor id.** The model sees only calendar and weather. It cannot
know that the previous hour was packed, and it pools all locations while
evaluating per sensor.
- **Weather fallback.** If an uploaded CSV does not overlap the baked weather
archive, missing weather is filled with 14 °C / 0 mm. That keeps the job
running but weakens the weather features; the returned metrics show it.
- **One chronological split.** Five months, one holdout. No cross-validation,
no annual seasonality claim, no per-weekday calibration.
- **Threshold metrics only.** This page reports hard 0.5-threshold metrics, not
a calibrated probability. The probability output is not a business forecast.
- **Counters are devices.** Footfall density is an index of estimated devices,
not a count of people. A "busy" flag inherits that uncertainty.
- **Small uploads are unstable.** A CSV with one sensor, only one class, or
fewer than four hourly timestamps fails with a clear error. A CSV with a
short window may train but produce meaningless metrics.
- **No causal claims.** Weather and holiday coefficients are associations in
this sample, not causal effects, and the classifier is not used for scenario
uplift (that overlay is explicit and separate).
+119
View File
@@ -0,0 +1,119 @@
# Training end-to-end evidence
The script starts a real `uvicorn pulsecast.server:app` process with
`CONTROL_TOKEN`, waits for startup, and drives the HTTP API only. It checks the
snapshot, every prediction/scenario path, auth and error paths, one job at a
time, and both training jobs; it exits non-zero on the first failed check.
Unit/contract tests (`make check`) are separate.
Reproduce with:
```bash
uv run python scripts/e2e_train.py --port 18000 --report /tmp/e2e.json
```
## Run
| Field | Value |
|---|---|
| Date | 2026-11-14, 05:33 UTC |
| Branch | `feat/ml-final` |
| Port | **18000** |
| Checks | **38 passed, 0 failed** |
| Total wall time | 10.4 s |
| Forecast version before | `histgb-2026-11-14` |
| Data version | `2026-11-07T23:51:00Z` |
| Context version | `2026-11-14T03:47:09Z` |
## Snapshot and robustness (20 checks)
- `/healthz` → `{"ok":true}`; `/readyz` → 200 `model=true`.
- `/meta` covers 8 sensors and the baked model beats the seasonal-naive
baseline on the holdout.
- `/bootstrap` → 8 sensors, all 8 hourly series (23,952 points), 168 forecast
weather hours, 24 holidays, versions equal to `/meta`.
- `/predict`: **128/128** combinations (8 sensors × 4 horizons × 4 scenarios)
returned the contract point counts (`15m` 1, `1h` 4, `48h` 48, `7d` 168),
monotonic UTC timestamps, non-negative values, and `lo ≤ v ≤ hi`.
- `/simulate`: all 4 scenarios × 8 sensors × 4 horizons returned valid points.
- Weather override changes the forecast (`-8 °C / 6 mm` vs `30 °C / 0 mm`).
- Unknown sensor → 404; missing token → 401 `unauthorized`; wrong token → 401;
unknown job → 404 `not_found`; invalid model → 400 `bad_request`; missing
JSON path → 400.
## Small training job (real 60-day CSV subset)
Input: 36,846 density rows, 8 sensors, 2026-09-08T23:53Z → 2026-11-07T23:51Z.
| Field | Value |
|---|---|
| Job ID | `0b35347963974cbbbe584b77ecd4b846` |
| Status | `succeeded` |
| Model version | `logit-20261114-053344` |
| Wall time (submit → terminal) | **0.172 s** (contract < 5 s) |
| n_train / n_test | 8,151 / 2,301 |
| Base rate | 0.180 |
| Accuracy | 0.824 |
| Precision | 0.542 |
| Recall | 0.157 |
| F1 | 0.243 |
| ROC-AUC | 0.813 |
This is a different window from the baked full-window model in
`docs/small-model.md`, so the numbers are not directly comparable: the p75
target and the 80/20 split are recomputed inside the 60-day upload. The low
recall illustrates the limit noted in the small-model doc: a short, recent
window with a different threshold is a different classification problem.
Post-conditions verified:
- `small_latest.joblib` and `<job_id>-small.joblib` exist under
`PULSECAST_MODELS`.
- `/meta.small_model.model_version` equals the job version and its metrics
match the job exactly (in-memory hot-swap).
- `/predict` still returns 168 valid points after the swap.
## Forecast training job (full challenge CSV) and 409
Input: full `schwarz-pax.csv`, 19,383,568 bytes.
| Field | Value |
|---|---|
| Job ID | `235209a0d00f449ba8be717eb76386d6` |
| Status | `succeeded` |
| Model version | `histgb-20261114-053349` |
| Wall time | **4.380 s** |
| MAE | 3.32 |
| WAPE | 27.5% |
| R² | 0.777 |
| Baseline MAE / WAPE / R² | 5.40 / 44.7% / 0.462 |
| n_eval | 5,350 |
The ablation in the job payload matches `artifacts/metrics.json` row for row
(3.96, 3.84, 3.36, 4.08, 3.32). While this job was in `running`, a second
`POST /train` returned **409** `conflict`, proving one job at a time.
Post-conditions verified:
- `forecast_latest.joblib` and `<job_id>-forecast.joblib` exist under
`PULSECAST_MODELS`.
- `/meta.model_version` equals the retrained version and
`/meta.model_source` is `retrained`.
- `/predict` reports the retrained version; `/readyz` stays 200.
- `data_version` after the retrain is `2026-11-07T23:00:00Z` (the hourly input
maximum) rather than the raw-file floor `23:51:00Z`; both describe the same
observed window.
## Failure isolation
A malformed CSV (`not,a,pulsecast,file`) was accepted as a job (202), then
failed with a clear error:
```text
CSV needs a time column (time, timestamp, datetime, date, hour) and a value
column (value, density, footfall, count, pax, visitors); got ['not', 'a',
'pulsecast', 'file']
```
`/healthz` still returned 200 afterward: a bad upload cannot take down the
service or block `/predict`.
+63 -18
View File
@@ -33,7 +33,11 @@ Day-ahead ablation models additionally use `lag_day`, `lag_2d`, `roll_24`,
## Target and data ## Target and data
- Target: mean of 15-minute `densityData` readings in a UTC hour, per sensor. - Target: mean of 15-minute `densityData` readings in a UTC hour, per sensor.
- Training window: 2026-06-12 → 2026-10-10; holdout: 2026-10-10 → 2026-11-07. - Training rows: 2026-06-12T00:00:00Z → 2026-10-10T22:00:00Z, 18,602 rows.
- Holdout: 2026-10-10T23:00:00Z → 2026-11-07T23:00:00Z, 5,350
evaluation rows (hours with a seasonal-naive baseline).
- The production bundle is refit on all observed rows (through
2026-11-07T23:00:00Z) only after the holdout is evaluated.
- Eight usable sensors after excluding `maker-fair` (73 rows, Hannover), - Eight usable sensors after excluding `maker-fair` (73 rows, Hannover),
`niklas-01` (8 rows), `hn-bollwerksturm-01` (5 rows). `niklas-01` (8 rows), `hn-bollwerksturm-01` (5 rows).
- `density-01` merges two physical devices (`863f75b0`, `f8c19730`) into one - `density-01` merges two physical devices (`863f75b0`, `f8c19730`) into one
@@ -57,35 +61,76 @@ Time-based 28-day holdout, no shuffling. Seasonal naive is the baseline. WAPE is
| 7-day-safe, no weather | 4.08 | 33.8% | 0.697 | | 7-day-safe, no weather | 4.08 | 33.8% | 0.697 |
| **7-day-safe, + temperature/rain (production)** | **3.32** | **27.5%** | **0.777** | | **7-day-safe, + temperature/rain (production)** | **3.32** | **27.5%** | **0.777** |
Weather is the useful signal: adding temperature and rain cuts MAE by ~19% Weather is the useful signal: adding temperature and rain cuts MAE by 18.6%
versus the 7-day no-weather model and ~38% versus seasonal naive. The holiday versus the 7-day no-weather model and 38.5% versus seasonal naive. The holiday
flag alone changes day-ahead MAE by −0.12 in this run; holidays are context, not flag alone changes day-ahead MAE by −0.12 in this run; holidays are context, not
a headline accuracy claim. a headline accuracy claim.
The seasonal-naive row reproduces the mission reference exactly The seasonal-naive row reproduces the mission reference exactly
(5.40 / 44.7% / 0.462), and the day-ahead weather row matches 3.36 / 27.8%. The (5.40 / 44.7% / 0.462), and the day-ahead weather row matches 3.36 / 27.8%. The
production model is slightly better than the reference because `lag_week` uses production 7-day model edges that day-ahead reference on MAE and WAPE
the previous *observed* same weekday/hour across sensor gaps, while the baseline (3.32 / 27.5%) but is slightly lower on R² (0.777 vs 0.779). The two use
uses the exact 168-row lag. R² differs slightly from the reference 0.779 different features; `lag_week` uses the previous *observed* same weekday/hour
(0.777 here) because of that feature choice. across sensor gaps, while the seasonal-naive baseline uses the exact 168-row
lag.
Per-sensor holdout MAE improves over each sensor's own seasonal naive by
27.7%–43.9%; the full table, figures, and provenance are in
[`docs/metrics.md`](docs/metrics.md). The largest absolute error is
Stadtgalerie (MAE 5.83, its baseline 10.39); Soleo has the smallest relative
gain (27.7%) and lowest R² (0.548). Every sensor has a positive R² under the
model; two seasonal-naive baselines are negative.
## Uncertainty band ## Uncertainty band
The `lo`/`hi` band is the 10th/90th percentile of holdout residuals added to the The `lo`/`hi` band is the 10th/90th percentile of holdout residuals added to the
point forecast, clipped at zero. It is an empirical 80% band, not a calibrated point forecast, clipped at zero: p10 −4.83, p50 −0.59, p90 +4.88. It is an
guarantee. empirical 80% band, not a calibrated guarantee; the negative median residual
means the point forecast is slightly optimistic on the holdout.
## Small classifier ## Small classifier ("I coded an AI" artifact)
This model is separate from the production forecaster and powers the
`model=small` retraining demo. Full definition, thresholds, confusion matrix
interpretation, and limits: [`docs/small-model.md`](docs/small-model.md).
- Target: `busy` = hourly footfall strictly above that sensor's p75, where the - Target: `busy` = hourly footfall strictly above that sensor's p75, where the
p75 is computed on the training window only. p75 is computed on the training window only and then frozen for the holdout.
- Features: `hour`, `weekday`, `is_weekend`, `temp_c`, `precip_mm`, `is_holiday`. - Features: `hour`, `weekday`, `is_weekend`, `temp_c`, `precip_mm`, `is_holiday`
- Model: standardized logistic regression, `class_weight="balanced"`. (no lags, no sensor identifier, no target history).
- Split: last 20% of hourly timestamps, time-ordered. - Model: standardized logistic regression, `class_weight="balanced"`, `C=1.0`,
- Result: accuracy 0.708, precision 0.360, recall 0.574, F1 0.443, ROC-AUC 0.726. `max_iter=1000`, `random_state=42`.
- Trains in ~0.25 s on the challenge CSV and under 1 s on a small upload. - Split: last 20% of hourly timestamps, time-ordered. Train
- The low precision reflects a hard p75 threshold and a deliberately tiny model; 2026-06-12T00:00Z → 2026-10-09T01:00Z (18,242 rows); test
it is a demo artifact, not the production forecast. 2026-10-09T02:00Z → 2026-11-07T23:00Z (5,710 rows).
- Result (`logit-20261114-052723`): accuracy 0.708, precision 0.360, recall
0.574, F1 0.443, ROC-AUC 0.726, base rate 0.202.
- Confusion matrix: TN 3,377, FP 1,178, FN 492, TP 663. Specificity 0.741,
negative predictive value 0.873.
- Fit time: 0.30 s on the full challenge CSV; the API job (parse + fit) must
stay under the 5 s contract. The live e2e run trained a 60-day upload subset
in 0.17 s wall time.
- Per-sensor F1 ranges from 0.230 (Marahaus, 5.5% positive) to 0.592 (Soleo,
70.4% positive). The balanced weights over-flag low-base-rate sensors; accuracy
is lowest on Soleo (0.581) and recall is lowest on Stadtgalerie (0.374).
- Accuracy is below the 0.798 majority-class baseline. Do not use accuracy as
the headline: the model finds 663 of 1,155 busy hours, which the majority
baseline never finds. The low precision reflects a hard p75 threshold, a
deliberately tiny model, and balanced class weights.
- This is a demo artifact, not the production forecast. Thresholds move with
the upload window, and a CSV without weather overlap falls back to
14 °C / 0 mm.
## Artifacts and reproduction
- `artifacts/metrics.json` — versions, headline, ablation, per-sensor, small
model summary.
- `artifacts/small_metrics.json` — full small-model payload including the
confusion matrix, per-sensor table, and thresholds.
- `make train` regenerates models and metrics; `make small` prints the
small-model numbers; `make figures` regenerates `docs/figures/*.svg`.
- `docs/metrics.md` is the paper support sheet; it quotes the JSON and does not
invent numbers.
## No-leakage rules ## No-leakage rules
+523
View File
@@ -0,0 +1,523 @@
#!/usr/bin/env python3
"""End-to-end acceptance run against a real FastAPI process on port 18000.
Starts uvicorn with ``CONTROL_TOKEN``, then exercises robustness and the full
training lifecycle:
- /healthz, /readyz, /meta, /bootstrap (8 sensors, hourly series, context)
- /predict for every sensor x horizon x scenario (128 calls)
- /simulate for every scenario with all sensors
- auth, unknown job, unknown model, malformed CSV, one-job-at-a-time
- POST /train model=small on a real CSV subset; poll to success; hot-swap
- POST /train model=forecast on the full CSV; poll to success; hot-swap
- /predict still works after each swap
Exit code is 0 only when every check passes. Figures printed to stdout are the
measured numbers; ``--report`` writes the same result as JSON.
"""
from __future__ import annotations
import argparse
import json
import os
import shutil
import socket
import subprocess
import sys
import tempfile
import time
from datetime import UTC, datetime
from pathlib import Path
import httpx
import pandas as pd
from pulsecast import config
HORIZON_SIZES = {"15m": 1, "1h": 4, "48h": 48, "7d": 168}
SCENARIOS = tuple(config.SCENARIOS)
SENSORS = tuple(sorted(config.USABLE_SENSORS))
class CheckFailed(RuntimeError):
pass
class Runner:
def __init__(self) -> None:
self.checks = 0
self.results: dict = {}
def check(self, condition: bool, message: str) -> None:
self.checks += 1
if not condition:
raise CheckFailed(message)
print(f" [ok] {message}")
def _utc_now() -> str:
return datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ")
def _parse_ts(value: str) -> pd.Timestamp:
return pd.Timestamp(value)
def _seconds(start: str | None, end: str | None) -> float | None:
if not start or not end:
return None
return round((_parse_ts(end) - _parse_ts(start)).total_seconds(), 3)
def _wait_http(client: httpx.Client, url: str, timeout: float = 30.0) -> None:
deadline = time.monotonic() + timeout
last: Exception | None = None
while time.monotonic() < deadline:
try:
response = client.get(url, timeout=2.0)
if response.status_code == 200:
return
except Exception as exc: # noqa: BLE001 - startup race
last = exc
time.sleep(0.1)
raise CheckFailed(f"server did not answer {url} within {timeout}s: {last}")
def _wait_job(
client: httpx.Client,
job_id: str,
headers: dict[str, str],
*,
timeout: float = 180.0,
state: str | None = None,
) -> dict:
deadline = time.monotonic() + timeout
last_status = None
while time.monotonic() < deadline:
response = client.get(f"/train/{job_id}", headers=headers)
if response.status_code != 200:
raise CheckFailed(f"GET /train/{job_id} -> {response.status_code}: {response.text}")
job = response.json()
last_status = job["status"]
if state is not None and last_status == state:
return job
if state is None and last_status in {"succeeded", "failed"}:
return job
time.sleep(0.05)
raise CheckFailed(f"job {job_id} did not reach {state or 'terminal'} (last={last_status})")
def _job_headers(token: str) -> dict[str, str]:
return {"Authorization": f"Bearer {token}"}
def _submit(
client: httpx.Client, headers: dict[str, str], path: Path, model: str
) -> httpx.Response:
return client.post(
"/train",
headers=headers,
files={"file": (path.name, path.read_bytes(), "text/csv")},
data={"model": model},
timeout=120.0,
)
def _assert_points(points: list[dict], expected: int, label: str) -> None:
if len(points) != expected:
raise CheckFailed(f"{label}: expected {expected} points, got {len(points)}")
previous = None
for point in points:
if set(point) != {"t", "v", "lo", "hi"}:
raise CheckFailed(f"{label}: unexpected point keys {sorted(point)}")
if not point["t"].endswith("Z"):
raise CheckFailed(f"{label}: timestamp is not UTC RFC 3339: {point['t']}")
if point["v"] < 0 or point["lo"] > point["v"] or point["v"] > point["hi"]:
raise CheckFailed(f"{label}: invalid band {point}")
stamp = _parse_ts(point["t"])
if previous is not None and stamp <= previous:
raise CheckFailed(f"{label}: timestamps are not increasing at {point['t']}")
previous = stamp
def _versions(payload: dict) -> dict:
return {
key: payload.get(key)
for key in ("data_version", "model_version", "context_version")
}
def _build_subset(raw: Path, destination: Path, days: int = 60) -> tuple[Path, dict]:
frame = pd.read_csv(raw)
frame = frame[frame["type"].astype(str) == "densityData"].copy()
frame["time"] = pd.to_datetime(frame["time"], utc=True)
end = frame["time"].max()
cut = end - pd.Timedelta(days=days)
frame = frame[frame["time"] >= cut][["time", "name", "value"]]
frame["time"] = frame["time"].dt.strftime("%Y-%m-%dT%H:%M:%SZ")
frame.to_csv(destination, index=False)
return destination, {
"rows": len(frame),
"sensors": int(frame["name"].nunique()),
"from": frame["time"].min(),
"to": frame["time"].max(),
"days": days,
}
def _port_free(port: int) -> bool:
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.settimeout(0.5)
return sock.connect_ex(("127.0.0.1", port)) != 0
def run(args: argparse.Namespace, runner: Runner) -> dict:
raw = Path(args.csv)
if not raw.exists():
raise CheckFailed(f"CSV not found: {raw}")
if not _port_free(args.port):
raise CheckFailed(f"port {args.port} is already in use")
workdir = Path(tempfile.mkdtemp(prefix="pulsecast-e2e-"))
models_dir = workdir / "models"
uploads_dir = workdir / "uploads"
subset_path, subset_info = _build_subset(raw, workdir / "small-subset.csv", args.days)
garbage_path = workdir / "garbage.csv"
garbage_path.write_text("not,a,pulsecast,file\n1,2,3,4\n")
env = os.environ.copy()
env.update(
CONTROL_TOKEN=args.token,
PULSECAST_MODELS=str(models_dir),
PULSECAST_UPLOADS=str(uploads_dir),
)
log_path = workdir / "server.log"
log_handle = log_path.open("wb")
command = [
sys.executable,
"-m",
"uvicorn",
"pulsecast.server:app",
"--host",
"127.0.0.1",
"--port",
str(args.port),
"--log-level",
"warning",
]
process = subprocess.Popen(command, env=env, stdout=log_handle, stderr=subprocess.STDOUT)
report: dict = {
"started_at": _utc_now(),
"base_url": f"http://127.0.0.1:{args.port}",
"subset": subset_info,
"raw_csv_bytes": raw.stat().st_size,
}
try:
with httpx.Client(base_url=f"http://127.0.0.1:{args.port}", timeout=120.0) as client:
_wait_http(client, "/healthz")
headers = _job_headers(args.token)
print("Boot and snapshot")
health = client.get("/healthz").json()
runner.check(health == {"ok": True}, "/healthz returns {ok:true}")
ready = client.get("/readyz")
runner.check(ready.status_code == 200 and ready.json()["model"] is True, "/readyz 200 model=true")
meta_before = client.get("/meta").json()
runner.check(meta_before["coverage"]["sensors"] == 8, "/meta covers 8 sensors")
runner.check(
meta_before["metrics"]["mae"] < meta_before["metrics"]["baseline_mae"],
"baked model beats seasonal naive on the holdout",
)
bootstrap = client.get("/bootstrap").json()
runner.check(len(bootstrap["sensors"]) == 8, "/bootstrap returns 8 sensors")
runner.check(
set(bootstrap["series"]) == set(SENSORS),
"/bootstrap series covers all 8 sensors",
)
series_points = sum(len(points) for points in bootstrap["series"].values())
runner.check(series_points > 20_000, f"/bootstrap carries {series_points} hourly points")
context = bootstrap["context"]
runner.check(
len(context["weather"]["forecast"]) == 168,
"/bootstrap context has 168 forecast weather hours",
)
runner.check(bool(context["holidays"]), "/bootstrap context has holidays")
runner.check(
_versions(bootstrap) == _versions(meta_before),
"/bootstrap versions match /meta",
)
report["versions_before"] = _versions(meta_before)
report["bootstrap"] = {"sensors": 8, "series_points": series_points}
print("Robustness: /predict every sensor x horizon x scenario")
predict_calls = 0
for sensor in SENSORS:
for horizon, count in HORIZON_SIZES.items():
for scenario in SCENARIOS:
response = client.post(
"/predict",
json={"sensor": sensor, "horizon": horizon, "scenario": scenario},
)
if response.status_code != 200:
raise CheckFailed(
f"/predict {sensor}/{horizon}/{scenario} -> "
f"{response.status_code}: {response.text}"
)
payload = response.json()
_assert_points(
payload["points"],
count,
f"/predict {sensor}/{horizon}/{scenario}",
)
predict_calls += 1
runner.check(predict_calls == 128, "128 /predict combinations returned valid points")
print("Robustness: /simulate every scenario, all sensors")
simulate_calls = 0
for scenario in SCENARIOS:
response = client.post(
"/simulate", json={"scenario": scenario, "sensors": "all"}
)
if response.status_code != 200:
raise CheckFailed(f"/simulate {scenario} -> {response.status_code}: {response.text}")
payload = response.json()
if set(payload["forecasts"]) != set(SENSORS):
raise CheckFailed(f"/simulate {scenario}: sensor set mismatch")
for sensor, by_horizon in payload["forecasts"].items():
if set(by_horizon) != set(HORIZON_SIZES):
raise CheckFailed(f"/simulate {scenario}/{sensor}: horizon set mismatch")
for horizon, count in HORIZON_SIZES.items():
_assert_points(
by_horizon[horizon],
count,
f"/simulate {scenario}/{sensor}/{horizon}",
)
simulate_calls += 1
runner.check(simulate_calls == 4, "4 /simulate scenarios x 8 sensors x 4 horizons")
cold = client.post(
"/predict",
json={"sensor": SENSORS[0], "horizon": "48h", "weather": {"temp_c": -8.0, "precip_mm": 6.0}},
).json()["points"]
warm = client.post(
"/predict",
json={"sensor": SENSORS[0], "horizon": "48h", "weather": {"temp_c": 30.0, "precip_mm": 0.0}},
).json()["points"]
runner.check(cold != warm, "weather override changes the forecast")
print("Error handling")
no_token = client.post("/predict", json={"sensor": "nope"})
runner.check(no_token.status_code == 404, "/predict unknown sensor -> 404")
unauthorized = _submit(client, {}, subset_path, "small")
runner.check(
unauthorized.status_code == 401
and unauthorized.json()["error"]["code"] == "unauthorized",
"/train without token -> 401 unauthorized",
)
bad_token = _submit(client, _job_headers("wrong-token"), subset_path, "small")
runner.check(bad_token.status_code == 401, "/train with wrong token -> 401")
unknown_job = client.get("/train/nope", headers=headers)
runner.check(
unknown_job.status_code == 404
and unknown_job.json()["error"]["code"] == "not_found",
"/train/nope -> 404 not_found",
)
bad_model = _submit(client, headers, subset_path, "giant")
runner.check(
bad_model.status_code == 400
and bad_model.json()["error"]["code"] == "bad_request",
"/train invalid model -> 400 bad_request",
)
missing_path = client.post(
"/train", headers=headers, json={"path": "/tmp/no-such-pulsecast.csv", "model": "small"}
)
runner.check(missing_path.status_code == 400, "/train missing JSON path -> 400")
print("Small model: real CSV subset through the API")
started = time.monotonic()
submitted = _submit(client, headers, subset_path, "small")
if submitted.status_code != 202:
raise CheckFailed(f"/train small -> {submitted.status_code}: {submitted.text}")
small_submitted = submitted.json()
runner.check(
small_submitted["status"] == "queued" and small_submitted["model"] == "small",
f"/train small queued as job {small_submitted['job_id']}",
)
small_job = _wait_job(client, small_submitted["job_id"], headers)
wall = round(time.monotonic() - started, 3)
if small_job["status"] != "succeeded":
raise CheckFailed(f"small job failed: {small_job}")
small_metrics = small_job["metrics"]
for key in ("accuracy", "precision", "recall", "f1", "roc_auc", "n_train", "n_test", "positive_rate"):
if key not in small_metrics:
raise CheckFailed(f"small metrics missing {key}: {small_metrics}")
train_seconds = _seconds(small_job["started_at"], small_job["finished_at"])
runner.check(
small_job["model_version"].startswith("logit-"),
f"small model_version {small_job['model_version']}",
)
runner.check(
wall < 5.0,
f"small job took {wall}s wall time including upload (contract < 5 s; "
f"server timestamps have 1 s resolution, train_seconds={train_seconds})",
)
runner.check(small_metrics["accuracy"] > 0.6, f"small accuracy {small_metrics['accuracy']}")
runner.check(
(models_dir / "small_latest.joblib").exists(),
"small model persisted to PULSECAST_MODELS",
)
meta_after_small = client.get("/meta").json()
runner.check(
meta_after_small["small_model"]["model_version"] == small_job["model_version"],
"small model hot-swapped into /meta",
)
runner.check(
meta_after_small["small_model"]["metrics"]["accuracy"] == small_metrics["accuracy"],
"hot-swapped small metrics match the job",
)
predict_after_small = client.post(
"/predict", json={"sensor": SENSORS[0], "horizon": "7d", "scenario": "base"}
)
runner.check(
predict_after_small.status_code == 200
and len(predict_after_small.json()["points"]) == 168,
"/predict still works after the small swap",
)
report["jobs"] = {
"small": {
"job": small_job,
"wall_seconds": wall,
"train_seconds": train_seconds,
"input": subset_info,
}
}
print("Forecast model: full CSV, hot-swap, 409 while busy")
forecast_started = time.monotonic()
forecast_response = _submit(client, headers, raw, "forecast")
if forecast_response.status_code != 202:
raise CheckFailed(
f"/train forecast -> {forecast_response.status_code}: {forecast_response.text}"
)
forecast_submitted = forecast_response.json()
runner.check(
forecast_submitted["model"] == "forecast",
f"/train forecast queued as job {forecast_submitted['job_id']}",
)
_wait_job(client, forecast_submitted["job_id"], headers, state="running", timeout=30)
busy = _submit(client, headers, subset_path, "small")
runner.check(
busy.status_code == 409 and busy.json()["error"]["code"] == "conflict",
"second /train while forecast is running -> 409 conflict",
)
forecast_job = _wait_job(client, forecast_submitted["job_id"], headers, timeout=300)
forecast_wall = round(time.monotonic() - forecast_started, 3)
if forecast_job["status"] != "succeeded":
raise CheckFailed(f"forecast job failed: {forecast_job}")
forecast_metrics = forecast_job["metrics"]
for key in ("mae", "wape", "r2", "baseline_mae", "n_eval"):
if key not in forecast_metrics:
raise CheckFailed(f"forecast metrics missing {key}: {forecast_metrics}")
runner.check(
forecast_job["model_version"].startswith("histgb-"),
f"forecast model_version {forecast_job['model_version']}",
)
runner.check(
forecast_metrics["mae"] < forecast_metrics["baseline_mae"],
f"retrained forecast MAE {forecast_metrics['mae']} < baseline {forecast_metrics['baseline_mae']}",
)
runner.check(
(models_dir / "forecast_latest.joblib").exists(),
"forecast model persisted to PULSECAST_MODELS",
)
meta_after_forecast = client.get("/meta").json()
runner.check(
meta_after_forecast["model_version"] == forecast_job["model_version"]
and meta_after_forecast["model_source"] == "retrained",
"forecast model hot-swapped into /meta as retrained",
)
predict_after_forecast = client.post(
"/predict", json={"sensor": SENSORS[-1], "horizon": "48h", "scenario": "rainy-saturday"}
)
runner.check(
predict_after_forecast.status_code == 200
and predict_after_forecast.json()["model_version"] == forecast_job["model_version"],
"/predict uses the retrained forecast model",
)
runner.check(client.get("/readyz").status_code == 200, "/readyz stays 200 after hot-swap")
report["jobs"]["forecast"] = {
"job": forecast_job,
"wall_seconds": forecast_wall,
"input": {"rows": "full challenge CSV", "bytes": raw.stat().st_size},
}
print("Malformed CSV fails the job (not the process)")
bad_csv = _submit(client, headers, garbage_path, "small")
runner.check(bad_csv.status_code == 202, "malformed CSV is accepted as a job")
bad_job = _wait_job(client, bad_csv.json()["job_id"], headers)
runner.check(
bad_job["status"] == "failed" and bool(bad_job["error"]),
f"malformed CSV job failed with {bad_job['error']!r}",
)
runner.check(
client.get("/healthz").status_code == 200,
"/healthz still 200 after a failed job",
)
report["versions_after_small"] = _versions(meta_after_small)
report["versions_after_forecast"] = _versions(meta_after_forecast)
report["checks"] = runner.checks
report["ok"] = True
finally:
process.terminate()
try:
process.wait(timeout=10)
except subprocess.TimeoutExpired:
process.kill()
process.wait(timeout=5)
log_handle.close()
if process.returncode not in (0, -15):
print(f"server exited with {process.returncode}; log: {log_path}", file=sys.stderr)
print(log_path.read_text(errors="replace")[-4000:], file=sys.stderr)
report["server_log_tail"] = log_path.read_text(errors="replace")[-1000:]
if args.keep:
print(f"kept workdir {workdir}")
else:
shutil.rmtree(workdir, ignore_errors=True)
report["finished_at"] = _utc_now()
return report
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--port", type=int, default=18000)
parser.add_argument("--token", default=os.environ.get("CONTROL_TOKEN") or "e2e-local-token")
parser.add_argument("--csv", default=str(config.raw_csv_path()))
parser.add_argument("--days", type=int, default=60, help="days in the small-model CSV subset")
parser.add_argument("--report", type=Path, help="write the JSON result here")
parser.add_argument("--keep", action="store_true", help="keep the temporary workdir")
args = parser.parse_args(argv)
runner = Runner()
print(f"PulseCast end-to-end check on port {args.port}")
started = time.monotonic()
try:
report = run(args, runner)
except CheckFailed as exc:
print(f"\nFAIL after {runner.checks} checks: {exc}", file=sys.stderr)
return 1
elapsed = round(time.monotonic() - started, 1)
report["elapsed_seconds"] = elapsed
print(f"\nPASS: {runner.checks} checks in {elapsed}s")
if args.report:
args.report.parent.mkdir(parents=True, exist_ok=True)
args.report.write_text(json.dumps(report, indent=2, ensure_ascii=False) + "\n")
print(f"report: {args.report}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+262
View File
@@ -0,0 +1,262 @@
#!/usr/bin/env python3
"""Render dependency-free SVG figures from ``artifacts/metrics.json``.
The paper figures are generated from the measured holdout metrics, never
hand-written. Run ``make figures`` after ``make train``.
"""
from __future__ import annotations
import argparse
import json
from pathlib import Path
from xml.sax.saxutils import escape
from pulsecast import config
MODEL_COLOR = "#2563eb"
BASELINE_COLOR = "#94a3b8"
PRODUCTION_COLOR = "#0f766e"
TEXT_COLOR = "#0f172a"
GRID_COLOR = "#e2e8f0"
def _svg_document(
title: str,
subtitle: str,
body: list[str],
width: int,
height: int,
footer: str = "Source: artifacts/metrics.json · PulseCast 28-day time-based holdout",
) -> str:
return "\n".join(
[
f'<svg xmlns="http://www.w3.org/2000/svg" width="{width}" height="{height}" '
f'viewBox="0 0 {width} {height}" role="img" aria-label="{escape(title)}">',
'<rect width="100%" height="100%" fill="#ffffff"/>',
'<g font-family="-apple-system, Segoe UI, Helvetica, Arial, sans-serif">',
f'<text x="24" y="34" font-size="20" font-weight="600" fill="{TEXT_COLOR}">'
f"{escape(title)}</text>",
f'<text x="24" y="56" font-size="13" fill="#475569">{escape(subtitle)}</text>',
*body,
f'<text x="24" y="{height - 12}" font-size="11" fill="#64748b">'
f"{escape(footer)}</text>",
"</g>",
"</svg>",
"",
]
)
def _bar_chart(
title: str,
subtitle: str,
rows: list[dict],
*,
value_key: str = "mae",
secondary_key: str | None = None,
secondary_label: str | None = None,
primary_label: str = "MAE",
production: str | None = None,
) -> str:
width = 960
left = 230
right = 96
top = 86
row_height = 38
height = top + row_height * len(rows) + 56
plot_width = width - left - right
values = [row[value_key] for row in rows]
if secondary_key:
values += [row[secondary_key] for row in rows]
maximum = max(values) * 1.08 if values else 1.0
body: list[str] = []
for tick in range(5):
x = left + plot_width * tick / 4
value = maximum * tick / 4
body.append(
f'<line x1="{x:.1f}" y1="{top - 18}" x2="{x:.1f}" y2="{top + row_height * len(rows)}" '
f'stroke="{GRID_COLOR}" stroke-width="1"/>'
)
body.append(
f'<text x="{x:.1f}" y="{top - 24}" font-size="11" fill="#64748b" '
f'text-anchor="middle">{value:.1f}</text>'
)
legend_y = top - 34
if secondary_key:
body.append(
f'<rect x="{left}" y="{legend_y}" width="10" height="10" fill="{BASELINE_COLOR}"/>'
f'<text x="{left + 16}" y="{legend_y + 9}" font-size="12" fill="#475569">'
f"{escape(secondary_label or 'baseline')}</text>"
)
body.append(
f'<rect x="{left + 140}" y="{legend_y}" width="10" height="10" fill="{MODEL_COLOR}"/>'
f'<text x="{left + 156}" y="{legend_y + 9}" font-size="12" fill="#475569">'
f"{escape(primary_label)}</text>"
)
for index, row in enumerate(rows):
y = top + index * row_height
label = str(row.get("name") or row.get("sensor"))
body.append(
f'<text x="{left - 12}" y="{y + 20}" font-size="12" fill="{TEXT_COLOR}" '
f'text-anchor="end">{escape(label)}</text>'
)
if secondary_key:
baseline_width = plot_width * row[secondary_key] / maximum
body.append(
f'<rect x="{left}" y="{y + 3}" width="{baseline_width:.1f}" height="12" '
f'fill="{BASELINE_COLOR}"/>'
)
body.append(
f'<text x="{left + baseline_width + 6:.1f}" y="{y + 13}" font-size="11" '
f'fill="#64748b">{row[secondary_key]:.2f}</text>'
)
model_width = plot_width * row[value_key] / maximum
body.append(
f'<rect x="{left}" y="{y + 18}" width="{model_width:.1f}" height="12" '
f'fill="{MODEL_COLOR}"/>'
)
body.append(
f'<text x="{left + model_width + 6:.1f}" y="{y + 28}" font-size="11" '
f'fill="{TEXT_COLOR}">{row[value_key]:.2f}</text>'
)
else:
color = PRODUCTION_COLOR if production and row.get("name") == production else MODEL_COLOR
bar_width = plot_width * row[value_key] / maximum
body.append(
f'<rect x="{left}" y="{y + 6}" width="{bar_width:.1f}" height="22" fill="{color}"/>'
)
body.append(
f'<text x="{left + bar_width + 6:.1f}" y="{y + 21}" font-size="12" '
f'fill="{TEXT_COLOR}">{row[value_key]:.2f}</text>'
)
return _svg_document(title, subtitle, body, width, height)
def _confusion_svg(small: dict) -> str:
matrix = small["confusion_matrix"]
metrics = small["metrics"]
width, height = 760, 520
left, top, cell = 200, 130, 190
values = {
(0, 0): ("TN", matrix["tn"]),
(0, 1): ("FP", matrix["fp"]),
(1, 0): ("FN", matrix["fn"]),
(1, 1): ("TP", matrix["tp"]),
}
maximum = max(value for _, value in values.values())
body: list[str] = []
body.append(
f'<text x="{left + cell / 2}" y="{top - 16}" font-size="13" fill="#475569" '
f'text-anchor="middle">Predicted not busy</text>'
)
body.append(
f'<text x="{left + cell + cell / 2}" y="{top - 16}" font-size="13" fill="#475569" '
f'text-anchor="middle">Predicted busy</text>'
)
row_labels = ("Actually\nnot busy", "Actually\nbusy")
for row, label in enumerate(row_labels):
y = top + row * cell
body.append(
f'<text x="{left - 18}" y="{y + cell / 2}" font-size="13" fill="#475569" '
f'text-anchor="end">{escape(label).replace(chr(10), " ")}</text>'
)
for (row, column), (code, value) in values.items():
x = left + column * cell
y = top + row * cell
intensity = 0.12 + 0.88 * (value / maximum)
red = round(219 + (29 - 219) * intensity)
green = round(234 + (78 - 234) * intensity)
blue = round(254 + (216 - 254) * intensity)
fill = f"#{red:02x}{green:02x}{blue:02x}"
text_color = "#ffffff" if intensity > 0.55 else TEXT_COLOR
row_total = matrix["tn"] + matrix["fp"] if row == 0 else matrix["fn"] + matrix["tp"]
share = 100.0 * value / row_total if row_total else 0.0
body.append(
f'<rect x="{x}" y="{y}" width="{cell}" height="{cell}" fill="{fill}" '
f'stroke="#ffffff" stroke-width="2"/>'
)
body.append(
f'<text x="{x + cell / 2}" y="{y + cell / 2 - 8}" font-size="30" '
f'font-weight="600" fill="{text_color}" text-anchor="middle">{value:,}</text>'
)
body.append(
f'<text x="{x + cell / 2}" y="{y + cell / 2 + 20}" font-size="13" '
f'fill="{text_color}" text-anchor="middle">{code} · {share:.1f}% of row</text>'
)
subtitle = (
f"n={metrics['n_test']:,} · accuracy {metrics['accuracy']:.3f} · "
f"precision {metrics['precision']:.3f} · recall {metrics['recall']:.3f} · "
f"F1 {metrics['f1']:.3f} · ROC-AUC {metrics['roc_auc']:.3f}"
)
return _svg_document(
"Small busy-hour classifier: holdout confusion matrix",
subtitle,
body,
width,
height,
footer="Source: artifacts/small_metrics.json · chronological 80/20 split",
)
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--artifacts",
type=Path,
default=config.artifacts_dir(),
help="artifact directory containing metrics.json (default: %(default)s)",
)
parser.add_argument(
"--out",
type=Path,
default=Path(__file__).resolve().parents[1] / "docs" / "figures",
help="output directory for SVG files (default: %(default)s)",
)
args = parser.parse_args(argv)
metrics = json.loads((args.artifacts / "metrics.json").read_text())
holdout = f"Holdout {metrics['test_from']} .. {metrics['test_to']}, n={metrics['metrics']['n_eval']}"
args.out.mkdir(parents=True, exist_ok=True)
ablation = _bar_chart(
"Ablation: holdout MAE by feature set",
holdout,
[
{
"name": row["name"].replace("_", " "),
"mae": row["mae"],
}
for row in metrics["ablation"]
],
production="week weather",
)
ablation_path = args.out / "ablation_mae.svg"
ablation_path.write_text(ablation)
per_sensor = _bar_chart(
"Per-sensor holdout MAE: production model vs seasonal naive",
holdout,
metrics["per_sensor"],
secondary_key="baseline_mae",
secondary_label="seasonal naive",
primary_label="weather-7d",
)
per_sensor_path = args.out / "per_sensor_mae.svg"
per_sensor_path.write_text(per_sensor)
confusion_path = args.out / "small_model_confusion.svg"
confusion_path.write_text(_confusion_svg(metrics["small"]))
print(f"wrote {ablation_path}")
print(f"wrote {per_sensor_path}")
print(f"wrote {confusion_path}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+153
View File
@@ -0,0 +1,153 @@
#!/usr/bin/env python3
"""Train and evaluate the small "busy hour" classifier from a CSV.
This is the repeatable artifact behind ``make small`` and the runtime
``model=small`` training job. It prints the headline metrics and per-sensor
table used in ``docs/small-model.md``; ``--save`` bakes the bundle and metrics
into ``artifacts/`` exactly like ``make train`` does.
Examples:
uv run python scripts/train_small.py
uv run python scripts/train_small.py --csv new-data.csv --all
uv run python scripts/train_small.py --save --json
"""
from __future__ import annotations
import argparse
import json
import sys
import time
from pathlib import Path
import joblib
import pandas as pd
from pulsecast import config
from pulsecast.small import load_hourly_csv, train_small_from_csv
def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument(
"csv_positional",
nargs="?",
metavar="CSV",
help=f"CSV to train from (default: {config.raw_csv_path()})",
)
parser.add_argument(
"--csv",
dest="csv_option",
help="CSV to train from; overrides the positional argument",
)
parser.add_argument(
"--start",
default=config.MULTI_SENSOR_START,
help=(
"drop rows before this UTC timestamp (default: %(default)s, the "
"8-sensor challenge window); ignored with --all"
),
)
parser.add_argument(
"--all",
action="store_true",
help="use every row in the CSV, including density-01's older history",
)
parser.add_argument(
"--save",
action="store_true",
help="write artifacts/small_model.joblib and artifacts/small_metrics.json",
)
parser.add_argument(
"--json",
action="store_true",
help="print the full metrics payload as JSON after the summary",
)
return parser.parse_args(argv)
def _confusion_lines(matrix: dict) -> list[str]:
tn, fp, fn, tp = matrix["tn"], matrix["fp"], matrix["fn"], matrix["tp"]
total = tn + fp + fn + tp
specificity = tn / (tn + fp) if tn + fp else 0.0
negative_predictive = tn / (tn + fn) if tn + fn else 0.0
return [
"Confusion matrix (rows = truth, columns = prediction)",
f" not busy: {tn:5d} {fp:5d}",
f" busy : {fn:5d} {tp:5d}",
f" n={total} specificity={specificity:.3f} NPV={negative_predictive:.3f}",
]
def main(argv: list[str] | None = None) -> int:
args = _parse_args(argv)
path = Path(args.csv_option or args.csv_positional or config.raw_csv_path())
if not path.exists():
print(f"CSV not found: {path}", file=sys.stderr)
return 2
start = None if args.all else pd.Timestamp(args.start)
started = time.perf_counter()
try:
bundle, payload = train_small_from_csv(path, start=start)
except (ValueError, FileNotFoundError) as exc:
print(f"training failed: {exc}", file=sys.stderr)
return 1
elapsed = time.perf_counter() - started
hourly = load_hourly_csv(path)
if start is not None:
hourly = hourly[hourly["t"] >= start]
metrics = payload["metrics"]
print(f"CSV {path}")
print(f"Rows {len(hourly):,} hourly rows, {metrics['sensors']} sensor(s)")
print(f"Window {payload['train_from']} .. {payload['test_to']}")
print(f"Split (time) train {payload['train_from']} .. {payload['train_to']}"
f" ({metrics['n_train']:,} rows)")
print(f" test {payload['test_from']} .. {payload['test_to']}"
f" ({metrics['n_test']:,} rows)")
print("Target busy = hourly density > sensor p75 (training window only)")
print(f"Features {', '.join(payload['features'])}")
print("Model standardized logistic regression (class_weight=balanced)")
print(f"Model version {payload['model_version']}")
print()
print("Metrics")
print(f" accuracy {metrics['accuracy']:.3f}")
print(f" precision {metrics['precision']:.3f}")
print(f" recall {metrics['recall']:.3f}")
print(f" f1 {metrics['f1']:.3f}")
print(f" roc_auc {metrics['roc_auc'] if metrics['roc_auc'] is not None else 'n/a'}")
print(f" base rate {metrics['positive_rate']:.3f}")
print()
print("\n".join(_confusion_lines(payload["confusion_matrix"])))
print()
print("Per sensor (holdout)")
print(f" {'sensor':<24} {'n':>6} {'pos':>6} {'acc':>6} {'prec':>6} {'rec':>6} {'f1':>6}")
for row in payload.get("per_sensor", []):
print(
f" {row['sensor']:<24} {row['n']:>6} {row['positive_rate']:>6.3f}"
f" {row['accuracy']:>6.3f} {row['precision']:>6.3f}"
f" {row['recall']:>6.3f} {row['f1']:>6.3f}"
)
print()
print(f"Fit time {elapsed:.2f} s (contract: < 5 s)")
if args.save:
artifacts = config.artifacts_dir()
artifacts.mkdir(parents=True, exist_ok=True)
joblib.dump(bundle, artifacts / "small_model.joblib")
(artifacts / "small_metrics.json").write_text(
json.dumps(payload, indent=2, ensure_ascii=False) + "\n"
)
print(f"Saved {artifacts / 'small_model.joblib'}")
print(f" {artifacts / 'small_metrics.json'}")
if args.json:
print()
print(json.dumps(payload, indent=2, ensure_ascii=False))
return 0
if __name__ == "__main__":
raise SystemExit(main())
+5 -6
View File
@@ -1,4 +1,4 @@
"""Configuration: paths, sensors, holidays, scenarios, contract versions. """Configuration: paths, sensors, holidays, features, scenarios.
All timestamps are UTC internally; Europe/Berlin is used only for calendar All timestamps are UTC internally; Europe/Berlin is used only for calendar
features and display-oriented payloads. features and display-oriented payloads.
@@ -65,8 +65,8 @@ def control_token() -> str:
# --------------------------------------------------------------------------- data # --------------------------------------------------------------------------- data
# Multi-sensor window documented in mission/data.md. density-01 has older # Multi-sensor window of the published dataset. density-01 has older history,
# history but the other seven locations start on 2026-06-12. # but the other seven locations start on 2026-06-12.
MULTI_SENSOR_START = "2026-06-12T00:00:00Z" MULTI_SENSOR_START = "2026-06-12T00:00:00Z"
# 28-day time-based holdout, measured back from the last observed hour. # 28-day time-based holdout, measured back from the last observed hour.
@@ -134,8 +134,8 @@ WEATHER_COLUMNS = {
"relative_humidity_2m": "humidity_pct", "relative_humidity_2m": "humidity_pct",
} }
# Baden-Württemberg public holidays, 2025-2026 (mission/data.md). Kept explicit # Baden-Württemberg public holidays, 2025-2026. Kept explicit so the build is
# so the build is deterministic and needs no holiday library. # deterministic and needs no holiday library.
HOLIDAYS: dict[str, str] = { HOLIDAYS: dict[str, str] = {
"2025-01-01": "Neujahr", "2025-01-01": "Neujahr",
"2025-01-06": "Heilige Drei Könige", "2025-01-06": "Heilige Drei Könige",
@@ -185,7 +185,6 @@ FEATURE_WEATHER = ["temp_c", "precip_mm"]
FEATURE_DAY_NO_HOLIDAY = FEATURE_CALENDAR_NO_HOLIDAY + FEATURE_DAY[6:] FEATURE_DAY_NO_HOLIDAY = FEATURE_CALENDAR_NO_HOLIDAY + FEATURE_DAY[6:]
FEATURE_DAY_NO_WEATHER = FEATURE_DAY FEATURE_DAY_NO_WEATHER = FEATURE_DAY
FEATURE_DAY_WEATHER = FEATURE_DAY + FEATURE_WEATHER FEATURE_DAY_WEATHER = FEATURE_DAY + FEATURE_WEATHER
FEATURE_DAY_HOLIDAY = FEATURE_DAY + ["is_holiday"] # is_holiday already present; kept for clarity
FEATURE_WEEK_NO_WEATHER = FEATURE_WEEK FEATURE_WEEK_NO_WEATHER = FEATURE_WEEK
FEATURE_WEEK_WEATHER = FEATURE_WEEK + FEATURE_WEATHER FEATURE_WEEK_WEATHER = FEATURE_WEEK + FEATURE_WEATHER
+2 -2
View File
@@ -16,7 +16,7 @@ from . import config
from .features import SENSOR_CODES, future_calendar from .features import SENSOR_CODES, future_calendar
# Fixed, deliberately small model. No tuning beyond a short sanity pass: the # Fixed, deliberately small model. No tuning beyond a short sanity pass: the
# 28-day holdout is the honest signal, not the training loss. # 28-day holdout decides, not the training loss.
HISTGB_PARAMS = { HISTGB_PARAMS = {
"loss": "squared_error", "loss": "squared_error",
"learning_rate": 0.06, "learning_rate": 0.06,
@@ -58,7 +58,7 @@ def metrics(y_true, y_pred) -> dict:
def residual_quantiles(eval_frame: pd.DataFrame, pred: np.ndarray) -> dict: def residual_quantiles(eval_frame: pd.DataFrame, pred: np.ndarray) -> dict:
"""80% prediction band offsets from the honest holdout residuals.""" """80% prediction band offsets from the holdout residuals."""
residual = np.asarray(eval_frame["y"], dtype=float) - np.asarray(pred, dtype=float) residual = np.asarray(eval_frame["y"], dtype=float) - np.asarray(pred, dtype=float)
return { return {
"p10": round(float(np.quantile(residual, 0.10)), 4), "p10": round(float(np.quantile(residual, 0.10)), 4),
+4 -3
View File
@@ -1,8 +1,9 @@
"""Parse and clean the raw PAX CSV. """Parse and clean the raw PAX CSV.
Verifies the claims in mission/data.md: 192,464 rows, 12 device IDs, 8 usable The audit checks the counts the pipeline depends on: 192,464 rows, 12 device
logical locations, three excluded test devices, and the density-01 merge. IDs, 8 usable logical locations, three excluded test devices, and the
The raw file is never written to git; a cached parquet copy lives in data/cache/. density-01 merge. The raw file is never written to git; a cached parquet copy
lives in data/cache/.
""" """
from __future__ import annotations from __future__ import annotations
+7 -6
View File
@@ -1,7 +1,7 @@
"""FastAPI ML service: /healthz, /readyz, /meta, /bootstrap, /predict, /simulate, /train. """FastAPI ML service: /healthz, /readyz, /meta, /bootstrap, /predict, /simulate, /train.
Shapes follow CONTRACT.md. The service is internal: only Go exposes it publicly, The service is internal: only Go exposes it publicly, and the training routes
and the training routes require a bearer control token. require a bearer control token.
""" """
from __future__ import annotations from __future__ import annotations
@@ -79,10 +79,9 @@ def _load_runtime(runtime: Runtime) -> None:
json.loads(meta_path.read_text()) if meta_path.exists() else None json.loads(meta_path.read_text()) if meta_path.exists() else None
) )
# Baked snapshot artifacts served by GET /bootstrap. Go caches this once at # Go caches /bootstrap once at startup, so load it at boot and keep the
# startup, so loading them at boot keeps the endpoint a pure in-memory read. # endpoint a pure in-memory read. Project to contract fields because the
# Project to the contract fields; artifacts carry extra audit fields # artifacts carry extra audit fields (last_seen).
# (last_seen) that Go does not need.
sensors_path = artifacts / "sensors.json" sensors_path = artifacts / "sensors.json"
if sensors_path.exists(): if sensors_path.exists():
sensors = json.loads(sensors_path.read_text()) sensors = json.loads(sensors_path.read_text())
@@ -653,6 +652,8 @@ def create_app() -> FastAPI:
state = runtime() state = runtime()
model, path = await _parse_train_request(request) model, path = await _parse_train_request(request)
# Serialize check-and-start so two requests cannot both pass the
# "no active job" check.
with state.schedule_lock: with state.schedule_lock:
with state.jobs_lock: with state.jobs_lock:
if any(job["status"] in ACTIVE_STATUSES for job in state.jobs.values()): if any(job["status"] in ACTIVE_STATUSES for job in state.jobs.values()):
+33 -2
View File
@@ -17,6 +17,7 @@ import pandas as pd
from sklearn.linear_model import LogisticRegression from sklearn.linear_model import LogisticRegression
from sklearn.metrics import ( from sklearn.metrics import (
accuracy_score, accuracy_score,
confusion_matrix,
f1_score, f1_score,
precision_score, precision_score,
recall_score, recall_score,
@@ -92,8 +93,8 @@ def _attach_weather(hourly: pd.DataFrame, archive: pd.DataFrame | None) -> pd.Da
frame[column] = frame[column].where(frame[column].notna(), frame[archive_column]) frame[column] = frame[column].where(frame[column].notna(), frame[archive_column])
frame = frame.drop(columns=[archive_column]) frame = frame.drop(columns=[archive_column])
# Honest fallback: no weather available for the uploaded window. Constant # No weather for the uploaded window: constant values keep the other
# values keep the other features useful; the job metrics show the effect. # features usable, and the job metrics show the effect.
frame["temp_c"] = frame["temp_c"].fillna(14.0) frame["temp_c"] = frame["temp_c"].fillna(14.0)
frame["precip_mm"] = frame["precip_mm"].fillna(0.0) frame["precip_mm"] = frame["precip_mm"].fillna(0.0)
return frame return frame
@@ -254,6 +255,28 @@ def train_small_from_csv(
except ValueError: except ValueError:
roc_auc = None roc_auc = None
tn, fp, fn, tp = confusion_matrix(truth, predicted, labels=[0, 1]).ravel()
scored = test.assign(_predicted=predicted)
per_sensor = []
for sensor, group in scored.groupby("sensor", observed=True):
sensor_truth = group["busy"].to_numpy(dtype=int)
sensor_pred = group["_predicted"].to_numpy(dtype=int)
per_sensor.append(
{
"sensor": str(sensor),
"n": len(group),
"positive_rate": round(float(sensor_truth.mean()), 3),
"accuracy": round(float(accuracy_score(sensor_truth, sensor_pred)), 3),
"precision": round(
float(precision_score(sensor_truth, sensor_pred, zero_division=0)), 3
),
"recall": round(
float(recall_score(sensor_truth, sensor_pred, zero_division=0)), 3
),
"f1": round(float(f1_score(sensor_truth, sensor_pred, zero_division=0)), 3),
}
)
trained_at = _utc_now() trained_at = _utc_now()
model_version = f"logit-{trained_at[:10].replace('-', '')}-{trained_at[11:19].replace(':', '')}" model_version = f"logit-{trained_at[:10].replace('-', '')}-{trained_at[11:19].replace(':', '')}"
metrics_payload = { metrics_payload = {
@@ -278,6 +301,14 @@ def train_small_from_csv(
"sensors": int(hourly["sensor"].nunique()), "sensors": int(hourly["sensor"].nunique()),
} }
), ),
"confusion_matrix": {
"labels": {"0": "not busy", "1": "busy"},
"tn": int(tn),
"fp": int(fp),
"fn": int(fn),
"tp": int(tp),
},
"per_sensor": per_sensor,
} }
bundle = { bundle = {
"model": model, "model": model,
+43 -5
View File
@@ -142,7 +142,35 @@ def train_bundle(
baseline = by_name["seasonal_naive"] baseline = by_name["seasonal_naive"]
headline = by_name["week_weather"] headline = by_name["week_weather"]
# Production model: refit on all observed rows after the honest holdout. # Per-sensor holdout breakdown for the paper; same rows as the headline.
scored = eval_frame.copy()
scored["_model"] = preds["week_weather"]
scored["_baseline"] = preds["seasonal_naive"]
per_sensor = []
for sensor, group in scored.groupby("sensor", observed=True):
model_row = metrics(group["y"], group["_model"])
baseline_row = metrics(group["y"], group["_baseline"])
improvement = (
100.0 * (baseline_row["mae"] - model_row["mae"]) / baseline_row["mae"]
if baseline_row["mae"]
else 0.0
)
per_sensor.append(
{
"sensor": str(sensor),
"name": config.DISPLAY_NAMES.get(str(sensor), str(sensor)),
"n": model_row["n"],
"mae": round(model_row["mae"], 2),
"wape": round(model_row["wape"], 1),
"r2": round(model_row["r2"], 3),
"baseline_mae": round(baseline_row["mae"], 2),
"baseline_wape": round(baseline_row["wape"], 1),
"baseline_r2": round(baseline_row["r2"], 3),
"mae_improvement_pct": round(improvement, 1),
}
)
# Production model: refit on all observed rows after the holdout is scored.
production = fit(features, config.FEATURE_WEEK_WEATHER) production = fit(features, config.FEATURE_WEEK_WEATHER)
residual = residual_quantiles(eval_frame, preds["week_weather"]) residual = residual_quantiles(eval_frame, preds["week_weather"])
sensor_mean = hourly.groupby("sensor", observed=True)["v"].mean().round(4).to_dict() sensor_mean = hourly.groupby("sensor", observed=True)["v"].mean().round(4).to_dict()
@@ -207,6 +235,7 @@ def train_bundle(
"features": list(config.FEATURE_WEEK_WEATHER), "features": list(config.FEATURE_WEEK_WEATHER),
"holiday_effect": holiday_effect, "holiday_effect": holiday_effect,
"ablation": [_round_row(row) for row in results], "ablation": [_round_row(row) for row in results],
"per_sensor": per_sensor,
"limits": list(config.LIMITS), "limits": list(config.LIMITS),
} }
return bundle, metrics_payload, results return bundle, metrics_payload, results
@@ -267,10 +296,6 @@ def run_train() -> dict:
bundle, metrics_payload, _ = train_bundle(hourly, features, archive, meta, audit) bundle, metrics_payload, _ = train_bundle(hourly, features, archive, meta, audit)
joblib.dump(bundle, artifacts / "model.joblib") joblib.dump(bundle, artifacts / "model.joblib")
(artifacts / "metrics.json").write_text(
json.dumps(metrics_payload, indent=2, ensure_ascii=False) + "\n"
)
start = pd.Timestamp.now(tz="UTC").ceil("h") start = pd.Timestamp.now(tz="UTC").ceil("h")
forecasts = _export_forecasts(bundle, forecast, start) forecasts = _export_forecasts(bundle, forecast, start)
(artifacts / "forecasts.json").write_text( (artifacts / "forecasts.json").write_text(
@@ -289,6 +314,19 @@ def run_train() -> dict:
json.dumps(small_metrics, indent=2, ensure_ascii=False) + "\n" json.dumps(small_metrics, indent=2, ensure_ascii=False) + "\n"
) )
# One self-contained file for the paper: ablation + per-sensor + small model.
metrics_payload["small"] = {
"model_version": small_metrics["model_version"],
"trained_at": small_metrics["trained_at"],
"features": small_metrics["features"],
"metrics": small_metrics["metrics"],
"confusion_matrix": small_metrics["confusion_matrix"],
"per_sensor": small_metrics["per_sensor"],
}
(artifacts / "metrics.json").write_text(
json.dumps(metrics_payload, indent=2, ensure_ascii=False) + "\n"
)
log.info( log.info(
"wrote metrics.json, forecasts.json (%s..%s), model.joblib, small_model.joblib", "wrote metrics.json, forecasts.json (%s..%s), model.joblib, small_model.joblib",
start, start,
+4 -4
View File
@@ -1,8 +1,8 @@
"""Open-Meteo weather: historical archive for training, forecast for scenarios. """Open-Meteo weather: historical archive for training, forecast for scenarios.
No API key. Responses are cached under data/cache/ so `make train` and the No API key. Responses are cached under data/cache/ so `make train` needs no
Docker image need no network. The raw CSV and cache stay out of git; the final network after the first fetch. The raw CSV and cache stay out of git; compact
context.json is committed. frames and context.json are committed under artifacts/.
""" """
from __future__ import annotations from __future__ import annotations
@@ -128,7 +128,7 @@ def ensure_weather(
*, *,
refresh: bool = False, refresh: bool = False,
) -> tuple[pd.DataFrame, pd.DataFrame, dict]: ) -> tuple[pd.DataFrame, pd.DataFrame, dict]:
"""Load cached weather, fetching only when missing, stale, or refresh=True.""" """Load cached weather, fetching only when the cache is missing or refresh=True."""
cache = config.cache_dir() cache = config.cache_dir()
cache.mkdir(parents=True, exist_ok=True) cache.mkdir(parents=True, exist_ok=True)
archive_path = cache / ARCHIVE_FILE archive_path = cache / ARCHIVE_FILE
+18
View File
@@ -67,6 +67,24 @@ def test_metrics_and_audit():
assert metrics["metrics"]["wape"] < metrics["metrics"]["baseline_wape"] assert metrics["metrics"]["wape"] < metrics["metrics"]["baseline_wape"]
assert len(metrics["features"]) == 10 assert len(metrics["features"]) == 10
assert len(metrics["limits"]) >= 4 assert len(metrics["limits"]) >= 4
assert len(metrics["per_sensor"]) == 8
assert sum(row["n"] for row in metrics["per_sensor"]) == metrics["metrics"]["n_eval"]
assert {row["sensor"] for row in metrics["per_sensor"]} == set(config.USABLE_SENSORS)
for row in metrics["per_sensor"]:
assert row["mae"] >= 0
assert row["baseline_mae"] >= 0
assert 0 <= row["wape"] < 100
assert row["mae_improvement_pct"] > 0
small = _load("small_metrics.json")
matrix = small["confusion_matrix"]
total = matrix["tn"] + matrix["fp"] + matrix["fn"] + matrix["tp"]
assert total == small["metrics"]["n_test"]
assert sum(row["n"] for row in small["per_sensor"]) == total
recomputed = (matrix["tn"] + matrix["tp"]) / total
assert abs(recomputed - small["metrics"]["accuracy"]) < 0.001
assert metrics["small"]["metrics"] == small["metrics"]
assert set(matrix) == {"labels", "tn", "fp", "fn", "tp"}
audit = _load("data_audit.json") audit = _load("data_audit.json")
assert audit["all_checks_ok"] is True assert audit["all_checks_ok"] is True