feat: ci docs #3
No files matched your search
@@ -18,13 +18,14 @@ jobs:
|
|||||||
- name: Check out
|
- name: Check out
|
||||||
uses: actions/checkout@v4
|
uses: actions/checkout@v4
|
||||||
|
|
||||||
- name: Set up Python
|
# setup-python needs the runner toolcache and lsb_release; this Gitea
|
||||||
uses: actions/setup-python@v5
|
# runner has neither. uv installs its own Python 3.12.
|
||||||
|
- name: Set up uv
|
||||||
|
uses: astral-sh/setup-uv@v6
|
||||||
with:
|
with:
|
||||||
|
version: "0.10.4"
|
||||||
python-version: "3.12"
|
python-version: "3.12"
|
||||||
|
enable-cache: false
|
||||||
- name: Install uv
|
|
||||||
run: python -m pip install "uv==0.10.4"
|
|
||||||
|
|
||||||
# Same checks as local `make check`, split into steps for readable CI
|
# Same checks as local `make check`, split into steps for readable CI
|
||||||
# failures. No secrets, no image push, no cluster access.
|
# failures. No secrets, no image push, no cluster access.
|
||||||
@@ -32,10 +33,10 @@ jobs:
|
|||||||
run: uv sync --frozen
|
run: uv sync --frozen
|
||||||
|
|
||||||
- name: Static analysis (ruff)
|
- name: Static analysis (ruff)
|
||||||
run: uv run ruff check src tests main.py
|
run: uv run ruff check src tests scripts main.py
|
||||||
|
|
||||||
- name: Byte-compile
|
- name: Byte-compile
|
||||||
run: uv run python -m compileall src
|
run: uv run python -m compileall src scripts
|
||||||
|
|
||||||
- name: Tests
|
- name: Tests
|
||||||
run: uv run pytest -q
|
run: uv run pytest -q
|
||||||
@@ -1,11 +1,13 @@
|
|||||||
UV ?= uv
|
UV ?= uv
|
||||||
|
|
||||||
.PHONY: help data train test lint check serve docker clean
|
.PHONY: help data train small figures test lint check serve docker clean
|
||||||
|
|
||||||
help:
|
help:
|
||||||
@printf '%s\n' \
|
@printf '%s\n' \
|
||||||
'make data parse/clean the raw CSV, join weather + holidays, export JSON' \
|
'make data parse/clean the raw CSV, join weather + holidays, export JSON' \
|
||||||
'make train fit the 28-day holdout model, export metrics/forecasts/model' \
|
'make train fit the 28-day holdout model, export metrics/forecasts/model' \
|
||||||
|
'make small train + print the busy-hour classifier metrics from the raw CSV' \
|
||||||
|
'make figures regenerate docs/figures/*.svg from artifacts/metrics.json' \
|
||||||
'make test run pytest' \
|
'make test run pytest' \
|
||||||
'make lint run ruff static checks' \
|
'make lint run ruff static checks' \
|
||||||
'make check CI gate: sync --frozen, ruff, compileall, pytest' \
|
'make check CI gate: sync --frozen, ruff, compileall, pytest' \
|
||||||
@@ -19,16 +21,22 @@ data:
|
|||||||
train:
|
train:
|
||||||
$(UV) run python -m pulsecast.train
|
$(UV) run python -m pulsecast.train
|
||||||
|
|
||||||
|
small:
|
||||||
|
$(UV) run python scripts/train_small.py
|
||||||
|
|
||||||
|
figures:
|
||||||
|
$(UV) run python scripts/plot_metrics.py
|
||||||
|
|
||||||
test:
|
test:
|
||||||
$(UV) run pytest
|
$(UV) run pytest
|
||||||
|
|
||||||
lint:
|
lint:
|
||||||
$(UV) run ruff check src tests main.py
|
$(UV) run ruff check src tests scripts main.py
|
||||||
|
|
||||||
check:
|
check:
|
||||||
$(UV) sync --frozen
|
$(UV) sync --frozen
|
||||||
$(UV) run ruff check src tests main.py
|
$(UV) run ruff check src tests scripts main.py
|
||||||
$(UV) run python -m compileall src
|
$(UV) run python -m compileall src scripts
|
||||||
$(UV) run pytest -q
|
$(UV) run pytest -q
|
||||||
|
|
||||||
serve:
|
serve:
|
||||||
|
|||||||
@@ -1,11 +1,10 @@
|
|||||||
# PulseCast ML
|
# PulseCast ML
|
||||||
|
|
||||||
Footfall pipeline, weather-aware 7-day forecast, scenario engine, and FastAPI
|
Footfall pipeline, weather-aware 7-day forecast, scenario engine, and FastAPI
|
||||||
service for the PulseCast staffing copilot.
|
service for the PulseCast staffing copilot. It turns raw 15-minute PAX counter
|
||||||
|
rows into hourly series, joins weather and Baden-Württemberg holidays, trains
|
||||||
The reusable asset is the pipeline: raw 15-minute PAX counters → clean hourly
|
one global model, evaluates it on a 28-day time-based holdout, and serves the
|
||||||
series → observed weather + Baden-Württemberg holidays → global model →
|
resulting JSON artifacts.
|
||||||
honest holdout metrics → JSON artifacts → decision-ready API.
|
|
||||||
|
|
||||||
## Results
|
## Results
|
||||||
|
|
||||||
@@ -24,9 +23,10 @@ honest holdout metrics → JSON artifacts → decision-ready API.
|
|||||||
The seasonal-naive row reproduces the mission reference exactly
|
The seasonal-naive row reproduces the mission reference exactly
|
||||||
(5.40 / 44.7% / 0.462), and the day-ahead weather row matches 3.36 / 27.8%.
|
(5.40 / 44.7% / 0.462), and the day-ahead weather row matches 3.36 / 27.8%.
|
||||||
Weather cuts error by ~19% versus the no-weather 7-day model and ~38% versus
|
Weather cuts error by ~19% versus the no-weather 7-day model and ~38% versus
|
||||||
seasonal naive. The production model is slightly better than the reference
|
seasonal naive. The production 7-day model edges the reference day-ahead
|
||||||
because its seasonal feature uses the previous *observed* same weekday/hour
|
weather row (3.32 vs 3.36 MAE). The two use different features; `lag_week` uses
|
||||||
across sensor gaps; the baseline uses the exact 168-row lag.
|
the previous *observed* same weekday/hour across sensor gaps, while the
|
||||||
|
seasonal-naive baseline uses the exact 168-row lag.
|
||||||
|
|
||||||
Small busy-hour classifier (baked for the "train new data" demo):
|
Small busy-hour classifier (baked for the "train new data" demo):
|
||||||
|
|
||||||
@@ -37,30 +37,35 @@ Small busy-hour classifier (baked for the "train new data" demo):
|
|||||||
The classifier predicts `busy` = hourly footfall above the sensor's p75 using
|
The classifier predicts `busy` = hourly footfall above the sensor's p75 using
|
||||||
hour, weekday, weekend, temperature, rain, and holiday. It trains in under a
|
hour, weekday, weekend, temperature, rain, and holiday. It trains in under a
|
||||||
second on the challenge CSV and is retrained from an uploaded CSV through
|
second on the challenge CSV and is retrained from an uploaded CSV through
|
||||||
`POST /train`.
|
`POST /train`. Full definition, confusion matrix, per-sensor table, and limits:
|
||||||
|
[`docs/small-model.md`](docs/small-model.md). Paper tables and figures:
|
||||||
|
[`docs/metrics.md`](docs/metrics.md).
|
||||||
|
|
||||||
## Quick start
|
## Quick start
|
||||||
|
|
||||||
Requirements: Python 3.12 and `uv` (the repo also works with a plain venv).
|
Requires Python 3.12 and `uv`.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv sync
|
uv sync
|
||||||
make data # needs the raw CSV, first run also fetches Open-Meteo weather
|
make serve # uses the committed artifacts; http://localhost:8000
|
||||||
|
make check # same gate as CI: sync --frozen, ruff, compileall, pytest
|
||||||
|
```
|
||||||
|
|
||||||
|
To rebuild the artifacts from raw data:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make data # needs the raw CSV; first run also fetches Open-Meteo weather
|
||||||
make train # writes artifacts/model.joblib, metrics.json, forecasts.json
|
make train # writes artifacts/model.joblib, metrics.json, forecasts.json
|
||||||
|
make small # prints the busy-hour classifier metrics + confusion matrix
|
||||||
|
make figures # regenerates docs/figures/*.svg from metrics.json
|
||||||
make test
|
make test
|
||||||
make check # CI gate: sync --frozen, ruff, compileall, pytest
|
|
||||||
make serve # http://localhost:8000
|
|
||||||
```
|
```
|
||||||
|
|
||||||
The raw challenge CSV is **never committed**. Put it at:
|
The raw challenge CSV is **never committed**. The default path is
|
||||||
|
`/home/noel/Documents/Hackathon_2026/data/raw/schwarz-pax.csv`; set
|
||||||
```text
|
`PULSECAST_RAW_CSV` to use another file. `data/raw/` and `data/cache/` are
|
||||||
/home/noel/Documents/Hackathon_2026/data/raw/schwarz-pax.csv
|
git-ignored. The committed `artifacts/` (JSON + joblib + compact weather frames)
|
||||||
```
|
let the API and Docker image run without the raw file or network.
|
||||||
|
|
||||||
or point `PULSECAST_RAW_CSV` at it. `data/raw/` and `data/cache/` are git-ignored.
|
|
||||||
The generated `artifacts/` (JSON + joblib + compact weather frames) are
|
|
||||||
committed so the API and Docker image work without the raw file or network.
|
|
||||||
|
|
||||||
## Artifacts
|
## Artifacts
|
||||||
|
|
||||||
@@ -69,16 +74,18 @@ committed so the API and Docker image work without the raw file or network.
|
|||||||
| `artifacts/sensors.json` | 8 sensors: id, name, lat/lon, mean, p95, battery |
|
| `artifacts/sensors.json` | 8 sensors: id, name, lat/lon, mean, p95, battery |
|
||||||
| `artifacts/series.json` + `series/<id>.json` | Hourly series per sensor |
|
| `artifacts/series.json` + `series/<id>.json` | Hourly series per sensor |
|
||||||
| `artifacts/forecasts.json` | 48h + 7d forecasts for 4 scenarios |
|
| `artifacts/forecasts.json` | 48h + 7d forecasts for 4 scenarios |
|
||||||
| `artifacts/metrics.json` | Versions, coverage, headline metrics, ablation, limits |
|
| `artifacts/metrics.json` | Versions, coverage, headline metrics, ablation, per-sensor, small summary, limits |
|
||||||
|
| `artifacts/small_metrics.json` | Busy classifier: metrics, confusion matrix, per-sensor, thresholds |
|
||||||
| `artifacts/context.json` | Current + 7-day weather, BW holidays |
|
| `artifacts/context.json` | Current + 7-day weather, BW holidays |
|
||||||
| `artifacts/data_audit.json` | Row/sensor checks against `mission/data.md` |
|
| `artifacts/data_audit.json` | Row/device checks for the raw CSV |
|
||||||
| `artifacts/model.joblib` | Baked HistGB forecast bundle |
|
| `artifacts/model.joblib` | Baked HistGB forecast bundle |
|
||||||
| `artifacts/small_model.joblib` | Baked busy classifier |
|
| `artifacts/small_model.joblib` | Baked busy classifier |
|
||||||
| `artifacts/weather_*.parquet` | Baked weather frames for retraining/inference |
|
| `artifacts/weather_*.parquet` | Baked weather frames for retraining/inference |
|
||||||
|
|
||||||
## API
|
## API
|
||||||
|
|
||||||
Shapes follow `CONTRACT.md`. The service is internal; Go exposes it publicly.
|
Response shapes match the Go service contract. The service is internal; Go
|
||||||
|
exposes it publicly.
|
||||||
|
|
||||||
| Method | Path | Notes |
|
| Method | Path | Notes |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
@@ -118,6 +125,29 @@ and persists to `PULSECAST_MODELS` (default `/tmp/pulsecast/models`). A restart
|
|||||||
falls back to the baked model. `model=small` is the fast demo; `model=forecast`
|
falls back to the baked model. `model=small` is the fast demo; `model=forecast`
|
||||||
retrains the global HistGB (slower, still background).
|
retrains the global HistGB (slower, still background).
|
||||||
|
|
||||||
|
## End-to-end check
|
||||||
|
|
||||||
|
`scripts/e2e_train.py` starts a real uvicorn process on port 18000 with a
|
||||||
|
control token, then checks the snapshot, every `/predict` combination, every
|
||||||
|
`/simulate` scenario, auth and error paths, one-job-at-a-time, and both real
|
||||||
|
training jobs (small + forecast) including hot-swap and `/predict` afterward:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python scripts/e2e_train.py --port 18000 --report /tmp/e2e.json
|
||||||
|
```
|
||||||
|
|
||||||
|
The unit and contract tests are `make test`; `make check` is the CI gate.
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
| File | Contents |
|
||||||
|
|---|---|
|
||||||
|
| `docs/metrics.md` | Paper tables/figures: protocol, headline, ablation, per sensor, uncertainty, small model, claim limits |
|
||||||
|
| `docs/small-model.md` | Busy-hour target, thresholds, split, metrics, confusion matrix, per sensor, limits |
|
||||||
|
| `docs/train-e2e.md` | Exact live training-job evidence (small + forecast, hot-swap, errors) |
|
||||||
|
| `docs/figures/*.svg` | Ablation MAE, per-sensor MAE, small-model confusion matrix (generated) |
|
||||||
|
| `model_card.md` | Model card: features, split, baseline, ablation, leakage, limitations |
|
||||||
|
|
||||||
## Docker
|
## Docker
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -132,9 +162,11 @@ artifacts baked in, writable paths only under `/tmp`.
|
|||||||
|
|
||||||
| Variable | Default | Use |
|
| Variable | Default | Use |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
|
| `PULSECAST_ROOT` | first parent with `pyproject.toml` | Project root |
|
||||||
| `PULSECAST_RAW_CSV` | challenge path | Raw CSV location |
|
| `PULSECAST_RAW_CSV` | challenge path | Raw CSV location |
|
||||||
| `PULSECAST_ARTIFACTS` | `./artifacts` | Baked JSON/model output |
|
| `PULSECAST_ARTIFACTS` | `./artifacts` | Baked JSON/model output |
|
||||||
| `PULSECAST_CACHE` | `./data/cache` | Parquet/weather cache |
|
| `PULSECAST_CACHE` | `./data/cache` | Parquet/weather cache |
|
||||||
|
| `PULSECAST_WEATHER` | `PULSECAST_ARTIFACTS` | Baked weather frame location |
|
||||||
| `PULSECAST_MODELS` | `/tmp/pulsecast/models` | Retrained model persistence |
|
| `PULSECAST_MODELS` | `/tmp/pulsecast/models` | Retrained model persistence |
|
||||||
| `PULSECAST_UPLOADS` | `/tmp/pulsecast/uploads` | Multipart upload landing |
|
| `PULSECAST_UPLOADS` | `/tmp/pulsecast/uploads` | Multipart upload landing |
|
||||||
| `CONTROL_TOKEN` | unset | Required by `/train`; service returns 503 if unset |
|
| `CONTROL_TOKEN` | unset | Required by `/train`; service returns 503 if unset |
|
||||||
@@ -153,6 +185,8 @@ src/pulsecast/
|
|||||||
small.py busy classifier + flexible CSV parser
|
small.py busy classifier + flexible CSV parser
|
||||||
retrain.py runtime forecast retraining from upload
|
retrain.py runtime forecast retraining from upload
|
||||||
server.py FastAPI app and training job manager
|
server.py FastAPI app and training job manager
|
||||||
|
scripts/ train_small.py, plot_metrics.py, e2e_train.py
|
||||||
|
docs/ metrics.md, small-model.md, train-e2e.md, generated figures
|
||||||
tests/ schema, bootstrap, health, prediction, small-training job
|
tests/ schema, bootstrap, health, prediction, small-training job
|
||||||
model_card.md features, split, baseline, ablation, limits, leakage
|
model_card.md features, split, baseline, ablation, limits, leakage
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -1 +1 @@
|
|||||||
{"data_version": "2026-11-07T23:51:00Z", "model_version": "histgb-2026-11-14", "context_version": "2026-11-14T03:47:09Z", "generated_at": "2026-11-14T04:03:38Z", "forecast_start": "2026-11-14T05:00:00Z", "forecasts": {"density-01": {"48h": {"base": [{"t": "2026-11-14T05:00:00Z", "v": 1.17, "lo": 0.0, "hi": 6.04}, {"t": "2026-11-14T06:00:00Z", "v": 1.95, "lo": 0.0, "hi": 6.83}, {"t": "2026-11-14T07:00:00Z", "v": 2.94, "lo": 0.0, "hi": 7.81}, {"t": "2026-11-14T08:00:00Z", "v": 4.08, "lo": 0.0, "hi": 8.96}, {"t": "2026-11-14T09:00:00Z", "v": 5.68, "lo": 0.85, "hi": 10.55}, {"t": "2026-11-14T10:00:00Z", "v": 8.5, "lo": 3.67, "hi": 13.38}, {"t": "2026-11-14T11:00:00Z", "v": 9.66, "lo": 4.84, "hi": 14.54}, {"t": "2026-11-14T12:00:00Z", "v": 11.39, "lo": 6.56, "hi": 16.26}, {"t": "2026-11-14T13:00:00Z", "v": 11.32, "lo": 6.5, "hi": 16.2}, {"t": "2026-11-14T14:00:00Z", "v": 12.63, "lo": 7.81, "hi": 17.51}, {"t": "2026-11-14T15:00:00Z", "v": 14.78, "lo": 9.96, "hi": 19.66}, {"t": "2026-11-14T16:00:00Z", "v": 14.21, "lo": 9.39, "hi": 19.09}, {"t": "2026-11-14T17:00:00Z", "v": 14.52, "lo": 9.7, "hi": 19.4}, {"t": "2026-11-14T18:00:00Z", "v": 15.67, "lo": 10.84, "hi": 20.55}, {"t": "2026-11-14T19:00:00Z", "v": 14.37, "lo": 9.55, "hi": 19.25}, {"t": "2026-11-14T20:00:00Z", "v": 12.64, "lo": 7.81, "hi": 17.51}, {"t": "2026-11-14T21:00:00Z", "v": 14.52, "lo": 9.7, "hi": 19.4}, {"t": "2026-11-14T22:00:00Z", "v": 12.33, "lo": 7.5, "hi": 17.21}, {"t": "2026-11-14T23:00:00Z", "v": 9.87, "lo": 5.04, "hi": 14.74}, {"t": "2026-11-15T00:00:00Z", "v": 3.94, "lo": 0.0, "hi": 8.82}, {"t": "2026-11-15T01:00:00Z", "v": 1.34, "lo": 0.0, "hi": 6.22}, {"t": "2026-11-15T02:00:00Z", "v": 1.13, "lo": 0.0, "hi": 6.01}, {"t": "2026-11-15T03:00:00Z", "v": 0.99, "lo": 0.0, "hi": 5.86}, {"t": "2026-11-15T04:00:00Z", "v": 0.99, "lo": 0.0, "hi": 5.86}, {"t": "2026-11-15T05:00:00Z", "v": 1.01, "lo": 0.0, "hi": 5.88}, {"t": "2026-11-15T06:00:00Z", "v": 1.04, "lo": 0.0, "hi": 5.92}, {"t": "2026-11-15T07:00:00Z", "v": 1.55, "lo": 0.0, "hi": 6.43}, {"t": "2026-11-15T08:00:00Z", "v": 1.65, "lo": 0.0, "hi": 6.53}, {"t": "2026-11-15T09:00:00Z", "v": 5.61, "lo": 0.78, "hi": 10.48}, {"t": "2026-11-15T10:00:00Z", "v": 6.5, "lo": 1.68, "hi": 11.38}, {"t": "2026-11-15T11:00:00Z", "v": 7.51, "lo": 2.69, "hi": 12.39}, {"t": "2026-11-15T12:00:00Z", "v": 9.58, "lo": 4.76, "hi": 14.46}, {"t": "2026-11-15T13:00:00Z", "v": 11.7, "lo": 6.88, "hi": 16.58}, {"t": "2026-11-15T14:00:00Z", "v": 18.36, "lo": 13.54, "hi": 23.24}, {"t": "2026-11-15T15:00:00Z", "v": 17.33, "lo": 12.5, "hi": 22.21}, {"t": "2026-11-15T16:00:00Z", "v": 16.04, "lo": 11.21, "hi": 20.91}, {"t": "2026-11-15T17:00:00Z", "v": 11.34, "lo": 6.51, "hi": 16.21}, {"t": "2026-11-15T18:00:00Z", "v": 10.87, "lo": 6.05, "hi": 15.75}, {"t": "2026-11-15T19:00:00Z", "v": 8.45, "lo": 3.63, "hi": 13.33}, {"t": "2026-11-15T20:00:00Z", "v": 5.82, "lo": 0.99, "hi": 10.7}, {"t": "2026-11-15T21:00:00Z", "v": 3.71, "lo": 0.0, "hi": 8.59}, {"t": "2026-11-15T22:00:00Z", "v": 2.11, "lo": 0.0, "hi": 6.99}, {"t": "2026-11-15T23:00:00Z", "v": 1.47, "lo": 0.0, "hi": 6.34}, {"t": "2026-11-16T00:00:00Z", "v": 0.91, "lo": 0.0, "hi": 5.79}, {"t": "2026-11-16T01:00:00Z", "v": 0.88, "lo": 0.0, "hi": 5.76}, {"t": "2026-11-16T02:00:00Z", "v": 1.15, "lo": 0.0, "hi": 6.03}, {"t": "2026-11-16T03:00:00Z", "v": 1.15, "lo": 0.0, "hi": 6.03}, {"t": "2026-11-16T04:00:00Z", "v": 1.74, "lo": 0.0, "hi": 6.62}], "sunny-saturday-market": [{"t": "2026-11-14T05:00:00Z", "v": 1.17, "lo": 0.0, "hi": 6.04}, {"t": "2026-11-14T06:00:00Z", "v": 1.95, "lo": 0.0, "hi": 6.83}, {"t": "2026-11-14T07:00:00Z", "v": 2.94, "lo": 0.0, "hi": 7.81}, {"t": "2026-11-14T08:00:00Z", "v": 4.08, "lo": 0.0, "hi": 8.96}, {"t": "2026-11-14T09:00:00Z", "v": 7.63, "lo": 2.81, "hi": 12.51}, {"t": "2026-11-14T10:00:00Z", "v": 10.14, "lo": 5.32, "hi": 15.02}, {"t": "2026-11-14T11:00:00Z", "v": 11.16, "lo": 6.34, "hi": 16.04}, {"t": "2026-11-14T12:00:00Z", "v": 12.88, "lo": 8.05, "hi": 17.76}, {"t": "2026-11-14T13:00:00Z", "v": 12.02, "lo": 7.2, "hi": 16.9}, {"t": "2026-11-14T14:00:00Z", "v": 17.24, "lo": 12.42, "hi": 22.12}, {"t": "2026-11-14T15:00:00Z", "v": 15.8, "lo": 10.97, "hi": 20.67}, {"t": "2026-11-14T16:00:00Z", "v": 15.97, "lo": 11.15, "hi": 20.85}, {"t": "2026-11-14T17:00:00Z", "v": 15.77, "lo": 10.94, "hi": 20.65}, {"t": "2026-11-14T18:00:00Z", "v": 16.1, "lo": 11.27, "hi": 20.97}, {"t": "2026-11-14T19:00:00Z", "v": 17.49, "lo": 12.67, "hi": 22.37}, {"t": "2026-11-14T20:00:00Z", "v": 12.64, "lo": 7.81, "hi": 17.51}, {"t": "2026-11-14T21:00:00Z", "v": 14.52, "lo": 9.7, "hi": 19.4}, {"t": "2026-11-14T22:00:00Z", "v": 12.33, "lo": 7.5, "hi": 17.21}, {"t": "2026-11-14T23:00:00Z", "v": 9.87, "lo": 5.04, "hi": 14.74}, {"t": "2026-11-15T00:00:00Z", "v": 3.94, "lo": 0.0, "hi": 8.82}, {"t": "2026-11-15T01:00:00Z", "v": 1.34, "lo": 0.0, "hi": 6.22}, {"t": "2026-11-15T02:00:00Z", "v": 1.13, "lo": 0.0, "hi": 6.01}, {"t": "2026-11-15T03:00:00Z", "v": 0.99, "lo": 0.0, "hi": 5.86}, {"Line truncated
|
{"data_version": "2026-11-07T23:51:00Z", "model_version": "histgb-2026-11-14", "context_version": "2026-11-14T03:47:09Z", "generated_at": "2026-11-14T05:27:22Z", "forecast_start": "2026-11-14T06:00:00Z", "forecasts": {"density-01": {"48h": {"base": [{"t": "2026-11-14T06:00:00Z", "v": 1.95, "lo": 0.0, "hi": 6.83}, {"t": "2026-11-14T07:00:00Z", "v": 2.94, "lo": 0.0, "hi": 7.81}, {"t": "2026-11-14T08:00:00Z", "v": 4.08, "lo": 0.0, "hi": 8.96}, {"t": "2026-11-14T09:00:00Z", "v": 5.68, "lo": 0.85, "hi": 10.55}, {"t": "2026-11-14T10:00:00Z", "v": 8.5, "lo": 3.67, "hi": 13.38}, {"t": "2026-11-14T11:00:00Z", "v": 9.66, "lo": 4.84, "hi": 14.54}, {"t": "2026-11-14T12:00:00Z", "v": 11.39, "lo": 6.56, "hi": 16.26}, {"t": "2026-11-14T13:00:00Z", "v": 11.32, "lo": 6.5, "hi": 16.2}, {"t": "2026-11-14T14:00:00Z", "v": 12.63, "lo": 7.81, "hi": 17.51}, {"t": "2026-11-14T15:00:00Z", "v": 14.78, "lo": 9.96, "hi": 19.66}, {"t": "2026-11-14T16:00:00Z", "v": 14.21, "lo": 9.39, "hi": 19.09}, {"t": "2026-11-14T17:00:00Z", "v": 14.52, "lo": 9.7, "hi": 19.4}, {"t": "2026-11-14T18:00:00Z", "v": 15.67, "lo": 10.84, "hi": 20.55}, {"t": "2026-11-14T19:00:00Z", "v": 14.37, "lo": 9.55, "hi": 19.25}, {"t": "2026-11-14T20:00:00Z", "v": 12.64, "lo": 7.81, "hi": 17.51}, {"t": "2026-11-14T21:00:00Z", "v": 14.52, "lo": 9.7, "hi": 19.4}, {"t": "2026-11-14T22:00:00Z", "v": 12.33, "lo": 7.5, "hi": 17.21}, {"t": "2026-11-14T23:00:00Z", "v": 9.87, "lo": 5.04, "hi": 14.74}, {"t": "2026-11-15T00:00:00Z", "v": 3.94, "lo": 0.0, "hi": 8.82}, {"t": "2026-11-15T01:00:00Z", "v": 1.34, "lo": 0.0, "hi": 6.22}, {"t": "2026-11-15T02:00:00Z", "v": 1.13, "lo": 0.0, "hi": 6.01}, {"t": "2026-11-15T03:00:00Z", "v": 0.99, "lo": 0.0, "hi": 5.86}, {"t": "2026-11-15T04:00:00Z", "v": 0.99, "lo": 0.0, "hi": 5.86}, {"t": "2026-11-15T05:00:00Z", "v": 1.01, "lo": 0.0, "hi": 5.88}, {"t": "2026-11-15T06:00:00Z", "v": 1.04, "lo": 0.0, "hi": 5.92}, {"t": "2026-11-15T07:00:00Z", "v": 1.55, "lo": 0.0, "hi": 6.43}, {"t": "2026-11-15T08:00:00Z", "v": 1.65, "lo": 0.0, "hi": 6.53}, {"t": "2026-11-15T09:00:00Z", "v": 5.61, "lo": 0.78, "hi": 10.48}, {"t": "2026-11-15T10:00:00Z", "v": 6.5, "lo": 1.68, "hi": 11.38}, {"t": "2026-11-15T11:00:00Z", "v": 7.51, "lo": 2.69, "hi": 12.39}, {"t": "2026-11-15T12:00:00Z", "v": 9.58, "lo": 4.76, "hi": 14.46}, {"t": "2026-11-15T13:00:00Z", "v": 11.7, "lo": 6.88, "hi": 16.58}, {"t": "2026-11-15T14:00:00Z", "v": 18.36, "lo": 13.54, "hi": 23.24}, {"t": "2026-11-15T15:00:00Z", "v": 17.33, "lo": 12.5, "hi": 22.21}, {"t": "2026-11-15T16:00:00Z", "v": 16.04, "lo": 11.21, "hi": 20.91}, {"t": "2026-11-15T17:00:00Z", "v": 11.34, "lo": 6.51, "hi": 16.21}, {"t": "2026-11-15T18:00:00Z", "v": 10.87, "lo": 6.05, "hi": 15.75}, {"t": "2026-11-15T19:00:00Z", "v": 8.45, "lo": 3.63, "hi": 13.33}, {"t": "2026-11-15T20:00:00Z", "v": 5.82, "lo": 0.99, "hi": 10.7}, {"t": "2026-11-15T21:00:00Z", "v": 3.71, "lo": 0.0, "hi": 8.59}, {"t": "2026-11-15T22:00:00Z", "v": 2.11, "lo": 0.0, "hi": 6.99}, {"t": "2026-11-15T23:00:00Z", "v": 1.47, "lo": 0.0, "hi": 6.34}, {"t": "2026-11-16T00:00:00Z", "v": 0.91, "lo": 0.0, "hi": 5.79}, {"t": "2026-11-16T01:00:00Z", "v": 0.88, "lo": 0.0, "hi": 5.76}, {"t": "2026-11-16T02:00:00Z", "v": 1.15, "lo": 0.0, "hi": 6.03}, {"t": "2026-11-16T03:00:00Z", "v": 1.15, "lo": 0.0, "hi": 6.03}, {"t": "2026-11-16T04:00:00Z", "v": 1.74, "lo": 0.0, "hi": 6.62}, {"t": "2026-11-16T05:00:00Z", "v": 1.58, "lo": 0.0, "hi": 6.46}], "sunny-saturday-market": [{"t": "2026-11-14T06:00:00Z", "v": 1.95, "lo": 0.0, "hi": 6.83}, {"t": "2026-11-14T07:00:00Z", "v": 2.94, "lo": 0.0, "hi": 7.81}, {"t": "2026-11-14T08:00:00Z", "v": 4.08, "lo": 0.0, "hi": 8.96}, {"t": "2026-11-14T09:00:00Z", "v": 7.63, "lo": 2.81, "hi": 12.51}, {"t": "2026-11-14T10:00:00Z", "v": 10.14, "lo": 5.32, "hi": 15.02}, {"t": "2026-11-14T11:00:00Z", "v": 11.16, "lo": 6.34, "hi": 16.04}, {"t": "2026-11-14T12:00:00Z", "v": 12.88, "lo": 8.05, "hi": 17.76}, {"t": "2026-11-14T13:00:00Z", "v": 12.02, "lo": 7.2, "hi": 16.9}, {"t": "2026-11-14T14:00:00Z", "v": 17.24, "lo": 12.42, "hi": 22.12}, {"t": "2026-11-14T15:00:00Z", "v": 15.8, "lo": 10.97, "hi": 20.67}, {"t": "2026-11-14T16:00:00Z", "v": 15.97, "lo": 11.15, "hi": 20.85}, {"t": "2026-11-14T17:00:00Z", "v": 15.77, "lo": 10.94, "hi": 20.65}, {"t": "2026-11-14T18:00:00Z", "v": 16.1, "lo": 11.27, "hi": 20.97}, {"t": "2026-11-14T19:00:00Z", "v": 17.49, "lo": 12.67, "hi": 22.37}, {"t": "2026-11-14T20:00:00Z", "v": 12.64, "lo": 7.81, "hi": 17.51}, {"t": "2026-11-14T21:00:00Z", "v": 14.52, "lo": 9.7, "hi": 19.4}, {"t": "2026-11-14T22:00:00Z", "v": 12.33, "lo": 7.5, "hi": 17.21}, {"t": "2026-11-14T23:00:00Z", "v": 9.87, "lo": 5.04, "hi": 14.74}, {"t": "2026-11-15T00:00:00Z", "v": 3.94, "lo": 0.0, "hi": 8.82}, {"t": "2026-11-15T01:00:00Z", "v": 1.34, "lo": 0.0, "hi": 6.22}, {"t": "2026-11-15T02:00:00Z", "v": 1.13, "lo": 0.0, "hi": 6.01}, {"t": "2026-11-15T03:00:00Z", "v": 0.99, "lo": 0.0, "hi": 5.86}, {"t": "2026-11-15T04:00:00Z", "v": 0.99, "lo": 0.0, "hi": 5.86}, {"Line truncated
|
||||||
+207
-2
@@ -19,7 +19,7 @@
|
|||||||
"baseline_r2": 0.462,
|
"baseline_r2": 0.462,
|
||||||
"n_eval": 5350
|
"n_eval": 5350
|
||||||
},
|
},
|
||||||
"trained_at": "2026-11-14T04:03:37Z",
|
"trained_at": "2026-11-14T05:27:21Z",
|
||||||
"train_from": "2026-06-12T00:00:00Z",
|
"train_from": "2026-06-12T00:00:00Z",
|
||||||
"train_to": "2026-11-07T23:00:00Z",
|
"train_to": "2026-11-07T23:00:00Z",
|
||||||
"test_from": "2026-10-10T23:00:00Z",
|
"test_from": "2026-10-10T23:00:00Z",
|
||||||
@@ -90,11 +90,216 @@
|
|||||||
"n": 5350
|
"n": 5350
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
|
"per_sensor": [
|
||||||
|
{
|
||||||
|
"sensor": "density-01",
|
||||||
|
"name": "Density 01",
|
||||||
|
"n": 669,
|
||||||
|
"mae": 2.26,
|
||||||
|
"wape": 30.8,
|
||||||
|
"r2": 0.722,
|
||||||
|
"baseline_mae": 3.58,
|
||||||
|
"baseline_wape": 48.8,
|
||||||
|
"baseline_r2": 0.238,
|
||||||
|
"mae_improvement_pct": 36.8
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"sensor": "hn-goetzentuem-01",
|
||||||
|
"name": "Götzenturm",
|
||||||
|
"n": 669,
|
||||||
|
"mae": 2.4,
|
||||||
|
"wape": 26.9,
|
||||||
|
"r2": 0.718,
|
||||||
|
"baseline_mae": 3.84,
|
||||||
|
"baseline_wape": 43.0,
|
||||||
|
"baseline_r2": 0.281,
|
||||||
|
"mae_improvement_pct": 37.5
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"sensor": "hn-hagenbuchersee-01",
|
||||||
|
"name": "Hagenbuchersee 1",
|
||||||
|
"n": 669,
|
||||||
|
"mae": 2.47,
|
||||||
|
"wape": 38.6,
|
||||||
|
"r2": 0.602,
|
||||||
|
"baseline_mae": 4.01,
|
||||||
|
"baseline_wape": 62.8,
|
||||||
|
"baseline_r2": -0.212,
|
||||||
|
"mae_improvement_pct": 38.5
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"sensor": "hn-hagenbuchersee-02",
|
||||||
|
"name": "Hagenbuchersee 2",
|
||||||
|
"n": 669,
|
||||||
|
"mae": 4.12,
|
||||||
|
"wape": 31.1,
|
||||||
|
"r2": 0.731,
|
||||||
|
"baseline_mae": 6.49,
|
||||||
|
"baseline_wape": 48.9,
|
||||||
|
"baseline_r2": 0.337,
|
||||||
|
"mae_improvement_pct": 36.5
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"sensor": "hn-marahaus-01",
|
||||||
|
"name": "Marahaus",
|
||||||
|
"n": 669,
|
||||||
|
"mae": 2.67,
|
||||||
|
"wape": 29.8,
|
||||||
|
"r2": 0.671,
|
||||||
|
"baseline_mae": 4.57,
|
||||||
|
"baseline_wape": 51.0,
|
||||||
|
"baseline_r2": -0.045,
|
||||||
|
"mae_improvement_pct": 41.7
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"sensor": "hn-neckarufer-01",
|
||||||
|
"name": "Neckarufer",
|
||||||
|
"n": 668,
|
||||||
|
"mae": 3.63,
|
||||||
|
"wape": 28.2,
|
||||||
|
"r2": 0.695,
|
||||||
|
"baseline_mae": 5.89,
|
||||||
|
"baseline_wape": 45.8,
|
||||||
|
"baseline_r2": 0.11,
|
||||||
|
"mae_improvement_pct": 38.4
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"sensor": "hn-soleo-01",
|
||||||
|
"name": "Soleo",
|
||||||
|
"n": 669,
|
||||||
|
"mae": 3.19,
|
||||||
|
"wape": 20.2,
|
||||||
|
"r2": 0.548,
|
||||||
|
"baseline_mae": 4.4,
|
||||||
|
"baseline_wape": 28.0,
|
||||||
|
"baseline_r2": 0.163,
|
||||||
|
"mae_improvement_pct": 27.7
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"sensor": "hn-stadtgalerie-01",
|
||||||
|
"name": "Stadtgalerie",
|
||||||
|
"n": 668,
|
||||||
|
"mae": 5.83,
|
||||||
|
"wape": 25.3,
|
||||||
|
"r2": 0.765,
|
||||||
|
"baseline_mae": 10.39,
|
||||||
|
"baseline_wape": 45.1,
|
||||||
|
"baseline_r2": 0.48,
|
||||||
|
"mae_improvement_pct": 43.9
|
||||||
|
}
|
||||||
|
],
|
||||||
"limits": [
|
"limits": [
|
||||||
"estimated devices, not people",
|
"estimated devices, not people",
|
||||||
"8 sensors in a ~580 m inner-city cluster",
|
"8 sensors in a ~580 m inner-city cluster",
|
||||||
"5 months of multi-sensor data; no annual seasonality claim",
|
"5 months of multi-sensor data; no annual seasonality claim",
|
||||||
"observed weather in training, forecast weather in scenarios",
|
"observed weather in training, forecast weather in scenarios",
|
||||||
"holidays are context, not a large accuracy claim"
|
"holidays are context, not a large accuracy claim"
|
||||||
]
|
],
|
||||||
|
"small": {
|
||||||
|
"model_version": "logit-20261114-052723",
|
||||||
|
"trained_at": "2026-11-14T05:27:23Z",
|
||||||
|
"features": [
|
||||||
|
"hour",
|
||||||
|
"weekday",
|
||||||
|
"is_weekend",
|
||||||
|
"temp_c",
|
||||||
|
"precip_mm",
|
||||||
|
"is_holiday"
|
||||||
|
],
|
||||||
|
"metrics": {
|
||||||
|
"accuracy": 0.708,
|
||||||
|
"precision": 0.36,
|
||||||
|
"recall": 0.574,
|
||||||
|
"f1": 0.443,
|
||||||
|
"roc_auc": 0.726,
|
||||||
|
"n_train": 18242,
|
||||||
|
"n_test": 5710,
|
||||||
|
"positive_rate": 0.202,
|
||||||
|
"sensors": 8
|
||||||
|
},
|
||||||
|
"confusion_matrix": {
|
||||||
|
"labels": {
|
||||||
|
"0": "not busy",
|
||||||
|
"1": "busy"
|
||||||
|
},
|
||||||
|
"tn": 3377,
|
||||||
|
"fp": 1178,
|
||||||
|
"fn": 492,
|
||||||
|
"tp": 663
|
||||||
|
},
|
||||||
|
"per_sensor": [
|
||||||
|
{
|
||||||
|
"sensor": "density-01",
|
||||||
|
"n": 714,
|
||||||
|
"positive_rate": 0.111,
|
||||||
|
"accuracy": 0.761,
|
||||||
|
"precision": 0.3,
|
||||||
|
"recall": 0.873,
|
||||||
|
"f1": 0.447
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"sensor": "hn-goetzentuem-01",
|
||||||
|
"n": 714,
|
||||||
|
"positive_rate": 0.197,
|
||||||
|
"accuracy": 0.732,
|
||||||
|
"precision": 0.391,
|
||||||
|
"recall": 0.638,
|
||||||
|
"f1": 0.485
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"sensor": "hn-hagenbuchersee-01",
|
||||||
|
"n": 714,
|
||||||
|
"positive_rate": 0.049,
|
||||||
|
"accuracy": 0.724,
|
||||||
|
"precision": 0.148,
|
||||||
|
"recall": 0.971,
|
||||||
|
"f1": 0.257
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"sensor": "hn-hagenbuchersee-02",
|
||||||
|
"n": 714,
|
||||||
|
"positive_rate": 0.123,
|
||||||
|
"accuracy": 0.783,
|
||||||
|
"precision": 0.355,
|
||||||
|
"recall": 0.932,
|
||||||
|
"f1": 0.514
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"sensor": "hn-marahaus-01",
|
||||||
|
"n": 714,
|
||||||
|
"positive_rate": 0.055,
|
||||||
|
"accuracy": 0.71,
|
||||||
|
"precision": 0.135,
|
||||||
|
"recall": 0.795,
|
||||||
|
"f1": 0.23
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"sensor": "hn-neckarufer-01",
|
||||||
|
"n": 713,
|
||||||
|
"positive_rate": 0.09,
|
||||||
|
"accuracy": 0.764,
|
||||||
|
"precision": 0.274,
|
||||||
|
"recall": 0.984,
|
||||||
|
"f1": 0.429
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"sensor": "hn-soleo-01",
|
||||||
|
"n": 714,
|
||||||
|
"positive_rate": 0.704,
|
||||||
|
"accuracy": 0.581,
|
||||||
|
"precision": 0.943,
|
||||||
|
"recall": 0.431,
|
||||||
|
"f1": 0.592
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"sensor": "hn-stadtgalerie-01",
|
||||||
|
"n": 713,
|
||||||
|
"positive_rate": 0.289,
|
||||||
|
"accuracy": 0.604,
|
||||||
|
"precision": 0.335,
|
||||||
|
"recall": 0.374,
|
||||||
|
"f1": 0.353
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
}
|
}
|
||||||
Binary file not shown.
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"model_version": "logit-20261114-040339",
|
"model_version": "logit-20261114-052723",
|
||||||
"trained_at": "2026-11-14T04:03:39Z",
|
"trained_at": "2026-11-14T05:27:23Z",
|
||||||
"features": [
|
"features": [
|
||||||
"hour",
|
"hour",
|
||||||
"weekday",
|
"weekday",
|
||||||
@@ -33,5 +33,89 @@
|
|||||||
"n_test": 5710,
|
"n_test": 5710,
|
||||||
"positive_rate": 0.202,
|
"positive_rate": 0.202,
|
||||||
"sensors": 8
|
"sensors": 8
|
||||||
}
|
},
|
||||||
|
"confusion_matrix": {
|
||||||
|
"labels": {
|
||||||
|
"0": "not busy",
|
||||||
|
"1": "busy"
|
||||||
|
},
|
||||||
|
"tn": 3377,
|
||||||
|
"fp": 1178,
|
||||||
|
"fn": 492,
|
||||||
|
"tp": 663
|
||||||
|
},
|
||||||
|
"per_sensor": [
|
||||||
|
{
|
||||||
|
"sensor": "density-01",
|
||||||
|
"n": 714,
|
||||||
|
"positive_rate": 0.111,
|
||||||
|
"accuracy": 0.761,
|
||||||
|
"precision": 0.3,
|
||||||
|
"recall": 0.873,
|
||||||
|
"f1": 0.447
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"sensor": "hn-goetzentuem-01",
|
||||||
|
"n": 714,
|
||||||
|
"positive_rate": 0.197,
|
||||||
|
"accuracy": 0.732,
|
||||||
|
"precision": 0.391,
|
||||||
|
"recall": 0.638,
|
||||||
|
"f1": 0.485
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"sensor": "hn-hagenbuchersee-01",
|
||||||
|
"n": 714,
|
||||||
|
"positive_rate": 0.049,
|
||||||
|
"accuracy": 0.724,
|
||||||
|
"precision": 0.148,
|
||||||
|
"recall": 0.971,
|
||||||
|
"f1": 0.257
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"sensor": "hn-hagenbuchersee-02",
|
||||||
|
"n": 714,
|
||||||
|
"positive_rate": 0.123,
|
||||||
|
"accuracy": 0.783,
|
||||||
|
"precision": 0.355,
|
||||||
|
"recall": 0.932,
|
||||||
|
"f1": 0.514
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"sensor": "hn-marahaus-01",
|
||||||
|
"n": 714,
|
||||||
|
"positive_rate": 0.055,
|
||||||
|
"accuracy": 0.71,
|
||||||
|
"precision": 0.135,
|
||||||
|
"recall": 0.795,
|
||||||
|
"f1": 0.23
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"sensor": "hn-neckarufer-01",
|
||||||
|
"n": 713,
|
||||||
|
"positive_rate": 0.09,
|
||||||
|
"accuracy": 0.764,
|
||||||
|
"precision": 0.274,
|
||||||
|
"recall": 0.984,
|
||||||
|
"f1": 0.429
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"sensor": "hn-soleo-01",
|
||||||
|
"n": 714,
|
||||||
|
"positive_rate": 0.704,
|
||||||
|
"accuracy": 0.581,
|
||||||
|
"precision": 0.943,
|
||||||
|
"recall": 0.431,
|
||||||
|
"f1": 0.592
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"sensor": "hn-stadtgalerie-01",
|
||||||
|
"n": 713,
|
||||||
|
"positive_rate": 0.289,
|
||||||
|
"accuracy": 0.604,
|
||||||
|
"precision": 0.335,
|
||||||
|
"recall": 0.374,
|
||||||
|
"f1": 0.353
|
||||||
|
}
|
||||||
|
]
|
||||||
}
|
}
|
||||||
Binary file not shown.
@@ -0,0 +1,36 @@
|
|||||||
|
<svg xmlns="http://www.w3.org/2000/svg" width="960" height="370" viewBox="0 0 960 370" role="img" aria-label="Ablation: holdout MAE by feature set">
|
||||||
|
<rect width="100%" height="100%" fill="#ffffff"/>
|
||||||
|
<g font-family="-apple-system, Segoe UI, Helvetica, Arial, sans-serif">
|
||||||
|
<text x="24" y="34" font-size="20" font-weight="600" fill="#0f172a">Ablation: holdout MAE by feature set</text>
|
||||||
|
<text x="24" y="56" font-size="13" fill="#475569">Holdout 2026-10-10T23:00:00Z .. 2026-11-07T23:00:00Z, n=5350</text>
|
||||||
|
<line x1="230.0" y1="68" x2="230.0" y2="314" stroke="#e2e8f0" stroke-width="1"/>
|
||||||
|
<text x="230.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">0.0</text>
|
||||||
|
<line x1="388.5" y1="68" x2="388.5" y2="314" stroke="#e2e8f0" stroke-width="1"/>
|
||||||
|
<text x="388.5" y="62" font-size="11" fill="#64748b" text-anchor="middle">1.5</text>
|
||||||
|
<line x1="547.0" y1="68" x2="547.0" y2="314" stroke="#e2e8f0" stroke-width="1"/>
|
||||||
|
<text x="547.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">2.9</text>
|
||||||
|
<line x1="705.5" y1="68" x2="705.5" y2="314" stroke="#e2e8f0" stroke-width="1"/>
|
||||||
|
<text x="705.5" y="62" font-size="11" fill="#64748b" text-anchor="middle">4.4</text>
|
||||||
|
<line x1="864.0" y1="68" x2="864.0" y2="314" stroke="#e2e8f0" stroke-width="1"/>
|
||||||
|
<text x="864.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">5.8</text>
|
||||||
|
<text x="218" y="106" font-size="12" fill="#0f172a" text-anchor="end">seasonal naive</text>
|
||||||
|
<rect x="230" y="92" width="587.0" height="22" fill="#2563eb"/>
|
||||||
|
<text x="823.0" y="107" font-size="12" fill="#0f172a">5.40</text>
|
||||||
|
<text x="218" y="144" font-size="12" fill="#0f172a" text-anchor="end">day no weather</text>
|
||||||
|
<rect x="230" y="130" width="430.5" height="22" fill="#2563eb"/>
|
||||||
|
<text x="666.5" y="145" font-size="12" fill="#0f172a">3.96</text>
|
||||||
|
<text x="218" y="182" font-size="12" fill="#0f172a" text-anchor="end">day holiday</text>
|
||||||
|
<rect x="230" y="168" width="417.4" height="22" fill="#2563eb"/>
|
||||||
|
<text x="653.4" y="183" font-size="12" fill="#0f172a">3.84</text>
|
||||||
|
<text x="218" y="220" font-size="12" fill="#0f172a" text-anchor="end">day weather</text>
|
||||||
|
<rect x="230" y="206" width="365.3" height="22" fill="#2563eb"/>
|
||||||
|
<text x="601.3" y="221" font-size="12" fill="#0f172a">3.36</text>
|
||||||
|
<text x="218" y="258" font-size="12" fill="#0f172a" text-anchor="end">week no weather</text>
|
||||||
|
<rect x="230" y="244" width="443.5" height="22" fill="#2563eb"/>
|
||||||
|
<text x="679.5" y="259" font-size="12" fill="#0f172a">4.08</text>
|
||||||
|
<text x="218" y="296" font-size="12" fill="#0f172a" text-anchor="end">week weather</text>
|
||||||
|
<rect x="230" y="282" width="360.9" height="22" fill="#0f766e"/>
|
||||||
|
<text x="596.9" y="297" font-size="12" fill="#0f172a">3.32</text>
|
||||||
|
<text x="24" y="358" font-size="11" fill="#64748b">Source: artifacts/metrics.json · PulseCast 28-day time-based holdout</text>
|
||||||
|
</g>
|
||||||
|
</svg>
|
||||||
|
After Width: | Height: | Size: 2.7 KiB |
@@ -0,0 +1,60 @@
|
|||||||
|
<svg xmlns="http://www.w3.org/2000/svg" width="960" height="446" viewBox="0 0 960 446" role="img" aria-label="Per-sensor holdout MAE: production model vs seasonal naive">
|
||||||
|
<rect width="100%" height="100%" fill="#ffffff"/>
|
||||||
|
<g font-family="-apple-system, Segoe UI, Helvetica, Arial, sans-serif">
|
||||||
|
<text x="24" y="34" font-size="20" font-weight="600" fill="#0f172a">Per-sensor holdout MAE: production model vs seasonal naive</text>
|
||||||
|
<text x="24" y="56" font-size="13" fill="#475569">Holdout 2026-10-10T23:00:00Z .. 2026-11-07T23:00:00Z, n=5350</text>
|
||||||
|
<line x1="230.0" y1="68" x2="230.0" y2="390" stroke="#e2e8f0" stroke-width="1"/>
|
||||||
|
<text x="230.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">0.0</text>
|
||||||
|
<line x1="388.5" y1="68" x2="388.5" y2="390" stroke="#e2e8f0" stroke-width="1"/>
|
||||||
|
<text x="388.5" y="62" font-size="11" fill="#64748b" text-anchor="middle">2.8</text>
|
||||||
|
<line x1="547.0" y1="68" x2="547.0" y2="390" stroke="#e2e8f0" stroke-width="1"/>
|
||||||
|
<text x="547.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">5.6</text>
|
||||||
|
<line x1="705.5" y1="68" x2="705.5" y2="390" stroke="#e2e8f0" stroke-width="1"/>
|
||||||
|
<text x="705.5" y="62" font-size="11" fill="#64748b" text-anchor="middle">8.4</text>
|
||||||
|
<line x1="864.0" y1="68" x2="864.0" y2="390" stroke="#e2e8f0" stroke-width="1"/>
|
||||||
|
<text x="864.0" y="62" font-size="11" fill="#64748b" text-anchor="middle">11.2</text>
|
||||||
|
<rect x="230" y="52" width="10" height="10" fill="#94a3b8"/><text x="246" y="61" font-size="12" fill="#475569">seasonal naive</text>
|
||||||
|
<rect x="370" y="52" width="10" height="10" fill="#2563eb"/><text x="386" y="61" font-size="12" fill="#475569">weather-7d</text>
|
||||||
|
<text x="218" y="106" font-size="12" fill="#0f172a" text-anchor="end">Density 01</text>
|
||||||
|
<rect x="230" y="89" width="202.3" height="12" fill="#94a3b8"/>
|
||||||
|
<text x="438.3" y="99" font-size="11" fill="#64748b">3.58</text>
|
||||||
|
<rect x="230" y="104" width="127.7" height="12" fill="#2563eb"/>
|
||||||
|
<text x="363.7" y="114" font-size="11" fill="#0f172a">2.26</text>
|
||||||
|
<text x="218" y="144" font-size="12" fill="#0f172a" text-anchor="end">Götzenturm</text>
|
||||||
|
<rect x="230" y="127" width="217.0" height="12" fill="#94a3b8"/>
|
||||||
|
<text x="453.0" y="137" font-size="11" fill="#64748b">3.84</text>
|
||||||
|
<rect x="230" y="142" width="135.6" height="12" fill="#2563eb"/>
|
||||||
|
<text x="371.6" y="152" font-size="11" fill="#0f172a">2.40</text>
|
||||||
|
<text x="218" y="182" font-size="12" fill="#0f172a" text-anchor="end">Hagenbuchersee 1</text>
|
||||||
|
<rect x="230" y="165" width="226.6" height="12" fill="#94a3b8"/>
|
||||||
|
<text x="462.6" y="175" font-size="11" fill="#64748b">4.01</text>
|
||||||
|
<rect x="230" y="180" width="139.6" height="12" fill="#2563eb"/>
|
||||||
|
<text x="375.6" y="190" font-size="11" fill="#0f172a">2.47</text>
|
||||||
|
<text x="218" y="220" font-size="12" fill="#0f172a" text-anchor="end">Hagenbuchersee 2</text>
|
||||||
|
<rect x="230" y="203" width="366.7" height="12" fill="#94a3b8"/>
|
||||||
|
<text x="602.7" y="213" font-size="11" fill="#64748b">6.49</text>
|
||||||
|
<rect x="230" y="218" width="232.8" height="12" fill="#2563eb"/>
|
||||||
|
<text x="468.8" y="228" font-size="11" fill="#0f172a">4.12</text>
|
||||||
|
<text x="218" y="258" font-size="12" fill="#0f172a" text-anchor="end">Marahaus</text>
|
||||||
|
<rect x="230" y="241" width="258.2" height="12" fill="#94a3b8"/>
|
||||||
|
<text x="494.2" y="251" font-size="11" fill="#64748b">4.57</text>
|
||||||
|
<rect x="230" y="256" width="150.9" height="12" fill="#2563eb"/>
|
||||||
|
<text x="386.9" y="266" font-size="11" fill="#0f172a">2.67</text>
|
||||||
|
<text x="218" y="296" font-size="12" fill="#0f172a" text-anchor="end">Neckarufer</text>
|
||||||
|
<rect x="230" y="279" width="332.8" height="12" fill="#94a3b8"/>
|
||||||
|
<text x="568.8" y="289" font-size="11" fill="#64748b">5.89</text>
|
||||||
|
<rect x="230" y="294" width="205.1" height="12" fill="#2563eb"/>
|
||||||
|
<text x="441.1" y="304" font-size="11" fill="#0f172a">3.63</text>
|
||||||
|
<text x="218" y="334" font-size="12" fill="#0f172a" text-anchor="end">Soleo</text>
|
||||||
|
<rect x="230" y="317" width="248.6" height="12" fill="#94a3b8"/>
|
||||||
|
<text x="484.6" y="327" font-size="11" fill="#64748b">4.40</text>
|
||||||
|
<rect x="230" y="332" width="180.2" height="12" fill="#2563eb"/>
|
||||||
|
<text x="416.2" y="342" font-size="11" fill="#0f172a">3.19</text>
|
||||||
|
<text x="218" y="372" font-size="12" fill="#0f172a" text-anchor="end">Stadtgalerie</text>
|
||||||
|
<rect x="230" y="355" width="587.0" height="12" fill="#94a3b8"/>
|
||||||
|
<text x="823.0" y="365" font-size="11" fill="#64748b">10.39</text>
|
||||||
|
<rect x="230" y="370" width="329.4" height="12" fill="#2563eb"/>
|
||||||
|
<text x="565.4" y="380" font-size="11" fill="#0f172a">5.83</text>
|
||||||
|
<text x="24" y="434" font-size="11" fill="#64748b">Source: artifacts/metrics.json · PulseCast 28-day time-based holdout</text>
|
||||||
|
</g>
|
||||||
|
</svg>
|
||||||
|
After Width: | Height: | Size: 4.5 KiB |
@@ -0,0 +1,24 @@
|
|||||||
|
<svg xmlns="http://www.w3.org/2000/svg" width="760" height="520" viewBox="0 0 760 520" role="img" aria-label="Small busy-hour classifier: holdout confusion matrix">
|
||||||
|
<rect width="100%" height="100%" fill="#ffffff"/>
|
||||||
|
<g font-family="-apple-system, Segoe UI, Helvetica, Arial, sans-serif">
|
||||||
|
<text x="24" y="34" font-size="20" font-weight="600" fill="#0f172a">Small busy-hour classifier: holdout confusion matrix</text>
|
||||||
|
<text x="24" y="56" font-size="13" fill="#475569">n=5,710 · accuracy 0.708 · precision 0.360 · recall 0.574 · F1 0.443 · ROC-AUC 0.726</text>
|
||||||
|
<text x="295.0" y="114" font-size="13" fill="#475569" text-anchor="middle">Predicted not busy</text>
|
||||||
|
<text x="485.0" y="114" font-size="13" fill="#475569" text-anchor="middle">Predicted busy</text>
|
||||||
|
<text x="182" y="225.0" font-size="13" fill="#475569" text-anchor="end">Actually not busy</text>
|
||||||
|
<text x="182" y="415.0" font-size="13" fill="#475569" text-anchor="end">Actually busy</text>
|
||||||
|
<rect x="200" y="130" width="190" height="190" fill="#1d4ed8" stroke="#ffffff" stroke-width="2"/>
|
||||||
|
<text x="295.0" y="217.0" font-size="30" font-weight="600" fill="#ffffff" text-anchor="middle">3,377</text>
|
||||||
|
<text x="295.0" y="245.0" font-size="13" fill="#ffffff" text-anchor="middle">TN · 74.1% of row</text>
|
||||||
|
<rect x="390" y="130" width="190" height="190" fill="#8aa7ee" stroke="#ffffff" stroke-width="2"/>
|
||||||
|
<text x="485.0" y="217.0" font-size="30" font-weight="600" fill="#0f172a" text-anchor="middle">1,178</text>
|
||||||
|
<text x="485.0" y="245.0" font-size="13" fill="#0f172a" text-anchor="middle">FP · 25.9% of row</text>
|
||||||
|
<rect x="200" y="320" width="190" height="190" fill="#acc3f5" stroke="#ffffff" stroke-width="2"/>
|
||||||
|
<text x="295.0" y="407.0" font-size="30" font-weight="600" fill="#0f172a" text-anchor="middle">492</text>
|
||||||
|
<text x="295.0" y="435.0" font-size="13" fill="#0f172a" text-anchor="middle">FN · 42.6% of row</text>
|
||||||
|
<rect x="390" y="320" width="190" height="190" fill="#a3bcf3" stroke="#ffffff" stroke-width="2"/>
|
||||||
|
<text x="485.0" y="407.0" font-size="30" font-weight="600" fill="#0f172a" text-anchor="middle">663</text>
|
||||||
|
<text x="485.0" y="435.0" font-size="13" fill="#0f172a" text-anchor="middle">TP · 57.4% of row</text>
|
||||||
|
<text x="24" y="508" font-size="11" fill="#64748b">Source: artifacts/small_metrics.json · chronological 80/20 split</text>
|
||||||
|
</g>
|
||||||
|
</svg>
|
||||||
|
After Width: | Height: | Size: 2.3 KiB |
+188
@@ -0,0 +1,188 @@
|
|||||||
|
# Metrics — paper support
|
||||||
|
|
||||||
|
Every value in this file comes from the committed artifacts:
|
||||||
|
`artifacts/metrics.json` (forecast + ablation + per-sensor + small model),
|
||||||
|
`artifacts/small_metrics.json` (small-model detail), and
|
||||||
|
`artifacts/data_audit.json` (raw-input counts). To refresh a table, rerun
|
||||||
|
`make train && make figures` and quote the new JSON.
|
||||||
|
|
||||||
|
## Run provenance
|
||||||
|
|
||||||
|
| Field | Value |
|
||||||
|
|---|---|
|
||||||
|
| Model version | `histgb-2026-11-14` |
|
||||||
|
| Trained at | 2026-11-14T05:27:21Z |
|
||||||
|
| Data version | 2026-11-07T23:51:00Z |
|
||||||
|
| Context (weather) version | 2026-11-14T03:47:09Z |
|
||||||
|
| Train rows | 18,602 hourly sensor rows |
|
||||||
|
| Holdout | 2026-10-10T23:00:00Z → 2026-11-07T23:00:00Z |
|
||||||
|
| Evaluation rows | 5,350 (8 sensors, hours with a seasonal-naive baseline) |
|
||||||
|
| Sensors | 8 (one merged `density-01`, seven `hn-*`) |
|
||||||
|
| Raw input | 192,464 rows, 96,232 density rows, 96,232 battery rows |
|
||||||
|
| Weather | Open-Meteo historical archive (training), forecast (scenarios) |
|
||||||
|
| Calendar | Baden-Württemberg holidays 2025–2026 |
|
||||||
|
|
||||||
|
## Protocol
|
||||||
|
|
||||||
|
- One global `HistGradientBoostingRegressor` over all eight sensors. Target is
|
||||||
|
the mean of 15-minute `densityData` readings in a UTC hour.
|
||||||
|
- Time-based 28-day holdout measured back from the last observed hour. No
|
||||||
|
shuffling, no random split.
|
||||||
|
- Seasonal naive baseline: the exact 168-row lag (same observed hour one week
|
||||||
|
earlier), evaluated on the same rows.
|
||||||
|
- The production bundle is refit on all observed rows **after** the holdout is
|
||||||
|
evaluated; the holdout is never in the production fit.
|
||||||
|
- WAPE = `sum(|error|) / sum(|actual|)`. R² is computed on the same rows.
|
||||||
|
- The production 7-day model uses only features at least 168 hours old, so a
|
||||||
|
168-hour forecast needs no recursion.
|
||||||
|
- Weather in the holdout is observed, not forecast. Live accuracy will be lower
|
||||||
|
than the table because the dashboard uses forecast weather.
|
||||||
|
|
||||||
|
## Headline
|
||||||
|
|
||||||
|
| Model | MAE | WAPE | R² |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| Seasonal naive (exact 168-row lag) | 5.40 | 44.7% | 0.462 |
|
||||||
|
| **Weather-aware 7-day (production)** | **3.32** | **27.5%** | **0.777** |
|
||||||
|
|
||||||
|
- MAE improvement over seasonal naive: **38.5%**.
|
||||||
|
- WAPE improvement: 17.2 percentage points.
|
||||||
|
- The mission reference numbers for seasonal naive (5.40 / 44.7% / 0.462) are
|
||||||
|
reproduced exactly.
|
||||||
|
- The production row also beats the mission reference 7-day no-weather model
|
||||||
|
(4.37 / 36.2% / 0.672): weather supplies most of the gain, and `lag_week`
|
||||||
|
uses the previous *observed* same weekday/hour across sensor gaps instead of
|
||||||
|
recursion.
|
||||||
|
|
||||||
|
## Ablation
|
||||||
|
|
||||||
|
| Model | Description | MAE | WAPE | R² | n |
|
||||||
|
|---|---|---|---:|---:|---:|---:|
|
||||||
|
| `seasonal_naive` | exact 168-row lag (same observed hour last week) | 5.40 | 44.7% | 0.462 | 5,350 |
|
||||||
|
| `day_no_weather` | calendar + lags 24/48/168 + 24h/168h rolling means | 3.96 | 32.8% | 0.706 | 5,350 |
|
||||||
|
| `day_holiday` | day-ahead + Baden-Württemberg holiday flag | 3.84 | 31.8% | 0.732 | 5,350 |
|
||||||
|
| `day_weather` | day-ahead + temperature + rain (observed weather) | 3.36 | 27.8% | 0.763 | 5,350 |
|
||||||
|
| `week_no_weather` | 7-day-safe calendar + previous same weekday/hour values | 4.08 | 33.8% | 0.697 | 5,350 |
|
||||||
|
| **`week_weather`** | **7-day-safe + temperature + rain (production)** | **3.32** | **27.5%** | **0.777** | **5,350** |
|
||||||
|
|
||||||
|
Derived effects:
|
||||||
|
|
||||||
|
- Weather at day-ahead: 3.96 → 3.36 MAE, **−15.2%**.
|
||||||
|
- Weather at 7-day-safe: 4.08 → 3.32 MAE, **−18.6%**.
|
||||||
|
- Holiday flag at day-ahead: 3.96 → 3.84 MAE, **−0.12 MAE**. Holidays are
|
||||||
|
context, not a headline accuracy claim.
|
||||||
|
- The mission reference day-ahead weather row (3.36 / 27.8%) is reproduced.
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
*Figure 1. Holdout MAE for each feature set; teal is the production model.
|
||||||
|
Weather is the largest single gain; the holiday flag is a small addition.*
|
||||||
|
|
||||||
|
## Per sensor
|
||||||
|
|
||||||
|
Production model and seasonal naive on the same holdout rows.
|
||||||
|
|
||||||
|
| Sensor | Name | n | MAE | WAPE | R² | Baseline MAE | Baseline WAPE | Baseline R² | MAE improvement |
|
||||||
|
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||||
|
| `density-01` | Density 01 | 669 | 2.26 | 30.8% | 0.722 | 3.58 | 48.8% | 0.238 | 36.8% |
|
||||||
|
| `hn-goetzentuem-01` | Götzenturm | 669 | 2.40 | 26.9% | 0.718 | 3.84 | 43.0% | 0.281 | 37.5% |
|
||||||
|
| `hn-hagenbuchersee-01` | Hagenbuchersee 1 | 669 | 2.47 | 38.6% | 0.602 | 4.01 | 62.8% | −0.212 | 38.5% |
|
||||||
|
| `hn-hagenbuchersee-02` | Hagenbuchersee 2 | 669 | 4.12 | 31.1% | 0.731 | 6.49 | 48.9% | 0.337 | 36.5% |
|
||||||
|
| `hn-marahaus-01` | Marahaus | 669 | 2.67 | 29.8% | 0.671 | 4.57 | 51.0% | −0.045 | 41.7% |
|
||||||
|
| `hn-neckarufer-01` | Neckarufer | 668 | 3.63 | 28.2% | 0.695 | 5.89 | 45.8% | 0.110 | 38.4% |
|
||||||
|
| `hn-soleo-01` | Soleo | 669 | 3.19 | 20.2% | 0.548 | 4.40 | 28.0% | 0.163 | 27.7% |
|
||||||
|
| `hn-stadtgalerie-01` | Stadtgalerie | 668 | 5.83 | 25.3% | 0.765 | 10.39 | 45.1% | 0.480 | 43.9% |
|
||||||
|
|
||||||
|
- Every sensor improves over its own seasonal naive baseline (27.7%–43.9%).
|
||||||
|
- Stadtgalerie has the largest absolute error (MAE 5.83) and the largest
|
||||||
|
volume; Soleo has the smallest relative gain and the lowest R² (0.548).
|
||||||
|
- Two baselines have negative R² (`hagenbuchersee-01`, `marahaus-01`), which is
|
||||||
|
expected for a naive lag on a short, variable holdout; the model is positive
|
||||||
|
on all eight.
|
||||||
|
- Sensor totals: 669 × 6 + 668 × 2 = 5,350, matching the headline.
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
*Figure 2. Per-sensor holdout MAE, production model (blue) vs seasonal naive
|
||||||
|
(grey). Every sensor improves; the model is not carried by one large sensor.*
|
||||||
|
|
||||||
|
## Uncertainty band
|
||||||
|
|
||||||
|
The `lo`/`hi` band is the holdout residual p10/p90 added to the point forecast,
|
||||||
|
clipped at zero:
|
||||||
|
|
||||||
|
| p10 | p50 | p90 |
|
||||||
|
|---:|---:|---:|
|
||||||
|
| −4.83 | −0.59 | +4.88 |
|
||||||
|
|
||||||
|
This is an empirical 80% band, not a calibrated guarantee, and the median
|
||||||
|
residual being −0.59 shows a small optimistic bias on the holdout.
|
||||||
|
|
||||||
|
## Small busy-hour classifier
|
||||||
|
|
||||||
|
Full definition and limitations: [`docs/small-model.md`](small-model.md).
|
||||||
|
Summary from the same baked run (`logit-20261114-052723`):
|
||||||
|
|
||||||
|
| Accuracy | Precision | Recall | F1 | ROC-AUC | Train rows | Test rows | Base rate |
|
||||||
|
|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||||
|
| 0.708 | 0.360 | 0.574 | 0.443 | 0.726 | 18,242 | 5,710 | 0.202 |
|
||||||
|
|
||||||
|
| | Predicted not busy | Predicted busy |
|
||||||
|
|---|---:|---:|
|
||||||
|
| **Actually not busy** | 3,377 (TN) | 1,178 (FP) |
|
||||||
|
| **Actually busy** | 492 (FN) | 663 (TP) |
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
*Figure 3. Holdout confusion matrix for the small busy-hour classifier. The
|
||||||
|
balanced weights trade precision for recall; accuracy 0.708 is below the 0.798
|
||||||
|
majority-class baseline, so report recall/F1 and ROC-AUC instead.*
|
||||||
|
|
||||||
|
Per-sensor small-model holdout:
|
||||||
|
|
||||||
|
| Sensor | n | Positive rate | Accuracy | Precision | Recall | F1 |
|
||||||
|
|---|---:|---:|---:|---:|---:|---:|
|
||||||
|
| `density-01` | 714 | 0.111 | 0.761 | 0.300 | 0.873 | 0.447 |
|
||||||
|
| `hn-goetzentuem-01` | 714 | 0.197 | 0.732 | 0.391 | 0.638 | 0.485 |
|
||||||
|
| `hn-hagenbuchersee-01` | 714 | 0.049 | 0.724 | 0.148 | 0.971 | 0.257 |
|
||||||
|
| `hn-hagenbuchersee-02` | 714 | 0.123 | 0.783 | 0.355 | 0.932 | 0.514 |
|
||||||
|
| `hn-marahaus-01` | 714 | 0.055 | 0.710 | 0.135 | 0.795 | 0.230 |
|
||||||
|
| `hn-neckarufer-01` | 713 | 0.090 | 0.764 | 0.274 | 0.984 | 0.429 |
|
||||||
|
| `hn-soleo-01` | 714 | 0.704 | 0.581 | 0.943 | 0.431 | 0.592 |
|
||||||
|
| `hn-stadtgalerie-01` | 713 | 0.289 | 0.604 | 0.335 | 0.374 | 0.353 |
|
||||||
|
|
||||||
|
## Claims the data supports
|
||||||
|
|
||||||
|
- A global, weather-aware HistGB forecasts hourly footfall for all eight
|
||||||
|
inner-city sensors with MAE 3.32 / WAPE 27.5% / R² 0.777 on a 28-day
|
||||||
|
time-based holdout.
|
||||||
|
- Weather features are the main measurable gain: −18.6% MAE versus the 7-day
|
||||||
|
no-weather model, −38.5% versus seasonal naive.
|
||||||
|
- Every sensor improves over its own seasonal naive baseline (27.7%–43.9%).
|
||||||
|
- A tiny logistic model can flag busy hours with ROC-AUC 0.726 and 57.4%
|
||||||
|
recall at 36.0% precision, retrained from a CSV in under 5 seconds.
|
||||||
|
|
||||||
|
## Claims the data does not support
|
||||||
|
|
||||||
|
- People counts. Values are an index of estimated devices; penetration is
|
||||||
|
unknown.
|
||||||
|
- Annual seasonality, city-wide surfaces, or store-location recommendations
|
||||||
|
from five months at eight sensors in ~580 m.
|
||||||
|
- Causal weather effects. The ablation is predictive, not a treatment effect.
|
||||||
|
- Live accuracy equal to the holdout. The holdout uses observed weather; the
|
||||||
|
demo uses forecast weather and `predict` adds scenario overlays.
|
||||||
|
- Holidays as a large driver. The flag moves day-ahead MAE by only −0.12.
|
||||||
|
- The small classifier as the production forecast. It is a demo artifact with
|
||||||
|
six features and a per-window p75 target.
|
||||||
|
- Tuned hyperparameters. The model configuration is fixed; the 28-day holdout
|
||||||
|
is the held-out result, not a search result.
|
||||||
|
|
||||||
|
## Reproduce
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make data # parse + weather + holidays + JSON artifacts
|
||||||
|
make train # holdout, ablation, per-sensor, small model, forecasts
|
||||||
|
make small # print the small-model metrics and confusion matrix
|
||||||
|
make figures # regenerate docs/figures/*.svg from artifacts/metrics.json
|
||||||
|
make check # sync --frozen, ruff, compileall, pytest
|
||||||
|
```
|
||||||
@@ -0,0 +1,186 @@
|
|||||||
|
# Small model — busy-hour classifier
|
||||||
|
|
||||||
|
A logistic regression that labels an hour `busy` when footfall exceeds that
|
||||||
|
sensor's p75. It powers the `model=small` retraining demo, not the dashboard
|
||||||
|
forecast: six cheap features, a time-based split, and under a second to fit.
|
||||||
|
|
||||||
|
Every number below comes from the baked artifact
|
||||||
|
`artifacts/small_metrics.json` (version `logit-20261114-052723`, trained
|
||||||
|
2026-11-14T05:27:23Z). `make small` prints the headline metrics and per-sensor
|
||||||
|
table; `scripts/train_small.py --json` prints the full payload.
|
||||||
|
|
||||||
|
## Target
|
||||||
|
|
||||||
|
`busy` is a binary label for one sensor-hour:
|
||||||
|
|
||||||
|
```text
|
||||||
|
busy = 1 if hourly_mean_density > that sensor's 75th percentile
|
||||||
|
busy = 0 otherwise
|
||||||
|
```
|
||||||
|
|
||||||
|
- The p75 is computed **per sensor on the training window only**, then applied
|
||||||
|
unchanged to the holdout. Computing it on the full dataset would leak the
|
||||||
|
holdout distribution into the label.
|
||||||
|
- A p75 threshold makes "busy" mean "busier than three of four hours at this
|
||||||
|
location", not an absolute footfall claim. The same absolute value can be
|
||||||
|
busy at Soleo and quiet at Stadtgalerie.
|
||||||
|
- Hours above the threshold are exactly 25% of the training hours by
|
||||||
|
construction, before missing hours are dropped.
|
||||||
|
|
||||||
|
| Sensor | Training-window p75 |
|
||||||
|
|---|---:|
|
||||||
|
| `density-01` | 15.23 |
|
||||||
|
| `hn-goetzentuem-01` | 15.40 |
|
||||||
|
| `hn-hagenbuchersee-01` | 18.20 |
|
||||||
|
| `hn-hagenbuchersee-02` | 29.17 |
|
||||||
|
| `hn-marahaus-01` | 22.23 |
|
||||||
|
| `hn-neckarufer-01` | 28.35 |
|
||||||
|
| `hn-soleo-01` | 11.38 |
|
||||||
|
| `hn-stadtgalerie-01` | 38.41 |
|
||||||
|
|
||||||
|
## Data and split
|
||||||
|
|
||||||
|
- Input: the challenge CSV, `densityData` rows only, aggregated to the mean of
|
||||||
|
the 15-minute readings in each UTC hour.
|
||||||
|
- Window: 2026-06-12T00:00:00Z → 2026-11-07T23:00:00Z.
|
||||||
|
- 23,952 hourly sensor rows across all 8 usable sensors.
|
||||||
|
- Split is chronological, not random: the first 80% of **timestamps** train,
|
||||||
|
the last 20% test.
|
||||||
|
- Train: 2026-06-12T00:00:00Z → 2026-10-09T01:00:00Z, 18,242 rows.
|
||||||
|
- Test: 2026-10-09T02:00:00Z → 2026-11-07T23:00:00Z, 5,710 rows.
|
||||||
|
- Excluded devices: `maker-fair`, `niklas-01`, `hn-bollwerksturm-01` (test
|
||||||
|
installations / insufficient coverage). See `artifacts/data_audit.json`.
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
| Feature | Source | Note |
|
||||||
|
|---|---|---|
|
||||||
|
| `hour` | local Europe/Berlin hour | 0–23, captures the daily curve |
|
||||||
|
| `weekday` | local Europe/Berlin weekday | 0 = Monday |
|
||||||
|
| `is_weekend` | weekday ≥ 5 | |
|
||||||
|
| `is_holiday` | Baden-Württemberg calendar | context, not a headline claim |
|
||||||
|
| `temp_c` | observed Open-Meteo archive | joined to the nearest hour (±3 h) |
|
||||||
|
| `precip_mm` | observed Open-Meteo archive | same join |
|
||||||
|
|
||||||
|
No lags, no target history, no sensor identifier. That is intentional: the
|
||||||
|
classifier must retrain from a small user upload that may not contain a long
|
||||||
|
history, and it must not depend on future rows. Sensor identity is still
|
||||||
|
implicit in the per-sensor threshold.
|
||||||
|
|
||||||
|
## Model
|
||||||
|
|
||||||
|
```text
|
||||||
|
StandardScaler -> LogisticRegression(
|
||||||
|
C=1.0, max_iter=1000, class_weight="balanced",
|
||||||
|
solver="lbfgs", random_state=42)
|
||||||
|
```
|
||||||
|
|
||||||
|
`class_weight="balanced"` compensates for the 25/75 class split caused by the
|
||||||
|
p75 threshold. It buys recall at the cost of precision, which is the right
|
||||||
|
trade for a "flag the busy hours" demo.
|
||||||
|
|
||||||
|
Fit time: **0.30 s** on the full challenge CSV in this run (contract: < 5 s,
|
||||||
|
including the API job's CSV parsing).
|
||||||
|
|
||||||
|
## Metrics
|
||||||
|
|
||||||
|
Time-based holdout, 5,710 rows, base rate 20.2% positive.
|
||||||
|
|
||||||
|
| Accuracy | Precision | Recall | F1 | ROC-AUC |
|
||||||
|
|---:|---:|---:|---:|---:|
|
||||||
|
| 0.708 | 0.360 | 0.574 | 0.443 | 0.726 |
|
||||||
|
|
||||||
|
Confusion matrix:
|
||||||
|
|
||||||
|
| | Predicted not busy | Predicted busy |
|
||||||
|
|---|---:|---:|
|
||||||
|
| **Actually not busy** | 3,377 (TN) | 1,178 (FP) |
|
||||||
|
| **Actually busy** | 492 (FN) | 663 (TP) |
|
||||||
|
|
||||||
|
Derived:
|
||||||
|
|
||||||
|
- Specificity (true-negative rate): 0.741
|
||||||
|
- Negative predictive value: 0.873
|
||||||
|
- False-positive rate: 0.259
|
||||||
|
- False-negative rate: 0.426
|
||||||
|
- Precision lift over the 20.2% base rate: 1.78×
|
||||||
|
- Accuracy of the majority-class baseline ("always not busy"): 0.798
|
||||||
|
|
||||||
|
The model is **less accurate than always saying "not busy"**, but it finds 663
|
||||||
|
of 1,155 busy hours (57.4% recall) that the majority baseline never finds. The
|
||||||
|
balanced weights push it toward over-flagging: 1,178 false positives for 1,155
|
||||||
|
actual busy hours. That is why precision is 0.36 and accuracy is not the right
|
||||||
|
headline.
|
||||||
|
|
||||||
|
Per-sensor holdout:
|
||||||
|
|
||||||
|
| Sensor | n | Pos. rate | Accuracy | Precision | Recall | F1 |
|
||||||
|
|---|---:|---:|---:|---:|---:|---:|
|
||||||
|
| `density-01` | 714 | 0.111 | 0.761 | 0.300 | 0.873 | 0.447 |
|
||||||
|
| `hn-goetzentuem-01` | 714 | 0.197 | 0.732 | 0.391 | 0.638 | 0.485 |
|
||||||
|
| `hn-hagenbuchersee-01` | 714 | 0.049 | 0.724 | 0.148 | 0.971 | 0.257 |
|
||||||
|
| `hn-hagenbuchersee-02` | 714 | 0.123 | 0.783 | 0.355 | 0.932 | 0.514 |
|
||||||
|
| `hn-marahaus-01` | 714 | 0.055 | 0.710 | 0.135 | 0.795 | 0.230 |
|
||||||
|
| `hn-neckarufer-01` | 713 | 0.090 | 0.764 | 0.274 | 0.984 | 0.429 |
|
||||||
|
| `hn-soleo-01` | 714 | 0.704 | 0.581 | 0.943 | 0.431 | 0.592 |
|
||||||
|
| `hn-stadtgalerie-01` | 713 | 0.289 | 0.604 | 0.335 | 0.374 | 0.353 |
|
||||||
|
|
||||||
|
The pooled model has the lowest accuracy on `hn-soleo-01` (0.581, an unusually
|
||||||
|
high base rate of 70.4%) and the lowest recall on `hn-stadtgalerie-01` (0.374).
|
||||||
|
The high recall on low-base-rate sensors is mostly the balanced weights firing;
|
||||||
|
it is not evidence of a strong signal there.
|
||||||
|
|
||||||
|
## Reproduce
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Full challenge CSV, prints the metrics and per-sensor table above.
|
||||||
|
make small
|
||||||
|
|
||||||
|
# Explicit CSV, all rows, save the baked artifact.
|
||||||
|
uv run python scripts/train_small.py --csv new-data.csv --all --save
|
||||||
|
|
||||||
|
# Machine-readable payload.
|
||||||
|
uv run python scripts/train_small.py --json
|
||||||
|
```
|
||||||
|
|
||||||
|
The runtime path is the same function (`pulsecast.small.train_small_from_csv`):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -X POST localhost:18000/train \
|
||||||
|
-H "Authorization: Bearer $CONTROL_TOKEN" \
|
||||||
|
-F model=small -F file=@new-data.csv
|
||||||
|
curl -s localhost:18000/train/<job_id> \
|
||||||
|
-H "Authorization: Bearer $CONTROL_TOKEN"
|
||||||
|
```
|
||||||
|
|
||||||
|
The job returns `accuracy`, `precision`, `recall`, `f1`, `roc_auc`, `n_train`,
|
||||||
|
`n_test`, `positive_rate`, and `sensors`; the full confusion matrix and
|
||||||
|
per-sensor table are stored in `artifacts/small_metrics.json`. On success the
|
||||||
|
bundle is hot-swapped and persisted to `PULSECAST_MODELS/small_latest.joblib`.
|
||||||
|
A restart falls back to the baked artifact.
|
||||||
|
|
||||||
|
## Honest limits
|
||||||
|
|
||||||
|
- **It is a demo, not the production forecast.** The dashboard uses the global
|
||||||
|
HistGB 7-day model, not this classifier.
|
||||||
|
- **Threshold convention.** "Busy" is the sensor's own p75, recomputed from the
|
||||||
|
upload's training window. A different upload can move the thresholds and
|
||||||
|
change every metric. Do not compare runs trained on different windows.
|
||||||
|
- **No lags or sensor id.** The model sees only calendar and weather. It cannot
|
||||||
|
know that the previous hour was packed, and it pools all locations while
|
||||||
|
evaluating per sensor.
|
||||||
|
- **Weather fallback.** If an uploaded CSV does not overlap the baked weather
|
||||||
|
archive, missing weather is filled with 14 °C / 0 mm. That keeps the job
|
||||||
|
running but weakens the weather features; the returned metrics show it.
|
||||||
|
- **One chronological split.** Five months, one holdout. No cross-validation,
|
||||||
|
no annual seasonality claim, no per-weekday calibration.
|
||||||
|
- **Threshold metrics only.** This page reports hard 0.5-threshold metrics, not
|
||||||
|
a calibrated probability. The probability output is not a business forecast.
|
||||||
|
- **Counters are devices.** Footfall density is an index of estimated devices,
|
||||||
|
not a count of people. A "busy" flag inherits that uncertainty.
|
||||||
|
- **Small uploads are unstable.** A CSV with one sensor, only one class, or
|
||||||
|
fewer than four hourly timestamps fails with a clear error. A CSV with a
|
||||||
|
short window may train but produce meaningless metrics.
|
||||||
|
- **No causal claims.** Weather and holiday coefficients are associations in
|
||||||
|
this sample, not causal effects, and the classifier is not used for scenario
|
||||||
|
uplift (that overlay is explicit and separate).
|
||||||
@@ -0,0 +1,119 @@
|
|||||||
|
# Training end-to-end evidence
|
||||||
|
|
||||||
|
The script starts a real `uvicorn pulsecast.server:app` process with
|
||||||
|
`CONTROL_TOKEN`, waits for startup, and drives the HTTP API only. It checks the
|
||||||
|
snapshot, every prediction/scenario path, auth and error paths, one job at a
|
||||||
|
time, and both training jobs; it exits non-zero on the first failed check.
|
||||||
|
Unit/contract tests (`make check`) are separate.
|
||||||
|
|
||||||
|
Reproduce with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python scripts/e2e_train.py --port 18000 --report /tmp/e2e.json
|
||||||
|
```
|
||||||
|
|
||||||
|
## Run
|
||||||
|
|
||||||
|
| Field | Value |
|
||||||
|
|---|---|
|
||||||
|
| Date | 2026-11-14, 05:33 UTC |
|
||||||
|
| Branch | `feat/ml-final` |
|
||||||
|
| Port | **18000** |
|
||||||
|
| Checks | **38 passed, 0 failed** |
|
||||||
|
| Total wall time | 10.4 s |
|
||||||
|
| Forecast version before | `histgb-2026-11-14` |
|
||||||
|
| Data version | `2026-11-07T23:51:00Z` |
|
||||||
|
| Context version | `2026-11-14T03:47:09Z` |
|
||||||
|
|
||||||
|
## Snapshot and robustness (20 checks)
|
||||||
|
|
||||||
|
- `/healthz` → `{"ok":true}`; `/readyz` → 200 `model=true`.
|
||||||
|
- `/meta` covers 8 sensors and the baked model beats the seasonal-naive
|
||||||
|
baseline on the holdout.
|
||||||
|
- `/bootstrap` → 8 sensors, all 8 hourly series (23,952 points), 168 forecast
|
||||||
|
weather hours, 24 holidays, versions equal to `/meta`.
|
||||||
|
- `/predict`: **128/128** combinations (8 sensors × 4 horizons × 4 scenarios)
|
||||||
|
returned the contract point counts (`15m` 1, `1h` 4, `48h` 48, `7d` 168),
|
||||||
|
monotonic UTC timestamps, non-negative values, and `lo ≤ v ≤ hi`.
|
||||||
|
- `/simulate`: all 4 scenarios × 8 sensors × 4 horizons returned valid points.
|
||||||
|
- Weather override changes the forecast (`-8 °C / 6 mm` vs `30 °C / 0 mm`).
|
||||||
|
- Unknown sensor → 404; missing token → 401 `unauthorized`; wrong token → 401;
|
||||||
|
unknown job → 404 `not_found`; invalid model → 400 `bad_request`; missing
|
||||||
|
JSON path → 400.
|
||||||
|
|
||||||
|
## Small training job (real 60-day CSV subset)
|
||||||
|
|
||||||
|
Input: 36,846 density rows, 8 sensors, 2026-09-08T23:53Z → 2026-11-07T23:51Z.
|
||||||
|
|
||||||
|
| Field | Value |
|
||||||
|
|---|---|
|
||||||
|
| Job ID | `0b35347963974cbbbe584b77ecd4b846` |
|
||||||
|
| Status | `succeeded` |
|
||||||
|
| Model version | `logit-20261114-053344` |
|
||||||
|
| Wall time (submit → terminal) | **0.172 s** (contract < 5 s) |
|
||||||
|
| n_train / n_test | 8,151 / 2,301 |
|
||||||
|
| Base rate | 0.180 |
|
||||||
|
| Accuracy | 0.824 |
|
||||||
|
| Precision | 0.542 |
|
||||||
|
| Recall | 0.157 |
|
||||||
|
| F1 | 0.243 |
|
||||||
|
| ROC-AUC | 0.813 |
|
||||||
|
|
||||||
|
This is a different window from the baked full-window model in
|
||||||
|
`docs/small-model.md`, so the numbers are not directly comparable: the p75
|
||||||
|
target and the 80/20 split are recomputed inside the 60-day upload. The low
|
||||||
|
recall illustrates the limit noted in the small-model doc: a short, recent
|
||||||
|
window with a different threshold is a different classification problem.
|
||||||
|
|
||||||
|
Post-conditions verified:
|
||||||
|
|
||||||
|
- `small_latest.joblib` and `<job_id>-small.joblib` exist under
|
||||||
|
`PULSECAST_MODELS`.
|
||||||
|
- `/meta.small_model.model_version` equals the job version and its metrics
|
||||||
|
match the job exactly (in-memory hot-swap).
|
||||||
|
- `/predict` still returns 168 valid points after the swap.
|
||||||
|
|
||||||
|
## Forecast training job (full challenge CSV) and 409
|
||||||
|
|
||||||
|
Input: full `schwarz-pax.csv`, 19,383,568 bytes.
|
||||||
|
|
||||||
|
| Field | Value |
|
||||||
|
|---|---|
|
||||||
|
| Job ID | `235209a0d00f449ba8be717eb76386d6` |
|
||||||
|
| Status | `succeeded` |
|
||||||
|
| Model version | `histgb-20261114-053349` |
|
||||||
|
| Wall time | **4.380 s** |
|
||||||
|
| MAE | 3.32 |
|
||||||
|
| WAPE | 27.5% |
|
||||||
|
| R² | 0.777 |
|
||||||
|
| Baseline MAE / WAPE / R² | 5.40 / 44.7% / 0.462 |
|
||||||
|
| n_eval | 5,350 |
|
||||||
|
|
||||||
|
The ablation in the job payload matches `artifacts/metrics.json` row for row
|
||||||
|
(3.96, 3.84, 3.36, 4.08, 3.32). While this job was in `running`, a second
|
||||||
|
`POST /train` returned **409** `conflict`, proving one job at a time.
|
||||||
|
|
||||||
|
Post-conditions verified:
|
||||||
|
|
||||||
|
- `forecast_latest.joblib` and `<job_id>-forecast.joblib` exist under
|
||||||
|
`PULSECAST_MODELS`.
|
||||||
|
- `/meta.model_version` equals the retrained version and
|
||||||
|
`/meta.model_source` is `retrained`.
|
||||||
|
- `/predict` reports the retrained version; `/readyz` stays 200.
|
||||||
|
- `data_version` after the retrain is `2026-11-07T23:00:00Z` (the hourly input
|
||||||
|
maximum) rather than the raw-file floor `23:51:00Z`; both describe the same
|
||||||
|
observed window.
|
||||||
|
|
||||||
|
## Failure isolation
|
||||||
|
|
||||||
|
A malformed CSV (`not,a,pulsecast,file`) was accepted as a job (202), then
|
||||||
|
failed with a clear error:
|
||||||
|
|
||||||
|
```text
|
||||||
|
CSV needs a time column (time, timestamp, datetime, date, hour) and a value
|
||||||
|
column (value, density, footfall, count, pax, visitors); got ['not', 'a',
|
||||||
|
'pulsecast', 'file']
|
||||||
|
```
|
||||||
|
|
||||||
|
`/healthz` still returned 200 afterward: a bad upload cannot take down the
|
||||||
|
service or block `/predict`.
|
||||||
+63
-18
@@ -33,7 +33,11 @@ Day-ahead ablation models additionally use `lag_day`, `lag_2d`, `roll_24`,
|
|||||||
## Target and data
|
## Target and data
|
||||||
|
|
||||||
- Target: mean of 15-minute `densityData` readings in a UTC hour, per sensor.
|
- Target: mean of 15-minute `densityData` readings in a UTC hour, per sensor.
|
||||||
- Training window: 2026-06-12 → 2026-10-10; holdout: 2026-10-10 → 2026-11-07.
|
- Training rows: 2026-06-12T00:00:00Z → 2026-10-10T22:00:00Z, 18,602 rows.
|
||||||
|
- Holdout: 2026-10-10T23:00:00Z → 2026-11-07T23:00:00Z, 5,350
|
||||||
|
evaluation rows (hours with a seasonal-naive baseline).
|
||||||
|
- The production bundle is refit on all observed rows (through
|
||||||
|
2026-11-07T23:00:00Z) only after the holdout is evaluated.
|
||||||
- Eight usable sensors after excluding `maker-fair` (73 rows, Hannover),
|
- Eight usable sensors after excluding `maker-fair` (73 rows, Hannover),
|
||||||
`niklas-01` (8 rows), `hn-bollwerksturm-01` (5 rows).
|
`niklas-01` (8 rows), `hn-bollwerksturm-01` (5 rows).
|
||||||
- `density-01` merges two physical devices (`863f75b0`, `f8c19730`) into one
|
- `density-01` merges two physical devices (`863f75b0`, `f8c19730`) into one
|
||||||
@@ -57,35 +61,76 @@ Time-based 28-day holdout, no shuffling. Seasonal naive is the baseline. WAPE is
|
|||||||
| 7-day-safe, no weather | 4.08 | 33.8% | 0.697 |
|
| 7-day-safe, no weather | 4.08 | 33.8% | 0.697 |
|
||||||
| **7-day-safe, + temperature/rain (production)** | **3.32** | **27.5%** | **0.777** |
|
| **7-day-safe, + temperature/rain (production)** | **3.32** | **27.5%** | **0.777** |
|
||||||
|
|
||||||
Weather is the useful signal: adding temperature and rain cuts MAE by ~19%
|
Weather is the useful signal: adding temperature and rain cuts MAE by 18.6%
|
||||||
versus the 7-day no-weather model and ~38% versus seasonal naive. The holiday
|
versus the 7-day no-weather model and 38.5% versus seasonal naive. The holiday
|
||||||
flag alone changes day-ahead MAE by −0.12 in this run; holidays are context, not
|
flag alone changes day-ahead MAE by −0.12 in this run; holidays are context, not
|
||||||
a headline accuracy claim.
|
a headline accuracy claim.
|
||||||
|
|
||||||
The seasonal-naive row reproduces the mission reference exactly
|
The seasonal-naive row reproduces the mission reference exactly
|
||||||
(5.40 / 44.7% / 0.462), and the day-ahead weather row matches 3.36 / 27.8%. The
|
(5.40 / 44.7% / 0.462), and the day-ahead weather row matches 3.36 / 27.8%. The
|
||||||
production model is slightly better than the reference because `lag_week` uses
|
production 7-day model edges that day-ahead reference on MAE and WAPE
|
||||||
the previous *observed* same weekday/hour across sensor gaps, while the baseline
|
(3.32 / 27.5%) but is slightly lower on R² (0.777 vs 0.779). The two use
|
||||||
uses the exact 168-row lag. R² differs slightly from the reference 0.779
|
different features; `lag_week` uses the previous *observed* same weekday/hour
|
||||||
(0.777 here) because of that feature choice.
|
across sensor gaps, while the seasonal-naive baseline uses the exact 168-row
|
||||||
|
lag.
|
||||||
|
|
||||||
|
Per-sensor holdout MAE improves over each sensor's own seasonal naive by
|
||||||
|
27.7%–43.9%; the full table, figures, and provenance are in
|
||||||
|
[`docs/metrics.md`](docs/metrics.md). The largest absolute error is
|
||||||
|
Stadtgalerie (MAE 5.83, its baseline 10.39); Soleo has the smallest relative
|
||||||
|
gain (27.7%) and lowest R² (0.548). Every sensor has a positive R² under the
|
||||||
|
model; two seasonal-naive baselines are negative.
|
||||||
|
|
||||||
## Uncertainty band
|
## Uncertainty band
|
||||||
|
|
||||||
The `lo`/`hi` band is the 10th/90th percentile of holdout residuals added to the
|
The `lo`/`hi` band is the 10th/90th percentile of holdout residuals added to the
|
||||||
point forecast, clipped at zero. It is an empirical 80% band, not a calibrated
|
point forecast, clipped at zero: p10 −4.83, p50 −0.59, p90 +4.88. It is an
|
||||||
guarantee.
|
empirical 80% band, not a calibrated guarantee; the negative median residual
|
||||||
|
means the point forecast is slightly optimistic on the holdout.
|
||||||
|
|
||||||
## Small classifier
|
## Small classifier ("I coded an AI" artifact)
|
||||||
|
|
||||||
|
This model is separate from the production forecaster and powers the
|
||||||
|
`model=small` retraining demo. Full definition, thresholds, confusion matrix
|
||||||
|
interpretation, and limits: [`docs/small-model.md`](docs/small-model.md).
|
||||||
|
|
||||||
- Target: `busy` = hourly footfall strictly above that sensor's p75, where the
|
- Target: `busy` = hourly footfall strictly above that sensor's p75, where the
|
||||||
p75 is computed on the training window only.
|
p75 is computed on the training window only and then frozen for the holdout.
|
||||||
- Features: `hour`, `weekday`, `is_weekend`, `temp_c`, `precip_mm`, `is_holiday`.
|
- Features: `hour`, `weekday`, `is_weekend`, `temp_c`, `precip_mm`, `is_holiday`
|
||||||
- Model: standardized logistic regression, `class_weight="balanced"`.
|
(no lags, no sensor identifier, no target history).
|
||||||
- Split: last 20% of hourly timestamps, time-ordered.
|
- Model: standardized logistic regression, `class_weight="balanced"`, `C=1.0`,
|
||||||
- Result: accuracy 0.708, precision 0.360, recall 0.574, F1 0.443, ROC-AUC 0.726.
|
`max_iter=1000`, `random_state=42`.
|
||||||
- Trains in ~0.25 s on the challenge CSV and under 1 s on a small upload.
|
- Split: last 20% of hourly timestamps, time-ordered. Train
|
||||||
- The low precision reflects a hard p75 threshold and a deliberately tiny model;
|
2026-06-12T00:00Z → 2026-10-09T01:00Z (18,242 rows); test
|
||||||
it is a demo artifact, not the production forecast.
|
2026-10-09T02:00Z → 2026-11-07T23:00Z (5,710 rows).
|
||||||
|
- Result (`logit-20261114-052723`): accuracy 0.708, precision 0.360, recall
|
||||||
|
0.574, F1 0.443, ROC-AUC 0.726, base rate 0.202.
|
||||||
|
- Confusion matrix: TN 3,377, FP 1,178, FN 492, TP 663. Specificity 0.741,
|
||||||
|
negative predictive value 0.873.
|
||||||
|
- Fit time: 0.30 s on the full challenge CSV; the API job (parse + fit) must
|
||||||
|
stay under the 5 s contract. The live e2e run trained a 60-day upload subset
|
||||||
|
in 0.17 s wall time.
|
||||||
|
- Per-sensor F1 ranges from 0.230 (Marahaus, 5.5% positive) to 0.592 (Soleo,
|
||||||
|
70.4% positive). The balanced weights over-flag low-base-rate sensors; accuracy
|
||||||
|
is lowest on Soleo (0.581) and recall is lowest on Stadtgalerie (0.374).
|
||||||
|
- Accuracy is below the 0.798 majority-class baseline. Do not use accuracy as
|
||||||
|
the headline: the model finds 663 of 1,155 busy hours, which the majority
|
||||||
|
baseline never finds. The low precision reflects a hard p75 threshold, a
|
||||||
|
deliberately tiny model, and balanced class weights.
|
||||||
|
- This is a demo artifact, not the production forecast. Thresholds move with
|
||||||
|
the upload window, and a CSV without weather overlap falls back to
|
||||||
|
14 °C / 0 mm.
|
||||||
|
|
||||||
|
## Artifacts and reproduction
|
||||||
|
|
||||||
|
- `artifacts/metrics.json` — versions, headline, ablation, per-sensor, small
|
||||||
|
model summary.
|
||||||
|
- `artifacts/small_metrics.json` — full small-model payload including the
|
||||||
|
confusion matrix, per-sensor table, and thresholds.
|
||||||
|
- `make train` regenerates models and metrics; `make small` prints the
|
||||||
|
small-model numbers; `make figures` regenerates `docs/figures/*.svg`.
|
||||||
|
- `docs/metrics.md` is the paper support sheet; it quotes the JSON and does not
|
||||||
|
invent numbers.
|
||||||
|
|
||||||
## No-leakage rules
|
## No-leakage rules
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,523 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""End-to-end acceptance run against a real FastAPI process on port 18000.
|
||||||
|
|
||||||
|
Starts uvicorn with ``CONTROL_TOKEN``, then exercises robustness and the full
|
||||||
|
training lifecycle:
|
||||||
|
|
||||||
|
- /healthz, /readyz, /meta, /bootstrap (8 sensors, hourly series, context)
|
||||||
|
- /predict for every sensor x horizon x scenario (128 calls)
|
||||||
|
- /simulate for every scenario with all sensors
|
||||||
|
- auth, unknown job, unknown model, malformed CSV, one-job-at-a-time
|
||||||
|
- POST /train model=small on a real CSV subset; poll to success; hot-swap
|
||||||
|
- POST /train model=forecast on the full CSV; poll to success; hot-swap
|
||||||
|
- /predict still works after each swap
|
||||||
|
|
||||||
|
Exit code is 0 only when every check passes. Figures printed to stdout are the
|
||||||
|
measured numbers; ``--report`` writes the same result as JSON.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
import socket
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import time
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from pulsecast import config
|
||||||
|
|
||||||
|
HORIZON_SIZES = {"15m": 1, "1h": 4, "48h": 48, "7d": 168}
|
||||||
|
SCENARIOS = tuple(config.SCENARIOS)
|
||||||
|
SENSORS = tuple(sorted(config.USABLE_SENSORS))
|
||||||
|
|
||||||
|
|
||||||
|
class CheckFailed(RuntimeError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class Runner:
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.checks = 0
|
||||||
|
self.results: dict = {}
|
||||||
|
|
||||||
|
def check(self, condition: bool, message: str) -> None:
|
||||||
|
self.checks += 1
|
||||||
|
if not condition:
|
||||||
|
raise CheckFailed(message)
|
||||||
|
print(f" [ok] {message}")
|
||||||
|
|
||||||
|
|
||||||
|
def _utc_now() -> str:
|
||||||
|
return datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_ts(value: str) -> pd.Timestamp:
|
||||||
|
return pd.Timestamp(value)
|
||||||
|
|
||||||
|
|
||||||
|
def _seconds(start: str | None, end: str | None) -> float | None:
|
||||||
|
if not start or not end:
|
||||||
|
return None
|
||||||
|
return round((_parse_ts(end) - _parse_ts(start)).total_seconds(), 3)
|
||||||
|
|
||||||
|
|
||||||
|
def _wait_http(client: httpx.Client, url: str, timeout: float = 30.0) -> None:
|
||||||
|
deadline = time.monotonic() + timeout
|
||||||
|
last: Exception | None = None
|
||||||
|
while time.monotonic() < deadline:
|
||||||
|
try:
|
||||||
|
response = client.get(url, timeout=2.0)
|
||||||
|
if response.status_code == 200:
|
||||||
|
return
|
||||||
|
except Exception as exc: # noqa: BLE001 - startup race
|
||||||
|
last = exc
|
||||||
|
time.sleep(0.1)
|
||||||
|
raise CheckFailed(f"server did not answer {url} within {timeout}s: {last}")
|
||||||
|
|
||||||
|
|
||||||
|
def _wait_job(
|
||||||
|
client: httpx.Client,
|
||||||
|
job_id: str,
|
||||||
|
headers: dict[str, str],
|
||||||
|
*,
|
||||||
|
timeout: float = 180.0,
|
||||||
|
state: str | None = None,
|
||||||
|
) -> dict:
|
||||||
|
deadline = time.monotonic() + timeout
|
||||||
|
last_status = None
|
||||||
|
while time.monotonic() < deadline:
|
||||||
|
response = client.get(f"/train/{job_id}", headers=headers)
|
||||||
|
if response.status_code != 200:
|
||||||
|
raise CheckFailed(f"GET /train/{job_id} -> {response.status_code}: {response.text}")
|
||||||
|
job = response.json()
|
||||||
|
last_status = job["status"]
|
||||||
|
if state is not None and last_status == state:
|
||||||
|
return job
|
||||||
|
if state is None and last_status in {"succeeded", "failed"}:
|
||||||
|
return job
|
||||||
|
time.sleep(0.05)
|
||||||
|
raise CheckFailed(f"job {job_id} did not reach {state or 'terminal'} (last={last_status})")
|
||||||
|
|
||||||
|
|
||||||
|
def _job_headers(token: str) -> dict[str, str]:
|
||||||
|
return {"Authorization": f"Bearer {token}"}
|
||||||
|
|
||||||
|
|
||||||
|
def _submit(
|
||||||
|
client: httpx.Client, headers: dict[str, str], path: Path, model: str
|
||||||
|
) -> httpx.Response:
|
||||||
|
return client.post(
|
||||||
|
"/train",
|
||||||
|
headers=headers,
|
||||||
|
files={"file": (path.name, path.read_bytes(), "text/csv")},
|
||||||
|
data={"model": model},
|
||||||
|
timeout=120.0,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_points(points: list[dict], expected: int, label: str) -> None:
|
||||||
|
if len(points) != expected:
|
||||||
|
raise CheckFailed(f"{label}: expected {expected} points, got {len(points)}")
|
||||||
|
previous = None
|
||||||
|
for point in points:
|
||||||
|
if set(point) != {"t", "v", "lo", "hi"}:
|
||||||
|
raise CheckFailed(f"{label}: unexpected point keys {sorted(point)}")
|
||||||
|
if not point["t"].endswith("Z"):
|
||||||
|
raise CheckFailed(f"{label}: timestamp is not UTC RFC 3339: {point['t']}")
|
||||||
|
if point["v"] < 0 or point["lo"] > point["v"] or point["v"] > point["hi"]:
|
||||||
|
raise CheckFailed(f"{label}: invalid band {point}")
|
||||||
|
stamp = _parse_ts(point["t"])
|
||||||
|
if previous is not None and stamp <= previous:
|
||||||
|
raise CheckFailed(f"{label}: timestamps are not increasing at {point['t']}")
|
||||||
|
previous = stamp
|
||||||
|
|
||||||
|
|
||||||
|
def _versions(payload: dict) -> dict:
|
||||||
|
return {
|
||||||
|
key: payload.get(key)
|
||||||
|
for key in ("data_version", "model_version", "context_version")
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _build_subset(raw: Path, destination: Path, days: int = 60) -> tuple[Path, dict]:
|
||||||
|
frame = pd.read_csv(raw)
|
||||||
|
frame = frame[frame["type"].astype(str) == "densityData"].copy()
|
||||||
|
frame["time"] = pd.to_datetime(frame["time"], utc=True)
|
||||||
|
end = frame["time"].max()
|
||||||
|
cut = end - pd.Timedelta(days=days)
|
||||||
|
frame = frame[frame["time"] >= cut][["time", "name", "value"]]
|
||||||
|
frame["time"] = frame["time"].dt.strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||||
|
frame.to_csv(destination, index=False)
|
||||||
|
return destination, {
|
||||||
|
"rows": len(frame),
|
||||||
|
"sensors": int(frame["name"].nunique()),
|
||||||
|
"from": frame["time"].min(),
|
||||||
|
"to": frame["time"].max(),
|
||||||
|
"days": days,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _port_free(port: int) -> bool:
|
||||||
|
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
|
||||||
|
sock.settimeout(0.5)
|
||||||
|
return sock.connect_ex(("127.0.0.1", port)) != 0
|
||||||
|
|
||||||
|
|
||||||
|
def run(args: argparse.Namespace, runner: Runner) -> dict:
|
||||||
|
raw = Path(args.csv)
|
||||||
|
if not raw.exists():
|
||||||
|
raise CheckFailed(f"CSV not found: {raw}")
|
||||||
|
if not _port_free(args.port):
|
||||||
|
raise CheckFailed(f"port {args.port} is already in use")
|
||||||
|
|
||||||
|
workdir = Path(tempfile.mkdtemp(prefix="pulsecast-e2e-"))
|
||||||
|
models_dir = workdir / "models"
|
||||||
|
uploads_dir = workdir / "uploads"
|
||||||
|
subset_path, subset_info = _build_subset(raw, workdir / "small-subset.csv", args.days)
|
||||||
|
garbage_path = workdir / "garbage.csv"
|
||||||
|
garbage_path.write_text("not,a,pulsecast,file\n1,2,3,4\n")
|
||||||
|
|
||||||
|
env = os.environ.copy()
|
||||||
|
env.update(
|
||||||
|
CONTROL_TOKEN=args.token,
|
||||||
|
PULSECAST_MODELS=str(models_dir),
|
||||||
|
PULSECAST_UPLOADS=str(uploads_dir),
|
||||||
|
)
|
||||||
|
log_path = workdir / "server.log"
|
||||||
|
log_handle = log_path.open("wb")
|
||||||
|
command = [
|
||||||
|
sys.executable,
|
||||||
|
"-m",
|
||||||
|
"uvicorn",
|
||||||
|
"pulsecast.server:app",
|
||||||
|
"--host",
|
||||||
|
"127.0.0.1",
|
||||||
|
"--port",
|
||||||
|
str(args.port),
|
||||||
|
"--log-level",
|
||||||
|
"warning",
|
||||||
|
]
|
||||||
|
process = subprocess.Popen(command, env=env, stdout=log_handle, stderr=subprocess.STDOUT)
|
||||||
|
report: dict = {
|
||||||
|
"started_at": _utc_now(),
|
||||||
|
"base_url": f"http://127.0.0.1:{args.port}",
|
||||||
|
"subset": subset_info,
|
||||||
|
"raw_csv_bytes": raw.stat().st_size,
|
||||||
|
}
|
||||||
|
|
||||||
|
try:
|
||||||
|
with httpx.Client(base_url=f"http://127.0.0.1:{args.port}", timeout=120.0) as client:
|
||||||
|
_wait_http(client, "/healthz")
|
||||||
|
headers = _job_headers(args.token)
|
||||||
|
|
||||||
|
print("Boot and snapshot")
|
||||||
|
health = client.get("/healthz").json()
|
||||||
|
runner.check(health == {"ok": True}, "/healthz returns {ok:true}")
|
||||||
|
ready = client.get("/readyz")
|
||||||
|
runner.check(ready.status_code == 200 and ready.json()["model"] is True, "/readyz 200 model=true")
|
||||||
|
meta_before = client.get("/meta").json()
|
||||||
|
runner.check(meta_before["coverage"]["sensors"] == 8, "/meta covers 8 sensors")
|
||||||
|
runner.check(
|
||||||
|
meta_before["metrics"]["mae"] < meta_before["metrics"]["baseline_mae"],
|
||||||
|
"baked model beats seasonal naive on the holdout",
|
||||||
|
)
|
||||||
|
bootstrap = client.get("/bootstrap").json()
|
||||||
|
runner.check(len(bootstrap["sensors"]) == 8, "/bootstrap returns 8 sensors")
|
||||||
|
runner.check(
|
||||||
|
set(bootstrap["series"]) == set(SENSORS),
|
||||||
|
"/bootstrap series covers all 8 sensors",
|
||||||
|
)
|
||||||
|
series_points = sum(len(points) for points in bootstrap["series"].values())
|
||||||
|
runner.check(series_points > 20_000, f"/bootstrap carries {series_points} hourly points")
|
||||||
|
context = bootstrap["context"]
|
||||||
|
runner.check(
|
||||||
|
len(context["weather"]["forecast"]) == 168,
|
||||||
|
"/bootstrap context has 168 forecast weather hours",
|
||||||
|
)
|
||||||
|
runner.check(bool(context["holidays"]), "/bootstrap context has holidays")
|
||||||
|
runner.check(
|
||||||
|
_versions(bootstrap) == _versions(meta_before),
|
||||||
|
"/bootstrap versions match /meta",
|
||||||
|
)
|
||||||
|
report["versions_before"] = _versions(meta_before)
|
||||||
|
report["bootstrap"] = {"sensors": 8, "series_points": series_points}
|
||||||
|
|
||||||
|
print("Robustness: /predict every sensor x horizon x scenario")
|
||||||
|
predict_calls = 0
|
||||||
|
for sensor in SENSORS:
|
||||||
|
for horizon, count in HORIZON_SIZES.items():
|
||||||
|
for scenario in SCENARIOS:
|
||||||
|
response = client.post(
|
||||||
|
"/predict",
|
||||||
|
json={"sensor": sensor, "horizon": horizon, "scenario": scenario},
|
||||||
|
)
|
||||||
|
if response.status_code != 200:
|
||||||
|
raise CheckFailed(
|
||||||
|
f"/predict {sensor}/{horizon}/{scenario} -> "
|
||||||
|
f"{response.status_code}: {response.text}"
|
||||||
|
)
|
||||||
|
payload = response.json()
|
||||||
|
_assert_points(
|
||||||
|
payload["points"],
|
||||||
|
count,
|
||||||
|
f"/predict {sensor}/{horizon}/{scenario}",
|
||||||
|
)
|
||||||
|
predict_calls += 1
|
||||||
|
runner.check(predict_calls == 128, "128 /predict combinations returned valid points")
|
||||||
|
|
||||||
|
print("Robustness: /simulate every scenario, all sensors")
|
||||||
|
simulate_calls = 0
|
||||||
|
for scenario in SCENARIOS:
|
||||||
|
response = client.post(
|
||||||
|
"/simulate", json={"scenario": scenario, "sensors": "all"}
|
||||||
|
)
|
||||||
|
if response.status_code != 200:
|
||||||
|
raise CheckFailed(f"/simulate {scenario} -> {response.status_code}: {response.text}")
|
||||||
|
payload = response.json()
|
||||||
|
if set(payload["forecasts"]) != set(SENSORS):
|
||||||
|
raise CheckFailed(f"/simulate {scenario}: sensor set mismatch")
|
||||||
|
for sensor, by_horizon in payload["forecasts"].items():
|
||||||
|
if set(by_horizon) != set(HORIZON_SIZES):
|
||||||
|
raise CheckFailed(f"/simulate {scenario}/{sensor}: horizon set mismatch")
|
||||||
|
for horizon, count in HORIZON_SIZES.items():
|
||||||
|
_assert_points(
|
||||||
|
by_horizon[horizon],
|
||||||
|
count,
|
||||||
|
f"/simulate {scenario}/{sensor}/{horizon}",
|
||||||
|
)
|
||||||
|
simulate_calls += 1
|
||||||
|
runner.check(simulate_calls == 4, "4 /simulate scenarios x 8 sensors x 4 horizons")
|
||||||
|
|
||||||
|
cold = client.post(
|
||||||
|
"/predict",
|
||||||
|
json={"sensor": SENSORS[0], "horizon": "48h", "weather": {"temp_c": -8.0, "precip_mm": 6.0}},
|
||||||
|
).json()["points"]
|
||||||
|
warm = client.post(
|
||||||
|
"/predict",
|
||||||
|
json={"sensor": SENSORS[0], "horizon": "48h", "weather": {"temp_c": 30.0, "precip_mm": 0.0}},
|
||||||
|
).json()["points"]
|
||||||
|
runner.check(cold != warm, "weather override changes the forecast")
|
||||||
|
|
||||||
|
print("Error handling")
|
||||||
|
no_token = client.post("/predict", json={"sensor": "nope"})
|
||||||
|
runner.check(no_token.status_code == 404, "/predict unknown sensor -> 404")
|
||||||
|
unauthorized = _submit(client, {}, subset_path, "small")
|
||||||
|
runner.check(
|
||||||
|
unauthorized.status_code == 401
|
||||||
|
and unauthorized.json()["error"]["code"] == "unauthorized",
|
||||||
|
"/train without token -> 401 unauthorized",
|
||||||
|
)
|
||||||
|
bad_token = _submit(client, _job_headers("wrong-token"), subset_path, "small")
|
||||||
|
runner.check(bad_token.status_code == 401, "/train with wrong token -> 401")
|
||||||
|
unknown_job = client.get("/train/nope", headers=headers)
|
||||||
|
runner.check(
|
||||||
|
unknown_job.status_code == 404
|
||||||
|
and unknown_job.json()["error"]["code"] == "not_found",
|
||||||
|
"/train/nope -> 404 not_found",
|
||||||
|
)
|
||||||
|
bad_model = _submit(client, headers, subset_path, "giant")
|
||||||
|
runner.check(
|
||||||
|
bad_model.status_code == 400
|
||||||
|
and bad_model.json()["error"]["code"] == "bad_request",
|
||||||
|
"/train invalid model -> 400 bad_request",
|
||||||
|
)
|
||||||
|
missing_path = client.post(
|
||||||
|
"/train", headers=headers, json={"path": "/tmp/no-such-pulsecast.csv", "model": "small"}
|
||||||
|
)
|
||||||
|
runner.check(missing_path.status_code == 400, "/train missing JSON path -> 400")
|
||||||
|
|
||||||
|
print("Small model: real CSV subset through the API")
|
||||||
|
started = time.monotonic()
|
||||||
|
submitted = _submit(client, headers, subset_path, "small")
|
||||||
|
if submitted.status_code != 202:
|
||||||
|
raise CheckFailed(f"/train small -> {submitted.status_code}: {submitted.text}")
|
||||||
|
small_submitted = submitted.json()
|
||||||
|
runner.check(
|
||||||
|
small_submitted["status"] == "queued" and small_submitted["model"] == "small",
|
||||||
|
f"/train small queued as job {small_submitted['job_id']}",
|
||||||
|
)
|
||||||
|
small_job = _wait_job(client, small_submitted["job_id"], headers)
|
||||||
|
wall = round(time.monotonic() - started, 3)
|
||||||
|
if small_job["status"] != "succeeded":
|
||||||
|
raise CheckFailed(f"small job failed: {small_job}")
|
||||||
|
small_metrics = small_job["metrics"]
|
||||||
|
for key in ("accuracy", "precision", "recall", "f1", "roc_auc", "n_train", "n_test", "positive_rate"):
|
||||||
|
if key not in small_metrics:
|
||||||
|
raise CheckFailed(f"small metrics missing {key}: {small_metrics}")
|
||||||
|
train_seconds = _seconds(small_job["started_at"], small_job["finished_at"])
|
||||||
|
runner.check(
|
||||||
|
small_job["model_version"].startswith("logit-"),
|
||||||
|
f"small model_version {small_job['model_version']}",
|
||||||
|
)
|
||||||
|
runner.check(
|
||||||
|
wall < 5.0,
|
||||||
|
f"small job took {wall}s wall time including upload (contract < 5 s; "
|
||||||
|
f"server timestamps have 1 s resolution, train_seconds={train_seconds})",
|
||||||
|
)
|
||||||
|
runner.check(small_metrics["accuracy"] > 0.6, f"small accuracy {small_metrics['accuracy']}")
|
||||||
|
runner.check(
|
||||||
|
(models_dir / "small_latest.joblib").exists(),
|
||||||
|
"small model persisted to PULSECAST_MODELS",
|
||||||
|
)
|
||||||
|
meta_after_small = client.get("/meta").json()
|
||||||
|
runner.check(
|
||||||
|
meta_after_small["small_model"]["model_version"] == small_job["model_version"],
|
||||||
|
"small model hot-swapped into /meta",
|
||||||
|
)
|
||||||
|
runner.check(
|
||||||
|
meta_after_small["small_model"]["metrics"]["accuracy"] == small_metrics["accuracy"],
|
||||||
|
"hot-swapped small metrics match the job",
|
||||||
|
)
|
||||||
|
predict_after_small = client.post(
|
||||||
|
"/predict", json={"sensor": SENSORS[0], "horizon": "7d", "scenario": "base"}
|
||||||
|
)
|
||||||
|
runner.check(
|
||||||
|
predict_after_small.status_code == 200
|
||||||
|
and len(predict_after_small.json()["points"]) == 168,
|
||||||
|
"/predict still works after the small swap",
|
||||||
|
)
|
||||||
|
report["jobs"] = {
|
||||||
|
"small": {
|
||||||
|
"job": small_job,
|
||||||
|
"wall_seconds": wall,
|
||||||
|
"train_seconds": train_seconds,
|
||||||
|
"input": subset_info,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
print("Forecast model: full CSV, hot-swap, 409 while busy")
|
||||||
|
forecast_started = time.monotonic()
|
||||||
|
forecast_response = _submit(client, headers, raw, "forecast")
|
||||||
|
if forecast_response.status_code != 202:
|
||||||
|
raise CheckFailed(
|
||||||
|
f"/train forecast -> {forecast_response.status_code}: {forecast_response.text}"
|
||||||
|
)
|
||||||
|
forecast_submitted = forecast_response.json()
|
||||||
|
runner.check(
|
||||||
|
forecast_submitted["model"] == "forecast",
|
||||||
|
f"/train forecast queued as job {forecast_submitted['job_id']}",
|
||||||
|
)
|
||||||
|
_wait_job(client, forecast_submitted["job_id"], headers, state="running", timeout=30)
|
||||||
|
busy = _submit(client, headers, subset_path, "small")
|
||||||
|
runner.check(
|
||||||
|
busy.status_code == 409 and busy.json()["error"]["code"] == "conflict",
|
||||||
|
"second /train while forecast is running -> 409 conflict",
|
||||||
|
)
|
||||||
|
forecast_job = _wait_job(client, forecast_submitted["job_id"], headers, timeout=300)
|
||||||
|
forecast_wall = round(time.monotonic() - forecast_started, 3)
|
||||||
|
if forecast_job["status"] != "succeeded":
|
||||||
|
raise CheckFailed(f"forecast job failed: {forecast_job}")
|
||||||
|
forecast_metrics = forecast_job["metrics"]
|
||||||
|
for key in ("mae", "wape", "r2", "baseline_mae", "n_eval"):
|
||||||
|
if key not in forecast_metrics:
|
||||||
|
raise CheckFailed(f"forecast metrics missing {key}: {forecast_metrics}")
|
||||||
|
runner.check(
|
||||||
|
forecast_job["model_version"].startswith("histgb-"),
|
||||||
|
f"forecast model_version {forecast_job['model_version']}",
|
||||||
|
)
|
||||||
|
runner.check(
|
||||||
|
forecast_metrics["mae"] < forecast_metrics["baseline_mae"],
|
||||||
|
f"retrained forecast MAE {forecast_metrics['mae']} < baseline {forecast_metrics['baseline_mae']}",
|
||||||
|
)
|
||||||
|
runner.check(
|
||||||
|
(models_dir / "forecast_latest.joblib").exists(),
|
||||||
|
"forecast model persisted to PULSECAST_MODELS",
|
||||||
|
)
|
||||||
|
meta_after_forecast = client.get("/meta").json()
|
||||||
|
runner.check(
|
||||||
|
meta_after_forecast["model_version"] == forecast_job["model_version"]
|
||||||
|
and meta_after_forecast["model_source"] == "retrained",
|
||||||
|
"forecast model hot-swapped into /meta as retrained",
|
||||||
|
)
|
||||||
|
predict_after_forecast = client.post(
|
||||||
|
"/predict", json={"sensor": SENSORS[-1], "horizon": "48h", "scenario": "rainy-saturday"}
|
||||||
|
)
|
||||||
|
runner.check(
|
||||||
|
predict_after_forecast.status_code == 200
|
||||||
|
and predict_after_forecast.json()["model_version"] == forecast_job["model_version"],
|
||||||
|
"/predict uses the retrained forecast model",
|
||||||
|
)
|
||||||
|
runner.check(client.get("/readyz").status_code == 200, "/readyz stays 200 after hot-swap")
|
||||||
|
report["jobs"]["forecast"] = {
|
||||||
|
"job": forecast_job,
|
||||||
|
"wall_seconds": forecast_wall,
|
||||||
|
"input": {"rows": "full challenge CSV", "bytes": raw.stat().st_size},
|
||||||
|
}
|
||||||
|
|
||||||
|
print("Malformed CSV fails the job (not the process)")
|
||||||
|
bad_csv = _submit(client, headers, garbage_path, "small")
|
||||||
|
runner.check(bad_csv.status_code == 202, "malformed CSV is accepted as a job")
|
||||||
|
bad_job = _wait_job(client, bad_csv.json()["job_id"], headers)
|
||||||
|
runner.check(
|
||||||
|
bad_job["status"] == "failed" and bool(bad_job["error"]),
|
||||||
|
f"malformed CSV job failed with {bad_job['error']!r}",
|
||||||
|
)
|
||||||
|
runner.check(
|
||||||
|
client.get("/healthz").status_code == 200,
|
||||||
|
"/healthz still 200 after a failed job",
|
||||||
|
)
|
||||||
|
|
||||||
|
report["versions_after_small"] = _versions(meta_after_small)
|
||||||
|
report["versions_after_forecast"] = _versions(meta_after_forecast)
|
||||||
|
report["checks"] = runner.checks
|
||||||
|
report["ok"] = True
|
||||||
|
finally:
|
||||||
|
process.terminate()
|
||||||
|
try:
|
||||||
|
process.wait(timeout=10)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
process.kill()
|
||||||
|
process.wait(timeout=5)
|
||||||
|
log_handle.close()
|
||||||
|
if process.returncode not in (0, -15):
|
||||||
|
print(f"server exited with {process.returncode}; log: {log_path}", file=sys.stderr)
|
||||||
|
print(log_path.read_text(errors="replace")[-4000:], file=sys.stderr)
|
||||||
|
report["server_log_tail"] = log_path.read_text(errors="replace")[-1000:]
|
||||||
|
if args.keep:
|
||||||
|
print(f"kept workdir {workdir}")
|
||||||
|
else:
|
||||||
|
shutil.rmtree(workdir, ignore_errors=True)
|
||||||
|
|
||||||
|
report["finished_at"] = _utc_now()
|
||||||
|
return report
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str] | None = None) -> int:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||||
|
parser.add_argument("--port", type=int, default=18000)
|
||||||
|
parser.add_argument("--token", default=os.environ.get("CONTROL_TOKEN") or "e2e-local-token")
|
||||||
|
parser.add_argument("--csv", default=str(config.raw_csv_path()))
|
||||||
|
parser.add_argument("--days", type=int, default=60, help="days in the small-model CSV subset")
|
||||||
|
parser.add_argument("--report", type=Path, help="write the JSON result here")
|
||||||
|
parser.add_argument("--keep", action="store_true", help="keep the temporary workdir")
|
||||||
|
args = parser.parse_args(argv)
|
||||||
|
|
||||||
|
runner = Runner()
|
||||||
|
print(f"PulseCast end-to-end check on port {args.port}")
|
||||||
|
started = time.monotonic()
|
||||||
|
try:
|
||||||
|
report = run(args, runner)
|
||||||
|
except CheckFailed as exc:
|
||||||
|
print(f"\nFAIL after {runner.checks} checks: {exc}", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
elapsed = round(time.monotonic() - started, 1)
|
||||||
|
report["elapsed_seconds"] = elapsed
|
||||||
|
print(f"\nPASS: {runner.checks} checks in {elapsed}s")
|
||||||
|
if args.report:
|
||||||
|
args.report.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
args.report.write_text(json.dumps(report, indent=2, ensure_ascii=False) + "\n")
|
||||||
|
print(f"report: {args.report}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,262 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Render dependency-free SVG figures from ``artifacts/metrics.json``.
|
||||||
|
|
||||||
|
The paper figures are generated from the measured holdout metrics, never
|
||||||
|
hand-written. Run ``make figures`` after ``make train``.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
from xml.sax.saxutils import escape
|
||||||
|
|
||||||
|
from pulsecast import config
|
||||||
|
|
||||||
|
MODEL_COLOR = "#2563eb"
|
||||||
|
BASELINE_COLOR = "#94a3b8"
|
||||||
|
PRODUCTION_COLOR = "#0f766e"
|
||||||
|
TEXT_COLOR = "#0f172a"
|
||||||
|
GRID_COLOR = "#e2e8f0"
|
||||||
|
|
||||||
|
|
||||||
|
def _svg_document(
|
||||||
|
title: str,
|
||||||
|
subtitle: str,
|
||||||
|
body: list[str],
|
||||||
|
width: int,
|
||||||
|
height: int,
|
||||||
|
footer: str = "Source: artifacts/metrics.json · PulseCast 28-day time-based holdout",
|
||||||
|
) -> str:
|
||||||
|
return "\n".join(
|
||||||
|
[
|
||||||
|
f'<svg xmlns="http://www.w3.org/2000/svg" width="{width}" height="{height}" '
|
||||||
|
f'viewBox="0 0 {width} {height}" role="img" aria-label="{escape(title)}">',
|
||||||
|
'<rect width="100%" height="100%" fill="#ffffff"/>',
|
||||||
|
'<g font-family="-apple-system, Segoe UI, Helvetica, Arial, sans-serif">',
|
||||||
|
f'<text x="24" y="34" font-size="20" font-weight="600" fill="{TEXT_COLOR}">'
|
||||||
|
f"{escape(title)}</text>",
|
||||||
|
f'<text x="24" y="56" font-size="13" fill="#475569">{escape(subtitle)}</text>',
|
||||||
|
*body,
|
||||||
|
f'<text x="24" y="{height - 12}" font-size="11" fill="#64748b">'
|
||||||
|
f"{escape(footer)}</text>",
|
||||||
|
"</g>",
|
||||||
|
"</svg>",
|
||||||
|
"",
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _bar_chart(
|
||||||
|
title: str,
|
||||||
|
subtitle: str,
|
||||||
|
rows: list[dict],
|
||||||
|
*,
|
||||||
|
value_key: str = "mae",
|
||||||
|
secondary_key: str | None = None,
|
||||||
|
secondary_label: str | None = None,
|
||||||
|
primary_label: str = "MAE",
|
||||||
|
production: str | None = None,
|
||||||
|
) -> str:
|
||||||
|
width = 960
|
||||||
|
left = 230
|
||||||
|
right = 96
|
||||||
|
top = 86
|
||||||
|
row_height = 38
|
||||||
|
height = top + row_height * len(rows) + 56
|
||||||
|
plot_width = width - left - right
|
||||||
|
values = [row[value_key] for row in rows]
|
||||||
|
if secondary_key:
|
||||||
|
values += [row[secondary_key] for row in rows]
|
||||||
|
maximum = max(values) * 1.08 if values else 1.0
|
||||||
|
|
||||||
|
body: list[str] = []
|
||||||
|
for tick in range(5):
|
||||||
|
x = left + plot_width * tick / 4
|
||||||
|
value = maximum * tick / 4
|
||||||
|
body.append(
|
||||||
|
f'<line x1="{x:.1f}" y1="{top - 18}" x2="{x:.1f}" y2="{top + row_height * len(rows)}" '
|
||||||
|
f'stroke="{GRID_COLOR}" stroke-width="1"/>'
|
||||||
|
)
|
||||||
|
body.append(
|
||||||
|
f'<text x="{x:.1f}" y="{top - 24}" font-size="11" fill="#64748b" '
|
||||||
|
f'text-anchor="middle">{value:.1f}</text>'
|
||||||
|
)
|
||||||
|
|
||||||
|
legend_y = top - 34
|
||||||
|
if secondary_key:
|
||||||
|
body.append(
|
||||||
|
f'<rect x="{left}" y="{legend_y}" width="10" height="10" fill="{BASELINE_COLOR}"/>'
|
||||||
|
f'<text x="{left + 16}" y="{legend_y + 9}" font-size="12" fill="#475569">'
|
||||||
|
f"{escape(secondary_label or 'baseline')}</text>"
|
||||||
|
)
|
||||||
|
body.append(
|
||||||
|
f'<rect x="{left + 140}" y="{legend_y}" width="10" height="10" fill="{MODEL_COLOR}"/>'
|
||||||
|
f'<text x="{left + 156}" y="{legend_y + 9}" font-size="12" fill="#475569">'
|
||||||
|
f"{escape(primary_label)}</text>"
|
||||||
|
)
|
||||||
|
|
||||||
|
for index, row in enumerate(rows):
|
||||||
|
y = top + index * row_height
|
||||||
|
label = str(row.get("name") or row.get("sensor"))
|
||||||
|
body.append(
|
||||||
|
f'<text x="{left - 12}" y="{y + 20}" font-size="12" fill="{TEXT_COLOR}" '
|
||||||
|
f'text-anchor="end">{escape(label)}</text>'
|
||||||
|
)
|
||||||
|
if secondary_key:
|
||||||
|
baseline_width = plot_width * row[secondary_key] / maximum
|
||||||
|
body.append(
|
||||||
|
f'<rect x="{left}" y="{y + 3}" width="{baseline_width:.1f}" height="12" '
|
||||||
|
f'fill="{BASELINE_COLOR}"/>'
|
||||||
|
)
|
||||||
|
body.append(
|
||||||
|
f'<text x="{left + baseline_width + 6:.1f}" y="{y + 13}" font-size="11" '
|
||||||
|
f'fill="#64748b">{row[secondary_key]:.2f}</text>'
|
||||||
|
)
|
||||||
|
model_width = plot_width * row[value_key] / maximum
|
||||||
|
body.append(
|
||||||
|
f'<rect x="{left}" y="{y + 18}" width="{model_width:.1f}" height="12" '
|
||||||
|
f'fill="{MODEL_COLOR}"/>'
|
||||||
|
)
|
||||||
|
body.append(
|
||||||
|
f'<text x="{left + model_width + 6:.1f}" y="{y + 28}" font-size="11" '
|
||||||
|
f'fill="{TEXT_COLOR}">{row[value_key]:.2f}</text>'
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
color = PRODUCTION_COLOR if production and row.get("name") == production else MODEL_COLOR
|
||||||
|
bar_width = plot_width * row[value_key] / maximum
|
||||||
|
body.append(
|
||||||
|
f'<rect x="{left}" y="{y + 6}" width="{bar_width:.1f}" height="22" fill="{color}"/>'
|
||||||
|
)
|
||||||
|
body.append(
|
||||||
|
f'<text x="{left + bar_width + 6:.1f}" y="{y + 21}" font-size="12" '
|
||||||
|
f'fill="{TEXT_COLOR}">{row[value_key]:.2f}</text>'
|
||||||
|
)
|
||||||
|
|
||||||
|
return _svg_document(title, subtitle, body, width, height)
|
||||||
|
|
||||||
|
|
||||||
|
def _confusion_svg(small: dict) -> str:
|
||||||
|
matrix = small["confusion_matrix"]
|
||||||
|
metrics = small["metrics"]
|
||||||
|
width, height = 760, 520
|
||||||
|
left, top, cell = 200, 130, 190
|
||||||
|
values = {
|
||||||
|
(0, 0): ("TN", matrix["tn"]),
|
||||||
|
(0, 1): ("FP", matrix["fp"]),
|
||||||
|
(1, 0): ("FN", matrix["fn"]),
|
||||||
|
(1, 1): ("TP", matrix["tp"]),
|
||||||
|
}
|
||||||
|
maximum = max(value for _, value in values.values())
|
||||||
|
body: list[str] = []
|
||||||
|
body.append(
|
||||||
|
f'<text x="{left + cell / 2}" y="{top - 16}" font-size="13" fill="#475569" '
|
||||||
|
f'text-anchor="middle">Predicted not busy</text>'
|
||||||
|
)
|
||||||
|
body.append(
|
||||||
|
f'<text x="{left + cell + cell / 2}" y="{top - 16}" font-size="13" fill="#475569" '
|
||||||
|
f'text-anchor="middle">Predicted busy</text>'
|
||||||
|
)
|
||||||
|
row_labels = ("Actually\nnot busy", "Actually\nbusy")
|
||||||
|
for row, label in enumerate(row_labels):
|
||||||
|
y = top + row * cell
|
||||||
|
body.append(
|
||||||
|
f'<text x="{left - 18}" y="{y + cell / 2}" font-size="13" fill="#475569" '
|
||||||
|
f'text-anchor="end">{escape(label).replace(chr(10), " ")}</text>'
|
||||||
|
)
|
||||||
|
for (row, column), (code, value) in values.items():
|
||||||
|
x = left + column * cell
|
||||||
|
y = top + row * cell
|
||||||
|
intensity = 0.12 + 0.88 * (value / maximum)
|
||||||
|
red = round(219 + (29 - 219) * intensity)
|
||||||
|
green = round(234 + (78 - 234) * intensity)
|
||||||
|
blue = round(254 + (216 - 254) * intensity)
|
||||||
|
fill = f"#{red:02x}{green:02x}{blue:02x}"
|
||||||
|
text_color = "#ffffff" if intensity > 0.55 else TEXT_COLOR
|
||||||
|
row_total = matrix["tn"] + matrix["fp"] if row == 0 else matrix["fn"] + matrix["tp"]
|
||||||
|
share = 100.0 * value / row_total if row_total else 0.0
|
||||||
|
body.append(
|
||||||
|
f'<rect x="{x}" y="{y}" width="{cell}" height="{cell}" fill="{fill}" '
|
||||||
|
f'stroke="#ffffff" stroke-width="2"/>'
|
||||||
|
)
|
||||||
|
body.append(
|
||||||
|
f'<text x="{x + cell / 2}" y="{y + cell / 2 - 8}" font-size="30" '
|
||||||
|
f'font-weight="600" fill="{text_color}" text-anchor="middle">{value:,}</text>'
|
||||||
|
)
|
||||||
|
body.append(
|
||||||
|
f'<text x="{x + cell / 2}" y="{y + cell / 2 + 20}" font-size="13" '
|
||||||
|
f'fill="{text_color}" text-anchor="middle">{code} · {share:.1f}% of row</text>'
|
||||||
|
)
|
||||||
|
subtitle = (
|
||||||
|
f"n={metrics['n_test']:,} · accuracy {metrics['accuracy']:.3f} · "
|
||||||
|
f"precision {metrics['precision']:.3f} · recall {metrics['recall']:.3f} · "
|
||||||
|
f"F1 {metrics['f1']:.3f} · ROC-AUC {metrics['roc_auc']:.3f}"
|
||||||
|
)
|
||||||
|
return _svg_document(
|
||||||
|
"Small busy-hour classifier: holdout confusion matrix",
|
||||||
|
subtitle,
|
||||||
|
body,
|
||||||
|
width,
|
||||||
|
height,
|
||||||
|
footer="Source: artifacts/small_metrics.json · chronological 80/20 split",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str] | None = None) -> int:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument(
|
||||||
|
"--artifacts",
|
||||||
|
type=Path,
|
||||||
|
default=config.artifacts_dir(),
|
||||||
|
help="artifact directory containing metrics.json (default: %(default)s)",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--out",
|
||||||
|
type=Path,
|
||||||
|
default=Path(__file__).resolve().parents[1] / "docs" / "figures",
|
||||||
|
help="output directory for SVG files (default: %(default)s)",
|
||||||
|
)
|
||||||
|
args = parser.parse_args(argv)
|
||||||
|
|
||||||
|
metrics = json.loads((args.artifacts / "metrics.json").read_text())
|
||||||
|
holdout = f"Holdout {metrics['test_from']} .. {metrics['test_to']}, n={metrics['metrics']['n_eval']}"
|
||||||
|
args.out.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
ablation = _bar_chart(
|
||||||
|
"Ablation: holdout MAE by feature set",
|
||||||
|
holdout,
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"name": row["name"].replace("_", " "),
|
||||||
|
"mae": row["mae"],
|
||||||
|
}
|
||||||
|
for row in metrics["ablation"]
|
||||||
|
],
|
||||||
|
production="week weather",
|
||||||
|
)
|
||||||
|
ablation_path = args.out / "ablation_mae.svg"
|
||||||
|
ablation_path.write_text(ablation)
|
||||||
|
|
||||||
|
per_sensor = _bar_chart(
|
||||||
|
"Per-sensor holdout MAE: production model vs seasonal naive",
|
||||||
|
holdout,
|
||||||
|
metrics["per_sensor"],
|
||||||
|
secondary_key="baseline_mae",
|
||||||
|
secondary_label="seasonal naive",
|
||||||
|
primary_label="weather-7d",
|
||||||
|
)
|
||||||
|
per_sensor_path = args.out / "per_sensor_mae.svg"
|
||||||
|
per_sensor_path.write_text(per_sensor)
|
||||||
|
|
||||||
|
confusion_path = args.out / "small_model_confusion.svg"
|
||||||
|
confusion_path.write_text(_confusion_svg(metrics["small"]))
|
||||||
|
|
||||||
|
print(f"wrote {ablation_path}")
|
||||||
|
print(f"wrote {per_sensor_path}")
|
||||||
|
print(f"wrote {confusion_path}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,153 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Train and evaluate the small "busy hour" classifier from a CSV.
|
||||||
|
|
||||||
|
This is the repeatable artifact behind ``make small`` and the runtime
|
||||||
|
``model=small`` training job. It prints the headline metrics and per-sensor
|
||||||
|
table used in ``docs/small-model.md``; ``--save`` bakes the bundle and metrics
|
||||||
|
into ``artifacts/`` exactly like ``make train`` does.
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
uv run python scripts/train_small.py
|
||||||
|
uv run python scripts/train_small.py --csv new-data.csv --all
|
||||||
|
uv run python scripts/train_small.py --save --json
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import joblib
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from pulsecast import config
|
||||||
|
from pulsecast.small import load_hourly_csv, train_small_from_csv
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||||
|
parser.add_argument(
|
||||||
|
"csv_positional",
|
||||||
|
nargs="?",
|
||||||
|
metavar="CSV",
|
||||||
|
help=f"CSV to train from (default: {config.raw_csv_path()})",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--csv",
|
||||||
|
dest="csv_option",
|
||||||
|
help="CSV to train from; overrides the positional argument",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--start",
|
||||||
|
default=config.MULTI_SENSOR_START,
|
||||||
|
help=(
|
||||||
|
"drop rows before this UTC timestamp (default: %(default)s, the "
|
||||||
|
"8-sensor challenge window); ignored with --all"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--all",
|
||||||
|
action="store_true",
|
||||||
|
help="use every row in the CSV, including density-01's older history",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--save",
|
||||||
|
action="store_true",
|
||||||
|
help="write artifacts/small_model.joblib and artifacts/small_metrics.json",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--json",
|
||||||
|
action="store_true",
|
||||||
|
help="print the full metrics payload as JSON after the summary",
|
||||||
|
)
|
||||||
|
return parser.parse_args(argv)
|
||||||
|
|
||||||
|
|
||||||
|
def _confusion_lines(matrix: dict) -> list[str]:
|
||||||
|
tn, fp, fn, tp = matrix["tn"], matrix["fp"], matrix["fn"], matrix["tp"]
|
||||||
|
total = tn + fp + fn + tp
|
||||||
|
specificity = tn / (tn + fp) if tn + fp else 0.0
|
||||||
|
negative_predictive = tn / (tn + fn) if tn + fn else 0.0
|
||||||
|
return [
|
||||||
|
"Confusion matrix (rows = truth, columns = prediction)",
|
||||||
|
f" not busy: {tn:5d} {fp:5d}",
|
||||||
|
f" busy : {fn:5d} {tp:5d}",
|
||||||
|
f" n={total} specificity={specificity:.3f} NPV={negative_predictive:.3f}",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str] | None = None) -> int:
|
||||||
|
args = _parse_args(argv)
|
||||||
|
path = Path(args.csv_option or args.csv_positional or config.raw_csv_path())
|
||||||
|
if not path.exists():
|
||||||
|
print(f"CSV not found: {path}", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
start = None if args.all else pd.Timestamp(args.start)
|
||||||
|
started = time.perf_counter()
|
||||||
|
try:
|
||||||
|
bundle, payload = train_small_from_csv(path, start=start)
|
||||||
|
except (ValueError, FileNotFoundError) as exc:
|
||||||
|
print(f"training failed: {exc}", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
elapsed = time.perf_counter() - started
|
||||||
|
|
||||||
|
hourly = load_hourly_csv(path)
|
||||||
|
if start is not None:
|
||||||
|
hourly = hourly[hourly["t"] >= start]
|
||||||
|
metrics = payload["metrics"]
|
||||||
|
|
||||||
|
print(f"CSV {path}")
|
||||||
|
print(f"Rows {len(hourly):,} hourly rows, {metrics['sensors']} sensor(s)")
|
||||||
|
print(f"Window {payload['train_from']} .. {payload['test_to']}")
|
||||||
|
print(f"Split (time) train {payload['train_from']} .. {payload['train_to']}"
|
||||||
|
f" ({metrics['n_train']:,} rows)")
|
||||||
|
print(f" test {payload['test_from']} .. {payload['test_to']}"
|
||||||
|
f" ({metrics['n_test']:,} rows)")
|
||||||
|
print("Target busy = hourly density > sensor p75 (training window only)")
|
||||||
|
print(f"Features {', '.join(payload['features'])}")
|
||||||
|
print("Model standardized logistic regression (class_weight=balanced)")
|
||||||
|
print(f"Model version {payload['model_version']}")
|
||||||
|
print()
|
||||||
|
print("Metrics")
|
||||||
|
print(f" accuracy {metrics['accuracy']:.3f}")
|
||||||
|
print(f" precision {metrics['precision']:.3f}")
|
||||||
|
print(f" recall {metrics['recall']:.3f}")
|
||||||
|
print(f" f1 {metrics['f1']:.3f}")
|
||||||
|
print(f" roc_auc {metrics['roc_auc'] if metrics['roc_auc'] is not None else 'n/a'}")
|
||||||
|
print(f" base rate {metrics['positive_rate']:.3f}")
|
||||||
|
print()
|
||||||
|
print("\n".join(_confusion_lines(payload["confusion_matrix"])))
|
||||||
|
print()
|
||||||
|
print("Per sensor (holdout)")
|
||||||
|
print(f" {'sensor':<24} {'n':>6} {'pos':>6} {'acc':>6} {'prec':>6} {'rec':>6} {'f1':>6}")
|
||||||
|
for row in payload.get("per_sensor", []):
|
||||||
|
print(
|
||||||
|
f" {row['sensor']:<24} {row['n']:>6} {row['positive_rate']:>6.3f}"
|
||||||
|
f" {row['accuracy']:>6.3f} {row['precision']:>6.3f}"
|
||||||
|
f" {row['recall']:>6.3f} {row['f1']:>6.3f}"
|
||||||
|
)
|
||||||
|
print()
|
||||||
|
print(f"Fit time {elapsed:.2f} s (contract: < 5 s)")
|
||||||
|
|
||||||
|
if args.save:
|
||||||
|
artifacts = config.artifacts_dir()
|
||||||
|
artifacts.mkdir(parents=True, exist_ok=True)
|
||||||
|
joblib.dump(bundle, artifacts / "small_model.joblib")
|
||||||
|
(artifacts / "small_metrics.json").write_text(
|
||||||
|
json.dumps(payload, indent=2, ensure_ascii=False) + "\n"
|
||||||
|
)
|
||||||
|
print(f"Saved {artifacts / 'small_model.joblib'}")
|
||||||
|
print(f" {artifacts / 'small_metrics.json'}")
|
||||||
|
|
||||||
|
if args.json:
|
||||||
|
print()
|
||||||
|
print(json.dumps(payload, indent=2, ensure_ascii=False))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
"""Configuration: paths, sensors, holidays, scenarios, contract versions.
|
"""Configuration: paths, sensors, holidays, features, scenarios.
|
||||||
|
|
||||||
All timestamps are UTC internally; Europe/Berlin is used only for calendar
|
All timestamps are UTC internally; Europe/Berlin is used only for calendar
|
||||||
features and display-oriented payloads.
|
features and display-oriented payloads.
|
||||||
@@ -65,8 +65,8 @@ def control_token() -> str:
|
|||||||
|
|
||||||
# --------------------------------------------------------------------------- data
|
# --------------------------------------------------------------------------- data
|
||||||
|
|
||||||
# Multi-sensor window documented in mission/data.md. density-01 has older
|
# Multi-sensor window of the published dataset. density-01 has older history,
|
||||||
# history but the other seven locations start on 2026-06-12.
|
# but the other seven locations start on 2026-06-12.
|
||||||
MULTI_SENSOR_START = "2026-06-12T00:00:00Z"
|
MULTI_SENSOR_START = "2026-06-12T00:00:00Z"
|
||||||
|
|
||||||
# 28-day time-based holdout, measured back from the last observed hour.
|
# 28-day time-based holdout, measured back from the last observed hour.
|
||||||
@@ -134,8 +134,8 @@ WEATHER_COLUMNS = {
|
|||||||
"relative_humidity_2m": "humidity_pct",
|
"relative_humidity_2m": "humidity_pct",
|
||||||
}
|
}
|
||||||
|
|
||||||
# Baden-Württemberg public holidays, 2025-2026 (mission/data.md). Kept explicit
|
# Baden-Württemberg public holidays, 2025-2026. Kept explicit so the build is
|
||||||
# so the build is deterministic and needs no holiday library.
|
# deterministic and needs no holiday library.
|
||||||
HOLIDAYS: dict[str, str] = {
|
HOLIDAYS: dict[str, str] = {
|
||||||
"2025-01-01": "Neujahr",
|
"2025-01-01": "Neujahr",
|
||||||
"2025-01-06": "Heilige Drei Könige",
|
"2025-01-06": "Heilige Drei Könige",
|
||||||
@@ -185,7 +185,6 @@ FEATURE_WEATHER = ["temp_c", "precip_mm"]
|
|||||||
FEATURE_DAY_NO_HOLIDAY = FEATURE_CALENDAR_NO_HOLIDAY + FEATURE_DAY[6:]
|
FEATURE_DAY_NO_HOLIDAY = FEATURE_CALENDAR_NO_HOLIDAY + FEATURE_DAY[6:]
|
||||||
FEATURE_DAY_NO_WEATHER = FEATURE_DAY
|
FEATURE_DAY_NO_WEATHER = FEATURE_DAY
|
||||||
FEATURE_DAY_WEATHER = FEATURE_DAY + FEATURE_WEATHER
|
FEATURE_DAY_WEATHER = FEATURE_DAY + FEATURE_WEATHER
|
||||||
FEATURE_DAY_HOLIDAY = FEATURE_DAY + ["is_holiday"] # is_holiday already present; kept for clarity
|
|
||||||
FEATURE_WEEK_NO_WEATHER = FEATURE_WEEK
|
FEATURE_WEEK_NO_WEATHER = FEATURE_WEEK
|
||||||
FEATURE_WEEK_WEATHER = FEATURE_WEEK + FEATURE_WEATHER
|
FEATURE_WEEK_WEATHER = FEATURE_WEEK + FEATURE_WEATHER
|
||||||
|
|
||||||
|
|||||||
@@ -16,7 +16,7 @@ from . import config
|
|||||||
from .features import SENSOR_CODES, future_calendar
|
from .features import SENSOR_CODES, future_calendar
|
||||||
|
|
||||||
# Fixed, deliberately small model. No tuning beyond a short sanity pass: the
|
# Fixed, deliberately small model. No tuning beyond a short sanity pass: the
|
||||||
# 28-day holdout is the honest signal, not the training loss.
|
# 28-day holdout decides, not the training loss.
|
||||||
HISTGB_PARAMS = {
|
HISTGB_PARAMS = {
|
||||||
"loss": "squared_error",
|
"loss": "squared_error",
|
||||||
"learning_rate": 0.06,
|
"learning_rate": 0.06,
|
||||||
@@ -58,7 +58,7 @@ def metrics(y_true, y_pred) -> dict:
|
|||||||
|
|
||||||
|
|
||||||
def residual_quantiles(eval_frame: pd.DataFrame, pred: np.ndarray) -> dict:
|
def residual_quantiles(eval_frame: pd.DataFrame, pred: np.ndarray) -> dict:
|
||||||
"""80% prediction band offsets from the honest holdout residuals."""
|
"""80% prediction band offsets from the holdout residuals."""
|
||||||
residual = np.asarray(eval_frame["y"], dtype=float) - np.asarray(pred, dtype=float)
|
residual = np.asarray(eval_frame["y"], dtype=float) - np.asarray(pred, dtype=float)
|
||||||
return {
|
return {
|
||||||
"p10": round(float(np.quantile(residual, 0.10)), 4),
|
"p10": round(float(np.quantile(residual, 0.10)), 4),
|
||||||
|
|||||||
@@ -1,8 +1,9 @@
|
|||||||
"""Parse and clean the raw PAX CSV.
|
"""Parse and clean the raw PAX CSV.
|
||||||
|
|
||||||
Verifies the claims in mission/data.md: 192,464 rows, 12 device IDs, 8 usable
|
The audit checks the counts the pipeline depends on: 192,464 rows, 12 device
|
||||||
logical locations, three excluded test devices, and the density-01 merge.
|
IDs, 8 usable logical locations, three excluded test devices, and the
|
||||||
The raw file is never written to git; a cached parquet copy lives in data/cache/.
|
density-01 merge. The raw file is never written to git; a cached parquet copy
|
||||||
|
lives in data/cache/.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
"""FastAPI ML service: /healthz, /readyz, /meta, /bootstrap, /predict, /simulate, /train.
|
"""FastAPI ML service: /healthz, /readyz, /meta, /bootstrap, /predict, /simulate, /train.
|
||||||
|
|
||||||
Shapes follow CONTRACT.md. The service is internal: only Go exposes it publicly,
|
The service is internal: only Go exposes it publicly, and the training routes
|
||||||
and the training routes require a bearer control token.
|
require a bearer control token.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
@@ -79,10 +79,9 @@ def _load_runtime(runtime: Runtime) -> None:
|
|||||||
json.loads(meta_path.read_text()) if meta_path.exists() else None
|
json.loads(meta_path.read_text()) if meta_path.exists() else None
|
||||||
)
|
)
|
||||||
|
|
||||||
# Baked snapshot artifacts served by GET /bootstrap. Go caches this once at
|
# Go caches /bootstrap once at startup, so load it at boot and keep the
|
||||||
# startup, so loading them at boot keeps the endpoint a pure in-memory read.
|
# endpoint a pure in-memory read. Project to contract fields because the
|
||||||
# Project to the contract fields; artifacts carry extra audit fields
|
# artifacts carry extra audit fields (last_seen).
|
||||||
# (last_seen) that Go does not need.
|
|
||||||
sensors_path = artifacts / "sensors.json"
|
sensors_path = artifacts / "sensors.json"
|
||||||
if sensors_path.exists():
|
if sensors_path.exists():
|
||||||
sensors = json.loads(sensors_path.read_text())
|
sensors = json.loads(sensors_path.read_text())
|
||||||
@@ -653,6 +652,8 @@ def create_app() -> FastAPI:
|
|||||||
state = runtime()
|
state = runtime()
|
||||||
model, path = await _parse_train_request(request)
|
model, path = await _parse_train_request(request)
|
||||||
|
|
||||||
|
# Serialize check-and-start so two requests cannot both pass the
|
||||||
|
# "no active job" check.
|
||||||
with state.schedule_lock:
|
with state.schedule_lock:
|
||||||
with state.jobs_lock:
|
with state.jobs_lock:
|
||||||
if any(job["status"] in ACTIVE_STATUSES for job in state.jobs.values()):
|
if any(job["status"] in ACTIVE_STATUSES for job in state.jobs.values()):
|
||||||
|
|||||||
+33
-2
@@ -17,6 +17,7 @@ import pandas as pd
|
|||||||
from sklearn.linear_model import LogisticRegression
|
from sklearn.linear_model import LogisticRegression
|
||||||
from sklearn.metrics import (
|
from sklearn.metrics import (
|
||||||
accuracy_score,
|
accuracy_score,
|
||||||
|
confusion_matrix,
|
||||||
f1_score,
|
f1_score,
|
||||||
precision_score,
|
precision_score,
|
||||||
recall_score,
|
recall_score,
|
||||||
@@ -92,8 +93,8 @@ def _attach_weather(hourly: pd.DataFrame, archive: pd.DataFrame | None) -> pd.Da
|
|||||||
frame[column] = frame[column].where(frame[column].notna(), frame[archive_column])
|
frame[column] = frame[column].where(frame[column].notna(), frame[archive_column])
|
||||||
frame = frame.drop(columns=[archive_column])
|
frame = frame.drop(columns=[archive_column])
|
||||||
|
|
||||||
# Honest fallback: no weather available for the uploaded window. Constant
|
# No weather for the uploaded window: constant values keep the other
|
||||||
# values keep the other features useful; the job metrics show the effect.
|
# features usable, and the job metrics show the effect.
|
||||||
frame["temp_c"] = frame["temp_c"].fillna(14.0)
|
frame["temp_c"] = frame["temp_c"].fillna(14.0)
|
||||||
frame["precip_mm"] = frame["precip_mm"].fillna(0.0)
|
frame["precip_mm"] = frame["precip_mm"].fillna(0.0)
|
||||||
return frame
|
return frame
|
||||||
@@ -254,6 +255,28 @@ def train_small_from_csv(
|
|||||||
except ValueError:
|
except ValueError:
|
||||||
roc_auc = None
|
roc_auc = None
|
||||||
|
|
||||||
|
tn, fp, fn, tp = confusion_matrix(truth, predicted, labels=[0, 1]).ravel()
|
||||||
|
scored = test.assign(_predicted=predicted)
|
||||||
|
per_sensor = []
|
||||||
|
for sensor, group in scored.groupby("sensor", observed=True):
|
||||||
|
sensor_truth = group["busy"].to_numpy(dtype=int)
|
||||||
|
sensor_pred = group["_predicted"].to_numpy(dtype=int)
|
||||||
|
per_sensor.append(
|
||||||
|
{
|
||||||
|
"sensor": str(sensor),
|
||||||
|
"n": len(group),
|
||||||
|
"positive_rate": round(float(sensor_truth.mean()), 3),
|
||||||
|
"accuracy": round(float(accuracy_score(sensor_truth, sensor_pred)), 3),
|
||||||
|
"precision": round(
|
||||||
|
float(precision_score(sensor_truth, sensor_pred, zero_division=0)), 3
|
||||||
|
),
|
||||||
|
"recall": round(
|
||||||
|
float(recall_score(sensor_truth, sensor_pred, zero_division=0)), 3
|
||||||
|
),
|
||||||
|
"f1": round(float(f1_score(sensor_truth, sensor_pred, zero_division=0)), 3),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
trained_at = _utc_now()
|
trained_at = _utc_now()
|
||||||
model_version = f"logit-{trained_at[:10].replace('-', '')}-{trained_at[11:19].replace(':', '')}"
|
model_version = f"logit-{trained_at[:10].replace('-', '')}-{trained_at[11:19].replace(':', '')}"
|
||||||
metrics_payload = {
|
metrics_payload = {
|
||||||
@@ -278,6 +301,14 @@ def train_small_from_csv(
|
|||||||
"sensors": int(hourly["sensor"].nunique()),
|
"sensors": int(hourly["sensor"].nunique()),
|
||||||
}
|
}
|
||||||
),
|
),
|
||||||
|
"confusion_matrix": {
|
||||||
|
"labels": {"0": "not busy", "1": "busy"},
|
||||||
|
"tn": int(tn),
|
||||||
|
"fp": int(fp),
|
||||||
|
"fn": int(fn),
|
||||||
|
"tp": int(tp),
|
||||||
|
},
|
||||||
|
"per_sensor": per_sensor,
|
||||||
}
|
}
|
||||||
bundle = {
|
bundle = {
|
||||||
"model": model,
|
"model": model,
|
||||||
|
|||||||
+43
-5
@@ -142,7 +142,35 @@ def train_bundle(
|
|||||||
baseline = by_name["seasonal_naive"]
|
baseline = by_name["seasonal_naive"]
|
||||||
headline = by_name["week_weather"]
|
headline = by_name["week_weather"]
|
||||||
|
|
||||||
# Production model: refit on all observed rows after the honest holdout.
|
# Per-sensor holdout breakdown for the paper; same rows as the headline.
|
||||||
|
scored = eval_frame.copy()
|
||||||
|
scored["_model"] = preds["week_weather"]
|
||||||
|
scored["_baseline"] = preds["seasonal_naive"]
|
||||||
|
per_sensor = []
|
||||||
|
for sensor, group in scored.groupby("sensor", observed=True):
|
||||||
|
model_row = metrics(group["y"], group["_model"])
|
||||||
|
baseline_row = metrics(group["y"], group["_baseline"])
|
||||||
|
improvement = (
|
||||||
|
100.0 * (baseline_row["mae"] - model_row["mae"]) / baseline_row["mae"]
|
||||||
|
if baseline_row["mae"]
|
||||||
|
else 0.0
|
||||||
|
)
|
||||||
|
per_sensor.append(
|
||||||
|
{
|
||||||
|
"sensor": str(sensor),
|
||||||
|
"name": config.DISPLAY_NAMES.get(str(sensor), str(sensor)),
|
||||||
|
"n": model_row["n"],
|
||||||
|
"mae": round(model_row["mae"], 2),
|
||||||
|
"wape": round(model_row["wape"], 1),
|
||||||
|
"r2": round(model_row["r2"], 3),
|
||||||
|
"baseline_mae": round(baseline_row["mae"], 2),
|
||||||
|
"baseline_wape": round(baseline_row["wape"], 1),
|
||||||
|
"baseline_r2": round(baseline_row["r2"], 3),
|
||||||
|
"mae_improvement_pct": round(improvement, 1),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
# Production model: refit on all observed rows after the holdout is scored.
|
||||||
production = fit(features, config.FEATURE_WEEK_WEATHER)
|
production = fit(features, config.FEATURE_WEEK_WEATHER)
|
||||||
residual = residual_quantiles(eval_frame, preds["week_weather"])
|
residual = residual_quantiles(eval_frame, preds["week_weather"])
|
||||||
sensor_mean = hourly.groupby("sensor", observed=True)["v"].mean().round(4).to_dict()
|
sensor_mean = hourly.groupby("sensor", observed=True)["v"].mean().round(4).to_dict()
|
||||||
@@ -207,6 +235,7 @@ def train_bundle(
|
|||||||
"features": list(config.FEATURE_WEEK_WEATHER),
|
"features": list(config.FEATURE_WEEK_WEATHER),
|
||||||
"holiday_effect": holiday_effect,
|
"holiday_effect": holiday_effect,
|
||||||
"ablation": [_round_row(row) for row in results],
|
"ablation": [_round_row(row) for row in results],
|
||||||
|
"per_sensor": per_sensor,
|
||||||
"limits": list(config.LIMITS),
|
"limits": list(config.LIMITS),
|
||||||
}
|
}
|
||||||
return bundle, metrics_payload, results
|
return bundle, metrics_payload, results
|
||||||
@@ -267,10 +296,6 @@ def run_train() -> dict:
|
|||||||
bundle, metrics_payload, _ = train_bundle(hourly, features, archive, meta, audit)
|
bundle, metrics_payload, _ = train_bundle(hourly, features, archive, meta, audit)
|
||||||
joblib.dump(bundle, artifacts / "model.joblib")
|
joblib.dump(bundle, artifacts / "model.joblib")
|
||||||
|
|
||||||
(artifacts / "metrics.json").write_text(
|
|
||||||
json.dumps(metrics_payload, indent=2, ensure_ascii=False) + "\n"
|
|
||||||
)
|
|
||||||
|
|
||||||
start = pd.Timestamp.now(tz="UTC").ceil("h")
|
start = pd.Timestamp.now(tz="UTC").ceil("h")
|
||||||
forecasts = _export_forecasts(bundle, forecast, start)
|
forecasts = _export_forecasts(bundle, forecast, start)
|
||||||
(artifacts / "forecasts.json").write_text(
|
(artifacts / "forecasts.json").write_text(
|
||||||
@@ -289,6 +314,19 @@ def run_train() -> dict:
|
|||||||
json.dumps(small_metrics, indent=2, ensure_ascii=False) + "\n"
|
json.dumps(small_metrics, indent=2, ensure_ascii=False) + "\n"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# One self-contained file for the paper: ablation + per-sensor + small model.
|
||||||
|
metrics_payload["small"] = {
|
||||||
|
"model_version": small_metrics["model_version"],
|
||||||
|
"trained_at": small_metrics["trained_at"],
|
||||||
|
"features": small_metrics["features"],
|
||||||
|
"metrics": small_metrics["metrics"],
|
||||||
|
"confusion_matrix": small_metrics["confusion_matrix"],
|
||||||
|
"per_sensor": small_metrics["per_sensor"],
|
||||||
|
}
|
||||||
|
(artifacts / "metrics.json").write_text(
|
||||||
|
json.dumps(metrics_payload, indent=2, ensure_ascii=False) + "\n"
|
||||||
|
)
|
||||||
|
|
||||||
log.info(
|
log.info(
|
||||||
"wrote metrics.json, forecasts.json (%s..%s), model.joblib, small_model.joblib",
|
"wrote metrics.json, forecasts.json (%s..%s), model.joblib, small_model.joblib",
|
||||||
start,
|
start,
|
||||||
|
|||||||
@@ -1,8 +1,8 @@
|
|||||||
"""Open-Meteo weather: historical archive for training, forecast for scenarios.
|
"""Open-Meteo weather: historical archive for training, forecast for scenarios.
|
||||||
|
|
||||||
No API key. Responses are cached under data/cache/ so `make train` and the
|
No API key. Responses are cached under data/cache/ so `make train` needs no
|
||||||
Docker image need no network. The raw CSV and cache stay out of git; the final
|
network after the first fetch. The raw CSV and cache stay out of git; compact
|
||||||
context.json is committed.
|
frames and context.json are committed under artifacts/.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
@@ -128,7 +128,7 @@ def ensure_weather(
|
|||||||
*,
|
*,
|
||||||
refresh: bool = False,
|
refresh: bool = False,
|
||||||
) -> tuple[pd.DataFrame, pd.DataFrame, dict]:
|
) -> tuple[pd.DataFrame, pd.DataFrame, dict]:
|
||||||
"""Load cached weather, fetching only when missing, stale, or refresh=True."""
|
"""Load cached weather, fetching only when the cache is missing or refresh=True."""
|
||||||
cache = config.cache_dir()
|
cache = config.cache_dir()
|
||||||
cache.mkdir(parents=True, exist_ok=True)
|
cache.mkdir(parents=True, exist_ok=True)
|
||||||
archive_path = cache / ARCHIVE_FILE
|
archive_path = cache / ARCHIVE_FILE
|
||||||
|
|||||||
@@ -67,6 +67,24 @@ def test_metrics_and_audit():
|
|||||||
assert metrics["metrics"]["wape"] < metrics["metrics"]["baseline_wape"]
|
assert metrics["metrics"]["wape"] < metrics["metrics"]["baseline_wape"]
|
||||||
assert len(metrics["features"]) == 10
|
assert len(metrics["features"]) == 10
|
||||||
assert len(metrics["limits"]) >= 4
|
assert len(metrics["limits"]) >= 4
|
||||||
|
assert len(metrics["per_sensor"]) == 8
|
||||||
|
assert sum(row["n"] for row in metrics["per_sensor"]) == metrics["metrics"]["n_eval"]
|
||||||
|
assert {row["sensor"] for row in metrics["per_sensor"]} == set(config.USABLE_SENSORS)
|
||||||
|
for row in metrics["per_sensor"]:
|
||||||
|
assert row["mae"] >= 0
|
||||||
|
assert row["baseline_mae"] >= 0
|
||||||
|
assert 0 <= row["wape"] < 100
|
||||||
|
assert row["mae_improvement_pct"] > 0
|
||||||
|
|
||||||
|
small = _load("small_metrics.json")
|
||||||
|
matrix = small["confusion_matrix"]
|
||||||
|
total = matrix["tn"] + matrix["fp"] + matrix["fn"] + matrix["tp"]
|
||||||
|
assert total == small["metrics"]["n_test"]
|
||||||
|
assert sum(row["n"] for row in small["per_sensor"]) == total
|
||||||
|
recomputed = (matrix["tn"] + matrix["tp"]) / total
|
||||||
|
assert abs(recomputed - small["metrics"]["accuracy"]) < 0.001
|
||||||
|
assert metrics["small"]["metrics"] == small["metrics"]
|
||||||
|
assert set(matrix) == {"labels", "tn", "fp", "fn", "tp"}
|
||||||
|
|
||||||
audit = _load("data_audit.json")
|
audit = _load("data_audit.json")
|
||||||
assert audit["all_checks_ok"] is True
|
assert audit["all_checks_ok"] is True
|
||||||
|
|||||||
Reference in new issue
Block a user