docs: small-model artifact, paper metrics, train e2e evidence

- docs/small-model.md: target, features, split, metrics, full confusion
  matrix, per-sensor table, honest limits
- docs/metrics.md: provenance, protocol, headline, ablation, per-sensor,
  uncertainty, small model, supported/unsupported claims
- docs/figures: generated ablation, per-sensor, confusion-matrix SVGs
- docs/train-e2e.md: exact 38-check live run, small + forecast job results
- model_card.md and README updated with final metrics and links
This commit is contained in:
Noel Sabia committed 2026-11-14 06:35:22 +01:00
1 parent 2bcf66c550
commit c7ea524b5b
8 files changed
+702 -16

No files matched your search

+32 -2
View File
@@ -37,7 +37,9 @@ Small busy-hour classifier (baked for the "train new data" demo):
The classifier predicts `busy` = hourly footfall above the sensor's p75 using
hour, weekday, weekend, temperature, rain, and holiday. It trains in under a
second on the challenge CSV and is retrained from an uploaded CSV through
`POST /train`.
`POST /train`. Full definition, confusion matrix, per-sensor table, and honest
limits: [`docs/small-model.md`](docs/small-model.md). Paper tables and figures:
[`docs/metrics.md`](docs/metrics.md).
## Quick start
@@ -47,6 +49,8 @@ Requirements: Python 3.12 and `uv` (the repo also works with a plain venv).
uv sync
make data # needs the raw CSV, first run also fetches Open-Meteo weather
make train # writes artifacts/model.joblib, metrics.json, forecasts.json
make small # prints the busy-hour classifier metrics + confusion matrix
make figures # regenerates docs/figures/*.svg from metrics.json
make test
make check # CI gate: sync --frozen, ruff, compileall, pytest
make serve # http://localhost:8000
@@ -69,7 +73,8 @@ committed so the API and Docker image work without the raw file or network.
| `artifacts/sensors.json` | 8 sensors: id, name, lat/lon, mean, p95, battery |
| `artifacts/series.json` + `series/<id>.json` | Hourly series per sensor |
| `artifacts/forecasts.json` | 48h + 7d forecasts for 4 scenarios |
| `artifacts/metrics.json` | Versions, coverage, headline metrics, ablation, limits |
| `artifacts/metrics.json` | Versions, coverage, headline metrics, ablation, per-sensor, small summary, limits |
| `artifacts/small_metrics.json` | Busy classifier: metrics, confusion matrix, per-sensor, thresholds |
| `artifacts/context.json` | Current + 7-day weather, BW holidays |
| `artifacts/data_audit.json` | Row/sensor checks against `mission/data.md` |
| `artifacts/model.joblib` | Baked HistGB forecast bundle |
@@ -118,6 +123,29 @@ and persists to `PULSECAST_MODELS` (default `/tmp/pulsecast/models`). A restart
falls back to the baked model. `model=small` is the fast demo; `model=forecast`
retrains the global HistGB (slower, still background).
## End-to-end check
`scripts/e2e_train.py` starts a real uvicorn process on port 18000 with a
control token, then checks the snapshot, every `/predict` combination, every
`/simulate` scenario, auth and error paths, one-job-at-a-time, and both real
training jobs (small + forecast) including hot-swap and `/predict` afterward:
```bash
uv run python scripts/e2e_train.py --port 18000 --report /tmp/e2e.json
```
The unit and contract tests are `make test`; `make check` is the CI gate.
## Docs
| File | Contents |
|---|---|
| `docs/metrics.md` | Paper tables/figures: protocol, headline, ablation, per sensor, uncertainty, small model, claim limits |
| `docs/small-model.md` | Busy-hour target, thresholds, split, metrics, confusion matrix, per sensor, limits |
| `docs/train-e2e.md` | Exact live training-job evidence (small + forecast, hot-swap, errors) |
| `docs/figures/*.svg` | Ablation MAE, per-sensor MAE, small-model confusion matrix (generated) |
| `model_card.md` | Model card: features, split, baseline, ablation, leakage, limitations |
## Docker
```bash
@@ -153,6 +181,8 @@ src/pulsecast/
small.py busy classifier + flexible CSV parser
retrain.py runtime forecast retraining from upload
server.py FastAPI app and training job manager
scripts/ train_small.py, plot_metrics.py, e2e_train.py
docs/ metrics.md, small-model.md, generated figures
tests/ schema, bootstrap, health, prediction, small-training job
model_card.md features, split, baseline, ablation, limits, leakage
```