docs: small-model artifact, paper metrics, train e2e evidence
- docs/small-model.md: target, features, split, metrics, full confusion matrix, per-sensor table, honest limits - docs/metrics.md: provenance, protocol, headline, ablation, per-sensor, uncertainty, small model, supported/unsupported claims - docs/figures: generated ablation, per-sensor, confusion-matrix SVGs - docs/train-e2e.md: exact 38-check live run, small + forecast job results - model_card.md and README updated with final metrics and links
This commit is contained in:
1 parent
2bcf66c550
commit
c7ea524b5b
8 files changed
+702
-16
No files matched your search
@@ -37,7 +37,9 @@ Small busy-hour classifier (baked for the "train new data" demo):
|
||||
The classifier predicts `busy` = hourly footfall above the sensor's p75 using
|
||||
hour, weekday, weekend, temperature, rain, and holiday. It trains in under a
|
||||
second on the challenge CSV and is retrained from an uploaded CSV through
|
||||
`POST /train`.
|
||||
`POST /train`. Full definition, confusion matrix, per-sensor table, and honest
|
||||
limits: [`docs/small-model.md`](docs/small-model.md). Paper tables and figures:
|
||||
[`docs/metrics.md`](docs/metrics.md).
|
||||
|
||||
## Quick start
|
||||
|
||||
@@ -47,6 +49,8 @@ Requirements: Python 3.12 and `uv` (the repo also works with a plain venv).
|
||||
uv sync
|
||||
make data # needs the raw CSV, first run also fetches Open-Meteo weather
|
||||
make train # writes artifacts/model.joblib, metrics.json, forecasts.json
|
||||
make small # prints the busy-hour classifier metrics + confusion matrix
|
||||
make figures # regenerates docs/figures/*.svg from metrics.json
|
||||
make test
|
||||
make check # CI gate: sync --frozen, ruff, compileall, pytest
|
||||
make serve # http://localhost:8000
|
||||
@@ -69,7 +73,8 @@ committed so the API and Docker image work without the raw file or network.
|
||||
| `artifacts/sensors.json` | 8 sensors: id, name, lat/lon, mean, p95, battery |
|
||||
| `artifacts/series.json` + `series/<id>.json` | Hourly series per sensor |
|
||||
| `artifacts/forecasts.json` | 48h + 7d forecasts for 4 scenarios |
|
||||
| `artifacts/metrics.json` | Versions, coverage, headline metrics, ablation, limits |
|
||||
| `artifacts/metrics.json` | Versions, coverage, headline metrics, ablation, per-sensor, small summary, limits |
|
||||
| `artifacts/small_metrics.json` | Busy classifier: metrics, confusion matrix, per-sensor, thresholds |
|
||||
| `artifacts/context.json` | Current + 7-day weather, BW holidays |
|
||||
| `artifacts/data_audit.json` | Row/sensor checks against `mission/data.md` |
|
||||
| `artifacts/model.joblib` | Baked HistGB forecast bundle |
|
||||
@@ -118,6 +123,29 @@ and persists to `PULSECAST_MODELS` (default `/tmp/pulsecast/models`). A restart
|
||||
falls back to the baked model. `model=small` is the fast demo; `model=forecast`
|
||||
retrains the global HistGB (slower, still background).
|
||||
|
||||
## End-to-end check
|
||||
|
||||
`scripts/e2e_train.py` starts a real uvicorn process on port 18000 with a
|
||||
control token, then checks the snapshot, every `/predict` combination, every
|
||||
`/simulate` scenario, auth and error paths, one-job-at-a-time, and both real
|
||||
training jobs (small + forecast) including hot-swap and `/predict` afterward:
|
||||
|
||||
```bash
|
||||
uv run python scripts/e2e_train.py --port 18000 --report /tmp/e2e.json
|
||||
```
|
||||
|
||||
The unit and contract tests are `make test`; `make check` is the CI gate.
|
||||
|
||||
## Docs
|
||||
|
||||
| File | Contents |
|
||||
|---|---|
|
||||
| `docs/metrics.md` | Paper tables/figures: protocol, headline, ablation, per sensor, uncertainty, small model, claim limits |
|
||||
| `docs/small-model.md` | Busy-hour target, thresholds, split, metrics, confusion matrix, per sensor, limits |
|
||||
| `docs/train-e2e.md` | Exact live training-job evidence (small + forecast, hot-swap, errors) |
|
||||
| `docs/figures/*.svg` | Ablation MAE, per-sensor MAE, small-model confusion matrix (generated) |
|
||||
| `model_card.md` | Model card: features, split, baseline, ablation, leakage, limitations |
|
||||
|
||||
## Docker
|
||||
|
||||
```bash
|
||||
@@ -153,6 +181,8 @@ src/pulsecast/
|
||||
small.py busy classifier + flexible CSV parser
|
||||
retrain.py runtime forecast retraining from upload
|
||||
server.py FastAPI app and training job manager
|
||||
scripts/ train_small.py, plot_metrics.py, e2e_train.py
|
||||
docs/ metrics.md, small-model.md, generated figures
|
||||
tests/ schema, bootstrap, health, prediction, small-training job
|
||||
model_card.md features, split, baseline, ablation, limits, leakage
|
||||
```
|
||||
|
||||
Reference in new issue
Block a user