docs: small-model artifact, paper metrics, train e2e evidence

- docs/small-model.md: target, features, split, metrics, full confusion
  matrix, per-sensor table, honest limits
- docs/metrics.md: provenance, protocol, headline, ablation, per-sensor,
  uncertainty, small model, supported/unsupported claims
- docs/figures: generated ablation, per-sensor, confusion-matrix SVGs
- docs/train-e2e.md: exact 38-check live run, small + forecast job results
- model_card.md and README updated with final metrics and links
This commit is contained in:
Noel Sabia committed 2026-11-14 06:35:22 +01:00
1 parent 2bcf66c550
commit c7ea524b5b
8 files changed
+702 -16

No files matched your search

+118
View File
@@ -0,0 +1,118 @@
# Training end-to-end evidence
Reproduce with:
```bash
uv run python scripts/e2e_train.py --port 18000 --report /tmp/e2e.json
```
The script starts a real `uvicorn pulsecast.server:app` process with
`CONTROL_TOKEN`, waits for startup, and drives the HTTP API only. It exits
non-zero on the first failed check. Unit/contract tests (`make check`) are
separate.
## Run
| Field | Value |
|---|---|
| Date | 2026-11-14, 05:33 UTC |
| Branch | `feat/ml-final` |
| Port | **18000** |
| Checks | **38 passed, 0 failed** |
| Total wall time | 10.4 s |
| Forecast version before | `histgb-2026-11-14` |
| Data version | `2026-11-07T23:51:00Z` |
| Context version | `2026-11-14T03:47:09Z` |
## Snapshot and robustness (20 checks)
- `/healthz` → `{"ok":true}`; `/readyz` → 200 `model=true`.
- `/meta` covers 8 sensors and the baked model beats the seasonal-naive
baseline on the holdout.
- `/bootstrap` → 8 sensors, all 8 hourly series (23,952 points), 168 forecast
weather hours, 24 holidays, versions equal to `/meta`.
- `/predict`: **128/128** combinations (8 sensors × 4 horizons × 4 scenarios)
returned the contract point counts (`15m` 1, `1h` 4, `48h` 48, `7d` 168),
monotonic UTC timestamps, non-negative values, and `lo ≤ v ≤ hi`.
- `/simulate`: all 4 scenarios × 8 sensors × 4 horizons returned valid points.
- Weather override changes the forecast (`-8 °C / 6 mm` vs `30 °C / 0 mm`).
- Unknown sensor → 404; missing token → 401 `unauthorized`; wrong token → 401;
unknown job → 404 `not_found`; invalid model → 400 `bad_request`; missing
JSON path → 400.
## Small training job (real 60-day CSV subset)
Input: 36,846 density rows, 8 sensors, 2026-09-08T23:53Z → 2026-11-07T23:51Z.
| Field | Value |
|---|---|
| Job ID | `0b35347963974cbbbe584b77ecd4b846` |
| Status | `succeeded` |
| Model version | `logit-20261114-053344` |
| Wall time (submit → terminal) | **0.172 s** (contract < 5 s) |
| n_train / n_test | 8,151 / 2,301 |
| Base rate | 0.180 |
| Accuracy | 0.824 |
| Precision | 0.542 |
| Recall | 0.157 |
| F1 | 0.243 |
| ROC-AUC | 0.813 |
This is a different window from the baked full-window model in
`docs/small-model.md`, so the numbers are not directly comparable: the p75
target and the 80/20 split are recomputed inside the 60-day upload. The
low recall illustrates the honest limit noted in the small-model doc — a short,
recent window with a different threshold is a different classification problem.
Post-conditions verified:
- `small_latest.joblib` and `<job_id>-small.joblib` exist under
`PULSECAST_MODELS`.
- `/meta.small_model.model_version` equals the job version and its metrics
match the job exactly (in-memory hot-swap).
- `/predict` still returns 168 valid points after the swap.
## Forecast training job (full challenge CSV) and 409
Input: full `schwarz-pax.csv`, 19,383,568 bytes.
| Field | Value |
|---|---|
| Job ID | `235209a0d00f449ba8be717eb76386d6` |
| Status | `succeeded` |
| Model version | `histgb-20261114-053349` |
| Wall time | **4.380 s** |
| MAE | 3.32 |
| WAPE | 27.5% |
| R² | 0.777 |
| Baseline MAE / WAPE / R² | 5.40 / 44.7% / 0.462 |
| n_eval | 5,350 |
The ablation in the job payload matches `artifacts/metrics.json` row for row
(3.96, 3.84, 3.36, 4.08, 3.32). While this job was in `running`, a second
`POST /train` returned **409** `conflict`, proving one job at a time.
Post-conditions verified:
- `forecast_latest.joblib` and `<job_id>-forecast.joblib` exist under
`PULSECAST_MODELS`.
- `/meta.model_version` equals the retrained version and
`/meta.model_source` is `retrained`.
- `/predict` reports the retrained version; `/readyz` stays 200.
- `data_version` after the retrain is `2026-11-07T23:00:00Z` (the hourly input
maximum) rather than the raw-file floor `23:51:00Z`; both describe the same
observed window.
## Failure isolation
A malformed CSV (`not,a,pulsecast,file`) was accepted as a job (202), then
failed with a clear error:
```text
CSV needs a time column (time, timestamp, datetime, date, hour) and a value
column (value, density, footfall, count, pax, visitors); got ['not', 'a',
'pulsecast', 'file']
```
`/healthz` still returned 200 afterward: a bad upload cannot take down the
service or block `/predict`.