Files
ML-py/docs/train-e2e.md
T
Noel Sabia c7ea524b5b docs: small-model artifact, paper metrics, train e2e evidence
- docs/small-model.md: target, features, split, metrics, full confusion
  matrix, per-sensor table, honest limits
- docs/metrics.md: provenance, protocol, headline, ablation, per-sensor,
  uncertainty, small model, supported/unsupported claims
- docs/figures: generated ablation, per-sensor, confusion-matrix SVGs
- docs/train-e2e.md: exact 38-check live run, small + forecast job results
- model_card.md and README updated with final metrics and links
2026-11-14 06:35:22 +01:00

119 lines
4.2 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Training end-to-end evidence
Reproduce with:
```bash
uv run python scripts/e2e_train.py --port 18000 --report /tmp/e2e.json
```
The script starts a real `uvicorn pulsecast.server:app` process with
`CONTROL_TOKEN`, waits for startup, and drives the HTTP API only. It exits
non-zero on the first failed check. Unit/contract tests (`make check`) are
separate.
## Run
| Field | Value |
|---|---|
| Date | 2026-11-14, 05:33 UTC |
| Branch | `feat/ml-final` |
| Port | **18000** |
| Checks | **38 passed, 0 failed** |
| Total wall time | 10.4 s |
| Forecast version before | `histgb-2026-11-14` |
| Data version | `2026-11-07T23:51:00Z` |
| Context version | `2026-11-14T03:47:09Z` |
## Snapshot and robustness (20 checks)
- `/healthz` → `{"ok":true}`; `/readyz` → 200 `model=true`.
- `/meta` covers 8 sensors and the baked model beats the seasonal-naive
baseline on the holdout.
- `/bootstrap` → 8 sensors, all 8 hourly series (23,952 points), 168 forecast
weather hours, 24 holidays, versions equal to `/meta`.
- `/predict`: **128/128** combinations (8 sensors × 4 horizons × 4 scenarios)
returned the contract point counts (`15m` 1, `1h` 4, `48h` 48, `7d` 168),
monotonic UTC timestamps, non-negative values, and `lo ≤ v ≤ hi`.
- `/simulate`: all 4 scenarios × 8 sensors × 4 horizons returned valid points.
- Weather override changes the forecast (`-8 °C / 6 mm` vs `30 °C / 0 mm`).
- Unknown sensor → 404; missing token → 401 `unauthorized`; wrong token → 401;
unknown job → 404 `not_found`; invalid model → 400 `bad_request`; missing
JSON path → 400.
## Small training job (real 60-day CSV subset)
Input: 36,846 density rows, 8 sensors, 2026-09-08T23:53Z → 2026-11-07T23:51Z.
| Field | Value |
|---|---|
| Job ID | `0b35347963974cbbbe584b77ecd4b846` |
| Status | `succeeded` |
| Model version | `logit-20261114-053344` |
| Wall time (submit → terminal) | **0.172 s** (contract < 5 s) |
| n_train / n_test | 8,151 / 2,301 |
| Base rate | 0.180 |
| Accuracy | 0.824 |
| Precision | 0.542 |
| Recall | 0.157 |
| F1 | 0.243 |
| ROC-AUC | 0.813 |
This is a different window from the baked full-window model in
`docs/small-model.md`, so the numbers are not directly comparable: the p75
target and the 80/20 split are recomputed inside the 60-day upload. The
low recall illustrates the honest limit noted in the small-model doc — a short,
recent window with a different threshold is a different classification problem.
Post-conditions verified:
- `small_latest.joblib` and `<job_id>-small.joblib` exist under
`PULSECAST_MODELS`.
- `/meta.small_model.model_version` equals the job version and its metrics
match the job exactly (in-memory hot-swap).
- `/predict` still returns 168 valid points after the swap.
## Forecast training job (full challenge CSV) and 409
Input: full `schwarz-pax.csv`, 19,383,568 bytes.
| Field | Value |
|---|---|
| Job ID | `235209a0d00f449ba8be717eb76386d6` |
| Status | `succeeded` |
| Model version | `histgb-20261114-053349` |
| Wall time | **4.380 s** |
| MAE | 3.32 |
| WAPE | 27.5% |
| R² | 0.777 |
| Baseline MAE / WAPE / R² | 5.40 / 44.7% / 0.462 |
| n_eval | 5,350 |
The ablation in the job payload matches `artifacts/metrics.json` row for row
(3.96, 3.84, 3.36, 4.08, 3.32). While this job was in `running`, a second
`POST /train` returned **409** `conflict`, proving one job at a time.
Post-conditions verified:
- `forecast_latest.joblib` and `<job_id>-forecast.joblib` exist under
`PULSECAST_MODELS`.
- `/meta.model_version` equals the retrained version and
`/meta.model_source` is `retrained`.
- `/predict` reports the retrained version; `/readyz` stays 200.
- `data_version` after the retrain is `2026-11-07T23:00:00Z` (the hourly input
maximum) rather than the raw-file floor `23:51:00Z`; both describe the same
observed window.
## Failure isolation
A malformed CSV (`not,a,pulsecast,file`) was accepted as a job (202), then
failed with a clear error:
```text
CSV needs a time column (time, timestamp, datetime, date, hour) and a value
column (value, density, footfall, count, pax, visitors); got ['not', 'a',
'pulsecast', 'file']
```
`/healthz` still returned 200 afterward: a bad upload cannot take down the
service or block `/predict`.