The small-model page said make small reproduced every number, but the threshold table and several derived values only appear in the JSON payload. Say what the commands actually print, fix the stale weather-cache docstring, and lead the e2e evidence page with what it checks.
120 lines
4.3 KiB
Markdown
120 lines
4.3 KiB
Markdown
# Training end-to-end evidence
|
||
|
||
The script starts a real `uvicorn pulsecast.server:app` process with
|
||
`CONTROL_TOKEN`, waits for startup, and drives the HTTP API only. It checks the
|
||
snapshot, every prediction/scenario path, auth and error paths, one job at a
|
||
time, and both training jobs; it exits non-zero on the first failed check.
|
||
Unit/contract tests (`make check`) are separate.
|
||
|
||
Reproduce with:
|
||
|
||
```bash
|
||
uv run python scripts/e2e_train.py --port 18000 --report /tmp/e2e.json
|
||
```
|
||
|
||
## Run
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Date | 2026-11-14, 05:33 UTC |
|
||
| Branch | `feat/ml-final` |
|
||
| Port | **18000** |
|
||
| Checks | **38 passed, 0 failed** |
|
||
| Total wall time | 10.4 s |
|
||
| Forecast version before | `histgb-2026-11-14` |
|
||
| Data version | `2026-11-07T23:51:00Z` |
|
||
| Context version | `2026-11-14T03:47:09Z` |
|
||
|
||
## Snapshot and robustness (20 checks)
|
||
|
||
- `/healthz` → `{"ok":true}`; `/readyz` → 200 `model=true`.
|
||
- `/meta` covers 8 sensors and the baked model beats the seasonal-naive
|
||
baseline on the holdout.
|
||
- `/bootstrap` → 8 sensors, all 8 hourly series (23,952 points), 168 forecast
|
||
weather hours, 24 holidays, versions equal to `/meta`.
|
||
- `/predict`: **128/128** combinations (8 sensors × 4 horizons × 4 scenarios)
|
||
returned the contract point counts (`15m` 1, `1h` 4, `48h` 48, `7d` 168),
|
||
monotonic UTC timestamps, non-negative values, and `lo ≤ v ≤ hi`.
|
||
- `/simulate`: all 4 scenarios × 8 sensors × 4 horizons returned valid points.
|
||
- Weather override changes the forecast (`-8 °C / 6 mm` vs `30 °C / 0 mm`).
|
||
- Unknown sensor → 404; missing token → 401 `unauthorized`; wrong token → 401;
|
||
unknown job → 404 `not_found`; invalid model → 400 `bad_request`; missing
|
||
JSON path → 400.
|
||
|
||
## Small training job (real 60-day CSV subset)
|
||
|
||
Input: 36,846 density rows, 8 sensors, 2026-09-08T23:53Z → 2026-11-07T23:51Z.
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Job ID | `0b35347963974cbbbe584b77ecd4b846` |
|
||
| Status | `succeeded` |
|
||
| Model version | `logit-20261114-053344` |
|
||
| Wall time (submit → terminal) | **0.172 s** (contract < 5 s) |
|
||
| n_train / n_test | 8,151 / 2,301 |
|
||
| Base rate | 0.180 |
|
||
| Accuracy | 0.824 |
|
||
| Precision | 0.542 |
|
||
| Recall | 0.157 |
|
||
| F1 | 0.243 |
|
||
| ROC-AUC | 0.813 |
|
||
|
||
This is a different window from the baked full-window model in
|
||
`docs/small-model.md`, so the numbers are not directly comparable: the p75
|
||
target and the 80/20 split are recomputed inside the 60-day upload. The low
|
||
recall illustrates the limit noted in the small-model doc: a short, recent
|
||
window with a different threshold is a different classification problem.
|
||
|
||
Post-conditions verified:
|
||
|
||
- `small_latest.joblib` and `<job_id>-small.joblib` exist under
|
||
`PULSECAST_MODELS`.
|
||
- `/meta.small_model.model_version` equals the job version and its metrics
|
||
match the job exactly (in-memory hot-swap).
|
||
- `/predict` still returns 168 valid points after the swap.
|
||
|
||
## Forecast training job (full challenge CSV) and 409
|
||
|
||
Input: full `schwarz-pax.csv`, 19,383,568 bytes.
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Job ID | `235209a0d00f449ba8be717eb76386d6` |
|
||
| Status | `succeeded` |
|
||
| Model version | `histgb-20261114-053349` |
|
||
| Wall time | **4.380 s** |
|
||
| MAE | 3.32 |
|
||
| WAPE | 27.5% |
|
||
| R² | 0.777 |
|
||
| Baseline MAE / WAPE / R² | 5.40 / 44.7% / 0.462 |
|
||
| n_eval | 5,350 |
|
||
|
||
The ablation in the job payload matches `artifacts/metrics.json` row for row
|
||
(3.96, 3.84, 3.36, 4.08, 3.32). While this job was in `running`, a second
|
||
`POST /train` returned **409** `conflict`, proving one job at a time.
|
||
|
||
Post-conditions verified:
|
||
|
||
- `forecast_latest.joblib` and `<job_id>-forecast.joblib` exist under
|
||
`PULSECAST_MODELS`.
|
||
- `/meta.model_version` equals the retrained version and
|
||
`/meta.model_source` is `retrained`.
|
||
- `/predict` reports the retrained version; `/readyz` stays 200.
|
||
- `data_version` after the retrain is `2026-11-07T23:00:00Z` (the hourly input
|
||
maximum) rather than the raw-file floor `23:51:00Z`; both describe the same
|
||
observed window.
|
||
|
||
## Failure isolation
|
||
|
||
A malformed CSV (`not,a,pulsecast,file`) was accepted as a job (202), then
|
||
failed with a clear error:
|
||
|
||
```text
|
||
CSV needs a time column (time, timestamp, datetime, date, hour) and a value
|
||
column (value, density, footfall, count, pax, visitors); got ['not', 'a',
|
||
'pulsecast', 'file']
|
||
```
|
||
|
||
`/healthz` still returned 200 afterward: a bad upload cannot take down the
|
||
service or block `/predict`.
|