Files
Noel Sabia ac0d8984bb
ci / check (push) Successful in 28s
ci / check (pull_request) Successful in 30s
docs: correct reproduction and cache claims
The small-model page said make small reproduced every number, but the
threshold table and several derived values only appear in the JSON payload.
Say what the commands actually print, fix the stale weather-cache docstring,
and lead the e2e evidence page with what it checks.
2026-11-14 09:12:29 +01:00

120 lines
4.3 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Training end-to-end evidence
The script starts a real `uvicorn pulsecast.server:app` process with
`CONTROL_TOKEN`, waits for startup, and drives the HTTP API only. It checks the
snapshot, every prediction/scenario path, auth and error paths, one job at a
time, and both training jobs; it exits non-zero on the first failed check.
Unit/contract tests (`make check`) are separate.
Reproduce with:
```bash
uv run python scripts/e2e_train.py --port 18000 --report /tmp/e2e.json
```
## Run
| Field | Value |
|---|---|
| Date | 2026-11-14, 05:33 UTC |
| Branch | `feat/ml-final` |
| Port | **18000** |
| Checks | **38 passed, 0 failed** |
| Total wall time | 10.4 s |
| Forecast version before | `histgb-2026-11-14` |
| Data version | `2026-11-07T23:51:00Z` |
| Context version | `2026-11-14T03:47:09Z` |
## Snapshot and robustness (20 checks)
- `/healthz` → `{"ok":true}`; `/readyz` → 200 `model=true`.
- `/meta` covers 8 sensors and the baked model beats the seasonal-naive
baseline on the holdout.
- `/bootstrap` → 8 sensors, all 8 hourly series (23,952 points), 168 forecast
weather hours, 24 holidays, versions equal to `/meta`.
- `/predict`: **128/128** combinations (8 sensors × 4 horizons × 4 scenarios)
returned the contract point counts (`15m` 1, `1h` 4, `48h` 48, `7d` 168),
monotonic UTC timestamps, non-negative values, and `lo ≤ v ≤ hi`.
- `/simulate`: all 4 scenarios × 8 sensors × 4 horizons returned valid points.
- Weather override changes the forecast (`-8 °C / 6 mm` vs `30 °C / 0 mm`).
- Unknown sensor → 404; missing token → 401 `unauthorized`; wrong token → 401;
unknown job → 404 `not_found`; invalid model → 400 `bad_request`; missing
JSON path → 400.
## Small training job (real 60-day CSV subset)
Input: 36,846 density rows, 8 sensors, 2026-09-08T23:53Z → 2026-11-07T23:51Z.
| Field | Value |
|---|---|
| Job ID | `0b35347963974cbbbe584b77ecd4b846` |
| Status | `succeeded` |
| Model version | `logit-20261114-053344` |
| Wall time (submit → terminal) | **0.172 s** (contract < 5 s) |
| n_train / n_test | 8,151 / 2,301 |
| Base rate | 0.180 |
| Accuracy | 0.824 |
| Precision | 0.542 |
| Recall | 0.157 |
| F1 | 0.243 |
| ROC-AUC | 0.813 |
This is a different window from the baked full-window model in
`docs/small-model.md`, so the numbers are not directly comparable: the p75
target and the 80/20 split are recomputed inside the 60-day upload. The low
recall illustrates the limit noted in the small-model doc: a short, recent
window with a different threshold is a different classification problem.
Post-conditions verified:
- `small_latest.joblib` and `<job_id>-small.joblib` exist under
`PULSECAST_MODELS`.
- `/meta.small_model.model_version` equals the job version and its metrics
match the job exactly (in-memory hot-swap).
- `/predict` still returns 168 valid points after the swap.
## Forecast training job (full challenge CSV) and 409
Input: full `schwarz-pax.csv`, 19,383,568 bytes.
| Field | Value |
|---|---|
| Job ID | `235209a0d00f449ba8be717eb76386d6` |
| Status | `succeeded` |
| Model version | `histgb-20261114-053349` |
| Wall time | **4.380 s** |
| MAE | 3.32 |
| WAPE | 27.5% |
| R² | 0.777 |
| Baseline MAE / WAPE / R² | 5.40 / 44.7% / 0.462 |
| n_eval | 5,350 |
The ablation in the job payload matches `artifacts/metrics.json` row for row
(3.96, 3.84, 3.36, 4.08, 3.32). While this job was in `running`, a second
`POST /train` returned **409** `conflict`, proving one job at a time.
Post-conditions verified:
- `forecast_latest.joblib` and `<job_id>-forecast.joblib` exist under
`PULSECAST_MODELS`.
- `/meta.model_version` equals the retrained version and
`/meta.model_source` is `retrained`.
- `/predict` reports the retrained version; `/readyz` stays 200.
- `data_version` after the retrain is `2026-11-07T23:00:00Z` (the hourly input
maximum) rather than the raw-file floor `23:51:00Z`; both describe the same
observed window.
## Failure isolation
A malformed CSV (`not,a,pulsecast,file`) was accepted as a job (202), then
failed with a clear error:
```text
CSV needs a time column (time, timestamp, datetime, date, hour) and a value
column (value, density, footfall, count, pax, visitors); got ['not', 'a',
'pulsecast', 'file']
```
`/healthz` still returned 200 afterward: a bad upload cannot take down the
service or block `/predict`.