The small-model page said make small reproduced every number, but the threshold table and several derived values only appear in the JSON payload. Say what the commands actually print, fix the stale weather-cache docstring, and lead the e2e evidence page with what it checks.
4.3 KiB
Training end-to-end evidence
The script starts a real uvicorn pulsecast.server:app process with
CONTROL_TOKEN, waits for startup, and drives the HTTP API only. It checks the
snapshot, every prediction/scenario path, auth and error paths, one job at a
time, and both training jobs; it exits non-zero on the first failed check.
Unit/contract tests (make check) are separate.
Reproduce with:
uv run python scripts/e2e_train.py --port 18000 --report /tmp/e2e.json
Run
| Field | Value |
|---|---|
| Date | 2026-11-14, 05:33 UTC |
| Branch | feat/ml-final |
| Port | 18000 |
| Checks | 38 passed, 0 failed |
| Total wall time | 10.4 s |
| Forecast version before | histgb-2026-11-14 |
| Data version | 2026-11-07T23:51:00Z |
| Context version | 2026-11-14T03:47:09Z |
Snapshot and robustness (20 checks)
/healthz→{"ok":true};/readyz→ 200model=true./metacovers 8 sensors and the baked model beats the seasonal-naive baseline on the holdout./bootstrap→ 8 sensors, all 8 hourly series (23,952 points), 168 forecast weather hours, 24 holidays, versions equal to/meta./predict: 128/128 combinations (8 sensors × 4 horizons × 4 scenarios) returned the contract point counts (15m1,1h4,48h48,7d168), monotonic UTC timestamps, non-negative values, andlo ≤ v ≤ hi./simulate: all 4 scenarios × 8 sensors × 4 horizons returned valid points.- Weather override changes the forecast (
-8 °C / 6 mmvs30 °C / 0 mm). - Unknown sensor → 404; missing token → 401
unauthorized; wrong token → 401; unknown job → 404not_found; invalid model → 400bad_request; missing JSON path → 400.
Small training job (real 60-day CSV subset)
Input: 36,846 density rows, 8 sensors, 2026-09-08T23:53Z → 2026-11-07T23:51Z.
| Field | Value |
|---|---|
| Job ID | 0b35347963974cbbbe584b77ecd4b846 |
| Status | succeeded |
| Model version | logit-20261114-053344 |
| Wall time (submit → terminal) | 0.172 s (contract < 5 s) |
| n_train / n_test | 8,151 / 2,301 |
| Base rate | 0.180 |
| Accuracy | 0.824 |
| Precision | 0.542 |
| Recall | 0.157 |
| F1 | 0.243 |
| ROC-AUC | 0.813 |
This is a different window from the baked full-window model in
docs/small-model.md, so the numbers are not directly comparable: the p75
target and the 80/20 split are recomputed inside the 60-day upload. The low
recall illustrates the limit noted in the small-model doc: a short, recent
window with a different threshold is a different classification problem.
Post-conditions verified:
small_latest.jobliband<job_id>-small.joblibexist underPULSECAST_MODELS./meta.small_model.model_versionequals the job version and its metrics match the job exactly (in-memory hot-swap)./predictstill returns 168 valid points after the swap.
Forecast training job (full challenge CSV) and 409
Input: full schwarz-pax.csv, 19,383,568 bytes.
| Field | Value |
|---|---|
| Job ID | 235209a0d00f449ba8be717eb76386d6 |
| Status | succeeded |
| Model version | histgb-20261114-053349 |
| Wall time | 4.380 s |
| MAE | 3.32 |
| WAPE | 27.5% |
| R² | 0.777 |
| Baseline MAE / WAPE / R² | 5.40 / 44.7% / 0.462 |
| n_eval | 5,350 |
The ablation in the job payload matches artifacts/metrics.json row for row
(3.96, 3.84, 3.36, 4.08, 3.32). While this job was in running, a second
POST /train returned 409 conflict, proving one job at a time.
Post-conditions verified:
forecast_latest.jobliband<job_id>-forecast.joblibexist underPULSECAST_MODELS./meta.model_versionequals the retrained version and/meta.model_sourceisretrained./predictreports the retrained version;/readyzstays 200.data_versionafter the retrain is2026-11-07T23:00:00Z(the hourly input maximum) rather than the raw-file floor23:51:00Z; both describe the same observed window.
Failure isolation
A malformed CSV (not,a,pulsecast,file) was accepted as a job (202), then
failed with a clear error:
CSV needs a time column (time, timestamp, datetime, date, hour) and a value
column (value, density, footfall, count, pax, visitors); got ['not', 'a',
'pulsecast', 'file']
/healthz still returned 200 afterward: a bad upload cannot take down the
service or block /predict.