Files
ML-py/docs/train-e2e.md
Noel Sabia ac0d8984bb
ci / check (push) Successful in 28s
ci / check (pull_request) Successful in 30s
docs: correct reproduction and cache claims
The small-model page said make small reproduced every number, but the
threshold table and several derived values only appear in the JSON payload.
Say what the commands actually print, fix the stale weather-cache docstring,
and lead the e2e evidence page with what it checks.
2026-11-14 09:12:29 +01:00

4.3 KiB
Raw Permalink Blame History

Training end-to-end evidence

The script starts a real uvicorn pulsecast.server:app process with CONTROL_TOKEN, waits for startup, and drives the HTTP API only. It checks the snapshot, every prediction/scenario path, auth and error paths, one job at a time, and both training jobs; it exits non-zero on the first failed check. Unit/contract tests (make check) are separate.

Reproduce with:

uv run python scripts/e2e_train.py --port 18000 --report /tmp/e2e.json

Run

Field Value
Date 2026-11-14, 05:33 UTC
Branch feat/ml-final
Port 18000
Checks 38 passed, 0 failed
Total wall time 10.4 s
Forecast version before histgb-2026-11-14
Data version 2026-11-07T23:51:00Z
Context version 2026-11-14T03:47:09Z

Snapshot and robustness (20 checks)

  • /healthz → {"ok":true}; /readyz → 200 model=true.
  • /meta covers 8 sensors and the baked model beats the seasonal-naive baseline on the holdout.
  • /bootstrap → 8 sensors, all 8 hourly series (23,952 points), 168 forecast weather hours, 24 holidays, versions equal to /meta.
  • /predict: 128/128 combinations (8 sensors × 4 horizons × 4 scenarios) returned the contract point counts (15m 1, 1h 4, 48h 48, 7d 168), monotonic UTC timestamps, non-negative values, and lo ≤ v ≤ hi.
  • /simulate: all 4 scenarios × 8 sensors × 4 horizons returned valid points.
  • Weather override changes the forecast (-8 °C / 6 mm vs 30 °C / 0 mm).
  • Unknown sensor → 404; missing token → 401 unauthorized; wrong token → 401; unknown job → 404 not_found; invalid model → 400 bad_request; missing JSON path → 400.

Small training job (real 60-day CSV subset)

Input: 36,846 density rows, 8 sensors, 2026-09-08T23:53Z → 2026-11-07T23:51Z.

Field Value
Job ID 0b35347963974cbbbe584b77ecd4b846
Status succeeded
Model version logit-20261114-053344
Wall time (submit → terminal) 0.172 s (contract < 5 s)
n_train / n_test 8,151 / 2,301
Base rate 0.180
Accuracy 0.824
Precision 0.542
Recall 0.157
F1 0.243
ROC-AUC 0.813

This is a different window from the baked full-window model in docs/small-model.md, so the numbers are not directly comparable: the p75 target and the 80/20 split are recomputed inside the 60-day upload. The low recall illustrates the limit noted in the small-model doc: a short, recent window with a different threshold is a different classification problem.

Post-conditions verified:

  • small_latest.joblib and <job_id>-small.joblib exist under PULSECAST_MODELS.
  • /meta.small_model.model_version equals the job version and its metrics match the job exactly (in-memory hot-swap).
  • /predict still returns 168 valid points after the swap.

Forecast training job (full challenge CSV) and 409

Input: full schwarz-pax.csv, 19,383,568 bytes.

Field Value
Job ID 235209a0d00f449ba8be717eb76386d6
Status succeeded
Model version histgb-20261114-053349
Wall time 4.380 s
MAE 3.32
WAPE 27.5%
R² 0.777
Baseline MAE / WAPE / R² 5.40 / 44.7% / 0.462
n_eval 5,350

The ablation in the job payload matches artifacts/metrics.json row for row (3.96, 3.84, 3.36, 4.08, 3.32). While this job was in running, a second POST /train returned 409 conflict, proving one job at a time.

Post-conditions verified:

  • forecast_latest.joblib and <job_id>-forecast.joblib exist under PULSECAST_MODELS.
  • /meta.model_version equals the retrained version and /meta.model_source is retrained.
  • /predict reports the retrained version; /readyz stays 200.
  • data_version after the retrain is 2026-11-07T23:00:00Z (the hourly input maximum) rather than the raw-file floor 23:51:00Z; both describe the same observed window.

Failure isolation

A malformed CSV (not,a,pulsecast,file) was accepted as a job (202), then failed with a clear error:

CSV needs a time column (time, timestamp, datetime, date, hour) and a value
column (value, density, footfall, count, pax, visitors); got ['not', 'a',
'pulsecast', 'file']

/healthz still returned 200 afterward: a bad upload cannot take down the service or block /predict.