Files
ML-py/docs/train-e2e.md
T
Noel Sabia c7ea524b5b docs: small-model artifact, paper metrics, train e2e evidence
- docs/small-model.md: target, features, split, metrics, full confusion
  matrix, per-sensor table, honest limits
- docs/metrics.md: provenance, protocol, headline, ablation, per-sensor,
  uncertainty, small model, supported/unsupported claims
- docs/figures: generated ablation, per-sensor, confusion-matrix SVGs
- docs/train-e2e.md: exact 38-check live run, small + forecast job results
- model_card.md and README updated with final metrics and links
2026-11-14 06:35:22 +01:00

4.2 KiB
Raw Blame History

Training end-to-end evidence

Reproduce with:

uv run python scripts/e2e_train.py --port 18000 --report /tmp/e2e.json

The script starts a real uvicorn pulsecast.server:app process with CONTROL_TOKEN, waits for startup, and drives the HTTP API only. It exits non-zero on the first failed check. Unit/contract tests (make check) are separate.

Run

Field Value
Date 2026-11-14, 05:33 UTC
Branch feat/ml-final
Port 18000
Checks 38 passed, 0 failed
Total wall time 10.4 s
Forecast version before histgb-2026-11-14
Data version 2026-11-07T23:51:00Z
Context version 2026-11-14T03:47:09Z

Snapshot and robustness (20 checks)

  • /healthz → {"ok":true}; /readyz → 200 model=true.
  • /meta covers 8 sensors and the baked model beats the seasonal-naive baseline on the holdout.
  • /bootstrap → 8 sensors, all 8 hourly series (23,952 points), 168 forecast weather hours, 24 holidays, versions equal to /meta.
  • /predict: 128/128 combinations (8 sensors × 4 horizons × 4 scenarios) returned the contract point counts (15m 1, 1h 4, 48h 48, 7d 168), monotonic UTC timestamps, non-negative values, and lo ≤ v ≤ hi.
  • /simulate: all 4 scenarios × 8 sensors × 4 horizons returned valid points.
  • Weather override changes the forecast (-8 °C / 6 mm vs 30 °C / 0 mm).
  • Unknown sensor → 404; missing token → 401 unauthorized; wrong token → 401; unknown job → 404 not_found; invalid model → 400 bad_request; missing JSON path → 400.

Small training job (real 60-day CSV subset)

Input: 36,846 density rows, 8 sensors, 2026-09-08T23:53Z → 2026-11-07T23:51Z.

Field Value
Job ID 0b35347963974cbbbe584b77ecd4b846
Status succeeded
Model version logit-20261114-053344
Wall time (submit → terminal) 0.172 s (contract < 5 s)
n_train / n_test 8,151 / 2,301
Base rate 0.180
Accuracy 0.824
Precision 0.542
Recall 0.157
F1 0.243
ROC-AUC 0.813

This is a different window from the baked full-window model in docs/small-model.md, so the numbers are not directly comparable: the p75 target and the 80/20 split are recomputed inside the 60-day upload. The low recall illustrates the honest limit noted in the small-model doc — a short, recent window with a different threshold is a different classification problem.

Post-conditions verified:

  • small_latest.joblib and <job_id>-small.joblib exist under PULSECAST_MODELS.
  • /meta.small_model.model_version equals the job version and its metrics match the job exactly (in-memory hot-swap).
  • /predict still returns 168 valid points after the swap.

Forecast training job (full challenge CSV) and 409

Input: full schwarz-pax.csv, 19,383,568 bytes.

Field Value
Job ID 235209a0d00f449ba8be717eb76386d6
Status succeeded
Model version histgb-20261114-053349
Wall time 4.380 s
MAE 3.32
WAPE 27.5%
R² 0.777
Baseline MAE / WAPE / R² 5.40 / 44.7% / 0.462
n_eval 5,350

The ablation in the job payload matches artifacts/metrics.json row for row (3.96, 3.84, 3.36, 4.08, 3.32). While this job was in running, a second POST /train returned 409 conflict, proving one job at a time.

Post-conditions verified:

  • forecast_latest.joblib and <job_id>-forecast.joblib exist under PULSECAST_MODELS.
  • /meta.model_version equals the retrained version and /meta.model_source is retrained.
  • /predict reports the retrained version; /readyz stays 200.
  • data_version after the retrain is 2026-11-07T23:00:00Z (the hourly input maximum) rather than the raw-file floor 23:51:00Z; both describe the same observed window.

Failure isolation

A malformed CSV (not,a,pulsecast,file) was accepted as a job (202), then failed with a clear error:

CSV needs a time column (time, timestamp, datetime, date, hour) and a value
column (value, density, footfall, count, pax, visitors); got ['not', 'a',
'pulsecast', 'file']

/healthz still returned 200 afterward: a bad upload cannot take down the service or block /predict.