# Training end-to-end evidence The script starts a real `uvicorn pulsecast.server:app` process with `CONTROL_TOKEN`, waits for startup, and drives the HTTP API only. It checks the snapshot, every prediction/scenario path, auth and error paths, one job at a time, and both training jobs; it exits non-zero on the first failed check. Unit/contract tests (`make check`) are separate. Reproduce with: ```bash uv run python scripts/e2e_train.py --port 18000 --report /tmp/e2e.json ``` ## Run | Field | Value | |---|---| | Date | 2026-11-14, 05:33 UTC | | Branch | `feat/ml-final` | | Port | **18000** | | Checks | **38 passed, 0 failed** | | Total wall time | 10.4 s | | Forecast version before | `histgb-2026-11-14` | | Data version | `2026-11-07T23:51:00Z` | | Context version | `2026-11-14T03:47:09Z` | ## Snapshot and robustness (20 checks) - `/healthz` → `{"ok":true}`; `/readyz` → 200 `model=true`. - `/meta` covers 8 sensors and the baked model beats the seasonal-naive baseline on the holdout. - `/bootstrap` → 8 sensors, all 8 hourly series (23,952 points), 168 forecast weather hours, 24 holidays, versions equal to `/meta`. - `/predict`: **128/128** combinations (8 sensors × 4 horizons × 4 scenarios) returned the contract point counts (`15m` 1, `1h` 4, `48h` 48, `7d` 168), monotonic UTC timestamps, non-negative values, and `lo ≤ v ≤ hi`. - `/simulate`: all 4 scenarios × 8 sensors × 4 horizons returned valid points. - Weather override changes the forecast (`-8 °C / 6 mm` vs `30 °C / 0 mm`). - Unknown sensor → 404; missing token → 401 `unauthorized`; wrong token → 401; unknown job → 404 `not_found`; invalid model → 400 `bad_request`; missing JSON path → 400. ## Small training job (real 60-day CSV subset) Input: 36,846 density rows, 8 sensors, 2026-09-08T23:53Z → 2026-11-07T23:51Z. | Field | Value | |---|---| | Job ID | `0b35347963974cbbbe584b77ecd4b846` | | Status | `succeeded` | | Model version | `logit-20261114-053344` | | Wall time (submit → terminal) | **0.172 s** (contract < 5 s) | | n_train / n_test | 8,151 / 2,301 | | Base rate | 0.180 | | Accuracy | 0.824 | | Precision | 0.542 | | Recall | 0.157 | | F1 | 0.243 | | ROC-AUC | 0.813 | This is a different window from the baked full-window model in `docs/small-model.md`, so the numbers are not directly comparable: the p75 target and the 80/20 split are recomputed inside the 60-day upload. The low recall illustrates the limit noted in the small-model doc: a short, recent window with a different threshold is a different classification problem. Post-conditions verified: - `small_latest.joblib` and `-small.joblib` exist under `PULSECAST_MODELS`. - `/meta.small_model.model_version` equals the job version and its metrics match the job exactly (in-memory hot-swap). - `/predict` still returns 168 valid points after the swap. ## Forecast training job (full challenge CSV) and 409 Input: full `schwarz-pax.csv`, 19,383,568 bytes. | Field | Value | |---|---| | Job ID | `235209a0d00f449ba8be717eb76386d6` | | Status | `succeeded` | | Model version | `histgb-20261114-053349` | | Wall time | **4.380 s** | | MAE | 3.32 | | WAPE | 27.5% | | R² | 0.777 | | Baseline MAE / WAPE / R² | 5.40 / 44.7% / 0.462 | | n_eval | 5,350 | The ablation in the job payload matches `artifacts/metrics.json` row for row (3.96, 3.84, 3.36, 4.08, 3.32). While this job was in `running`, a second `POST /train` returned **409** `conflict`, proving one job at a time. Post-conditions verified: - `forecast_latest.joblib` and `-forecast.joblib` exist under `PULSECAST_MODELS`. - `/meta.model_version` equals the retrained version and `/meta.model_source` is `retrained`. - `/predict` reports the retrained version; `/readyz` stays 200. - `data_version` after the retrain is `2026-11-07T23:00:00Z` (the hourly input maximum) rather than the raw-file floor `23:51:00Z`; both describe the same observed window. ## Failure isolation A malformed CSV (`not,a,pulsecast,file`) was accepted as a job (202), then failed with a clear error: ```text CSV needs a time column (time, timestamp, datetime, date, hour) and a value column (value, density, footfall, count, pax, visitors); got ['not', 'a', 'pulsecast', 'file'] ``` `/healthz` still returned 200 afterward: a bad upload cannot take down the service or block `/predict`.