docs: small-model artifact, paper metrics, train e2e evidence
- docs/small-model.md: target, features, split, metrics, full confusion matrix, per-sensor table, honest limits - docs/metrics.md: provenance, protocol, headline, ablation, per-sensor, uncertainty, small model, supported/unsupported claims - docs/figures: generated ablation, per-sensor, confusion-matrix SVGs - docs/train-e2e.md: exact 38-check live run, small + forecast job results - model_card.md and README updated with final metrics and links
This commit is contained in:
1 parent
2bcf66c550
commit
c7ea524b5b
8 files changed
+702
-16
No files matched your search
@@ -0,0 +1,118 @@
|
||||
# Training end-to-end evidence
|
||||
|
||||
Reproduce with:
|
||||
|
||||
```bash
|
||||
uv run python scripts/e2e_train.py --port 18000 --report /tmp/e2e.json
|
||||
```
|
||||
|
||||
The script starts a real `uvicorn pulsecast.server:app` process with
|
||||
`CONTROL_TOKEN`, waits for startup, and drives the HTTP API only. It exits
|
||||
non-zero on the first failed check. Unit/contract tests (`make check`) are
|
||||
separate.
|
||||
|
||||
## Run
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Date | 2026-11-14, 05:33 UTC |
|
||||
| Branch | `feat/ml-final` |
|
||||
| Port | **18000** |
|
||||
| Checks | **38 passed, 0 failed** |
|
||||
| Total wall time | 10.4 s |
|
||||
| Forecast version before | `histgb-2026-11-14` |
|
||||
| Data version | `2026-11-07T23:51:00Z` |
|
||||
| Context version | `2026-11-14T03:47:09Z` |
|
||||
|
||||
## Snapshot and robustness (20 checks)
|
||||
|
||||
- `/healthz` → `{"ok":true}`; `/readyz` → 200 `model=true`.
|
||||
- `/meta` covers 8 sensors and the baked model beats the seasonal-naive
|
||||
baseline on the holdout.
|
||||
- `/bootstrap` → 8 sensors, all 8 hourly series (23,952 points), 168 forecast
|
||||
weather hours, 24 holidays, versions equal to `/meta`.
|
||||
- `/predict`: **128/128** combinations (8 sensors × 4 horizons × 4 scenarios)
|
||||
returned the contract point counts (`15m` 1, `1h` 4, `48h` 48, `7d` 168),
|
||||
monotonic UTC timestamps, non-negative values, and `lo ≤ v ≤ hi`.
|
||||
- `/simulate`: all 4 scenarios × 8 sensors × 4 horizons returned valid points.
|
||||
- Weather override changes the forecast (`-8 °C / 6 mm` vs `30 °C / 0 mm`).
|
||||
- Unknown sensor → 404; missing token → 401 `unauthorized`; wrong token → 401;
|
||||
unknown job → 404 `not_found`; invalid model → 400 `bad_request`; missing
|
||||
JSON path → 400.
|
||||
|
||||
## Small training job (real 60-day CSV subset)
|
||||
|
||||
Input: 36,846 density rows, 8 sensors, 2026-09-08T23:53Z → 2026-11-07T23:51Z.
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Job ID | `0b35347963974cbbbe584b77ecd4b846` |
|
||||
| Status | `succeeded` |
|
||||
| Model version | `logit-20261114-053344` |
|
||||
| Wall time (submit → terminal) | **0.172 s** (contract < 5 s) |
|
||||
| n_train / n_test | 8,151 / 2,301 |
|
||||
| Base rate | 0.180 |
|
||||
| Accuracy | 0.824 |
|
||||
| Precision | 0.542 |
|
||||
| Recall | 0.157 |
|
||||
| F1 | 0.243 |
|
||||
| ROC-AUC | 0.813 |
|
||||
|
||||
This is a different window from the baked full-window model in
|
||||
`docs/small-model.md`, so the numbers are not directly comparable: the p75
|
||||
target and the 80/20 split are recomputed inside the 60-day upload. The
|
||||
low recall illustrates the honest limit noted in the small-model doc — a short,
|
||||
recent window with a different threshold is a different classification problem.
|
||||
|
||||
Post-conditions verified:
|
||||
|
||||
- `small_latest.joblib` and `<job_id>-small.joblib` exist under
|
||||
`PULSECAST_MODELS`.
|
||||
- `/meta.small_model.model_version` equals the job version and its metrics
|
||||
match the job exactly (in-memory hot-swap).
|
||||
- `/predict` still returns 168 valid points after the swap.
|
||||
|
||||
## Forecast training job (full challenge CSV) and 409
|
||||
|
||||
Input: full `schwarz-pax.csv`, 19,383,568 bytes.
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Job ID | `235209a0d00f449ba8be717eb76386d6` |
|
||||
| Status | `succeeded` |
|
||||
| Model version | `histgb-20261114-053349` |
|
||||
| Wall time | **4.380 s** |
|
||||
| MAE | 3.32 |
|
||||
| WAPE | 27.5% |
|
||||
| R² | 0.777 |
|
||||
| Baseline MAE / WAPE / R² | 5.40 / 44.7% / 0.462 |
|
||||
| n_eval | 5,350 |
|
||||
|
||||
The ablation in the job payload matches `artifacts/metrics.json` row for row
|
||||
(3.96, 3.84, 3.36, 4.08, 3.32). While this job was in `running`, a second
|
||||
`POST /train` returned **409** `conflict`, proving one job at a time.
|
||||
|
||||
Post-conditions verified:
|
||||
|
||||
- `forecast_latest.joblib` and `<job_id>-forecast.joblib` exist under
|
||||
`PULSECAST_MODELS`.
|
||||
- `/meta.model_version` equals the retrained version and
|
||||
`/meta.model_source` is `retrained`.
|
||||
- `/predict` reports the retrained version; `/readyz` stays 200.
|
||||
- `data_version` after the retrain is `2026-11-07T23:00:00Z` (the hourly input
|
||||
maximum) rather than the raw-file floor `23:51:00Z`; both describe the same
|
||||
observed window.
|
||||
|
||||
## Failure isolation
|
||||
|
||||
A malformed CSV (`not,a,pulsecast,file`) was accepted as a job (202), then
|
||||
failed with a clear error:
|
||||
|
||||
```text
|
||||
CSV needs a time column (time, timestamp, datetime, date, hour) and a value
|
||||
column (value, density, footfall, count, pax, visitors); got ['not', 'a',
|
||||
'pulsecast', 'file']
|
||||
```
|
||||
|
||||
`/healthz` still returned 200 afterward: a bad upload cannot take down the
|
||||
service or block `/predict`.
|
||||
Reference in new issue
Block a user