Add staffing timing to the integration harness and commit the verified run: small train 171ms, forecast train 3.6s, SSE propagation 15ms, 120/120 clients for 300s with 0 drops and p95 5.6ms, 20 heartbeats at 15s, FastAPI-down fallback from the snapshot, and recovery in 1.6s.
6.6 KiB
Integration and load-test report
Run: 2026-11-14, Go API worktree feat/integration.
FastAPI: real ML service from ML-py.git/main on 127.0.0.1:18000.
Go API: this worktree on 127.0.0.1:18080, FASTAPI_URL=http://127.0.0.1:18000.
Everything below is reproducible with make integration-load; raw evidence is
in docs/results/.
Command
WORKDIR=/tmp/pulsecast-full-run RESULTS_DIR="$PWD/docs/results" \
scripts/integration.sh --load
The script owns :18000 and :18080 (it refuses to run if either is busy).
It uses a 24,636-row CSV subset (2026-09-01 .. 2026-10-16, all 8 sensors,
densityData rows only) generated from the raw challenge CSV into /tmp.
The subset is test input only; it is not committed and not baked into the image.
Train end-to-end (real FastAPI, through the Go proxy)
Both jobs were uploaded as multipart CSV to POST /api/train with control auth,
then polled through GET /api/train/{job_id}.
| Job | Result | Model version | Submit to terminal | Headline metrics |
|---|---|---|---|---|
small |
succeeded, error: null |
logit-20261114-053410 |
171 ms (POST itself 25 ms) | accuracy 0.824, precision 0.589, recall 0.420, F1 0.490, ROC-AUC 0.849, n_train 5,363, n_test 1,728, 8 sensors |
forecast |
succeeded, error: null |
histgb-20261114-053413 |
3,571 ms | MAE 4.83, WAPE 33.0, R² 0.703, n_eval 4,919 |
smallis a logistic-regression busy classifier;forecastretrains the global HistGB. Both ran in FastAPI; the Go process only proxied and polled.- The forecast job trained on the small subset, so its MAE 4.83 is not the baked model's quality (3.32 on the full data). It proves the upload, queue, poll and hot-swap path, not model quality.
/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=basereturned 200, 168 points,cached: falseimmediately after the small job.- After the forecast hot-swap, the Go snapshot refresh picked up the new
model_version(histgb-20261114-053413) in/api/meta.
SSE scenario, heartbeat and reconnect
| Check | Result |
|---|---|
| Connect payload | retry: 3000, current scenario event immediately |
| Scenario propagation | 15 ms from POST /api/scenario to the client event (limit: 2 s) |
| Heartbeat | received during the check; server interval exactly 15,000 ms throughout the load test |
| Reconnect | closed and reopened the stream; current scenario (v2) arrived in 18 ms |
| Event shape | event: scenario, id: <version>, JSON body with version, scenario, at |
Staffing plan generation (contract: < 2 s)
A follow-up run on the same commit (ports 18100/18180, because another
verification process held 18000/18080) timed GET /api/staffing:
| Check | Result |
|---|---|
Cold call (live /predict + plan build) |
19 ms |
| Warm call (cached by `sensor | scenario |
| Plan size | 70 store hours (7 days × 10:00–20:00), 21 shift blocks |
| Coverage | 100%, cost €3,360 |
cached on the live plan |
false |
The repeat run also reproduced the other checks: small train 169 ms, scenario propagation 13 ms, reconnect 16 ms, recovery 1,339 ms.
120-client / 5-minute load test
scripts/sse-load: 120 concurrent streams, 300.008 s, scenario change every
30 s (rotating sunny-saturday-market, rainy-saturday, weekday-event,
base).
| Metric | Result |
|---|---|
| Peak connections | 120 / 120 |
| Connected after warmup | 120 / 120 |
| Dropped or reconnected streams | 0 |
| Scenario changes | 9 |
| Scenario events received | 1,080 / 1,080 (120 per change) |
| Missing scenario events | 0 |
| Scenario latency samples | 1,080 |
| p50 | 1.3 ms |
| p95 | 5.6 ms |
| p99 | 5.8 ms |
| max | 5.812 ms |
| Heartbeat pings | 20 unique server timestamps, every 15,000 ms (min = median = p95 = max) |
| Heartbeats per client | 20 / 20 / 20 (min / median / max) |
Per-change p95 ranged from 1.2 ms to 5.8 ms; the highest latencies were the first change after warmup. The tool exits non-zero if a client drops, an event is missing, or p95 exceeds the contract's 2 s limit; this run passed.
Fallback and recovery (runtime cache)
Sequence: warm a live base forecast, stop FastAPI, observe the cache failure, serve the snapshot/last-good data, restart FastAPI, wait for recovery.
While FastAPI was down:
| Check | Result |
|---|---|
/api/meta.cache.failures |
1 |
/api/meta.cache.last_error |
ml: /bootstrap: Get "http://127.0.0.1:18000/bootstrap": dial tcp 127.0.0.1:18000: connect: connection refused |
/api/meta.cache.next_retry_at |
2026-11-14T05:34:31Z (1 s after the first failure; schedule is 1/2/4/8/16/30 s) |
/api/meta.cache.last_success_at |
2026-11-14T05:34:20Z (unchanged) |
/api/readyz |
200 — a loaded snapshot keeps readiness even with FastAPI down |
/api/sensors |
200, 8 sensors, from the last good snapshot |
/api/context |
200, cached: true |
/api/series |
200, from the last good snapshot |
/api/forecast (warm key) |
200, 168 points, cached: true |
/api/staffing |
200, built from the last good forecast, cached: true |
/api/forecast (never-warmed key) |
503 forecast_unavailable — no invented data |
After restarting FastAPI:
| Check | Result |
|---|---|
| Cache recovery | 1,622 ms after restart; failures: 0 |
last_success_at |
advanced from 05:34:20Z to 05:34:33Z |
/api/forecast |
200, cached: false again |
The integration script used BOOTSTRAP_REFRESH=10s to keep the observation
window short. The production default is 60 s; the failure path, the
1/2/4/8/16/30 s retry schedule and the recovery semantics are unchanged.
Evidence files
docs/results/train-small.json,train-forecast.json— raw job responses and timingsdocs/results/staffing.json— cold/warm plan timing from the follow-up rundocs/results/sse.json,sse-stream.log— propagation, heartbeat, reconnectdocs/results/load.json,load.txt— all load-test samples and per-change resultsdocs/results/fallback.json— exact cache/fallback/recovery valuesdocs/results/summary.txt— one-screen summary of the same rundocs/results/meta-initial.json,forecast-after-small.json— baseline evidence
Notes and blockers
- At one point
:18000was held by another agent's ML end-to-end run. The test was not started against that process; it was rerun once the port was free. No blocker remains. - FastAPI was trained on a subset, so the retrained forecast metrics are lower than the baked full-data model. The baked model is restored on FastAPI restart, and recovery was verified with that baked model.