Files
api-go/docs/integration.md
T
Noel Sabia b3146e0862 docs: record 120-client SSE load, train and fallback results
Add staffing timing to the integration harness and commit the verified
run: small train 171ms, forecast train 3.6s, SSE propagation 15ms,
120/120 clients for 300s with 0 drops and p95 5.6ms, 20 heartbeats at
15s, FastAPI-down fallback from the snapshot, and recovery in 1.6s.
2026-11-14 06:43:30 +01:00

6.6 KiB
Raw Blame History

Integration and load-test report

Run: 2026-11-14, Go API worktree feat/integration. FastAPI: real ML service from ML-py.git/main on 127.0.0.1:18000. Go API: this worktree on 127.0.0.1:18080, FASTAPI_URL=http://127.0.0.1:18000. Everything below is reproducible with make integration-load; raw evidence is in docs/results/.

Command

WORKDIR=/tmp/pulsecast-full-run RESULTS_DIR="$PWD/docs/results" \
  scripts/integration.sh --load

The script owns :18000 and :18080 (it refuses to run if either is busy). It uses a 24,636-row CSV subset (2026-09-01 .. 2026-10-16, all 8 sensors, densityData rows only) generated from the raw challenge CSV into /tmp. The subset is test input only; it is not committed and not baked into the image.

Train end-to-end (real FastAPI, through the Go proxy)

Both jobs were uploaded as multipart CSV to POST /api/train with control auth, then polled through GET /api/train/{job_id}.

Job Result Model version Submit to terminal Headline metrics
small succeeded, error: null logit-20261114-053410 171 ms (POST itself 25 ms) accuracy 0.824, precision 0.589, recall 0.420, F1 0.490, ROC-AUC 0.849, n_train 5,363, n_test 1,728, 8 sensors
forecast succeeded, error: null histgb-20261114-053413 3,571 ms MAE 4.83, WAPE 33.0, R² 0.703, n_eval 4,919
  • small is a logistic-regression busy classifier; forecast retrains the global HistGB. Both ran in FastAPI; the Go process only proxied and polled.
  • The forecast job trained on the small subset, so its MAE 4.83 is not the baked model's quality (3.32 on the full data). It proves the upload, queue, poll and hot-swap path, not model quality.
  • /api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base returned 200, 168 points, cached: false immediately after the small job.
  • After the forecast hot-swap, the Go snapshot refresh picked up the new model_version (histgb-20261114-053413) in /api/meta.

SSE scenario, heartbeat and reconnect

Check Result
Connect payload retry: 3000, current scenario event immediately
Scenario propagation 15 ms from POST /api/scenario to the client event (limit: 2 s)
Heartbeat received during the check; server interval exactly 15,000 ms throughout the load test
Reconnect closed and reopened the stream; current scenario (v2) arrived in 18 ms
Event shape event: scenario, id: <version>, JSON body with version, scenario, at

Staffing plan generation (contract: < 2 s)

A follow-up run on the same commit (ports 18100/18180, because another verification process held 18000/18080) timed GET /api/staffing:

Check Result
Cold call (live /predict + plan build) 19 ms
Warm call (cached by `sensor scenario
Plan size 70 store hours (7 days × 10:00–20:00), 21 shift blocks
Coverage 100%, cost €3,360
cached on the live plan false

The repeat run also reproduced the other checks: small train 169 ms, scenario propagation 13 ms, reconnect 16 ms, recovery 1,339 ms.

120-client / 5-minute load test

scripts/sse-load: 120 concurrent streams, 300.008 s, scenario change every 30 s (rotating sunny-saturday-market, rainy-saturday, weekday-event, base).

Metric Result
Peak connections 120 / 120
Connected after warmup 120 / 120
Dropped or reconnected streams 0
Scenario changes 9
Scenario events received 1,080 / 1,080 (120 per change)
Missing scenario events 0
Scenario latency samples 1,080
p50 1.3 ms
p95 5.6 ms
p99 5.8 ms
max 5.812 ms
Heartbeat pings 20 unique server timestamps, every 15,000 ms (min = median = p95 = max)
Heartbeats per client 20 / 20 / 20 (min / median / max)

Per-change p95 ranged from 1.2 ms to 5.8 ms; the highest latencies were the first change after warmup. The tool exits non-zero if a client drops, an event is missing, or p95 exceeds the contract's 2 s limit; this run passed.

Fallback and recovery (runtime cache)

Sequence: warm a live base forecast, stop FastAPI, observe the cache failure, serve the snapshot/last-good data, restart FastAPI, wait for recovery.

While FastAPI was down:

Check Result
/api/meta.cache.failures 1
/api/meta.cache.last_error ml: /bootstrap: Get "http://127.0.0.1:18000/bootstrap": dial tcp 127.0.0.1:18000: connect: connection refused
/api/meta.cache.next_retry_at 2026-11-14T05:34:31Z (1 s after the first failure; schedule is 1/2/4/8/16/30 s)
/api/meta.cache.last_success_at 2026-11-14T05:34:20Z (unchanged)
/api/readyz 200 — a loaded snapshot keeps readiness even with FastAPI down
/api/sensors 200, 8 sensors, from the last good snapshot
/api/context 200, cached: true
/api/series 200, from the last good snapshot
/api/forecast (warm key) 200, 168 points, cached: true
/api/staffing 200, built from the last good forecast, cached: true
/api/forecast (never-warmed key) 503 forecast_unavailable — no invented data

After restarting FastAPI:

Check Result
Cache recovery 1,622 ms after restart; failures: 0
last_success_at advanced from 05:34:20Z to 05:34:33Z
/api/forecast 200, cached: false again

The integration script used BOOTSTRAP_REFRESH=10s to keep the observation window short. The production default is 60 s; the failure path, the 1/2/4/8/16/30 s retry schedule and the recovery semantics are unchanged.

Evidence files

  • docs/results/train-small.json, train-forecast.json — raw job responses and timings
  • docs/results/staffing.json — cold/warm plan timing from the follow-up run
  • docs/results/sse.json, sse-stream.log — propagation, heartbeat, reconnect
  • docs/results/load.json, load.txt — all load-test samples and per-change results
  • docs/results/fallback.json — exact cache/fallback/recovery values
  • docs/results/summary.txt — one-screen summary of the same run
  • docs/results/meta-initial.json, forecast-after-small.json — baseline evidence

Notes and blockers

  • At one point :18000 was held by another agent's ML end-to-end run. The test was not started against that process; it was rerun once the port was free. No blocker remains.
  • FastAPI was trained on a subset, so the retrained forecast metrics are lower than the baked full-data model. The baked model is restored on FastAPI restart, and recovery was verified with that baked model.