# Integration and load-test report Run: **2026-11-14**, Go API worktree `feat/integration`. FastAPI: real ML service from `ML-py.git/main` on `127.0.0.1:18000`. Go API: this worktree on `127.0.0.1:18080`, `FASTAPI_URL=http://127.0.0.1:18000`. Everything below is reproducible with `make integration-load`; raw evidence is in `docs/results/`. ## Command ```sh WORKDIR=/tmp/pulsecast-full-run RESULTS_DIR="$PWD/docs/results" \ scripts/integration.sh --load ``` The script owns `:18000` and `:18080` (it refuses to run if either is busy). It uses a 24,636-row CSV subset (`2026-09-01 .. 2026-10-16`, all 8 sensors, `densityData` rows only) generated from the raw challenge CSV into `/tmp`. The subset is test input only; it is not committed and not baked into the image. ## Train end-to-end (real FastAPI, through the Go proxy) Both jobs were uploaded as multipart CSV to `POST /api/train` with control auth, then polled through `GET /api/train/{job_id}`. | Job | Result | Model version | Submit to terminal | Headline metrics | |---|---|---|---|---| | `small` | `succeeded`, `error: null` | `logit-20261114-053410` | **171 ms** (POST itself 25 ms) | accuracy 0.824, precision 0.589, recall 0.420, F1 0.490, ROC-AUC 0.849, n_train 5,363, n_test 1,728, 8 sensors | | `forecast` | `succeeded`, `error: null` | `histgb-20261114-053413` | **3,571 ms** | MAE 4.83, WAPE 33.0, R² 0.703, n_eval 4,919 | - `small` is a logistic-regression busy classifier; `forecast` retrains the global HistGB. Both ran in FastAPI; the Go process only proxied and polled. - The forecast job trained on the **small subset**, so its MAE 4.83 is not the baked model's quality (3.32 on the full data). It proves the upload, queue, poll and hot-swap path, not model quality. - `/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base` returned **200, 168 points, `cached: false`** immediately after the small job. - After the forecast hot-swap, the Go snapshot refresh picked up the new `model_version` (`histgb-20261114-053413`) in `/api/meta`. ## SSE scenario, heartbeat and reconnect | Check | Result | |---|---| | Connect payload | `retry: 3000`, current `scenario` event immediately | | Scenario propagation | **15 ms** from `POST /api/scenario` to the client event (limit: 2 s) | | Heartbeat | received during the check; server interval exactly **15,000 ms** throughout the load test | | Reconnect | closed and reopened the stream; current scenario (v2) arrived in **18 ms** | | Event shape | `event: scenario`, `id: `, JSON body with `version`, `scenario`, `at` | ## Staffing plan generation (contract: < 2 s) A follow-up run on the same commit (ports `18100`/`18180`, because another verification process held `18000`/`18080`) timed `GET /api/staffing`: | Check | Result | |---|---| | Cold call (live `/predict` + plan build) | **19 ms** | | Warm call (same plan key) | **< 1 ms** | | Plan size | 70 store hours (7 days × 10:00–20:00), 21 shift blocks | | Coverage | 100%, cost €3,360 | | `cached` on the live plan | `false` | The repeat run also reproduced the other checks: small train 169 ms, scenario propagation 13 ms, reconnect 16 ms, recovery 1,339 ms. ## 120-client / 5-minute load test `scripts/sse-load`: 120 concurrent streams, 300.008 s, scenario change every 30 s (rotating `sunny-saturday-market`, `rainy-saturday`, `weekday-event`, `base`). | Metric | Result | |---|---| | Peak connections | **120 / 120** | | Connected after warmup | 120 / 120 | | Dropped or reconnected streams | **0** | | Scenario changes | 9 | | Scenario events received | **1,080 / 1,080** (120 per change) | | Missing scenario events | **0** | | Scenario latency samples | 1,080 | | p50 | **1.3 ms** | | p95 | **5.6 ms** | | p99 | **5.8 ms** | | max | **5.812 ms** | | Heartbeat pings | 20 unique server timestamps, every **15,000 ms** (min = median = p95 = max) | | Heartbeats per client | 20 / 20 / 20 (min / median / max) | Per-change p95 ranged from 1.2 ms to 5.8 ms; the highest latencies were the first change after warmup. The tool exits non-zero if a client drops, an event is missing, or p95 exceeds the contract's 2 s limit; this run passed. ## Fallback and recovery (runtime cache) Sequence: warm a live base forecast, stop FastAPI, observe the cache failure, serve the snapshot/last-good data, restart FastAPI, wait for recovery. While FastAPI was down: | Check | Result | |---|---| | `/api/meta.cache.failures` | **1** | | `/api/meta.cache.last_error` | `ml: /bootstrap: Get "http://127.0.0.1:18000/bootstrap": dial tcp 127.0.0.1:18000: connect: connection refused` | | `/api/meta.cache.next_retry_at` | `2026-11-14T05:34:31Z` (1 s after the first failure; schedule is 1/2/4/8/16/30 s) | | `/api/meta.cache.last_success_at` | `2026-11-14T05:34:20Z` (unchanged) | | `/api/readyz` | **200** — a loaded snapshot keeps readiness even with FastAPI down | | `/api/sensors` | **200**, 8 sensors, from the last good snapshot | | `/api/context` | **200**, `cached: true` | | `/api/series` | **200**, from the last good snapshot | | `/api/forecast` (warm key) | **200**, 168 points, **`cached: true`** | | `/api/staffing` | **200**, built from the last good forecast, **`cached: true`** | | `/api/forecast` (never-warmed key) | **503** `forecast_unavailable` — no invented data | After restarting FastAPI: | Check | Result | |---|---| | Cache recovery | **1,622 ms** after restart; `failures: 0` | | `last_success_at` | advanced from `05:34:20Z` to `05:34:33Z` | | `/api/forecast` | **200**, `cached: false` again | The integration script used `BOOTSTRAP_REFRESH=10s` to keep the observation window short. The production default is 60 s; the failure path, the 1/2/4/8/16/30 s retry schedule and the recovery semantics are unchanged. ## Evidence files - `docs/results/train-small.json`, `train-forecast.json` — raw job responses and timings - `docs/results/staffing.json` — cold/warm plan timing from the follow-up run - `docs/results/sse.json`, `sse-stream.log` — propagation, heartbeat, reconnect - `docs/results/load.json`, `load.txt` — all load-test samples and per-change results - `docs/results/fallback.json` — exact cache/fallback/recovery values - `docs/results/summary.txt` — one-screen summary of the same run - `docs/results/meta-initial.json`, `forecast-after-small.json` — baseline evidence ## Notes and blockers - At one point `:18000` was held by another agent's ML end-to-end run. The test was not started against that process; it was rerun once the port was free. No blocker remains. - FastAPI was trained on a subset, so the retrained forecast metrics are lower than the baked full-data model. The baked model is restored on FastAPI restart, and recovery was verified with that baked model.