Add staffing timing to the integration harness and commit the verified run: small train 171ms, forecast train 3.6s, SSE propagation 15ms, 120/120 clients for 300s with 0 drops and p95 5.6ms, 20 heartbeats at 15s, FastAPI-down fallback from the snapshot, and recovery in 1.6s.
144 lines
6.6 KiB
Markdown
144 lines
6.6 KiB
Markdown
# Integration and load-test report
|
||
|
||
Run: **2026-11-14**, Go API worktree `feat/integration`.
|
||
FastAPI: real ML service from `ML-py.git/main` on `127.0.0.1:18000`.
|
||
Go API: this worktree on `127.0.0.1:18080`, `FASTAPI_URL=http://127.0.0.1:18000`.
|
||
Everything below is reproducible with `make integration-load`; raw evidence is
|
||
in `docs/results/`.
|
||
|
||
## Command
|
||
|
||
```sh
|
||
WORKDIR=/tmp/pulsecast-full-run RESULTS_DIR="$PWD/docs/results" \
|
||
scripts/integration.sh --load
|
||
```
|
||
|
||
The script owns `:18000` and `:18080` (it refuses to run if either is busy).
|
||
It uses a 24,636-row CSV subset (`2026-09-01 .. 2026-10-16`, all 8 sensors,
|
||
`densityData` rows only) generated from the raw challenge CSV into `/tmp`.
|
||
The subset is test input only; it is not committed and not baked into the image.
|
||
|
||
## Train end-to-end (real FastAPI, through the Go proxy)
|
||
|
||
Both jobs were uploaded as multipart CSV to `POST /api/train` with control auth,
|
||
then polled through `GET /api/train/{job_id}`.
|
||
|
||
| Job | Result | Model version | Submit to terminal | Headline metrics |
|
||
|---|---|---|---|---|
|
||
| `small` | `succeeded`, `error: null` | `logit-20261114-053410` | **171 ms** (POST itself 25 ms) | accuracy 0.824, precision 0.589, recall 0.420, F1 0.490, ROC-AUC 0.849, n_train 5,363, n_test 1,728, 8 sensors |
|
||
| `forecast` | `succeeded`, `error: null` | `histgb-20261114-053413` | **3,571 ms** | MAE 4.83, WAPE 33.0, R² 0.703, n_eval 4,919 |
|
||
|
||
- `small` is a logistic-regression busy classifier; `forecast` retrains the
|
||
global HistGB. Both ran in FastAPI; the Go process only proxied and polled.
|
||
- The forecast job trained on the **small subset**, so its MAE 4.83 is not the
|
||
baked model's quality (3.32 on the full data). It proves the upload, queue,
|
||
poll and hot-swap path, not model quality.
|
||
- `/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base` returned
|
||
**200, 168 points, `cached: false`** immediately after the small job.
|
||
- After the forecast hot-swap, the Go snapshot refresh picked up the new
|
||
`model_version` (`histgb-20261114-053413`) in `/api/meta`.
|
||
|
||
## SSE scenario, heartbeat and reconnect
|
||
|
||
| Check | Result |
|
||
|---|---|
|
||
| Connect payload | `retry: 3000`, current `scenario` event immediately |
|
||
| Scenario propagation | **15 ms** from `POST /api/scenario` to the client event (limit: 2 s) |
|
||
| Heartbeat | received during the check; server interval exactly **15,000 ms** throughout the load test |
|
||
| Reconnect | closed and reopened the stream; current scenario (v2) arrived in **18 ms** |
|
||
| Event shape | `event: scenario`, `id: <version>`, JSON body with `version`, `scenario`, `at` |
|
||
|
||
## Staffing plan generation (contract: < 2 s)
|
||
|
||
A follow-up run on the same commit (ports `18100`/`18180`, because another
|
||
verification process held `18000`/`18080`) timed `GET /api/staffing`:
|
||
|
||
| Check | Result |
|
||
|---|---|
|
||
| Cold call (live `/predict` + plan build) | **19 ms** |
|
||
| Warm call (cached by `sensor|scenario|standard|version|weather`) | **< 1 ms** |
|
||
| Plan size | 70 store hours (7 days × 10:00–20:00), 21 shift blocks |
|
||
| Coverage | 100%, cost €3,360 |
|
||
| `cached` on the live plan | `false` |
|
||
|
||
The repeat run also reproduced the other checks: small train 169 ms, scenario
|
||
propagation 13 ms, reconnect 16 ms, recovery 1,339 ms.
|
||
|
||
## 120-client / 5-minute load test
|
||
|
||
`scripts/sse-load`: 120 concurrent streams, 300.008 s, scenario change every
|
||
30 s (rotating `sunny-saturday-market`, `rainy-saturday`, `weekday-event`,
|
||
`base`).
|
||
|
||
| Metric | Result |
|
||
|---|---|
|
||
| Peak connections | **120 / 120** |
|
||
| Connected after warmup | 120 / 120 |
|
||
| Dropped or reconnected streams | **0** |
|
||
| Scenario changes | 9 |
|
||
| Scenario events received | **1,080 / 1,080** (120 per change) |
|
||
| Missing scenario events | **0** |
|
||
| Scenario latency samples | 1,080 |
|
||
| p50 | **1.3 ms** |
|
||
| p95 | **5.6 ms** |
|
||
| p99 | **5.8 ms** |
|
||
| max | **5.812 ms** |
|
||
| Heartbeat pings | 20 unique server timestamps, every **15,000 ms** (min = median = p95 = max) |
|
||
| Heartbeats per client | 20 / 20 / 20 (min / median / max) |
|
||
|
||
Per-change p95 ranged from 1.2 ms to 5.8 ms; the highest latencies were the
|
||
first change after warmup. The tool exits non-zero if a client drops, an event
|
||
is missing, or p95 exceeds the contract's 2 s limit; this run passed.
|
||
|
||
## Fallback and recovery (runtime cache)
|
||
|
||
Sequence: warm a live base forecast, stop FastAPI, observe the cache failure,
|
||
serve the snapshot/last-good data, restart FastAPI, wait for recovery.
|
||
|
||
While FastAPI was down:
|
||
|
||
| Check | Result |
|
||
|---|---|
|
||
| `/api/meta.cache.failures` | **1** |
|
||
| `/api/meta.cache.last_error` | `ml: /bootstrap: Get "http://127.0.0.1:18000/bootstrap": dial tcp 127.0.0.1:18000: connect: connection refused` |
|
||
| `/api/meta.cache.next_retry_at` | `2026-11-14T05:34:31Z` (1 s after the first failure; schedule is 1/2/4/8/16/30 s) |
|
||
| `/api/meta.cache.last_success_at` | `2026-11-14T05:34:20Z` (unchanged) |
|
||
| `/api/readyz` | **200** — a loaded snapshot keeps readiness even with FastAPI down |
|
||
| `/api/sensors` | **200**, 8 sensors, from the last good snapshot |
|
||
| `/api/context` | **200**, `cached: true` |
|
||
| `/api/series` | **200**, from the last good snapshot |
|
||
| `/api/forecast` (warm key) | **200**, 168 points, **`cached: true`** |
|
||
| `/api/staffing` | **200**, built from the last good forecast, **`cached: true`** |
|
||
| `/api/forecast` (never-warmed key) | **503** `forecast_unavailable` — no invented data |
|
||
|
||
After restarting FastAPI:
|
||
|
||
| Check | Result |
|
||
|---|---|
|
||
| Cache recovery | **1,622 ms** after restart; `failures: 0` |
|
||
| `last_success_at` | advanced from `05:34:20Z` to `05:34:33Z` |
|
||
| `/api/forecast` | **200**, `cached: false` again |
|
||
|
||
The integration script used `BOOTSTRAP_REFRESH=10s` to keep the observation
|
||
window short. The production default is 60 s; the failure path, the
|
||
1/2/4/8/16/30 s retry schedule and the recovery semantics are unchanged.
|
||
|
||
## Evidence files
|
||
|
||
- `docs/results/train-small.json`, `train-forecast.json` — raw job responses and timings
|
||
- `docs/results/staffing.json` — cold/warm plan timing from the follow-up run
|
||
- `docs/results/sse.json`, `sse-stream.log` — propagation, heartbeat, reconnect
|
||
- `docs/results/load.json`, `load.txt` — all load-test samples and per-change results
|
||
- `docs/results/fallback.json` — exact cache/fallback/recovery values
|
||
- `docs/results/summary.txt` — one-screen summary of the same run
|
||
- `docs/results/meta-initial.json`, `forecast-after-small.json` — baseline evidence
|
||
|
||
## Notes and blockers
|
||
|
||
- At one point `:18000` was held by another agent's ML end-to-end run. The test
|
||
was not started against that process; it was rerun once the port was free.
|
||
No blocker remains.
|
||
- FastAPI was trained on a subset, so the retrained forecast metrics are lower
|
||
than the baked full-data model. The baked model is restored on FastAPI
|
||
restart, and recovery was verified with that baked model.
|