Files
api-go/docs/integration.md
T
Noel Sabia f4693af748
ci / check (push) Successful in 54s
ci / check (pull_request) Successful in 56s
ci: fix setup-go cache and refresh docs
setup-go@v5 aborts when go.sum is missing, and this stdlib-only module has none. Disable the action cache and cap the job at 10 minutes.

Lead the README with what the service does and a runnable quick start, fix stale contract and cache comments, and remove comments that only restate the code.
2026-11-14 09:14:09 +01:00

144 lines
6.6 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Integration and load-test report
Run: **2026-11-14**, Go API worktree `feat/integration`.
FastAPI: real ML service from `ML-py.git/main` on `127.0.0.1:18000`.
Go API: this worktree on `127.0.0.1:18080`, `FASTAPI_URL=http://127.0.0.1:18000`.
Everything below is reproducible with `make integration-load`; raw evidence is
in `docs/results/`.
## Command
```sh
WORKDIR=/tmp/pulsecast-full-run RESULTS_DIR="$PWD/docs/results" \
scripts/integration.sh --load
```
The script owns `:18000` and `:18080` (it refuses to run if either is busy).
It uses a 24,636-row CSV subset (`2026-09-01 .. 2026-10-16`, all 8 sensors,
`densityData` rows only) generated from the raw challenge CSV into `/tmp`.
The subset is test input only; it is not committed and not baked into the image.
## Train end-to-end (real FastAPI, through the Go proxy)
Both jobs were uploaded as multipart CSV to `POST /api/train` with control auth,
then polled through `GET /api/train/{job_id}`.
| Job | Result | Model version | Submit to terminal | Headline metrics |
|---|---|---|---|---|
| `small` | `succeeded`, `error: null` | `logit-20261114-053410` | **171 ms** (POST itself 25 ms) | accuracy 0.824, precision 0.589, recall 0.420, F1 0.490, ROC-AUC 0.849, n_train 5,363, n_test 1,728, 8 sensors |
| `forecast` | `succeeded`, `error: null` | `histgb-20261114-053413` | **3,571 ms** | MAE 4.83, WAPE 33.0, R² 0.703, n_eval 4,919 |
- `small` is a logistic-regression busy classifier; `forecast` retrains the
global HistGB. Both ran in FastAPI; the Go process only proxied and polled.
- The forecast job trained on the **small subset**, so its MAE 4.83 is not the
baked model's quality (3.32 on the full data). It proves the upload, queue,
poll and hot-swap path, not model quality.
- `/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base` returned
**200, 168 points, `cached: false`** immediately after the small job.
- After the forecast hot-swap, the Go snapshot refresh picked up the new
`model_version` (`histgb-20261114-053413`) in `/api/meta`.
## SSE scenario, heartbeat and reconnect
| Check | Result |
|---|---|
| Connect payload | `retry: 3000`, current `scenario` event immediately |
| Scenario propagation | **15 ms** from `POST /api/scenario` to the client event (limit: 2 s) |
| Heartbeat | received during the check; server interval exactly **15,000 ms** throughout the load test |
| Reconnect | closed and reopened the stream; current scenario (v2) arrived in **18 ms** |
| Event shape | `event: scenario`, `id: <version>`, JSON body with `version`, `scenario`, `at` |
## Staffing plan generation (contract: < 2 s)
A follow-up run on the same commit (ports `18100`/`18180`, because another
verification process held `18000`/`18080`) timed `GET /api/staffing`:
| Check | Result |
|---|---|
| Cold call (live `/predict` + plan build) | **19 ms** |
| Warm call (same plan key) | **< 1 ms** |
| Plan size | 70 store hours (7 days × 10:00–20:00), 21 shift blocks |
| Coverage | 100%, cost €3,360 |
| `cached` on the live plan | `false` |
The repeat run also reproduced the other checks: small train 169 ms, scenario
propagation 13 ms, reconnect 16 ms, recovery 1,339 ms.
## 120-client / 5-minute load test
`scripts/sse-load`: 120 concurrent streams, 300.008 s, scenario change every
30 s (rotating `sunny-saturday-market`, `rainy-saturday`, `weekday-event`,
`base`).
| Metric | Result |
|---|---|
| Peak connections | **120 / 120** |
| Connected after warmup | 120 / 120 |
| Dropped or reconnected streams | **0** |
| Scenario changes | 9 |
| Scenario events received | **1,080 / 1,080** (120 per change) |
| Missing scenario events | **0** |
| Scenario latency samples | 1,080 |
| p50 | **1.3 ms** |
| p95 | **5.6 ms** |
| p99 | **5.8 ms** |
| max | **5.812 ms** |
| Heartbeat pings | 20 unique server timestamps, every **15,000 ms** (min = median = p95 = max) |
| Heartbeats per client | 20 / 20 / 20 (min / median / max) |
Per-change p95 ranged from 1.2 ms to 5.8 ms; the highest latencies were the
first change after warmup. The tool exits non-zero if a client drops, an event
is missing, or p95 exceeds the contract's 2 s limit; this run passed.
## Fallback and recovery (runtime cache)
Sequence: warm a live base forecast, stop FastAPI, observe the cache failure,
serve the snapshot/last-good data, restart FastAPI, wait for recovery.
While FastAPI was down:
| Check | Result |
|---|---|
| `/api/meta.cache.failures` | **1** |
| `/api/meta.cache.last_error` | `ml: /bootstrap: Get "http://127.0.0.1:18000/bootstrap": dial tcp 127.0.0.1:18000: connect: connection refused` |
| `/api/meta.cache.next_retry_at` | `2026-11-14T05:34:31Z` (1 s after the first failure; schedule is 1/2/4/8/16/30 s) |
| `/api/meta.cache.last_success_at` | `2026-11-14T05:34:20Z` (unchanged) |
| `/api/readyz` | **200** — a loaded snapshot keeps readiness even with FastAPI down |
| `/api/sensors` | **200**, 8 sensors, from the last good snapshot |
| `/api/context` | **200**, `cached: true` |
| `/api/series` | **200**, from the last good snapshot |
| `/api/forecast` (warm key) | **200**, 168 points, **`cached: true`** |
| `/api/staffing` | **200**, built from the last good forecast, **`cached: true`** |
| `/api/forecast` (never-warmed key) | **503** `forecast_unavailable` — no invented data |
After restarting FastAPI:
| Check | Result |
|---|---|
| Cache recovery | **1,622 ms** after restart; `failures: 0` |
| `last_success_at` | advanced from `05:34:20Z` to `05:34:33Z` |
| `/api/forecast` | **200**, `cached: false` again |
The integration script used `BOOTSTRAP_REFRESH=10s` to keep the observation
window short. The production default is 60 s; the failure path, the
1/2/4/8/16/30 s retry schedule and the recovery semantics are unchanged.
## Evidence files
- `docs/results/train-small.json`, `train-forecast.json` — raw job responses and timings
- `docs/results/staffing.json` — cold/warm plan timing from the follow-up run
- `docs/results/sse.json`, `sse-stream.log` — propagation, heartbeat, reconnect
- `docs/results/load.json`, `load.txt` — all load-test samples and per-change results
- `docs/results/fallback.json` — exact cache/fallback/recovery values
- `docs/results/summary.txt` — one-screen summary of the same run
- `docs/results/meta-initial.json`, `forecast-after-small.json` — baseline evidence
## Notes and blockers
- At one point `:18000` was held by another agent's ML end-to-end run. The test
was not started against that process; it was rerun once the port was free.
No blocker remains.
- FastAPI was trained on a subset, so the retrained forecast metrics are lower
than the baked full-data model. The baked model is restored on FastAPI
restart, and recovery was verified with that baked model.