test: repeatable integration harness and 120-client SSE load tool

Add scripts/integration.sh, which starts the real FastAPI and Go API,
trains small and forecast through POST /api/train, checks SSE scenario
propagation, heartbeat, reconnect, then kills FastAPI to verify snapshot
and last-good forecast fallback before restarting it for recovery.

Add scripts/sse-load, a standard-library load generator that holds 120
concurrent SSE streams for five minutes, posts scenario changes, records
p50/p95/p99 event latency, drops and heartbeat cadence, and exits
non-zero when a contract threshold is missed. Add unit tests, make
targets and README documentation.
This commit is contained in:
Noel Sabia committed 2026-11-14 06:33:29 +01:00
1 parent e9f80b9ce7
commit ac7ceca7ec
5 files changed
+1273 -1

No files matched your search

+27
View File
@@ -36,6 +36,33 @@ the last good snapshot and retries with exponential backoff
| `BOOTSTRAP_REFRESH` | `60s` | Healthy snapshot refresh interval |
| `BOOTSTRAP_TIMEOUT` | `20s` | Per-request `/bootstrap` timeout |
| `ML_TIMEOUT` | `1500ms` | Per-request `/predict` timeout |
| `CONTROL_USER` / `CONTROL_PASS` | unset | Basic auth for `POST /api/scenario` and `/api/train*` |
| `CONTROL_TOKEN` | unset | Optional Bearer token forwarded to FastAPI `/train*` |
## Integration and load test
`make integration` runs the real stack (it refuses to start if `:18000` or
`:18080` is busy):
1. starts FastAPI from the ML checkout on `:18000` and the Go API on `:18080`;
2. uploads a 24k-row CSV subset to `POST /api/train` and polls the job
(`small`, then `forecast` by default);
3. verifies the live forecast still answers, scenario propagation, the 15 s
heartbeat and reconnect behaviour;
4. stops FastAPI, verifies `/api/meta.cache`, snapshot-served sensors, the
last-good forecast (`cached: true`) and staffing fallback;
5. restarts FastAPI and waits for the cache to recover.
`make integration-load` additionally runs `scripts/sse-load`: 120 concurrent
SSE clients for 5 minutes with a scenario change every 30 s. It records peak
connections, drops, per-change and overall p95 latency, and heartbeat cadence.
The tool exits non-zero when a client drops, an event is missing, or p95
scenario latency exceeds the contract's 2 s threshold.
Results land in `/tmp/pulsecast-integration/results` by default. Override with
`WORKDIR=/path` or `RESULTS_DIR=/path`. `make load` runs only the load test
against an API that is already up.
## Container