Files
infra-k8/docs/load-test.md
T
Noel Sabia b98a1bab42
infra-ci / manifests (push) Failing after 4s
infra-ci / manifests (pull_request) Failing after 3s
feat: prepare v1.1 deploy and presenter load test
- build and push v1.1 images from the service main worktrees
- switch manifests to :v1.1; add web control Secret refs
- configure API BOOTSTRAP_REFRESH=60s and remove unused DATA_DIR
- add k6 presenter scenario with a mid-run control POST
- require control credentials in make load; update runbook and drills
2026-11-14 08:00:58 +01:00

129 lines
5.1 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# k6 load test
`k6/sse.js` holds 120 concurrent Server-Sent Events connections for 5 minutes
and reports connection count plus p95 event latency. A separate `presenter`
scenario POSTs one scenario change at 2m30s so the test measures event
propagation through Cloudflare, not just heartbeat latency.
## Why a custom k6 binary
Native k6 buffers a complete HTTP response, so it cannot parse a stream that
never ends. The script imports the community `k6/x/sse` extension
(`github.com/phymbert/xk6-sse`), which a stock k6 binary does not include.
Build the pinned binary once:
```bash
make k6-build # creates k6/k6-sse (git-ignored)
./k6/k6-sse version # expect k6 v2.0.0 + xk6-sse v0.2.0
```
The build pins **k6 v2.0.0**. On k6 v2.4.0 the same `xk6-sse` v0.2.0 build
fails at runtime with `TypeError: Object has no member 'on'` (the extension's
client methods are not exposed); v2.0.0 was verified locally on 2026-11-14. If
you bump `K6_BUILD_VERSION`, rerun the script against a test stream first.
`make load` uses, in order: `K6` if set, `k6/k6-sse`, then `k6` on `PATH`.
A stock `grafana/k6` image is **not** a fallback: it lacks the SSE extension.
## Procedure
1. Confirm the API is healthy and the SSE route works:
```bash
curl -sS https://api-pulse.noelsabia.dev/api/healthz
curl -sS -N --max-time 20 https://api-pulse.noelsabia.dev/api/stream
```
You should see `retry: 3000` and a `scenario` event immediately, then a
`ping` every 15 s.
2. Load the control credentials and run the load test. The presenter scenario
requires `CONTROL_USER` and `CONTROL_PASS`; `make load` refuses to start
without them.
```bash
set -a
source ~/.config/pulsecast/control-auth.env
set +a
make load
# or: API_URL=https://api-pulse.noelsabia.dev \
# CONTROL_USER="$CONTROL_USER" CONTROL_PASS="$CONTROL_PASS" \
# ./k6/k6-sse run k6/sse.js
```
`make load` writes `k6/results/summary-<timestamp>.json` (git-ignored).
3. The `presenter` scenario starts at `2m30s`, POSTs
`PRESENTER_SCENARIO=rainy-saturday` once to `/api/scenario`, and records the
request duration plus success. All 120 SSE clients must receive the
resulting `scenario` event. Override the timing/scenario if needed:
```bash
make load PRESENTER_AT=3m PRESENTER_SCENARIO=sunny-saturday-market
```
4. Record these numbers in the paper/README from the saved summary:
- `sse_connections_opened` count (must be 120)
- `sse_connections_closed` count (must be >= 120)
- `sse_open_success` rate (must be >= 99%)
- `sse_event_latency` p95 (target < 2000 ms)
- `sse_scenario_latency` p95 (target < 2000 ms)
- `presenter_post_success` rate (must be 100%)
- `presenter_post_duration` (record p95/max)
- `sse_connection_errors` (target 0)
- `sse_events_received` total
## Local validation before the cluster run
The script was validated locally on 2026-11-14 with the pinned k6 v2.0.0 build
against a mock SSE stream and the real v1.1 API image (3 VUs, 1 heartbeat
each): connections opened and closed cleanly, open success 100%, presenter
POST 200, and scenario latency sampled. The real 120-client run still requires
the deployed API, because localhost results say nothing about Cloudflare
buffering or SKE scheduling.
## Pass criteria
| Metric | Target |
|---|---|
| Connections opened | 120 |
| Connections closed (stream ended) | >= 120 |
| Open success rate | >= 99% |
| p95 event latency | < 2000 ms |
| p95 scenario latency | < 2000 ms |
| Presenter POST success | 100% |
| Unexpected connection errors | 0 |
| Duration | 5 minutes |
The k6 thresholds enforce connections opened/closed, open success, p95 event
latency, p95 scenario latency, presenter success, and zero connection errors.
A p95 above 2 s means Cloudflare, the Go SSE hub, or the client is
buffering/delaying events; check `X-Accel-Buffering: no`, the 15 s heartbeat,
and whether the connection is HTTP/2-buffered at the edge.
## Results
Populate from the saved `k6/results/summary-*.json`; never estimate.
| Run | Date | Opened | Closed | Open success | p95 event | p95 scenario | Presenter success | Errors |
|---|---|---|---|---|---|---|---|---|
| v1.1 through Cloudflare | pending post-merge run | | | | | | | |
## Notes
- `sse_event_latency` uses the server timestamp in `data.t` (ping) or `data.at`
(scenario). It includes clock skew between the k6 host and the pod; keep both
on NTP. Negative or > 1 h values are ignored.
- The initial `scenario` event on connect replays the current state and carries
the original state-change time, not the send time. The script excludes that
first scenario event per VU; the presenter POST is the scenario sample for
`sse_scenario_latency`. Heartbeat pings always count toward
`sse_event_latency`.
- Each VU closes after 20 heartbeats (20 × 15 s = 5 min), so the run ends
cleanly instead of waiting for the 7-minute request timeout.
- One API replica means all clients share one SSE hub; a pod restart drops
every stream and clients reconnect after `retry: 3000`.
- k6 itself does not collect server CPU/memory. Use `kubectl top pods` (if
metrics-server is available) or watch the pod during the run.