Files
infra-k8/docs/load-test.md
Noel Sabia e8e048866c
infra-ci / manifests (push) Successful in 8s
infra-ci / manifests (pull_request) Successful in 8s
ci: run manifest checks on the Gitea runner
The node:20-bookworm runner has python3 but no pip, so the PyYAML install failed before any check ran. Install python3-yaml with apt instead, remove the duplicate cluster-only kubectl step, and keep the two cluster-free Make targets.

Also remove the stale v1.0/pending-review claims, drop the old api.pulse hostname, and move the deploy status below the runbook.
2026-11-14 09:13:27 +01:00

143 lines
5.9 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# k6 load test
`k6/sse.js` holds 120 concurrent Server-Sent Events connections for 5 minutes
and reports connection count plus p95 event latency. A separate `presenter`
scenario POSTs one scenario change at 2m30s so the test measures event
propagation through Cloudflare, not just heartbeat latency.
## Why a custom k6 binary
Native k6 buffers a complete HTTP response, so it cannot parse a stream that
never ends. The script imports the community `k6/x/sse` extension
(`github.com/phymbert/xk6-sse`), which a stock k6 binary does not include.
Build the pinned binary once:
```bash
make k6-build # creates k6/k6-sse (git-ignored)
./k6/k6-sse version # expect k6 v2.0.0 + the current xk6-sse
```
The build pins **k6 v2.0.0**; `xk6-sse` v0.2.0 is the version used for the
recorded runs. On k6 v2.4.0 that extension build fails at runtime with
`TypeError: Object has no member 'on'` (the client methods are not exposed);
v2.0.0 was verified locally on 2026-11-14. If you bump `K6_BUILD_VERSION`,
rerun the script against a test stream first.
`make load` uses, in order: `K6` if set, `k6/k6-sse`, then `k6` on `PATH`.
A stock `grafana/k6` image is **not** a fallback: it lacks the SSE extension.
## Procedure
1. Confirm the API is healthy and the SSE route works:
```bash
curl -sS https://api-pulse.noelsabia.dev/api/healthz
curl -sS -N --max-time 20 https://api-pulse.noelsabia.dev/api/stream
```
You should see `retry: 3000` and a `scenario` event immediately, then a
`ping` every 15 s.
2. Load the control credentials and run the load test. The presenter scenario
requires `CONTROL_USER` and `CONTROL_PASS`; `make load` refuses to start
without them.
```bash
set -a
source ~/.config/pulsecast/control-auth.env
set +a
make load
# or: API_URL=https://api-pulse.noelsabia.dev \
# CONTROL_USER="$CONTROL_USER" CONTROL_PASS="$CONTROL_PASS" \
# ./k6/k6-sse run k6/sse.js
```
`make load` writes `k6/results/summary-<timestamp>.json` (git-ignored).
3. The `presenter` scenario starts at `2m30s`, POSTs
`PRESENTER_SCENARIO=rainy-saturday` once to `/api/scenario`, and records the
request duration plus success. All 120 SSE clients must receive the
resulting `scenario` event. Override the timing/scenario if needed:
```bash
make load PRESENTER_AT=3m PRESENTER_SCENARIO=sunny-saturday-market
```
4. Record these numbers in the paper/README from the saved summary:
- `sse_connections_opened` count (must be 120)
- `sse_connections_closed` count (must be >= 120)
- `sse_open_success` rate (must be >= 99%)
- `sse_event_latency` p95 (target < 2000 ms)
- `sse_scenario_latency` p95 (target < 2000 ms)
- `presenter_post_success` rate (must be 100%)
- `presenter_post_duration` (record p95/max)
- `sse_connection_errors` (target 0)
- `sse_events_received` total
## Local validation before the cluster run
The script was validated locally on 2026-11-14 with the pinned k6 v2.0.0 build
against a mock SSE stream and the real v1.1 API image (3 VUs, 1 heartbeat
each): connections opened and closed cleanly, open success 100%, presenter
POST 200, and scenario latency sampled. Run the 120-client test against the
deployed API; localhost says nothing about Cloudflare buffering or SKE
scheduling.
## Pass criteria
| Metric | Target |
|---|---|
| Connections opened | 120 |
| Connections closed (stream ended) | >= 120 |
| Open success rate | >= 99% |
| p95 event latency | < 2000 ms |
| p95 scenario latency | < 2000 ms |
| Presenter POST success | 100% |
| Unexpected connection errors | 0 |
| Duration | 5 minutes |
The k6 thresholds enforce connections opened/closed, open success, p95 event
latency, p95 scenario latency, presenter success, and zero connection errors.
A p95 above 2 s means Cloudflare, the Go SSE hub, or the client is
buffering/delaying events; check `X-Accel-Buffering: no`, the 15 s heartbeat,
and whether the connection is HTTP/2-buffered at the edge.
## Results
Populate from the saved `k6/results/summary-*.json`; never estimate.
| Run | Date | Opened | Closed | Open success | p95 event | p95 scenario | Presenter success | Errors |
|---|---|---|---|---|---|---|---|---|
| v1.0 baseline through Cloudflare | 2026-11-14 08:07 CET | 120 | 120 | 100% | 1.03 s | 194 ms | 100% (82.69 ms) | 0 |
| v1.1 through Cloudflare | 2026-11-14 08:21 CET | 120 | 120 | 100% | 331 ms | 646 ms | 100% (49.60 ms) | 0 |
Raw baseline result: `k6/results/summary-20261114T080747.json` (git-ignored).
Raw v1.1 result: `k6/results/summary-20261114T082121.json` (git-ignored).
Baseline details: 361/361 checks passed, 2,760 events received, 2,400
heartbeat pings, 120 VUs, test duration 4m53s, presenter POST 82.69 ms.
`p95 ping latency` was 1,045 ms.
v1.1 details: 361/361 checks passed, 2,760 events received, 2,400 heartbeat
pings, 120 VUs, test duration 4m57.9s, presenter POST 49.60 ms, `p95 ping
latency` 268 ms. All thresholds passed. This run used the presenter POST at
2m30s through Cloudflare and the same `make load` path.
## Notes
- `sse_event_latency` uses the server timestamp in `data.t` (ping) or `data.at`
(scenario). It includes clock skew between the k6 host and the pod; keep both
on NTP. Negative or > 1 h values are ignored.
- The initial `scenario` event on connect replays the current state and carries
the original state-change time, not the send time. The script excludes that
first scenario event per VU; the presenter POST is the scenario sample for
`sse_scenario_latency`. Heartbeat pings always count toward
`sse_event_latency`.
- Each VU closes after 20 heartbeats (20 × 15 s = 5 min), so the run ends
cleanly instead of waiting for the 7-minute request timeout.
- One API replica means all clients share one SSE hub; a pod restart drops
every stream and clients reconnect after `retry: 3000`.
- k6 itself does not collect server CPU/memory. Use `kubectl top pods` (if
metrics-server is available) or watch the pod during the run.