Files
infra-k8/docs/load-test.md
T

92 lines
3.2 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# k6 load test
`k6/sse.js` holds 120 concurrent Server-Sent Events connections for 5 minutes
and reports connection count plus p95 event latency.
## Why a custom k6 binary
Native k6 buffers a complete HTTP response, so it cannot parse a stream that
never ends. The script imports the community `k6/x/sse` extension
(`github.com/phymbert/xk6-sse`). Newer k6 releases resolve community extensions
automatically. If the stock `grafana/k6` image rejects the import, build a
binary with the extension:
```bash
make k6-build # creates k6/k6-sse (git-ignored)
K6=./k6/k6-sse make load
```
`make load` uses, in order: `k6` on `PATH`, `k6/k6-sse`, then
`docker run --rm -i grafana/k6`.
## Procedure
1. Confirm the API is healthy and the SSE route works:
```bash
curl -sS https://api.pulse.noelsabia.dev/api/healthz
curl -sS -N --max-time 20 https://api.pulse.noelsabia.dev/api/stream
```
You should see `retry: 3000` and a `scenario` event immediately, then a
`ping` every 15 s.
2. Run the load test:
```bash
make load
# or: API_URL=https://api.pulse.noelsabia.dev k6 run k6/sse.js
```
`make load` writes `k6/results/summary-<timestamp>.json` (git-ignored).
3. During the run, post a scenario change so the scenario-event latency has
samples and the propagation path is exercised:
```bash
curl -u "$CONTROL_USER:$CONTROL_PASS" -X POST \
-H 'Content-Type: application/json' \
-d '{"scenario":"sunny-saturday-market","weather":{"temp_delta_c":4,"precip_mm":0}}' \
https://api.pulse.noelsabia.dev/api/scenario
```
Repeat every 30–60 s with different scenarios if you want a distribution.
Without a change, `sse_event_latency` is measured from heartbeat `ping`
events, which still proves the stream is live.
4. Record these numbers in the paper/README from the saved summary:
- `sse_connections_opened` count (must be 120)
- `sse_connections_active` max (must reach 120)
- `sse_open_success` rate (must be >= 99%)
- `sse_event_latency` p95 (target < 2000 ms)
- `sse_connection_errors` (target 0)
- `sse_events_received` total
## Pass criteria
| Metric | Target |
|---|---|
| Connections opened | 120 |
| Peak active connections | 120 |
| Open success rate | >= 99% |
| p95 event latency | < 2000 ms |
| Unexpected connection errors | 0 |
| Duration | 5 minutes |
The k6 thresholds enforce the first, third, and fourth; the others are read
from the summary. A p95 above 2 s means Cloudflare, the Go SSE hub, or the
client is buffering/delaying events; check `X-Accel-Buffering: no`, the 15 s
heartbeat, and whether the connection is HTTP/2-buffered at the edge.
## Notes
- `sse_event_latency` uses the server timestamp in `data.t` (ping) or `data.at`
(scenario). It includes clock skew between the k6 host and the pod; keep both
on NTP. Negative or > 1 h values are ignored.
- Each VU closes after 20 heartbeats (20 × 15 s = 5 min), so the run ends
cleanly instead of waiting for the 7-minute request timeout.
- One API replica means all clients share one SSE hub; a pod restart drops
every stream and clients reconnect after `retry: 3000`.
- k6 itself does not collect server CPU/memory. Use `kubectl top pods` (if
metrics-server is available) or watch the pod during the run.