The node:20-bookworm runner has python3 but no pip, so the PyYAML install failed before any check ran. Install python3-yaml with apt instead, remove the duplicate cluster-only kubectl step, and keep the two cluster-free Make targets. Also remove the stale v1.0/pending-review claims, drop the old api.pulse hostname, and move the deploy status below the runbook.
143 lines
5.9 KiB
Markdown
143 lines
5.9 KiB
Markdown
# k6 load test
|
||
|
||
`k6/sse.js` holds 120 concurrent Server-Sent Events connections for 5 minutes
|
||
and reports connection count plus p95 event latency. A separate `presenter`
|
||
scenario POSTs one scenario change at 2m30s so the test measures event
|
||
propagation through Cloudflare, not just heartbeat latency.
|
||
|
||
## Why a custom k6 binary
|
||
|
||
Native k6 buffers a complete HTTP response, so it cannot parse a stream that
|
||
never ends. The script imports the community `k6/x/sse` extension
|
||
(`github.com/phymbert/xk6-sse`), which a stock k6 binary does not include.
|
||
|
||
Build the pinned binary once:
|
||
|
||
```bash
|
||
make k6-build # creates k6/k6-sse (git-ignored)
|
||
./k6/k6-sse version # expect k6 v2.0.0 + the current xk6-sse
|
||
```
|
||
|
||
The build pins **k6 v2.0.0**; `xk6-sse` v0.2.0 is the version used for the
|
||
recorded runs. On k6 v2.4.0 that extension build fails at runtime with
|
||
`TypeError: Object has no member 'on'` (the client methods are not exposed);
|
||
v2.0.0 was verified locally on 2026-11-14. If you bump `K6_BUILD_VERSION`,
|
||
rerun the script against a test stream first.
|
||
|
||
`make load` uses, in order: `K6` if set, `k6/k6-sse`, then `k6` on `PATH`.
|
||
A stock `grafana/k6` image is **not** a fallback: it lacks the SSE extension.
|
||
|
||
## Procedure
|
||
|
||
1. Confirm the API is healthy and the SSE route works:
|
||
|
||
```bash
|
||
curl -sS https://api-pulse.noelsabia.dev/api/healthz
|
||
curl -sS -N --max-time 20 https://api-pulse.noelsabia.dev/api/stream
|
||
```
|
||
|
||
You should see `retry: 3000` and a `scenario` event immediately, then a
|
||
`ping` every 15 s.
|
||
|
||
2. Load the control credentials and run the load test. The presenter scenario
|
||
requires `CONTROL_USER` and `CONTROL_PASS`; `make load` refuses to start
|
||
without them.
|
||
|
||
```bash
|
||
set -a
|
||
source ~/.config/pulsecast/control-auth.env
|
||
set +a
|
||
make load
|
||
# or: API_URL=https://api-pulse.noelsabia.dev \
|
||
# CONTROL_USER="$CONTROL_USER" CONTROL_PASS="$CONTROL_PASS" \
|
||
# ./k6/k6-sse run k6/sse.js
|
||
```
|
||
|
||
`make load` writes `k6/results/summary-<timestamp>.json` (git-ignored).
|
||
|
||
3. The `presenter` scenario starts at `2m30s`, POSTs
|
||
`PRESENTER_SCENARIO=rainy-saturday` once to `/api/scenario`, and records the
|
||
request duration plus success. All 120 SSE clients must receive the
|
||
resulting `scenario` event. Override the timing/scenario if needed:
|
||
|
||
```bash
|
||
make load PRESENTER_AT=3m PRESENTER_SCENARIO=sunny-saturday-market
|
||
```
|
||
|
||
4. Record these numbers in the paper/README from the saved summary:
|
||
- `sse_connections_opened` count (must be 120)
|
||
- `sse_connections_closed` count (must be >= 120)
|
||
- `sse_open_success` rate (must be >= 99%)
|
||
- `sse_event_latency` p95 (target < 2000 ms)
|
||
- `sse_scenario_latency` p95 (target < 2000 ms)
|
||
- `presenter_post_success` rate (must be 100%)
|
||
- `presenter_post_duration` (record p95/max)
|
||
- `sse_connection_errors` (target 0)
|
||
- `sse_events_received` total
|
||
|
||
## Local validation before the cluster run
|
||
|
||
The script was validated locally on 2026-11-14 with the pinned k6 v2.0.0 build
|
||
against a mock SSE stream and the real v1.1 API image (3 VUs, 1 heartbeat
|
||
each): connections opened and closed cleanly, open success 100%, presenter
|
||
POST 200, and scenario latency sampled. Run the 120-client test against the
|
||
deployed API; localhost says nothing about Cloudflare buffering or SKE
|
||
scheduling.
|
||
|
||
## Pass criteria
|
||
|
||
| Metric | Target |
|
||
|---|---|
|
||
| Connections opened | 120 |
|
||
| Connections closed (stream ended) | >= 120 |
|
||
| Open success rate | >= 99% |
|
||
| p95 event latency | < 2000 ms |
|
||
| p95 scenario latency | < 2000 ms |
|
||
| Presenter POST success | 100% |
|
||
| Unexpected connection errors | 0 |
|
||
| Duration | 5 minutes |
|
||
|
||
The k6 thresholds enforce connections opened/closed, open success, p95 event
|
||
latency, p95 scenario latency, presenter success, and zero connection errors.
|
||
A p95 above 2 s means Cloudflare, the Go SSE hub, or the client is
|
||
buffering/delaying events; check `X-Accel-Buffering: no`, the 15 s heartbeat,
|
||
and whether the connection is HTTP/2-buffered at the edge.
|
||
|
||
## Results
|
||
|
||
Populate from the saved `k6/results/summary-*.json`; never estimate.
|
||
|
||
| Run | Date | Opened | Closed | Open success | p95 event | p95 scenario | Presenter success | Errors |
|
||
|---|---|---|---|---|---|---|---|---|
|
||
| v1.0 baseline through Cloudflare | 2026-11-14 08:07 CET | 120 | 120 | 100% | 1.03 s | 194 ms | 100% (82.69 ms) | 0 |
|
||
| v1.1 through Cloudflare | 2026-11-14 08:21 CET | 120 | 120 | 100% | 331 ms | 646 ms | 100% (49.60 ms) | 0 |
|
||
|
||
Raw baseline result: `k6/results/summary-20261114T080747.json` (git-ignored).
|
||
Raw v1.1 result: `k6/results/summary-20261114T082121.json` (git-ignored).
|
||
|
||
Baseline details: 361/361 checks passed, 2,760 events received, 2,400
|
||
heartbeat pings, 120 VUs, test duration 4m53s, presenter POST 82.69 ms.
|
||
`p95 ping latency` was 1,045 ms.
|
||
|
||
v1.1 details: 361/361 checks passed, 2,760 events received, 2,400 heartbeat
|
||
pings, 120 VUs, test duration 4m57.9s, presenter POST 49.60 ms, `p95 ping
|
||
latency` 268 ms. All thresholds passed. This run used the presenter POST at
|
||
2m30s through Cloudflare and the same `make load` path.
|
||
|
||
## Notes
|
||
|
||
- `sse_event_latency` uses the server timestamp in `data.t` (ping) or `data.at`
|
||
(scenario). It includes clock skew between the k6 host and the pod; keep both
|
||
on NTP. Negative or > 1 h values are ignored.
|
||
- The initial `scenario` event on connect replays the current state and carries
|
||
the original state-change time, not the send time. The script excludes that
|
||
first scenario event per VU; the presenter POST is the scenario sample for
|
||
`sse_scenario_latency`. Heartbeat pings always count toward
|
||
`sse_event_latency`.
|
||
- Each VU closes after 20 heartbeats (20 × 15 s = 5 min), so the run ends
|
||
cleanly instead of waiting for the 7-minute request timeout.
|
||
- One API replica means all clients share one SSE hub; a pod restart drops
|
||
every stream and clients reconnect after `retry: 3000`.
|
||
- k6 itself does not collect server CPU/memory. Use `kubectl top pods` (if
|
||
metrics-server is available) or watch the pod during the run.
|