Files
infra-k8/docs/load-test.md
Noel Sabia e8e048866c
infra-ci / manifests (push) Successful in 8s
infra-ci / manifests (pull_request) Successful in 8s
ci: run manifest checks on the Gitea runner
The node:20-bookworm runner has python3 but no pip, so the PyYAML install failed before any check ran. Install python3-yaml with apt instead, remove the duplicate cluster-only kubectl step, and keep the two cluster-free Make targets.

Also remove the stale v1.0/pending-review claims, drop the old api.pulse hostname, and move the deploy status below the runbook.
2026-11-14 09:13:27 +01:00

5.9 KiB
Raw Permalink Blame History

k6 load test

k6/sse.js holds 120 concurrent Server-Sent Events connections for 5 minutes and reports connection count plus p95 event latency. A separate presenter scenario POSTs one scenario change at 2m30s so the test measures event propagation through Cloudflare, not just heartbeat latency.

Why a custom k6 binary

Native k6 buffers a complete HTTP response, so it cannot parse a stream that never ends. The script imports the community k6/x/sse extension (github.com/phymbert/xk6-sse), which a stock k6 binary does not include.

Build the pinned binary once:

make k6-build          # creates k6/k6-sse (git-ignored)
./k6/k6-sse version    # expect k6 v2.0.0 + the current xk6-sse

The build pins k6 v2.0.0; xk6-sse v0.2.0 is the version used for the recorded runs. On k6 v2.4.0 that extension build fails at runtime with TypeError: Object has no member 'on' (the client methods are not exposed); v2.0.0 was verified locally on 2026-11-14. If you bump K6_BUILD_VERSION, rerun the script against a test stream first.

make load uses, in order: K6 if set, k6/k6-sse, then k6 on PATH. A stock grafana/k6 image is not a fallback: it lacks the SSE extension.

Procedure

  1. Confirm the API is healthy and the SSE route works:

    curl -sS https://api-pulse.noelsabia.dev/api/healthz
    curl -sS -N --max-time 20 https://api-pulse.noelsabia.dev/api/stream
    

    You should see retry: 3000 and a scenario event immediately, then a ping every 15 s.

  2. Load the control credentials and run the load test. The presenter scenario requires CONTROL_USER and CONTROL_PASS; make load refuses to start without them.

    set -a
    source ~/.config/pulsecast/control-auth.env
    set +a
    make load
    # or: API_URL=https://api-pulse.noelsabia.dev \
    #     CONTROL_USER="$CONTROL_USER" CONTROL_PASS="$CONTROL_PASS" \
    #     ./k6/k6-sse run k6/sse.js
    

    make load writes k6/results/summary-<timestamp>.json (git-ignored).

  3. The presenter scenario starts at 2m30s, POSTs PRESENTER_SCENARIO=rainy-saturday once to /api/scenario, and records the request duration plus success. All 120 SSE clients must receive the resulting scenario event. Override the timing/scenario if needed:

    make load PRESENTER_AT=3m PRESENTER_SCENARIO=sunny-saturday-market
    
  4. Record these numbers in the paper/README from the saved summary:

    • sse_connections_opened count (must be 120)
    • sse_connections_closed count (must be >= 120)
    • sse_open_success rate (must be >= 99%)
    • sse_event_latency p95 (target < 2000 ms)
    • sse_scenario_latency p95 (target < 2000 ms)
    • presenter_post_success rate (must be 100%)
    • presenter_post_duration (record p95/max)
    • sse_connection_errors (target 0)
    • sse_events_received total

Local validation before the cluster run

The script was validated locally on 2026-11-14 with the pinned k6 v2.0.0 build against a mock SSE stream and the real v1.1 API image (3 VUs, 1 heartbeat each): connections opened and closed cleanly, open success 100%, presenter POST 200, and scenario latency sampled. Run the 120-client test against the deployed API; localhost says nothing about Cloudflare buffering or SKE scheduling.

Pass criteria

Metric Target
Connections opened 120
Connections closed (stream ended) >= 120
Open success rate >= 99%
p95 event latency < 2000 ms
p95 scenario latency < 2000 ms
Presenter POST success 100%
Unexpected connection errors 0
Duration 5 minutes

The k6 thresholds enforce connections opened/closed, open success, p95 event latency, p95 scenario latency, presenter success, and zero connection errors. A p95 above 2 s means Cloudflare, the Go SSE hub, or the client is buffering/delaying events; check X-Accel-Buffering: no, the 15 s heartbeat, and whether the connection is HTTP/2-buffered at the edge.

Results

Populate from the saved k6/results/summary-*.json; never estimate.

Run Date Opened Closed Open success p95 event p95 scenario Presenter success Errors
v1.0 baseline through Cloudflare 2026-11-14 08:07 CET 120 120 100% 1.03 s 194 ms 100% (82.69 ms) 0
v1.1 through Cloudflare 2026-11-14 08:21 CET 120 120 100% 331 ms 646 ms 100% (49.60 ms) 0

Raw baseline result: k6/results/summary-20261114T080747.json (git-ignored). Raw v1.1 result: k6/results/summary-20261114T082121.json (git-ignored).

Baseline details: 361/361 checks passed, 2,760 events received, 2,400 heartbeat pings, 120 VUs, test duration 4m53s, presenter POST 82.69 ms. p95 ping latency was 1,045 ms.

v1.1 details: 361/361 checks passed, 2,760 events received, 2,400 heartbeat pings, 120 VUs, test duration 4m57.9s, presenter POST 49.60 ms, p95 ping latency 268 ms. All thresholds passed. This run used the presenter POST at 2m30s through Cloudflare and the same make load path.

Notes

  • sse_event_latency uses the server timestamp in data.t (ping) or data.at (scenario). It includes clock skew between the k6 host and the pod; keep both on NTP. Negative or > 1 h values are ignored.
  • The initial scenario event on connect replays the current state and carries the original state-change time, not the send time. The script excludes that first scenario event per VU; the presenter POST is the scenario sample for sse_scenario_latency. Heartbeat pings always count toward sse_event_latency.
  • Each VU closes after 20 heartbeats (20 × 15 s = 5 min), so the run ends cleanly instead of waiting for the 7-minute request timeout.
  • One API replica means all clients share one SSE hub; a pod restart drops every stream and clients reconnect after retry: 3000.
  • k6 itself does not collect server CPU/memory. Use kubectl top pods (if metrics-server is available) or watch the pod during the run.