Files
infra-k8/docs/load-test.md
T
Noel Sabia b98a1bab42
infra-ci / manifests (push) Failing after 4s
infra-ci / manifests (pull_request) Failing after 3s
feat: prepare v1.1 deploy and presenter load test
- build and push v1.1 images from the service main worktrees
- switch manifests to :v1.1; add web control Secret refs
- configure API BOOTSTRAP_REFRESH=60s and remove unused DATA_DIR
- add k6 presenter scenario with a mid-run control POST
- require control credentials in make load; update runbook and drills
2026-11-14 08:00:58 +01:00

5.1 KiB
Raw Blame History

k6 load test

k6/sse.js holds 120 concurrent Server-Sent Events connections for 5 minutes and reports connection count plus p95 event latency. A separate presenter scenario POSTs one scenario change at 2m30s so the test measures event propagation through Cloudflare, not just heartbeat latency.

Why a custom k6 binary

Native k6 buffers a complete HTTP response, so it cannot parse a stream that never ends. The script imports the community k6/x/sse extension (github.com/phymbert/xk6-sse), which a stock k6 binary does not include.

Build the pinned binary once:

make k6-build          # creates k6/k6-sse (git-ignored)
./k6/k6-sse version    # expect k6 v2.0.0 + xk6-sse v0.2.0

The build pins k6 v2.0.0. On k6 v2.4.0 the same xk6-sse v0.2.0 build fails at runtime with TypeError: Object has no member 'on' (the extension's client methods are not exposed); v2.0.0 was verified locally on 2026-11-14. If you bump K6_BUILD_VERSION, rerun the script against a test stream first.

make load uses, in order: K6 if set, k6/k6-sse, then k6 on PATH. A stock grafana/k6 image is not a fallback: it lacks the SSE extension.

Procedure

  1. Confirm the API is healthy and the SSE route works:

    curl -sS https://api-pulse.noelsabia.dev/api/healthz
    curl -sS -N --max-time 20 https://api-pulse.noelsabia.dev/api/stream
    

    You should see retry: 3000 and a scenario event immediately, then a ping every 15 s.

  2. Load the control credentials and run the load test. The presenter scenario requires CONTROL_USER and CONTROL_PASS; make load refuses to start without them.

    set -a
    source ~/.config/pulsecast/control-auth.env
    set +a
    make load
    # or: API_URL=https://api-pulse.noelsabia.dev \
    #     CONTROL_USER="$CONTROL_USER" CONTROL_PASS="$CONTROL_PASS" \
    #     ./k6/k6-sse run k6/sse.js
    

    make load writes k6/results/summary-<timestamp>.json (git-ignored).

  3. The presenter scenario starts at 2m30s, POSTs PRESENTER_SCENARIO=rainy-saturday once to /api/scenario, and records the request duration plus success. All 120 SSE clients must receive the resulting scenario event. Override the timing/scenario if needed:

    make load PRESENTER_AT=3m PRESENTER_SCENARIO=sunny-saturday-market
    
  4. Record these numbers in the paper/README from the saved summary:

    • sse_connections_opened count (must be 120)
    • sse_connections_closed count (must be >= 120)
    • sse_open_success rate (must be >= 99%)
    • sse_event_latency p95 (target < 2000 ms)
    • sse_scenario_latency p95 (target < 2000 ms)
    • presenter_post_success rate (must be 100%)
    • presenter_post_duration (record p95/max)
    • sse_connection_errors (target 0)
    • sse_events_received total

Local validation before the cluster run

The script was validated locally on 2026-11-14 with the pinned k6 v2.0.0 build against a mock SSE stream and the real v1.1 API image (3 VUs, 1 heartbeat each): connections opened and closed cleanly, open success 100%, presenter POST 200, and scenario latency sampled. The real 120-client run still requires the deployed API, because localhost results say nothing about Cloudflare buffering or SKE scheduling.

Pass criteria

Metric Target
Connections opened 120
Connections closed (stream ended) >= 120
Open success rate >= 99%
p95 event latency < 2000 ms
p95 scenario latency < 2000 ms
Presenter POST success 100%
Unexpected connection errors 0
Duration 5 minutes

The k6 thresholds enforce connections opened/closed, open success, p95 event latency, p95 scenario latency, presenter success, and zero connection errors. A p95 above 2 s means Cloudflare, the Go SSE hub, or the client is buffering/delaying events; check X-Accel-Buffering: no, the 15 s heartbeat, and whether the connection is HTTP/2-buffered at the edge.

Results

Populate from the saved k6/results/summary-*.json; never estimate.

Run Date Opened Closed Open success p95 event p95 scenario Presenter success Errors
v1.1 through Cloudflare pending post-merge run

Notes

  • sse_event_latency uses the server timestamp in data.t (ping) or data.at (scenario). It includes clock skew between the k6 host and the pod; keep both on NTP. Negative or > 1 h values are ignored.
  • The initial scenario event on connect replays the current state and carries the original state-change time, not the send time. The script excludes that first scenario event per VU; the presenter POST is the scenario sample for sse_scenario_latency. Heartbeat pings always count toward sse_event_latency.
  • Each VU closes after 20 heartbeats (20 × 15 s = 5 min), so the run ends cleanly instead of waiting for the 7-minute request timeout.
  • One API replica means all clients share one SSE hub; a pod restart drops every stream and clients reconnect after retry: 3000.
  • k6 itself does not collect server CPU/memory. Use kubectl top pods (if metrics-server is available) or watch the pod during the run.