Files
infra-k8/docs/load-test.md
T
Noel Sabia e8e0f2035b fix: use certificate-safe api-pulse hostname
Cloudflare Universal SSL covers *.noelsabia.dev only one level deep.
api.pulse.noelsabia.dev is two levels and fails the certificate check.
- switch the API base/SSE URLs to api-pulse.noelsabia.dev
- update smoke, k6, runbook, and failure drills
- record the verified DNS/certificate diagnosis and local cache flush
2026-11-14 06:40:35 +01:00

4.4 KiB
Raw Blame History

k6 load test

k6/sse.js holds 120 concurrent Server-Sent Events connections for 5 minutes and reports connection count plus p95 event latency.

Why a custom k6 binary

Native k6 buffers a complete HTTP response, so it cannot parse a stream that never ends. The script imports the community k6/x/sse extension (github.com/phymbert/xk6-sse), which a stock k6 binary does not include.

Build the pinned binary once:

make k6-build          # creates k6/k6-sse (git-ignored)
./k6/k6-sse version    # expect k6 v2.0.0 + xk6-sse v0.2.0

The build pins k6 v2.0.0. On k6 v2.4.0 the same xk6-sse v0.2.0 build fails at runtime with TypeError: Object has no member 'on' (the extension's client methods are not exposed); v2.0.0 was verified locally on 2026-11-14. If you bump K6_BUILD_VERSION, rerun the script against a test stream first.

make load uses, in order: K6 if set, k6/k6-sse, then k6 on PATH. A stock grafana/k6 image is not a fallback: it lacks the SSE extension.

Procedure

  1. Confirm the API is healthy and the SSE route works:

    curl -sS https://api-pulse.noelsabia.dev/api/healthz
    curl -sS -N --max-time 20 https://api-pulse.noelsabia.dev/api/stream
    

    You should see retry: 3000 and a scenario event immediately, then a ping every 15 s.

  2. Run the load test:

    make load
    # or: API_URL=https://api-pulse.noelsabia.dev ./k6/k6-sse run k6/sse.js
    

    make load writes k6/results/summary-<timestamp>.json (git-ignored).

  3. During the run, post a scenario change so the scenario-event latency has samples and the propagation path is exercised:

    curl -u "$CONTROL_USER:$CONTROL_PASS" -X POST \
      -H 'Content-Type: application/json' \
      -d '{"scenario":"sunny-saturday-market","weather":{"temp_delta_c":4,"precip_mm":0}}' \
      https://api-pulse.noelsabia.dev/api/scenario
    

    Repeat every 30–60 s with different scenarios if you want a distribution. Without a change, sse_event_latency is measured from heartbeat ping events, which still proves the stream is live.

  4. Record these numbers in the paper/README from the saved summary:

    • sse_connections_opened count (must be 120)
    • sse_connections_closed count (must equal connections opened)
    • sse_open_success rate (must be >= 99%)
    • sse_event_latency p95 (target < 2000 ms)
    • sse_connection_errors (target 0)
    • sse_events_received total

Local validation before the cluster run

The script was validated locally on 2026-11-14 with the pinned k6 v2.0.0 build against a mock SSE stream and the real API image (3 VUs, 1 heartbeat each): connections opened and closed cleanly, open success 100%. The real 120-client run still requires the deployed API, because localhost results say nothing about Cloudflare buffering or SKE scheduling.

Pass criteria

Metric Target
Connections opened 120
Connections closed (stream ended) 120
Open success rate >= 99%
p95 event latency < 2000 ms
Unexpected connection errors 0
Duration 5 minutes

The k6 thresholds enforce connections opened, open success, and p95 latency; the others are read from the summary. A final sse_connections_closed below 120 means at least one opened VU did not finish; inspect sse_connection_errors for unexpected errors. A p95 above 2 s means Cloudflare, the Go SSE hub, or the client is buffering/delaying events; check X-Accel-Buffering: no, the 15 s heartbeat, and whether the connection is HTTP/2-buffered at the edge.

Notes

  • sse_event_latency uses the server timestamp in data.t (ping) or data.at (scenario). It includes clock skew between the k6 host and the pod; keep both on NTP. Negative or > 1 h values are ignored.
  • The initial scenario event on connect replays the current state and carries the original state-change time, not the send time. The script excludes that first scenario event per VU; only scenario changes broadcast during the run are sampled by sse_scenario_latency. Heartbeat pings always count.
  • Each VU closes after 20 heartbeats (20 × 15 s = 5 min), so the run ends cleanly instead of waiting for the 7-minute request timeout.
  • One API replica means all clients share one SSE hub; a pod restart drops every stream and clients reconnect after retry: 3000.
  • k6 itself does not collect server CPU/memory. Use kubectl top pods (if metrics-server is available) or watch the pod during the run.