feat: prepare v1.1 deploy and presenter load test
infra-ci / manifests (push) Failing after 4s
infra-ci / manifests (pull_request) Failing after 3s

- build and push v1.1 images from the service main worktrees
- switch manifests to :v1.1; add web control Secret refs
- configure API BOOTSTRAP_REFRESH=60s and remove unused DATA_DIR
- add k6 presenter scenario with a mid-run control POST
- require control credentials in make load; update runbook and drills
This commit is contained in:
Noel Sabia committed 2026-11-14 08:00:58 +01:00
1 parent 40bc8e47d0
commit b98a1bab42
12 files changed
+264 -117

No files matched your search

+45 -27
View File
@@ -1,7 +1,9 @@
# k6 load test
`k6/sse.js` holds 120 concurrent Server-Sent Events connections for 5 minutes
and reports connection count plus p95 event latency.
and reports connection count plus p95 event latency. A separate `presenter`
scenario POSTs one scenario change at 2m30s so the test measures event
propagation through Cloudflare, not just heartbeat latency.
## Why a custom k6 binary
@@ -36,62 +38,77 @@ A stock `grafana/k6` image is **not** a fallback: it lacks the SSE extension.
You should see `retry: 3000` and a `scenario` event immediately, then a
`ping` every 15 s.
2. Run the load test:
2. Load the control credentials and run the load test. The presenter scenario
requires `CONTROL_USER` and `CONTROL_PASS`; `make load` refuses to start
without them.
```bash
set -a
source ~/.config/pulsecast/control-auth.env
set +a
make load
# or: API_URL=https://api-pulse.noelsabia.dev ./k6/k6-sse run k6/sse.js
# or: API_URL=https://api-pulse.noelsabia.dev \
# CONTROL_USER="$CONTROL_USER" CONTROL_PASS="$CONTROL_PASS" \
# ./k6/k6-sse run k6/sse.js
```
`make load` writes `k6/results/summary-<timestamp>.json` (git-ignored).
3. During the run, post a scenario change so the scenario-event latency has
samples and the propagation path is exercised:
3. The `presenter` scenario starts at `2m30s`, POSTs
`PRESENTER_SCENARIO=rainy-saturday` once to `/api/scenario`, and records the
request duration plus success. All 120 SSE clients must receive the
resulting `scenario` event. Override the timing/scenario if needed:
```bash
curl -u "$CONTROL_USER:$CONTROL_PASS" -X POST \
-H 'Content-Type: application/json' \
-d '{"scenario":"sunny-saturday-market","weather":{"temp_delta_c":4,"precip_mm":0}}' \
https://api-pulse.noelsabia.dev/api/scenario
make load PRESENTER_AT=3m PRESENTER_SCENARIO=sunny-saturday-market
```
Repeat every 30–60 s with different scenarios if you want a distribution.
Without a change, `sse_event_latency` is measured from heartbeat `ping`
events, which still proves the stream is live.
4. Record these numbers in the paper/README from the saved summary:
- `sse_connections_opened` count (must be 120)
- `sse_connections_closed` count (must equal connections opened)
- `sse_connections_closed` count (must be >= 120)
- `sse_open_success` rate (must be >= 99%)
- `sse_event_latency` p95 (target < 2000 ms)
- `sse_scenario_latency` p95 (target < 2000 ms)
- `presenter_post_success` rate (must be 100%)
- `presenter_post_duration` (record p95/max)
- `sse_connection_errors` (target 0)
- `sse_events_received` total
## Local validation before the cluster run
The script was validated locally on 2026-11-14 with the pinned k6 v2.0.0 build
against a mock SSE stream and the real API image (3 VUs, 1 heartbeat each):
connections opened and closed cleanly, open success 100%. The real 120-client
run still requires the deployed API, because localhost results say nothing
about Cloudflare buffering or SKE scheduling.
against a mock SSE stream and the real v1.1 API image (3 VUs, 1 heartbeat
each): connections opened and closed cleanly, open success 100%, presenter
POST 200, and scenario latency sampled. The real 120-client run still requires
the deployed API, because localhost results say nothing about Cloudflare
buffering or SKE scheduling.
## Pass criteria
| Metric | Target |
|---|---|
| Connections opened | 120 |
| Connections closed (stream ended) | 120 |
| Connections closed (stream ended) | >= 120 |
| Open success rate | >= 99% |
| p95 event latency | < 2000 ms |
| p95 scenario latency | < 2000 ms |
| Presenter POST success | 100% |
| Unexpected connection errors | 0 |
| Duration | 5 minutes |
The k6 thresholds enforce connections opened, open success, and p95 latency;
the others are read from the summary. A final `sse_connections_closed` below
120 means at least one opened VU did not finish; inspect
`sse_connection_errors` for unexpected errors. A p95 above 2 s means Cloudflare, the Go SSE hub, or the
client is buffering/delaying events; check `X-Accel-Buffering: no`, the 15 s
heartbeat, and whether the connection is HTTP/2-buffered at the edge.
The k6 thresholds enforce connections opened/closed, open success, p95 event
latency, p95 scenario latency, presenter success, and zero connection errors.
A p95 above 2 s means Cloudflare, the Go SSE hub, or the client is
buffering/delaying events; check `X-Accel-Buffering: no`, the 15 s heartbeat,
and whether the connection is HTTP/2-buffered at the edge.
## Results
Populate from the saved `k6/results/summary-*.json`; never estimate.
| Run | Date | Opened | Closed | Open success | p95 event | p95 scenario | Presenter success | Errors |
|---|---|---|---|---|---|---|---|---|
| v1.1 through Cloudflare | pending post-merge run | | | | | | | |
## Notes
@@ -100,8 +117,9 @@ heartbeat, and whether the connection is HTTP/2-buffered at the edge.
on NTP. Negative or > 1 h values are ignored.
- The initial `scenario` event on connect replays the current state and carries
the original state-change time, not the send time. The script excludes that
first scenario event per VU; only scenario changes broadcast during the run
are sampled by `sse_scenario_latency`. Heartbeat pings always count.
first scenario event per VU; the presenter POST is the scenario sample for
`sse_scenario_latency`. Heartbeat pings always count toward
`sse_event_latency`.
- Each VU closes after 20 heartbeats (20 × 15 s = 5 min), so the run ends
cleanly instead of waiting for the 7-minute request timeout.
- One API replica means all clients share one SSE hub; a pod restart drops