feat: prepare v1.1 deploy and presenter load test
- build and push v1.1 images from the service main worktrees - switch manifests to :v1.1; add web control Secret refs - configure API BOOTSTRAP_REFRESH=60s and remove unused DATA_DIR - add k6 presenter scenario with a mid-run control POST - require control credentials in make load; update runbook and drills
This commit is contained in:
1 parent
40bc8e47d0
commit
b98a1bab42
12 files changed
+264
-117
No files matched your search
+45
-27
@@ -1,7 +1,9 @@
|
||||
# k6 load test
|
||||
|
||||
`k6/sse.js` holds 120 concurrent Server-Sent Events connections for 5 minutes
|
||||
and reports connection count plus p95 event latency.
|
||||
and reports connection count plus p95 event latency. A separate `presenter`
|
||||
scenario POSTs one scenario change at 2m30s so the test measures event
|
||||
propagation through Cloudflare, not just heartbeat latency.
|
||||
|
||||
## Why a custom k6 binary
|
||||
|
||||
@@ -36,62 +38,77 @@ A stock `grafana/k6` image is **not** a fallback: it lacks the SSE extension.
|
||||
You should see `retry: 3000` and a `scenario` event immediately, then a
|
||||
`ping` every 15 s.
|
||||
|
||||
2. Run the load test:
|
||||
2. Load the control credentials and run the load test. The presenter scenario
|
||||
requires `CONTROL_USER` and `CONTROL_PASS`; `make load` refuses to start
|
||||
without them.
|
||||
|
||||
```bash
|
||||
set -a
|
||||
source ~/.config/pulsecast/control-auth.env
|
||||
set +a
|
||||
make load
|
||||
# or: API_URL=https://api-pulse.noelsabia.dev ./k6/k6-sse run k6/sse.js
|
||||
# or: API_URL=https://api-pulse.noelsabia.dev \
|
||||
# CONTROL_USER="$CONTROL_USER" CONTROL_PASS="$CONTROL_PASS" \
|
||||
# ./k6/k6-sse run k6/sse.js
|
||||
```
|
||||
|
||||
`make load` writes `k6/results/summary-<timestamp>.json` (git-ignored).
|
||||
|
||||
3. During the run, post a scenario change so the scenario-event latency has
|
||||
samples and the propagation path is exercised:
|
||||
3. The `presenter` scenario starts at `2m30s`, POSTs
|
||||
`PRESENTER_SCENARIO=rainy-saturday` once to `/api/scenario`, and records the
|
||||
request duration plus success. All 120 SSE clients must receive the
|
||||
resulting `scenario` event. Override the timing/scenario if needed:
|
||||
|
||||
```bash
|
||||
curl -u "$CONTROL_USER:$CONTROL_PASS" -X POST \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"scenario":"sunny-saturday-market","weather":{"temp_delta_c":4,"precip_mm":0}}' \
|
||||
https://api-pulse.noelsabia.dev/api/scenario
|
||||
make load PRESENTER_AT=3m PRESENTER_SCENARIO=sunny-saturday-market
|
||||
```
|
||||
|
||||
Repeat every 30–60 s with different scenarios if you want a distribution.
|
||||
Without a change, `sse_event_latency` is measured from heartbeat `ping`
|
||||
events, which still proves the stream is live.
|
||||
|
||||
4. Record these numbers in the paper/README from the saved summary:
|
||||
- `sse_connections_opened` count (must be 120)
|
||||
- `sse_connections_closed` count (must equal connections opened)
|
||||
- `sse_connections_closed` count (must be >= 120)
|
||||
- `sse_open_success` rate (must be >= 99%)
|
||||
- `sse_event_latency` p95 (target < 2000 ms)
|
||||
- `sse_scenario_latency` p95 (target < 2000 ms)
|
||||
- `presenter_post_success` rate (must be 100%)
|
||||
- `presenter_post_duration` (record p95/max)
|
||||
- `sse_connection_errors` (target 0)
|
||||
- `sse_events_received` total
|
||||
|
||||
## Local validation before the cluster run
|
||||
|
||||
The script was validated locally on 2026-11-14 with the pinned k6 v2.0.0 build
|
||||
against a mock SSE stream and the real API image (3 VUs, 1 heartbeat each):
|
||||
connections opened and closed cleanly, open success 100%. The real 120-client
|
||||
run still requires the deployed API, because localhost results say nothing
|
||||
about Cloudflare buffering or SKE scheduling.
|
||||
against a mock SSE stream and the real v1.1 API image (3 VUs, 1 heartbeat
|
||||
each): connections opened and closed cleanly, open success 100%, presenter
|
||||
POST 200, and scenario latency sampled. The real 120-client run still requires
|
||||
the deployed API, because localhost results say nothing about Cloudflare
|
||||
buffering or SKE scheduling.
|
||||
|
||||
## Pass criteria
|
||||
|
||||
| Metric | Target |
|
||||
|---|---|
|
||||
| Connections opened | 120 |
|
||||
| Connections closed (stream ended) | 120 |
|
||||
| Connections closed (stream ended) | >= 120 |
|
||||
| Open success rate | >= 99% |
|
||||
| p95 event latency | < 2000 ms |
|
||||
| p95 scenario latency | < 2000 ms |
|
||||
| Presenter POST success | 100% |
|
||||
| Unexpected connection errors | 0 |
|
||||
| Duration | 5 minutes |
|
||||
|
||||
The k6 thresholds enforce connections opened, open success, and p95 latency;
|
||||
the others are read from the summary. A final `sse_connections_closed` below
|
||||
120 means at least one opened VU did not finish; inspect
|
||||
`sse_connection_errors` for unexpected errors. A p95 above 2 s means Cloudflare, the Go SSE hub, or the
|
||||
client is buffering/delaying events; check `X-Accel-Buffering: no`, the 15 s
|
||||
heartbeat, and whether the connection is HTTP/2-buffered at the edge.
|
||||
The k6 thresholds enforce connections opened/closed, open success, p95 event
|
||||
latency, p95 scenario latency, presenter success, and zero connection errors.
|
||||
A p95 above 2 s means Cloudflare, the Go SSE hub, or the client is
|
||||
buffering/delaying events; check `X-Accel-Buffering: no`, the 15 s heartbeat,
|
||||
and whether the connection is HTTP/2-buffered at the edge.
|
||||
|
||||
## Results
|
||||
|
||||
Populate from the saved `k6/results/summary-*.json`; never estimate.
|
||||
|
||||
| Run | Date | Opened | Closed | Open success | p95 event | p95 scenario | Presenter success | Errors |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| v1.1 through Cloudflare | pending post-merge run | | | | | | | |
|
||||
|
||||
## Notes
|
||||
|
||||
@@ -100,8 +117,9 @@ heartbeat, and whether the connection is HTTP/2-buffered at the edge.
|
||||
on NTP. Negative or > 1 h values are ignored.
|
||||
- The initial `scenario` event on connect replays the current state and carries
|
||||
the original state-change time, not the send time. The script excludes that
|
||||
first scenario event per VU; only scenario changes broadcast during the run
|
||||
are sampled by `sse_scenario_latency`. Heartbeat pings always count.
|
||||
first scenario event per VU; the presenter POST is the scenario sample for
|
||||
`sse_scenario_latency`. Heartbeat pings always count toward
|
||||
`sse_event_latency`.
|
||||
- Each VU closes after 20 heartbeats (20 × 15 s = 5 min), so the run ends
|
||||
cleanly instead of waiting for the 7-minute request timeout.
|
||||
- One API replica means all clients share one SSE hub; a pod restart drops
|
||||
|
||||
Reference in new issue
Block a user