feat: harden Gate 1 manifests and runbook
- add read-only make gate1-check: permissions, guardrails, dry-run, Secret keys - set numeric runAsUser/runAsGroup (SKE non-numeric user caveat), pin cloudflared and use HTTP/2 - align ML env with PULSECAST_* and wire CONTROL_TOKEN for API/ML training proxy - fix service worktree paths and smoke.sh executable bit - pin k6 v2.0.0 for xk6-sse, fix closed-connection metric and initial scenario latency - rewrite docs/deploy.md as an exact copy-paste Gate 1 checklist
This commit is contained in:
1 parent
55dfe8b616
commit
5b199f8fe3
13 files changed
+1231
-235
No files matched your search
+30
-11
@@ -7,17 +7,22 @@ and reports connection count plus p95 event latency.
|
||||
|
||||
Native k6 buffers a complete HTTP response, so it cannot parse a stream that
|
||||
never ends. The script imports the community `k6/x/sse` extension
|
||||
(`github.com/phymbert/xk6-sse`). Newer k6 releases resolve community extensions
|
||||
automatically. If the stock `grafana/k6` image rejects the import, build a
|
||||
binary with the extension:
|
||||
(`github.com/phymbert/xk6-sse`), which a stock k6 binary does not include.
|
||||
|
||||
Build the pinned binary once:
|
||||
|
||||
```bash
|
||||
make k6-build # creates k6/k6-sse (git-ignored)
|
||||
K6=./k6/k6-sse make load
|
||||
./k6/k6-sse version # expect k6 v2.0.0 + xk6-sse v0.2.0
|
||||
```
|
||||
|
||||
`make load` uses, in order: `k6` on `PATH`, `k6/k6-sse`, then
|
||||
`docker run --rm -i grafana/k6`.
|
||||
The build pins **k6 v2.0.0**. On k6 v2.4.0 the same `xk6-sse` v0.2.0 build
|
||||
fails at runtime with `TypeError: Object has no member 'on'` (the extension's
|
||||
client methods are not exposed); v2.0.0 was verified locally on 2026-11-14. If
|
||||
you bump `K6_BUILD_VERSION`, rerun the script against a test stream first.
|
||||
|
||||
`make load` uses, in order: `K6` if set, `k6/k6-sse`, then `k6` on `PATH`.
|
||||
A stock `grafana/k6` image is **not** a fallback: it lacks the SSE extension.
|
||||
|
||||
## Procedure
|
||||
|
||||
@@ -35,7 +40,7 @@ K6=./k6/k6-sse make load
|
||||
|
||||
```bash
|
||||
make load
|
||||
# or: API_URL=https://api.pulse.noelsabia.dev k6 run k6/sse.js
|
||||
# or: API_URL=https://api.pulse.noelsabia.dev ./k6/k6-sse run k6/sse.js
|
||||
```
|
||||
|
||||
`make load` writes `k6/results/summary-<timestamp>.json` (git-ignored).
|
||||
@@ -56,25 +61,35 @@ K6=./k6/k6-sse make load
|
||||
|
||||
4. Record these numbers in the paper/README from the saved summary:
|
||||
- `sse_connections_opened` count (must be 120)
|
||||
- `sse_connections_active` max (must reach 120)
|
||||
- `sse_connections_closed` count (must equal connections opened)
|
||||
- `sse_open_success` rate (must be >= 99%)
|
||||
- `sse_event_latency` p95 (target < 2000 ms)
|
||||
- `sse_connection_errors` (target 0)
|
||||
- `sse_events_received` total
|
||||
|
||||
## Local validation before the cluster run
|
||||
|
||||
The script was validated locally on 2026-11-14 with the pinned k6 v2.0.0 build
|
||||
against a mock SSE stream and the real API image (3 VUs, 1 heartbeat each):
|
||||
connections opened and closed cleanly, open success 100%. The real 120-client
|
||||
run still requires the deployed API, because localhost results say nothing
|
||||
about Cloudflare buffering or SKE scheduling.
|
||||
|
||||
## Pass criteria
|
||||
|
||||
| Metric | Target |
|
||||
|---|---|
|
||||
| Connections opened | 120 |
|
||||
| Peak active connections | 120 |
|
||||
| Connections closed (stream ended) | 120 |
|
||||
| Open success rate | >= 99% |
|
||||
| p95 event latency | < 2000 ms |
|
||||
| Unexpected connection errors | 0 |
|
||||
| Duration | 5 minutes |
|
||||
|
||||
The k6 thresholds enforce the first, third, and fourth; the others are read
|
||||
from the summary. A p95 above 2 s means Cloudflare, the Go SSE hub, or the
|
||||
The k6 thresholds enforce connections opened, open success, and p95 latency;
|
||||
the others are read from the summary. A final `sse_connections_closed` below
|
||||
120 means at least one opened VU did not finish; inspect
|
||||
`sse_connection_errors` for unexpected errors. A p95 above 2 s means Cloudflare, the Go SSE hub, or the
|
||||
client is buffering/delaying events; check `X-Accel-Buffering: no`, the 15 s
|
||||
heartbeat, and whether the connection is HTTP/2-buffered at the edge.
|
||||
|
||||
@@ -83,6 +98,10 @@ heartbeat, and whether the connection is HTTP/2-buffered at the edge.
|
||||
- `sse_event_latency` uses the server timestamp in `data.t` (ping) or `data.at`
|
||||
(scenario). It includes clock skew between the k6 host and the pod; keep both
|
||||
on NTP. Negative or > 1 h values are ignored.
|
||||
- The initial `scenario` event on connect replays the current state and carries
|
||||
the original state-change time, not the send time. The script excludes that
|
||||
first scenario event per VU; only scenario changes broadcast during the run
|
||||
are sampled by `sse_scenario_latency`. Heartbeat pings always count.
|
||||
- Each VU closes after 20 heartbeats (20 × 15 s = 5 min), so the run ends
|
||||
cleanly instead of waiting for the 7-minute request timeout.
|
||||
- One API replica means all clients share one SSE hub; a pod restart drops
|
||||
|
||||
Reference in new issue
Block a user