feat: prepare v1.1 deploy and presenter load test
- build and push v1.1 images from the service main worktrees - switch manifests to :v1.1; add web control Secret refs - configure API BOOTSTRAP_REFRESH=60s and remove unused DATA_DIR - add k6 presenter scenario with a mid-run control POST - require control credentials in make load; update runbook and drills
This commit is contained in:
1 parent
40bc8e47d0
commit
b98a1bab42
12 files changed
+264
-117
No files matched your search
+75
-54
@@ -24,7 +24,7 @@ Gate 1 passes only when **all** of these are true:
|
||||
Do not claim Gate 1 from a local `kubectl port-forward`, a green manifest
|
||||
dry-run, or a container that only runs on your laptop.
|
||||
|
||||
## Current external status (2026-11-14 ~06:50 CET)
|
||||
## Current external status (2026-11-14 ~08:00 CET)
|
||||
|
||||
Re-check before trusting this snapshot:
|
||||
|
||||
@@ -33,8 +33,8 @@ Re-check before trusting this snapshot:
|
||||
Open-Meteo, and `git.noelsabia.dev`.
|
||||
- **Tunnel active:** `pulsecast-gate1` has routes for `pulse.noelsabia.dev`
|
||||
and `api-pulse.noelsabia.dev`; `cloudflared` is `1/1` with four HTTP/2
|
||||
connections, and `https://pulse.noelsabia.dev` returns 200 from hello with
|
||||
valid TLS. `api-pulse` returns 502 until the API is deployed.
|
||||
connections. `api-pulse` serves the deployed API; `pulse` still points at
|
||||
the temporary hello Service until the web cutover.
|
||||
- **DNS works:** public resolvers return Cloudflare IPs for
|
||||
`pulse.noelsabia.dev` and `api-pulse.noelsabia.dev`. A local
|
||||
`systemd-resolved` negative cache can be cleared with
|
||||
@@ -42,20 +42,19 @@ Re-check before trusting this snapshot:
|
||||
- **API hostname is `api-pulse.noelsabia.dev`:** Universal SSL covers only
|
||||
`*.noelsabia.dev`; a two-level `api.pulse.noelsabia.dev` is not covered by
|
||||
that certificate.
|
||||
- **Gitea pull is proven (2026-11-14 ~06:16 CET):** `gitea-regcred` exists and
|
||||
the cluster pulled `git.noelsabia.dev/noel/hello:gate1` in 120 ms; the hello
|
||||
pod is `1/1 Running` and its Service returned HTTP 200. The Gitea owner is
|
||||
`noel`.
|
||||
- **Gitea pull is proven:** `gitea-regcred` exists; hello and the v1.0 service
|
||||
images pulled from Gitea. The Gitea owner is `noel`.
|
||||
- **v1.1 images pushed:** web/api/ml `:v1.1` were built from the service
|
||||
`main` worktrees, locally exercised under numeric UIDs/read-only root, and
|
||||
pushed. The cluster still runs the previous v1.0 images until the v1.1
|
||||
branch is reviewed/merged and explicitly applied.
|
||||
- **All three Secrets exist:** `gitea-regcred`, `cloudflared`, `control-auth`.
|
||||
- **QUIC/UDP 7844 is untested.** TCP/443 to Cloudflare works, so the
|
||||
`cloudflared` manifest uses `--protocol http2` instead of relying on
|
||||
unverified UDP.
|
||||
- **Numeric UIDs are required:** a non-numeric image user breaks
|
||||
`runAsNonRoot` on SKE (reproduced with `curlimages/curl`). Every pod in
|
||||
`deploy/k8s/` sets explicit numeric `runAsUser`/`runAsGroup`/`fsGroup`.
|
||||
- **Repo status (2026-11-14 ~06:50 CET):** all three images build and pass
|
||||
local manifest-equivalent runtime checks (numeric UID, read-only root, `/tmp`
|
||||
tmpfs). Remaining steps are the image push, `control-auth`, deploy, and
|
||||
public smoke.
|
||||
|
||||
## What is possible now vs what is blocked
|
||||
|
||||
@@ -63,12 +62,10 @@ Re-check before trusting this snapshot:
|
||||
|---|---|---|
|
||||
| Read-only cluster/manifest check | agent or human | **ready now** (`make gate1-check`) |
|
||||
| Gitea push login + hello-world push | human | **done**; hello pull and Service verified |
|
||||
| `gitea-regcred`, `cloudflared`, `control-auth` Secrets | human | `gitea-regcred` + `cloudflared` **done**; `control-auth` pending |
|
||||
| Cloudflare Tunnel + public hostnames/DNS | human | **done**; tunnel active, public hello 200 |
|
||||
| `api` image | human | **ready** — `make build-api` + manifest-equivalent run verified locally |
|
||||
| `ml` image | human | **ready** — `make build-ml` + manifest-equivalent run verified locally |
|
||||
| `web` image | human | **ready** — `make build-web` + manifest-equivalent run verified locally |
|
||||
| Full deploy + smoke + Gate 1 sign-off | human | **ready once the tunnel/secret/image steps are done** |
|
||||
| `gitea-regcred`, `cloudflared`, `control-auth` Secrets | human | **done** |
|
||||
| Cloudflare Tunnel + public hostnames/DNS | human | **done**; tunnel active; `pulse` still points at hello |
|
||||
| `api`, `ml`, `web` images | human | **v1.1 pushed**; local manifest-equivalent checks pass |
|
||||
| v1.1 apply + smoke + presenter k6 + drills | human | **blocked on branch review/merge** |
|
||||
|
||||
`make build` and `make push` cover all three services; no Dockerfile is
|
||||
missing now. Do not deploy placeholder images under the real service names.
|
||||
@@ -86,7 +83,7 @@ export NS="fch26-onemanarmy"
|
||||
export CTX="fch26-onemanarmy-sa-context"
|
||||
export REGISTRY="git.noelsabia.dev"
|
||||
export OWNER="noel"
|
||||
export TAG="gate1"
|
||||
export TAG="v1.1"
|
||||
|
||||
chmod 600 "$KUBECONFIG"
|
||||
```
|
||||
@@ -147,7 +144,7 @@ Run top to bottom. Stop at the first failure and use
|
||||
- [ ] 5. `cloudflared` Secret created; logs show `Registered tunnel connection`.
|
||||
- [ ] 6. Public hello route returns HTTP 200 (proves tunnel + DNS + TLS).
|
||||
- [ ] 7. `control-auth` Secret created.
|
||||
- [ ] 8. All three images built, tagged `gate1`, and pushed.
|
||||
- [ ] 8. All three v1.1 images built and pushed.
|
||||
- [ ] 9. `make validate` and `make gate1-check` pass.
|
||||
- [ ] 10. `make deploy CONFIRM=yes` completes; all four Deployments are Ready.
|
||||
- [ ] 11. `make smoke SMOKE_SSE=1` passes.
|
||||
@@ -411,10 +408,10 @@ kubectl --context "$CTX" -n "$NS" describe pod -l app.kubernetes.io/name=cloudfl
|
||||
|
||||
## Step 4 — control-auth Secret
|
||||
|
||||
The API reads `CONTROL_USER` and `CONTROL_PASS` for browser control auth.
|
||||
Both API and ML also need the same `CONTROL_TOKEN`: the API forwards it as a
|
||||
Bearer token to FastAPI's internal `/train` routes. Generate both locally;
|
||||
never commit them.
|
||||
The web server validates `/control` and POST `/api/scenario` with
|
||||
`CONTROL_USER`/`CONTROL_PASS`; web, API, and ML also need the same
|
||||
`CONTROL_TOKEN` (the API forwards it as a Bearer token to FastAPI's internal
|
||||
`/train` routes). Generate both locally; never commit them.
|
||||
|
||||
```bash
|
||||
read -rsp 'Control password: ' CONTROL_PASS; echo
|
||||
@@ -438,9 +435,9 @@ creates nothing.
|
||||
Check whether the Dockerfiles exist:
|
||||
|
||||
```bash
|
||||
test -f ../../web-nuxt.git/feat-web-bootstrap/Dockerfile && echo web-ok || echo web-blocked
|
||||
test -f ../../api-go.git/feat-api-bootstrap/Dockerfile && echo api-ok || echo api-blocked
|
||||
test -f ../../ML-py.git/feat-ml-bootstrap/Dockerfile && echo ml-ok || echo ml-blocked
|
||||
test -f ../../web-nuxt.git/main/Dockerfile && echo web-ok || echo web-blocked
|
||||
test -f ../../api-go.git/main/Dockerfile && echo api-ok || echo api-blocked
|
||||
test -f ../../ML-py.git/main/Dockerfile && echo ml-ok || echo ml-blocked
|
||||
```
|
||||
|
||||
Expected **now** (all three Dockerfiles landed on 2026-11-14):
|
||||
@@ -457,18 +454,20 @@ Build all three:
|
||||
make build
|
||||
```
|
||||
|
||||
All three images were verified locally with the manifest's runtime
|
||||
All three v1.1 images were verified locally with the manifest's runtime
|
||||
constraints:
|
||||
|
||||
- API: distroless `static:nonroot`, `--user 65532:65532`, read-only root,
|
||||
`/tmp` tmpfs; `/api/healthz` and `/api/readyz` returned 200.
|
||||
- ML: image `USER app` is non-numeric, but the manifest overrides it with
|
||||
`runAsUser: 10001`; with read-only root and `/tmp` tmpfs, `/healthz` and
|
||||
`/readyz` returned 200 and `/meta` loaded the baked model.
|
||||
- Web: image `USER node` is non-numeric, but the manifest overrides it with
|
||||
`runAsUser: 10001`; with read-only root and `/tmp` tmpfs, `/` returned
|
||||
HTTP 200 and the runtime `NUXT_PUBLIC_API_BASE` appeared in the HTML.
|
||||
- No image was pushed or deployed by the infra agent.
|
||||
`/tmp` tmpfs; `/api/healthz` and `/api/readyz` returned 200, and the API
|
||||
loaded ML `/bootstrap` into memory.
|
||||
- ML: image `USER 10001:10001`; with read-only root and `/tmp` tmpfs,
|
||||
`/healthz` and `/readyz` returned 200, `/bootstrap` returned 8 sensors, and
|
||||
`/meta` showed `mae=3.32`, `wape=27.5`, `r2=0.777`.
|
||||
- Web: image `USER 10001:10001`; with read-only root and `/tmp` tmpfs, `/`,
|
||||
`/stage`, and `/live` returned 200; `/control` returned 401 anonymous and
|
||||
200 with control auth; the same-origin SSE proxy emitted events.
|
||||
- v1.1 was pushed to Gitea. The cluster is not updated until this branch is
|
||||
reviewed/merged and explicitly applied.
|
||||
|
||||
Push:
|
||||
|
||||
@@ -482,9 +481,9 @@ repeat the push login from Step 1.
|
||||
The Makefile auto-detects the service worktrees:
|
||||
|
||||
```text
|
||||
../../web-nuxt.git/feat-web-bootstrap
|
||||
../../api-go.git/feat-api-bootstrap
|
||||
../../ML-py.git/feat-ml-bootstrap
|
||||
../../web-nuxt.git/main
|
||||
../../api-go.git/main
|
||||
../../ML-py.git/main
|
||||
```
|
||||
|
||||
Override when the layout differs:
|
||||
@@ -512,9 +511,9 @@ Each service image must:
|
||||
|
||||
| Service | Image | Port | UID:GID | Health paths |
|
||||
|---|---|---|---|---|
|
||||
| web | `git.noelsabia.dev/noel/pulsecast-web:gate1` | 3000 | `10001:10001` | `/` |
|
||||
| api | `git.noelsabia.dev/noel/pulsecast-api:gate1` | 8080 | `65532:65532` | `/api/healthz`, `/api/readyz` |
|
||||
| ml | `git.noelsabia.dev/noel/pulsecast-ml:gate1` | 8000 | `10001:10001` | `/healthz`, `/readyz` |
|
||||
| web | `git.noelsabia.dev/noel/pulsecast-web:v1.1` | 3000 | `10001:10001` | `/`, `/stage`, `/live`, `/control` |
|
||||
| api | `git.noelsabia.dev/noel/pulsecast-api:v1.1` | 8080 | `65532:65532` | `/api/healthz`, `/api/readyz` |
|
||||
| ml | `git.noelsabia.dev/noel/pulsecast-ml:v1.1` | 8000 | `10001:10001` | `/healthz`, `/readyz`, `/bootstrap` |
|
||||
|
||||
The manifests set `runAsUser`/`runAsGroup`/`fsGroup` explicitly, so the image's
|
||||
named user is irrelevant; the files must be readable by that numeric UID.
|
||||
@@ -522,17 +521,19 @@ named user is irrelevant; the files must be readable by that numeric UID.
|
||||
Dockerfile genuinely needs another UID, change the manifest and this table
|
||||
together, and keep the value numeric.
|
||||
|
||||
`api` must read `DATA_DIR` (baked JSON), `FASTAPI_URL`, `PORT`,
|
||||
`CONTROL_USER`, `CONTROL_PASS`, `CONTROL_TOKEN`, `SSE_HEARTBEAT`, and
|
||||
`ML_TIMEOUT`.
|
||||
`web` must have `NUXT_PUBLIC_API_BASE=https://api-pulse.noelsabia.dev` baked in
|
||||
or read it at runtime from the ConfigMap; the browser must use the public API
|
||||
hostname. `ml` reads `PULSECAST_ROOT`, `PULSECAST_ARTIFACTS`,
|
||||
`PULSECAST_WEATHER`, `PULSECAST_MODELS`, `PULSECAST_UPLOADS`, and
|
||||
`CONTROL_TOKEN`; its image CMD binds `0.0.0.0:8000`. The env names in the
|
||||
manifests are the integration point: change them here first if a service needs
|
||||
a different name. The ML image currently uses a named `app` user; the manifest
|
||||
overrides it with numeric UID 10001, which was verified locally.
|
||||
`api` v1.1 reads `FASTAPI_URL=http://ml:8000`, `BOOTSTRAP_REFRESH=60s`,
|
||||
`BOOTSTRAP_TIMEOUT`, `PORT`, `CONTROL_USER`, `CONTROL_PASS`, `CONTROL_TOKEN`,
|
||||
`SSE_HEARTBEAT`, and `ML_TIMEOUT`. It has **no baked data and no `DATA_DIR`**;
|
||||
it fetches ML `/bootstrap` into memory and refreshes it.
|
||||
`web` proxies Go and SSE server-side using `NUXT_API_INTERNAL_BASE=http://api:8080`;
|
||||
it also needs `CONTROL_USER`, `CONTROL_PASS`, and `CONTROL_TOKEN` to protect
|
||||
`/control` and POST `/api/scenario`. `NUXT_PUBLIC_API_BASE` and the public SSE
|
||||
URL are display/future configuration; the browser calls the Nuxt origin.
|
||||
`ml` reads `PULSECAST_ROOT`, `PULSECAST_ARTIFACTS`, `PULSECAST_WEATHER`,
|
||||
`PULSECAST_MODELS`, `PULSECAST_UPLOADS`, and `CONTROL_TOKEN`; its image CMD
|
||||
binds `0.0.0.0:8000`. The env names in the manifests are the integration
|
||||
point: change them here first if a service needs a different name. All three
|
||||
images use numeric UIDs: web/ml `10001:10001`, API `65532:65532`.
|
||||
|
||||
## Step 6 — prove public HTTPS with hello
|
||||
|
||||
@@ -623,7 +624,7 @@ Expected: `web`, `api`, `ml`, and `cloudflared` each `1/1`; Services `web`,
|
||||
|
||||
### Re-pushing the same tag
|
||||
|
||||
The manifests use `imagePullPolicy: Always` while the tag stays `gate1`. After
|
||||
The manifests use `imagePullPolicy: Always` while the tag stays `v1.1`. After
|
||||
rebuilding and pushing the same tag, force a re-pull:
|
||||
|
||||
```bash
|
||||
@@ -680,6 +681,26 @@ The SSE output must start with `retry: 3000`, contain a `scenario` event
|
||||
immediately, and show a `ping` within 20 seconds. If events are delayed
|
||||
through Cloudflare, see the SSE row in Troubleshooting.
|
||||
|
||||
### v1.1 product checks
|
||||
|
||||
```bash
|
||||
curl -fsS https://api-pulse.noelsabia.dev/api/sensors | jq 'length'
|
||||
curl -fsS https://api-pulse.noelsabia.dev/api/meta \
|
||||
| jq '{data_version,model_version,metrics,cache}'
|
||||
curl -fsS -o /dev/null -w 'stage %{http_code}\n' https://pulse.noelsabia.dev/stage
|
||||
curl -fsS -o /dev/null -w 'live %{http_code}\n' https://pulse.noelsabia.dev/live
|
||||
curl -sS -o /dev/null -w 'control anon %{http_code}\n' https://pulse.noelsabia.dev/control
|
||||
set -a; source ~/.config/pulsecast/control-auth.env; set +a
|
||||
curl -sS -u "$CONTROL_USER:$CONTROL_PASS" -o /dev/null \
|
||||
-w 'control auth %{http_code}\n' https://pulse.noelsabia.dev/control
|
||||
curl -sS -o /dev/null -w 'plan %{time_total}s\n' \
|
||||
'https://api-pulse.noelsabia.dev/api/staffing?sensor=hn-stadtgalerie-01&scenario=base&standard=standard'
|
||||
```
|
||||
|
||||
Expected: **8** sensors; metrics `mae=3.32`, `wape=27.5`, `r2=0.777` with a
|
||||
`cache` object; `/stage` and `/live` 200; `/control` 401 anonymous and 200
|
||||
with control auth; plan under 2 seconds.
|
||||
|
||||
In-cluster connectivity (optional; **creates and removes one short-lived
|
||||
pod**, so the human runs it):
|
||||
|
||||
@@ -738,7 +759,7 @@ you need to reproduce the exact deploy.
|
||||
| `cloudflared` → `CrashLoopBackOff` | `kubectl -n "$NS" logs deployment/cloudflared --previous` | Invalid/expired tunnel token, or the tunnel was deleted in Cloudflare. Recreate the Secret and restart. |
|
||||
| `cloudflared` logs QUIC/timeout errors after you changed the protocol | `kubectl -n "$NS" logs deployment/cloudflared --tail=50` | UDP/7844 is unverified on SKE. Put `--protocol http2` back (the committed manifest already sets it); TCP/443 is confirmed. |
|
||||
| Public `502` / `1033` / `530` | `kubectl -n "$NS" get endpoints web api hello`; cloudflared logs | No ready endpoints, wrong Service port, or the hostname route points at the wrong service. Fix the dashboard route or the Service. Never add an Ingress. |
|
||||
| `api` pod not ready | `kubectl -n "$NS" logs deployment/api`; `curl -fsS https://api-pulse.noelsabia.dev/api/readyz` | Baked JSON missing from the image or `DATA_DIR` wrong. Rebuild the API image with the artifacts. The distroless API image has no shell, so do not rely on `kubectl exec ... ls`. |
|
||||
| `api` pod not ready | `kubectl -n "$NS" logs deployment/api`; `curl -fsS http://ml:8000/bootstrap` from inside the namespace; `curl -fsS https://api-pulse.noelsabia.dev/api/readyz` | v1.1 has no baked data. It needs ML reachable at `FASTAPI_URL` and a successful `/bootstrap`; check `BOOTSTRAP_REFRESH` and the ML readiness. The distroless API image has no shell, so do not rely on `kubectl exec ... ls`. |
|
||||
| `ml` pod stuck in startup | `kubectl -n "$NS" logs deployment/ml` | Model load exceeds the 120 s startup budget or `MODEL_PATH` is wrong. Fix the image/path. |
|
||||
| `web` 500 | `kubectl -n "$NS" logs deployment/web` | Missing runtime API base or SSR error. Rebuild with `NUXT_PUBLIC_API_BASE` or fix the ConfigMap. |
|
||||
| SSE events arrive only after a delay | `curl -N` direct to the pod vs through Cloudflare | Cloudflare buffering. Keep the 15 s heartbeat and the web polling fallback (`/api/scenario` every 5 s). Do not remove the heartbeat. |
|
||||
|
||||
@@ -7,8 +7,9 @@ drill.
|
||||
|
||||
## 1. FastAPI down (cached fallback)
|
||||
|
||||
Goal: the Go API keeps serving forecasts from baked JSON with `"cached": true`
|
||||
and `/api/readyz` stays 200. FastAPI being down must not fail API readiness.
|
||||
Goal: the Go API keeps serving its last good in-memory `/bootstrap` snapshot
|
||||
with `"cached": true` and `/api/readyz` stays 200. FastAPI being down must not
|
||||
fail API readiness after the snapshot has loaded once.
|
||||
|
||||
```bash
|
||||
# Baseline
|
||||
@@ -26,6 +27,9 @@ curl -sS 'https://api-pulse.noelsabia.dev/api/forecast?sensor=hn-stadtgalerie-01
|
||||
# Recover
|
||||
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy scale deployment/ml --replicas=1
|
||||
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy rollout status deployment/ml --timeout=180s
|
||||
# The API refreshes within BOOTSTRAP_REFRESH=60s; confirm the cache is fresh again.
|
||||
sleep 65
|
||||
curl -sS https://api-pulse.noelsabia.dev/api/meta | jq '.cache'
|
||||
```
|
||||
|
||||
Pass: no 5xx from `/api/forecast`, `cached` flips to `true` within the 1.5 s
|
||||
@@ -65,7 +69,8 @@ the tunnel token means updating the Secret and restarting the Deployment.
|
||||
|
||||
Goal: a single API pod restart causes a short 502 window, not a stuck UI. SSE
|
||||
clients reconnect with `retry: 3000`; scenario state is in memory and resets to
|
||||
the default, which is documented.
|
||||
the default. The restarted API re-fetches ML `/bootstrap` before `/api/readyz`
|
||||
turns 200; keep ML up during this drill.
|
||||
|
||||
```bash
|
||||
# Watch from two terminals:
|
||||
|
||||
+45
-27
@@ -1,7 +1,9 @@
|
||||
# k6 load test
|
||||
|
||||
`k6/sse.js` holds 120 concurrent Server-Sent Events connections for 5 minutes
|
||||
and reports connection count plus p95 event latency.
|
||||
and reports connection count plus p95 event latency. A separate `presenter`
|
||||
scenario POSTs one scenario change at 2m30s so the test measures event
|
||||
propagation through Cloudflare, not just heartbeat latency.
|
||||
|
||||
## Why a custom k6 binary
|
||||
|
||||
@@ -36,62 +38,77 @@ A stock `grafana/k6` image is **not** a fallback: it lacks the SSE extension.
|
||||
You should see `retry: 3000` and a `scenario` event immediately, then a
|
||||
`ping` every 15 s.
|
||||
|
||||
2. Run the load test:
|
||||
2. Load the control credentials and run the load test. The presenter scenario
|
||||
requires `CONTROL_USER` and `CONTROL_PASS`; `make load` refuses to start
|
||||
without them.
|
||||
|
||||
```bash
|
||||
set -a
|
||||
source ~/.config/pulsecast/control-auth.env
|
||||
set +a
|
||||
make load
|
||||
# or: API_URL=https://api-pulse.noelsabia.dev ./k6/k6-sse run k6/sse.js
|
||||
# or: API_URL=https://api-pulse.noelsabia.dev \
|
||||
# CONTROL_USER="$CONTROL_USER" CONTROL_PASS="$CONTROL_PASS" \
|
||||
# ./k6/k6-sse run k6/sse.js
|
||||
```
|
||||
|
||||
`make load` writes `k6/results/summary-<timestamp>.json` (git-ignored).
|
||||
|
||||
3. During the run, post a scenario change so the scenario-event latency has
|
||||
samples and the propagation path is exercised:
|
||||
3. The `presenter` scenario starts at `2m30s`, POSTs
|
||||
`PRESENTER_SCENARIO=rainy-saturday` once to `/api/scenario`, and records the
|
||||
request duration plus success. All 120 SSE clients must receive the
|
||||
resulting `scenario` event. Override the timing/scenario if needed:
|
||||
|
||||
```bash
|
||||
curl -u "$CONTROL_USER:$CONTROL_PASS" -X POST \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"scenario":"sunny-saturday-market","weather":{"temp_delta_c":4,"precip_mm":0}}' \
|
||||
https://api-pulse.noelsabia.dev/api/scenario
|
||||
make load PRESENTER_AT=3m PRESENTER_SCENARIO=sunny-saturday-market
|
||||
```
|
||||
|
||||
Repeat every 30–60 s with different scenarios if you want a distribution.
|
||||
Without a change, `sse_event_latency` is measured from heartbeat `ping`
|
||||
events, which still proves the stream is live.
|
||||
|
||||
4. Record these numbers in the paper/README from the saved summary:
|
||||
- `sse_connections_opened` count (must be 120)
|
||||
- `sse_connections_closed` count (must equal connections opened)
|
||||
- `sse_connections_closed` count (must be >= 120)
|
||||
- `sse_open_success` rate (must be >= 99%)
|
||||
- `sse_event_latency` p95 (target < 2000 ms)
|
||||
- `sse_scenario_latency` p95 (target < 2000 ms)
|
||||
- `presenter_post_success` rate (must be 100%)
|
||||
- `presenter_post_duration` (record p95/max)
|
||||
- `sse_connection_errors` (target 0)
|
||||
- `sse_events_received` total
|
||||
|
||||
## Local validation before the cluster run
|
||||
|
||||
The script was validated locally on 2026-11-14 with the pinned k6 v2.0.0 build
|
||||
against a mock SSE stream and the real API image (3 VUs, 1 heartbeat each):
|
||||
connections opened and closed cleanly, open success 100%. The real 120-client
|
||||
run still requires the deployed API, because localhost results say nothing
|
||||
about Cloudflare buffering or SKE scheduling.
|
||||
against a mock SSE stream and the real v1.1 API image (3 VUs, 1 heartbeat
|
||||
each): connections opened and closed cleanly, open success 100%, presenter
|
||||
POST 200, and scenario latency sampled. The real 120-client run still requires
|
||||
the deployed API, because localhost results say nothing about Cloudflare
|
||||
buffering or SKE scheduling.
|
||||
|
||||
## Pass criteria
|
||||
|
||||
| Metric | Target |
|
||||
|---|---|
|
||||
| Connections opened | 120 |
|
||||
| Connections closed (stream ended) | 120 |
|
||||
| Connections closed (stream ended) | >= 120 |
|
||||
| Open success rate | >= 99% |
|
||||
| p95 event latency | < 2000 ms |
|
||||
| p95 scenario latency | < 2000 ms |
|
||||
| Presenter POST success | 100% |
|
||||
| Unexpected connection errors | 0 |
|
||||
| Duration | 5 minutes |
|
||||
|
||||
The k6 thresholds enforce connections opened, open success, and p95 latency;
|
||||
the others are read from the summary. A final `sse_connections_closed` below
|
||||
120 means at least one opened VU did not finish; inspect
|
||||
`sse_connection_errors` for unexpected errors. A p95 above 2 s means Cloudflare, the Go SSE hub, or the
|
||||
client is buffering/delaying events; check `X-Accel-Buffering: no`, the 15 s
|
||||
heartbeat, and whether the connection is HTTP/2-buffered at the edge.
|
||||
The k6 thresholds enforce connections opened/closed, open success, p95 event
|
||||
latency, p95 scenario latency, presenter success, and zero connection errors.
|
||||
A p95 above 2 s means Cloudflare, the Go SSE hub, or the client is
|
||||
buffering/delaying events; check `X-Accel-Buffering: no`, the 15 s heartbeat,
|
||||
and whether the connection is HTTP/2-buffered at the edge.
|
||||
|
||||
## Results
|
||||
|
||||
Populate from the saved `k6/results/summary-*.json`; never estimate.
|
||||
|
||||
| Run | Date | Opened | Closed | Open success | p95 event | p95 scenario | Presenter success | Errors |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| v1.1 through Cloudflare | pending post-merge run | | | | | | | |
|
||||
|
||||
## Notes
|
||||
|
||||
@@ -100,8 +117,9 @@ heartbeat, and whether the connection is HTTP/2-buffered at the edge.
|
||||
on NTP. Negative or > 1 h values are ignored.
|
||||
- The initial `scenario` event on connect replays the current state and carries
|
||||
the original state-change time, not the send time. The script excludes that
|
||||
first scenario event per VU; only scenario changes broadcast during the run
|
||||
are sampled by `sse_scenario_latency`. Heartbeat pings always count.
|
||||
first scenario event per VU; the presenter POST is the scenario sample for
|
||||
`sse_scenario_latency`. Heartbeat pings always count toward
|
||||
`sse_event_latency`.
|
||||
- Each VU closes after 20 heartbeats (20 × 15 s = 5 min), so the run ends
|
||||
cleanly instead of waiting for the 7-minute request timeout.
|
||||
- One API replica means all clients share one SSE hub; a pod restart drops
|
||||
|
||||
Reference in new issue
Block a user