docs: record v1.1 load test and failure drill results
v1.1 Cloudflare run: 120/120 connections, 0 errors, p95 event 331 ms, p95 scenario 646 ms, presenter 49.60 ms, 2,760 events, 4m57.9s. Failure drill: ML-down cache/readyz/web pass; API pod restart pass. Also record v1.1 deployment, stage/live/control verification and hello removal.
This commit is contained in:
1 parent
1171a15311
commit
bd3176bef0
4 files changed
+54
-39
No files matched your search
@@ -52,17 +52,19 @@ hostname, and verification steps.
|
|||||||
- Gitea pull is proven: `gitea-regcred` exists and the hello pod pulled from
|
- Gitea pull is proven: `gitea-regcred` exists and the hello pod pulled from
|
||||||
`git.noelsabia.dev/noel/hello:gate1`; the Service returned HTTP 200.
|
`git.noelsabia.dev/noel/hello:gate1`; the Service returned HTTP 200.
|
||||||
- Tunnel `pulsecast-gate1` is active with `pulse` and `api-pulse` routes.
|
- Tunnel `pulsecast-gate1` is active with `pulse` and `api-pulse` routes.
|
||||||
`pulse` already serves the v1.0 Nuxt app; no route switch remains. `/stage`
|
`pulse` serves the v1.1 Nuxt app; `/stage` and `/live` return 200, and
|
||||||
and `/live` are v1.1 pages. The API hostname is
|
`/control` requires auth. The API hostname is
|
||||||
`api-pulse.noelsabia.dev` because Universal SSL only covers
|
`api-pulse.noelsabia.dev` because Universal SSL only covers
|
||||||
`*.noelsabia.dev`.
|
`*.noelsabia.dev`.
|
||||||
- The 120-VU v1.0 baseline passed: 120/120 connections, 0 errors, p95 event
|
- v1.1 is deployed: web/api/ml `:v1.1`, cloudflared `2026.11.0`, all four
|
||||||
1.03 s, p95 scenario 194 ms. Results are in `docs/load-test.md`.
|
Deployments Ready.
|
||||||
- v1.1 images are built from the service `main` worktrees, locally checked,
|
- The v1.1 120-VU Cloudflare run passed: 120/120 connections, 0 errors, p95
|
||||||
and pushed. The cluster still runs the previous v1.0 images until this branch
|
event 331 ms, p95 scenario 646 ms, presenter 49.60 ms. The v1.0 baseline is
|
||||||
is reviewed/merged and explicitly applied.
|
retained for comparison in `docs/load-test.md`.
|
||||||
- The v1.1 branch adds web control credentials, API bootstrap config, and the
|
- The ML-down/API-restart failure drill passed; details are in
|
||||||
k6 presenter scenario; load-test results are recorded after the approved run.
|
`docs/failure-drills.md`.
|
||||||
|
- The results documentation still needs a new PR review/merge; the v1.1
|
||||||
|
runtime code is already merged in `main`.
|
||||||
|
|
||||||
## CI
|
## CI
|
||||||
|
|
||||||
|
|||||||
+21
-26
@@ -24,22 +24,25 @@ Gate 1 passes only when **all** of these are true:
|
|||||||
Do not claim Gate 1 from a local `kubectl port-forward`, a green manifest
|
Do not claim Gate 1 from a local `kubectl port-forward`, a green manifest
|
||||||
dry-run, or a container that only runs on your laptop.
|
dry-run, or a container that only runs on your laptop.
|
||||||
|
|
||||||
## Current external status (2026-11-14 ~08:00 CET)
|
## Current external status (2026-11-14 ~08:30 CET)
|
||||||
|
|
||||||
Re-check before trusting this snapshot:
|
Re-check before trusting this snapshot:
|
||||||
|
|
||||||
- **Verified:** cluster access, scheduling, Service DNS/HTTP, probes, Secrets,
|
- **v1.1 is deployed and Ready:** web/api/ml `:v1.1`, cloudflared
|
||||||
non-root + read-only-root + `emptyDir`, and egress to Cloudflare,
|
`2026.11.0`, all four Deployments `1/1`. Numeric UIDs are
|
||||||
Open-Meteo, and `git.noelsabia.dev`.
|
`10001:10001` for web/ml and `65532:65532` for API/cloudflared.
|
||||||
- **Tunnel active:** `pulsecast-gate1` has routes for `pulse.noelsabia.dev`
|
- **Tunnel active:** `pulsecast-gate1` serves `pulse.noelsabia.dev` through
|
||||||
and `api-pulse.noelsabia.dev`; `cloudflared` is `1/1` with four HTTP/2
|
Nuxt (`x-powered-by: Nuxt`) and `api-pulse.noelsabia.dev` through the API;
|
||||||
connections. `pulse` already serves the v1.0 Nuxt app
|
four HTTP/2 connections. `/stage` and `/live` return 200; `/control` is
|
||||||
(`x-powered-by: Nuxt`), so **no Cloudflare route switch remains**.
|
401 anonymous and 200 with control auth. The temporary hello is deleted.
|
||||||
`/stage` and `/live` return 404 on v1.0; v1.1 adds them. `api-pulse` serves
|
- **v1.1 120-VU Cloudflare run passed:** 120/120 SSE connections opened and
|
||||||
the deployed API.
|
closed, 0 errors, p95 event 331 ms, p95 scenario 646 ms, presenter POST
|
||||||
- **120-VU baseline passed:** 120/120 SSE connections opened/closed, 0
|
49.60 ms, 361/361 checks. See `docs/load-test.md`.
|
||||||
connection errors, p95 event latency 1.03 s, p95 scenario latency 194 ms;
|
- **Failure drill passed:** ML scaled to 0 while API readiness/forecast cache
|
||||||
see `docs/load-test.md`. This was against v1.0.
|
and web stayed available; ML restored and API pod restarted cleanly. See
|
||||||
|
`docs/failure-drills.md`.
|
||||||
|
- **API/product checks:** 8 sensors, `/api/meta` `mae=3.32`, `wape=27.5`,
|
||||||
|
`r2=0.777`, cache `source=fastapi`, staffing plan under 2 s.
|
||||||
- **DNS works:** public resolvers return Cloudflare IPs for
|
- **DNS works:** public resolvers return Cloudflare IPs for
|
||||||
`pulse.noelsabia.dev` and `api-pulse.noelsabia.dev`. A local
|
`pulse.noelsabia.dev` and `api-pulse.noelsabia.dev`. A local
|
||||||
`systemd-resolved` negative cache can be cleared with
|
`systemd-resolved` negative cache can be cleared with
|
||||||
@@ -47,30 +50,22 @@ Re-check before trusting this snapshot:
|
|||||||
- **API hostname is `api-pulse.noelsabia.dev`:** Universal SSL covers only
|
- **API hostname is `api-pulse.noelsabia.dev`:** Universal SSL covers only
|
||||||
`*.noelsabia.dev`; a two-level `api.pulse.noelsabia.dev` is not covered by
|
`*.noelsabia.dev`; a two-level `api.pulse.noelsabia.dev` is not covered by
|
||||||
that certificate.
|
that certificate.
|
||||||
- **Gitea pull is proven:** `gitea-regcred` exists; hello and the v1.0 service
|
|
||||||
images pulled from Gitea. The Gitea owner is `noel`.
|
|
||||||
- **v1.1 images pushed:** web/api/ml `:v1.1` were built from the service
|
|
||||||
`main` worktrees, locally exercised under numeric UIDs/read-only root, and
|
|
||||||
pushed. The cluster still runs the previous v1.0 images until the v1.1
|
|
||||||
branch is reviewed/merged and explicitly applied.
|
|
||||||
- **All three Secrets exist:** `gitea-regcred`, `cloudflared`, `control-auth`.
|
- **All three Secrets exist:** `gitea-regcred`, `cloudflared`, `control-auth`.
|
||||||
- **QUIC/UDP 7844 is untested.** TCP/443 to Cloudflare works, so the
|
- **QUIC/UDP 7844 is untested.** TCP/443 to Cloudflare works, so the
|
||||||
`cloudflared` manifest uses `--protocol http2` instead of relying on
|
`cloudflared` manifest uses `--protocol http2` instead of relying on
|
||||||
unverified UDP.
|
unverified UDP.
|
||||||
- **Numeric UIDs are required:** a non-numeric image user breaks
|
|
||||||
`runAsNonRoot` on SKE (reproduced with `curlimages/curl`). Every pod in
|
|
||||||
`deploy/k8s/` sets explicit numeric `runAsUser`/`runAsGroup`/`fsGroup`.
|
|
||||||
|
|
||||||
## What is possible now vs what is blocked
|
## What is possible now vs what is blocked
|
||||||
|
|
||||||
| Step | Who | Status |
|
| Step | Who | Status |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| Read-only cluster/manifest check | agent or human | **ready now** (`make gate1-check`) |
|
| Cluster/manifest check | agent or human | **done**; static and live checks pass |
|
||||||
| Gitea push login + hello-world push | human | **done**; hello pull and Service verified |
|
| Gitea pull + hello proof | human | **done**; hello deleted after cutover |
|
||||||
| `gitea-regcred`, `cloudflared`, `control-auth` Secrets | human | **done** |
|
| `gitea-regcred`, `cloudflared`, `control-auth` Secrets | human | **done** |
|
||||||
| Cloudflare Tunnel + public hostnames/DNS | human | **done**; tunnel active; `pulse` serves Nuxt |
|
| Cloudflare Tunnel + public hostnames/DNS | human | **done**; tunnel active; `pulse` serves Nuxt |
|
||||||
| `api`, `ml`, `web` images | human | **v1.1 pushed**; local manifest-equivalent checks pass |
|
| `api`, `ml`, `web` v1.1 images | human | **done**; pushed and deployed |
|
||||||
| v1.1 apply + smoke + presenter k6 + drills | human | **blocked on branch review/merge** |
|
| v1.1 apply + smoke + presenter k6 + drills | human | **done** |
|
||||||
|
| Results docs | agent | **pending new PR review/merge** |
|
||||||
|
|
||||||
`make build` and `make push` cover all three services; no Dockerfile is
|
`make build` and `make push` cover all three services; no Dockerfile is
|
||||||
missing now. Do not deploy placeholder images under the real service names.
|
missing now. Do not deploy placeholder images under the real service names.
|
||||||
|
|||||||
@@ -5,6 +5,20 @@ cluster mutation: the human runs it, not an agent. Record the result and the
|
|||||||
recovery time. Keep `make smoke` and a `curl -N .../api/stream` open while you
|
recovery time. Keep `make smoke` and a `curl -N .../api/stream` open while you
|
||||||
drill.
|
drill.
|
||||||
|
|
||||||
|
## Last run — 2026-11-14 07:26:56–07:29:27 UTC
|
||||||
|
|
||||||
|
v1.1, executed against the live cluster:
|
||||||
|
|
||||||
|
| Step | Observed | Result |
|
||||||
|
|---|---|---|
|
||||||
|
| ML scaled to 0 | `/api/readyz` 200; forecast 200, `cached=true`, 168 points in 111 ms; web 200 | PASS |
|
||||||
|
| Failed bootstrap refresh | cache `source=fastapi`, `last_error` connection refused, `failures=6`, `next_retry_at` set; web/API stayed available | PASS |
|
||||||
|
| ML scaled back to 1 | rollout 1/1; cache recovered to `last_error=null`, `failures=0`, `/api/readyz` 200 | PASS |
|
||||||
|
| API pod restart | pod deleted and recreated; rollout 1/1; `/api/healthz` 200, `/api/readyz` 200, 8 sensors, web 200 | PASS |
|
||||||
|
|
||||||
|
The drill took 2m31s including the two 65 s refresh windows. No hello
|
||||||
|
resources existed during this run; it was deleted after the v1.1 cutover.
|
||||||
|
|
||||||
## 1. FastAPI down (cached fallback)
|
## 1. FastAPI down (cached fallback)
|
||||||
|
|
||||||
Goal: the Go API keeps serving its last good in-memory `/bootstrap` snapshot
|
Goal: the Go API keeps serving its last good in-memory `/bootstrap` snapshot
|
||||||
|
|||||||
+8
-4
@@ -109,15 +109,19 @@ Populate from the saved `k6/results/summary-*.json`; never estimate.
|
|||||||
| Run | Date | Opened | Closed | Open success | p95 event | p95 scenario | Presenter success | Errors |
|
| Run | Date | Opened | Closed | Open success | p95 event | p95 scenario | Presenter success | Errors |
|
||||||
|---|---|---|---|---|---|---|---|---|
|
|---|---|---|---|---|---|---|---|---|
|
||||||
| v1.0 baseline through Cloudflare | 2026-11-14 08:07 CET | 120 | 120 | 100% | 1.03 s | 194 ms | 100% (82.69 ms) | 0 |
|
| v1.0 baseline through Cloudflare | 2026-11-14 08:07 CET | 120 | 120 | 100% | 1.03 s | 194 ms | 100% (82.69 ms) | 0 |
|
||||||
| v1.1 through Cloudflare | pending post-merge run | | | | | | | |
|
| v1.1 through Cloudflare | 2026-11-14 08:21 CET | 120 | 120 | 100% | 331 ms | 646 ms | 100% (49.60 ms) | 0 |
|
||||||
|
|
||||||
Raw baseline result: `k6/results/summary-20261114T080747.json` (git-ignored).
|
Raw baseline result: `k6/results/summary-20261114T080747.json` (git-ignored).
|
||||||
|
Raw v1.1 result: `k6/results/summary-20261114T082121.json` (git-ignored).
|
||||||
|
|
||||||
Baseline details: 361/361 checks passed, 2,760 events received, 2,400
|
Baseline details: 361/361 checks passed, 2,760 events received, 2,400
|
||||||
heartbeat pings, 120 VUs, test duration 4m53s, presenter POST 82.69 ms.
|
heartbeat pings, 120 VUs, test duration 4m53s, presenter POST 82.69 ms.
|
||||||
`p95 ping latency` was 1,045 ms. This run was against the live v1.0 API; rerun
|
`p95 ping latency` was 1,045 ms.
|
||||||
the same command after v1.1 is applied and replace the pending row with exact
|
|
||||||
numbers.
|
v1.1 details: 361/361 checks passed, 2,760 events received, 2,400 heartbeat
|
||||||
|
pings, 120 VUs, test duration 4m57.9s, presenter POST 49.60 ms, `p95 ping
|
||||||
|
latency` 268 ms. All thresholds passed. This run used the presenter POST at
|
||||||
|
2m30s through Cloudflare and the same `make load` path.
|
||||||
|
|
||||||
## Notes
|
## Notes
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user