From d72878f56083e1d11cf299509fe86da39404f140 Mon Sep 17 00:00:00 2001 From: Noel Sabia Date: Sat, 14 Nov 2026 08:14:45 +0100 Subject: [PATCH 1/2] docs: record 120-VU v1.0 Cloudflare baseline Baseline: 120/120 SSE connections opened/closed, 0 errors, p95 event 1.03 s, p95 scenario 194 ms, presenter POST 82.69 ms, 2,760 events, 2,400 pings, 4m53s. Raw summary is git-ignored. Also confirm pulse.noelsabia.dev already serves Nuxt; no route switch is needed. /stage and /live remain 404 on v1.0 until v1.1 is applied. --- README.md | 9 ++++++--- docs/deploy.md | 11 ++++++++--- docs/load-test.md | 9 +++++++++ 3 files changed, 23 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 2077d59..f04ecae 100644 --- a/README.md +++ b/README.md @@ -52,9 +52,12 @@ hostname, and verification steps. - Gitea pull is proven: `gitea-regcred` exists and the hello pod pulled from `git.noelsabia.dev/noel/hello:gate1`; the Service returned HTTP 200. - Tunnel `pulsecast-gate1` is active with `pulse` and `api-pulse` routes. - `pulse` still points at the temporary hello Service until the v1.1 cutover. - The API hostname is `api-pulse.noelsabia.dev` because Universal SSL only - covers `*.noelsabia.dev`. + `pulse` already serves the v1.0 Nuxt app; no route switch remains. `/stage` + and `/live` are v1.1 pages. The API hostname is + `api-pulse.noelsabia.dev` because Universal SSL only covers + `*.noelsabia.dev`. +- The 120-VU v1.0 baseline passed: 120/120 connections, 0 errors, p95 event + 1.03 s, p95 scenario 194 ms. Results are in `docs/load-test.md`. - v1.1 images are built from the service `main` worktrees, locally checked, and pushed. The cluster still runs the previous v1.0 images until this branch is reviewed/merged and explicitly applied. diff --git a/docs/deploy.md b/docs/deploy.md index c00fc6e..ab28335 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -33,8 +33,13 @@ Re-check before trusting this snapshot: Open-Meteo, and `git.noelsabia.dev`. - **Tunnel active:** `pulsecast-gate1` has routes for `pulse.noelsabia.dev` and `api-pulse.noelsabia.dev`; `cloudflared` is `1/1` with four HTTP/2 - connections. `api-pulse` serves the deployed API; `pulse` still points at - the temporary hello Service until the web cutover. + connections. `pulse` already serves the v1.0 Nuxt app + (`x-powered-by: Nuxt`), so **no Cloudflare route switch remains**. + `/stage` and `/live` return 404 on v1.0; v1.1 adds them. `api-pulse` serves + the deployed API. +- **120-VU baseline passed:** 120/120 SSE connections opened/closed, 0 + connection errors, p95 event latency 1.03 s, p95 scenario latency 194 ms; + see `docs/load-test.md`. This was against v1.0. - **DNS works:** public resolvers return Cloudflare IPs for `pulse.noelsabia.dev` and `api-pulse.noelsabia.dev`. A local `systemd-resolved` negative cache can be cleared with @@ -63,7 +68,7 @@ Re-check before trusting this snapshot: | Read-only cluster/manifest check | agent or human | **ready now** (`make gate1-check`) | | Gitea push login + hello-world push | human | **done**; hello pull and Service verified | | `gitea-regcred`, `cloudflared`, `control-auth` Secrets | human | **done** | -| Cloudflare Tunnel + public hostnames/DNS | human | **done**; tunnel active; `pulse` still points at hello | +| Cloudflare Tunnel + public hostnames/DNS | human | **done**; tunnel active; `pulse` serves Nuxt | | `api`, `ml`, `web` images | human | **v1.1 pushed**; local manifest-equivalent checks pass | | v1.1 apply + smoke + presenter k6 + drills | human | **blocked on branch review/merge** | diff --git a/docs/load-test.md b/docs/load-test.md index 3ea386a..6a8935c 100644 --- a/docs/load-test.md +++ b/docs/load-test.md @@ -108,8 +108,17 @@ Populate from the saved `k6/results/summary-*.json`; never estimate. | Run | Date | Opened | Closed | Open success | p95 event | p95 scenario | Presenter success | Errors | |---|---|---|---|---|---|---|---|---| +| v1.0 baseline through Cloudflare | 2026-11-14 08:07 CET | 120 | 120 | 100% | 1.03 s | 194 ms | 100% (82.69 ms) | 0 | | v1.1 through Cloudflare | pending post-merge run | | | | | | | | +Raw baseline result: `k6/results/summary-20261114T080747.json` (git-ignored). + +Baseline details: 361/361 checks passed, 2,760 events received, 2,400 +heartbeat pings, 120 VUs, test duration 4m53s, presenter POST 82.69 ms. +`p95 ping latency` was 1,045 ms. This run was against the live v1.0 API; rerun +the same command after v1.1 is applied and replace the pending row with exact +numbers. + ## Notes - `sse_event_latency` uses the server timestamp in `data.t` (ping) or `data.at` From bd3176bef014d4060003cf9a310fef27950fe938 Mon Sep 17 00:00:00 2001 From: Noel Sabia Date: Sat, 14 Nov 2026 08:32:12 +0100 Subject: [PATCH 2/2] docs: record v1.1 load test and failure drill results v1.1 Cloudflare run: 120/120 connections, 0 errors, p95 event 331 ms, p95 scenario 646 ms, presenter 49.60 ms, 2,760 events, 4m57.9s. Failure drill: ML-down cache/readyz/web pass; API pod restart pass. Also record v1.1 deployment, stage/live/control verification and hello removal. --- README.md | 20 ++++++++++-------- docs/deploy.md | 47 +++++++++++++++++++----------------------- docs/failure-drills.md | 14 +++++++++++++ docs/load-test.md | 12 +++++++---- 4 files changed, 54 insertions(+), 39 deletions(-) diff --git a/README.md b/README.md index f04ecae..9594d90 100644 --- a/README.md +++ b/README.md @@ -52,17 +52,19 @@ hostname, and verification steps. - Gitea pull is proven: `gitea-regcred` exists and the hello pod pulled from `git.noelsabia.dev/noel/hello:gate1`; the Service returned HTTP 200. - Tunnel `pulsecast-gate1` is active with `pulse` and `api-pulse` routes. - `pulse` already serves the v1.0 Nuxt app; no route switch remains. `/stage` - and `/live` are v1.1 pages. The API hostname is + `pulse` serves the v1.1 Nuxt app; `/stage` and `/live` return 200, and + `/control` requires auth. The API hostname is `api-pulse.noelsabia.dev` because Universal SSL only covers `*.noelsabia.dev`. -- The 120-VU v1.0 baseline passed: 120/120 connections, 0 errors, p95 event - 1.03 s, p95 scenario 194 ms. Results are in `docs/load-test.md`. -- v1.1 images are built from the service `main` worktrees, locally checked, - and pushed. The cluster still runs the previous v1.0 images until this branch - is reviewed/merged and explicitly applied. -- The v1.1 branch adds web control credentials, API bootstrap config, and the - k6 presenter scenario; load-test results are recorded after the approved run. +- v1.1 is deployed: web/api/ml `:v1.1`, cloudflared `2026.11.0`, all four + Deployments Ready. +- The v1.1 120-VU Cloudflare run passed: 120/120 connections, 0 errors, p95 + event 331 ms, p95 scenario 646 ms, presenter 49.60 ms. The v1.0 baseline is + retained for comparison in `docs/load-test.md`. +- The ML-down/API-restart failure drill passed; details are in + `docs/failure-drills.md`. +- The results documentation still needs a new PR review/merge; the v1.1 + runtime code is already merged in `main`. ## CI diff --git a/docs/deploy.md b/docs/deploy.md index ab28335..91d5008 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -24,22 +24,25 @@ Gate 1 passes only when **all** of these are true: Do not claim Gate 1 from a local `kubectl port-forward`, a green manifest dry-run, or a container that only runs on your laptop. -## Current external status (2026-11-14 ~08:00 CET) +## Current external status (2026-11-14 ~08:30 CET) Re-check before trusting this snapshot: -- **Verified:** cluster access, scheduling, Service DNS/HTTP, probes, Secrets, - non-root + read-only-root + `emptyDir`, and egress to Cloudflare, - Open-Meteo, and `git.noelsabia.dev`. -- **Tunnel active:** `pulsecast-gate1` has routes for `pulse.noelsabia.dev` - and `api-pulse.noelsabia.dev`; `cloudflared` is `1/1` with four HTTP/2 - connections. `pulse` already serves the v1.0 Nuxt app - (`x-powered-by: Nuxt`), so **no Cloudflare route switch remains**. - `/stage` and `/live` return 404 on v1.0; v1.1 adds them. `api-pulse` serves - the deployed API. -- **120-VU baseline passed:** 120/120 SSE connections opened/closed, 0 - connection errors, p95 event latency 1.03 s, p95 scenario latency 194 ms; - see `docs/load-test.md`. This was against v1.0. +- **v1.1 is deployed and Ready:** web/api/ml `:v1.1`, cloudflared + `2026.11.0`, all four Deployments `1/1`. Numeric UIDs are + `10001:10001` for web/ml and `65532:65532` for API/cloudflared. +- **Tunnel active:** `pulsecast-gate1` serves `pulse.noelsabia.dev` through + Nuxt (`x-powered-by: Nuxt`) and `api-pulse.noelsabia.dev` through the API; + four HTTP/2 connections. `/stage` and `/live` return 200; `/control` is + 401 anonymous and 200 with control auth. The temporary hello is deleted. +- **v1.1 120-VU Cloudflare run passed:** 120/120 SSE connections opened and + closed, 0 errors, p95 event 331 ms, p95 scenario 646 ms, presenter POST + 49.60 ms, 361/361 checks. See `docs/load-test.md`. +- **Failure drill passed:** ML scaled to 0 while API readiness/forecast cache + and web stayed available; ML restored and API pod restarted cleanly. See + `docs/failure-drills.md`. +- **API/product checks:** 8 sensors, `/api/meta` `mae=3.32`, `wape=27.5`, + `r2=0.777`, cache `source=fastapi`, staffing plan under 2 s. - **DNS works:** public resolvers return Cloudflare IPs for `pulse.noelsabia.dev` and `api-pulse.noelsabia.dev`. A local `systemd-resolved` negative cache can be cleared with @@ -47,30 +50,22 @@ Re-check before trusting this snapshot: - **API hostname is `api-pulse.noelsabia.dev`:** Universal SSL covers only `*.noelsabia.dev`; a two-level `api.pulse.noelsabia.dev` is not covered by that certificate. -- **Gitea pull is proven:** `gitea-regcred` exists; hello and the v1.0 service - images pulled from Gitea. The Gitea owner is `noel`. -- **v1.1 images pushed:** web/api/ml `:v1.1` were built from the service - `main` worktrees, locally exercised under numeric UIDs/read-only root, and - pushed. The cluster still runs the previous v1.0 images until the v1.1 - branch is reviewed/merged and explicitly applied. - **All three Secrets exist:** `gitea-regcred`, `cloudflared`, `control-auth`. - **QUIC/UDP 7844 is untested.** TCP/443 to Cloudflare works, so the `cloudflared` manifest uses `--protocol http2` instead of relying on unverified UDP. -- **Numeric UIDs are required:** a non-numeric image user breaks - `runAsNonRoot` on SKE (reproduced with `curlimages/curl`). Every pod in - `deploy/k8s/` sets explicit numeric `runAsUser`/`runAsGroup`/`fsGroup`. ## What is possible now vs what is blocked | Step | Who | Status | |---|---|---| -| Read-only cluster/manifest check | agent or human | **ready now** (`make gate1-check`) | -| Gitea push login + hello-world push | human | **done**; hello pull and Service verified | +| Cluster/manifest check | agent or human | **done**; static and live checks pass | +| Gitea pull + hello proof | human | **done**; hello deleted after cutover | | `gitea-regcred`, `cloudflared`, `control-auth` Secrets | human | **done** | | Cloudflare Tunnel + public hostnames/DNS | human | **done**; tunnel active; `pulse` serves Nuxt | -| `api`, `ml`, `web` images | human | **v1.1 pushed**; local manifest-equivalent checks pass | -| v1.1 apply + smoke + presenter k6 + drills | human | **blocked on branch review/merge** | +| `api`, `ml`, `web` v1.1 images | human | **done**; pushed and deployed | +| v1.1 apply + smoke + presenter k6 + drills | human | **done** | +| Results docs | agent | **pending new PR review/merge** | `make build` and `make push` cover all three services; no Dockerfile is missing now. Do not deploy placeholder images under the real service names. diff --git a/docs/failure-drills.md b/docs/failure-drills.md index 720c495..316fbeb 100644 --- a/docs/failure-drills.md +++ b/docs/failure-drills.md @@ -5,6 +5,20 @@ cluster mutation: the human runs it, not an agent. Record the result and the recovery time. Keep `make smoke` and a `curl -N .../api/stream` open while you drill. +## Last run — 2026-11-14 07:26:56–07:29:27 UTC + +v1.1, executed against the live cluster: + +| Step | Observed | Result | +|---|---|---| +| ML scaled to 0 | `/api/readyz` 200; forecast 200, `cached=true`, 168 points in 111 ms; web 200 | PASS | +| Failed bootstrap refresh | cache `source=fastapi`, `last_error` connection refused, `failures=6`, `next_retry_at` set; web/API stayed available | PASS | +| ML scaled back to 1 | rollout 1/1; cache recovered to `last_error=null`, `failures=0`, `/api/readyz` 200 | PASS | +| API pod restart | pod deleted and recreated; rollout 1/1; `/api/healthz` 200, `/api/readyz` 200, 8 sensors, web 200 | PASS | + +The drill took 2m31s including the two 65 s refresh windows. No hello +resources existed during this run; it was deleted after the v1.1 cutover. + ## 1. FastAPI down (cached fallback) Goal: the Go API keeps serving its last good in-memory `/bootstrap` snapshot diff --git a/docs/load-test.md b/docs/load-test.md index 6a8935c..b3a1721 100644 --- a/docs/load-test.md +++ b/docs/load-test.md @@ -109,15 +109,19 @@ Populate from the saved `k6/results/summary-*.json`; never estimate. | Run | Date | Opened | Closed | Open success | p95 event | p95 scenario | Presenter success | Errors | |---|---|---|---|---|---|---|---|---| | v1.0 baseline through Cloudflare | 2026-11-14 08:07 CET | 120 | 120 | 100% | 1.03 s | 194 ms | 100% (82.69 ms) | 0 | -| v1.1 through Cloudflare | pending post-merge run | | | | | | | | +| v1.1 through Cloudflare | 2026-11-14 08:21 CET | 120 | 120 | 100% | 331 ms | 646 ms | 100% (49.60 ms) | 0 | Raw baseline result: `k6/results/summary-20261114T080747.json` (git-ignored). +Raw v1.1 result: `k6/results/summary-20261114T082121.json` (git-ignored). Baseline details: 361/361 checks passed, 2,760 events received, 2,400 heartbeat pings, 120 VUs, test duration 4m53s, presenter POST 82.69 ms. -`p95 ping latency` was 1,045 ms. This run was against the live v1.0 API; rerun -the same command after v1.1 is applied and replace the pending row with exact -numbers. +`p95 ping latency` was 1,045 ms. + +v1.1 details: 361/361 checks passed, 2,760 events received, 2,400 heartbeat +pings, 120 VUs, test duration 4m57.9s, presenter POST 49.60 ms, `p95 ping +latency` 268 ms. All thresholds passed. This run used the presenter POST at +2m30s through Cloudflare and the same `make load` path. ## Notes