docs: record v1.1 load test and failure drill results
infra-ci / manifests (push) Failing after 3s
infra-ci / manifests (pull_request) Failing after 4s

v1.1 Cloudflare run: 120/120 connections, 0 errors, p95 event 331 ms,
p95 scenario 646 ms, presenter 49.60 ms, 2,760 events, 4m57.9s.
Failure drill: ML-down cache/readyz/web pass; API pod restart pass.
Also record v1.1 deployment, stage/live/control verification and hello
removal.
This commit is contained in:
Noel Sabia committed 2026-11-14 08:32:12 +01:00
1 parent 1171a15311
commit bd3176bef0
4 files changed
+54 -39

No files matched your search

+21 -26
View File
@@ -24,22 +24,25 @@ Gate 1 passes only when **all** of these are true:
Do not claim Gate 1 from a local `kubectl port-forward`, a green manifest
dry-run, or a container that only runs on your laptop.
## Current external status (2026-11-14 ~08:00 CET)
## Current external status (2026-11-14 ~08:30 CET)
Re-check before trusting this snapshot:
- **Verified:** cluster access, scheduling, Service DNS/HTTP, probes, Secrets,
non-root + read-only-root + `emptyDir`, and egress to Cloudflare,
Open-Meteo, and `git.noelsabia.dev`.
- **Tunnel active:** `pulsecast-gate1` has routes for `pulse.noelsabia.dev`
and `api-pulse.noelsabia.dev`; `cloudflared` is `1/1` with four HTTP/2
connections. `pulse` already serves the v1.0 Nuxt app
(`x-powered-by: Nuxt`), so **no Cloudflare route switch remains**.
`/stage` and `/live` return 404 on v1.0; v1.1 adds them. `api-pulse` serves
the deployed API.
- **120-VU baseline passed:** 120/120 SSE connections opened/closed, 0
connection errors, p95 event latency 1.03 s, p95 scenario latency 194 ms;
see `docs/load-test.md`. This was against v1.0.
- **v1.1 is deployed and Ready:** web/api/ml `:v1.1`, cloudflared
`2026.11.0`, all four Deployments `1/1`. Numeric UIDs are
`10001:10001` for web/ml and `65532:65532` for API/cloudflared.
- **Tunnel active:** `pulsecast-gate1` serves `pulse.noelsabia.dev` through
Nuxt (`x-powered-by: Nuxt`) and `api-pulse.noelsabia.dev` through the API;
four HTTP/2 connections. `/stage` and `/live` return 200; `/control` is
401 anonymous and 200 with control auth. The temporary hello is deleted.
- **v1.1 120-VU Cloudflare run passed:** 120/120 SSE connections opened and
closed, 0 errors, p95 event 331 ms, p95 scenario 646 ms, presenter POST
49.60 ms, 361/361 checks. See `docs/load-test.md`.
- **Failure drill passed:** ML scaled to 0 while API readiness/forecast cache
and web stayed available; ML restored and API pod restarted cleanly. See
`docs/failure-drills.md`.
- **API/product checks:** 8 sensors, `/api/meta` `mae=3.32`, `wape=27.5`,
`r2=0.777`, cache `source=fastapi`, staffing plan under 2 s.
- **DNS works:** public resolvers return Cloudflare IPs for
`pulse.noelsabia.dev` and `api-pulse.noelsabia.dev`. A local
`systemd-resolved` negative cache can be cleared with
@@ -47,30 +50,22 @@ Re-check before trusting this snapshot:
- **API hostname is `api-pulse.noelsabia.dev`:** Universal SSL covers only
`*.noelsabia.dev`; a two-level `api.pulse.noelsabia.dev` is not covered by
that certificate.
- **Gitea pull is proven:** `gitea-regcred` exists; hello and the v1.0 service
images pulled from Gitea. The Gitea owner is `noel`.
- **v1.1 images pushed:** web/api/ml `:v1.1` were built from the service
`main` worktrees, locally exercised under numeric UIDs/read-only root, and
pushed. The cluster still runs the previous v1.0 images until the v1.1
branch is reviewed/merged and explicitly applied.
- **All three Secrets exist:** `gitea-regcred`, `cloudflared`, `control-auth`.
- **QUIC/UDP 7844 is untested.** TCP/443 to Cloudflare works, so the
`cloudflared` manifest uses `--protocol http2` instead of relying on
unverified UDP.
- **Numeric UIDs are required:** a non-numeric image user breaks
`runAsNonRoot` on SKE (reproduced with `curlimages/curl`). Every pod in
`deploy/k8s/` sets explicit numeric `runAsUser`/`runAsGroup`/`fsGroup`.
## What is possible now vs what is blocked
| Step | Who | Status |
|---|---|---|
| Read-only cluster/manifest check | agent or human | **ready now** (`make gate1-check`) |
| Gitea push login + hello-world push | human | **done**; hello pull and Service verified |
| Cluster/manifest check | agent or human | **done**; static and live checks pass |
| Gitea pull + hello proof | human | **done**; hello deleted after cutover |
| `gitea-regcred`, `cloudflared`, `control-auth` Secrets | human | **done** |
| Cloudflare Tunnel + public hostnames/DNS | human | **done**; tunnel active; `pulse` serves Nuxt |
| `api`, `ml`, `web` images | human | **v1.1 pushed**; local manifest-equivalent checks pass |
| v1.1 apply + smoke + presenter k6 + drills | human | **blocked on branch review/merge** |
| `api`, `ml`, `web` v1.1 images | human | **done**; pushed and deployed |
| v1.1 apply + smoke + presenter k6 + drills | human | **done** |
| Results docs | agent | **pending new PR review/merge** |
`make build` and `make push` cover all three services; no Dockerfile is
missing now. Do not deploy placeholder images under the real service names.