feat: infra v1.1 results #3
No files matched your search
@@ -52,14 +52,19 @@ hostname, and verification steps.
|
||||
- Gitea pull is proven: `gitea-regcred` exists and the hello pod pulled from
|
||||
`git.noelsabia.dev/noel/hello:gate1`; the Service returned HTTP 200.
|
||||
- Tunnel `pulsecast-gate1` is active with `pulse` and `api-pulse` routes.
|
||||
`pulse` still points at the temporary hello Service until the v1.1 cutover.
|
||||
The API hostname is `api-pulse.noelsabia.dev` because Universal SSL only
|
||||
covers `*.noelsabia.dev`.
|
||||
- v1.1 images are built from the service `main` worktrees, locally checked,
|
||||
and pushed. The cluster still runs the previous v1.0 images until this branch
|
||||
is reviewed/merged and explicitly applied.
|
||||
- The v1.1 branch adds web control credentials, API bootstrap config, and the
|
||||
k6 presenter scenario; load-test results are recorded after the approved run.
|
||||
`pulse` serves the v1.1 Nuxt app; `/stage` and `/live` return 200, and
|
||||
`/control` requires auth. The API hostname is
|
||||
`api-pulse.noelsabia.dev` because Universal SSL only covers
|
||||
`*.noelsabia.dev`.
|
||||
- v1.1 is deployed: web/api/ml `:v1.1`, cloudflared `2026.11.0`, all four
|
||||
Deployments Ready.
|
||||
- The v1.1 120-VU Cloudflare run passed: 120/120 connections, 0 errors, p95
|
||||
event 331 ms, p95 scenario 646 ms, presenter 49.60 ms. The v1.0 baseline is
|
||||
retained for comparison in `docs/load-test.md`.
|
||||
- The ML-down/API-restart failure drill passed; details are in
|
||||
`docs/failure-drills.md`.
|
||||
- The results documentation still needs a new PR review/merge; the v1.1
|
||||
runtime code is already merged in `main`.
|
||||
|
||||
## CI
|
||||
|
||||
|
||||
+22
-22
@@ -24,17 +24,25 @@ Gate 1 passes only when **all** of these are true:
|
||||
Do not claim Gate 1 from a local `kubectl port-forward`, a green manifest
|
||||
dry-run, or a container that only runs on your laptop.
|
||||
|
||||
## Current external status (2026-11-14 ~08:00 CET)
|
||||
## Current external status (2026-11-14 ~08:30 CET)
|
||||
|
||||
Re-check before trusting this snapshot:
|
||||
|
||||
- **Verified:** cluster access, scheduling, Service DNS/HTTP, probes, Secrets,
|
||||
non-root + read-only-root + `emptyDir`, and egress to Cloudflare,
|
||||
Open-Meteo, and `git.noelsabia.dev`.
|
||||
- **Tunnel active:** `pulsecast-gate1` has routes for `pulse.noelsabia.dev`
|
||||
and `api-pulse.noelsabia.dev`; `cloudflared` is `1/1` with four HTTP/2
|
||||
connections. `api-pulse` serves the deployed API; `pulse` still points at
|
||||
the temporary hello Service until the web cutover.
|
||||
- **v1.1 is deployed and Ready:** web/api/ml `:v1.1`, cloudflared
|
||||
`2026.11.0`, all four Deployments `1/1`. Numeric UIDs are
|
||||
`10001:10001` for web/ml and `65532:65532` for API/cloudflared.
|
||||
- **Tunnel active:** `pulsecast-gate1` serves `pulse.noelsabia.dev` through
|
||||
Nuxt (`x-powered-by: Nuxt`) and `api-pulse.noelsabia.dev` through the API;
|
||||
four HTTP/2 connections. `/stage` and `/live` return 200; `/control` is
|
||||
401 anonymous and 200 with control auth. The temporary hello is deleted.
|
||||
- **v1.1 120-VU Cloudflare run passed:** 120/120 SSE connections opened and
|
||||
closed, 0 errors, p95 event 331 ms, p95 scenario 646 ms, presenter POST
|
||||
49.60 ms, 361/361 checks. See `docs/load-test.md`.
|
||||
- **Failure drill passed:** ML scaled to 0 while API readiness/forecast cache
|
||||
and web stayed available; ML restored and API pod restarted cleanly. See
|
||||
`docs/failure-drills.md`.
|
||||
- **API/product checks:** 8 sensors, `/api/meta` `mae=3.32`, `wape=27.5`,
|
||||
`r2=0.777`, cache `source=fastapi`, staffing plan under 2 s.
|
||||
- **DNS works:** public resolvers return Cloudflare IPs for
|
||||
`pulse.noelsabia.dev` and `api-pulse.noelsabia.dev`. A local
|
||||
`systemd-resolved` negative cache can be cleared with
|
||||
@@ -42,30 +50,22 @@ Re-check before trusting this snapshot:
|
||||
- **API hostname is `api-pulse.noelsabia.dev`:** Universal SSL covers only
|
||||
`*.noelsabia.dev`; a two-level `api.pulse.noelsabia.dev` is not covered by
|
||||
that certificate.
|
||||
- **Gitea pull is proven:** `gitea-regcred` exists; hello and the v1.0 service
|
||||
images pulled from Gitea. The Gitea owner is `noel`.
|
||||
- **v1.1 images pushed:** web/api/ml `:v1.1` were built from the service
|
||||
`main` worktrees, locally exercised under numeric UIDs/read-only root, and
|
||||
pushed. The cluster still runs the previous v1.0 images until the v1.1
|
||||
branch is reviewed/merged and explicitly applied.
|
||||
- **All three Secrets exist:** `gitea-regcred`, `cloudflared`, `control-auth`.
|
||||
- **QUIC/UDP 7844 is untested.** TCP/443 to Cloudflare works, so the
|
||||
`cloudflared` manifest uses `--protocol http2` instead of relying on
|
||||
unverified UDP.
|
||||
- **Numeric UIDs are required:** a non-numeric image user breaks
|
||||
`runAsNonRoot` on SKE (reproduced with `curlimages/curl`). Every pod in
|
||||
`deploy/k8s/` sets explicit numeric `runAsUser`/`runAsGroup`/`fsGroup`.
|
||||
|
||||
## What is possible now vs what is blocked
|
||||
|
||||
| Step | Who | Status |
|
||||
|---|---|---|
|
||||
| Read-only cluster/manifest check | agent or human | **ready now** (`make gate1-check`) |
|
||||
| Gitea push login + hello-world push | human | **done**; hello pull and Service verified |
|
||||
| Cluster/manifest check | agent or human | **done**; static and live checks pass |
|
||||
| Gitea pull + hello proof | human | **done**; hello deleted after cutover |
|
||||
| `gitea-regcred`, `cloudflared`, `control-auth` Secrets | human | **done** |
|
||||
| Cloudflare Tunnel + public hostnames/DNS | human | **done**; tunnel active; `pulse` still points at hello |
|
||||
| `api`, `ml`, `web` images | human | **v1.1 pushed**; local manifest-equivalent checks pass |
|
||||
| v1.1 apply + smoke + presenter k6 + drills | human | **blocked on branch review/merge** |
|
||||
| Cloudflare Tunnel + public hostnames/DNS | human | **done**; tunnel active; `pulse` serves Nuxt |
|
||||
| `api`, `ml`, `web` v1.1 images | human | **done**; pushed and deployed |
|
||||
| v1.1 apply + smoke + presenter k6 + drills | human | **done** |
|
||||
| Results docs | agent | **pending new PR review/merge** |
|
||||
|
||||
`make build` and `make push` cover all three services; no Dockerfile is
|
||||
missing now. Do not deploy placeholder images under the real service names.
|
||||
|
||||
@@ -5,6 +5,20 @@ cluster mutation: the human runs it, not an agent. Record the result and the
|
||||
recovery time. Keep `make smoke` and a `curl -N .../api/stream` open while you
|
||||
drill.
|
||||
|
||||
## Last run — 2026-11-14 07:26:56–07:29:27 UTC
|
||||
|
||||
v1.1, executed against the live cluster:
|
||||
|
||||
| Step | Observed | Result |
|
||||
|---|---|---|
|
||||
| ML scaled to 0 | `/api/readyz` 200; forecast 200, `cached=true`, 168 points in 111 ms; web 200 | PASS |
|
||||
| Failed bootstrap refresh | cache `source=fastapi`, `last_error` connection refused, `failures=6`, `next_retry_at` set; web/API stayed available | PASS |
|
||||
| ML scaled back to 1 | rollout 1/1; cache recovered to `last_error=null`, `failures=0`, `/api/readyz` 200 | PASS |
|
||||
| API pod restart | pod deleted and recreated; rollout 1/1; `/api/healthz` 200, `/api/readyz` 200, 8 sensors, web 200 | PASS |
|
||||
|
||||
The drill took 2m31s including the two 65 s refresh windows. No hello
|
||||
resources existed during this run; it was deleted after the v1.1 cutover.
|
||||
|
||||
## 1. FastAPI down (cached fallback)
|
||||
|
||||
Goal: the Go API keeps serving its last good in-memory `/bootstrap` snapshot
|
||||
|
||||
+14
-1
@@ -108,7 +108,20 @@ Populate from the saved `k6/results/summary-*.json`; never estimate.
|
||||
|
||||
| Run | Date | Opened | Closed | Open success | p95 event | p95 scenario | Presenter success | Errors |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| v1.1 through Cloudflare | pending post-merge run | | | | | | | |
|
||||
| v1.0 baseline through Cloudflare | 2026-11-14 08:07 CET | 120 | 120 | 100% | 1.03 s | 194 ms | 100% (82.69 ms) | 0 |
|
||||
| v1.1 through Cloudflare | 2026-11-14 08:21 CET | 120 | 120 | 100% | 331 ms | 646 ms | 100% (49.60 ms) | 0 |
|
||||
|
||||
Raw baseline result: `k6/results/summary-20261114T080747.json` (git-ignored).
|
||||
Raw v1.1 result: `k6/results/summary-20261114T082121.json` (git-ignored).
|
||||
|
||||
Baseline details: 361/361 checks passed, 2,760 events received, 2,400
|
||||
heartbeat pings, 120 VUs, test duration 4m53s, presenter POST 82.69 ms.
|
||||
`p95 ping latency` was 1,045 ms.
|
||||
|
||||
v1.1 details: 361/361 checks passed, 2,760 events received, 2,400 heartbeat
|
||||
pings, 120 VUs, test duration 4m57.9s, presenter POST 49.60 ms, `p95 ping
|
||||
latency` 268 ms. All thresholds passed. This run used the presenter POST at
|
||||
2m30s through Cloudflare and the same `make load` path.
|
||||
|
||||
## Notes
|
||||
|
||||
|
||||
Reference in new issue
Block a user