docs: record v1.1 load test and failure drill results
infra-ci / manifests (push) Failing after 3s
infra-ci / manifests (pull_request) Failing after 4s

v1.1 Cloudflare run: 120/120 connections, 0 errors, p95 event 331 ms,
p95 scenario 646 ms, presenter 49.60 ms, 2,760 events, 4m57.9s.
Failure drill: ML-down cache/readyz/web pass; API pod restart pass.
Also record v1.1 deployment, stage/live/control verification and hello
removal.
This commit is contained in:
Noel Sabia committed 2026-11-14 08:32:12 +01:00
1 parent 1171a15311
commit bd3176bef0
4 files changed
+54 -39

No files matched your search

+14
View File
@@ -5,6 +5,20 @@ cluster mutation: the human runs it, not an agent. Record the result and the
recovery time. Keep `make smoke` and a `curl -N .../api/stream` open while you
drill.
## Last run — 2026-11-14 07:26:56–07:29:27 UTC
v1.1, executed against the live cluster:
| Step | Observed | Result |
|---|---|---|
| ML scaled to 0 | `/api/readyz` 200; forecast 200, `cached=true`, 168 points in 111 ms; web 200 | PASS |
| Failed bootstrap refresh | cache `source=fastapi`, `last_error` connection refused, `failures=6`, `next_retry_at` set; web/API stayed available | PASS |
| ML scaled back to 1 | rollout 1/1; cache recovered to `last_error=null`, `failures=0`, `/api/readyz` 200 | PASS |
| API pod restart | pod deleted and recreated; rollout 1/1; `/api/healthz` 200, `/api/readyz` 200, 8 sensors, web 200 | PASS |
The drill took 2m31s including the two 65 s refresh windows. No hello
resources existed during this run; it was deleted after the v1.1 cutover.
## 1. FastAPI down (cached fallback)
Goal: the Go API keeps serving its last good in-memory `/bootstrap` snapshot