Files
infra-k8/docs/failure-drills.md
T
Noel Sabia bd3176bef0
infra-ci / manifests (push) Failing after 3s
infra-ci / manifests (pull_request) Failing after 4s
docs: record v1.1 load test and failure drill results
v1.1 Cloudflare run: 120/120 connections, 0 errors, p95 event 331 ms,
p95 scenario 646 ms, presenter 49.60 ms, 2,760 events, 4m57.9s.
Failure drill: ML-down cache/readyz/web pass; API pod restart pass.
Also record v1.1 deployment, stage/live/control verification and hello
removal.
2026-11-14 08:32:12 +01:00

116 lines
5.2 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Failure drills
Run these **after** Gate 1 and outside the demo window. Every command is a
cluster mutation: the human runs it, not an agent. Record the result and the
recovery time. Keep `make smoke` and a `curl -N .../api/stream` open while you
drill.
## Last run — 2026-11-14 07:26:56–07:29:27 UTC
v1.1, executed against the live cluster:
| Step | Observed | Result |
|---|---|---|
| ML scaled to 0 | `/api/readyz` 200; forecast 200, `cached=true`, 168 points in 111 ms; web 200 | PASS |
| Failed bootstrap refresh | cache `source=fastapi`, `last_error` connection refused, `failures=6`, `next_retry_at` set; web/API stayed available | PASS |
| ML scaled back to 1 | rollout 1/1; cache recovered to `last_error=null`, `failures=0`, `/api/readyz` 200 | PASS |
| API pod restart | pod deleted and recreated; rollout 1/1; `/api/healthz` 200, `/api/readyz` 200, 8 sensors, web 200 | PASS |
The drill took 2m31s including the two 65 s refresh windows. No hello
resources existed during this run; it was deleted after the v1.1 cutover.
## 1. FastAPI down (cached fallback)
Goal: the Go API keeps serving its last good in-memory `/bootstrap` snapshot
with `"cached": true` and `/api/readyz` stays 200. FastAPI being down must not
fail API readiness after the snapshot has loaded once.
```bash
# Baseline
curl -sS https://api-pulse.noelsabia.dev/api/readyz
curl -sS 'https://api-pulse.noelsabia.dev/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base' | jq '.cached'
# Stop ML
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy scale deployment/ml --replicas=0
# Verify fallback (allow up to 2 s for the ML timeout)
curl -sS https://api-pulse.noelsabia.dev/api/readyz
curl -sS 'https://api-pulse.noelsabia.dev/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base' | jq '{cached, points: (.points|length)}'
# Expected: readyz 200; cached true; points still present
# Recover
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy scale deployment/ml --replicas=1
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy rollout status deployment/ml --timeout=180s
# The API refreshes within BOOTSTRAP_REFRESH=60s; confirm the cache is fresh again.
sleep 65
curl -sS https://api-pulse.noelsabia.dev/api/meta | jq '.cache'
```
Pass: no 5xx from `/api/forecast`, `cached` flips to `true` within the 1.5 s
timeout budget, and the UI shows the cached-forecast banner. Fail: a 502/503 or
an empty forecast.
## 2. Tunnel down
Goal: detect a public outage quickly, restart `cloudflared`, and verify both
public hostnames recover. Local `kubectl port-forward` is a diagnostic only,
never the public demo path.
```bash
# Stop the tunnel
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy scale deployment/cloudflared --replicas=0
# Verify the outage from outside
curl -sS -o /dev/null -w '%{http_code}\n' https://pulse.noelsabia.dev
curl -sS -o /dev/null -w '%{http_code}\n' https://api-pulse.noelsabia.dev/api/healthz
# Expected: 502/503/530-style edge error, not a valid 200
# Recover
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy scale deployment/cloudflared --replicas=1
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy rollout status deployment/cloudflared --timeout=120s
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy logs deployment/cloudflared --tail=50
# Look for: "Registered tunnel connection"
make smoke
```
If the tunnel pod is Running but the URL is still 502, check the dashboard
routes (`pulse` -> `http://web:3000`, `api-pulse` -> `http://api:8080`) and the
Secret token. A bad token keeps the pod restarting with an auth error. Rotating
the tunnel token means updating the Secret and restarting the Deployment.
## 3. Pod restart (API and SSE reconnect)
Goal: a single API pod restart causes a short 502 window, not a stuck UI. SSE
clients reconnect with `retry: 3000`; scenario state is in memory and resets to
the default. The restarted API re-fetches ML `/bootstrap` before `/api/readyz`
turns 200; keep ML up during this drill.
```bash
# Watch from two terminals:
# curl -sS -N https://api-pulse.noelsabia.dev/api/stream
# while true; do date +%T; curl -sS -o /dev/null -w '%{http_code}\n' https://api-pulse.noelsabia.dev/api/healthz; sleep 1; done
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy delete pod -l app.kubernetes.io/name=api
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy rollout status deployment/api --timeout=180s
curl -sS https://api-pulse.noelsabia.dev/api/healthz
curl -sS https://api-pulse.noelsabia.dev/api/readyz
```
Pass: the health loop shows at most a few seconds of errors, the deployment
returns to 1/1, `/api/stream` reconnects and emits a `scenario` event, and the
UI recovers without a manual reload. Note the actual outage duration in the
paper's reliability section.
Repeat for `web` (public page 502 briefly, then reloads) and `ml` (API serves
cached forecasts, no user-visible failure).
## Cleanup / invariants
- All deployments back to 1 replica.
- `make smoke` passes.
- No Ingress, Jobs, CronJobs, NetworkPolicies, or cluster-scoped resources were
created during the drills.
- Never leave `ml` or `cloudflared` scaled to zero.