# Failure drills Run these **after** Gate 1 and outside the demo window. Every command is a cluster mutation: the human runs it, not an agent. Record the result and the recovery time. Keep `make smoke` and a `curl -N .../api/stream` open while you drill. ## Last run — 2026-11-14 07:26:56–07:29:27 UTC v1.1, executed against the live cluster: | Step | Observed | Result | |---|---|---| | ML scaled to 0 | `/api/readyz` 200; forecast 200, `cached=true`, 168 points in 111 ms; web 200 | PASS | | Failed bootstrap refresh | cache `source=fastapi`, `last_error` connection refused, `failures=6`, `next_retry_at` set; web/API stayed available | PASS | | ML scaled back to 1 | rollout 1/1; cache recovered to `last_error=null`, `failures=0`, `/api/readyz` 200 | PASS | | API pod restart | pod deleted and recreated; rollout 1/1; `/api/healthz` 200, `/api/readyz` 200, 8 sensors, web 200 | PASS | The drill took 2m31s including the two 65 s refresh windows. No hello resources existed during this run; it was deleted after the v1.1 cutover. ## 1. FastAPI down (cached fallback) Goal: the Go API keeps serving its last good in-memory `/bootstrap` snapshot with `"cached": true` and `/api/readyz` stays 200. FastAPI being down must not fail API readiness after the snapshot has loaded once. ```bash # Baseline curl -sS https://api-pulse.noelsabia.dev/api/readyz curl -sS 'https://api-pulse.noelsabia.dev/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base' | jq '.cached' # Stop ML kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy scale deployment/ml --replicas=0 # Verify fallback (allow up to 2 s for the ML timeout) curl -sS https://api-pulse.noelsabia.dev/api/readyz curl -sS 'https://api-pulse.noelsabia.dev/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base' | jq '{cached, points: (.points|length)}' # Expected: readyz 200; cached true; points still present # Recover kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy scale deployment/ml --replicas=1 kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy rollout status deployment/ml --timeout=180s # The API refreshes within BOOTSTRAP_REFRESH=60s; confirm the cache is fresh again. sleep 65 curl -sS https://api-pulse.noelsabia.dev/api/meta | jq '.cache' ``` Pass: no 5xx from `/api/forecast`, `cached` flips to `true` within the 1.5 s timeout budget, and the UI shows the cached-forecast banner. Fail: a 502/503 or an empty forecast. ## 2. Tunnel down Goal: detect a public outage quickly, restart `cloudflared`, and verify both public hostnames recover. Local `kubectl port-forward` is a diagnostic only, never the public demo path. ```bash # Stop the tunnel kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy scale deployment/cloudflared --replicas=0 # Verify the outage from outside curl -sS -o /dev/null -w '%{http_code}\n' https://pulse.noelsabia.dev curl -sS -o /dev/null -w '%{http_code}\n' https://api-pulse.noelsabia.dev/api/healthz # Expected: 502/503/530-style edge error, not a valid 200 # Recover kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy scale deployment/cloudflared --replicas=1 kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy rollout status deployment/cloudflared --timeout=120s kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy logs deployment/cloudflared --tail=50 # Look for: "Registered tunnel connection" make smoke ``` If the tunnel pod is Running but the URL is still 502, check the dashboard routes (`pulse` -> `http://web:3000`, `api-pulse` -> `http://api:8080`) and the Secret token. A bad token keeps the pod restarting with an auth error. Rotating the tunnel token means updating the Secret and restarting the Deployment. ## 3. Pod restart (API and SSE reconnect) Goal: a single API pod restart causes a short 502 window, not a stuck UI. SSE clients reconnect with `retry: 3000`; scenario state is in memory and resets to the default. The restarted API re-fetches ML `/bootstrap` before `/api/readyz` turns 200; keep ML up during this drill. ```bash # Watch from two terminals: # curl -sS -N https://api-pulse.noelsabia.dev/api/stream # while true; do date +%T; curl -sS -o /dev/null -w '%{http_code}\n' https://api-pulse.noelsabia.dev/api/healthz; sleep 1; done kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy delete pod -l app.kubernetes.io/name=api kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy rollout status deployment/api --timeout=180s curl -sS https://api-pulse.noelsabia.dev/api/healthz curl -sS https://api-pulse.noelsabia.dev/api/readyz ``` Pass: the health loop shows at most a few seconds of errors, the deployment returns to 1/1, `/api/stream` reconnects and emits a `scenario` event, and the UI recovers without a manual reload. Note the actual outage duration in the paper's reliability section. Repeat for `web` (public page 502 briefly, then reloads) and `ml` (API serves cached forecasts, no user-visible failure). ## Cleanup / invariants - All deployments back to 1 replica. - `make smoke` passes. - No Ingress, Jobs, CronJobs, NetworkPolicies, or cluster-scoped resources were created during the drills. - Never leave `ml` or `cloudflared` scaled to zero.