Files
infra-k8/docs/failure-drills.md
Noel Sabia bd3176bef0
infra-ci / manifests (push) Failing after 3s
infra-ci / manifests (pull_request) Failing after 4s
docs: record v1.1 load test and failure drill results
v1.1 Cloudflare run: 120/120 connections, 0 errors, p95 event 331 ms,
p95 scenario 646 ms, presenter 49.60 ms, 2,760 events, 4m57.9s.
Failure drill: ML-down cache/readyz/web pass; API pod restart pass.
Also record v1.1 deployment, stage/live/control verification and hello
removal.
2026-11-14 08:32:12 +01:00

5.2 KiB
Raw Permalink Blame History

Failure drills

Run these after Gate 1 and outside the demo window. Every command is a cluster mutation: the human runs it, not an agent. Record the result and the recovery time. Keep make smoke and a curl -N .../api/stream open while you drill.

Last run — 2026-11-14 07:26:56–07:29:27 UTC

v1.1, executed against the live cluster:

Step Observed Result
ML scaled to 0 /api/readyz 200; forecast 200, cached=true, 168 points in 111 ms; web 200 PASS
Failed bootstrap refresh cache source=fastapi, last_error connection refused, failures=6, next_retry_at set; web/API stayed available PASS
ML scaled back to 1 rollout 1/1; cache recovered to last_error=null, failures=0, /api/readyz 200 PASS
API pod restart pod deleted and recreated; rollout 1/1; /api/healthz 200, /api/readyz 200, 8 sensors, web 200 PASS

The drill took 2m31s including the two 65 s refresh windows. No hello resources existed during this run; it was deleted after the v1.1 cutover.

1. FastAPI down (cached fallback)

Goal: the Go API keeps serving its last good in-memory /bootstrap snapshot with "cached": true and /api/readyz stays 200. FastAPI being down must not fail API readiness after the snapshot has loaded once.

# Baseline
curl -sS https://api-pulse.noelsabia.dev/api/readyz
curl -sS 'https://api-pulse.noelsabia.dev/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base' | jq '.cached'

# Stop ML
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy scale deployment/ml --replicas=0

# Verify fallback (allow up to 2 s for the ML timeout)
curl -sS https://api-pulse.noelsabia.dev/api/readyz
curl -sS 'https://api-pulse.noelsabia.dev/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base' | jq '{cached, points: (.points|length)}'
# Expected: readyz 200; cached true; points still present

# Recover
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy scale deployment/ml --replicas=1
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy rollout status deployment/ml --timeout=180s
# The API refreshes within BOOTSTRAP_REFRESH=60s; confirm the cache is fresh again.
sleep 65
curl -sS https://api-pulse.noelsabia.dev/api/meta | jq '.cache'

Pass: no 5xx from /api/forecast, cached flips to true within the 1.5 s timeout budget, and the UI shows the cached-forecast banner. Fail: a 502/503 or an empty forecast.

2. Tunnel down

Goal: detect a public outage quickly, restart cloudflared, and verify both public hostnames recover. Local kubectl port-forward is a diagnostic only, never the public demo path.

# Stop the tunnel
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy scale deployment/cloudflared --replicas=0

# Verify the outage from outside
curl -sS -o /dev/null -w '%{http_code}\n' https://pulse.noelsabia.dev
curl -sS -o /dev/null -w '%{http_code}\n' https://api-pulse.noelsabia.dev/api/healthz
# Expected: 502/503/530-style edge error, not a valid 200

# Recover
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy scale deployment/cloudflared --replicas=1
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy rollout status deployment/cloudflared --timeout=120s
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy logs deployment/cloudflared --tail=50
# Look for: "Registered tunnel connection"

make smoke

If the tunnel pod is Running but the URL is still 502, check the dashboard routes (pulse -> http://web:3000, api-pulse -> http://api:8080) and the Secret token. A bad token keeps the pod restarting with an auth error. Rotating the tunnel token means updating the Secret and restarting the Deployment.

3. Pod restart (API and SSE reconnect)

Goal: a single API pod restart causes a short 502 window, not a stuck UI. SSE clients reconnect with retry: 3000; scenario state is in memory and resets to the default. The restarted API re-fetches ML /bootstrap before /api/readyz turns 200; keep ML up during this drill.

# Watch from two terminals:
#   curl -sS -N https://api-pulse.noelsabia.dev/api/stream
#   while true; do date +%T; curl -sS -o /dev/null -w '%{http_code}\n' https://api-pulse.noelsabia.dev/api/healthz; sleep 1; done

kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy delete pod -l app.kubernetes.io/name=api

kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy rollout status deployment/api --timeout=180s
curl -sS https://api-pulse.noelsabia.dev/api/healthz
curl -sS https://api-pulse.noelsabia.dev/api/readyz

Pass: the health loop shows at most a few seconds of errors, the deployment returns to 1/1, /api/stream reconnects and emits a scenario event, and the UI recovers without a manual reload. Note the actual outage duration in the paper's reliability section.

Repeat for web (public page 502 briefly, then reloads) and ml (API serves cached forecasts, no user-visible failure).

Cleanup / invariants

  • All deployments back to 1 replica.
  • make smoke passes.
  • No Ingress, Jobs, CronJobs, NetworkPolicies, or cluster-scoped resources were created during the drills.
  • Never leave ml or cloudflared scaled to zero.