Cloudflare Universal SSL covers *.noelsabia.dev only one level deep. api.pulse.noelsabia.dev is two levels and fails the certificate check. - switch the API base/SSE URLs to api-pulse.noelsabia.dev - update smoke, k6, runbook, and failure drills - record the verified DNS/certificate diagnosis and local cache flush
97 lines
4.1 KiB
Markdown
97 lines
4.1 KiB
Markdown
# Failure drills
|
|
|
|
Run these **after** Gate 1 and outside the demo window. Every command is a
|
|
cluster mutation: the human runs it, not an agent. Record the result and the
|
|
recovery time. Keep `make smoke` and a `curl -N .../api/stream` open while you
|
|
drill.
|
|
|
|
## 1. FastAPI down (cached fallback)
|
|
|
|
Goal: the Go API keeps serving forecasts from baked JSON with `"cached": true`
|
|
and `/api/readyz` stays 200. FastAPI being down must not fail API readiness.
|
|
|
|
```bash
|
|
# Baseline
|
|
curl -sS https://api-pulse.noelsabia.dev/api/readyz
|
|
curl -sS 'https://api-pulse.noelsabia.dev/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base' | jq '.cached'
|
|
|
|
# Stop ML
|
|
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy scale deployment/ml --replicas=0
|
|
|
|
# Verify fallback (allow up to 2 s for the ML timeout)
|
|
curl -sS https://api-pulse.noelsabia.dev/api/readyz
|
|
curl -sS 'https://api-pulse.noelsabia.dev/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base' | jq '{cached, points: (.points|length)}'
|
|
# Expected: readyz 200; cached true; points still present
|
|
|
|
# Recover
|
|
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy scale deployment/ml --replicas=1
|
|
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy rollout status deployment/ml --timeout=180s
|
|
```
|
|
|
|
Pass: no 5xx from `/api/forecast`, `cached` flips to `true` within the 1.5 s
|
|
timeout budget, and the UI shows the cached-forecast banner. Fail: a 502/503 or
|
|
an empty forecast.
|
|
|
|
## 2. Tunnel down
|
|
|
|
Goal: detect a public outage quickly, restart `cloudflared`, and verify both
|
|
public hostnames recover. Local `kubectl port-forward` is a diagnostic only,
|
|
never the public demo path.
|
|
|
|
```bash
|
|
# Stop the tunnel
|
|
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy scale deployment/cloudflared --replicas=0
|
|
|
|
# Verify the outage from outside
|
|
curl -sS -o /dev/null -w '%{http_code}\n' https://pulse.noelsabia.dev
|
|
curl -sS -o /dev/null -w '%{http_code}\n' https://api-pulse.noelsabia.dev/api/healthz
|
|
# Expected: 502/503/530-style edge error, not a valid 200
|
|
|
|
# Recover
|
|
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy scale deployment/cloudflared --replicas=1
|
|
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy rollout status deployment/cloudflared --timeout=120s
|
|
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy logs deployment/cloudflared --tail=50
|
|
# Look for: "Registered tunnel connection"
|
|
|
|
make smoke
|
|
```
|
|
|
|
If the tunnel pod is Running but the URL is still 502, check the dashboard
|
|
routes (`pulse` -> `http://web:3000`, `api-pulse` -> `http://api:8080`) and the
|
|
Secret token. A bad token keeps the pod restarting with an auth error. Rotating
|
|
the tunnel token means updating the Secret and restarting the Deployment.
|
|
|
|
## 3. Pod restart (API and SSE reconnect)
|
|
|
|
Goal: a single API pod restart causes a short 502 window, not a stuck UI. SSE
|
|
clients reconnect with `retry: 3000`; scenario state is in memory and resets to
|
|
the default, which is documented.
|
|
|
|
```bash
|
|
# Watch from two terminals:
|
|
# curl -sS -N https://api-pulse.noelsabia.dev/api/stream
|
|
# while true; do date +%T; curl -sS -o /dev/null -w '%{http_code}\n' https://api-pulse.noelsabia.dev/api/healthz; sleep 1; done
|
|
|
|
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy delete pod -l app.kubernetes.io/name=api
|
|
|
|
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy rollout status deployment/api --timeout=180s
|
|
curl -sS https://api-pulse.noelsabia.dev/api/healthz
|
|
curl -sS https://api-pulse.noelsabia.dev/api/readyz
|
|
```
|
|
|
|
Pass: the health loop shows at most a few seconds of errors, the deployment
|
|
returns to 1/1, `/api/stream` reconnects and emits a `scenario` event, and the
|
|
UI recovers without a manual reload. Note the actual outage duration in the
|
|
paper's reliability section.
|
|
|
|
Repeat for `web` (public page 502 briefly, then reloads) and `ml` (API serves
|
|
cached forecasts, no user-visible failure).
|
|
|
|
## Cleanup / invariants
|
|
|
|
- All deployments back to 1 replica.
|
|
- `make smoke` passes.
|
|
- No Ingress, Jobs, CronJobs, NetworkPolicies, or cluster-scoped resources were
|
|
created during the drills.
|
|
- Never leave `ml` or `cloudflared` scaled to zero.
|