chore: bootstrap infra-k8 scaffold
This commit is contained in:
commit
aeee20cb7f
14 files changed
+1256
No files matched your search
@@ -0,0 +1,96 @@
|
||||
# Failure drills
|
||||
|
||||
Run these **after** Gate 1 and outside the demo window. Every command is a
|
||||
cluster mutation: the human runs it, not an agent. Record the result and the
|
||||
recovery time. Keep `make smoke` and a `curl -N .../api/stream` open while you
|
||||
drill.
|
||||
|
||||
## 1. FastAPI down (cached fallback)
|
||||
|
||||
Goal: the Go API keeps serving forecasts from baked JSON with `"cached": true`
|
||||
and `/api/readyz` stays 200. FastAPI being down must not fail API readiness.
|
||||
|
||||
```bash
|
||||
# Baseline
|
||||
curl -sS https://api.pulse.noelsabia.dev/api/readyz
|
||||
curl -sS 'https://api.pulse.noelsabia.dev/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base' | jq '.cached'
|
||||
|
||||
# Stop ML
|
||||
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy scale deployment/ml --replicas=0
|
||||
|
||||
# Verify fallback (allow up to 2 s for the ML timeout)
|
||||
curl -sS https://api.pulse.noelsabia.dev/api/readyz
|
||||
curl -sS 'https://api.pulse.noelsabia.dev/api/forecast?sensor=hn-stadtgalerie-01&horizon=7d&scenario=base' | jq '{cached, points: (.points|length)}'
|
||||
# Expected: readyz 200; cached true; points still present
|
||||
|
||||
# Recover
|
||||
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy scale deployment/ml --replicas=1
|
||||
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy rollout status deployment/ml --timeout=180s
|
||||
```
|
||||
|
||||
Pass: no 5xx from `/api/forecast`, `cached` flips to `true` within the 1.5 s
|
||||
timeout budget, and the UI shows the cached-forecast banner. Fail: a 502/503 or
|
||||
an empty forecast.
|
||||
|
||||
## 2. Tunnel down
|
||||
|
||||
Goal: detect a public outage quickly, restart `cloudflared`, and verify both
|
||||
public hostnames recover. Local `kubectl port-forward` is a diagnostic only,
|
||||
never the public demo path.
|
||||
|
||||
```bash
|
||||
# Stop the tunnel
|
||||
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy scale deployment/cloudflared --replicas=0
|
||||
|
||||
# Verify the outage from outside
|
||||
curl -sS -o /dev/null -w '%{http_code}\n' https://pulse.noelsabia.dev
|
||||
curl -sS -o /dev/null -w '%{http_code}\n' https://api.pulse.noelsabia.dev/api/healthz
|
||||
# Expected: 502/503/530-style edge error, not a valid 200
|
||||
|
||||
# Recover
|
||||
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy scale deployment/cloudflared --replicas=1
|
||||
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy rollout status deployment/cloudflared --timeout=120s
|
||||
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy logs deployment/cloudflared --tail=50
|
||||
# Look for: "Registered tunnel connection"
|
||||
|
||||
make smoke
|
||||
```
|
||||
|
||||
If the tunnel pod is Running but the URL is still 502, check the dashboard
|
||||
routes (`pulse` -> `http://web:3000`, `api.pulse` -> `http://api:8080`) and the
|
||||
Secret token. A bad token keeps the pod restarting with an auth error. Rotating
|
||||
the tunnel token means updating the Secret and restarting the Deployment.
|
||||
|
||||
## 3. Pod restart (API and SSE reconnect)
|
||||
|
||||
Goal: a single API pod restart causes a short 502 window, not a stuck UI. SSE
|
||||
clients reconnect with `retry: 3000`; scenario state is in memory and resets to
|
||||
the default, which is documented.
|
||||
|
||||
```bash
|
||||
# Watch from two terminals:
|
||||
# curl -sS -N https://api.pulse.noelsabia.dev/api/stream
|
||||
# while true; do date +%T; curl -sS -o /dev/null -w '%{http_code}\n' https://api.pulse.noelsabia.dev/api/healthz; sleep 1; done
|
||||
|
||||
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy delete pod -l app.kubernetes.io/name=api
|
||||
|
||||
kubectl --context fch26-onemanarmy-sa-context -n fch26-onemanarmy rollout status deployment/api --timeout=180s
|
||||
curl -sS https://api.pulse.noelsabia.dev/api/healthz
|
||||
curl -sS https://api.pulse.noelsabia.dev/api/readyz
|
||||
```
|
||||
|
||||
Pass: the health loop shows at most a few seconds of errors, the deployment
|
||||
returns to 1/1, `/api/stream` reconnects and emits a `scenario` event, and the
|
||||
UI recovers without a manual reload. Note the actual outage duration in the
|
||||
paper's reliability section.
|
||||
|
||||
Repeat for `web` (public page 502 briefly, then reloads) and `ml` (API serves
|
||||
cached forecasts, no user-visible failure).
|
||||
|
||||
## Cleanup / invariants
|
||||
|
||||
- All deployments back to 1 replica.
|
||||
- `make smoke` passes.
|
||||
- No Ingress, Jobs, CronJobs, NetworkPolicies, or cluster-scoped resources were
|
||||
created during the drills.
|
||||
- Never leave `ml` or `cloudflared` scaled to zero.
|
||||
@@ -0,0 +1,91 @@
|
||||
# k6 load test
|
||||
|
||||
`k6/sse.js` holds 120 concurrent Server-Sent Events connections for 5 minutes
|
||||
and reports connection count plus p95 event latency.
|
||||
|
||||
## Why a custom k6 binary
|
||||
|
||||
Native k6 buffers a complete HTTP response, so it cannot parse a stream that
|
||||
never ends. The script imports the community `k6/x/sse` extension
|
||||
(`github.com/phymbert/xk6-sse`). Newer k6 releases resolve community extensions
|
||||
automatically. If the stock `grafana/k6` image rejects the import, build a
|
||||
binary with the extension:
|
||||
|
||||
```bash
|
||||
make k6-build # creates k6/k6-sse (git-ignored)
|
||||
K6=./k6/k6-sse make load
|
||||
```
|
||||
|
||||
`make load` uses, in order: `k6` on `PATH`, `k6/k6-sse`, then
|
||||
`docker run --rm -i grafana/k6`.
|
||||
|
||||
## Procedure
|
||||
|
||||
1. Confirm the API is healthy and the SSE route works:
|
||||
|
||||
```bash
|
||||
curl -sS https://api.pulse.noelsabia.dev/api/healthz
|
||||
curl -sS -N --max-time 20 https://api.pulse.noelsabia.dev/api/stream
|
||||
```
|
||||
|
||||
You should see `retry: 3000` and a `scenario` event immediately, then a
|
||||
`ping` every 15 s.
|
||||
|
||||
2. Run the load test:
|
||||
|
||||
```bash
|
||||
make load
|
||||
# or: API_URL=https://api.pulse.noelsabia.dev k6 run k6/sse.js
|
||||
```
|
||||
|
||||
`make load` writes `k6/results/summary-<timestamp>.json` (git-ignored).
|
||||
|
||||
3. During the run, post a scenario change so the scenario-event latency has
|
||||
samples and the propagation path is exercised:
|
||||
|
||||
```bash
|
||||
curl -u "$CONTROL_USER:$CONTROL_PASS" -X POST \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"scenario":"sunny-saturday-market","weather":{"temp_delta_c":4,"precip_mm":0}}' \
|
||||
https://api.pulse.noelsabia.dev/api/scenario
|
||||
```
|
||||
|
||||
Repeat every 30–60 s with different scenarios if you want a distribution.
|
||||
Without a change, `sse_event_latency` is measured from heartbeat `ping`
|
||||
events, which still proves the stream is live.
|
||||
|
||||
4. Record these numbers in the paper/README from the saved summary:
|
||||
- `sse_connections_opened` count (must be 120)
|
||||
- `sse_connections_active` max (must reach 120)
|
||||
- `sse_open_success` rate (must be >= 99%)
|
||||
- `sse_event_latency` p95 (target < 2000 ms)
|
||||
- `sse_connection_errors` (target 0)
|
||||
- `sse_events_received` total
|
||||
|
||||
## Pass criteria
|
||||
|
||||
| Metric | Target |
|
||||
|---|---|
|
||||
| Connections opened | 120 |
|
||||
| Peak active connections | 120 |
|
||||
| Open success rate | >= 99% |
|
||||
| p95 event latency | < 2000 ms |
|
||||
| Unexpected connection errors | 0 |
|
||||
| Duration | 5 minutes |
|
||||
|
||||
The k6 thresholds enforce the first, third, and fourth; the others are read
|
||||
from the summary. A p95 above 2 s means Cloudflare, the Go SSE hub, or the
|
||||
client is buffering/delaying events; check `X-Accel-Buffering: no`, the 15 s
|
||||
heartbeat, and whether the connection is HTTP/2-buffered at the edge.
|
||||
|
||||
## Notes
|
||||
|
||||
- `sse_event_latency` uses the server timestamp in `data.t` (ping) or `data.at`
|
||||
(scenario). It includes clock skew between the k6 host and the pod; keep both
|
||||
on NTP. Negative or > 1 h values are ignored.
|
||||
- Each VU closes after 20 heartbeats (20 × 15 s = 5 min), so the run ends
|
||||
cleanly instead of waiting for the 7-minute request timeout.
|
||||
- One API replica means all clients share one SSE hub; a pod restart drops
|
||||
every stream and clients reconnect after `retry: 3000`.
|
||||
- k6 itself does not collect server CPU/memory. Use `kubectl top pods` (if
|
||||
metrics-server is available) or watch the pod during the run.
|
||||
Reference in new issue
Block a user