v1.1 Cloudflare run: 120/120 connections, 0 errors, p95 event 331 ms, p95 scenario 646 ms, presenter 49.60 ms, 2,760 events, 4m57.9s. Failure drill: ML-down cache/readyz/web pass; API pod restart pass. Also record v1.1 deployment, stage/live/control verification and hello removal.
96 lines
4.5 KiB
Markdown
96 lines
4.5 KiB
Markdown
# PulseCast infra/QA
|
|
|
|
Namespace-only Kubernetes manifests, smoke tests, and the k6 SSE load test for
|
|
PulseCast. This repo never applies cluster changes by itself: the human runs the
|
|
commands in `docs/deploy.md`.
|
|
|
|
## Layout
|
|
|
|
```text
|
|
deploy/k8s/ web, api, ml + cloudflared Deployments and Services, ConfigMaps
|
|
deploy/examples/ example Secret shapes (placeholders only, never applied)
|
|
scripts/smoke.sh public + in-cluster smoke checks
|
|
scripts/gate1-check.sh read-only permissions/manifest/secret readiness check
|
|
scripts/check_manifests.py cluster-free manifest guardrails used by CI
|
|
.gitea/workflows/ci.yml manifest validation; no secrets, no cluster writes
|
|
k6/sse.js 120 concurrent SSE clients for 5 minutes
|
|
docs/deploy.md Gate 1 runbook and exact human checklist
|
|
docs/load-test.md load-test procedure and how to record results
|
|
docs/failure-drills.md FastAPI-down, tunnel-down, pod-restart drills
|
|
Makefile validate, build, push, deploy, smoke, load helpers
|
|
```
|
|
|
|
## Quick start
|
|
|
|
```bash
|
|
export KUBECONFIG=/path/to/fch26-onemanarmy-sa-kubeconfig.yaml # chmod 600
|
|
make gate1-check # read-only: permissions, guardrails, dry-run, Secrets
|
|
make validate # client-side dry run only, never applies
|
|
make build # all three images
|
|
make push # after docker login to Gitea
|
|
make deploy CONFIRM=yes # human only; requires the three Secrets first
|
|
make smoke # public HTTPS + API + ML health checks
|
|
make k6-build # once: pinned k6 + SSE extension
|
|
make load # 120 SSE clients for 5 minutes
|
|
```
|
|
|
|
`make gate1-check` exits 0 when ready, 1 on a hard permission/manifest failure,
|
|
and 2 when only human prerequisites (Secrets, Deployments) are pending. It
|
|
never mutates the cluster. DNS and public HTTPS are checked separately in the
|
|
runbook and by `make smoke`.
|
|
|
|
Read `docs/deploy.md` before the first deploy. It contains the Gitea token,
|
|
`docker login`, hello-world push, `gitea-regcred`, Cloudflare Tunnel, public
|
|
hostname, and verification steps.
|
|
|
|
## Gate 1 status
|
|
|
|
- Static `make gate1-check STATIC_ONLY=1`/CI passes. The live check correctly
|
|
flags the cluster's v1.0 images until the reviewed v1.1 branch is applied.
|
|
- All three v1.1 images build and pass local manifest-equivalent runtime checks
|
|
(read-only root, `/tmp` tmpfs, numeric UID). There is no Dockerfile blocker.
|
|
- Gitea pull is proven: `gitea-regcred` exists and the hello pod pulled from
|
|
`git.noelsabia.dev/noel/hello:gate1`; the Service returned HTTP 200.
|
|
- Tunnel `pulsecast-gate1` is active with `pulse` and `api-pulse` routes.
|
|
`pulse` serves the v1.1 Nuxt app; `/stage` and `/live` return 200, and
|
|
`/control` requires auth. The API hostname is
|
|
`api-pulse.noelsabia.dev` because Universal SSL only covers
|
|
`*.noelsabia.dev`.
|
|
- v1.1 is deployed: web/api/ml `:v1.1`, cloudflared `2026.11.0`, all four
|
|
Deployments Ready.
|
|
- The v1.1 120-VU Cloudflare run passed: 120/120 connections, 0 errors, p95
|
|
event 331 ms, p95 scenario 646 ms, presenter 49.60 ms. The v1.0 baseline is
|
|
retained for comparison in `docs/load-test.md`.
|
|
- The ML-down/API-restart failure drill passed; details are in
|
|
`docs/failure-drills.md`.
|
|
- The results documentation still needs a new PR review/merge; the v1.1
|
|
runtime code is already merged in `main`.
|
|
|
|
## CI
|
|
|
|
`.gitea/workflows/ci.yml` runs `make gate1-check STATIC_ONLY=1` and
|
|
`make validate`. It uses no Secrets, pushes no images, and never mutates a
|
|
cluster. If the Gitea runner is unavailable, run the same two commands
|
|
locally; `STATIC_ONLY=1` needs no kubeconfig.
|
|
|
|
## Git workflow
|
|
|
|
`main` is integration. New work happens on a `feat/<topic>` branch in its own
|
|
worktree. Before every push: `git fetch origin && git merge origin/main`.
|
|
Never force-push `main`. The initial scaffold is the one bootstrap exception
|
|
committed directly to `main`.
|
|
|
|
## Safety rules
|
|
|
|
- No Ingress, cert-manager, external-dns, Jobs, CronJobs, CRDs, Roles,
|
|
RoleBindings, NetworkPolicies, or any cluster-scoped resource.
|
|
- Exposure only through the `cloudflared` Deployment.
|
|
- No secrets, tokens, kubeconfig, or raw CSV in git.
|
|
- The human applies cluster mutations. `make deploy` refuses to run without
|
|
`CONFIRM=yes`.
|
|
- Every pod sets explicit numeric `runAsUser`/`runAsGroup`/`fsGroup`: web/ml
|
|
`10001`, api/cloudflared `65532`. SKE rejects a non-numeric image user under
|
|
`runAsNonRoot`.
|
|
- The tunnel uses HTTP/2 (`--protocol http2`) because TCP/443 to Cloudflare is
|
|
verified and UDP/7844 is not.
|