# Deploy runbook — Gate 1 Target: three services (`web`, `api`, `ml`) plus one `cloudflared` Deployment in namespace `fch26-onemanarmy`, reachable over public HTTPS, with Gitea image pulls working. **The human runs every cluster-mutating command in this document.** The infra agent only writes and validates manifests. No Ingress, cert-manager, external-dns, Jobs, CronJobs, CRDs, Roles, RoleBindings, NetworkPolicies, or other cluster-scoped resources are used. ## Gate 1 checklist Run top to bottom. Stop at the first failure and use [Troubleshooting](#troubleshooting). - [ ] 1. Kubeconfig exported, `600`, correct context, namespace reachable. - [ ] 2. Gitea **read-only package token** created and `docker login` succeeds. - [ ] 3. Hello-world image pushed and pulled by the cluster with `gitea-regcred`. - [ ] 4. Secrets created: `gitea-regcred`, `cloudflared`, `control-auth`. - [ ] 5. Real `web`/`api`/`ml` images built, tagged `gate1`, pushed to Gitea. - [ ] 6. `make validate` passes locally. - [ ] 7. Cloudflare Tunnel exists; both public hostnames configured. - [ ] 8. `make deploy CONFIRM=yes` completes and all four Deployments are ready. - [ ] 9. `make smoke SMOKE_SSE=1` passes. - [ ] 10. `https://pulse.noelsabia.dev` opens on a phone over mobile data. - [ ] 11. `https://api.pulse.noelsabia.dev/api/healthz` returns 200. - [ ] 12. Record the deployed image tags and the time in the status note. ## 0. Prerequisites The kubeconfig is **not in this repo**. It lives outside git: ```bash export KUBECONFIG="/home/noel/Documents/Hackathon_2026/OKF_Vault/fch26-onemanarmy-sa-kubeconfig.yaml" export NS=fch26-onemanarmy export CTX=fch26-onemanarmy-sa-context export REGISTRY=git.noelsabia.dev export OWNER=noelsabia export TAG=gate1 chmod 600 "$KUBECONFIG" ``` Verify read-only first: ```bash kubectl config current-context # fch26-onemanarmy-sa-context kubectl -n "$NS" auth can-i create deployments # yes kubectl -n "$NS" auth can-i create services # yes kubectl -n "$NS" auth can-i create ingresses # no kubectl -n "$NS" auth can-i create jobs # no kubectl -n "$NS" get pods,svc,secrets ``` If any of the first three answers differ, stop and report; do not work around permissions with cluster-scoped resources. Local tools: `git`, `make`, `bash`, `curl`, `docker`, `kubectl`. Optional: `kubeconform` (schema validation), `jq`, `k6` built with the SSE extension. ## 1. Gitea read-only package token In Gitea: **Settings → Applications → Generate New Token** (or the package token screen). Scope: read-only package access. Do not reuse the admin token. ```bash read -rsp 'Gitea package token: ' GITEA_TOKEN; echo printf '%s' "$GITEA_TOKEN" | docker login "$REGISTRY" -u "$OWNER" --password-stdin ``` A successful login prints `Login Succeeded`. Never put the token in a file, a command that lands in shell history, or this repo. ## 2. Hello-world image pull (prove the registry path) ```bash docker pull nginxinc/nginx-unprivileged:alpine docker tag nginxinc/nginx-unprivileged:alpine "$REGISTRY/$OWNER/hello:gate1" docker push "$REGISTRY/$OWNER/hello:gate1" kubectl -n "$NS" create secret docker-registry gitea-regcred \ --docker-server="$REGISTRY" \ --docker-username="$OWNER" \ --docker-password="$GITEA_TOKEN" kubectl -n "$NS" create deployment hello \ --image="$REGISTRY/$OWNER/hello:gate1" kubectl -n "$NS" patch deployment hello --type=merge -p \ '{"spec":{"template":{"spec":{"imagePullSecrets":[{"name":"gitea-regcred"}]}}}}' kubectl -n "$NS" rollout status deployment/hello --timeout=180s kubectl -n "$NS" get pods -l app=hello ``` Pass: the pod is `Running` and `kubectl describe pod` shows the image pulled. Clean up before the real deploy: ```bash kubectl -n "$NS" delete deployment hello ``` If this fails, fix the token/secret before building the application images. ## 3. Create the remaining Secrets Generate values locally; never commit them. ```bash read -rsp 'Control password: ' CONTROL_PASS; echo kubectl -n "$NS" create secret generic control-auth \ --from-literal=CONTROL_USER=control \ --from-literal=CONTROL_PASS="$CONTROL_PASS" # Token from Cloudflare Zero Trust → Networks → Tunnels → your tunnel. read -rsp 'Cloudflare tunnel token: ' TUNNEL_TOKEN; echo kubectl -n "$NS" create secret generic cloudflared \ --from-literal=TUNNEL_TOKEN="$TUNNEL_TOKEN" kubectl -n "$NS" get secrets ``` `make secret-commands` prints the same commands with placeholders. ## 4. Build and push the three images The manifests expect: | Service | Image | Port | Health paths | |---|---|---|---| | web | `git.noelsabia.dev/noelsabia/pulsecast-web:gate1` | 3000 | `/` | | api | `git.noelsabia.dev/noelsabia/pulsecast-api:gate1` | 8080 | `/api/healthz`, `/api/readyz` | | ml | `git.noelsabia.dev/noelsabia/pulsecast-ml:gate1` | 8000 | `/healthz`, `/readyz` | ```bash make build # builds all three from the service worktrees make push # docker push to Gitea ``` `make build` auto-detects the service worktrees under `/home/noel/Documents/Hackathon_2026/worktrees/`. Override when needed: ```bash make build WEB_DIR=/path/to/web-nuxt API_DIR=/path/to/api-go ML_DIR=/path/to/ML-py ``` If `OWNER` or `TAG` differs from the manifest defaults, either update `deploy/k8s/*.yaml` before `make deploy`, or deploy first and then run the guarded image update: ```bash make set-image CONFIRM=yes OWNER= TAG= ``` ### Dockerfile contract (coordinate through the orchestrator) Each service image must: - listen on `0.0.0.0` at the port above; - run as a non-root user (numeric UID, no shell required); - write only to `/tmp` (the pod root filesystem is read-only); - start with the environment in `deploy/k8s/00-config.yaml`; - be self-contained: no runtime download of data, model, fonts, or packages. `api` must read `DATA_DIR` (baked JSON), `FASTAPI_URL`, `PORT`, `CONTROL_USER`, `CONTROL_PASS`, `SSE_HEARTBEAT`, and `ML_TIMEOUT`. `web` must have `NUXT_PUBLIC_API_BASE=https://api.pulse.noelsabia.dev` baked in or provided at runtime; `ml` must find its model and artifacts under the paths its Dockerfile defines. The env names in the manifests are the integration point: change them here first if a service needs a different name. ## 5. Cloudflare Tunnel 1. Zero Trust → **Networks → Tunnels → Create a tunnel**. 2. Connector: **Cloudflared**. Name: `pulsecast-gate1`. 3. Copy the tunnel token (it is shown once). Store it in the `cloudflared` Secret (step 3). 4. Add public hostnames: | Public hostname | Service | |---|---| | `pulse.noelsabia.dev` | `http://web:3000` | | `api.pulse.noelsabia.dev` | `http://api:8080` | The hostnames must be in the Cloudflare zone for `noelsabia.dev`. The service URL uses the Kubernetes Service name and port inside the namespace; no Ingress is involved. 5. For the first smoke test before the app images exist, temporarily point `pulse.noelsabia.dev` at `http://hello:8080`, then switch it to `http://web:3000` after step 7. DNS records for the public hostnames are created by Cloudflare; do not add manual A/AAAA records for a tunnel hostname. ## 6. Validate before applying ```bash make validate ``` This is read-only: - `kubeconform -strict` validates the resource schemas if installed; - otherwise, if a cluster is reachable, `kubectl apply --dry-run=client` validates against the API server; - if neither is available it parses the YAML and prints a warning that schema validation was skipped. Never run `kubectl apply` / `delete` from an agent session. The commands below are for the human. ## 7. Deploy ```bash make deploy-check # read-only: required Secrets exist make deploy CONFIRM=yes # applies deploy/k8s and waits for rollouts ``` `make deploy` fails closed: without `CONFIRM=yes` it changes nothing. It does not create the namespace or secrets. If the three service images are not available yet, deploy only the tunnel and use the hello service to prove public HTTPS; return to this step when the real images exist. Do not deploy a placeholder image under the real service names. Check the result: ```bash kubectl -n "$NS" get deploy,pods,svc kubectl -n "$NS" get pods -o wide kubectl -n "$NS" logs deployment/cloudflared --tail=50 | grep 'Registered tunnel connection' ``` Expected: `web`, `api`, `ml`, `cloudflared` each `1/1`; Services `web`, `api`, `ml` are `ClusterIP`; `cloudflared` has no Service (it dials out). ## 8. Verify Public checks (also runnable from a phone on mobile data): ```bash curl -fsS -o /dev/null -w 'web %{http_code}\n' https://pulse.noelsabia.dev/ curl -fsS -o /dev/null -w 'api %{http_code}\n' https://api.pulse.noelsabia.dev/api/healthz curl -fsS -o /dev/null -w 'ready %{http_code}\n' https://api.pulse.noelsabia.dev/api/readyz curl -fsS https://api.pulse.noelsabia.dev/api/meta | jq '{data_version,model_version}' curl -sS -N --max-time 20 https://api.pulse.noelsabia.dev/api/stream | head -20 ``` The SSE output must start with `retry: 3000`, contain a `scenario` event immediately, and show a `ping` within 20 s. Full smoke test, including the ML service through a temporary `kubectl port-forward`: ```bash make smoke SMOKE_SSE=1 ``` Local variant once the three dev servers are running: ```bash make smoke-local ``` In-cluster connectivity (optional; creates and removes one short-lived pod): ```bash kubectl -n "$NS" run gate1-netcheck --rm -it --restart=Never \ --image=curlimages/curl:latest --command -- sh -c \ 'curl -fsS http://api:8080/api/healthz && \ curl -fsS http://ml:8000/healthz && \ curl -fsS http://web:3000/ >/dev/null && echo connectivity-ok' ``` TLS is verified by `make smoke` (`ssl_verify_result=0`). Certificates are issued by Cloudflare at the edge; there is no cert-manager resource. ## 9. Record the Gate 1 result Write down, in the status note or README: - deploy time and commit hash; - image tags actually deployed (`kubectl -n "$NS" get deploy -o jsonpath=...`); - output of `make smoke SMOKE_SSE=1`; - public URLs and the device/network used; - any warning that remains. ## Troubleshooting | Symptom | Check | Fix | |---|---|---| | `ImagePullBackOff` / `ErrImagePull` | `kubectl -n "$NS" describe pod `; `kubectl -n "$NS" get secret gitea-regcred` | Token is read-only package scoped; image tag exists; secret is in the same namespace. | | `CreateContainerConfigError` | `kubectl -n "$NS" describe pod ` | A referenced Secret key is missing (`control-auth` or `cloudflared`). Recreate the Secret. | | `cloudflared` CrashLoopBackOff | `kubectl -n "$NS" logs deployment/cloudflared --previous` | Invalid/expired tunnel token, or the tunnel was deleted in Cloudflare. Recreate the Secret and restart. | | Public 502 / 1033 | `kubectl -n "$NS" get endpoints web api`; cloudflared logs | Service has no ready endpoints, wrong port, or hostname route points at the wrong service. | | `api` pod not ready | `kubectl -n "$NS" logs deployment/api`; `kubectl -n "$NS" exec deploy/api -- ls /app/data` | Baked JSON missing from the image or `DATA_DIR` wrong. Rebuild the API image with the artifacts. | | `ml` pod stuck in startup | `kubectl -n "$NS" logs deployment/ml` | Model load exceeds 120 s or the model path is wrong. Fix the image/path; startupProbe allows 120 s. | | `web` 500 | `kubectl -n "$NS" logs deployment/web` | Missing build-time API base or server error. Rebuild with `NUXT_PUBLIC_API_BASE`. | | SSE events arrive only after a delay | `curl -N` direct to the pod vs through Cloudflare | Cloudflare buffering; the web client falls back to polling `/api/scenario` every 5 s. Keep the heartbeat at 15 s. | | `make deploy-check` fails | `kubectl -n "$NS" get secrets` | Create the missing Secret; do not commit it. | ## Rollback No cluster-scoped resources are created, so rollback is namespace-local: ```bash kubectl -n "$NS" rollout undo deployment/web kubectl -n "$NS" rollout undo deployment/api kubectl -n "$NS" rollout undo deployment/ml kubectl -n "$NS" rollout undo deployment/cloudflared kubectl -n "$NS" rollout status deployment/web --timeout=180s ``` If the tunnel token or route is wrong, fix the Secret / Cloudflare dashboard and restart `cloudflared`; do not create an Ingress as a workaround. ## Security notes - Never commit the kubeconfig, tunnel token, control password, Gitea token, or any `.env` file. `.gitignore` covers the usual filenames; check `git status` before every commit. - Rotate the Gitea package token and the control password after the event. Rotate the service-account token after the event; it was shared in chat. - The kubeconfig must stay at mode `600` and outside the repos. - `make smoke` prints no secrets; do not add `set -x` to the scripts with a live `KUBECONFIG`.