Files
infra-k8/docs/deploy.md
T

13 KiB

Deploy runbook — Gate 1

Target: three services (web, api, ml) plus one cloudflared Deployment in namespace fch26-onemanarmy, reachable over public HTTPS, with Gitea image pulls working.

The human runs every cluster-mutating command in this document. The infra agent only writes and validates manifests. No Ingress, cert-manager, external-dns, Jobs, CronJobs, CRDs, Roles, RoleBindings, NetworkPolicies, or other cluster-scoped resources are used.

Gate 1 checklist

Run top to bottom. Stop at the first failure and use Troubleshooting.

  • 1. Kubeconfig exported, 600, correct context, namespace reachable.
  • 2. Gitea read-only package token created and docker login succeeds.
  • 3. Hello-world image pushed and pulled by the cluster with gitea-regcred.
  • 4. Secrets created: gitea-regcred, cloudflared, control-auth.
  • 5. Real web/api/ml images built, tagged gate1, pushed to Gitea.
  • 6. make validate passes locally.
  • 7. Cloudflare Tunnel exists; both public hostnames configured.
  • 8. make deploy CONFIRM=yes completes and all four Deployments are ready.
  • 9. make smoke SMOKE_SSE=1 passes.
  • 10. https://pulse.noelsabia.dev opens on a phone over mobile data.
  • 11. https://api.pulse.noelsabia.dev/api/healthz returns 200.
  • 12. Record the deployed image tags and the time in the status note.

0. Prerequisites

The kubeconfig is not in this repo. It lives outside git:

export KUBECONFIG="/home/noel/Documents/Hackathon_2026/OKF_Vault/fch26-onemanarmy-sa-kubeconfig.yaml"
export NS=fch26-onemanarmy
export CTX=fch26-onemanarmy-sa-context
export REGISTRY=git.noelsabia.dev
export OWNER=noelsabia
export TAG=gate1

chmod 600 "$KUBECONFIG"

Verify read-only first:

kubectl config current-context                     # fch26-onemanarmy-sa-context
kubectl -n "$NS" auth can-i create deployments     # yes
kubectl -n "$NS" auth can-i create services        # yes
kubectl -n "$NS" auth can-i create ingresses       # no
kubectl -n "$NS" auth can-i create jobs            # no
kubectl -n "$NS" get pods,svc,secrets

If any of the first three answers differ, stop and report; do not work around permissions with cluster-scoped resources.

Local tools: git, make, bash, curl, docker, kubectl. Optional: kubeconform (schema validation), jq, k6 built with the SSE extension.

1. Gitea read-only package token

In Gitea: Settings → Applications → Generate New Token (or the package token screen). Scope: read-only package access. Do not reuse the admin token.

read -rsp 'Gitea package token: ' GITEA_TOKEN; echo
printf '%s' "$GITEA_TOKEN" | docker login "$REGISTRY" -u "$OWNER" --password-stdin

A successful login prints Login Succeeded. Never put the token in a file, a command that lands in shell history, or this repo.

2. Hello-world image pull (prove the registry path)

docker pull nginxinc/nginx-unprivileged:alpine
docker tag nginxinc/nginx-unprivileged:alpine "$REGISTRY/$OWNER/hello:gate1"
docker push "$REGISTRY/$OWNER/hello:gate1"

kubectl -n "$NS" create secret docker-registry gitea-regcred \
  --docker-server="$REGISTRY" \
  --docker-username="$OWNER" \
  --docker-password="$GITEA_TOKEN"

kubectl -n "$NS" create deployment hello \
  --image="$REGISTRY/$OWNER/hello:gate1"
kubectl -n "$NS" patch deployment hello --type=merge -p \
  '{"spec":{"template":{"spec":{"imagePullSecrets":[{"name":"gitea-regcred"}]}}}}'
kubectl -n "$NS" rollout status deployment/hello --timeout=180s
kubectl -n "$NS" get pods -l app=hello

Pass: the pod is Running and kubectl describe pod shows the image pulled. Clean up before the real deploy:

kubectl -n "$NS" delete deployment hello

If this fails, fix the token/secret before building the application images.

3. Create the remaining Secrets

Generate values locally; never commit them.

read -rsp 'Control password: ' CONTROL_PASS; echo
kubectl -n "$NS" create secret generic control-auth \
  --from-literal=CONTROL_USER=control \
  --from-literal=CONTROL_PASS="$CONTROL_PASS"

# Token from Cloudflare Zero Trust → Networks → Tunnels → your tunnel.
read -rsp 'Cloudflare tunnel token: ' TUNNEL_TOKEN; echo
kubectl -n "$NS" create secret generic cloudflared \
  --from-literal=TUNNEL_TOKEN="$TUNNEL_TOKEN"

kubectl -n "$NS" get secrets

make secret-commands prints the same commands with placeholders.

4. Build and push the three images

The manifests expect:

Service Image Port Health paths
web git.noelsabia.dev/noelsabia/pulsecast-web:gate1 3000 /
api git.noelsabia.dev/noelsabia/pulsecast-api:gate1 8080 /api/healthz, /api/readyz
ml git.noelsabia.dev/noelsabia/pulsecast-ml:gate1 8000 /healthz, /readyz
make build          # builds all three from the service worktrees
make push           # docker push to Gitea

make build auto-detects the service worktrees under /home/noel/Documents/Hackathon_2026/worktrees/. Override when needed:

make build WEB_DIR=/path/to/web-nuxt API_DIR=/path/to/api-go ML_DIR=/path/to/ML-py

If OWNER or TAG differs from the manifest defaults, either update deploy/k8s/*.yaml before make deploy, or deploy first and then run the guarded image update:

make set-image CONFIRM=yes OWNER=<owner> TAG=<tag>

Dockerfile contract (coordinate through the orchestrator)

Each service image must:

  • listen on 0.0.0.0 at the port above;
  • run as a non-root user (numeric UID, no shell required);
  • write only to /tmp (the pod root filesystem is read-only);
  • start with the environment in deploy/k8s/00-config.yaml;
  • be self-contained: no runtime download of data, model, fonts, or packages.

api must read DATA_DIR (baked JSON), FASTAPI_URL, PORT, CONTROL_USER, CONTROL_PASS, SSE_HEARTBEAT, and ML_TIMEOUT. web must have NUXT_PUBLIC_API_BASE=https://api.pulse.noelsabia.dev baked in or provided at runtime; ml must find its model and artifacts under the paths its Dockerfile defines. The env names in the manifests are the integration point: change them here first if a service needs a different name.

5. Cloudflare Tunnel

  1. Zero Trust → Networks → Tunnels → Create a tunnel.

  2. Connector: Cloudflared. Name: pulsecast-gate1.

  3. Copy the tunnel token (it is shown once). Store it in the cloudflared Secret (step 3).

  4. Add public hostnames:

    Public hostname Service
    pulse.noelsabia.dev http://web:3000
    api.pulse.noelsabia.dev http://api:8080

    The hostnames must be in the Cloudflare zone for noelsabia.dev. The service URL uses the Kubernetes Service name and port inside the namespace; no Ingress is involved.

  5. For the first smoke test before the app images exist, temporarily point pulse.noelsabia.dev at http://hello:8080, then switch it to http://web:3000 after step 7.

DNS records for the public hostnames are created by Cloudflare; do not add manual A/AAAA records for a tunnel hostname.

6. Validate before applying

make validate

This is read-only:

  • kubeconform -strict validates the resource schemas if installed;
  • otherwise, if a cluster is reachable, kubectl apply --dry-run=client validates against the API server;
  • if neither is available it parses the YAML and prints a warning that schema validation was skipped.

Never run kubectl apply / delete from an agent session. The commands below are for the human.

7. Deploy

make deploy-check                 # read-only: required Secrets exist
make deploy CONFIRM=yes           # applies deploy/k8s and waits for rollouts

make deploy fails closed: without CONFIRM=yes it changes nothing. It does not create the namespace or secrets.

If the three service images are not available yet, deploy only the tunnel and use the hello service to prove public HTTPS; return to this step when the real images exist. Do not deploy a placeholder image under the real service names.

Check the result:

kubectl -n "$NS" get deploy,pods,svc
kubectl -n "$NS" get pods -o wide
kubectl -n "$NS" logs deployment/cloudflared --tail=50 | grep 'Registered tunnel connection'

Expected: web, api, ml, cloudflared each 1/1; Services web, api, ml are ClusterIP; cloudflared has no Service (it dials out).

8. Verify

Public checks (also runnable from a phone on mobile data):

curl -fsS -o /dev/null -w 'web %{http_code}\n'        https://pulse.noelsabia.dev/
curl -fsS -o /dev/null -w 'api %{http_code}\n'        https://api.pulse.noelsabia.dev/api/healthz
curl -fsS -o /dev/null -w 'ready %{http_code}\n'      https://api.pulse.noelsabia.dev/api/readyz
curl -fsS https://api.pulse.noelsabia.dev/api/meta | jq '{data_version,model_version}'
curl -sS -N --max-time 20 https://api.pulse.noelsabia.dev/api/stream | head -20

The SSE output must start with retry: 3000, contain a scenario event immediately, and show a ping within 20 s.

Full smoke test, including the ML service through a temporary kubectl port-forward:

make smoke SMOKE_SSE=1

Local variant once the three dev servers are running:

make smoke-local

In-cluster connectivity (optional; creates and removes one short-lived pod):

kubectl -n "$NS" run gate1-netcheck --rm -it --restart=Never \
  --image=curlimages/curl:latest --command -- sh -c \
  'curl -fsS http://api:8080/api/healthz && \
   curl -fsS http://ml:8000/healthz && \
   curl -fsS http://web:3000/ >/dev/null && echo connectivity-ok'

TLS is verified by make smoke (ssl_verify_result=0). Certificates are issued by Cloudflare at the edge; there is no cert-manager resource.

9. Record the Gate 1 result

Write down, in the status note or README:

  • deploy time and commit hash;
  • image tags actually deployed (kubectl -n "$NS" get deploy -o jsonpath=...);
  • output of make smoke SMOKE_SSE=1;
  • public URLs and the device/network used;
  • any warning that remains.

Troubleshooting

Symptom Check Fix
ImagePullBackOff / ErrImagePull kubectl -n "$NS" describe pod <pod>; kubectl -n "$NS" get secret gitea-regcred Token is read-only package scoped; image tag exists; secret is in the same namespace.
CreateContainerConfigError kubectl -n "$NS" describe pod <pod> A referenced Secret key is missing (control-auth or cloudflared). Recreate the Secret.
cloudflared CrashLoopBackOff kubectl -n "$NS" logs deployment/cloudflared --previous Invalid/expired tunnel token, or the tunnel was deleted in Cloudflare. Recreate the Secret and restart.
Public 502 / 1033 kubectl -n "$NS" get endpoints web api; cloudflared logs Service has no ready endpoints, wrong port, or hostname route points at the wrong service.
api pod not ready kubectl -n "$NS" logs deployment/api; kubectl -n "$NS" exec deploy/api -- ls /app/data Baked JSON missing from the image or DATA_DIR wrong. Rebuild the API image with the artifacts.
ml pod stuck in startup kubectl -n "$NS" logs deployment/ml Model load exceeds 120 s or the model path is wrong. Fix the image/path; startupProbe allows 120 s.
web 500 kubectl -n "$NS" logs deployment/web Missing build-time API base or server error. Rebuild with NUXT_PUBLIC_API_BASE.
SSE events arrive only after a delay curl -N direct to the pod vs through Cloudflare Cloudflare buffering; the web client falls back to polling /api/scenario every 5 s. Keep the heartbeat at 15 s.
make deploy-check fails kubectl -n "$NS" get secrets Create the missing Secret; do not commit it.

Rollback

No cluster-scoped resources are created, so rollback is namespace-local:

kubectl -n "$NS" rollout undo deployment/web
kubectl -n "$NS" rollout undo deployment/api
kubectl -n "$NS" rollout undo deployment/ml
kubectl -n "$NS" rollout undo deployment/cloudflared
kubectl -n "$NS" rollout status deployment/web --timeout=180s

If the tunnel token or route is wrong, fix the Secret / Cloudflare dashboard and restart cloudflared; do not create an Ingress as a workaround.

Security notes

  • Never commit the kubeconfig, tunnel token, control password, Gitea token, or any .env file. .gitignore covers the usual filenames; check git status before every commit.
  • Rotate the Gitea package token and the control password after the event. Rotate the service-account token after the event; it was shared in chat.
  • The kubeconfig must stay at mode 600 and outside the repos.
  • make smoke prints no secrets; do not add set -x to the scripts with a live KUBECONFIG.