Files
infra-k8/docs/deploy.md
T
Noel Sabia e8e048866c
infra-ci / manifests (push) Successful in 8s
infra-ci / manifests (pull_request) Successful in 8s
ci: run manifest checks on the Gitea runner
The node:20-bookworm runner has python3 but no pip, so the PyYAML install failed before any check ran. Install python3-yaml with apt instead, remove the duplicate cluster-only kubectl step, and keep the two cluster-free Make targets.

Also remove the stale v1.0/pending-review claims, drop the old api.pulse hostname, and move the deploy status below the runbook.
2026-11-14 09:13:27 +01:00

31 KiB
Raw Blame History

Deploy runbook — Gate 1

Target: three services (web, api, ml) plus one cloudflared Deployment in namespace fch26-onemanarmy, reachable over public HTTPS, with Gitea image pulls proven before the real images are deployed.

The human runs every cluster-mutating command in this document. The infra agent only writes and validates manifests. No Ingress, cert-manager, external-dns, Jobs, CronJobs, CRDs, Roles, RoleBindings, NetworkPolicies, or other cluster-scoped resources are used.

Run it

From this repo root, with KUBECONFIG exported (see Before you start):

make validate
make gate1-check
make build
make push
make deploy CONFIRM=yes
make smoke SMOKE_SSE=1

The first deploy also needs the Gitea pull proof, Tunnel, and Secrets in Steps 1–6. Every cluster mutation is human-run.

What Gate 1 means

Gate 1 passes only when all of these are true:

  1. The cluster pulls an image from Gitea in fch26-onemanarmy using gitea-regcred (proven with a hello-world before the real images).
  2. A Cloudflare Tunnel serves a public HTTPS URL from this namespace.
  3. web, api, ml, and cloudflared are Ready.
  4. make smoke SMOKE_SSE=1 passes and a phone on mobile data can open https://pulse.noelsabia.dev.
  5. https://api-pulse.noelsabia.dev/api/healthz returns HTTP 200.

Do not claim Gate 1 from a local kubectl port-forward, a green manifest dry-run, or a container that only runs on your laptop.

Before you start

Run every block in this document in one shell, from the repo root. The commands are bash/zsh compatible.

cd /path/to/infra-k8

export KUBECONFIG="/home/noel/Documents/Hackathon_2026/OKF_Vault/fch26-onemanarmy-sa-kubeconfig.yaml"
export NS="fch26-onemanarmy"
export CTX="fch26-onemanarmy-sa-context"
export REGISTRY="git.noelsabia.dev"
export OWNER="noel"
export TAG="v1.1"

chmod 600 "$KUBECONFIG"

The kubeconfig is not in this repo. If it moved, change the path; never commit it or paste its contents anywhere. The Gitea owner for this workspace is noel; if your account differs, set OWNER and the image paths consistently.

Check the local tools:

for tool in kubectl docker curl jq make; do
  command -v "$tool" >/dev/null || echo "MISSING: $tool"
done

Expected: no MISSING lines. kubeconform, k6, and yamllint are optional.

Check the context and the namespace permissions (read-only):

kubectl config current-context
kubectl --context "$CTX" -n "$NS" auth can-i create deployments
kubectl --context "$CTX" -n "$NS" auth can-i create services
kubectl --context "$CTX" -n "$NS" auth can-i create ingresses
kubectl --context "$CTX" -n "$NS" auth can-i create jobs
kubectl --context "$CTX" -n "$NS" get deployments

Expected:

fch26-onemanarmy-sa-context
yes
yes
no
no
No resources found in fch26-onemanarmy namespace.

If deployments or services is not yes, or ingresses/jobs is not no, stop and report. Do not work around permissions with cluster-scoped resources.

Gate 1 checklist

Run top to bottom. Stop at the first failure and use Troubleshooting.

  • 0. make gate1-check shows 0 hard failures.
  • 1. Gitea push login succeeds (Login Succeeded).
  • 2. Read-only package token created; hello image pushed to Gitea.
  • 3. gitea-regcred created; hello pod is 1/1 Running from Gitea (the Secret existing is not proof).
  • 4. Cloudflare Tunnel created; both public hostnames configured and DNS records resolve.
  • 5. cloudflared Secret created; logs show Registered tunnel connection.
  • 6. Public hello route returns HTTP 200 (proves tunnel + DNS + TLS).
  • 7. control-auth Secret created.
  • 8. All three v1.1 images built and pushed.
  • 9. make validate and make gate1-check pass.
  • 10. make deploy CONFIRM=yes completes; all four Deployments are Ready.
  • 11. make smoke SMOKE_SSE=1 passes.
  • 12. Phone over mobile data opens https://pulse.noelsabia.dev; https://api-pulse.noelsabia.dev/api/healthz returns 200.
  • 13. Image tags/digests and the time are recorded.

Step 0 — read-only readiness and dry-run

make validate
make gate1-check

Expected on a fresh cluster (before any human work):

  • With a reachable cluster, make validate prints == kubectl client dry-run (read-only) and 11 created (dry run) lines. Without one it prints 11 manifests parsed.
  • make gate1-check reports 0 hard failures, the expected permission matrix, and lists missing Secrets/Deployments as [pending].
  • Exit code 2 from make gate1-check is correct at this stage: it means only human prerequisites remain, not that the manifests are broken.

Exit codes: 0 ready, 1 hard failure (permissions or manifests), 2 only pending human prerequisites. Use make gate1-check ALLOW_PENDING=1 when you want exit 0 while prerequisites are still being prepared.

Neither target mutates the cluster. make validate uses kubectl apply --dry-run=client; make gate1-check only reads configuration/RBAC/secrets and runs the same dry-run. CI runs the same static guardrails without a cluster:

make gate1-check STATIC_ONLY=1
make validate

Step 1 — Gitea credentials

You need two different credentials. Do not reuse the read-only token for pushing.

  1. Push credential: your normal Gitea account password, or a token with write:package. Use a token if the account has 2FA.
  2. Cluster pull credential: a separate token with only read:package. This becomes gitea-regcred; it never needs write access.

Create the read-only token in Gitea: Settings → Applications → Generate New Token (or the package-token screen). Name it pulsecast-k8s-pull; scope read:package. Copy it once.

Log in for pushing:

read -rsp 'Gitea push password or write:package token: ' GITEA_PUSH_SECRET; echo
printf '%s' "$GITEA_PUSH_SECRET" | docker login "$REGISTRY" -u "$OWNER" --password-stdin

Expected:

Login Succeeded

Store the read-only token in a shell variable for the Secret step:

read -rsp 'Gitea read-only package token: ' GITEA_RO_TOKEN; echo

Expected: no output (the value is not echoed).

If login fails with 401 Unauthorized, the password/token or username is wrong, or 2FA requires a token. Do not put either credential in a file, a command that lands in shell history, or this repo.

Step 2 — hello-world image push and pull

This proves the registry path before spending time on application images.

docker pull nginxinc/nginx-unprivileged:alpine
docker tag nginxinc/nginx-unprivileged:alpine "$REGISTRY/$OWNER/hello:gate1"
docker push "$REGISTRY/$OWNER/hello:gate1"

Expected: the push ends with a digest line, for example:

gate1: digest: sha256:... size: ...

If you see denied: requested access to the resource is denied, the push credential lacks write:package or the owner/package path is wrong. Fix that before continuing.

Create the pull secret with the read-only token:

kubectl --context "$CTX" -n "$NS" create secret docker-registry gitea-regcred \
  --docker-server="$REGISTRY" \
  --docker-username="$OWNER" \
  --docker-password="$GITEA_RO_TOKEN"

Expected: secret/gitea-regcred created. If it already exists, delete and recreate it after reading the troubleshooting note: kubectl --context "$CTX" -n "$NS" delete secret gitea-regcred.

Deploy the hello image and make the cluster pull it:

kubectl --context "$CTX" -n "$NS" create deployment hello \
  --image="$REGISTRY/$OWNER/hello:gate1"

kubectl --context "$CTX" -n "$NS" patch deployment hello --type=merge -p \
  '{"spec":{"template":{"spec":{"imagePullSecrets":[{"name":"gitea-regcred"}]}}}}'

kubectl --context "$CTX" -n "$NS" rollout status deployment/hello --timeout=180s
kubectl --context "$CTX" -n "$NS" get pods -l app=hello -o wide

Expected:

deployment "hello" successfully rolled out
NAME                     READY   STATUS    RESTARTS   AGE
hello-xxxxxxxxxx-xxxxx   1/1     Running   0          20s

This is the first real pass/fail. A gitea-regcred Secret existing is not proof: the Gate 1 Gitea item is complete only when this hello pod is 1/1 Running after pulling from Gitea. If the pod is not Running, stop. ImagePullBackOff means the token, secret namespace, image tag, or registry DNS is wrong. Diagnose with:

kubectl --context "$CTX" -n "$NS" describe pod -l app=hello

Keep the hello Deployment until Step 6 proves the public route. Create a Service for it so Cloudflare has something to reach:

kubectl --context "$CTX" -n "$NS" expose deployment hello \
  --port=8080 --target-port=8080

Expected: service/hello exposed.

Step 3 — Cloudflare Tunnel

The tunnel used here is pulsecast-gate1. pulse.noelsabia.dev resolves; api-pulse.noelsabia.dev resolves after its route is saved. Universal SSL covers only *.noelsabia.dev, so the API host must stay one label under the zone: api-pulse.noelsabia.dev.

getent hosts pulse.noelsabia.dev
getent hosts api-pulse.noelsabia.dev || echo 'api-pulse: no DNS record yet (expected until the route exists)'

If DNS looks missing locally but public DNS has it, flush the local cache:

resolvectl flush-caches

If getent still misses the name while dig +short NAME @1.1.1.1 works, force a fresh upstream query and flush again:

resolvectl query --cache=no api-pulse.noelsabia.dev
resolvectl flush-caches
getent hosts api-pulse.noelsabia.dev
  1. Open Cloudflare Zero Trust → Networks → Tunnels → Create a tunnel.

  2. Connector: Cloudflared. Name: pulsecast-gate1.

  3. Copy the tunnel token from the connector install command. It is the long eyJ... string after --token (or after service install). It is sensitive: never paste it into chat, a commit, or a screenshot. The tunnel stays Inactive with 0 routes until the connector Deployment is running; routes can be added while it is inactive.

  4. Open the tunnel and click the Routes tab (not Overview, and not Add a replica). Click Add route or Add a public hostname, choose Public hostname, and add:

    Subdomain Domain Path Service type Service URL
    pulse noelsabia.dev (empty) HTTP hello:8080
    api-pulse noelsabia.dev (empty) HTTP api:8080

    The first route is temporary for Gate 1-A. For the final Gate 1-B route, change it to web:3000 after the web image is deployed. If you already have the web image, you may set web:3000 now and skip the hello route; then the public hello check in Step 6 is not available, but the tunnel proof still happens when the real web pod is Ready.

  5. The hostnames must be in the Cloudflare zone for noelsabia.dev. Cloudflare creates the DNS records for a tunnel hostname; do not add manual A/AAAA records. If the dashboard refuses to save a hostname because a conflicting A/AAAA/CNAME already exists for that exact name, delete the conflicting record first.

  6. Do not add an Ingress. The tunnel is the only public path.

After saving both hostnames, DNS should resolve to Cloudflare:

getent hosts pulse.noelsabia.dev
getent hosts api-pulse.noelsabia.dev

Expected: one or more Cloudflare IPv4/IPv6 addresses. If there is still no record, the public hostname was not saved, it is in the wrong zone, or a conflicting record exists. Fix that in the dashboard before testing HTTPS; curl cannot succeed without DNS.

Create the tunnel Secret (the token is not echoed):

read -rsp 'Cloudflare tunnel token: ' TUNNEL_TOKEN; echo
kubectl --context "$CTX" -n "$NS" create secret generic cloudflared \
  --from-literal=TUNNEL_TOKEN="$TUNNEL_TOKEN"

Expected: secret/cloudflared created.

Deploy only the tunnel first. This does not need the application images:

kubectl --context "$CTX" -n "$NS" apply -f deploy/k8s/40-cloudflared.yaml
kubectl --context "$CTX" -n "$NS" rollout status deployment/cloudflared --timeout=180s
kubectl --context "$CTX" -n "$NS" logs deployment/cloudflared --tail=100 \
  | grep 'Registered tunnel connection' || echo 'NO TUNNEL CONNECTION'

Expected:

deployment.apps/cloudflared created
deployment "cloudflared" successfully rolled out
... Registered tunnel connection ...

(Usually four Registered tunnel connection lines, one per edge.)

The manifest deliberately uses --protocol http2: TCP/443 to Cloudflare is verified, UDP/7844 (QUIC) is not. Do not switch to QUIC during Gate 1. If you do test QUIC later, revert to HTTP/2 on any connection error.

If cloudflared is in CrashLoopBackOff, the token is invalid/expired or the tunnel was deleted:

kubectl --context "$CTX" -n "$NS" logs deployment/cloudflared --previous
kubectl --context "$CTX" -n "$NS" describe pod -l app.kubernetes.io/name=cloudflared

Step 4 — control-auth Secret

The web server validates /control and POST /api/scenario with CONTROL_USER/CONTROL_PASS; web, API, and ML also need the same CONTROL_TOKEN (the API forwards it as a Bearer token to FastAPI's internal /train routes). Generate both locally; never commit them.

read -rsp 'Control password: ' CONTROL_PASS; echo
CONTROL_TOKEN="$(python3 -c 'import secrets; print(secrets.token_hex(24))')"
kubectl --context "$CTX" -n "$NS" create secret generic control-auth \
  --from-literal=CONTROL_USER=control \
  --from-literal=CONTROL_PASS="$CONTROL_PASS" \
  --from-literal=CONTROL_TOKEN="$CONTROL_TOKEN"

Expected: secret/control-auth created.

The token is shared by API and ML, not the browser password. Without it FastAPI /train returns 503 and the API's /api/train proxy cannot work.

make secret-commands prints all three Secret commands with placeholders; it creates nothing.

Step 5 — build and push the three images (Phase B)

Check whether the Dockerfiles exist:

test -f ../../web-nuxt.git/main/Dockerfile && echo web-ok || echo web-blocked
test -f ../../api-go.git/main/Dockerfile && echo api-ok || echo api-blocked
test -f ../../ML-py.git/main/Dockerfile && echo ml-ok || echo ml-blocked

Expected:

web-ok
api-ok
ml-ok

Build all three:

make build

All three v1.1 images were verified locally with the manifest's runtime constraints:

  • API: distroless static:nonroot, --user 65532:65532, read-only root, /tmp tmpfs; /api/healthz and /api/readyz returned 200, and the API loaded ML /bootstrap into memory.
  • ML: image USER 10001:10001; with read-only root and /tmp tmpfs, /healthz and /readyz returned 200, /bootstrap returned 8 sensors, and /meta showed mae=3.32, wape=27.5, r2=0.777.
  • Web: image USER 10001:10001; with read-only root and /tmp tmpfs, /, /stage, and /live returned 200; /control returned 401 anonymous and 200 with control auth; the same-origin SSE proxy emitted events.
  • v1.1 is pushed to Gitea and deployed in the cluster; re-apply only after a reviewed image change.

Push:

make push

Expected: each docker push ends with a digest. If authentication expired, repeat the push login from Step 1.

The Makefile auto-detects the service worktrees:

../../web-nuxt.git/main
../../api-go.git/main
../../ML-py.git/main

Override when the layout differs:

make build WEB_DIR=/path/to/web-nuxt API_DIR=/path/to/api-go ML_DIR=/path/to/ML-py

If OWNER or TAG differs from the manifests, update deploy/k8s/*.yaml before make deploy, or deploy first and then run the guarded image update:

make set-image CONFIRM=yes OWNER=<owner> TAG=<tag>

Dockerfile contract (coordinate through the orchestrator)

Each service image must:

  • listen on 0.0.0.0 at the port below;
  • run as a non-root numeric UID (no named user, no shell required);
  • write only to /tmp (the pod root filesystem is read-only);
  • start with the environment in deploy/k8s/00-config.yaml;
  • be self-contained: no runtime download of data, model, fonts, or packages.
Service Image Port UID:GID Health paths
web git.noelsabia.dev/noel/pulsecast-web:v1.1 3000 10001:10001 /, /stage, /live, /control
api git.noelsabia.dev/noel/pulsecast-api:v1.1 8080 65532:65532 /api/healthz, /api/readyz
ml git.noelsabia.dev/noel/pulsecast-ml:v1.1 8000 10001:10001 /healthz, /readyz, /bootstrap

The manifests set runAsUser/runAsGroup/fsGroup explicitly, so the image's named user is irrelevant; the files must be readable by that numeric UID. api uses distroless static:nonroot (65532); web and ml use 10001. If a Dockerfile genuinely needs another UID, change the manifest and this table together, and keep the value numeric.

api v1.1 reads FASTAPI_URL=http://ml:8000, BOOTSTRAP_REFRESH=60s, BOOTSTRAP_TIMEOUT, PORT, CONTROL_USER, CONTROL_PASS, CONTROL_TOKEN, SSE_HEARTBEAT, and ML_TIMEOUT. It has no baked data and no DATA_DIR; it fetches ML /bootstrap into memory and refreshes it. web proxies Go and SSE server-side using NUXT_API_INTERNAL_BASE=http://api:8080; it also needs CONTROL_USER, CONTROL_PASS, and CONTROL_TOKEN to protect /control and POST /api/scenario. NUXT_PUBLIC_API_BASE and the public SSE URL are display/future configuration; the browser calls the Nuxt origin. ml reads PULSECAST_ROOT, PULSECAST_ARTIFACTS, PULSECAST_WEATHER, PULSECAST_MODELS, PULSECAST_UPLOADS, and CONTROL_TOKEN; its image CMD binds 0.0.0.0:8000. The env names in the manifests are the integration point: change them here first if a service needs a different name. All three images use numeric UIDs: web/ml 10001:10001, API 65532:65532.

Step 6 — prove public HTTPS with hello

Do this before deleting hello. It proves tunnel + DNS + TLS without depending on web/ml.

curl -fsS -o /dev/null -w 'hello %{http_code}\n' https://pulse.noelsabia.dev/

Expected: hello 200.

If you get 502, 530, or 1033, the edge cannot reach the Service. Check:

kubectl --context "$CTX" -n "$NS" get endpoints hello
kubectl --context "$CTX" -n "$NS" get svc hello
kubectl --context "$CTX" -n "$NS" logs deployment/cloudflared --tail=50

The hostname route must be http://hello:8080 and the hello Service must have an endpoint. Fix the dashboard route or the Service; do not add an Ingress.

After the public hello check passes, delete it so it cannot mask a web problem:

kubectl --context "$CTX" -n "$NS" delete service hello
kubectl --context "$CTX" -n "$NS" delete deployment hello

Expected: service "hello" deleted and deployment.apps "hello" deleted.

https://pulse.noelsabia.dev will now return an edge error until the real web Deployment is Ready. That is expected; do not "fix" it with an Ingress.

Step 7 — validate and check readiness again

make validate
make gate1-check

Expected once Secrets and Deployments exist:

  • make validate: 11 resources created (dry run);
  • make gate1-check: 0 hard failures, 0 pending, RESULT: READY for Gate 1 (exit 0).

Before the deploy, pending Deployments keep it at exit 2. That is fine.

Step 8 — deploy

make deploy-check
make deploy CONFIRM=yes

Expected:

required Secrets and keys present
... created/configured ...
deployment "ml" successfully rolled out
deployment "api" successfully rolled out
deployment "web" successfully rolled out
deployment "cloudflared" successfully rolled out

make deploy fails closed: without CONFIRM=yes it changes nothing. It does not create the namespace or Secrets. It applies all 11 manifests, including the ConfigMaps.

If the real images do not exist, the rollouts fail with ImagePullBackOff. Stop and finish Step 5; do not deploy a placeholder under web, api, or ml.

Check the state:

make status
kubectl --context "$CTX" -n "$NS" get pods -o wide
kubectl --context "$CTX" -n "$NS" logs deployment/cloudflared --tail=50 \
  | grep 'Registered tunnel connection' || echo 'NO TUNNEL CONNECTION'

Expected: web, api, ml, and cloudflared each 1/1; Services web, api, and ml are ClusterIP; cloudflared has no Service.

Re-pushing the same tag

The manifests use imagePullPolicy: Always while the tag stays v1.1. After rebuilding and pushing the same tag, force a re-pull:

make restart CONFIRM=yes

This restarts web, api, and ml and waits for the rollout. Do not use it during the demo window unless you are fixing a blocker.

Step 9 — verify public HTTPS and SSE

Full smoke, including the ML service through a short kubectl port-forward:

make smoke SMOKE_SSE=1

Expected: each check prints PASS, the summary ends with 0 failed, and TLS verification passes (ssl_verify_result=0). If the ML port-forward cannot start, set ML_URL or SKIP_ML=1; do not silently drop the ML check from the sign-off.

Manual public checks (also run from a phone on mobile data):

curl -fsS -o /dev/null -w 'web %{http_code}\n'     https://pulse.noelsabia.dev/
curl -fsS -o /dev/null -w 'api %{http_code}\n'     https://api-pulse.noelsabia.dev/api/healthz
curl -fsS -o /dev/null -w 'ready %{http_code}\n'   https://api-pulse.noelsabia.dev/api/readyz
curl -fsS https://api-pulse.noelsabia.dev/api/meta | jq '{data_version,model_version}'
curl -sS -N --max-time 20 https://api-pulse.noelsabia.dev/api/stream | head -20

Expected:

web 200
api 200
ready 200
{
  "data_version": "...",
  "model_version": "..."
}
retry: 3000

event: scenario
id: ...
data: {...}

event: ping
data: {"t":"..."}

The SSE output must start with retry: 3000, contain a scenario event immediately, and show a ping within 20 seconds. If events are delayed through Cloudflare, see the SSE row in Troubleshooting.

v1.1 product checks

curl -fsS https://api-pulse.noelsabia.dev/api/sensors | jq 'length'
curl -fsS https://api-pulse.noelsabia.dev/api/meta \
  | jq '{data_version,model_version,metrics,cache}'
curl -fsS -o /dev/null -w 'stage %{http_code}\n'   https://pulse.noelsabia.dev/stage
curl -fsS -o /dev/null -w 'live %{http_code}\n'    https://pulse.noelsabia.dev/live
curl -sS  -o /dev/null -w 'control anon %{http_code}\n' https://pulse.noelsabia.dev/control
set -a; source ~/.config/pulsecast/control-auth.env; set +a
curl -sS -u "$CONTROL_USER:$CONTROL_PASS" -o /dev/null \
  -w 'control auth %{http_code}\n' https://pulse.noelsabia.dev/control
curl -sS -o /dev/null -w 'plan %{time_total}s\n' \
  'https://api-pulse.noelsabia.dev/api/staffing?sensor=hn-stadtgalerie-01&scenario=base&standard=standard'

Expected: 8 sensors; metrics mae=3.32, wape=27.5, r2=0.777 with a cache object; /stage and /live 200; /control 401 anonymous and 200 with control auth; plan under 2 seconds.

In-cluster connectivity (optional; creates and removes one short-lived pod, so the human runs it):

kubectl --context "$CTX" -n "$NS" run gate1-netcheck --rm -i --restart=Never \
  --image=curlimages/curl:latest \
  --overrides='{"spec":{"securityContext":{"runAsNonRoot":true,"runAsUser":1000,"runAsGroup":1000}}}' \
  --command -- sh -c \
  'curl -fsS http://api:8080/api/healthz && \
   curl -fsS http://ml:8000/healthz && \
   curl -fsS http://web:3000/ >/dev/null && echo connectivity-ok'

Expected: {"ok":true}{"ok":true}connectivity-ok (order may vary with the shell output). If the namespace's Pod Security Admission blocks this ad-hoc pod, skip it and use kubectl exec deploy/web -- ... or the smoke test; do not weaken the application Deployments to make a diagnostic pod run.

Finally, confirm the checker is green end-to-end:

make gate1-check CHECK_PUBLIC=1

Expected: RESULT: READY for Gate 1 (exit 0).

Step 10 — record the Gate 1 result

Record in the status note, README, or commit message:

date -u +'%Y-%m-%dT%H:%M:%SZ'
git rev-parse HEAD
kubectl --context "$CTX" -n "$NS" get deploy -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.template.spec.containers[0].image}{"\n"}{end}'

Also save:

  • the output of make smoke SMOKE_SSE=1;
  • the public URLs and the device/network used (phone on mobile data);
  • any warning that remains.

Image digests are in the docker push output; tags alone are not enough if you need to reproduce the exact deploy.

Current status (2026-11-14)

Re-check before trusting this snapshot:

  • v1.1 is deployed and Ready: web/api/ml :v1.1, cloudflared 2026.11.0, all four Deployments 1/1. Numeric UIDs are 10001:10001 for web/ml and 65532:65532 for API/cloudflared.
  • Tunnel active: pulsecast-gate1 serves pulse.noelsabia.dev through Nuxt (x-powered-by: Nuxt) and api-pulse.noelsabia.dev through the API; four HTTP/2 connections. /stage and /live return 200; /control is 401 anonymous and 200 with control auth. The temporary hello is deleted.
  • v1.1 120-VU Cloudflare run passed: 120/120 SSE connections opened and closed, 0 errors, p95 event 331 ms, p95 scenario 646 ms, presenter POST 49.60 ms, 361/361 checks. See docs/load-test.md.
  • Failure drill passed: ML scaled to 0 while API readiness/forecast cache and web stayed available; ML restored and API pod restarted cleanly. See docs/failure-drills.md.
  • API/product checks: 8 sensors, /api/meta mae=3.32, wape=27.5, r2=0.777, cache source=fastapi, staffing plan under 2 s.
  • DNS works: public resolvers return Cloudflare IPs for pulse.noelsabia.dev and api-pulse.noelsabia.dev. A local systemd-resolved negative cache can be cleared with resolvectl query --cache=no NAME followed by resolvectl flush-caches.
  • API hostname is api-pulse.noelsabia.dev: Universal SSL covers only *.noelsabia.dev, so the host must stay one label under the zone.
  • All three Secrets exist: gitea-regcred, cloudflared, control-auth.
  • QUIC/UDP 7844 is untested. TCP/443 to Cloudflare works, so the cloudflared manifest uses --protocol http2 instead of relying on unverified UDP.
  • Results docs are merged in main. This branch only updates CI/docs.

Troubleshooting

Symptom Check Fix
docker login → 401 Unauthorized Username and credential; 2FA Use a token with write:package (not the read-only token), or the account password if 2FA is off.
docker push → denied: requested access to the resource is denied Push credential scope; owner/package path Use a write:package token; make sure $OWNER matches the Gitea image owner.
ImagePullBackOff / ErrImagePull kubectl -n "$NS" describe pod -l app=hello (and the real pod); kubectl -n "$NS" get secret gitea-regcred The token must have read:package; the Secret must be in fch26-onemanarmy; the tag must exist in Gitea. Do not print the secret data.
CreateContainerConfigError kubectl -n "$NS" describe pod <pod> A referenced Secret key is missing (control-auth: CONTROL_USER, CONTROL_PASS, CONTROL_TOKEN) or the cloudflared Secret lacks TUNNEL_TOKEN. Recreate the Secret.
container has runAsNonRoot and image will run as root or a non-numeric-user error kubectl -n "$NS" get pod <pod> -o yaml; inspect securityContext Keep runAsNonRoot: true and set a positive numeric runAsUser/runAsGroup (SKE caveat). The manifests already do; do not "fix" this by disabling runAsNonRoot.
Public hostname has no DNS record getent hosts pulse.noelsabia.dev / getent hosts api-pulse.noelsabia.dev Save the public hostname in the tunnel dashboard in the noelsabia.dev zone; remove a conflicting A/AAAA/CNAME for that exact name. Do not add manual records for a tunnel hostname.
cloudflared → CrashLoopBackOff kubectl -n "$NS" logs deployment/cloudflared --previous Invalid/expired tunnel token, or the tunnel was deleted in Cloudflare. Recreate the Secret and restart.
cloudflared logs QUIC/timeout errors after you changed the protocol kubectl -n "$NS" logs deployment/cloudflared --tail=50 UDP/7844 is unverified on SKE. Put --protocol http2 back (the committed manifest already sets it); TCP/443 is confirmed.
Public 502 / 1033 / 530 kubectl -n "$NS" get endpoints web api hello; cloudflared logs No ready endpoints, wrong Service port, or the hostname route points at the wrong service. Fix the dashboard route or the Service. Never add an Ingress.
api pod not ready kubectl -n "$NS" logs deployment/api; curl -fsS http://ml:8000/bootstrap from inside the namespace; curl -fsS https://api-pulse.noelsabia.dev/api/readyz v1.1 has no baked data. It needs ML reachable at FASTAPI_URL and a successful /bootstrap; check BOOTSTRAP_REFRESH and the ML readiness. The distroless API image has no shell, so do not rely on kubectl exec ... ls.
ml pod stuck in startup kubectl -n "$NS" logs deployment/ml Model load exceeds the 120 s startup budget or MODEL_PATH is wrong. Fix the image/path.
web 500 kubectl -n "$NS" logs deployment/web Missing runtime API base or SSR error. Rebuild with NUXT_PUBLIC_API_BASE or fix the ConfigMap.
SSE events arrive only after a delay curl -N direct to the pod vs through Cloudflare Cloudflare buffering. Keep the 15 s heartbeat and the web polling fallback (/api/scenario every 5 s). Do not remove the heartbeat.
make gate1-check exits 2 Output lists [pending] Secrets/Deployments Complete the human steps above. Exit 2 is not a manifest failure.
make deploy refuses CONFIRM is empty Re-run make deploy CONFIRM=yes; this is the safety guard.
kubectl → Forbidden kubectl config current-context Use --context "$CTX". Do not create Roles/Ingress/Jobs as a workaround; escalate instead.
make validate fails Dry-run output Fix the manifest; do not apply it to see whether it works.

Rollback

No cluster-scoped resources are created, so rollback is namespace-local:

kubectl --context "$CTX" -n "$NS" rollout undo deployment/web
kubectl --context "$CTX" -n "$NS" rollout undo deployment/api
kubectl --context "$CTX" -n "$NS" rollout undo deployment/ml
kubectl --context "$CTX" -n "$NS" rollout undo deployment/cloudflared
kubectl --context "$CTX" -n "$NS" rollout status deployment/web --timeout=180s

If the tunnel token or route is wrong, fix the Secret / Cloudflare dashboard and restart cloudflared; do not create an Ingress as a workaround. Delete any leftover hello resources:

kubectl --context "$CTX" -n "$NS" delete service hello --ignore-not-found
kubectl --context "$CTX" -n "$NS" delete deployment hello --ignore-not-found

Security notes

  • Never commit the kubeconfig, tunnel token, control password, Gitea token, or any .env file. .gitignore covers the usual filenames; check git status before every commit.
  • The kubeconfig must stay at mode 600 and outside the repos. It was shared in chat: rotate/revoke the service-account token after the event.
  • Rotate the Gitea package token and the control password after the event.
  • Use the read-only package token only for gitea-regcred; use a separate write credential for pushes.
  • make smoke and make gate1-check print no secrets; do not add set -x to scripts while a live KUBECONFIG or token is exported.
  • Stop the tunnel and rotate the Gitea token if either leaks. Deleting the tunnel in Cloudflare invalidates its token.