The node:20-bookworm runner has python3 but no pip, so the PyYAML install failed before any check ran. Install python3-yaml with apt instead, remove the duplicate cluster-only kubectl step, and keep the two cluster-free Make targets. Also remove the stale v1.0/pending-review claims, drop the old api.pulse hostname, and move the deploy status below the runbook.
807 lines
31 KiB
Markdown
807 lines
31 KiB
Markdown
# Deploy runbook — Gate 1
|
||
|
||
Target: three services (`web`, `api`, `ml`) plus one `cloudflared` Deployment in
|
||
namespace `fch26-onemanarmy`, reachable over public HTTPS, with Gitea image
|
||
pulls proven before the real images are deployed.
|
||
|
||
**The human runs every cluster-mutating command in this document.** The infra
|
||
agent only writes and validates manifests. No Ingress, cert-manager,
|
||
external-dns, Jobs, CronJobs, CRDs, Roles, RoleBindings, NetworkPolicies, or
|
||
other cluster-scoped resources are used.
|
||
|
||
## Run it
|
||
|
||
From this repo root, with `KUBECONFIG` exported (see
|
||
[Before you start](#before-you-start)):
|
||
|
||
```bash
|
||
make validate
|
||
make gate1-check
|
||
make build
|
||
make push
|
||
make deploy CONFIRM=yes
|
||
make smoke SMOKE_SSE=1
|
||
```
|
||
|
||
The first deploy also needs the Gitea pull proof, Tunnel, and Secrets in
|
||
Steps 1–6. Every cluster mutation is human-run.
|
||
|
||
## What Gate 1 means
|
||
|
||
Gate 1 passes only when **all** of these are true:
|
||
|
||
1. The cluster pulls an image from Gitea in `fch26-onemanarmy` using
|
||
`gitea-regcred` (proven with a hello-world before the real images).
|
||
2. A Cloudflare Tunnel serves a public HTTPS URL from this namespace.
|
||
3. `web`, `api`, `ml`, and `cloudflared` are Ready.
|
||
4. `make smoke SMOKE_SSE=1` passes and a phone on mobile data can open
|
||
`https://pulse.noelsabia.dev`.
|
||
5. `https://api-pulse.noelsabia.dev/api/healthz` returns HTTP 200.
|
||
|
||
Do not claim Gate 1 from a local `kubectl port-forward`, a green manifest
|
||
dry-run, or a container that only runs on your laptop.
|
||
|
||
## Before you start
|
||
|
||
Run every block in this document in **one shell**, from the repo root. The
|
||
commands are bash/zsh compatible.
|
||
|
||
```bash
|
||
cd /path/to/infra-k8
|
||
|
||
export KUBECONFIG="/home/noel/Documents/Hackathon_2026/OKF_Vault/fch26-onemanarmy-sa-kubeconfig.yaml"
|
||
export NS="fch26-onemanarmy"
|
||
export CTX="fch26-onemanarmy-sa-context"
|
||
export REGISTRY="git.noelsabia.dev"
|
||
export OWNER="noel"
|
||
export TAG="v1.1"
|
||
|
||
chmod 600 "$KUBECONFIG"
|
||
```
|
||
|
||
The kubeconfig is **not in this repo**. If it moved, change the path; never
|
||
commit it or paste its contents anywhere. The Gitea owner for this workspace
|
||
is `noel`; if your account differs, set `OWNER` and the image paths
|
||
consistently.
|
||
|
||
Check the local tools:
|
||
|
||
```bash
|
||
for tool in kubectl docker curl jq make; do
|
||
command -v "$tool" >/dev/null || echo "MISSING: $tool"
|
||
done
|
||
```
|
||
|
||
Expected: no `MISSING` lines. `kubeconform`, `k6`, and `yamllint` are optional.
|
||
|
||
Check the context and the namespace permissions (read-only):
|
||
|
||
```bash
|
||
kubectl config current-context
|
||
kubectl --context "$CTX" -n "$NS" auth can-i create deployments
|
||
kubectl --context "$CTX" -n "$NS" auth can-i create services
|
||
kubectl --context "$CTX" -n "$NS" auth can-i create ingresses
|
||
kubectl --context "$CTX" -n "$NS" auth can-i create jobs
|
||
kubectl --context "$CTX" -n "$NS" get deployments
|
||
```
|
||
|
||
Expected:
|
||
|
||
```text
|
||
fch26-onemanarmy-sa-context
|
||
yes
|
||
yes
|
||
no
|
||
no
|
||
No resources found in fch26-onemanarmy namespace.
|
||
```
|
||
|
||
If `deployments` or `services` is not `yes`, or `ingresses`/`jobs` is not
|
||
`no`, stop and report. Do not work around permissions with cluster-scoped
|
||
resources.
|
||
|
||
## Gate 1 checklist
|
||
|
||
Run top to bottom. Stop at the first failure and use
|
||
[Troubleshooting](#troubleshooting).
|
||
|
||
- [ ] 0. `make gate1-check` shows 0 hard failures.
|
||
- [ ] 1. Gitea push login succeeds (`Login Succeeded`).
|
||
- [ ] 2. Read-only package token created; hello image pushed to Gitea.
|
||
- [ ] 3. `gitea-regcred` created; hello pod is `1/1 Running` from Gitea
|
||
(the Secret existing is not proof).
|
||
- [ ] 4. Cloudflare Tunnel created; both public hostnames configured and DNS
|
||
records resolve.
|
||
- [ ] 5. `cloudflared` Secret created; logs show `Registered tunnel connection`.
|
||
- [ ] 6. Public hello route returns HTTP 200 (proves tunnel + DNS + TLS).
|
||
- [ ] 7. `control-auth` Secret created.
|
||
- [ ] 8. All three v1.1 images built and pushed.
|
||
- [ ] 9. `make validate` and `make gate1-check` pass.
|
||
- [ ] 10. `make deploy CONFIRM=yes` completes; all four Deployments are Ready.
|
||
- [ ] 11. `make smoke SMOKE_SSE=1` passes.
|
||
- [ ] 12. Phone over mobile data opens `https://pulse.noelsabia.dev`;
|
||
`https://api-pulse.noelsabia.dev/api/healthz` returns 200.
|
||
- [ ] 13. Image tags/digests and the time are recorded.
|
||
|
||
## Step 0 — read-only readiness and dry-run
|
||
|
||
```bash
|
||
make validate
|
||
make gate1-check
|
||
```
|
||
|
||
Expected on a fresh cluster (before any human work):
|
||
|
||
- With a reachable cluster, `make validate` prints
|
||
`== kubectl client dry-run (read-only)` and 11 `created (dry run)` lines.
|
||
Without one it prints `11 manifests parsed`.
|
||
- `make gate1-check` reports **0 hard failures**, the expected permission
|
||
matrix, and lists missing Secrets/Deployments as `[pending]`.
|
||
- Exit code 2 from `make gate1-check` is correct at this stage: it means only
|
||
human prerequisites remain, not that the manifests are broken.
|
||
|
||
Exit codes: `0` ready, `1` hard failure (permissions or manifests), `2` only
|
||
pending human prerequisites. Use `make gate1-check ALLOW_PENDING=1` when you
|
||
want exit 0 while prerequisites are still being prepared.
|
||
|
||
Neither target mutates the cluster. `make validate` uses
|
||
`kubectl apply --dry-run=client`; `make gate1-check` only reads
|
||
configuration/RBAC/secrets and runs the same dry-run. CI runs the same static
|
||
guardrails without a cluster:
|
||
|
||
```bash
|
||
make gate1-check STATIC_ONLY=1
|
||
make validate
|
||
```
|
||
|
||
## Step 1 — Gitea credentials
|
||
|
||
You need **two different credentials**. Do not reuse the read-only token for
|
||
pushing.
|
||
|
||
1. **Push credential:** your normal Gitea account password, or a token with
|
||
`write:package`. Use a token if the account has 2FA.
|
||
2. **Cluster pull credential:** a separate token with **only**
|
||
`read:package`. This becomes `gitea-regcred`; it never needs write access.
|
||
|
||
Create the read-only token in Gitea: **Settings → Applications → Generate New
|
||
Token** (or the package-token screen). Name it `pulsecast-k8s-pull`; scope
|
||
`read:package`. Copy it once.
|
||
|
||
Log in for pushing:
|
||
|
||
```bash
|
||
read -rsp 'Gitea push password or write:package token: ' GITEA_PUSH_SECRET; echo
|
||
printf '%s' "$GITEA_PUSH_SECRET" | docker login "$REGISTRY" -u "$OWNER" --password-stdin
|
||
```
|
||
|
||
Expected:
|
||
|
||
```text
|
||
Login Succeeded
|
||
```
|
||
|
||
Store the read-only token in a shell variable for the Secret step:
|
||
|
||
```bash
|
||
read -rsp 'Gitea read-only package token: ' GITEA_RO_TOKEN; echo
|
||
```
|
||
|
||
Expected: no output (the value is not echoed).
|
||
|
||
If login fails with `401 Unauthorized`, the password/token or username is
|
||
wrong, or 2FA requires a token. Do not put either credential in a file, a
|
||
command that lands in shell history, or this repo.
|
||
|
||
## Step 2 — hello-world image push and pull
|
||
|
||
This proves the registry path before spending time on application images.
|
||
|
||
```bash
|
||
docker pull nginxinc/nginx-unprivileged:alpine
|
||
docker tag nginxinc/nginx-unprivileged:alpine "$REGISTRY/$OWNER/hello:gate1"
|
||
docker push "$REGISTRY/$OWNER/hello:gate1"
|
||
```
|
||
|
||
Expected: the push ends with a digest line, for example:
|
||
|
||
```text
|
||
gate1: digest: sha256:... size: ...
|
||
```
|
||
|
||
If you see `denied: requested access to the resource is denied`, the push
|
||
credential lacks `write:package` or the owner/package path is wrong. Fix that
|
||
before continuing.
|
||
|
||
Create the pull secret with the **read-only** token:
|
||
|
||
```bash
|
||
kubectl --context "$CTX" -n "$NS" create secret docker-registry gitea-regcred \
|
||
--docker-server="$REGISTRY" \
|
||
--docker-username="$OWNER" \
|
||
--docker-password="$GITEA_RO_TOKEN"
|
||
```
|
||
|
||
Expected: `secret/gitea-regcred created`. If it already exists, delete and
|
||
recreate it after reading the troubleshooting note:
|
||
`kubectl --context "$CTX" -n "$NS" delete secret gitea-regcred`.
|
||
|
||
Deploy the hello image and make the cluster pull it:
|
||
|
||
```bash
|
||
kubectl --context "$CTX" -n "$NS" create deployment hello \
|
||
--image="$REGISTRY/$OWNER/hello:gate1"
|
||
|
||
kubectl --context "$CTX" -n "$NS" patch deployment hello --type=merge -p \
|
||
'{"spec":{"template":{"spec":{"imagePullSecrets":[{"name":"gitea-regcred"}]}}}}'
|
||
|
||
kubectl --context "$CTX" -n "$NS" rollout status deployment/hello --timeout=180s
|
||
kubectl --context "$CTX" -n "$NS" get pods -l app=hello -o wide
|
||
```
|
||
|
||
Expected:
|
||
|
||
```text
|
||
deployment "hello" successfully rolled out
|
||
NAME READY STATUS RESTARTS AGE
|
||
hello-xxxxxxxxxx-xxxxx 1/1 Running 0 20s
|
||
```
|
||
|
||
**This is the first real pass/fail.** A `gitea-regcred` Secret existing is not
|
||
proof: the Gate 1 Gitea item is complete only when this hello pod is `1/1
|
||
Running` after pulling from Gitea. If the pod is not `Running`, stop.
|
||
`ImagePullBackOff` means the token, secret namespace, image tag, or registry
|
||
DNS is wrong. Diagnose with:
|
||
|
||
```bash
|
||
kubectl --context "$CTX" -n "$NS" describe pod -l app=hello
|
||
```
|
||
|
||
Keep the hello Deployment until Step 6 proves the public route. Create a
|
||
Service for it so Cloudflare has something to reach:
|
||
|
||
```bash
|
||
kubectl --context "$CTX" -n "$NS" expose deployment hello \
|
||
--port=8080 --target-port=8080
|
||
```
|
||
|
||
Expected: `service/hello exposed`.
|
||
|
||
## Step 3 — Cloudflare Tunnel
|
||
|
||
The tunnel used here is `pulsecast-gate1`. `pulse.noelsabia.dev` resolves;
|
||
`api-pulse.noelsabia.dev` resolves after its route is saved. Universal SSL
|
||
covers only `*.noelsabia.dev`, so the API host must stay one label under the
|
||
zone: `api-pulse.noelsabia.dev`.
|
||
|
||
```bash
|
||
getent hosts pulse.noelsabia.dev
|
||
getent hosts api-pulse.noelsabia.dev || echo 'api-pulse: no DNS record yet (expected until the route exists)'
|
||
```
|
||
|
||
If DNS looks missing locally but public DNS has it, flush the local cache:
|
||
|
||
```bash
|
||
resolvectl flush-caches
|
||
```
|
||
|
||
If `getent` still misses the name while `dig +short NAME @1.1.1.1` works, force
|
||
a fresh upstream query and flush again:
|
||
|
||
```bash
|
||
resolvectl query --cache=no api-pulse.noelsabia.dev
|
||
resolvectl flush-caches
|
||
getent hosts api-pulse.noelsabia.dev
|
||
```
|
||
|
||
1. Open Cloudflare **Zero Trust → Networks → Tunnels → Create a tunnel**.
|
||
2. Connector: **Cloudflared**. Name: `pulsecast-gate1`.
|
||
3. Copy the tunnel token from the connector install command. It is the long
|
||
`eyJ...` string after `--token` (or after `service install`). It is
|
||
sensitive: never paste it into chat, a commit, or a screenshot. The tunnel
|
||
stays **Inactive** with 0 routes until the connector Deployment is running;
|
||
routes can be added while it is inactive.
|
||
4. Open the tunnel and click the **Routes** tab (not Overview, and not
|
||
**Add a replica**). Click **Add route** or **Add a public hostname**, choose
|
||
**Public hostname**, and add:
|
||
|
||
| Subdomain | Domain | Path | Service type | Service URL |
|
||
|---|---|---|---|---|
|
||
| `pulse` | `noelsabia.dev` | *(empty)* | HTTP | `hello:8080` |
|
||
| `api-pulse` | `noelsabia.dev` | *(empty)* | HTTP | `api:8080` |
|
||
|
||
The first route is temporary for Gate 1-A. For the final Gate 1-B route,
|
||
change it to `web:3000` after the web image is deployed. If you already have
|
||
the web image, you may set `web:3000` now and skip the hello route; then the
|
||
public hello check in Step 6 is not available, but the tunnel proof still
|
||
happens when the real web pod is Ready.
|
||
|
||
5. The hostnames must be in the Cloudflare zone for `noelsabia.dev`.
|
||
Cloudflare creates the DNS records for a tunnel hostname; do **not** add
|
||
manual A/AAAA records. If the dashboard refuses to save a hostname because
|
||
a conflicting A/AAAA/CNAME already exists for that exact name, delete the
|
||
conflicting record first.
|
||
6. Do not add an Ingress. The tunnel is the only public path.
|
||
|
||
After saving both hostnames, DNS should resolve to Cloudflare:
|
||
|
||
```bash
|
||
getent hosts pulse.noelsabia.dev
|
||
getent hosts api-pulse.noelsabia.dev
|
||
```
|
||
|
||
Expected: one or more Cloudflare IPv4/IPv6 addresses. If there is still no
|
||
record, the public hostname was not saved, it is in the wrong zone, or a
|
||
conflicting record exists. Fix that in the dashboard before testing HTTPS;
|
||
`curl` cannot succeed without DNS.
|
||
|
||
Create the tunnel Secret (the token is not echoed):
|
||
|
||
```bash
|
||
read -rsp 'Cloudflare tunnel token: ' TUNNEL_TOKEN; echo
|
||
kubectl --context "$CTX" -n "$NS" create secret generic cloudflared \
|
||
--from-literal=TUNNEL_TOKEN="$TUNNEL_TOKEN"
|
||
```
|
||
|
||
Expected: `secret/cloudflared created`.
|
||
|
||
Deploy only the tunnel first. This does not need the application images:
|
||
|
||
```bash
|
||
kubectl --context "$CTX" -n "$NS" apply -f deploy/k8s/40-cloudflared.yaml
|
||
kubectl --context "$CTX" -n "$NS" rollout status deployment/cloudflared --timeout=180s
|
||
kubectl --context "$CTX" -n "$NS" logs deployment/cloudflared --tail=100 \
|
||
| grep 'Registered tunnel connection' || echo 'NO TUNNEL CONNECTION'
|
||
```
|
||
|
||
Expected:
|
||
|
||
```text
|
||
deployment.apps/cloudflared created
|
||
deployment "cloudflared" successfully rolled out
|
||
... Registered tunnel connection ...
|
||
```
|
||
|
||
(Usually four `Registered tunnel connection` lines, one per edge.)
|
||
|
||
The manifest deliberately uses `--protocol http2`: TCP/443 to Cloudflare is
|
||
verified, UDP/7844 (QUIC) is not. Do not switch to QUIC during Gate 1. If you
|
||
do test QUIC later, revert to HTTP/2 on any connection error.
|
||
|
||
If `cloudflared` is in `CrashLoopBackOff`, the token is invalid/expired or the
|
||
tunnel was deleted:
|
||
|
||
```bash
|
||
kubectl --context "$CTX" -n "$NS" logs deployment/cloudflared --previous
|
||
kubectl --context "$CTX" -n "$NS" describe pod -l app.kubernetes.io/name=cloudflared
|
||
```
|
||
|
||
## Step 4 — control-auth Secret
|
||
|
||
The web server validates `/control` and POST `/api/scenario` with
|
||
`CONTROL_USER`/`CONTROL_PASS`; web, API, and ML also need the same
|
||
`CONTROL_TOKEN` (the API forwards it as a Bearer token to FastAPI's internal
|
||
`/train` routes). Generate both locally; never commit them.
|
||
|
||
```bash
|
||
read -rsp 'Control password: ' CONTROL_PASS; echo
|
||
CONTROL_TOKEN="$(python3 -c 'import secrets; print(secrets.token_hex(24))')"
|
||
kubectl --context "$CTX" -n "$NS" create secret generic control-auth \
|
||
--from-literal=CONTROL_USER=control \
|
||
--from-literal=CONTROL_PASS="$CONTROL_PASS" \
|
||
--from-literal=CONTROL_TOKEN="$CONTROL_TOKEN"
|
||
```
|
||
|
||
Expected: `secret/control-auth created`.
|
||
|
||
The token is shared by API and ML, not the browser password. Without it
|
||
FastAPI `/train` returns 503 and the API's `/api/train` proxy cannot work.
|
||
|
||
`make secret-commands` prints all three Secret commands with placeholders; it
|
||
creates nothing.
|
||
|
||
## Step 5 — build and push the three images (Phase B)
|
||
|
||
Check whether the Dockerfiles exist:
|
||
|
||
```bash
|
||
test -f ../../web-nuxt.git/main/Dockerfile && echo web-ok || echo web-blocked
|
||
test -f ../../api-go.git/main/Dockerfile && echo api-ok || echo api-blocked
|
||
test -f ../../ML-py.git/main/Dockerfile && echo ml-ok || echo ml-blocked
|
||
```
|
||
|
||
Expected:
|
||
|
||
```text
|
||
web-ok
|
||
api-ok
|
||
ml-ok
|
||
```
|
||
|
||
Build all three:
|
||
|
||
```bash
|
||
make build
|
||
```
|
||
|
||
All three v1.1 images were verified locally with the manifest's runtime
|
||
constraints:
|
||
|
||
- API: distroless `static:nonroot`, `--user 65532:65532`, read-only root,
|
||
`/tmp` tmpfs; `/api/healthz` and `/api/readyz` returned 200, and the API
|
||
loaded ML `/bootstrap` into memory.
|
||
- ML: image `USER 10001:10001`; with read-only root and `/tmp` tmpfs,
|
||
`/healthz` and `/readyz` returned 200, `/bootstrap` returned 8 sensors, and
|
||
`/meta` showed `mae=3.32`, `wape=27.5`, `r2=0.777`.
|
||
- Web: image `USER 10001:10001`; with read-only root and `/tmp` tmpfs, `/`,
|
||
`/stage`, and `/live` returned 200; `/control` returned 401 anonymous and
|
||
200 with control auth; the same-origin SSE proxy emitted events.
|
||
- v1.1 is pushed to Gitea and deployed in the cluster; re-apply only after a
|
||
reviewed image change.
|
||
|
||
Push:
|
||
|
||
```bash
|
||
make push
|
||
```
|
||
|
||
Expected: each `docker push` ends with a digest. If authentication expired,
|
||
repeat the push login from Step 1.
|
||
|
||
The Makefile auto-detects the service worktrees:
|
||
|
||
```text
|
||
../../web-nuxt.git/main
|
||
../../api-go.git/main
|
||
../../ML-py.git/main
|
||
```
|
||
|
||
Override when the layout differs:
|
||
|
||
```bash
|
||
make build WEB_DIR=/path/to/web-nuxt API_DIR=/path/to/api-go ML_DIR=/path/to/ML-py
|
||
```
|
||
|
||
If `OWNER` or `TAG` differs from the manifests, update `deploy/k8s/*.yaml`
|
||
before `make deploy`, or deploy first and then run the guarded image update:
|
||
|
||
```bash
|
||
make set-image CONFIRM=yes OWNER=<owner> TAG=<tag>
|
||
```
|
||
|
||
### Dockerfile contract (coordinate through the orchestrator)
|
||
|
||
Each service image must:
|
||
|
||
- listen on `0.0.0.0` at the port below;
|
||
- run as a **non-root numeric UID** (no named user, no shell required);
|
||
- write only to `/tmp` (the pod root filesystem is read-only);
|
||
- start with the environment in `deploy/k8s/00-config.yaml`;
|
||
- be self-contained: no runtime download of data, model, fonts, or packages.
|
||
|
||
| Service | Image | Port | UID:GID | Health paths |
|
||
|---|---|---|---|---|
|
||
| web | `git.noelsabia.dev/noel/pulsecast-web:v1.1` | 3000 | `10001:10001` | `/`, `/stage`, `/live`, `/control` |
|
||
| api | `git.noelsabia.dev/noel/pulsecast-api:v1.1` | 8080 | `65532:65532` | `/api/healthz`, `/api/readyz` |
|
||
| ml | `git.noelsabia.dev/noel/pulsecast-ml:v1.1` | 8000 | `10001:10001` | `/healthz`, `/readyz`, `/bootstrap` |
|
||
|
||
The manifests set `runAsUser`/`runAsGroup`/`fsGroup` explicitly, so the image's
|
||
named user is irrelevant; the files must be readable by that numeric UID.
|
||
`api` uses distroless `static:nonroot` (65532); web and ml use 10001. If a
|
||
Dockerfile genuinely needs another UID, change the manifest and this table
|
||
together, and keep the value numeric.
|
||
|
||
`api` v1.1 reads `FASTAPI_URL=http://ml:8000`, `BOOTSTRAP_REFRESH=60s`,
|
||
`BOOTSTRAP_TIMEOUT`, `PORT`, `CONTROL_USER`, `CONTROL_PASS`, `CONTROL_TOKEN`,
|
||
`SSE_HEARTBEAT`, and `ML_TIMEOUT`. It has **no baked data and no `DATA_DIR`**;
|
||
it fetches ML `/bootstrap` into memory and refreshes it.
|
||
`web` proxies Go and SSE server-side using `NUXT_API_INTERNAL_BASE=http://api:8080`;
|
||
it also needs `CONTROL_USER`, `CONTROL_PASS`, and `CONTROL_TOKEN` to protect
|
||
`/control` and POST `/api/scenario`. `NUXT_PUBLIC_API_BASE` and the public SSE
|
||
URL are display/future configuration; the browser calls the Nuxt origin.
|
||
`ml` reads `PULSECAST_ROOT`, `PULSECAST_ARTIFACTS`, `PULSECAST_WEATHER`,
|
||
`PULSECAST_MODELS`, `PULSECAST_UPLOADS`, and `CONTROL_TOKEN`; its image CMD
|
||
binds `0.0.0.0:8000`. The env names in the manifests are the integration
|
||
point: change them here first if a service needs a different name. All three
|
||
images use numeric UIDs: web/ml `10001:10001`, API `65532:65532`.
|
||
|
||
## Step 6 — prove public HTTPS with hello
|
||
|
||
Do this **before** deleting hello. It proves tunnel + DNS + TLS without
|
||
depending on web/ml.
|
||
|
||
```bash
|
||
curl -fsS -o /dev/null -w 'hello %{http_code}\n' https://pulse.noelsabia.dev/
|
||
```
|
||
|
||
Expected: `hello 200`.
|
||
|
||
If you get `502`, `530`, or `1033`, the edge cannot reach the Service. Check:
|
||
|
||
```bash
|
||
kubectl --context "$CTX" -n "$NS" get endpoints hello
|
||
kubectl --context "$CTX" -n "$NS" get svc hello
|
||
kubectl --context "$CTX" -n "$NS" logs deployment/cloudflared --tail=50
|
||
```
|
||
|
||
The hostname route must be `http://hello:8080` and the hello Service must have
|
||
an endpoint. Fix the dashboard route or the Service; do not add an Ingress.
|
||
|
||
After the public hello check passes, delete it so it cannot mask a web problem:
|
||
|
||
```bash
|
||
kubectl --context "$CTX" -n "$NS" delete service hello
|
||
kubectl --context "$CTX" -n "$NS" delete deployment hello
|
||
```
|
||
|
||
Expected: `service "hello" deleted` and `deployment.apps "hello" deleted`.
|
||
|
||
`https://pulse.noelsabia.dev` will now return an edge error until the real web
|
||
Deployment is Ready. That is expected; do not "fix" it with an Ingress.
|
||
|
||
## Step 7 — validate and check readiness again
|
||
|
||
```bash
|
||
make validate
|
||
make gate1-check
|
||
```
|
||
|
||
Expected once Secrets and Deployments exist:
|
||
|
||
- `make validate`: 11 resources created (dry run);
|
||
- `make gate1-check`: 0 hard failures, 0 pending, `RESULT: READY for Gate 1`
|
||
(exit 0).
|
||
|
||
Before the deploy, pending Deployments keep it at exit 2. That is fine.
|
||
|
||
## Step 8 — deploy
|
||
|
||
```bash
|
||
make deploy-check
|
||
make deploy CONFIRM=yes
|
||
```
|
||
|
||
Expected:
|
||
|
||
```text
|
||
required Secrets and keys present
|
||
... created/configured ...
|
||
deployment "ml" successfully rolled out
|
||
deployment "api" successfully rolled out
|
||
deployment "web" successfully rolled out
|
||
deployment "cloudflared" successfully rolled out
|
||
```
|
||
|
||
`make deploy` fails closed: without `CONFIRM=yes` it changes nothing. It does
|
||
not create the namespace or Secrets. It applies all 11 manifests, including
|
||
the ConfigMaps.
|
||
|
||
If the real images do not exist, the rollouts fail with `ImagePullBackOff`.
|
||
Stop and finish Step 5; do not deploy a placeholder under `web`, `api`, or
|
||
`ml`.
|
||
|
||
Check the state:
|
||
|
||
```bash
|
||
make status
|
||
kubectl --context "$CTX" -n "$NS" get pods -o wide
|
||
kubectl --context "$CTX" -n "$NS" logs deployment/cloudflared --tail=50 \
|
||
| grep 'Registered tunnel connection' || echo 'NO TUNNEL CONNECTION'
|
||
```
|
||
|
||
Expected: `web`, `api`, `ml`, and `cloudflared` each `1/1`; Services `web`,
|
||
`api`, and `ml` are `ClusterIP`; `cloudflared` has no Service.
|
||
|
||
### Re-pushing the same tag
|
||
|
||
The manifests use `imagePullPolicy: Always` while the tag stays `v1.1`. After
|
||
rebuilding and pushing the same tag, force a re-pull:
|
||
|
||
```bash
|
||
make restart CONFIRM=yes
|
||
```
|
||
|
||
This restarts `web`, `api`, and `ml` and waits for the rollout. Do not use it
|
||
during the demo window unless you are fixing a blocker.
|
||
|
||
## Step 9 — verify public HTTPS and SSE
|
||
|
||
Full smoke, including the ML service through a short `kubectl port-forward`:
|
||
|
||
```bash
|
||
make smoke SMOKE_SSE=1
|
||
```
|
||
|
||
Expected: each check prints `PASS`, the summary ends with `0 failed`, and TLS
|
||
verification passes (`ssl_verify_result=0`). If the ML port-forward cannot
|
||
start, set `ML_URL` or `SKIP_ML=1`; do not silently drop the ML check from the
|
||
sign-off.
|
||
|
||
Manual public checks (also run from a phone on mobile data):
|
||
|
||
```bash
|
||
curl -fsS -o /dev/null -w 'web %{http_code}\n' https://pulse.noelsabia.dev/
|
||
curl -fsS -o /dev/null -w 'api %{http_code}\n' https://api-pulse.noelsabia.dev/api/healthz
|
||
curl -fsS -o /dev/null -w 'ready %{http_code}\n' https://api-pulse.noelsabia.dev/api/readyz
|
||
curl -fsS https://api-pulse.noelsabia.dev/api/meta | jq '{data_version,model_version}'
|
||
curl -sS -N --max-time 20 https://api-pulse.noelsabia.dev/api/stream | head -20
|
||
```
|
||
|
||
Expected:
|
||
|
||
```text
|
||
web 200
|
||
api 200
|
||
ready 200
|
||
{
|
||
"data_version": "...",
|
||
"model_version": "..."
|
||
}
|
||
retry: 3000
|
||
|
||
event: scenario
|
||
id: ...
|
||
data: {...}
|
||
|
||
event: ping
|
||
data: {"t":"..."}
|
||
```
|
||
|
||
The SSE output must start with `retry: 3000`, contain a `scenario` event
|
||
immediately, and show a `ping` within 20 seconds. If events are delayed
|
||
through Cloudflare, see the SSE row in Troubleshooting.
|
||
|
||
### v1.1 product checks
|
||
|
||
```bash
|
||
curl -fsS https://api-pulse.noelsabia.dev/api/sensors | jq 'length'
|
||
curl -fsS https://api-pulse.noelsabia.dev/api/meta \
|
||
| jq '{data_version,model_version,metrics,cache}'
|
||
curl -fsS -o /dev/null -w 'stage %{http_code}\n' https://pulse.noelsabia.dev/stage
|
||
curl -fsS -o /dev/null -w 'live %{http_code}\n' https://pulse.noelsabia.dev/live
|
||
curl -sS -o /dev/null -w 'control anon %{http_code}\n' https://pulse.noelsabia.dev/control
|
||
set -a; source ~/.config/pulsecast/control-auth.env; set +a
|
||
curl -sS -u "$CONTROL_USER:$CONTROL_PASS" -o /dev/null \
|
||
-w 'control auth %{http_code}\n' https://pulse.noelsabia.dev/control
|
||
curl -sS -o /dev/null -w 'plan %{time_total}s\n' \
|
||
'https://api-pulse.noelsabia.dev/api/staffing?sensor=hn-stadtgalerie-01&scenario=base&standard=standard'
|
||
```
|
||
|
||
Expected: **8** sensors; metrics `mae=3.32`, `wape=27.5`, `r2=0.777` with a
|
||
`cache` object; `/stage` and `/live` 200; `/control` 401 anonymous and 200
|
||
with control auth; plan under 2 seconds.
|
||
|
||
In-cluster connectivity (optional; **creates and removes one short-lived
|
||
pod**, so the human runs it):
|
||
|
||
```bash
|
||
kubectl --context "$CTX" -n "$NS" run gate1-netcheck --rm -i --restart=Never \
|
||
--image=curlimages/curl:latest \
|
||
--overrides='{"spec":{"securityContext":{"runAsNonRoot":true,"runAsUser":1000,"runAsGroup":1000}}}' \
|
||
--command -- sh -c \
|
||
'curl -fsS http://api:8080/api/healthz && \
|
||
curl -fsS http://ml:8000/healthz && \
|
||
curl -fsS http://web:3000/ >/dev/null && echo connectivity-ok'
|
||
```
|
||
|
||
Expected: `{"ok":true}{"ok":true}connectivity-ok` (order may vary with the
|
||
shell output). If the namespace's Pod Security Admission blocks this ad-hoc
|
||
pod, skip it and use `kubectl exec deploy/web -- ...` or the smoke test; do not
|
||
weaken the application Deployments to make a diagnostic pod run.
|
||
|
||
Finally, confirm the checker is green end-to-end:
|
||
|
||
```bash
|
||
make gate1-check CHECK_PUBLIC=1
|
||
```
|
||
|
||
Expected: `RESULT: READY for Gate 1` (exit 0).
|
||
|
||
## Step 10 — record the Gate 1 result
|
||
|
||
Record in the status note, README, or commit message:
|
||
|
||
```bash
|
||
date -u +'%Y-%m-%dT%H:%M:%SZ'
|
||
git rev-parse HEAD
|
||
kubectl --context "$CTX" -n "$NS" get deploy -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.template.spec.containers[0].image}{"\n"}{end}'
|
||
```
|
||
|
||
Also save:
|
||
|
||
- the output of `make smoke SMOKE_SSE=1`;
|
||
- the public URLs and the device/network used (phone on mobile data);
|
||
- any warning that remains.
|
||
|
||
Image digests are in the `docker push` output; tags alone are not enough if
|
||
you need to reproduce the exact deploy.
|
||
|
||
## Current status (2026-11-14)
|
||
|
||
Re-check before trusting this snapshot:
|
||
|
||
- **v1.1 is deployed and Ready:** web/api/ml `:v1.1`, cloudflared
|
||
`2026.11.0`, all four Deployments `1/1`. Numeric UIDs are
|
||
`10001:10001` for web/ml and `65532:65532` for API/cloudflared.
|
||
- **Tunnel active:** `pulsecast-gate1` serves `pulse.noelsabia.dev` through
|
||
Nuxt (`x-powered-by: Nuxt`) and `api-pulse.noelsabia.dev` through the API;
|
||
four HTTP/2 connections. `/stage` and `/live` return 200; `/control` is
|
||
401 anonymous and 200 with control auth. The temporary hello is deleted.
|
||
- **v1.1 120-VU Cloudflare run passed:** 120/120 SSE connections opened and
|
||
closed, 0 errors, p95 event 331 ms, p95 scenario 646 ms, presenter POST
|
||
49.60 ms, 361/361 checks. See `docs/load-test.md`.
|
||
- **Failure drill passed:** ML scaled to 0 while API readiness/forecast cache
|
||
and web stayed available; ML restored and API pod restarted cleanly. See
|
||
`docs/failure-drills.md`.
|
||
- **API/product checks:** 8 sensors, `/api/meta` `mae=3.32`, `wape=27.5`,
|
||
`r2=0.777`, cache `source=fastapi`, staffing plan under 2 s.
|
||
- **DNS works:** public resolvers return Cloudflare IPs for
|
||
`pulse.noelsabia.dev` and `api-pulse.noelsabia.dev`. A local
|
||
`systemd-resolved` negative cache can be cleared with
|
||
`resolvectl query --cache=no NAME` followed by `resolvectl flush-caches`.
|
||
- **API hostname is `api-pulse.noelsabia.dev`:** Universal SSL covers only
|
||
`*.noelsabia.dev`, so the host must stay one label under the zone.
|
||
- **All three Secrets exist:** `gitea-regcred`, `cloudflared`, `control-auth`.
|
||
- **QUIC/UDP 7844 is untested.** TCP/443 to Cloudflare works, so the
|
||
`cloudflared` manifest uses `--protocol http2` instead of relying on
|
||
unverified UDP.
|
||
- **Results docs are merged in `main`.** This branch only updates CI/docs.
|
||
|
||
## Troubleshooting
|
||
|
||
| Symptom | Check | Fix |
|
||
|---|---|---|
|
||
| `docker login` → `401 Unauthorized` | Username and credential; 2FA | Use a token with `write:package` (not the read-only token), or the account password if 2FA is off. |
|
||
| `docker push` → `denied: requested access to the resource is denied` | Push credential scope; owner/package path | Use a `write:package` token; make sure `$OWNER` matches the Gitea image owner. |
|
||
| `ImagePullBackOff` / `ErrImagePull` | `kubectl -n "$NS" describe pod -l app=hello` (and the real pod); `kubectl -n "$NS" get secret gitea-regcred` | The token must have `read:package`; the Secret must be in `fch26-onemanarmy`; the tag must exist in Gitea. Do not print the secret data. |
|
||
| `CreateContainerConfigError` | `kubectl -n "$NS" describe pod <pod>` | A referenced Secret key is missing (`control-auth`: `CONTROL_USER`, `CONTROL_PASS`, `CONTROL_TOKEN`) or the `cloudflared` Secret lacks `TUNNEL_TOKEN`. Recreate the Secret. |
|
||
| `container has runAsNonRoot and image will run as root` or a non-numeric-user error | `kubectl -n "$NS" get pod <pod> -o yaml`; inspect `securityContext` | Keep `runAsNonRoot: true` and set a positive numeric `runAsUser`/`runAsGroup` (SKE caveat). The manifests already do; do not "fix" this by disabling `runAsNonRoot`. |
|
||
| Public hostname has no DNS record | `getent hosts pulse.noelsabia.dev` / `getent hosts api-pulse.noelsabia.dev` | Save the public hostname in the tunnel dashboard in the `noelsabia.dev` zone; remove a conflicting A/AAAA/CNAME for that exact name. Do not add manual records for a tunnel hostname. |
|
||
| `cloudflared` → `CrashLoopBackOff` | `kubectl -n "$NS" logs deployment/cloudflared --previous` | Invalid/expired tunnel token, or the tunnel was deleted in Cloudflare. Recreate the Secret and restart. |
|
||
| `cloudflared` logs QUIC/timeout errors after you changed the protocol | `kubectl -n "$NS" logs deployment/cloudflared --tail=50` | UDP/7844 is unverified on SKE. Put `--protocol http2` back (the committed manifest already sets it); TCP/443 is confirmed. |
|
||
| Public `502` / `1033` / `530` | `kubectl -n "$NS" get endpoints web api hello`; cloudflared logs | No ready endpoints, wrong Service port, or the hostname route points at the wrong service. Fix the dashboard route or the Service. Never add an Ingress. |
|
||
| `api` pod not ready | `kubectl -n "$NS" logs deployment/api`; `curl -fsS http://ml:8000/bootstrap` from inside the namespace; `curl -fsS https://api-pulse.noelsabia.dev/api/readyz` | v1.1 has no baked data. It needs ML reachable at `FASTAPI_URL` and a successful `/bootstrap`; check `BOOTSTRAP_REFRESH` and the ML readiness. The distroless API image has no shell, so do not rely on `kubectl exec ... ls`. |
|
||
| `ml` pod stuck in startup | `kubectl -n "$NS" logs deployment/ml` | Model load exceeds the 120 s startup budget or `MODEL_PATH` is wrong. Fix the image/path. |
|
||
| `web` 500 | `kubectl -n "$NS" logs deployment/web` | Missing runtime API base or SSR error. Rebuild with `NUXT_PUBLIC_API_BASE` or fix the ConfigMap. |
|
||
| SSE events arrive only after a delay | `curl -N` direct to the pod vs through Cloudflare | Cloudflare buffering. Keep the 15 s heartbeat and the web polling fallback (`/api/scenario` every 5 s). Do not remove the heartbeat. |
|
||
| `make gate1-check` exits 2 | Output lists `[pending]` Secrets/Deployments | Complete the human steps above. Exit 2 is not a manifest failure. |
|
||
| `make deploy` refuses | `CONFIRM` is empty | Re-run `make deploy CONFIRM=yes`; this is the safety guard. |
|
||
| `kubectl` → `Forbidden` | `kubectl config current-context` | Use `--context "$CTX"`. Do not create Roles/Ingress/Jobs as a workaround; escalate instead. |
|
||
| `make validate` fails | Dry-run output | Fix the manifest; do not apply it to see whether it works. |
|
||
|
||
## Rollback
|
||
|
||
No cluster-scoped resources are created, so rollback is namespace-local:
|
||
|
||
```bash
|
||
kubectl --context "$CTX" -n "$NS" rollout undo deployment/web
|
||
kubectl --context "$CTX" -n "$NS" rollout undo deployment/api
|
||
kubectl --context "$CTX" -n "$NS" rollout undo deployment/ml
|
||
kubectl --context "$CTX" -n "$NS" rollout undo deployment/cloudflared
|
||
kubectl --context "$CTX" -n "$NS" rollout status deployment/web --timeout=180s
|
||
```
|
||
|
||
If the tunnel token or route is wrong, fix the Secret / Cloudflare dashboard
|
||
and restart `cloudflared`; do not create an Ingress as a workaround. Delete any
|
||
leftover hello resources:
|
||
|
||
```bash
|
||
kubectl --context "$CTX" -n "$NS" delete service hello --ignore-not-found
|
||
kubectl --context "$CTX" -n "$NS" delete deployment hello --ignore-not-found
|
||
```
|
||
|
||
## Security notes
|
||
|
||
- Never commit the kubeconfig, tunnel token, control password, Gitea token, or
|
||
any `.env` file. `.gitignore` covers the usual filenames; check
|
||
`git status` before every commit.
|
||
- The kubeconfig must stay at mode `600` and outside the repos. It was shared
|
||
in chat: rotate/revoke the service-account token after the event.
|
||
- Rotate the Gitea package token and the control password after the event.
|
||
- Use the read-only package token only for `gitea-regcred`; use a separate
|
||
write credential for pushes.
|
||
- `make smoke` and `make gate1-check` print no secrets; do not add `set -x` to
|
||
scripts while a live `KUBECONFIG` or token is exported.
|
||
- Stop the tunnel and rotate the Gitea token if either leaks. Deleting the
|
||
tunnel in Cloudflare invalidates its token.
|