Public hostnames are under the tunnel Routes tab, not an Overview action. Record that the tunnel stays Inactive until the connector runs and that the token comes from the install command.
765 lines
30 KiB
Markdown
765 lines
30 KiB
Markdown
# Deploy runbook — Gate 1
|
|
|
|
Target: three services (`web`, `api`, `ml`) plus one `cloudflared` Deployment in
|
|
namespace `fch26-onemanarmy`, reachable over public HTTPS, with Gitea image
|
|
pulls proven before the real images are deployed.
|
|
|
|
**The human runs every cluster-mutating command in this document.** The infra
|
|
agent only writes and validates manifests. No Ingress, cert-manager,
|
|
external-dns, Jobs, CronJobs, CRDs, Roles, RoleBindings, NetworkPolicies, or
|
|
other cluster-scoped resources are used.
|
|
|
|
## What Gate 1 means
|
|
|
|
Gate 1 passes only when **all** of these are true:
|
|
|
|
1. The cluster pulls an image from Gitea in `fch26-onemanarmy` using
|
|
`gitea-regcred` (proven with a hello-world before the real images).
|
|
2. A Cloudflare Tunnel serves a public HTTPS URL from this namespace.
|
|
3. `web`, `api`, `ml`, and `cloudflared` are Ready.
|
|
4. `make smoke SMOKE_SSE=1` passes and a phone on mobile data can open
|
|
`https://pulse.noelsabia.dev`.
|
|
5. `https://api.pulse.noelsabia.dev/api/healthz` returns HTTP 200.
|
|
|
|
Do not claim Gate 1 from a local `kubectl port-forward`, a green manifest
|
|
dry-run, or a container that only runs on your laptop.
|
|
|
|
## Current external status (2026-11-14 03:50 UTC smoke)
|
|
|
|
Findings from `CLUSTER-SMOKE.md`; re-check before trusting this snapshot:
|
|
|
|
- **Verified:** cluster access, scheduling, Service DNS/HTTP, probes, Secrets,
|
|
non-root + read-only-root + `emptyDir`, and egress to Cloudflare,
|
|
Open-Meteo, and `git.noelsabia.dev`.
|
|
- **No public DNS yet:** `pulse.noelsabia.dev` and
|
|
`api.pulse.noelsabia.dev` have no records. That is expected until the
|
|
Cloudflare Tunnel public hostnames are created.
|
|
- **No tunnel yet:** Cloudflare Tunnel has not been created.
|
|
- **Gitea pull is proven (2026-11-14 ~06:16 CET):** `gitea-regcred` exists and
|
|
the cluster pulled `git.noelsabia.dev/noel/hello:gate1` in 120 ms; the hello
|
|
pod is `1/1 Running` and its Service returned HTTP 200. The Gitea owner is
|
|
`noel`.
|
|
- **QUIC/UDP 7844 is untested.** TCP/443 to Cloudflare works, so the
|
|
`cloudflared` manifest uses `--protocol http2` instead of relying on
|
|
unverified UDP.
|
|
- **Numeric UIDs are required:** a non-numeric image user breaks
|
|
`runAsNonRoot` on SKE (reproduced with `curlimages/curl`). Every pod in
|
|
`deploy/k8s/` sets explicit numeric `runAsUser`/`runAsGroup`/`fsGroup`.
|
|
- **Repo status (2026-11-14 ~05:10 UTC):** all three images now build and pass
|
|
local manifest-equivalent runtime checks (numeric UID, read-only root, `/tmp`
|
|
tmpfs). There is no remaining Dockerfile blocker; the human Gitea/Tunnel
|
|
steps are what remain.
|
|
|
|
## What is possible now vs what is blocked
|
|
|
|
| Step | Who | Status |
|
|
|---|---|---|
|
|
| Read-only cluster/manifest check | agent or human | **ready now** (`make gate1-check`) |
|
|
| Gitea push login + hello-world push | human | **done**; hello pull and Service verified |
|
|
| `gitea-regcred`, `cloudflared`, `control-auth` Secrets | human | `gitea-regcred` **done**; `cloudflared`/`control-auth` pending |
|
|
| Cloudflare Tunnel + public hostnames/DNS | human | **blocked on human** — not created; DNS absent |
|
|
| `api` image | human | **ready** — `make build-api` + manifest-equivalent run verified locally |
|
|
| `ml` image | human | **ready** — `make build-ml` + manifest-equivalent run verified locally |
|
|
| `web` image | human | **ready** — `make build-web` + manifest-equivalent run verified locally |
|
|
| Full deploy + smoke + Gate 1 sign-off | human | **ready once the Gitea/Tunnel steps are done** |
|
|
|
|
`make build` and `make push` cover all three services; no Dockerfile is
|
|
missing now. Do not deploy placeholder images under the real service names.
|
|
|
|
## Before you start
|
|
|
|
Run every block in this document in **one shell**, from this worktree. The
|
|
commands are bash/zsh compatible.
|
|
|
|
```bash
|
|
cd /home/noel/Documents/Hackathon_2026/infra-k8.git/feat-infra-bootstrap
|
|
|
|
export KUBECONFIG="/home/noel/Documents/Hackathon_2026/OKF_Vault/fch26-onemanarmy-sa-kubeconfig.yaml"
|
|
export NS="fch26-onemanarmy"
|
|
export CTX="fch26-onemanarmy-sa-context"
|
|
export REGISTRY="git.noelsabia.dev"
|
|
export OWNER="noel"
|
|
export TAG="gate1"
|
|
|
|
chmod 600 "$KUBECONFIG"
|
|
```
|
|
|
|
The kubeconfig is **not in this repo**. If it moved, change the path; never
|
|
commit it or paste its contents anywhere. The Gitea owner for this workspace
|
|
is `noel`; if your account differs, set `OWNER` and the image paths
|
|
consistently.
|
|
|
|
Check the local tools:
|
|
|
|
```bash
|
|
for tool in kubectl docker curl jq make; do
|
|
command -v "$tool" >/dev/null || echo "MISSING: $tool"
|
|
done
|
|
```
|
|
|
|
Expected: no `MISSING` lines. `kubeconform`, `k6`, and `yamllint` are optional.
|
|
|
|
Check the context and the namespace permissions (read-only):
|
|
|
|
```bash
|
|
kubectl config current-context
|
|
kubectl --context "$CTX" -n "$NS" auth can-i create deployments
|
|
kubectl --context "$CTX" -n "$NS" auth can-i create services
|
|
kubectl --context "$CTX" -n "$NS" auth can-i create ingresses
|
|
kubectl --context "$CTX" -n "$NS" auth can-i create jobs
|
|
kubectl --context "$CTX" -n "$NS" get deployments
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
fch26-onemanarmy-sa-context
|
|
yes
|
|
yes
|
|
no
|
|
no
|
|
No resources found in fch26-onemanarmy namespace.
|
|
```
|
|
|
|
If `deployments` or `services` is not `yes`, or `ingresses`/`jobs` is not
|
|
`no`, stop and report. Do not work around permissions with cluster-scoped
|
|
resources.
|
|
|
|
## Gate 1 checklist
|
|
|
|
Run top to bottom. Stop at the first failure and use
|
|
[Troubleshooting](#troubleshooting).
|
|
|
|
- [ ] 0. `make gate1-check` shows 0 hard failures.
|
|
- [ ] 1. Gitea push login succeeds (`Login Succeeded`).
|
|
- [ ] 2. Read-only package token created; hello image pushed to Gitea.
|
|
- [ ] 3. `gitea-regcred` created; hello pod is `1/1 Running` from Gitea
|
|
(the Secret existing is not proof).
|
|
- [ ] 4. Cloudflare Tunnel created; both public hostnames configured and DNS
|
|
records resolve.
|
|
- [ ] 5. `cloudflared` Secret created; logs show `Registered tunnel connection`.
|
|
- [ ] 6. Public hello route returns HTTP 200 (proves tunnel + DNS + TLS).
|
|
- [ ] 7. `control-auth` Secret created.
|
|
- [ ] 8. All three images built, tagged `gate1`, and pushed.
|
|
- [ ] 9. `make validate` and `make gate1-check` pass.
|
|
- [ ] 10. `make deploy CONFIRM=yes` completes; all four Deployments are Ready.
|
|
- [ ] 11. `make smoke SMOKE_SSE=1` passes.
|
|
- [ ] 12. Phone over mobile data opens `https://pulse.noelsabia.dev`;
|
|
`https://api.pulse.noelsabia.dev/api/healthz` returns 200.
|
|
- [ ] 13. Image tags/digests and the time are recorded.
|
|
|
|
## Step 0 — read-only readiness and dry-run
|
|
|
|
```bash
|
|
make validate
|
|
make gate1-check
|
|
```
|
|
|
|
Expected now (before any human work):
|
|
|
|
- `make validate` prints `== kubectl client dry-run (read-only)` and 11
|
|
`created (dry run)` lines.
|
|
- `make gate1-check` reports **0 hard failures**, the expected permission
|
|
matrix, and lists missing Secrets/Deployments as `[pending]`.
|
|
- Exit code 2 from `make gate1-check` is correct at this stage: it means only
|
|
human prerequisites remain, not that the manifests are broken.
|
|
|
|
Exit codes: `0` ready, `1` hard failure (permissions or manifests), `2` only
|
|
pending human prerequisites. Use `make gate1-check ALLOW_PENDING=1` when you
|
|
want exit 0 while prerequisites are still being prepared.
|
|
|
|
Neither target mutates the cluster. `make validate` uses
|
|
`kubectl apply --dry-run=client`; `make gate1-check` only reads
|
|
configuration/RBAC/secrets and runs the same dry-run. CI runs the same static
|
|
guardrails without a cluster:
|
|
|
|
```bash
|
|
make gate1-check STATIC_ONLY=1
|
|
make validate
|
|
```
|
|
|
|
## Step 1 — Gitea credentials
|
|
|
|
You need **two different credentials**. Do not reuse the read-only token for
|
|
pushing.
|
|
|
|
1. **Push credential:** your normal Gitea account password, or a token with
|
|
`write:package`. Use a token if the account has 2FA.
|
|
2. **Cluster pull credential:** a separate token with **only**
|
|
`read:package`. This becomes `gitea-regcred`; it never needs write access.
|
|
|
|
Create the read-only token in Gitea: **Settings → Applications → Generate New
|
|
Token** (or the package-token screen). Name it `pulsecast-k8s-pull`; scope
|
|
`read:package`. Copy it once.
|
|
|
|
Log in for pushing:
|
|
|
|
```bash
|
|
read -rsp 'Gitea push password or write:package token: ' GITEA_PUSH_SECRET; echo
|
|
printf '%s' "$GITEA_PUSH_SECRET" | docker login "$REGISTRY" -u "$OWNER" --password-stdin
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
Login Succeeded
|
|
```
|
|
|
|
Store the read-only token in a shell variable for the Secret step:
|
|
|
|
```bash
|
|
read -rsp 'Gitea read-only package token: ' GITEA_RO_TOKEN; echo
|
|
```
|
|
|
|
Expected: no output (the value is not echoed).
|
|
|
|
If login fails with `401 Unauthorized`, the password/token or username is
|
|
wrong, or 2FA requires a token. Do not put either credential in a file, a
|
|
command that lands in shell history, or this repo.
|
|
|
|
## Step 2 — hello-world image push and pull
|
|
|
|
This proves the registry path before spending time on application images.
|
|
|
|
```bash
|
|
docker pull nginxinc/nginx-unprivileged:alpine
|
|
docker tag nginxinc/nginx-unprivileged:alpine "$REGISTRY/$OWNER/hello:gate1"
|
|
docker push "$REGISTRY/$OWNER/hello:gate1"
|
|
```
|
|
|
|
Expected: the push ends with a digest line, for example:
|
|
|
|
```text
|
|
gate1: digest: sha256:... size: ...
|
|
```
|
|
|
|
If you see `denied: requested access to the resource is denied`, the push
|
|
credential lacks `write:package` or the owner/package path is wrong. Fix that
|
|
before continuing.
|
|
|
|
Create the pull secret with the **read-only** token:
|
|
|
|
```bash
|
|
kubectl --context "$CTX" -n "$NS" create secret docker-registry gitea-regcred \
|
|
--docker-server="$REGISTRY" \
|
|
--docker-username="$OWNER" \
|
|
--docker-password="$GITEA_RO_TOKEN"
|
|
```
|
|
|
|
Expected: `secret/gitea-regcred created`. If it already exists, delete and
|
|
recreate it after reading the troubleshooting note:
|
|
`kubectl --context "$CTX" -n "$NS" delete secret gitea-regcred`.
|
|
|
|
Deploy the hello image and make the cluster pull it:
|
|
|
|
```bash
|
|
kubectl --context "$CTX" -n "$NS" create deployment hello \
|
|
--image="$REGISTRY/$OWNER/hello:gate1"
|
|
|
|
kubectl --context "$CTX" -n "$NS" patch deployment hello --type=merge -p \
|
|
'{"spec":{"template":{"spec":{"imagePullSecrets":[{"name":"gitea-regcred"}]}}}}'
|
|
|
|
kubectl --context "$CTX" -n "$NS" rollout status deployment/hello --timeout=180s
|
|
kubectl --context "$CTX" -n "$NS" get pods -l app=hello -o wide
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
deployment "hello" successfully rolled out
|
|
NAME READY STATUS RESTARTS AGE
|
|
hello-xxxxxxxxxx-xxxxx 1/1 Running 0 20s
|
|
```
|
|
|
|
**This is the first real pass/fail.** A `gitea-regcred` Secret existing is not
|
|
proof: the Gate 1 Gitea item is complete only when this hello pod is `1/1
|
|
Running` after pulling from Gitea. If the pod is not `Running`, stop.
|
|
`ImagePullBackOff` means the token, secret namespace, image tag, or registry
|
|
DNS is wrong. Diagnose with:
|
|
|
|
```bash
|
|
kubectl --context "$CTX" -n "$NS" describe pod -l app=hello
|
|
```
|
|
|
|
Keep the hello Deployment until Step 6 proves the public route. Create a
|
|
Service for it so Cloudflare has something to reach:
|
|
|
|
```bash
|
|
kubectl --context "$CTX" -n "$NS" expose deployment hello \
|
|
--port=8080 --target-port=8080
|
|
```
|
|
|
|
Expected: `service/hello exposed`.
|
|
|
|
## Step 3 — Cloudflare Tunnel
|
|
|
|
As of the 03:50 UTC smoke, neither public hostname has a DNS record. That is
|
|
expected before the tunnel route exists; do not add manual A/AAAA records.
|
|
|
|
```bash
|
|
getent hosts pulse.noelsabia.dev || echo 'pulse: no DNS record yet (expected)'
|
|
getent hosts api.pulse.noelsabia.dev || echo 'api: no DNS record yet (expected)'
|
|
```
|
|
|
|
1. Open Cloudflare **Zero Trust → Networks → Tunnels → Create a tunnel**.
|
|
2. Connector: **Cloudflared**. Name: `pulsecast-gate1`.
|
|
3. Copy the tunnel token from the connector install command. It is the long
|
|
`eyJ...` string after `--token` (or after `service install`). It is
|
|
sensitive: never paste it into chat, a commit, or a screenshot. The tunnel
|
|
stays **Inactive** with 0 routes until the connector Deployment is running;
|
|
routes can be added while it is inactive.
|
|
4. Open the tunnel and click the **Routes** tab (not Overview, and not
|
|
**Add a replica**). Click **Add route** or **Add a public hostname**, choose
|
|
**Public hostname**, and add:
|
|
|
|
| Subdomain | Domain | Path | Service type | Service URL |
|
|
|---|---|---|---|---|
|
|
| `pulse` | `noelsabia.dev` | *(empty)* | HTTP | `hello:8080` |
|
|
| `api.pulse` | `noelsabia.dev` | *(empty)* | HTTP | `api:8080` |
|
|
|
|
The first route is temporary for Gate 1-A. For the final Gate 1-B route,
|
|
change it to `web:3000` after the web image is deployed. If you already have
|
|
the web image, you may set `web:3000` now and skip the hello route; then the
|
|
public hello check in Step 6 is not available, but the tunnel proof still
|
|
happens when the real web pod is Ready.
|
|
|
|
For the final Gate 1-B route, change `pulse.noelsabia.dev` to
|
|
`http://web:3000` after the web image is deployed. If you already have the
|
|
web image, you may set `web:3000` now and skip the hello route; then the
|
|
public hello check in Step 6 is not available, but the tunnel proof still
|
|
happens when the real web pod is Ready.
|
|
|
|
5. The hostnames must be in the Cloudflare zone for `noelsabia.dev`.
|
|
Cloudflare creates the DNS records for a tunnel hostname; do **not** add
|
|
manual A/AAAA records. If the dashboard refuses to save a hostname because
|
|
a conflicting A/AAAA/CNAME already exists for that exact name, delete the
|
|
conflicting record first.
|
|
6. Do not add an Ingress. The tunnel is the only public path.
|
|
|
|
After saving both hostnames, DNS should resolve to Cloudflare:
|
|
|
|
```bash
|
|
getent hosts pulse.noelsabia.dev
|
|
getent hosts api.pulse.noelsabia.dev
|
|
```
|
|
|
|
Expected: one or more Cloudflare IPv4/IPv6 addresses. If there is still no
|
|
record, the public hostname was not saved, it is in the wrong zone, or a
|
|
conflicting record exists. Fix that in the dashboard before testing HTTPS;
|
|
`curl` cannot succeed without DNS.
|
|
|
|
Create the tunnel Secret (the token is not echoed):
|
|
|
|
```bash
|
|
read -rsp 'Cloudflare tunnel token: ' TUNNEL_TOKEN; echo
|
|
kubectl --context "$CTX" -n "$NS" create secret generic cloudflared \
|
|
--from-literal=TUNNEL_TOKEN="$TUNNEL_TOKEN"
|
|
```
|
|
|
|
Expected: `secret/cloudflared created`.
|
|
|
|
Deploy only the tunnel first. This does not need the application images:
|
|
|
|
```bash
|
|
kubectl --context "$CTX" -n "$NS" apply -f deploy/k8s/40-cloudflared.yaml
|
|
kubectl --context "$CTX" -n "$NS" rollout status deployment/cloudflared --timeout=180s
|
|
kubectl --context "$CTX" -n "$NS" logs deployment/cloudflared --tail=100 \
|
|
| grep 'Registered tunnel connection' || echo 'NO TUNNEL CONNECTION'
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
deployment.apps/cloudflared created
|
|
deployment "cloudflared" successfully rolled out
|
|
... Registered tunnel connection ...
|
|
```
|
|
|
|
(Usually four `Registered tunnel connection` lines, one per edge.)
|
|
|
|
The manifest deliberately uses `--protocol http2`: TCP/443 to Cloudflare is
|
|
verified, UDP/7844 (QUIC) is not. Do not switch to QUIC during Gate 1. If you
|
|
do test QUIC later, revert to HTTP/2 on any connection error.
|
|
|
|
If `cloudflared` is in `CrashLoopBackOff`, the token is invalid/expired or the
|
|
tunnel was deleted:
|
|
|
|
```bash
|
|
kubectl --context "$CTX" -n "$NS" logs deployment/cloudflared --previous
|
|
kubectl --context "$CTX" -n "$NS" describe pod -l app.kubernetes.io/name=cloudflared
|
|
```
|
|
|
|
## Step 4 — control-auth Secret
|
|
|
|
The API reads `CONTROL_USER` and `CONTROL_PASS` for browser control auth.
|
|
Both API and ML also need the same `CONTROL_TOKEN`: the API forwards it as a
|
|
Bearer token to FastAPI's internal `/train` routes. Generate both locally;
|
|
never commit them.
|
|
|
|
```bash
|
|
read -rsp 'Control password: ' CONTROL_PASS; echo
|
|
CONTROL_TOKEN="$(python3 -c 'import secrets; print(secrets.token_hex(24))')"
|
|
kubectl --context "$CTX" -n "$NS" create secret generic control-auth \
|
|
--from-literal=CONTROL_USER=control \
|
|
--from-literal=CONTROL_PASS="$CONTROL_PASS" \
|
|
--from-literal=CONTROL_TOKEN="$CONTROL_TOKEN"
|
|
```
|
|
|
|
Expected: `secret/control-auth created`.
|
|
|
|
The token is shared by API and ML, not the browser password. Without it
|
|
FastAPI `/train` returns 503 and the API's `/api/train` proxy cannot work.
|
|
|
|
`make secret-commands` prints all three Secret commands with placeholders; it
|
|
creates nothing.
|
|
|
|
## Step 5 — build and push the three images (Phase B)
|
|
|
|
Check whether the Dockerfiles exist:
|
|
|
|
```bash
|
|
test -f ../../web-nuxt.git/feat-web-bootstrap/Dockerfile && echo web-ok || echo web-blocked
|
|
test -f ../../api-go.git/feat-api-bootstrap/Dockerfile && echo api-ok || echo api-blocked
|
|
test -f ../../ML-py.git/feat-ml-bootstrap/Dockerfile && echo ml-ok || echo ml-blocked
|
|
```
|
|
|
|
Expected **now** (all three Dockerfiles landed on 2026-11-14):
|
|
|
|
```text
|
|
web-ok
|
|
api-ok
|
|
ml-ok
|
|
```
|
|
|
|
Build all three:
|
|
|
|
```bash
|
|
make build
|
|
```
|
|
|
|
All three images were verified locally with the manifest's runtime
|
|
constraints:
|
|
|
|
- API: distroless `static:nonroot`, `--user 65532:65532`, read-only root,
|
|
`/tmp` tmpfs; `/api/healthz` and `/api/readyz` returned 200.
|
|
- ML: image `USER app` is non-numeric, but the manifest overrides it with
|
|
`runAsUser: 10001`; with read-only root and `/tmp` tmpfs, `/healthz` and
|
|
`/readyz` returned 200 and `/meta` loaded the baked model.
|
|
- Web: image `USER node` is non-numeric, but the manifest overrides it with
|
|
`runAsUser: 10001`; with read-only root and `/tmp` tmpfs, `/` returned
|
|
HTTP 200 and the runtime `NUXT_PUBLIC_API_BASE` appeared in the HTML.
|
|
- No image was pushed or deployed by the infra agent.
|
|
|
|
Push:
|
|
|
|
```bash
|
|
make push
|
|
```
|
|
|
|
Expected: each `docker push` ends with a digest. If authentication expired,
|
|
repeat the push login from Step 1.
|
|
|
|
The Makefile auto-detects the service worktrees:
|
|
|
|
```text
|
|
../../web-nuxt.git/feat-web-bootstrap
|
|
../../api-go.git/feat-api-bootstrap
|
|
../../ML-py.git/feat-ml-bootstrap
|
|
```
|
|
|
|
Override when the layout differs:
|
|
|
|
```bash
|
|
make build WEB_DIR=/path/to/web-nuxt API_DIR=/path/to/api-go ML_DIR=/path/to/ML-py
|
|
```
|
|
|
|
If `OWNER` or `TAG` differs from the manifests, update `deploy/k8s/*.yaml`
|
|
before `make deploy`, or deploy first and then run the guarded image update:
|
|
|
|
```bash
|
|
make set-image CONFIRM=yes OWNER=<owner> TAG=<tag>
|
|
```
|
|
|
|
### Dockerfile contract (coordinate through the orchestrator)
|
|
|
|
Each service image must:
|
|
|
|
- listen on `0.0.0.0` at the port below;
|
|
- run as a **non-root numeric UID** (no named user, no shell required);
|
|
- write only to `/tmp` (the pod root filesystem is read-only);
|
|
- start with the environment in `deploy/k8s/00-config.yaml`;
|
|
- be self-contained: no runtime download of data, model, fonts, or packages.
|
|
|
|
| Service | Image | Port | UID:GID | Health paths |
|
|
|---|---|---|---|---|
|
|
| web | `git.noelsabia.dev/noel/pulsecast-web:gate1` | 3000 | `10001:10001` | `/` |
|
|
| api | `git.noelsabia.dev/noel/pulsecast-api:gate1` | 8080 | `65532:65532` | `/api/healthz`, `/api/readyz` |
|
|
| ml | `git.noelsabia.dev/noel/pulsecast-ml:gate1` | 8000 | `10001:10001` | `/healthz`, `/readyz` |
|
|
|
|
The manifests set `runAsUser`/`runAsGroup`/`fsGroup` explicitly, so the image's
|
|
named user is irrelevant; the files must be readable by that numeric UID.
|
|
`api` uses distroless `static:nonroot` (65532); web and ml use 10001. If a
|
|
Dockerfile genuinely needs another UID, change the manifest and this table
|
|
together, and keep the value numeric.
|
|
|
|
`api` must read `DATA_DIR` (baked JSON), `FASTAPI_URL`, `PORT`,
|
|
`CONTROL_USER`, `CONTROL_PASS`, `CONTROL_TOKEN`, `SSE_HEARTBEAT`, and
|
|
`ML_TIMEOUT`.
|
|
`web` must have `NUXT_PUBLIC_API_BASE=https://api.pulse.noelsabia.dev` baked in
|
|
or read it at runtime from the ConfigMap; the browser must use the public API
|
|
hostname. `ml` reads `PULSECAST_ROOT`, `PULSECAST_ARTIFACTS`,
|
|
`PULSECAST_WEATHER`, `PULSECAST_MODELS`, `PULSECAST_UPLOADS`, and
|
|
`CONTROL_TOKEN`; its image CMD binds `0.0.0.0:8000`. The env names in the
|
|
manifests are the integration point: change them here first if a service needs
|
|
a different name. The ML image currently uses a named `app` user; the manifest
|
|
overrides it with numeric UID 10001, which was verified locally.
|
|
|
|
## Step 6 — prove public HTTPS with hello
|
|
|
|
Do this **before** deleting hello. It proves tunnel + DNS + TLS without
|
|
depending on web/ml.
|
|
|
|
```bash
|
|
curl -fsS -o /dev/null -w 'hello %{http_code}\n' https://pulse.noelsabia.dev/
|
|
```
|
|
|
|
Expected: `hello 200`.
|
|
|
|
If you get `502`, `530`, or `1033`, the edge cannot reach the Service. Check:
|
|
|
|
```bash
|
|
kubectl --context "$CTX" -n "$NS" get endpoints hello
|
|
kubectl --context "$CTX" -n "$NS" get svc hello
|
|
kubectl --context "$CTX" -n "$NS" logs deployment/cloudflared --tail=50
|
|
```
|
|
|
|
The hostname route must be `http://hello:8080` and the hello Service must have
|
|
an endpoint. Fix the dashboard route or the Service; do not add an Ingress.
|
|
|
|
After the public hello check passes, delete it so it cannot mask a web problem:
|
|
|
|
```bash
|
|
kubectl --context "$CTX" -n "$NS" delete service hello
|
|
kubectl --context "$CTX" -n "$NS" delete deployment hello
|
|
```
|
|
|
|
Expected: `service "hello" deleted` and `deployment.apps "hello" deleted`.
|
|
|
|
`https://pulse.noelsabia.dev` will now return an edge error until the real web
|
|
Deployment is Ready. That is expected; do not "fix" it with an Ingress.
|
|
|
|
## Step 7 — validate and check readiness again
|
|
|
|
```bash
|
|
make validate
|
|
make gate1-check
|
|
```
|
|
|
|
Expected once Secrets and Deployments exist:
|
|
|
|
- `make validate`: 11 resources created (dry run);
|
|
- `make gate1-check`: 0 hard failures, 0 pending, `RESULT: READY for Gate 1`
|
|
(exit 0).
|
|
|
|
Before the deploy, pending Deployments keep it at exit 2. That is fine.
|
|
|
|
## Step 8 — deploy
|
|
|
|
```bash
|
|
make deploy-check
|
|
make deploy CONFIRM=yes
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
required Secrets and keys present
|
|
... created/configured ...
|
|
deployment "ml" successfully rolled out
|
|
deployment "api" successfully rolled out
|
|
deployment "web" successfully rolled out
|
|
deployment "cloudflared" successfully rolled out
|
|
```
|
|
|
|
`make deploy` fails closed: without `CONFIRM=yes` it changes nothing. It does
|
|
not create the namespace or Secrets. It applies all 11 manifests, including
|
|
the ConfigMaps.
|
|
|
|
If the real images do not exist, the rollouts fail with `ImagePullBackOff`.
|
|
Stop and finish Step 5; do not deploy a placeholder under `web`, `api`, or
|
|
`ml`.
|
|
|
|
Check the state:
|
|
|
|
```bash
|
|
make status
|
|
kubectl --context "$CTX" -n "$NS" get pods -o wide
|
|
kubectl --context "$CTX" -n "$NS" logs deployment/cloudflared --tail=50 \
|
|
| grep 'Registered tunnel connection' || echo 'NO TUNNEL CONNECTION'
|
|
```
|
|
|
|
Expected: `web`, `api`, `ml`, and `cloudflared` each `1/1`; Services `web`,
|
|
`api`, and `ml` are `ClusterIP`; `cloudflared` has no Service.
|
|
|
|
### Re-pushing the same tag
|
|
|
|
The manifests use `imagePullPolicy: Always` while the tag stays `gate1`. After
|
|
rebuilding and pushing the same tag, force a re-pull:
|
|
|
|
```bash
|
|
make restart CONFIRM=yes
|
|
```
|
|
|
|
This restarts `web`, `api`, and `ml` and waits for the rollout. Do not use it
|
|
during the demo window unless you are fixing a blocker.
|
|
|
|
## Step 9 — verify public HTTPS and SSE
|
|
|
|
Full smoke, including the ML service through a short `kubectl port-forward`:
|
|
|
|
```bash
|
|
make smoke SMOKE_SSE=1
|
|
```
|
|
|
|
Expected: each check prints `PASS`, the summary ends with `0 failed`, and TLS
|
|
verification passes (`ssl_verify_result=0`). If the ML port-forward cannot
|
|
start, set `ML_URL` or `SKIP_ML=1`; do not silently drop the ML check from the
|
|
sign-off.
|
|
|
|
Manual public checks (also run from a phone on mobile data):
|
|
|
|
```bash
|
|
curl -fsS -o /dev/null -w 'web %{http_code}\n' https://pulse.noelsabia.dev/
|
|
curl -fsS -o /dev/null -w 'api %{http_code}\n' https://api.pulse.noelsabia.dev/api/healthz
|
|
curl -fsS -o /dev/null -w 'ready %{http_code}\n' https://api.pulse.noelsabia.dev/api/readyz
|
|
curl -fsS https://api.pulse.noelsabia.dev/api/meta | jq '{data_version,model_version}'
|
|
curl -sS -N --max-time 20 https://api.pulse.noelsabia.dev/api/stream | head -20
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
web 200
|
|
api 200
|
|
ready 200
|
|
{
|
|
"data_version": "...",
|
|
"model_version": "..."
|
|
}
|
|
retry: 3000
|
|
|
|
event: scenario
|
|
id: ...
|
|
data: {...}
|
|
|
|
event: ping
|
|
data: {"t":"..."}
|
|
```
|
|
|
|
The SSE output must start with `retry: 3000`, contain a `scenario` event
|
|
immediately, and show a `ping` within 20 seconds. If events are delayed
|
|
through Cloudflare, see the SSE row in Troubleshooting.
|
|
|
|
In-cluster connectivity (optional; **creates and removes one short-lived
|
|
pod**, so the human runs it):
|
|
|
|
```bash
|
|
kubectl --context "$CTX" -n "$NS" run gate1-netcheck --rm -i --restart=Never \
|
|
--image=curlimages/curl:latest \
|
|
--overrides='{"spec":{"securityContext":{"runAsNonRoot":true,"runAsUser":1000,"runAsGroup":1000}}}' \
|
|
--command -- sh -c \
|
|
'curl -fsS http://api:8080/api/healthz && \
|
|
curl -fsS http://ml:8000/healthz && \
|
|
curl -fsS http://web:3000/ >/dev/null && echo connectivity-ok'
|
|
```
|
|
|
|
Expected: `{"ok":true}{"ok":true}connectivity-ok` (order may vary with the
|
|
shell output). If the namespace's Pod Security Admission blocks this ad-hoc
|
|
pod, skip it and use `kubectl exec deploy/web -- ...` or the smoke test; do not
|
|
weaken the application Deployments to make a diagnostic pod run.
|
|
|
|
Finally, confirm the checker is green end-to-end:
|
|
|
|
```bash
|
|
make gate1-check CHECK_PUBLIC=1
|
|
```
|
|
|
|
Expected: `RESULT: READY for Gate 1` (exit 0).
|
|
|
|
## Step 10 — record the Gate 1 result
|
|
|
|
Record in the status note, README, or commit message:
|
|
|
|
```bash
|
|
date -u +'%Y-%m-%dT%H:%M:%SZ'
|
|
git rev-parse HEAD
|
|
kubectl --context "$CTX" -n "$NS" get deploy -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.template.spec.containers[0].image}{"\n"}{end}'
|
|
```
|
|
|
|
Also save:
|
|
|
|
- the output of `make smoke SMOKE_SSE=1`;
|
|
- the public URLs and the device/network used (phone on mobile data);
|
|
- any warning that remains.
|
|
|
|
Image digests are in the `docker push` output; tags alone are not enough if
|
|
you need to reproduce the exact deploy.
|
|
|
|
## Troubleshooting
|
|
|
|
| Symptom | Check | Fix |
|
|
|---|---|---|
|
|
| `docker login` → `401 Unauthorized` | Username and credential; 2FA | Use a token with `write:package` (not the read-only token), or the account password if 2FA is off. |
|
|
| `docker push` → `denied: requested access to the resource is denied` | Push credential scope; owner/package path | Use a `write:package` token; make sure `$OWNER` matches the Gitea image owner. |
|
|
| `ImagePullBackOff` / `ErrImagePull` | `kubectl -n "$NS" describe pod -l app=hello` (and the real pod); `kubectl -n "$NS" get secret gitea-regcred` | The token must have `read:package`; the Secret must be in `fch26-onemanarmy`; the tag must exist in Gitea. Do not print the secret data. |
|
|
| `CreateContainerConfigError` | `kubectl -n "$NS" describe pod <pod>` | A referenced Secret key is missing (`control-auth`: `CONTROL_USER`, `CONTROL_PASS`, `CONTROL_TOKEN`) or the `cloudflared` Secret lacks `TUNNEL_TOKEN`. Recreate the Secret. |
|
|
| `container has runAsNonRoot and image will run as root` or a non-numeric-user error | `kubectl -n "$NS" get pod <pod> -o yaml`; inspect `securityContext` | Keep `runAsNonRoot: true` and set a positive numeric `runAsUser`/`runAsGroup` (SKE caveat). The manifests already do; do not "fix" this by disabling `runAsNonRoot`. |
|
|
| Public hostname has no DNS record | `getent hosts pulse.noelsabia.dev` / `getent hosts api.pulse.noelsabia.dev` | Save the public hostname in the tunnel dashboard in the `noelsabia.dev` zone; remove a conflicting A/AAAA/CNAME for that exact name. Do not add manual records for a tunnel hostname. |
|
|
| `cloudflared` → `CrashLoopBackOff` | `kubectl -n "$NS" logs deployment/cloudflared --previous` | Invalid/expired tunnel token, or the tunnel was deleted in Cloudflare. Recreate the Secret and restart. |
|
|
| `cloudflared` logs QUIC/timeout errors after you changed the protocol | `kubectl -n "$NS" logs deployment/cloudflared --tail=50` | UDP/7844 is unverified on SKE. Put `--protocol http2` back (the committed manifest already sets it); TCP/443 is confirmed. |
|
|
| Public `502` / `1033` / `530` | `kubectl -n "$NS" get endpoints web api hello`; cloudflared logs | No ready endpoints, wrong Service port, or the hostname route points at the wrong service. Fix the dashboard route or the Service. Never add an Ingress. |
|
|
| `api` pod not ready | `kubectl -n "$NS" logs deployment/api`; `curl -fsS https://api.pulse.noelsabia.dev/api/readyz` | Baked JSON missing from the image or `DATA_DIR` wrong. Rebuild the API image with the artifacts. The distroless API image has no shell, so do not rely on `kubectl exec ... ls`. |
|
|
| `ml` pod stuck in startup | `kubectl -n "$NS" logs deployment/ml` | Model load exceeds the 120 s startup budget or `MODEL_PATH` is wrong. Fix the image/path. |
|
|
| `web` 500 | `kubectl -n "$NS" logs deployment/web` | Missing runtime API base or SSR error. Rebuild with `NUXT_PUBLIC_API_BASE` or fix the ConfigMap. |
|
|
| SSE events arrive only after a delay | `curl -N` direct to the pod vs through Cloudflare | Cloudflare buffering. Keep the 15 s heartbeat and the web polling fallback (`/api/scenario` every 5 s). Do not remove the heartbeat. |
|
|
| `make gate1-check` exits 2 | Output lists `[pending]` Secrets/Deployments | Complete the human steps above. Exit 2 is not a manifest failure. |
|
|
| `make deploy` refuses | `CONFIRM` is empty | Re-run `make deploy CONFIRM=yes`; this is the safety guard. |
|
|
| `kubectl` → `Forbidden` | `kubectl config current-context` | Use `--context "$CTX"`. Do not create Roles/Ingress/Jobs as a workaround; escalate instead. |
|
|
| `make validate` fails | Dry-run output | Fix the manifest; do not apply it to see whether it works. |
|
|
|
|
## Rollback
|
|
|
|
No cluster-scoped resources are created, so rollback is namespace-local:
|
|
|
|
```bash
|
|
kubectl --context "$CTX" -n "$NS" rollout undo deployment/web
|
|
kubectl --context "$CTX" -n "$NS" rollout undo deployment/api
|
|
kubectl --context "$CTX" -n "$NS" rollout undo deployment/ml
|
|
kubectl --context "$CTX" -n "$NS" rollout undo deployment/cloudflared
|
|
kubectl --context "$CTX" -n "$NS" rollout status deployment/web --timeout=180s
|
|
```
|
|
|
|
If the tunnel token or route is wrong, fix the Secret / Cloudflare dashboard
|
|
and restart `cloudflared`; do not create an Ingress as a workaround. Delete any
|
|
leftover hello resources:
|
|
|
|
```bash
|
|
kubectl --context "$CTX" -n "$NS" delete service hello --ignore-not-found
|
|
kubectl --context "$CTX" -n "$NS" delete deployment hello --ignore-not-found
|
|
```
|
|
|
|
## Security notes
|
|
|
|
- Never commit the kubeconfig, tunnel token, control password, Gitea token, or
|
|
any `.env` file. `.gitignore` covers the usual filenames; check
|
|
`git status` before every commit.
|
|
- The kubeconfig must stay at mode `600` and outside the repos. It was shared
|
|
in chat: rotate/revoke the service-account token after the event.
|
|
- Rotate the Gitea package token and the control password after the event.
|
|
- Use the read-only package token only for `gitea-regcred`; use a separate
|
|
write credential for pushes.
|
|
- `make smoke` and `make gate1-check` print no secrets; do not add `set -x` to
|
|
scripts while a live `KUBECONFIG` or token is exported.
|
|
- Stop the tunnel and rotate the Gitea token if either leaks. Deleting the
|
|
tunnel in Cloudflare invalidates its token.
|