Baseline: 120/120 SSE connections opened/closed, 0 errors, p95 event 1.03 s, p95 scenario 194 ms, presenter POST 82.69 ms, 2,760 events, 2,400 pings, 4m53s. Raw summary is git-ignored. Also confirm pulse.noelsabia.dev already serves Nuxt; no route switch is needed. /stage and /live remain 404 on v1.0 until v1.1 is applied.
811 lines
32 KiB
Markdown
811 lines
32 KiB
Markdown
# Deploy runbook — Gate 1
|
|
|
|
Target: three services (`web`, `api`, `ml`) plus one `cloudflared` Deployment in
|
|
namespace `fch26-onemanarmy`, reachable over public HTTPS, with Gitea image
|
|
pulls proven before the real images are deployed.
|
|
|
|
**The human runs every cluster-mutating command in this document.** The infra
|
|
agent only writes and validates manifests. No Ingress, cert-manager,
|
|
external-dns, Jobs, CronJobs, CRDs, Roles, RoleBindings, NetworkPolicies, or
|
|
other cluster-scoped resources are used.
|
|
|
|
## What Gate 1 means
|
|
|
|
Gate 1 passes only when **all** of these are true:
|
|
|
|
1. The cluster pulls an image from Gitea in `fch26-onemanarmy` using
|
|
`gitea-regcred` (proven with a hello-world before the real images).
|
|
2. A Cloudflare Tunnel serves a public HTTPS URL from this namespace.
|
|
3. `web`, `api`, `ml`, and `cloudflared` are Ready.
|
|
4. `make smoke SMOKE_SSE=1` passes and a phone on mobile data can open
|
|
`https://pulse.noelsabia.dev`.
|
|
5. `https://api-pulse.noelsabia.dev/api/healthz` returns HTTP 200.
|
|
|
|
Do not claim Gate 1 from a local `kubectl port-forward`, a green manifest
|
|
dry-run, or a container that only runs on your laptop.
|
|
|
|
## Current external status (2026-11-14 ~08:00 CET)
|
|
|
|
Re-check before trusting this snapshot:
|
|
|
|
- **Verified:** cluster access, scheduling, Service DNS/HTTP, probes, Secrets,
|
|
non-root + read-only-root + `emptyDir`, and egress to Cloudflare,
|
|
Open-Meteo, and `git.noelsabia.dev`.
|
|
- **Tunnel active:** `pulsecast-gate1` has routes for `pulse.noelsabia.dev`
|
|
and `api-pulse.noelsabia.dev`; `cloudflared` is `1/1` with four HTTP/2
|
|
connections. `pulse` already serves the v1.0 Nuxt app
|
|
(`x-powered-by: Nuxt`), so **no Cloudflare route switch remains**.
|
|
`/stage` and `/live` return 404 on v1.0; v1.1 adds them. `api-pulse` serves
|
|
the deployed API.
|
|
- **120-VU baseline passed:** 120/120 SSE connections opened/closed, 0
|
|
connection errors, p95 event latency 1.03 s, p95 scenario latency 194 ms;
|
|
see `docs/load-test.md`. This was against v1.0.
|
|
- **DNS works:** public resolvers return Cloudflare IPs for
|
|
`pulse.noelsabia.dev` and `api-pulse.noelsabia.dev`. A local
|
|
`systemd-resolved` negative cache can be cleared with
|
|
`resolvectl query --cache=no NAME` followed by `resolvectl flush-caches`.
|
|
- **API hostname is `api-pulse.noelsabia.dev`:** Universal SSL covers only
|
|
`*.noelsabia.dev`; a two-level `api.pulse.noelsabia.dev` is not covered by
|
|
that certificate.
|
|
- **Gitea pull is proven:** `gitea-regcred` exists; hello and the v1.0 service
|
|
images pulled from Gitea. The Gitea owner is `noel`.
|
|
- **v1.1 images pushed:** web/api/ml `:v1.1` were built from the service
|
|
`main` worktrees, locally exercised under numeric UIDs/read-only root, and
|
|
pushed. The cluster still runs the previous v1.0 images until the v1.1
|
|
branch is reviewed/merged and explicitly applied.
|
|
- **All three Secrets exist:** `gitea-regcred`, `cloudflared`, `control-auth`.
|
|
- **QUIC/UDP 7844 is untested.** TCP/443 to Cloudflare works, so the
|
|
`cloudflared` manifest uses `--protocol http2` instead of relying on
|
|
unverified UDP.
|
|
- **Numeric UIDs are required:** a non-numeric image user breaks
|
|
`runAsNonRoot` on SKE (reproduced with `curlimages/curl`). Every pod in
|
|
`deploy/k8s/` sets explicit numeric `runAsUser`/`runAsGroup`/`fsGroup`.
|
|
|
|
## What is possible now vs what is blocked
|
|
|
|
| Step | Who | Status |
|
|
|---|---|---|
|
|
| Read-only cluster/manifest check | agent or human | **ready now** (`make gate1-check`) |
|
|
| Gitea push login + hello-world push | human | **done**; hello pull and Service verified |
|
|
| `gitea-regcred`, `cloudflared`, `control-auth` Secrets | human | **done** |
|
|
| Cloudflare Tunnel + public hostnames/DNS | human | **done**; tunnel active; `pulse` serves Nuxt |
|
|
| `api`, `ml`, `web` images | human | **v1.1 pushed**; local manifest-equivalent checks pass |
|
|
| v1.1 apply + smoke + presenter k6 + drills | human | **blocked on branch review/merge** |
|
|
|
|
`make build` and `make push` cover all three services; no Dockerfile is
|
|
missing now. Do not deploy placeholder images under the real service names.
|
|
|
|
## Before you start
|
|
|
|
Run every block in this document in **one shell**, from this worktree. The
|
|
commands are bash/zsh compatible.
|
|
|
|
```bash
|
|
cd /home/noel/Documents/Hackathon_2026/infra-k8.git/feat-infra-bootstrap
|
|
|
|
export KUBECONFIG="/home/noel/Documents/Hackathon_2026/OKF_Vault/fch26-onemanarmy-sa-kubeconfig.yaml"
|
|
export NS="fch26-onemanarmy"
|
|
export CTX="fch26-onemanarmy-sa-context"
|
|
export REGISTRY="git.noelsabia.dev"
|
|
export OWNER="noel"
|
|
export TAG="v1.1"
|
|
|
|
chmod 600 "$KUBECONFIG"
|
|
```
|
|
|
|
The kubeconfig is **not in this repo**. If it moved, change the path; never
|
|
commit it or paste its contents anywhere. The Gitea owner for this workspace
|
|
is `noel`; if your account differs, set `OWNER` and the image paths
|
|
consistently.
|
|
|
|
Check the local tools:
|
|
|
|
```bash
|
|
for tool in kubectl docker curl jq make; do
|
|
command -v "$tool" >/dev/null || echo "MISSING: $tool"
|
|
done
|
|
```
|
|
|
|
Expected: no `MISSING` lines. `kubeconform`, `k6`, and `yamllint` are optional.
|
|
|
|
Check the context and the namespace permissions (read-only):
|
|
|
|
```bash
|
|
kubectl config current-context
|
|
kubectl --context "$CTX" -n "$NS" auth can-i create deployments
|
|
kubectl --context "$CTX" -n "$NS" auth can-i create services
|
|
kubectl --context "$CTX" -n "$NS" auth can-i create ingresses
|
|
kubectl --context "$CTX" -n "$NS" auth can-i create jobs
|
|
kubectl --context "$CTX" -n "$NS" get deployments
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
fch26-onemanarmy-sa-context
|
|
yes
|
|
yes
|
|
no
|
|
no
|
|
No resources found in fch26-onemanarmy namespace.
|
|
```
|
|
|
|
If `deployments` or `services` is not `yes`, or `ingresses`/`jobs` is not
|
|
`no`, stop and report. Do not work around permissions with cluster-scoped
|
|
resources.
|
|
|
|
## Gate 1 checklist
|
|
|
|
Run top to bottom. Stop at the first failure and use
|
|
[Troubleshooting](#troubleshooting).
|
|
|
|
- [ ] 0. `make gate1-check` shows 0 hard failures.
|
|
- [ ] 1. Gitea push login succeeds (`Login Succeeded`).
|
|
- [ ] 2. Read-only package token created; hello image pushed to Gitea.
|
|
- [ ] 3. `gitea-regcred` created; hello pod is `1/1 Running` from Gitea
|
|
(the Secret existing is not proof).
|
|
- [ ] 4. Cloudflare Tunnel created; both public hostnames configured and DNS
|
|
records resolve.
|
|
- [ ] 5. `cloudflared` Secret created; logs show `Registered tunnel connection`.
|
|
- [ ] 6. Public hello route returns HTTP 200 (proves tunnel + DNS + TLS).
|
|
- [ ] 7. `control-auth` Secret created.
|
|
- [ ] 8. All three v1.1 images built and pushed.
|
|
- [ ] 9. `make validate` and `make gate1-check` pass.
|
|
- [ ] 10. `make deploy CONFIRM=yes` completes; all four Deployments are Ready.
|
|
- [ ] 11. `make smoke SMOKE_SSE=1` passes.
|
|
- [ ] 12. Phone over mobile data opens `https://pulse.noelsabia.dev`;
|
|
`https://api-pulse.noelsabia.dev/api/healthz` returns 200.
|
|
- [ ] 13. Image tags/digests and the time are recorded.
|
|
|
|
## Step 0 — read-only readiness and dry-run
|
|
|
|
```bash
|
|
make validate
|
|
make gate1-check
|
|
```
|
|
|
|
Expected now (before any human work):
|
|
|
|
- `make validate` prints `== kubectl client dry-run (read-only)` and 11
|
|
`created (dry run)` lines.
|
|
- `make gate1-check` reports **0 hard failures**, the expected permission
|
|
matrix, and lists missing Secrets/Deployments as `[pending]`.
|
|
- Exit code 2 from `make gate1-check` is correct at this stage: it means only
|
|
human prerequisites remain, not that the manifests are broken.
|
|
|
|
Exit codes: `0` ready, `1` hard failure (permissions or manifests), `2` only
|
|
pending human prerequisites. Use `make gate1-check ALLOW_PENDING=1` when you
|
|
want exit 0 while prerequisites are still being prepared.
|
|
|
|
Neither target mutates the cluster. `make validate` uses
|
|
`kubectl apply --dry-run=client`; `make gate1-check` only reads
|
|
configuration/RBAC/secrets and runs the same dry-run. CI runs the same static
|
|
guardrails without a cluster:
|
|
|
|
```bash
|
|
make gate1-check STATIC_ONLY=1
|
|
make validate
|
|
```
|
|
|
|
## Step 1 — Gitea credentials
|
|
|
|
You need **two different credentials**. Do not reuse the read-only token for
|
|
pushing.
|
|
|
|
1. **Push credential:** your normal Gitea account password, or a token with
|
|
`write:package`. Use a token if the account has 2FA.
|
|
2. **Cluster pull credential:** a separate token with **only**
|
|
`read:package`. This becomes `gitea-regcred`; it never needs write access.
|
|
|
|
Create the read-only token in Gitea: **Settings → Applications → Generate New
|
|
Token** (or the package-token screen). Name it `pulsecast-k8s-pull`; scope
|
|
`read:package`. Copy it once.
|
|
|
|
Log in for pushing:
|
|
|
|
```bash
|
|
read -rsp 'Gitea push password or write:package token: ' GITEA_PUSH_SECRET; echo
|
|
printf '%s' "$GITEA_PUSH_SECRET" | docker login "$REGISTRY" -u "$OWNER" --password-stdin
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
Login Succeeded
|
|
```
|
|
|
|
Store the read-only token in a shell variable for the Secret step:
|
|
|
|
```bash
|
|
read -rsp 'Gitea read-only package token: ' GITEA_RO_TOKEN; echo
|
|
```
|
|
|
|
Expected: no output (the value is not echoed).
|
|
|
|
If login fails with `401 Unauthorized`, the password/token or username is
|
|
wrong, or 2FA requires a token. Do not put either credential in a file, a
|
|
command that lands in shell history, or this repo.
|
|
|
|
## Step 2 — hello-world image push and pull
|
|
|
|
This proves the registry path before spending time on application images.
|
|
|
|
```bash
|
|
docker pull nginxinc/nginx-unprivileged:alpine
|
|
docker tag nginxinc/nginx-unprivileged:alpine "$REGISTRY/$OWNER/hello:gate1"
|
|
docker push "$REGISTRY/$OWNER/hello:gate1"
|
|
```
|
|
|
|
Expected: the push ends with a digest line, for example:
|
|
|
|
```text
|
|
gate1: digest: sha256:... size: ...
|
|
```
|
|
|
|
If you see `denied: requested access to the resource is denied`, the push
|
|
credential lacks `write:package` or the owner/package path is wrong. Fix that
|
|
before continuing.
|
|
|
|
Create the pull secret with the **read-only** token:
|
|
|
|
```bash
|
|
kubectl --context "$CTX" -n "$NS" create secret docker-registry gitea-regcred \
|
|
--docker-server="$REGISTRY" \
|
|
--docker-username="$OWNER" \
|
|
--docker-password="$GITEA_RO_TOKEN"
|
|
```
|
|
|
|
Expected: `secret/gitea-regcred created`. If it already exists, delete and
|
|
recreate it after reading the troubleshooting note:
|
|
`kubectl --context "$CTX" -n "$NS" delete secret gitea-regcred`.
|
|
|
|
Deploy the hello image and make the cluster pull it:
|
|
|
|
```bash
|
|
kubectl --context "$CTX" -n "$NS" create deployment hello \
|
|
--image="$REGISTRY/$OWNER/hello:gate1"
|
|
|
|
kubectl --context "$CTX" -n "$NS" patch deployment hello --type=merge -p \
|
|
'{"spec":{"template":{"spec":{"imagePullSecrets":[{"name":"gitea-regcred"}]}}}}'
|
|
|
|
kubectl --context "$CTX" -n "$NS" rollout status deployment/hello --timeout=180s
|
|
kubectl --context "$CTX" -n "$NS" get pods -l app=hello -o wide
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
deployment "hello" successfully rolled out
|
|
NAME READY STATUS RESTARTS AGE
|
|
hello-xxxxxxxxxx-xxxxx 1/1 Running 0 20s
|
|
```
|
|
|
|
**This is the first real pass/fail.** A `gitea-regcred` Secret existing is not
|
|
proof: the Gate 1 Gitea item is complete only when this hello pod is `1/1
|
|
Running` after pulling from Gitea. If the pod is not `Running`, stop.
|
|
`ImagePullBackOff` means the token, secret namespace, image tag, or registry
|
|
DNS is wrong. Diagnose with:
|
|
|
|
```bash
|
|
kubectl --context "$CTX" -n "$NS" describe pod -l app=hello
|
|
```
|
|
|
|
Keep the hello Deployment until Step 6 proves the public route. Create a
|
|
Service for it so Cloudflare has something to reach:
|
|
|
|
```bash
|
|
kubectl --context "$CTX" -n "$NS" expose deployment hello \
|
|
--port=8080 --target-port=8080
|
|
```
|
|
|
|
Expected: `service/hello exposed`.
|
|
|
|
## Step 3 — Cloudflare Tunnel
|
|
|
|
The tunnel `pulsecast-gate1` already exists. `pulse.noelsabia.dev` resolves;
|
|
`api-pulse.noelsabia.dev` resolves after its route is saved. The old
|
|
`api.pulse.noelsabia.dev` also resolves but is **not certificate-safe**: the
|
|
Universal SSL certificate is `*.noelsabia.dev` and covers only one subdomain
|
|
level. Use `api-pulse.noelsabia.dev`, not `api.pulse.noelsabia.dev`.
|
|
|
|
```bash
|
|
getent hosts pulse.noelsabia.dev
|
|
getent hosts api-pulse.noelsabia.dev || echo 'api-pulse: no DNS record yet (expected until the route exists)'
|
|
```
|
|
|
|
If DNS looks missing locally but public DNS has it, flush the local cache:
|
|
|
|
```bash
|
|
resolvectl flush-caches
|
|
```
|
|
|
|
If `getent` still misses the name while `dig +short NAME @1.1.1.1` works, force
|
|
a fresh upstream query and flush again:
|
|
|
|
```bash
|
|
resolvectl query --cache=no api-pulse.noelsabia.dev
|
|
resolvectl flush-caches
|
|
getent hosts api-pulse.noelsabia.dev
|
|
```
|
|
|
|
1. Open Cloudflare **Zero Trust → Networks → Tunnels → Create a tunnel**.
|
|
2. Connector: **Cloudflared**. Name: `pulsecast-gate1`.
|
|
3. Copy the tunnel token from the connector install command. It is the long
|
|
`eyJ...` string after `--token` (or after `service install`). It is
|
|
sensitive: never paste it into chat, a commit, or a screenshot. The tunnel
|
|
stays **Inactive** with 0 routes until the connector Deployment is running;
|
|
routes can be added while it is inactive.
|
|
4. Open the tunnel and click the **Routes** tab (not Overview, and not
|
|
**Add a replica**). Click **Add route** or **Add a public hostname**, choose
|
|
**Public hostname**, and add:
|
|
|
|
| Subdomain | Domain | Path | Service type | Service URL |
|
|
|---|---|---|---|---|
|
|
| `pulse` | `noelsabia.dev` | *(empty)* | HTTP | `hello:8080` |
|
|
| `api-pulse` | `noelsabia.dev` | *(empty)* | HTTP | `api:8080` |
|
|
|
|
The first route is temporary for Gate 1-A. For the final Gate 1-B route,
|
|
change it to `web:3000` after the web image is deployed. If you already have
|
|
the web image, you may set `web:3000` now and skip the hello route; then the
|
|
public hello check in Step 6 is not available, but the tunnel proof still
|
|
happens when the real web pod is Ready. Delete the old
|
|
`api.pulse.noelsabia.dev` route if it still exists.
|
|
|
|
5. The hostnames must be in the Cloudflare zone for `noelsabia.dev`.
|
|
Cloudflare creates the DNS records for a tunnel hostname; do **not** add
|
|
manual A/AAAA records. If the dashboard refuses to save a hostname because
|
|
a conflicting A/AAAA/CNAME already exists for that exact name, delete the
|
|
conflicting record first.
|
|
6. Do not add an Ingress. The tunnel is the only public path.
|
|
|
|
After saving both hostnames, DNS should resolve to Cloudflare:
|
|
|
|
```bash
|
|
getent hosts pulse.noelsabia.dev
|
|
getent hosts api-pulse.noelsabia.dev
|
|
```
|
|
|
|
Expected: one or more Cloudflare IPv4/IPv6 addresses. If there is still no
|
|
record, the public hostname was not saved, it is in the wrong zone, or a
|
|
conflicting record exists. Fix that in the dashboard before testing HTTPS;
|
|
`curl` cannot succeed without DNS.
|
|
|
|
Create the tunnel Secret (the token is not echoed):
|
|
|
|
```bash
|
|
read -rsp 'Cloudflare tunnel token: ' TUNNEL_TOKEN; echo
|
|
kubectl --context "$CTX" -n "$NS" create secret generic cloudflared \
|
|
--from-literal=TUNNEL_TOKEN="$TUNNEL_TOKEN"
|
|
```
|
|
|
|
Expected: `secret/cloudflared created`.
|
|
|
|
Deploy only the tunnel first. This does not need the application images:
|
|
|
|
```bash
|
|
kubectl --context "$CTX" -n "$NS" apply -f deploy/k8s/40-cloudflared.yaml
|
|
kubectl --context "$CTX" -n "$NS" rollout status deployment/cloudflared --timeout=180s
|
|
kubectl --context "$CTX" -n "$NS" logs deployment/cloudflared --tail=100 \
|
|
| grep 'Registered tunnel connection' || echo 'NO TUNNEL CONNECTION'
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
deployment.apps/cloudflared created
|
|
deployment "cloudflared" successfully rolled out
|
|
... Registered tunnel connection ...
|
|
```
|
|
|
|
(Usually four `Registered tunnel connection` lines, one per edge.)
|
|
|
|
The manifest deliberately uses `--protocol http2`: TCP/443 to Cloudflare is
|
|
verified, UDP/7844 (QUIC) is not. Do not switch to QUIC during Gate 1. If you
|
|
do test QUIC later, revert to HTTP/2 on any connection error.
|
|
|
|
If `cloudflared` is in `CrashLoopBackOff`, the token is invalid/expired or the
|
|
tunnel was deleted:
|
|
|
|
```bash
|
|
kubectl --context "$CTX" -n "$NS" logs deployment/cloudflared --previous
|
|
kubectl --context "$CTX" -n "$NS" describe pod -l app.kubernetes.io/name=cloudflared
|
|
```
|
|
|
|
## Step 4 — control-auth Secret
|
|
|
|
The web server validates `/control` and POST `/api/scenario` with
|
|
`CONTROL_USER`/`CONTROL_PASS`; web, API, and ML also need the same
|
|
`CONTROL_TOKEN` (the API forwards it as a Bearer token to FastAPI's internal
|
|
`/train` routes). Generate both locally; never commit them.
|
|
|
|
```bash
|
|
read -rsp 'Control password: ' CONTROL_PASS; echo
|
|
CONTROL_TOKEN="$(python3 -c 'import secrets; print(secrets.token_hex(24))')"
|
|
kubectl --context "$CTX" -n "$NS" create secret generic control-auth \
|
|
--from-literal=CONTROL_USER=control \
|
|
--from-literal=CONTROL_PASS="$CONTROL_PASS" \
|
|
--from-literal=CONTROL_TOKEN="$CONTROL_TOKEN"
|
|
```
|
|
|
|
Expected: `secret/control-auth created`.
|
|
|
|
The token is shared by API and ML, not the browser password. Without it
|
|
FastAPI `/train` returns 503 and the API's `/api/train` proxy cannot work.
|
|
|
|
`make secret-commands` prints all three Secret commands with placeholders; it
|
|
creates nothing.
|
|
|
|
## Step 5 — build and push the three images (Phase B)
|
|
|
|
Check whether the Dockerfiles exist:
|
|
|
|
```bash
|
|
test -f ../../web-nuxt.git/main/Dockerfile && echo web-ok || echo web-blocked
|
|
test -f ../../api-go.git/main/Dockerfile && echo api-ok || echo api-blocked
|
|
test -f ../../ML-py.git/main/Dockerfile && echo ml-ok || echo ml-blocked
|
|
```
|
|
|
|
Expected **now** (all three Dockerfiles landed on 2026-11-14):
|
|
|
|
```text
|
|
web-ok
|
|
api-ok
|
|
ml-ok
|
|
```
|
|
|
|
Build all three:
|
|
|
|
```bash
|
|
make build
|
|
```
|
|
|
|
All three v1.1 images were verified locally with the manifest's runtime
|
|
constraints:
|
|
|
|
- API: distroless `static:nonroot`, `--user 65532:65532`, read-only root,
|
|
`/tmp` tmpfs; `/api/healthz` and `/api/readyz` returned 200, and the API
|
|
loaded ML `/bootstrap` into memory.
|
|
- ML: image `USER 10001:10001`; with read-only root and `/tmp` tmpfs,
|
|
`/healthz` and `/readyz` returned 200, `/bootstrap` returned 8 sensors, and
|
|
`/meta` showed `mae=3.32`, `wape=27.5`, `r2=0.777`.
|
|
- Web: image `USER 10001:10001`; with read-only root and `/tmp` tmpfs, `/`,
|
|
`/stage`, and `/live` returned 200; `/control` returned 401 anonymous and
|
|
200 with control auth; the same-origin SSE proxy emitted events.
|
|
- v1.1 was pushed to Gitea. The cluster is not updated until this branch is
|
|
reviewed/merged and explicitly applied.
|
|
|
|
Push:
|
|
|
|
```bash
|
|
make push
|
|
```
|
|
|
|
Expected: each `docker push` ends with a digest. If authentication expired,
|
|
repeat the push login from Step 1.
|
|
|
|
The Makefile auto-detects the service worktrees:
|
|
|
|
```text
|
|
../../web-nuxt.git/main
|
|
../../api-go.git/main
|
|
../../ML-py.git/main
|
|
```
|
|
|
|
Override when the layout differs:
|
|
|
|
```bash
|
|
make build WEB_DIR=/path/to/web-nuxt API_DIR=/path/to/api-go ML_DIR=/path/to/ML-py
|
|
```
|
|
|
|
If `OWNER` or `TAG` differs from the manifests, update `deploy/k8s/*.yaml`
|
|
before `make deploy`, or deploy first and then run the guarded image update:
|
|
|
|
```bash
|
|
make set-image CONFIRM=yes OWNER=<owner> TAG=<tag>
|
|
```
|
|
|
|
### Dockerfile contract (coordinate through the orchestrator)
|
|
|
|
Each service image must:
|
|
|
|
- listen on `0.0.0.0` at the port below;
|
|
- run as a **non-root numeric UID** (no named user, no shell required);
|
|
- write only to `/tmp` (the pod root filesystem is read-only);
|
|
- start with the environment in `deploy/k8s/00-config.yaml`;
|
|
- be self-contained: no runtime download of data, model, fonts, or packages.
|
|
|
|
| Service | Image | Port | UID:GID | Health paths |
|
|
|---|---|---|---|---|
|
|
| web | `git.noelsabia.dev/noel/pulsecast-web:v1.1` | 3000 | `10001:10001` | `/`, `/stage`, `/live`, `/control` |
|
|
| api | `git.noelsabia.dev/noel/pulsecast-api:v1.1` | 8080 | `65532:65532` | `/api/healthz`, `/api/readyz` |
|
|
| ml | `git.noelsabia.dev/noel/pulsecast-ml:v1.1` | 8000 | `10001:10001` | `/healthz`, `/readyz`, `/bootstrap` |
|
|
|
|
The manifests set `runAsUser`/`runAsGroup`/`fsGroup` explicitly, so the image's
|
|
named user is irrelevant; the files must be readable by that numeric UID.
|
|
`api` uses distroless `static:nonroot` (65532); web and ml use 10001. If a
|
|
Dockerfile genuinely needs another UID, change the manifest and this table
|
|
together, and keep the value numeric.
|
|
|
|
`api` v1.1 reads `FASTAPI_URL=http://ml:8000`, `BOOTSTRAP_REFRESH=60s`,
|
|
`BOOTSTRAP_TIMEOUT`, `PORT`, `CONTROL_USER`, `CONTROL_PASS`, `CONTROL_TOKEN`,
|
|
`SSE_HEARTBEAT`, and `ML_TIMEOUT`. It has **no baked data and no `DATA_DIR`**;
|
|
it fetches ML `/bootstrap` into memory and refreshes it.
|
|
`web` proxies Go and SSE server-side using `NUXT_API_INTERNAL_BASE=http://api:8080`;
|
|
it also needs `CONTROL_USER`, `CONTROL_PASS`, and `CONTROL_TOKEN` to protect
|
|
`/control` and POST `/api/scenario`. `NUXT_PUBLIC_API_BASE` and the public SSE
|
|
URL are display/future configuration; the browser calls the Nuxt origin.
|
|
`ml` reads `PULSECAST_ROOT`, `PULSECAST_ARTIFACTS`, `PULSECAST_WEATHER`,
|
|
`PULSECAST_MODELS`, `PULSECAST_UPLOADS`, and `CONTROL_TOKEN`; its image CMD
|
|
binds `0.0.0.0:8000`. The env names in the manifests are the integration
|
|
point: change them here first if a service needs a different name. All three
|
|
images use numeric UIDs: web/ml `10001:10001`, API `65532:65532`.
|
|
|
|
## Step 6 — prove public HTTPS with hello
|
|
|
|
Do this **before** deleting hello. It proves tunnel + DNS + TLS without
|
|
depending on web/ml.
|
|
|
|
```bash
|
|
curl -fsS -o /dev/null -w 'hello %{http_code}\n' https://pulse.noelsabia.dev/
|
|
```
|
|
|
|
Expected: `hello 200`.
|
|
|
|
If you get `502`, `530`, or `1033`, the edge cannot reach the Service. Check:
|
|
|
|
```bash
|
|
kubectl --context "$CTX" -n "$NS" get endpoints hello
|
|
kubectl --context "$CTX" -n "$NS" get svc hello
|
|
kubectl --context "$CTX" -n "$NS" logs deployment/cloudflared --tail=50
|
|
```
|
|
|
|
The hostname route must be `http://hello:8080` and the hello Service must have
|
|
an endpoint. Fix the dashboard route or the Service; do not add an Ingress.
|
|
|
|
After the public hello check passes, delete it so it cannot mask a web problem:
|
|
|
|
```bash
|
|
kubectl --context "$CTX" -n "$NS" delete service hello
|
|
kubectl --context "$CTX" -n "$NS" delete deployment hello
|
|
```
|
|
|
|
Expected: `service "hello" deleted` and `deployment.apps "hello" deleted`.
|
|
|
|
`https://pulse.noelsabia.dev` will now return an edge error until the real web
|
|
Deployment is Ready. That is expected; do not "fix" it with an Ingress.
|
|
|
|
## Step 7 — validate and check readiness again
|
|
|
|
```bash
|
|
make validate
|
|
make gate1-check
|
|
```
|
|
|
|
Expected once Secrets and Deployments exist:
|
|
|
|
- `make validate`: 11 resources created (dry run);
|
|
- `make gate1-check`: 0 hard failures, 0 pending, `RESULT: READY for Gate 1`
|
|
(exit 0).
|
|
|
|
Before the deploy, pending Deployments keep it at exit 2. That is fine.
|
|
|
|
## Step 8 — deploy
|
|
|
|
```bash
|
|
make deploy-check
|
|
make deploy CONFIRM=yes
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
required Secrets and keys present
|
|
... created/configured ...
|
|
deployment "ml" successfully rolled out
|
|
deployment "api" successfully rolled out
|
|
deployment "web" successfully rolled out
|
|
deployment "cloudflared" successfully rolled out
|
|
```
|
|
|
|
`make deploy` fails closed: without `CONFIRM=yes` it changes nothing. It does
|
|
not create the namespace or Secrets. It applies all 11 manifests, including
|
|
the ConfigMaps.
|
|
|
|
If the real images do not exist, the rollouts fail with `ImagePullBackOff`.
|
|
Stop and finish Step 5; do not deploy a placeholder under `web`, `api`, or
|
|
`ml`.
|
|
|
|
Check the state:
|
|
|
|
```bash
|
|
make status
|
|
kubectl --context "$CTX" -n "$NS" get pods -o wide
|
|
kubectl --context "$CTX" -n "$NS" logs deployment/cloudflared --tail=50 \
|
|
| grep 'Registered tunnel connection' || echo 'NO TUNNEL CONNECTION'
|
|
```
|
|
|
|
Expected: `web`, `api`, `ml`, and `cloudflared` each `1/1`; Services `web`,
|
|
`api`, and `ml` are `ClusterIP`; `cloudflared` has no Service.
|
|
|
|
### Re-pushing the same tag
|
|
|
|
The manifests use `imagePullPolicy: Always` while the tag stays `v1.1`. After
|
|
rebuilding and pushing the same tag, force a re-pull:
|
|
|
|
```bash
|
|
make restart CONFIRM=yes
|
|
```
|
|
|
|
This restarts `web`, `api`, and `ml` and waits for the rollout. Do not use it
|
|
during the demo window unless you are fixing a blocker.
|
|
|
|
## Step 9 — verify public HTTPS and SSE
|
|
|
|
Full smoke, including the ML service through a short `kubectl port-forward`:
|
|
|
|
```bash
|
|
make smoke SMOKE_SSE=1
|
|
```
|
|
|
|
Expected: each check prints `PASS`, the summary ends with `0 failed`, and TLS
|
|
verification passes (`ssl_verify_result=0`). If the ML port-forward cannot
|
|
start, set `ML_URL` or `SKIP_ML=1`; do not silently drop the ML check from the
|
|
sign-off.
|
|
|
|
Manual public checks (also run from a phone on mobile data):
|
|
|
|
```bash
|
|
curl -fsS -o /dev/null -w 'web %{http_code}\n' https://pulse.noelsabia.dev/
|
|
curl -fsS -o /dev/null -w 'api %{http_code}\n' https://api-pulse.noelsabia.dev/api/healthz
|
|
curl -fsS -o /dev/null -w 'ready %{http_code}\n' https://api-pulse.noelsabia.dev/api/readyz
|
|
curl -fsS https://api-pulse.noelsabia.dev/api/meta | jq '{data_version,model_version}'
|
|
curl -sS -N --max-time 20 https://api-pulse.noelsabia.dev/api/stream | head -20
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
web 200
|
|
api 200
|
|
ready 200
|
|
{
|
|
"data_version": "...",
|
|
"model_version": "..."
|
|
}
|
|
retry: 3000
|
|
|
|
event: scenario
|
|
id: ...
|
|
data: {...}
|
|
|
|
event: ping
|
|
data: {"t":"..."}
|
|
```
|
|
|
|
The SSE output must start with `retry: 3000`, contain a `scenario` event
|
|
immediately, and show a `ping` within 20 seconds. If events are delayed
|
|
through Cloudflare, see the SSE row in Troubleshooting.
|
|
|
|
### v1.1 product checks
|
|
|
|
```bash
|
|
curl -fsS https://api-pulse.noelsabia.dev/api/sensors | jq 'length'
|
|
curl -fsS https://api-pulse.noelsabia.dev/api/meta \
|
|
| jq '{data_version,model_version,metrics,cache}'
|
|
curl -fsS -o /dev/null -w 'stage %{http_code}\n' https://pulse.noelsabia.dev/stage
|
|
curl -fsS -o /dev/null -w 'live %{http_code}\n' https://pulse.noelsabia.dev/live
|
|
curl -sS -o /dev/null -w 'control anon %{http_code}\n' https://pulse.noelsabia.dev/control
|
|
set -a; source ~/.config/pulsecast/control-auth.env; set +a
|
|
curl -sS -u "$CONTROL_USER:$CONTROL_PASS" -o /dev/null \
|
|
-w 'control auth %{http_code}\n' https://pulse.noelsabia.dev/control
|
|
curl -sS -o /dev/null -w 'plan %{time_total}s\n' \
|
|
'https://api-pulse.noelsabia.dev/api/staffing?sensor=hn-stadtgalerie-01&scenario=base&standard=standard'
|
|
```
|
|
|
|
Expected: **8** sensors; metrics `mae=3.32`, `wape=27.5`, `r2=0.777` with a
|
|
`cache` object; `/stage` and `/live` 200; `/control` 401 anonymous and 200
|
|
with control auth; plan under 2 seconds.
|
|
|
|
In-cluster connectivity (optional; **creates and removes one short-lived
|
|
pod**, so the human runs it):
|
|
|
|
```bash
|
|
kubectl --context "$CTX" -n "$NS" run gate1-netcheck --rm -i --restart=Never \
|
|
--image=curlimages/curl:latest \
|
|
--overrides='{"spec":{"securityContext":{"runAsNonRoot":true,"runAsUser":1000,"runAsGroup":1000}}}' \
|
|
--command -- sh -c \
|
|
'curl -fsS http://api:8080/api/healthz && \
|
|
curl -fsS http://ml:8000/healthz && \
|
|
curl -fsS http://web:3000/ >/dev/null && echo connectivity-ok'
|
|
```
|
|
|
|
Expected: `{"ok":true}{"ok":true}connectivity-ok` (order may vary with the
|
|
shell output). If the namespace's Pod Security Admission blocks this ad-hoc
|
|
pod, skip it and use `kubectl exec deploy/web -- ...` or the smoke test; do not
|
|
weaken the application Deployments to make a diagnostic pod run.
|
|
|
|
Finally, confirm the checker is green end-to-end:
|
|
|
|
```bash
|
|
make gate1-check CHECK_PUBLIC=1
|
|
```
|
|
|
|
Expected: `RESULT: READY for Gate 1` (exit 0).
|
|
|
|
## Step 10 — record the Gate 1 result
|
|
|
|
Record in the status note, README, or commit message:
|
|
|
|
```bash
|
|
date -u +'%Y-%m-%dT%H:%M:%SZ'
|
|
git rev-parse HEAD
|
|
kubectl --context "$CTX" -n "$NS" get deploy -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.template.spec.containers[0].image}{"\n"}{end}'
|
|
```
|
|
|
|
Also save:
|
|
|
|
- the output of `make smoke SMOKE_SSE=1`;
|
|
- the public URLs and the device/network used (phone on mobile data);
|
|
- any warning that remains.
|
|
|
|
Image digests are in the `docker push` output; tags alone are not enough if
|
|
you need to reproduce the exact deploy.
|
|
|
|
## Troubleshooting
|
|
|
|
| Symptom | Check | Fix |
|
|
|---|---|---|
|
|
| `docker login` → `401 Unauthorized` | Username and credential; 2FA | Use a token with `write:package` (not the read-only token), or the account password if 2FA is off. |
|
|
| `docker push` → `denied: requested access to the resource is denied` | Push credential scope; owner/package path | Use a `write:package` token; make sure `$OWNER` matches the Gitea image owner. |
|
|
| `ImagePullBackOff` / `ErrImagePull` | `kubectl -n "$NS" describe pod -l app=hello` (and the real pod); `kubectl -n "$NS" get secret gitea-regcred` | The token must have `read:package`; the Secret must be in `fch26-onemanarmy`; the tag must exist in Gitea. Do not print the secret data. |
|
|
| `CreateContainerConfigError` | `kubectl -n "$NS" describe pod <pod>` | A referenced Secret key is missing (`control-auth`: `CONTROL_USER`, `CONTROL_PASS`, `CONTROL_TOKEN`) or the `cloudflared` Secret lacks `TUNNEL_TOKEN`. Recreate the Secret. |
|
|
| `container has runAsNonRoot and image will run as root` or a non-numeric-user error | `kubectl -n "$NS" get pod <pod> -o yaml`; inspect `securityContext` | Keep `runAsNonRoot: true` and set a positive numeric `runAsUser`/`runAsGroup` (SKE caveat). The manifests already do; do not "fix" this by disabling `runAsNonRoot`. |
|
|
| Public hostname has no DNS record | `getent hosts pulse.noelsabia.dev` / `getent hosts api-pulse.noelsabia.dev` | Save the public hostname in the tunnel dashboard in the `noelsabia.dev` zone; remove a conflicting A/AAAA/CNAME for that exact name. Do not add manual records for a tunnel hostname. |
|
|
| `cloudflared` → `CrashLoopBackOff` | `kubectl -n "$NS" logs deployment/cloudflared --previous` | Invalid/expired tunnel token, or the tunnel was deleted in Cloudflare. Recreate the Secret and restart. |
|
|
| `cloudflared` logs QUIC/timeout errors after you changed the protocol | `kubectl -n "$NS" logs deployment/cloudflared --tail=50` | UDP/7844 is unverified on SKE. Put `--protocol http2` back (the committed manifest already sets it); TCP/443 is confirmed. |
|
|
| Public `502` / `1033` / `530` | `kubectl -n "$NS" get endpoints web api hello`; cloudflared logs | No ready endpoints, wrong Service port, or the hostname route points at the wrong service. Fix the dashboard route or the Service. Never add an Ingress. |
|
|
| `api` pod not ready | `kubectl -n "$NS" logs deployment/api`; `curl -fsS http://ml:8000/bootstrap` from inside the namespace; `curl -fsS https://api-pulse.noelsabia.dev/api/readyz` | v1.1 has no baked data. It needs ML reachable at `FASTAPI_URL` and a successful `/bootstrap`; check `BOOTSTRAP_REFRESH` and the ML readiness. The distroless API image has no shell, so do not rely on `kubectl exec ... ls`. |
|
|
| `ml` pod stuck in startup | `kubectl -n "$NS" logs deployment/ml` | Model load exceeds the 120 s startup budget or `MODEL_PATH` is wrong. Fix the image/path. |
|
|
| `web` 500 | `kubectl -n "$NS" logs deployment/web` | Missing runtime API base or SSR error. Rebuild with `NUXT_PUBLIC_API_BASE` or fix the ConfigMap. |
|
|
| SSE events arrive only after a delay | `curl -N` direct to the pod vs through Cloudflare | Cloudflare buffering. Keep the 15 s heartbeat and the web polling fallback (`/api/scenario` every 5 s). Do not remove the heartbeat. |
|
|
| `make gate1-check` exits 2 | Output lists `[pending]` Secrets/Deployments | Complete the human steps above. Exit 2 is not a manifest failure. |
|
|
| `make deploy` refuses | `CONFIRM` is empty | Re-run `make deploy CONFIRM=yes`; this is the safety guard. |
|
|
| `kubectl` → `Forbidden` | `kubectl config current-context` | Use `--context "$CTX"`. Do not create Roles/Ingress/Jobs as a workaround; escalate instead. |
|
|
| `make validate` fails | Dry-run output | Fix the manifest; do not apply it to see whether it works. |
|
|
|
|
## Rollback
|
|
|
|
No cluster-scoped resources are created, so rollback is namespace-local:
|
|
|
|
```bash
|
|
kubectl --context "$CTX" -n "$NS" rollout undo deployment/web
|
|
kubectl --context "$CTX" -n "$NS" rollout undo deployment/api
|
|
kubectl --context "$CTX" -n "$NS" rollout undo deployment/ml
|
|
kubectl --context "$CTX" -n "$NS" rollout undo deployment/cloudflared
|
|
kubectl --context "$CTX" -n "$NS" rollout status deployment/web --timeout=180s
|
|
```
|
|
|
|
If the tunnel token or route is wrong, fix the Secret / Cloudflare dashboard
|
|
and restart `cloudflared`; do not create an Ingress as a workaround. Delete any
|
|
leftover hello resources:
|
|
|
|
```bash
|
|
kubectl --context "$CTX" -n "$NS" delete service hello --ignore-not-found
|
|
kubectl --context "$CTX" -n "$NS" delete deployment hello --ignore-not-found
|
|
```
|
|
|
|
## Security notes
|
|
|
|
- Never commit the kubeconfig, tunnel token, control password, Gitea token, or
|
|
any `.env` file. `.gitignore` covers the usual filenames; check
|
|
`git status` before every commit.
|
|
- The kubeconfig must stay at mode `600` and outside the repos. It was shared
|
|
in chat: rotate/revoke the service-account token after the event.
|
|
- Rotate the Gitea package token and the control password after the event.
|
|
- Use the read-only package token only for `gitea-regcred`; use a separate
|
|
write credential for pushes.
|
|
- `make smoke` and `make gate1-check` print no secrets; do not add `set -x` to
|
|
scripts while a live `KUBECONFIG` or token is exported.
|
|
- Stop the tunnel and rotate the Gitea token if either leaks. Deleting the
|
|
tunnel in Cloudflare invalidates its token.
|