feat: prepare v1.1 deploy and presenter load test
infra-ci / manifests (push) Failing after 4s
infra-ci / manifests (pull_request) Failing after 3s

- build and push v1.1 images from the service main worktrees
- switch manifests to :v1.1; add web control Secret refs
- configure API BOOTSTRAP_REFRESH=60s and remove unused DATA_DIR
- add k6 presenter scenario with a mid-run control POST
- require control credentials in make load; update runbook and drills
This commit is contained in:
Noel Sabia committed 2026-11-14 08:00:58 +01:00
1 parent 40bc8e47d0
commit b98a1bab42
12 files changed
+264 -117

No files matched your search

+75 -54
View File
@@ -24,7 +24,7 @@ Gate 1 passes only when **all** of these are true:
Do not claim Gate 1 from a local `kubectl port-forward`, a green manifest
dry-run, or a container that only runs on your laptop.
## Current external status (2026-11-14 ~06:50 CET)
## Current external status (2026-11-14 ~08:00 CET)
Re-check before trusting this snapshot:
@@ -33,8 +33,8 @@ Re-check before trusting this snapshot:
Open-Meteo, and `git.noelsabia.dev`.
- **Tunnel active:** `pulsecast-gate1` has routes for `pulse.noelsabia.dev`
and `api-pulse.noelsabia.dev`; `cloudflared` is `1/1` with four HTTP/2
connections, and `https://pulse.noelsabia.dev` returns 200 from hello with
valid TLS. `api-pulse` returns 502 until the API is deployed.
connections. `api-pulse` serves the deployed API; `pulse` still points at
the temporary hello Service until the web cutover.
- **DNS works:** public resolvers return Cloudflare IPs for
`pulse.noelsabia.dev` and `api-pulse.noelsabia.dev`. A local
`systemd-resolved` negative cache can be cleared with
@@ -42,20 +42,19 @@ Re-check before trusting this snapshot:
- **API hostname is `api-pulse.noelsabia.dev`:** Universal SSL covers only
`*.noelsabia.dev`; a two-level `api.pulse.noelsabia.dev` is not covered by
that certificate.
- **Gitea pull is proven (2026-11-14 ~06:16 CET):** `gitea-regcred` exists and
the cluster pulled `git.noelsabia.dev/noel/hello:gate1` in 120 ms; the hello
pod is `1/1 Running` and its Service returned HTTP 200. The Gitea owner is
`noel`.
- **Gitea pull is proven:** `gitea-regcred` exists; hello and the v1.0 service
images pulled from Gitea. The Gitea owner is `noel`.
- **v1.1 images pushed:** web/api/ml `:v1.1` were built from the service
`main` worktrees, locally exercised under numeric UIDs/read-only root, and
pushed. The cluster still runs the previous v1.0 images until the v1.1
branch is reviewed/merged and explicitly applied.
- **All three Secrets exist:** `gitea-regcred`, `cloudflared`, `control-auth`.
- **QUIC/UDP 7844 is untested.** TCP/443 to Cloudflare works, so the
`cloudflared` manifest uses `--protocol http2` instead of relying on
unverified UDP.
- **Numeric UIDs are required:** a non-numeric image user breaks
`runAsNonRoot` on SKE (reproduced with `curlimages/curl`). Every pod in
`deploy/k8s/` sets explicit numeric `runAsUser`/`runAsGroup`/`fsGroup`.
- **Repo status (2026-11-14 ~06:50 CET):** all three images build and pass
local manifest-equivalent runtime checks (numeric UID, read-only root, `/tmp`
tmpfs). Remaining steps are the image push, `control-auth`, deploy, and
public smoke.
## What is possible now vs what is blocked
@@ -63,12 +62,10 @@ Re-check before trusting this snapshot:
|---|---|---|
| Read-only cluster/manifest check | agent or human | **ready now** (`make gate1-check`) |
| Gitea push login + hello-world push | human | **done**; hello pull and Service verified |
| `gitea-regcred`, `cloudflared`, `control-auth` Secrets | human | `gitea-regcred` + `cloudflared` **done**; `control-auth` pending |
| Cloudflare Tunnel + public hostnames/DNS | human | **done**; tunnel active, public hello 200 |
| `api` image | human | **ready** — `make build-api` + manifest-equivalent run verified locally |
| `ml` image | human | **ready** — `make build-ml` + manifest-equivalent run verified locally |
| `web` image | human | **ready** — `make build-web` + manifest-equivalent run verified locally |
| Full deploy + smoke + Gate 1 sign-off | human | **ready once the tunnel/secret/image steps are done** |
| `gitea-regcred`, `cloudflared`, `control-auth` Secrets | human | **done** |
| Cloudflare Tunnel + public hostnames/DNS | human | **done**; tunnel active; `pulse` still points at hello |
| `api`, `ml`, `web` images | human | **v1.1 pushed**; local manifest-equivalent checks pass |
| v1.1 apply + smoke + presenter k6 + drills | human | **blocked on branch review/merge** |
`make build` and `make push` cover all three services; no Dockerfile is
missing now. Do not deploy placeholder images under the real service names.
@@ -86,7 +83,7 @@ export NS="fch26-onemanarmy"
export CTX="fch26-onemanarmy-sa-context"
export REGISTRY="git.noelsabia.dev"
export OWNER="noel"
export TAG="gate1"
export TAG="v1.1"
chmod 600 "$KUBECONFIG"
```
@@ -147,7 +144,7 @@ Run top to bottom. Stop at the first failure and use
- [ ] 5. `cloudflared` Secret created; logs show `Registered tunnel connection`.
- [ ] 6. Public hello route returns HTTP 200 (proves tunnel + DNS + TLS).
- [ ] 7. `control-auth` Secret created.
- [ ] 8. All three images built, tagged `gate1`, and pushed.
- [ ] 8. All three v1.1 images built and pushed.
- [ ] 9. `make validate` and `make gate1-check` pass.
- [ ] 10. `make deploy CONFIRM=yes` completes; all four Deployments are Ready.
- [ ] 11. `make smoke SMOKE_SSE=1` passes.
@@ -411,10 +408,10 @@ kubectl --context "$CTX" -n "$NS" describe pod -l app.kubernetes.io/name=cloudfl
## Step 4 — control-auth Secret
The API reads `CONTROL_USER` and `CONTROL_PASS` for browser control auth.
Both API and ML also need the same `CONTROL_TOKEN`: the API forwards it as a
Bearer token to FastAPI's internal `/train` routes. Generate both locally;
never commit them.
The web server validates `/control` and POST `/api/scenario` with
`CONTROL_USER`/`CONTROL_PASS`; web, API, and ML also need the same
`CONTROL_TOKEN` (the API forwards it as a Bearer token to FastAPI's internal
`/train` routes). Generate both locally; never commit them.
```bash
read -rsp 'Control password: ' CONTROL_PASS; echo
@@ -438,9 +435,9 @@ creates nothing.
Check whether the Dockerfiles exist:
```bash
test -f ../../web-nuxt.git/feat-web-bootstrap/Dockerfile && echo web-ok || echo web-blocked
test -f ../../api-go.git/feat-api-bootstrap/Dockerfile && echo api-ok || echo api-blocked
test -f ../../ML-py.git/feat-ml-bootstrap/Dockerfile && echo ml-ok || echo ml-blocked
test -f ../../web-nuxt.git/main/Dockerfile && echo web-ok || echo web-blocked
test -f ../../api-go.git/main/Dockerfile && echo api-ok || echo api-blocked
test -f ../../ML-py.git/main/Dockerfile && echo ml-ok || echo ml-blocked
```
Expected **now** (all three Dockerfiles landed on 2026-11-14):
@@ -457,18 +454,20 @@ Build all three:
make build
```
All three images were verified locally with the manifest's runtime
All three v1.1 images were verified locally with the manifest's runtime
constraints:
- API: distroless `static:nonroot`, `--user 65532:65532`, read-only root,
`/tmp` tmpfs; `/api/healthz` and `/api/readyz` returned 200.
- ML: image `USER app` is non-numeric, but the manifest overrides it with
`runAsUser: 10001`; with read-only root and `/tmp` tmpfs, `/healthz` and
`/readyz` returned 200 and `/meta` loaded the baked model.
- Web: image `USER node` is non-numeric, but the manifest overrides it with
`runAsUser: 10001`; with read-only root and `/tmp` tmpfs, `/` returned
HTTP 200 and the runtime `NUXT_PUBLIC_API_BASE` appeared in the HTML.
- No image was pushed or deployed by the infra agent.
`/tmp` tmpfs; `/api/healthz` and `/api/readyz` returned 200, and the API
loaded ML `/bootstrap` into memory.
- ML: image `USER 10001:10001`; with read-only root and `/tmp` tmpfs,
`/healthz` and `/readyz` returned 200, `/bootstrap` returned 8 sensors, and
`/meta` showed `mae=3.32`, `wape=27.5`, `r2=0.777`.
- Web: image `USER 10001:10001`; with read-only root and `/tmp` tmpfs, `/`,
`/stage`, and `/live` returned 200; `/control` returned 401 anonymous and
200 with control auth; the same-origin SSE proxy emitted events.
- v1.1 was pushed to Gitea. The cluster is not updated until this branch is
reviewed/merged and explicitly applied.
Push:
@@ -482,9 +481,9 @@ repeat the push login from Step 1.
The Makefile auto-detects the service worktrees:
```text
../../web-nuxt.git/feat-web-bootstrap
../../api-go.git/feat-api-bootstrap
../../ML-py.git/feat-ml-bootstrap
../../web-nuxt.git/main
../../api-go.git/main
../../ML-py.git/main
```
Override when the layout differs:
@@ -512,9 +511,9 @@ Each service image must:
| Service | Image | Port | UID:GID | Health paths |
|---|---|---|---|---|
| web | `git.noelsabia.dev/noel/pulsecast-web:gate1` | 3000 | `10001:10001` | `/` |
| api | `git.noelsabia.dev/noel/pulsecast-api:gate1` | 8080 | `65532:65532` | `/api/healthz`, `/api/readyz` |
| ml | `git.noelsabia.dev/noel/pulsecast-ml:gate1` | 8000 | `10001:10001` | `/healthz`, `/readyz` |
| web | `git.noelsabia.dev/noel/pulsecast-web:v1.1` | 3000 | `10001:10001` | `/`, `/stage`, `/live`, `/control` |
| api | `git.noelsabia.dev/noel/pulsecast-api:v1.1` | 8080 | `65532:65532` | `/api/healthz`, `/api/readyz` |
| ml | `git.noelsabia.dev/noel/pulsecast-ml:v1.1` | 8000 | `10001:10001` | `/healthz`, `/readyz`, `/bootstrap` |
The manifests set `runAsUser`/`runAsGroup`/`fsGroup` explicitly, so the image's
named user is irrelevant; the files must be readable by that numeric UID.
@@ -522,17 +521,19 @@ named user is irrelevant; the files must be readable by that numeric UID.
Dockerfile genuinely needs another UID, change the manifest and this table
together, and keep the value numeric.
`api` must read `DATA_DIR` (baked JSON), `FASTAPI_URL`, `PORT`,
`CONTROL_USER`, `CONTROL_PASS`, `CONTROL_TOKEN`, `SSE_HEARTBEAT`, and
`ML_TIMEOUT`.
`web` must have `NUXT_PUBLIC_API_BASE=https://api-pulse.noelsabia.dev` baked in
or read it at runtime from the ConfigMap; the browser must use the public API
hostname. `ml` reads `PULSECAST_ROOT`, `PULSECAST_ARTIFACTS`,
`PULSECAST_WEATHER`, `PULSECAST_MODELS`, `PULSECAST_UPLOADS`, and
`CONTROL_TOKEN`; its image CMD binds `0.0.0.0:8000`. The env names in the
manifests are the integration point: change them here first if a service needs
a different name. The ML image currently uses a named `app` user; the manifest
overrides it with numeric UID 10001, which was verified locally.
`api` v1.1 reads `FASTAPI_URL=http://ml:8000`, `BOOTSTRAP_REFRESH=60s`,
`BOOTSTRAP_TIMEOUT`, `PORT`, `CONTROL_USER`, `CONTROL_PASS`, `CONTROL_TOKEN`,
`SSE_HEARTBEAT`, and `ML_TIMEOUT`. It has **no baked data and no `DATA_DIR`**;
it fetches ML `/bootstrap` into memory and refreshes it.
`web` proxies Go and SSE server-side using `NUXT_API_INTERNAL_BASE=http://api:8080`;
it also needs `CONTROL_USER`, `CONTROL_PASS`, and `CONTROL_TOKEN` to protect
`/control` and POST `/api/scenario`. `NUXT_PUBLIC_API_BASE` and the public SSE
URL are display/future configuration; the browser calls the Nuxt origin.
`ml` reads `PULSECAST_ROOT`, `PULSECAST_ARTIFACTS`, `PULSECAST_WEATHER`,
`PULSECAST_MODELS`, `PULSECAST_UPLOADS`, and `CONTROL_TOKEN`; its image CMD
binds `0.0.0.0:8000`. The env names in the manifests are the integration
point: change them here first if a service needs a different name. All three
images use numeric UIDs: web/ml `10001:10001`, API `65532:65532`.
## Step 6 — prove public HTTPS with hello
@@ -623,7 +624,7 @@ Expected: `web`, `api`, `ml`, and `cloudflared` each `1/1`; Services `web`,
### Re-pushing the same tag
The manifests use `imagePullPolicy: Always` while the tag stays `gate1`. After
The manifests use `imagePullPolicy: Always` while the tag stays `v1.1`. After
rebuilding and pushing the same tag, force a re-pull:
```bash
@@ -680,6 +681,26 @@ The SSE output must start with `retry: 3000`, contain a `scenario` event
immediately, and show a `ping` within 20 seconds. If events are delayed
through Cloudflare, see the SSE row in Troubleshooting.
### v1.1 product checks
```bash
curl -fsS https://api-pulse.noelsabia.dev/api/sensors | jq 'length'
curl -fsS https://api-pulse.noelsabia.dev/api/meta \
| jq '{data_version,model_version,metrics,cache}'
curl -fsS -o /dev/null -w 'stage %{http_code}\n' https://pulse.noelsabia.dev/stage
curl -fsS -o /dev/null -w 'live %{http_code}\n' https://pulse.noelsabia.dev/live
curl -sS -o /dev/null -w 'control anon %{http_code}\n' https://pulse.noelsabia.dev/control
set -a; source ~/.config/pulsecast/control-auth.env; set +a
curl -sS -u "$CONTROL_USER:$CONTROL_PASS" -o /dev/null \
-w 'control auth %{http_code}\n' https://pulse.noelsabia.dev/control
curl -sS -o /dev/null -w 'plan %{time_total}s\n' \
'https://api-pulse.noelsabia.dev/api/staffing?sensor=hn-stadtgalerie-01&scenario=base&standard=standard'
```
Expected: **8** sensors; metrics `mae=3.32`, `wape=27.5`, `r2=0.777` with a
`cache` object; `/stage` and `/live` 200; `/control` 401 anonymous and 200
with control auth; plan under 2 seconds.
In-cluster connectivity (optional; **creates and removes one short-lived
pod**, so the human runs it):
@@ -738,7 +759,7 @@ you need to reproduce the exact deploy.
| `cloudflared` → `CrashLoopBackOff` | `kubectl -n "$NS" logs deployment/cloudflared --previous` | Invalid/expired tunnel token, or the tunnel was deleted in Cloudflare. Recreate the Secret and restart. |
| `cloudflared` logs QUIC/timeout errors after you changed the protocol | `kubectl -n "$NS" logs deployment/cloudflared --tail=50` | UDP/7844 is unverified on SKE. Put `--protocol http2` back (the committed manifest already sets it); TCP/443 is confirmed. |
| Public `502` / `1033` / `530` | `kubectl -n "$NS" get endpoints web api hello`; cloudflared logs | No ready endpoints, wrong Service port, or the hostname route points at the wrong service. Fix the dashboard route or the Service. Never add an Ingress. |
| `api` pod not ready | `kubectl -n "$NS" logs deployment/api`; `curl -fsS https://api-pulse.noelsabia.dev/api/readyz` | Baked JSON missing from the image or `DATA_DIR` wrong. Rebuild the API image with the artifacts. The distroless API image has no shell, so do not rely on `kubectl exec ... ls`. |
| `api` pod not ready | `kubectl -n "$NS" logs deployment/api`; `curl -fsS http://ml:8000/bootstrap` from inside the namespace; `curl -fsS https://api-pulse.noelsabia.dev/api/readyz` | v1.1 has no baked data. It needs ML reachable at `FASTAPI_URL` and a successful `/bootstrap`; check `BOOTSTRAP_REFRESH` and the ML readiness. The distroless API image has no shell, so do not rely on `kubectl exec ... ls`. |
| `ml` pod stuck in startup | `kubectl -n "$NS" logs deployment/ml` | Model load exceeds the 120 s startup budget or `MODEL_PATH` is wrong. Fix the image/path. |
| `web` 500 | `kubectl -n "$NS" logs deployment/web` | Missing runtime API base or SSR error. Rebuild with `NUXT_PUBLIC_API_BASE` or fix the ConfigMap. |
| SSE events arrive only after a delay | `curl -N` direct to the pod vs through Cloudflare | Cloudflare buffering. Keep the 15 s heartbeat and the web polling fallback (`/api/scenario` every 5 s). Do not remove the heartbeat. |