ci: run manifest checks on the Gitea runner
The node:20-bookworm runner has python3 but no pip, so the PyYAML install failed before any check ran. Install python3-yaml with apt instead, remove the duplicate cluster-only kubectl step, and keep the two cluster-free Make targets. Also remove the stale v1.0/pending-review claims, drop the old api.pulse hostname, and move the deploy status below the runbook.
This commit is contained in:
1 parent
bd3176bef0
commit
e8e048866c
10 files changed
+102
-104
No files matched your search
+62
-61
@@ -9,6 +9,23 @@ agent only writes and validates manifests. No Ingress, cert-manager,
|
||||
external-dns, Jobs, CronJobs, CRDs, Roles, RoleBindings, NetworkPolicies, or
|
||||
other cluster-scoped resources are used.
|
||||
|
||||
## Run it
|
||||
|
||||
From this repo root, with `KUBECONFIG` exported (see
|
||||
[Before you start](#before-you-start)):
|
||||
|
||||
```bash
|
||||
make validate
|
||||
make gate1-check
|
||||
make build
|
||||
make push
|
||||
make deploy CONFIRM=yes
|
||||
make smoke SMOKE_SSE=1
|
||||
```
|
||||
|
||||
The first deploy also needs the Gitea pull proof, Tunnel, and Secrets in
|
||||
Steps 1–6. Every cluster mutation is human-run.
|
||||
|
||||
## What Gate 1 means
|
||||
|
||||
Gate 1 passes only when **all** of these are true:
|
||||
@@ -24,59 +41,13 @@ Gate 1 passes only when **all** of these are true:
|
||||
Do not claim Gate 1 from a local `kubectl port-forward`, a green manifest
|
||||
dry-run, or a container that only runs on your laptop.
|
||||
|
||||
## Current external status (2026-11-14 ~08:30 CET)
|
||||
|
||||
Re-check before trusting this snapshot:
|
||||
|
||||
- **v1.1 is deployed and Ready:** web/api/ml `:v1.1`, cloudflared
|
||||
`2026.11.0`, all four Deployments `1/1`. Numeric UIDs are
|
||||
`10001:10001` for web/ml and `65532:65532` for API/cloudflared.
|
||||
- **Tunnel active:** `pulsecast-gate1` serves `pulse.noelsabia.dev` through
|
||||
Nuxt (`x-powered-by: Nuxt`) and `api-pulse.noelsabia.dev` through the API;
|
||||
four HTTP/2 connections. `/stage` and `/live` return 200; `/control` is
|
||||
401 anonymous and 200 with control auth. The temporary hello is deleted.
|
||||
- **v1.1 120-VU Cloudflare run passed:** 120/120 SSE connections opened and
|
||||
closed, 0 errors, p95 event 331 ms, p95 scenario 646 ms, presenter POST
|
||||
49.60 ms, 361/361 checks. See `docs/load-test.md`.
|
||||
- **Failure drill passed:** ML scaled to 0 while API readiness/forecast cache
|
||||
and web stayed available; ML restored and API pod restarted cleanly. See
|
||||
`docs/failure-drills.md`.
|
||||
- **API/product checks:** 8 sensors, `/api/meta` `mae=3.32`, `wape=27.5`,
|
||||
`r2=0.777`, cache `source=fastapi`, staffing plan under 2 s.
|
||||
- **DNS works:** public resolvers return Cloudflare IPs for
|
||||
`pulse.noelsabia.dev` and `api-pulse.noelsabia.dev`. A local
|
||||
`systemd-resolved` negative cache can be cleared with
|
||||
`resolvectl query --cache=no NAME` followed by `resolvectl flush-caches`.
|
||||
- **API hostname is `api-pulse.noelsabia.dev`:** Universal SSL covers only
|
||||
`*.noelsabia.dev`; a two-level `api.pulse.noelsabia.dev` is not covered by
|
||||
that certificate.
|
||||
- **All three Secrets exist:** `gitea-regcred`, `cloudflared`, `control-auth`.
|
||||
- **QUIC/UDP 7844 is untested.** TCP/443 to Cloudflare works, so the
|
||||
`cloudflared` manifest uses `--protocol http2` instead of relying on
|
||||
unverified UDP.
|
||||
|
||||
## What is possible now vs what is blocked
|
||||
|
||||
| Step | Who | Status |
|
||||
|---|---|---|
|
||||
| Cluster/manifest check | agent or human | **done**; static and live checks pass |
|
||||
| Gitea pull + hello proof | human | **done**; hello deleted after cutover |
|
||||
| `gitea-regcred`, `cloudflared`, `control-auth` Secrets | human | **done** |
|
||||
| Cloudflare Tunnel + public hostnames/DNS | human | **done**; tunnel active; `pulse` serves Nuxt |
|
||||
| `api`, `ml`, `web` v1.1 images | human | **done**; pushed and deployed |
|
||||
| v1.1 apply + smoke + presenter k6 + drills | human | **done** |
|
||||
| Results docs | agent | **pending new PR review/merge** |
|
||||
|
||||
`make build` and `make push` cover all three services; no Dockerfile is
|
||||
missing now. Do not deploy placeholder images under the real service names.
|
||||
|
||||
## Before you start
|
||||
|
||||
Run every block in this document in **one shell**, from this worktree. The
|
||||
Run every block in this document in **one shell**, from the repo root. The
|
||||
commands are bash/zsh compatible.
|
||||
|
||||
```bash
|
||||
cd /home/noel/Documents/Hackathon_2026/infra-k8.git/feat-infra-bootstrap
|
||||
cd /path/to/infra-k8
|
||||
|
||||
export KUBECONFIG="/home/noel/Documents/Hackathon_2026/OKF_Vault/fch26-onemanarmy-sa-kubeconfig.yaml"
|
||||
export NS="fch26-onemanarmy"
|
||||
@@ -159,10 +130,11 @@ make validate
|
||||
make gate1-check
|
||||
```
|
||||
|
||||
Expected now (before any human work):
|
||||
Expected on a fresh cluster (before any human work):
|
||||
|
||||
- `make validate` prints `== kubectl client dry-run (read-only)` and 11
|
||||
`created (dry run)` lines.
|
||||
- With a reachable cluster, `make validate` prints
|
||||
`== kubectl client dry-run (read-only)` and 11 `created (dry run)` lines.
|
||||
Without one it prints `11 manifests parsed`.
|
||||
- `make gate1-check` reports **0 hard failures**, the expected permission
|
||||
matrix, and lists missing Secrets/Deployments as `[pending]`.
|
||||
- Exit code 2 from `make gate1-check` is correct at this stage: it means only
|
||||
@@ -297,11 +269,10 @@ Expected: `service/hello exposed`.
|
||||
|
||||
## Step 3 — Cloudflare Tunnel
|
||||
|
||||
The tunnel `pulsecast-gate1` already exists. `pulse.noelsabia.dev` resolves;
|
||||
`api-pulse.noelsabia.dev` resolves after its route is saved. The old
|
||||
`api.pulse.noelsabia.dev` also resolves but is **not certificate-safe**: the
|
||||
Universal SSL certificate is `*.noelsabia.dev` and covers only one subdomain
|
||||
level. Use `api-pulse.noelsabia.dev`, not `api.pulse.noelsabia.dev`.
|
||||
The tunnel used here is `pulsecast-gate1`. `pulse.noelsabia.dev` resolves;
|
||||
`api-pulse.noelsabia.dev` resolves after its route is saved. Universal SSL
|
||||
covers only `*.noelsabia.dev`, so the API host must stay one label under the
|
||||
zone: `api-pulse.noelsabia.dev`.
|
||||
|
||||
```bash
|
||||
getent hosts pulse.noelsabia.dev
|
||||
@@ -343,8 +314,7 @@ getent hosts api-pulse.noelsabia.dev
|
||||
change it to `web:3000` after the web image is deployed. If you already have
|
||||
the web image, you may set `web:3000` now and skip the hello route; then the
|
||||
public hello check in Step 6 is not available, but the tunnel proof still
|
||||
happens when the real web pod is Ready. Delete the old
|
||||
`api.pulse.noelsabia.dev` route if it still exists.
|
||||
happens when the real web pod is Ready.
|
||||
|
||||
5. The hostnames must be in the Cloudflare zone for `noelsabia.dev`.
|
||||
Cloudflare creates the DNS records for a tunnel hostname; do **not** add
|
||||
@@ -440,7 +410,7 @@ test -f ../../api-go.git/main/Dockerfile && echo api-ok || echo api-blocked
|
||||
test -f ../../ML-py.git/main/Dockerfile && echo ml-ok || echo ml-blocked
|
||||
```
|
||||
|
||||
Expected **now** (all three Dockerfiles landed on 2026-11-14):
|
||||
Expected:
|
||||
|
||||
```text
|
||||
web-ok
|
||||
@@ -466,8 +436,8 @@ constraints:
|
||||
- Web: image `USER 10001:10001`; with read-only root and `/tmp` tmpfs, `/`,
|
||||
`/stage`, and `/live` returned 200; `/control` returned 401 anonymous and
|
||||
200 with control auth; the same-origin SSE proxy emitted events.
|
||||
- v1.1 was pushed to Gitea. The cluster is not updated until this branch is
|
||||
reviewed/merged and explicitly applied.
|
||||
- v1.1 is pushed to Gitea and deployed in the cluster; re-apply only after a
|
||||
reviewed image change.
|
||||
|
||||
Push:
|
||||
|
||||
@@ -746,6 +716,37 @@ Also save:
|
||||
Image digests are in the `docker push` output; tags alone are not enough if
|
||||
you need to reproduce the exact deploy.
|
||||
|
||||
## Current status (2026-11-14)
|
||||
|
||||
Re-check before trusting this snapshot:
|
||||
|
||||
- **v1.1 is deployed and Ready:** web/api/ml `:v1.1`, cloudflared
|
||||
`2026.11.0`, all four Deployments `1/1`. Numeric UIDs are
|
||||
`10001:10001` for web/ml and `65532:65532` for API/cloudflared.
|
||||
- **Tunnel active:** `pulsecast-gate1` serves `pulse.noelsabia.dev` through
|
||||
Nuxt (`x-powered-by: Nuxt`) and `api-pulse.noelsabia.dev` through the API;
|
||||
four HTTP/2 connections. `/stage` and `/live` return 200; `/control` is
|
||||
401 anonymous and 200 with control auth. The temporary hello is deleted.
|
||||
- **v1.1 120-VU Cloudflare run passed:** 120/120 SSE connections opened and
|
||||
closed, 0 errors, p95 event 331 ms, p95 scenario 646 ms, presenter POST
|
||||
49.60 ms, 361/361 checks. See `docs/load-test.md`.
|
||||
- **Failure drill passed:** ML scaled to 0 while API readiness/forecast cache
|
||||
and web stayed available; ML restored and API pod restarted cleanly. See
|
||||
`docs/failure-drills.md`.
|
||||
- **API/product checks:** 8 sensors, `/api/meta` `mae=3.32`, `wape=27.5`,
|
||||
`r2=0.777`, cache `source=fastapi`, staffing plan under 2 s.
|
||||
- **DNS works:** public resolvers return Cloudflare IPs for
|
||||
`pulse.noelsabia.dev` and `api-pulse.noelsabia.dev`. A local
|
||||
`systemd-resolved` negative cache can be cleared with
|
||||
`resolvectl query --cache=no NAME` followed by `resolvectl flush-caches`.
|
||||
- **API hostname is `api-pulse.noelsabia.dev`:** Universal SSL covers only
|
||||
`*.noelsabia.dev`, so the host must stay one label under the zone.
|
||||
- **All three Secrets exist:** `gitea-regcred`, `cloudflared`, `control-auth`.
|
||||
- **QUIC/UDP 7844 is untested.** TCP/443 to Cloudflare works, so the
|
||||
`cloudflared` manifest uses `--protocol http2` instead of relying on
|
||||
unverified UDP.
|
||||
- **Results docs are merged in `main`.** This branch only updates CI/docs.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Check | Fix |
|
||||
|
||||
+9
-8
@@ -15,13 +15,14 @@ Build the pinned binary once:
|
||||
|
||||
```bash
|
||||
make k6-build # creates k6/k6-sse (git-ignored)
|
||||
./k6/k6-sse version # expect k6 v2.0.0 + xk6-sse v0.2.0
|
||||
./k6/k6-sse version # expect k6 v2.0.0 + the current xk6-sse
|
||||
```
|
||||
|
||||
The build pins **k6 v2.0.0**. On k6 v2.4.0 the same `xk6-sse` v0.2.0 build
|
||||
fails at runtime with `TypeError: Object has no member 'on'` (the extension's
|
||||
client methods are not exposed); v2.0.0 was verified locally on 2026-11-14. If
|
||||
you bump `K6_BUILD_VERSION`, rerun the script against a test stream first.
|
||||
The build pins **k6 v2.0.0**; `xk6-sse` v0.2.0 is the version used for the
|
||||
recorded runs. On k6 v2.4.0 that extension build fails at runtime with
|
||||
`TypeError: Object has no member 'on'` (the client methods are not exposed);
|
||||
v2.0.0 was verified locally on 2026-11-14. If you bump `K6_BUILD_VERSION`,
|
||||
rerun the script against a test stream first.
|
||||
|
||||
`make load` uses, in order: `K6` if set, `k6/k6-sse`, then `k6` on `PATH`.
|
||||
A stock `grafana/k6` image is **not** a fallback: it lacks the SSE extension.
|
||||
@@ -79,9 +80,9 @@ A stock `grafana/k6` image is **not** a fallback: it lacks the SSE extension.
|
||||
The script was validated locally on 2026-11-14 with the pinned k6 v2.0.0 build
|
||||
against a mock SSE stream and the real v1.1 API image (3 VUs, 1 heartbeat
|
||||
each): connections opened and closed cleanly, open success 100%, presenter
|
||||
POST 200, and scenario latency sampled. The real 120-client run still requires
|
||||
the deployed API, because localhost results say nothing about Cloudflare
|
||||
buffering or SKE scheduling.
|
||||
POST 200, and scenario latency sampled. Run the 120-client test against the
|
||||
deployed API; localhost says nothing about Cloudflare buffering or SKE
|
||||
scheduling.
|
||||
|
||||
## Pass criteria
|
||||
|
||||
|
||||
Reference in new issue
Block a user