fix: use certificate-safe api-pulse hostname
Cloudflare Universal SSL covers *.noelsabia.dev only one level deep. api.pulse.noelsabia.dev is two levels and fails the certificate check. - switch the API base/SSE URLs to api-pulse.noelsabia.dev - update smoke, k6, runbook, and failure drills - record the verified DNS/certificate diagnosis and local cache flush
This commit is contained in:
1 parent
2eacb62696
commit
e8e0f2035b
10 files changed
+68
-55
No files matched your search
+40
-30
@@ -19,22 +19,28 @@ Gate 1 passes only when **all** of these are true:
|
||||
3. `web`, `api`, `ml`, and `cloudflared` are Ready.
|
||||
4. `make smoke SMOKE_SSE=1` passes and a phone on mobile data can open
|
||||
`https://pulse.noelsabia.dev`.
|
||||
5. `https://api.pulse.noelsabia.dev/api/healthz` returns HTTP 200.
|
||||
5. `https://api-pulse.noelsabia.dev/api/healthz` returns HTTP 200.
|
||||
|
||||
Do not claim Gate 1 from a local `kubectl port-forward`, a green manifest
|
||||
dry-run, or a container that only runs on your laptop.
|
||||
|
||||
## Current external status (2026-11-14 03:50 UTC smoke)
|
||||
## Current external status (2026-11-14 ~06:40 CET)
|
||||
|
||||
Findings from `CLUSTER-SMOKE.md`; re-check before trusting this snapshot:
|
||||
Re-check before trusting this snapshot:
|
||||
|
||||
- **Verified:** cluster access, scheduling, Service DNS/HTTP, probes, Secrets,
|
||||
non-root + read-only-root + `emptyDir`, and egress to Cloudflare,
|
||||
Open-Meteo, and `git.noelsabia.dev`.
|
||||
- **No public DNS yet:** `pulse.noelsabia.dev` and
|
||||
`api.pulse.noelsabia.dev` have no records. That is expected until the
|
||||
Cloudflare Tunnel public hostnames are created.
|
||||
- **No tunnel yet:** Cloudflare Tunnel has not been created.
|
||||
- **Tunnel created, connector pending:** `pulsecast-gate1` has routes for
|
||||
`pulse.noelsabia.dev` and `api-pulse.noelsabia.dev`; it stays Inactive until
|
||||
the `cloudflared` Deployment runs.
|
||||
- **DNS works:** public resolvers return Cloudflare IPs for
|
||||
`pulse.noelsabia.dev` and the old `api.pulse.noelsabia.dev`. A local
|
||||
`systemd-resolved` NXDOMAIN cache can be flushed with
|
||||
`resolvectl flush-caches`.
|
||||
- **API hostname is `api-pulse.noelsabia.dev`:** Universal SSL covers only
|
||||
`*.noelsabia.dev`; a two-level `api.pulse.noelsabia.dev` is not covered by
|
||||
that certificate.
|
||||
- **Gitea pull is proven (2026-11-14 ~06:16 CET):** `gitea-regcred` exists and
|
||||
the cluster pulled `git.noelsabia.dev/noel/hello:gate1` in 120 ms; the hello
|
||||
pod is `1/1 Running` and its Service returned HTTP 200. The Gitea owner is
|
||||
@@ -57,11 +63,11 @@ Findings from `CLUSTER-SMOKE.md`; re-check before trusting this snapshot:
|
||||
| Read-only cluster/manifest check | agent or human | **ready now** (`make gate1-check`) |
|
||||
| Gitea push login + hello-world push | human | **done**; hello pull and Service verified |
|
||||
| `gitea-regcred`, `cloudflared`, `control-auth` Secrets | human | `gitea-regcred` **done**; `cloudflared`/`control-auth` pending |
|
||||
| Cloudflare Tunnel + public hostnames/DNS | human | **blocked on human** — not created; DNS absent |
|
||||
| Cloudflare Tunnel + public hostnames/DNS | human | tunnel created; routes added; **connector pending** |
|
||||
| `api` image | human | **ready** — `make build-api` + manifest-equivalent run verified locally |
|
||||
| `ml` image | human | **ready** — `make build-ml` + manifest-equivalent run verified locally |
|
||||
| `web` image | human | **ready** — `make build-web` + manifest-equivalent run verified locally |
|
||||
| Full deploy + smoke + Gate 1 sign-off | human | **ready once the Gitea/Tunnel steps are done** |
|
||||
| Full deploy + smoke + Gate 1 sign-off | human | **ready once the tunnel/secret/image steps are done** |
|
||||
|
||||
`make build` and `make push` cover all three services; no Dockerfile is
|
||||
missing now. Do not deploy placeholder images under the real service names.
|
||||
@@ -145,7 +151,7 @@ Run top to bottom. Stop at the first failure and use
|
||||
- [ ] 10. `make deploy CONFIRM=yes` completes; all four Deployments are Ready.
|
||||
- [ ] 11. `make smoke SMOKE_SSE=1` passes.
|
||||
- [ ] 12. Phone over mobile data opens `https://pulse.noelsabia.dev`;
|
||||
`https://api.pulse.noelsabia.dev/api/healthz` returns 200.
|
||||
`https://api-pulse.noelsabia.dev/api/healthz` returns 200.
|
||||
- [ ] 13. Image tags/digests and the time are recorded.
|
||||
|
||||
## Step 0 — read-only readiness and dry-run
|
||||
@@ -293,12 +299,21 @@ Expected: `service/hello exposed`.
|
||||
|
||||
## Step 3 — Cloudflare Tunnel
|
||||
|
||||
As of the 03:50 UTC smoke, neither public hostname has a DNS record. That is
|
||||
expected before the tunnel route exists; do not add manual A/AAAA records.
|
||||
The tunnel `pulsecast-gate1` already exists. `pulse.noelsabia.dev` resolves;
|
||||
`api-pulse.noelsabia.dev` resolves after its route is saved. The old
|
||||
`api.pulse.noelsabia.dev` also resolves but is **not certificate-safe**: the
|
||||
Universal SSL certificate is `*.noelsabia.dev` and covers only one subdomain
|
||||
level. Use `api-pulse.noelsabia.dev`, not `api.pulse.noelsabia.dev`.
|
||||
|
||||
```bash
|
||||
getent hosts pulse.noelsabia.dev || echo 'pulse: no DNS record yet (expected)'
|
||||
getent hosts api.pulse.noelsabia.dev || echo 'api: no DNS record yet (expected)'
|
||||
getent hosts pulse.noelsabia.dev
|
||||
getent hosts api-pulse.noelsabia.dev || echo 'api-pulse: no DNS record yet (expected until the route exists)'
|
||||
```
|
||||
|
||||
If DNS looks missing locally but public DNS has it, flush the local cache:
|
||||
|
||||
```bash
|
||||
resolvectl flush-caches
|
||||
```
|
||||
|
||||
1. Open Cloudflare **Zero Trust → Networks → Tunnels → Create a tunnel**.
|
||||
@@ -315,19 +330,14 @@ getent hosts api.pulse.noelsabia.dev || echo 'api: no DNS record yet (expected)'
|
||||
| Subdomain | Domain | Path | Service type | Service URL |
|
||||
|---|---|---|---|---|
|
||||
| `pulse` | `noelsabia.dev` | *(empty)* | HTTP | `hello:8080` |
|
||||
| `api.pulse` | `noelsabia.dev` | *(empty)* | HTTP | `api:8080` |
|
||||
| `api-pulse` | `noelsabia.dev` | *(empty)* | HTTP | `api:8080` |
|
||||
|
||||
The first route is temporary for Gate 1-A. For the final Gate 1-B route,
|
||||
change it to `web:3000` after the web image is deployed. If you already have
|
||||
the web image, you may set `web:3000` now and skip the hello route; then the
|
||||
public hello check in Step 6 is not available, but the tunnel proof still
|
||||
happens when the real web pod is Ready.
|
||||
|
||||
For the final Gate 1-B route, change `pulse.noelsabia.dev` to
|
||||
`http://web:3000` after the web image is deployed. If you already have the
|
||||
web image, you may set `web:3000` now and skip the hello route; then the
|
||||
public hello check in Step 6 is not available, but the tunnel proof still
|
||||
happens when the real web pod is Ready.
|
||||
happens when the real web pod is Ready. Delete the old
|
||||
`api.pulse.noelsabia.dev` route if it still exists.
|
||||
|
||||
5. The hostnames must be in the Cloudflare zone for `noelsabia.dev`.
|
||||
Cloudflare creates the DNS records for a tunnel hostname; do **not** add
|
||||
@@ -340,7 +350,7 @@ After saving both hostnames, DNS should resolve to Cloudflare:
|
||||
|
||||
```bash
|
||||
getent hosts pulse.noelsabia.dev
|
||||
getent hosts api.pulse.noelsabia.dev
|
||||
getent hosts api-pulse.noelsabia.dev
|
||||
```
|
||||
|
||||
Expected: one or more Cloudflare IPv4/IPv6 addresses. If there is still no
|
||||
@@ -505,7 +515,7 @@ together, and keep the value numeric.
|
||||
`api` must read `DATA_DIR` (baked JSON), `FASTAPI_URL`, `PORT`,
|
||||
`CONTROL_USER`, `CONTROL_PASS`, `CONTROL_TOKEN`, `SSE_HEARTBEAT`, and
|
||||
`ML_TIMEOUT`.
|
||||
`web` must have `NUXT_PUBLIC_API_BASE=https://api.pulse.noelsabia.dev` baked in
|
||||
`web` must have `NUXT_PUBLIC_API_BASE=https://api-pulse.noelsabia.dev` baked in
|
||||
or read it at runtime from the ConfigMap; the browser must use the public API
|
||||
hostname. `ml` reads `PULSECAST_ROOT`, `PULSECAST_ARTIFACTS`,
|
||||
`PULSECAST_WEATHER`, `PULSECAST_MODELS`, `PULSECAST_UPLOADS`, and
|
||||
@@ -630,10 +640,10 @@ Manual public checks (also run from a phone on mobile data):
|
||||
|
||||
```bash
|
||||
curl -fsS -o /dev/null -w 'web %{http_code}\n' https://pulse.noelsabia.dev/
|
||||
curl -fsS -o /dev/null -w 'api %{http_code}\n' https://api.pulse.noelsabia.dev/api/healthz
|
||||
curl -fsS -o /dev/null -w 'ready %{http_code}\n' https://api.pulse.noelsabia.dev/api/readyz
|
||||
curl -fsS https://api.pulse.noelsabia.dev/api/meta | jq '{data_version,model_version}'
|
||||
curl -sS -N --max-time 20 https://api.pulse.noelsabia.dev/api/stream | head -20
|
||||
curl -fsS -o /dev/null -w 'api %{http_code}\n' https://api-pulse.noelsabia.dev/api/healthz
|
||||
curl -fsS -o /dev/null -w 'ready %{http_code}\n' https://api-pulse.noelsabia.dev/api/readyz
|
||||
curl -fsS https://api-pulse.noelsabia.dev/api/meta | jq '{data_version,model_version}'
|
||||
curl -sS -N --max-time 20 https://api-pulse.noelsabia.dev/api/stream | head -20
|
||||
```
|
||||
|
||||
Expected:
|
||||
@@ -714,11 +724,11 @@ you need to reproduce the exact deploy.
|
||||
| `ImagePullBackOff` / `ErrImagePull` | `kubectl -n "$NS" describe pod -l app=hello` (and the real pod); `kubectl -n "$NS" get secret gitea-regcred` | The token must have `read:package`; the Secret must be in `fch26-onemanarmy`; the tag must exist in Gitea. Do not print the secret data. |
|
||||
| `CreateContainerConfigError` | `kubectl -n "$NS" describe pod <pod>` | A referenced Secret key is missing (`control-auth`: `CONTROL_USER`, `CONTROL_PASS`, `CONTROL_TOKEN`) or the `cloudflared` Secret lacks `TUNNEL_TOKEN`. Recreate the Secret. |
|
||||
| `container has runAsNonRoot and image will run as root` or a non-numeric-user error | `kubectl -n "$NS" get pod <pod> -o yaml`; inspect `securityContext` | Keep `runAsNonRoot: true` and set a positive numeric `runAsUser`/`runAsGroup` (SKE caveat). The manifests already do; do not "fix" this by disabling `runAsNonRoot`. |
|
||||
| Public hostname has no DNS record | `getent hosts pulse.noelsabia.dev` / `getent hosts api.pulse.noelsabia.dev` | Save the public hostname in the tunnel dashboard in the `noelsabia.dev` zone; remove a conflicting A/AAAA/CNAME for that exact name. Do not add manual records for a tunnel hostname. |
|
||||
| Public hostname has no DNS record | `getent hosts pulse.noelsabia.dev` / `getent hosts api-pulse.noelsabia.dev` | Save the public hostname in the tunnel dashboard in the `noelsabia.dev` zone; remove a conflicting A/AAAA/CNAME for that exact name. Do not add manual records for a tunnel hostname. |
|
||||
| `cloudflared` → `CrashLoopBackOff` | `kubectl -n "$NS" logs deployment/cloudflared --previous` | Invalid/expired tunnel token, or the tunnel was deleted in Cloudflare. Recreate the Secret and restart. |
|
||||
| `cloudflared` logs QUIC/timeout errors after you changed the protocol | `kubectl -n "$NS" logs deployment/cloudflared --tail=50` | UDP/7844 is unverified on SKE. Put `--protocol http2` back (the committed manifest already sets it); TCP/443 is confirmed. |
|
||||
| Public `502` / `1033` / `530` | `kubectl -n "$NS" get endpoints web api hello`; cloudflared logs | No ready endpoints, wrong Service port, or the hostname route points at the wrong service. Fix the dashboard route or the Service. Never add an Ingress. |
|
||||
| `api` pod not ready | `kubectl -n "$NS" logs deployment/api`; `curl -fsS https://api.pulse.noelsabia.dev/api/readyz` | Baked JSON missing from the image or `DATA_DIR` wrong. Rebuild the API image with the artifacts. The distroless API image has no shell, so do not rely on `kubectl exec ... ls`. |
|
||||
| `api` pod not ready | `kubectl -n "$NS" logs deployment/api`; `curl -fsS https://api-pulse.noelsabia.dev/api/readyz` | Baked JSON missing from the image or `DATA_DIR` wrong. Rebuild the API image with the artifacts. The distroless API image has no shell, so do not rely on `kubectl exec ... ls`. |
|
||||
| `ml` pod stuck in startup | `kubectl -n "$NS" logs deployment/ml` | Model load exceeds the 120 s startup budget or `MODEL_PATH` is wrong. Fix the image/path. |
|
||||
| `web` 500 | `kubectl -n "$NS" logs deployment/web` | Missing runtime API base or SSR error. Rebuild with `NUXT_PUBLIC_API_BASE` or fix the ConfigMap. |
|
||||
| SSE events arrive only after a delay | `curl -N` direct to the pod vs through Cloudflare | Cloudflare buffering. Keep the 15 s heartbeat and the web polling fallback (`/api/scenario` every 5 s). Do not remove the heartbeat. |
|
||||
|
||||
Reference in new issue
Block a user