326 lines
13 KiB
Markdown
326 lines
13 KiB
Markdown
# Deploy runbook — Gate 1
|
|
|
|
Target: three services (`web`, `api`, `ml`) plus one `cloudflared` Deployment in
|
|
namespace `fch26-onemanarmy`, reachable over public HTTPS, with Gitea image
|
|
pulls working.
|
|
|
|
**The human runs every cluster-mutating command in this document.** The infra
|
|
agent only writes and validates manifests. No Ingress, cert-manager,
|
|
external-dns, Jobs, CronJobs, CRDs, Roles, RoleBindings, NetworkPolicies, or
|
|
other cluster-scoped resources are used.
|
|
|
|
## Gate 1 checklist
|
|
|
|
Run top to bottom. Stop at the first failure and use
|
|
[Troubleshooting](#troubleshooting).
|
|
|
|
- [ ] 1. Kubeconfig exported, `600`, correct context, namespace reachable.
|
|
- [ ] 2. Gitea **read-only package token** created and `docker login` succeeds.
|
|
- [ ] 3. Hello-world image pushed and pulled by the cluster with `gitea-regcred`.
|
|
- [ ] 4. Secrets created: `gitea-regcred`, `cloudflared`, `control-auth`.
|
|
- [ ] 5. Real `web`/`api`/`ml` images built, tagged `gate1`, pushed to Gitea.
|
|
- [ ] 6. `make validate` passes locally.
|
|
- [ ] 7. Cloudflare Tunnel exists; both public hostnames configured.
|
|
- [ ] 8. `make deploy CONFIRM=yes` completes and all four Deployments are ready.
|
|
- [ ] 9. `make smoke SMOKE_SSE=1` passes.
|
|
- [ ] 10. `https://pulse.noelsabia.dev` opens on a phone over mobile data.
|
|
- [ ] 11. `https://api.pulse.noelsabia.dev/api/healthz` returns 200.
|
|
- [ ] 12. Record the deployed image tags and the time in the status note.
|
|
|
|
## 0. Prerequisites
|
|
|
|
The kubeconfig is **not in this repo**. It lives outside git:
|
|
|
|
```bash
|
|
export KUBECONFIG="/home/noel/Documents/Hackathon_2026/OKF_Vault/fch26-onemanarmy-sa-kubeconfig.yaml"
|
|
export NS=fch26-onemanarmy
|
|
export CTX=fch26-onemanarmy-sa-context
|
|
export REGISTRY=git.noelsabia.dev
|
|
export OWNER=noelsabia
|
|
export TAG=gate1
|
|
|
|
chmod 600 "$KUBECONFIG"
|
|
```
|
|
|
|
Verify read-only first:
|
|
|
|
```bash
|
|
kubectl config current-context # fch26-onemanarmy-sa-context
|
|
kubectl -n "$NS" auth can-i create deployments # yes
|
|
kubectl -n "$NS" auth can-i create services # yes
|
|
kubectl -n "$NS" auth can-i create ingresses # no
|
|
kubectl -n "$NS" auth can-i create jobs # no
|
|
kubectl -n "$NS" get pods,svc,secrets
|
|
```
|
|
|
|
If any of the first three answers differ, stop and report; do not work around
|
|
permissions with cluster-scoped resources.
|
|
|
|
Local tools: `git`, `make`, `bash`, `curl`, `docker`, `kubectl`. Optional:
|
|
`kubeconform` (schema validation), `jq`, `k6` built with the SSE extension.
|
|
|
|
## 1. Gitea read-only package token
|
|
|
|
In Gitea: **Settings → Applications → Generate New Token** (or the package
|
|
token screen). Scope: read-only package access. Do not reuse the admin token.
|
|
|
|
```bash
|
|
read -rsp 'Gitea package token: ' GITEA_TOKEN; echo
|
|
printf '%s' "$GITEA_TOKEN" | docker login "$REGISTRY" -u "$OWNER" --password-stdin
|
|
```
|
|
|
|
A successful login prints `Login Succeeded`. Never put the token in a file,
|
|
a command that lands in shell history, or this repo.
|
|
|
|
## 2. Hello-world image pull (prove the registry path)
|
|
|
|
```bash
|
|
docker pull nginxinc/nginx-unprivileged:alpine
|
|
docker tag nginxinc/nginx-unprivileged:alpine "$REGISTRY/$OWNER/hello:gate1"
|
|
docker push "$REGISTRY/$OWNER/hello:gate1"
|
|
|
|
kubectl -n "$NS" create secret docker-registry gitea-regcred \
|
|
--docker-server="$REGISTRY" \
|
|
--docker-username="$OWNER" \
|
|
--docker-password="$GITEA_TOKEN"
|
|
|
|
kubectl -n "$NS" create deployment hello \
|
|
--image="$REGISTRY/$OWNER/hello:gate1"
|
|
kubectl -n "$NS" patch deployment hello --type=merge -p \
|
|
'{"spec":{"template":{"spec":{"imagePullSecrets":[{"name":"gitea-regcred"}]}}}}'
|
|
kubectl -n "$NS" rollout status deployment/hello --timeout=180s
|
|
kubectl -n "$NS" get pods -l app=hello
|
|
```
|
|
|
|
Pass: the pod is `Running` and `kubectl describe pod` shows the image pulled.
|
|
Clean up before the real deploy:
|
|
|
|
```bash
|
|
kubectl -n "$NS" delete deployment hello
|
|
```
|
|
|
|
If this fails, fix the token/secret before building the application images.
|
|
|
|
## 3. Create the remaining Secrets
|
|
|
|
Generate values locally; never commit them.
|
|
|
|
```bash
|
|
read -rsp 'Control password: ' CONTROL_PASS; echo
|
|
kubectl -n "$NS" create secret generic control-auth \
|
|
--from-literal=CONTROL_USER=control \
|
|
--from-literal=CONTROL_PASS="$CONTROL_PASS"
|
|
|
|
# Token from Cloudflare Zero Trust → Networks → Tunnels → your tunnel.
|
|
read -rsp 'Cloudflare tunnel token: ' TUNNEL_TOKEN; echo
|
|
kubectl -n "$NS" create secret generic cloudflared \
|
|
--from-literal=TUNNEL_TOKEN="$TUNNEL_TOKEN"
|
|
|
|
kubectl -n "$NS" get secrets
|
|
```
|
|
|
|
`make secret-commands` prints the same commands with placeholders.
|
|
|
|
## 4. Build and push the three images
|
|
|
|
The manifests expect:
|
|
|
|
| Service | Image | Port | Health paths |
|
|
|---|---|---|---|
|
|
| web | `git.noelsabia.dev/noelsabia/pulsecast-web:gate1` | 3000 | `/` |
|
|
| api | `git.noelsabia.dev/noelsabia/pulsecast-api:gate1` | 8080 | `/api/healthz`, `/api/readyz` |
|
|
| ml | `git.noelsabia.dev/noelsabia/pulsecast-ml:gate1` | 8000 | `/healthz`, `/readyz` |
|
|
|
|
```bash
|
|
make build # builds all three from the service worktrees
|
|
make push # docker push to Gitea
|
|
```
|
|
|
|
`make build` auto-detects the service worktrees under
|
|
`/home/noel/Documents/Hackathon_2026/worktrees/`. Override when needed:
|
|
|
|
```bash
|
|
make build WEB_DIR=/path/to/web-nuxt API_DIR=/path/to/api-go ML_DIR=/path/to/ML-py
|
|
```
|
|
|
|
If `OWNER` or `TAG` differs from the manifest defaults, either update
|
|
`deploy/k8s/*.yaml` before `make deploy`, or deploy first and then run the
|
|
guarded image update:
|
|
|
|
```bash
|
|
make set-image CONFIRM=yes OWNER=<owner> TAG=<tag>
|
|
```
|
|
|
|
### Dockerfile contract (coordinate through the orchestrator)
|
|
|
|
Each service image must:
|
|
|
|
- listen on `0.0.0.0` at the port above;
|
|
- run as a non-root user (numeric UID, no shell required);
|
|
- write only to `/tmp` (the pod root filesystem is read-only);
|
|
- start with the environment in `deploy/k8s/00-config.yaml`;
|
|
- be self-contained: no runtime download of data, model, fonts, or packages.
|
|
|
|
`api` must read `DATA_DIR` (baked JSON), `FASTAPI_URL`, `PORT`,
|
|
`CONTROL_USER`, `CONTROL_PASS`, `SSE_HEARTBEAT`, and `ML_TIMEOUT`.
|
|
`web` must have `NUXT_PUBLIC_API_BASE=https://api.pulse.noelsabia.dev` baked in or
|
|
provided at runtime; `ml` must find its model and artifacts under the paths its
|
|
Dockerfile defines. The env names in the manifests are the integration point:
|
|
change them here first if a service needs a different name.
|
|
|
|
## 5. Cloudflare Tunnel
|
|
|
|
1. Zero Trust → **Networks → Tunnels → Create a tunnel**.
|
|
2. Connector: **Cloudflared**. Name: `pulsecast-gate1`.
|
|
3. Copy the tunnel token (it is shown once). Store it in the `cloudflared`
|
|
Secret (step 3).
|
|
4. Add public hostnames:
|
|
|
|
| Public hostname | Service |
|
|
|---|---|
|
|
| `pulse.noelsabia.dev` | `http://web:3000` |
|
|
| `api.pulse.noelsabia.dev` | `http://api:8080` |
|
|
|
|
The hostnames must be in the Cloudflare zone for `noelsabia.dev`. The
|
|
service URL uses the Kubernetes Service name and port inside the namespace;
|
|
no Ingress is involved.
|
|
5. For the first smoke test before the app images exist, temporarily point
|
|
`pulse.noelsabia.dev` at `http://hello:8080`, then switch it to
|
|
`http://web:3000` after step 7.
|
|
|
|
DNS records for the public hostnames are created by Cloudflare; do not add
|
|
manual A/AAAA records for a tunnel hostname.
|
|
|
|
## 6. Validate before applying
|
|
|
|
```bash
|
|
make validate
|
|
```
|
|
|
|
This is read-only:
|
|
|
|
- `kubeconform -strict` validates the resource schemas if installed;
|
|
- otherwise, if a cluster is reachable, `kubectl apply --dry-run=client`
|
|
validates against the API server;
|
|
- if neither is available it parses the YAML and prints a warning that schema
|
|
validation was skipped.
|
|
|
|
Never run `kubectl apply` / `delete` from an agent session. The commands below
|
|
are for the human.
|
|
|
|
## 7. Deploy
|
|
|
|
```bash
|
|
make deploy-check # read-only: required Secrets exist
|
|
make deploy CONFIRM=yes # applies deploy/k8s and waits for rollouts
|
|
```
|
|
|
|
`make deploy` fails closed: without `CONFIRM=yes` it changes nothing. It does
|
|
not create the namespace or secrets.
|
|
|
|
If the three service images are not available yet, deploy only the tunnel and
|
|
use the hello service to prove public HTTPS; return to this step when the real
|
|
images exist. Do not deploy a placeholder image under the real service names.
|
|
|
|
Check the result:
|
|
|
|
```bash
|
|
kubectl -n "$NS" get deploy,pods,svc
|
|
kubectl -n "$NS" get pods -o wide
|
|
kubectl -n "$NS" logs deployment/cloudflared --tail=50 | grep 'Registered tunnel connection'
|
|
```
|
|
|
|
Expected: `web`, `api`, `ml`, `cloudflared` each `1/1`; Services `web`, `api`,
|
|
`ml` are `ClusterIP`; `cloudflared` has no Service (it dials out).
|
|
|
|
## 8. Verify
|
|
|
|
Public checks (also runnable from a phone on mobile data):
|
|
|
|
```bash
|
|
curl -fsS -o /dev/null -w 'web %{http_code}\n' https://pulse.noelsabia.dev/
|
|
curl -fsS -o /dev/null -w 'api %{http_code}\n' https://api.pulse.noelsabia.dev/api/healthz
|
|
curl -fsS -o /dev/null -w 'ready %{http_code}\n' https://api.pulse.noelsabia.dev/api/readyz
|
|
curl -fsS https://api.pulse.noelsabia.dev/api/meta | jq '{data_version,model_version}'
|
|
curl -sS -N --max-time 20 https://api.pulse.noelsabia.dev/api/stream | head -20
|
|
```
|
|
|
|
The SSE output must start with `retry: 3000`, contain a `scenario` event
|
|
immediately, and show a `ping` within 20 s.
|
|
|
|
Full smoke test, including the ML service through a temporary
|
|
`kubectl port-forward`:
|
|
|
|
```bash
|
|
make smoke SMOKE_SSE=1
|
|
```
|
|
|
|
Local variant once the three dev servers are running:
|
|
|
|
```bash
|
|
make smoke-local
|
|
```
|
|
|
|
In-cluster connectivity (optional; creates and removes one short-lived pod):
|
|
|
|
```bash
|
|
kubectl -n "$NS" run gate1-netcheck --rm -it --restart=Never \
|
|
--image=curlimages/curl:latest --command -- sh -c \
|
|
'curl -fsS http://api:8080/api/healthz && \
|
|
curl -fsS http://ml:8000/healthz && \
|
|
curl -fsS http://web:3000/ >/dev/null && echo connectivity-ok'
|
|
```
|
|
|
|
TLS is verified by `make smoke` (`ssl_verify_result=0`). Certificates are
|
|
issued by Cloudflare at the edge; there is no cert-manager resource.
|
|
|
|
## 9. Record the Gate 1 result
|
|
|
|
Write down, in the status note or README:
|
|
|
|
- deploy time and commit hash;
|
|
- image tags actually deployed (`kubectl -n "$NS" get deploy -o jsonpath=...`);
|
|
- output of `make smoke SMOKE_SSE=1`;
|
|
- public URLs and the device/network used;
|
|
- any warning that remains.
|
|
|
|
## Troubleshooting
|
|
|
|
| Symptom | Check | Fix |
|
|
|---|---|---|
|
|
| `ImagePullBackOff` / `ErrImagePull` | `kubectl -n "$NS" describe pod <pod>`; `kubectl -n "$NS" get secret gitea-regcred` | Token is read-only package scoped; image tag exists; secret is in the same namespace. |
|
|
| `CreateContainerConfigError` | `kubectl -n "$NS" describe pod <pod>` | A referenced Secret key is missing (`control-auth` or `cloudflared`). Recreate the Secret. |
|
|
| `cloudflared` CrashLoopBackOff | `kubectl -n "$NS" logs deployment/cloudflared --previous` | Invalid/expired tunnel token, or the tunnel was deleted in Cloudflare. Recreate the Secret and restart. |
|
|
| Public 502 / 1033 | `kubectl -n "$NS" get endpoints web api`; cloudflared logs | Service has no ready endpoints, wrong port, or hostname route points at the wrong service. |
|
|
| `api` pod not ready | `kubectl -n "$NS" logs deployment/api`; `kubectl -n "$NS" exec deploy/api -- ls /app/data` | Baked JSON missing from the image or `DATA_DIR` wrong. Rebuild the API image with the artifacts. |
|
|
| `ml` pod stuck in startup | `kubectl -n "$NS" logs deployment/ml` | Model load exceeds 120 s or the model path is wrong. Fix the image/path; startupProbe allows 120 s. |
|
|
| `web` 500 | `kubectl -n "$NS" logs deployment/web` | Missing build-time API base or server error. Rebuild with `NUXT_PUBLIC_API_BASE`. |
|
|
| SSE events arrive only after a delay | `curl -N` direct to the pod vs through Cloudflare | Cloudflare buffering; the web client falls back to polling `/api/scenario` every 5 s. Keep the heartbeat at 15 s. |
|
|
| `make deploy-check` fails | `kubectl -n "$NS" get secrets` | Create the missing Secret; do not commit it. |
|
|
|
|
## Rollback
|
|
|
|
No cluster-scoped resources are created, so rollback is namespace-local:
|
|
|
|
```bash
|
|
kubectl -n "$NS" rollout undo deployment/web
|
|
kubectl -n "$NS" rollout undo deployment/api
|
|
kubectl -n "$NS" rollout undo deployment/ml
|
|
kubectl -n "$NS" rollout undo deployment/cloudflared
|
|
kubectl -n "$NS" rollout status deployment/web --timeout=180s
|
|
```
|
|
|
|
If the tunnel token or route is wrong, fix the Secret / Cloudflare dashboard
|
|
and restart `cloudflared`; do not create an Ingress as a workaround.
|
|
|
|
## Security notes
|
|
|
|
- Never commit the kubeconfig, tunnel token, control password, Gitea token, or
|
|
any `.env` file. `.gitignore` covers the usual filenames; check
|
|
`git status` before every commit.
|
|
- Rotate the Gitea package token and the control password after the event.
|
|
Rotate the service-account token after the event; it was shared in chat.
|
|
- The kubeconfig must stay at mode `600` and outside the repos.
|
|
- `make smoke` prints no secrets; do not add `set -x` to the scripts with a
|
|
live `KUBECONFIG`.
|