feat: deployment manifests and Gate 1 runbook
This commit is contained in:
1 parent
aeee20cb7f
commit
55dfe8b616
3 files changed
+402
-54
No files matched your search
+325
@@ -0,0 +1,325 @@
|
||||
# Deploy runbook — Gate 1
|
||||
|
||||
Target: three services (`web`, `api`, `ml`) plus one `cloudflared` Deployment in
|
||||
namespace `fch26-onemanarmy`, reachable over public HTTPS, with Gitea image
|
||||
pulls working.
|
||||
|
||||
**The human runs every cluster-mutating command in this document.** The infra
|
||||
agent only writes and validates manifests. No Ingress, cert-manager,
|
||||
external-dns, Jobs, CronJobs, CRDs, Roles, RoleBindings, NetworkPolicies, or
|
||||
other cluster-scoped resources are used.
|
||||
|
||||
## Gate 1 checklist
|
||||
|
||||
Run top to bottom. Stop at the first failure and use
|
||||
[Troubleshooting](#troubleshooting).
|
||||
|
||||
- [ ] 1. Kubeconfig exported, `600`, correct context, namespace reachable.
|
||||
- [ ] 2. Gitea **read-only package token** created and `docker login` succeeds.
|
||||
- [ ] 3. Hello-world image pushed and pulled by the cluster with `gitea-regcred`.
|
||||
- [ ] 4. Secrets created: `gitea-regcred`, `cloudflared`, `control-auth`.
|
||||
- [ ] 5. Real `web`/`api`/`ml` images built, tagged `gate1`, pushed to Gitea.
|
||||
- [ ] 6. `make validate` passes locally.
|
||||
- [ ] 7. Cloudflare Tunnel exists; both public hostnames configured.
|
||||
- [ ] 8. `make deploy CONFIRM=yes` completes and all four Deployments are ready.
|
||||
- [ ] 9. `make smoke SMOKE_SSE=1` passes.
|
||||
- [ ] 10. `https://pulse.noelsabia.dev` opens on a phone over mobile data.
|
||||
- [ ] 11. `https://api.pulse.noelsabia.dev/api/healthz` returns 200.
|
||||
- [ ] 12. Record the deployed image tags and the time in the status note.
|
||||
|
||||
## 0. Prerequisites
|
||||
|
||||
The kubeconfig is **not in this repo**. It lives outside git:
|
||||
|
||||
```bash
|
||||
export KUBECONFIG="/home/noel/Documents/Hackathon_2026/OKF_Vault/fch26-onemanarmy-sa-kubeconfig.yaml"
|
||||
export NS=fch26-onemanarmy
|
||||
export CTX=fch26-onemanarmy-sa-context
|
||||
export REGISTRY=git.noelsabia.dev
|
||||
export OWNER=noelsabia
|
||||
export TAG=gate1
|
||||
|
||||
chmod 600 "$KUBECONFIG"
|
||||
```
|
||||
|
||||
Verify read-only first:
|
||||
|
||||
```bash
|
||||
kubectl config current-context # fch26-onemanarmy-sa-context
|
||||
kubectl -n "$NS" auth can-i create deployments # yes
|
||||
kubectl -n "$NS" auth can-i create services # yes
|
||||
kubectl -n "$NS" auth can-i create ingresses # no
|
||||
kubectl -n "$NS" auth can-i create jobs # no
|
||||
kubectl -n "$NS" get pods,svc,secrets
|
||||
```
|
||||
|
||||
If any of the first three answers differ, stop and report; do not work around
|
||||
permissions with cluster-scoped resources.
|
||||
|
||||
Local tools: `git`, `make`, `bash`, `curl`, `docker`, `kubectl`. Optional:
|
||||
`kubeconform` (schema validation), `jq`, `k6` built with the SSE extension.
|
||||
|
||||
## 1. Gitea read-only package token
|
||||
|
||||
In Gitea: **Settings → Applications → Generate New Token** (or the package
|
||||
token screen). Scope: read-only package access. Do not reuse the admin token.
|
||||
|
||||
```bash
|
||||
read -rsp 'Gitea package token: ' GITEA_TOKEN; echo
|
||||
printf '%s' "$GITEA_TOKEN" | docker login "$REGISTRY" -u "$OWNER" --password-stdin
|
||||
```
|
||||
|
||||
A successful login prints `Login Succeeded`. Never put the token in a file,
|
||||
a command that lands in shell history, or this repo.
|
||||
|
||||
## 2. Hello-world image pull (prove the registry path)
|
||||
|
||||
```bash
|
||||
docker pull nginxinc/nginx-unprivileged:alpine
|
||||
docker tag nginxinc/nginx-unprivileged:alpine "$REGISTRY/$OWNER/hello:gate1"
|
||||
docker push "$REGISTRY/$OWNER/hello:gate1"
|
||||
|
||||
kubectl -n "$NS" create secret docker-registry gitea-regcred \
|
||||
--docker-server="$REGISTRY" \
|
||||
--docker-username="$OWNER" \
|
||||
--docker-password="$GITEA_TOKEN"
|
||||
|
||||
kubectl -n "$NS" create deployment hello \
|
||||
--image="$REGISTRY/$OWNER/hello:gate1"
|
||||
kubectl -n "$NS" patch deployment hello --type=merge -p \
|
||||
'{"spec":{"template":{"spec":{"imagePullSecrets":[{"name":"gitea-regcred"}]}}}}'
|
||||
kubectl -n "$NS" rollout status deployment/hello --timeout=180s
|
||||
kubectl -n "$NS" get pods -l app=hello
|
||||
```
|
||||
|
||||
Pass: the pod is `Running` and `kubectl describe pod` shows the image pulled.
|
||||
Clean up before the real deploy:
|
||||
|
||||
```bash
|
||||
kubectl -n "$NS" delete deployment hello
|
||||
```
|
||||
|
||||
If this fails, fix the token/secret before building the application images.
|
||||
|
||||
## 3. Create the remaining Secrets
|
||||
|
||||
Generate values locally; never commit them.
|
||||
|
||||
```bash
|
||||
read -rsp 'Control password: ' CONTROL_PASS; echo
|
||||
kubectl -n "$NS" create secret generic control-auth \
|
||||
--from-literal=CONTROL_USER=control \
|
||||
--from-literal=CONTROL_PASS="$CONTROL_PASS"
|
||||
|
||||
# Token from Cloudflare Zero Trust → Networks → Tunnels → your tunnel.
|
||||
read -rsp 'Cloudflare tunnel token: ' TUNNEL_TOKEN; echo
|
||||
kubectl -n "$NS" create secret generic cloudflared \
|
||||
--from-literal=TUNNEL_TOKEN="$TUNNEL_TOKEN"
|
||||
|
||||
kubectl -n "$NS" get secrets
|
||||
```
|
||||
|
||||
`make secret-commands` prints the same commands with placeholders.
|
||||
|
||||
## 4. Build and push the three images
|
||||
|
||||
The manifests expect:
|
||||
|
||||
| Service | Image | Port | Health paths |
|
||||
|---|---|---|---|
|
||||
| web | `git.noelsabia.dev/noelsabia/pulsecast-web:gate1` | 3000 | `/` |
|
||||
| api | `git.noelsabia.dev/noelsabia/pulsecast-api:gate1` | 8080 | `/api/healthz`, `/api/readyz` |
|
||||
| ml | `git.noelsabia.dev/noelsabia/pulsecast-ml:gate1` | 8000 | `/healthz`, `/readyz` |
|
||||
|
||||
```bash
|
||||
make build # builds all three from the service worktrees
|
||||
make push # docker push to Gitea
|
||||
```
|
||||
|
||||
`make build` auto-detects the service worktrees under
|
||||
`/home/noel/Documents/Hackathon_2026/worktrees/`. Override when needed:
|
||||
|
||||
```bash
|
||||
make build WEB_DIR=/path/to/web-nuxt API_DIR=/path/to/api-go ML_DIR=/path/to/ML-py
|
||||
```
|
||||
|
||||
If `OWNER` or `TAG` differs from the manifest defaults, either update
|
||||
`deploy/k8s/*.yaml` before `make deploy`, or deploy first and then run the
|
||||
guarded image update:
|
||||
|
||||
```bash
|
||||
make set-image CONFIRM=yes OWNER=<owner> TAG=<tag>
|
||||
```
|
||||
|
||||
### Dockerfile contract (coordinate through the orchestrator)
|
||||
|
||||
Each service image must:
|
||||
|
||||
- listen on `0.0.0.0` at the port above;
|
||||
- run as a non-root user (numeric UID, no shell required);
|
||||
- write only to `/tmp` (the pod root filesystem is read-only);
|
||||
- start with the environment in `deploy/k8s/00-config.yaml`;
|
||||
- be self-contained: no runtime download of data, model, fonts, or packages.
|
||||
|
||||
`api` must read `DATA_DIR` (baked JSON), `FASTAPI_URL`, `PORT`,
|
||||
`CONTROL_USER`, `CONTROL_PASS`, `SSE_HEARTBEAT`, and `ML_TIMEOUT`.
|
||||
`web` must have `NUXT_PUBLIC_API_BASE=https://api.pulse.noelsabia.dev` baked in or
|
||||
provided at runtime; `ml` must find its model and artifacts under the paths its
|
||||
Dockerfile defines. The env names in the manifests are the integration point:
|
||||
change them here first if a service needs a different name.
|
||||
|
||||
## 5. Cloudflare Tunnel
|
||||
|
||||
1. Zero Trust → **Networks → Tunnels → Create a tunnel**.
|
||||
2. Connector: **Cloudflared**. Name: `pulsecast-gate1`.
|
||||
3. Copy the tunnel token (it is shown once). Store it in the `cloudflared`
|
||||
Secret (step 3).
|
||||
4. Add public hostnames:
|
||||
|
||||
| Public hostname | Service |
|
||||
|---|---|
|
||||
| `pulse.noelsabia.dev` | `http://web:3000` |
|
||||
| `api.pulse.noelsabia.dev` | `http://api:8080` |
|
||||
|
||||
The hostnames must be in the Cloudflare zone for `noelsabia.dev`. The
|
||||
service URL uses the Kubernetes Service name and port inside the namespace;
|
||||
no Ingress is involved.
|
||||
5. For the first smoke test before the app images exist, temporarily point
|
||||
`pulse.noelsabia.dev` at `http://hello:8080`, then switch it to
|
||||
`http://web:3000` after step 7.
|
||||
|
||||
DNS records for the public hostnames are created by Cloudflare; do not add
|
||||
manual A/AAAA records for a tunnel hostname.
|
||||
|
||||
## 6. Validate before applying
|
||||
|
||||
```bash
|
||||
make validate
|
||||
```
|
||||
|
||||
This is read-only:
|
||||
|
||||
- `kubeconform -strict` validates the resource schemas if installed;
|
||||
- otherwise, if a cluster is reachable, `kubectl apply --dry-run=client`
|
||||
validates against the API server;
|
||||
- if neither is available it parses the YAML and prints a warning that schema
|
||||
validation was skipped.
|
||||
|
||||
Never run `kubectl apply` / `delete` from an agent session. The commands below
|
||||
are for the human.
|
||||
|
||||
## 7. Deploy
|
||||
|
||||
```bash
|
||||
make deploy-check # read-only: required Secrets exist
|
||||
make deploy CONFIRM=yes # applies deploy/k8s and waits for rollouts
|
||||
```
|
||||
|
||||
`make deploy` fails closed: without `CONFIRM=yes` it changes nothing. It does
|
||||
not create the namespace or secrets.
|
||||
|
||||
If the three service images are not available yet, deploy only the tunnel and
|
||||
use the hello service to prove public HTTPS; return to this step when the real
|
||||
images exist. Do not deploy a placeholder image under the real service names.
|
||||
|
||||
Check the result:
|
||||
|
||||
```bash
|
||||
kubectl -n "$NS" get deploy,pods,svc
|
||||
kubectl -n "$NS" get pods -o wide
|
||||
kubectl -n "$NS" logs deployment/cloudflared --tail=50 | grep 'Registered tunnel connection'
|
||||
```
|
||||
|
||||
Expected: `web`, `api`, `ml`, `cloudflared` each `1/1`; Services `web`, `api`,
|
||||
`ml` are `ClusterIP`; `cloudflared` has no Service (it dials out).
|
||||
|
||||
## 8. Verify
|
||||
|
||||
Public checks (also runnable from a phone on mobile data):
|
||||
|
||||
```bash
|
||||
curl -fsS -o /dev/null -w 'web %{http_code}\n' https://pulse.noelsabia.dev/
|
||||
curl -fsS -o /dev/null -w 'api %{http_code}\n' https://api.pulse.noelsabia.dev/api/healthz
|
||||
curl -fsS -o /dev/null -w 'ready %{http_code}\n' https://api.pulse.noelsabia.dev/api/readyz
|
||||
curl -fsS https://api.pulse.noelsabia.dev/api/meta | jq '{data_version,model_version}'
|
||||
curl -sS -N --max-time 20 https://api.pulse.noelsabia.dev/api/stream | head -20
|
||||
```
|
||||
|
||||
The SSE output must start with `retry: 3000`, contain a `scenario` event
|
||||
immediately, and show a `ping` within 20 s.
|
||||
|
||||
Full smoke test, including the ML service through a temporary
|
||||
`kubectl port-forward`:
|
||||
|
||||
```bash
|
||||
make smoke SMOKE_SSE=1
|
||||
```
|
||||
|
||||
Local variant once the three dev servers are running:
|
||||
|
||||
```bash
|
||||
make smoke-local
|
||||
```
|
||||
|
||||
In-cluster connectivity (optional; creates and removes one short-lived pod):
|
||||
|
||||
```bash
|
||||
kubectl -n "$NS" run gate1-netcheck --rm -it --restart=Never \
|
||||
--image=curlimages/curl:latest --command -- sh -c \
|
||||
'curl -fsS http://api:8080/api/healthz && \
|
||||
curl -fsS http://ml:8000/healthz && \
|
||||
curl -fsS http://web:3000/ >/dev/null && echo connectivity-ok'
|
||||
```
|
||||
|
||||
TLS is verified by `make smoke` (`ssl_verify_result=0`). Certificates are
|
||||
issued by Cloudflare at the edge; there is no cert-manager resource.
|
||||
|
||||
## 9. Record the Gate 1 result
|
||||
|
||||
Write down, in the status note or README:
|
||||
|
||||
- deploy time and commit hash;
|
||||
- image tags actually deployed (`kubectl -n "$NS" get deploy -o jsonpath=...`);
|
||||
- output of `make smoke SMOKE_SSE=1`;
|
||||
- public URLs and the device/network used;
|
||||
- any warning that remains.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Check | Fix |
|
||||
|---|---|---|
|
||||
| `ImagePullBackOff` / `ErrImagePull` | `kubectl -n "$NS" describe pod <pod>`; `kubectl -n "$NS" get secret gitea-regcred` | Token is read-only package scoped; image tag exists; secret is in the same namespace. |
|
||||
| `CreateContainerConfigError` | `kubectl -n "$NS" describe pod <pod>` | A referenced Secret key is missing (`control-auth` or `cloudflared`). Recreate the Secret. |
|
||||
| `cloudflared` CrashLoopBackOff | `kubectl -n "$NS" logs deployment/cloudflared --previous` | Invalid/expired tunnel token, or the tunnel was deleted in Cloudflare. Recreate the Secret and restart. |
|
||||
| Public 502 / 1033 | `kubectl -n "$NS" get endpoints web api`; cloudflared logs | Service has no ready endpoints, wrong port, or hostname route points at the wrong service. |
|
||||
| `api` pod not ready | `kubectl -n "$NS" logs deployment/api`; `kubectl -n "$NS" exec deploy/api -- ls /app/data` | Baked JSON missing from the image or `DATA_DIR` wrong. Rebuild the API image with the artifacts. |
|
||||
| `ml` pod stuck in startup | `kubectl -n "$NS" logs deployment/ml` | Model load exceeds 120 s or the model path is wrong. Fix the image/path; startupProbe allows 120 s. |
|
||||
| `web` 500 | `kubectl -n "$NS" logs deployment/web` | Missing build-time API base or server error. Rebuild with `NUXT_PUBLIC_API_BASE`. |
|
||||
| SSE events arrive only after a delay | `curl -N` direct to the pod vs through Cloudflare | Cloudflare buffering; the web client falls back to polling `/api/scenario` every 5 s. Keep the heartbeat at 15 s. |
|
||||
| `make deploy-check` fails | `kubectl -n "$NS" get secrets` | Create the missing Secret; do not commit it. |
|
||||
|
||||
## Rollback
|
||||
|
||||
No cluster-scoped resources are created, so rollback is namespace-local:
|
||||
|
||||
```bash
|
||||
kubectl -n "$NS" rollout undo deployment/web
|
||||
kubectl -n "$NS" rollout undo deployment/api
|
||||
kubectl -n "$NS" rollout undo deployment/ml
|
||||
kubectl -n "$NS" rollout undo deployment/cloudflared
|
||||
kubectl -n "$NS" rollout status deployment/web --timeout=180s
|
||||
```
|
||||
|
||||
If the tunnel token or route is wrong, fix the Secret / Cloudflare dashboard
|
||||
and restart `cloudflared`; do not create an Ingress as a workaround.
|
||||
|
||||
## Security notes
|
||||
|
||||
- Never commit the kubeconfig, tunnel token, control password, Gitea token, or
|
||||
any `.env` file. `.gitignore` covers the usual filenames; check
|
||||
`git status` before every commit.
|
||||
- Rotate the Gitea package token and the control password after the event.
|
||||
Rotate the service-account token after the event; it was shared in chat.
|
||||
- The kubeconfig must stay at mode `600` and outside the repos.
|
||||
- `make smoke` prints no secrets; do not add `set -x` to the scripts with a
|
||||
live `KUBECONFIG`.
|
||||
Reference in new issue
Block a user