Files
OKF_Vault/mission/architecture.md
T
Noel Sabia 1ff3bc8290
release-please / release-please (push) Failing after 3s
docs(mission): add weather and holiday context
2026-11-13 20:27:06 +01:00

240 lines
15 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Architecture — namespace-only, three services, Cloudflare Tunnel
## Recommendation in one line
Run **three namespace-scoped Deployments** — Nuxt SSR web, Go API/SSE, Python FastAPI ML — behind a **Cloudflare Tunnel**, with the model and data baked into images, no database, no Ingress, no cert-manager, no external-dns, and a cached fallback in Go if FastAPI is unavailable.
## What the kubeconfig actually allows
Verified against `fch26-onemanarmy-sa-kubeconfig.yaml` on 2026-11-13:
| Can create | Cannot create |
|---|---|
| Deployments, Services, ConfigMaps, Secrets, ServiceAccounts, Pods, PVCs | **Ingress**, **Jobs**, **CronJobs**, NetworkPolicies, Roles, RoleBindings |
- No cert-manager CRDs, no Gateway API, no `IngressClass` visibility.
- No ResourceQuota or LimitRange — three small pods fit.
- Public image pull works; a non-root `nginx-unprivileged` pod ran successfully.
- **Exposure:** Cloudflare Tunnel, because Ingress is not permitted.
- **Batch:** build-time, because Jobs/CronJobs are not permitted.
- **TLS:** terminated at Cloudflare; no cert-manager needed.
- **DNS:** Cloudflare Tunnel public hostnames; no external-dns needed.
## System
```text
┌────────────────────────────────────────────┐
phones / laptop │ Nuxt 4 SSR (Onyx, Chart.js, MapLibre) │
pulse.noelsabia │ Node server, server-rendered shell │
└───────────────┬────────────────────────────┘
│ HTTPS
┌───────────────▼────────────────────────────┐
│ Cloudflare edge + Tunnel │
│ pulse.noelsabia.dev -> web:3000 │
│ api.pulse.noelsabia.dev -> api:8080 │
└───────────────┬────────────────────────────┘
│ outbound from cloudflared pod
┌───────────────▼────────────────────────────┐
│ cloudflared Deployment (namespace-scoped) │
└───────┬───────────────────────┬────────────┘
│ │
┌─────────────▼────────────┐ ┌───────▼────────────────┐
│ Go API + SSE + cache │ │ Nuxt SSR server │
│ scenario state (1 pod) │ │ calls api:8080 │
└─────────────┬────────────┘ └────────────────────────┘
│ internal HTTP
┌─────────────▼────────────┐
│ FastAPI ML service │
│ /predict /simulate │
│ /healthz /meta │
└──────────────────────────┘
```
**Build-time:**
```text
CSV ──► Python batch (pandas, sklearn) ──► model + JSON forecasts
│
┌─────────────────────────┴──────────────────┐
│ baked into ml image │
│ copied into Go image as fallback JSON │
└────────────────────────────────────────────┘
```
## Why this shape
| Choice | Why | Cost |
|---|---|---|
| Nuxt SSR | Fast first paint on phones; you know Nuxt; Onyx SSR was tested and works. | A Node Deployment; hydration and SSE care. |
| Go API | Bonus stack; small binary; SSE fan-out; scenario state; cache. | One more service to deploy. |
| FastAPI | Your requested Python service; model inference and scenario queries. | Third Deployment; must not be a single point of failure. |
| Cloudflare Tunnel | The only exposure path the namespace allows; free TLS; no Ingress, no cert-manager. | One `cloudflared` pod and a tunnel token. |
| Baked model/data | No PVC, no database, no runtime data fetch. | Rebuild image to update data/model. Fine for 24 h. |
| Cached fallback in Go | If FastAPI is down, the core screens still work from precomputed JSON. | Slight duplication of data. Worth it. |
| One API replica | SSE state is in memory; 120 clients fit. | No HA. Documented. |
## Stack
- **Frontend:** Nuxt 4.6.4, `ssr: true`, `@sit-onyx/nuxt` 1.2.0, `sit-onyx` 1.22.0, `@sit-onyx/chartjs-plugin` 1.0.1, Chart.js, MapLibre GL JS (static basemap fallback).
- **Backend:** Go 1.27+, `net/http`; `chi` optional. SSE hub, scenario state, cache, health/ready.
- **ML:** Python 3.12, pandas, scikit-learn (`HistGradientBoostingRegressor`), FastAPI/Uvicorn. Horizons: 15 min, 1 h, 48 h, 7 d. Weather from Open-Meteo; German/Baden-Württemberg public holidays as a feature and dashboard context.
- **Registry:** Gitea at `git.noelsabia.dev`, Docker Registry v2, Let's Encrypt certificate. `imagePullSecret` in the namespace. Public Docker Hub pull also works, so `cloudflare/cloudflared` can be used directly.
- **K8s:** three Deployments + `cloudflared`, four Services, ConfigMaps/Secrets, no PVC, no CRDs, no cluster-scoped resources.
- **TLS/DNS:** Cloudflare Tunnel public hostnames. No cert-manager, no external-dns, no Ingress.
## Cloudflare Tunnel setup
1. Create a tunnel in the Cloudflare dashboard (Zero Trust → Networks → Tunnels).
2. Add two public hostnames:
- `pulse.noelsabia.dev` → `http://web:3000`
- `api.pulse.noelsabia.dev` → `http://api:8080`
(If the tunnel UI supports path routing, `pulse.noelsabia.dev/api` → `http://api:8080` is also possible; verify. Two hostnames are the reliable default.)
3. Store the tunnel token as a Secret:
```bash
kubectl -n fch26-onemanarmy create secret generic cloudflared \
--from-literal=TUNNEL_TOKEN='<token>'
```
4. Deploy `cloudflare/cloudflared:latest` with `tunnel --no-autoupdate run --token $(TUNNEL_TOKEN)`.
5. Verify `https://pulse.noelsabia.dev` and `https://api.pulse.noelsabia.dev/api/healthz`.
6. Test SSE through Cloudflare; heartbeat every 15 s. If buffering appears, fall back to polling.
**CORS:** if you use two hostnames, Go must allow `Origin: https://pulse.noelsabia.dev` and the `Authorization` header for `/control`. The browser uses `https://api.pulse.noelsabia.dev` as the API base. If you use path routing, same-origin and no CORS.
## Registry
```bash
docker login git.noelsabia.dev -u <user>
docker tag <image> git.noelsabia.dev/<owner>/<package>:<tag>
docker push git.noelsabia.dev/<owner>/<package>:<tag>
kubectl -n fch26-onemanarmy create secret docker-registry gitea-regcred \
--docker-server=git.noelsabia.dev \
--docker-username=<user> \
--docker-password=<read-only-package-token>
```
Reference `gitea-regcred` in each Deployment or the default ServiceAccount. Verify with a hello-world before building the real images. Public Docker Hub pull works as a fallback for `cloudflared`.
## Quotas and Pod Security
- No ResourceQuota or LimitRange was visible; three small pods fit.
- Still set requests/limits: web 100m/256Mi, api 100m/128Mi, ml 200m/512Mi, cloudflared 50m/64Mi.
- Run as non-root; drop all capabilities; `allowPrivilegeEscalation: false`; read-only root filesystem with an `emptyDir` at `/tmp`.
- The namespace has no explicit PSA labels; a non-root `nginx-unprivileged` pod ran. Design for restricted anyway.
- No NetworkPolicy permission; assume default allow. Test web → api and api → ml after deployment. If a default-deny policy blocks traffic, escalate; you cannot create NetworkPolicies.
## API contract
### Go API
```text
GET /api/healthz -> 200 {"ok":true}
GET /api/readyz -> 200 if data loaded
GET /api/meta -> data/model versions, coverage, metrics
GET /api/sensors -> [{id,name,lat,lon,mean,p95,battery}]
GET /api/series?sensor=&from=&to= -> [{t,v}]
GET /api/context -> {weather:{now,forecast[]}, holidays:[], updated_at}
GET /api/forecast?sensor=&horizon=15m|1h|48h|7d&scenario= -> [{t,v,lo,hi}]
GET /api/staffing?sensor=&scenario=&standard= -> {hours:[...],summary:{...}}
POST /api/scenario -> 202 {version, scenario}
GET /api/stream -> text/event-stream; events: scenario, pulse
```
- `scenario` enum: `base`, `sunny-saturday-market`, `rainy-saturday`, `weekday-event`; `standard`: `relaxed|standard|premium`.
- Go never computes the model. It calls FastAPI or serves the cached JSON.
- Every response carries `data_version`, `model_version` and `context_version`; the UI shows "last updated" for each.
- SSE: `event: scenario\ndata: {...}\n\n`, heartbeat 15 s, `retry: 3000`.
- **Plan generation:** `GET /api/staffing` returns a complete next-week plan in under 2 seconds; the heuristic is deterministic and cached by `sensor|scenario|standard`.
### FastAPI ML service
```text
GET /healthz -> 200
GET /readyz -> 200 if model loaded
GET /meta -> model version, metrics, training window, context version
POST /predict -> {sensor, horizon, scenario, weather, is_holiday} -> forecast
POST /simulate -> {scenario, weather, is_holiday} -> forecasts for all sensors/horizons
```
- Go calls `/simulate` on a scenario change; caches by `scenario|standard|weather|holiday`.
- If FastAPI times out (> 1.5 s) or is unhealthy, Go serves the cached JSON and marks the UI "cached forecast."
- The batch job is build-time; there is no Job/CronJob permission. A production path can use a CronJob in a less restricted namespace; document it, do not depend on it.
## Context features: weather and holidays
The dashboard shows the latest Heilbronn weather and the German/Baden-Württemberg public holidays from the past. The ML service uses the same context as features, and the what-if scenarios can change the weather.
```text
Open-Meteo historical ──► training/validation features
Open-Meteo current/7d ──► /api/context + dashboard cards + scenario input
BW holiday calendar ──► is_holiday feature + timeline badges
│
▼
FastAPI /predict + /simulate
│
▼
Go cache ──► Nuxt SSR dashboard
```
- **Weather source:** Open-Meteo archive for training; forecast API for the dashboard. No API key.
- **Weather fields:** temperature, precipitation, cloud cover, wind, humidity. Show all; use temperature/rain as model features (cloud/wind did not improve the short test).
- **Holidays:** 2025–2026 Baden-Württemberg public holidays; `is_holiday` feature plus `holiday_name` for display.
- **Dashboard:** current conditions, 7-day weather strip, holiday badges on the timeline, and a "context" card that explains spikes.
- **Reliability:** Go caches the last `/api/context` payload. If Open-Meteo is unreachable, the dashboard shows "last updated" and the forecast still renders. No runtime weather call is required for the demo. If the context is newer than the forecast, the UI labels it "context updated; forecast cached."
- **Honesty:** weather is the main driver; holidays are context and a small feature. Do not claim a large holiday accuracy lift.
## Horizon semantics
| Horizon | Use | Data reality |
|---|---|---|
| 15 min | "Now" / replay pulse | One-step nowcast; needs a live/replay clock. |
| 1 h | "Now" / short operational | 4 steps; useful for immediate staffing. |
| 48 h | Plan | Day-ahead; weather-sensitive. |
| 7 d | Plan | Weekly shift plan; the hero. |
The CSV is static. Without a live feed, 15 min/1 h are replay/nowcast features, not live sensor data. Label them accordingly.
## Failure modes
| Failure | Detection | Fallback |
|---|---|---|
| No SKE access | No kubeconfig by Sat 09:00 | Escalate to contact; build locally; manifests ready. |
| Gitea pull fails | `ImagePullBackOff` | Check `gitea-regcred`, DNS, NetworkPolicy; public Docker Hub pull works as fallback. |
| Cloudflare Tunnel down | URL 502 / no tunnel | Restart `cloudflared`; use a temporary `kubectl port-forward` for the demo? No — use a backup tunnel or local fallback. |
| SSE buffered | No events on phone | Poll `/api/scenario` every 5 s; keep UI. |
| FastAPI down | `/readyz` fails | Go serves cached JSON; banner "cached forecast." |
| Open-Meteo down | `/api/context` stale | Go serves cached context; show "last updated"; forecast unaffected. |
| Nuxt SSR down | 502 on `/` | Restart pod; local fallback; recorded video. |
| No Jobs/CronJobs | Batch cannot run in-cluster | Build-time pipeline; document production path. |
| No NetworkPolicy permission | Cannot fix a default-deny | Test early; escalate to the platform. |
| Venue network | Phones cannot load | Mobile data; laptop local build; video. |
| Single API pod restarts | Brief 502 | Probes, small image, `Recreate`; SSE clients reconnect. |
## Security and privacy
- No PII, no MAC addresses, no geolocation permission; audience view uses a random session ID.
- `/control` uses **basic auth** (Go middleware) or a token; public dashboard and `/live` stay open. No SSO.
- All images run as non-root. No privileged containers.
- Secrets: `gitea-regcred`, `cloudflared` token, control credentials. Never in git.
- Same-origin or explicit CORS; no third-party trackers; bundle fonts/assets.
- UI states the data limitation: "estimated devices, not people; 8 sensors; 5 months."
- **Cluster credential:** the provided kubeconfig is a secret. It is git-ignored and chmod 600. Rotate the token after the event; it was shared in chat.
## Degradation ladder
1. **Full:** Nuxt SSR + Go + FastAPI + `cloudflared`.
2. **Two services:** Nuxt SPA served by Go + FastAPI; drop SSR if resources/time are tight.
3. **One service:** Go serves SPA, API, SSE and cached forecasts; Python is build-time only.
4. **No tunnel:** temporary `kubectl port-forward` for local verification only; not a public demo.
5. **No SKE:** local kind + recorded demo; escalate; do not pretend it meets the requirement.
## Open decisions
1. Q1: **settled** — staffing copilot.
2. SSR: **settled** — yes.
3. Replay vs static for 15 min/1 h — default: replay P1, static P0.
4. Paper: **settled** — hackathon deliverable, 4–5 pages, AI draft + human verification.
5. Herdr: **settled** — four workstreams/worktrees.
6. Basic auth scope: `/control` only (default) vs whole app (would hurt the 120-phone demo).
7. Weather: use Open-Meteo/holidays (default) or only the provided CSV. See paper/data implications.