Files
Noel Sabia 1ff3bc8290
release-please / release-please (push) Failing after 3s
docs(mission): add weather and holiday context
2026-11-13 20:27:06 +01:00

15 KiB
Raw Permalink Blame History

Architecture — namespace-only, three services, Cloudflare Tunnel

Recommendation in one line

Run three namespace-scoped Deployments — Nuxt SSR web, Go API/SSE, Python FastAPI ML — behind a Cloudflare Tunnel, with the model and data baked into images, no database, no Ingress, no cert-manager, no external-dns, and a cached fallback in Go if FastAPI is unavailable.

What the kubeconfig actually allows

Verified against fch26-onemanarmy-sa-kubeconfig.yaml on 2026-11-13:

Can create Cannot create
Deployments, Services, ConfigMaps, Secrets, ServiceAccounts, Pods, PVCs Ingress, Jobs, CronJobs, NetworkPolicies, Roles, RoleBindings
  • No cert-manager CRDs, no Gateway API, no IngressClass visibility.
  • No ResourceQuota or LimitRange — three small pods fit.
  • Public image pull works; a non-root nginx-unprivileged pod ran successfully.
  • Exposure: Cloudflare Tunnel, because Ingress is not permitted.
  • Batch: build-time, because Jobs/CronJobs are not permitted.
  • TLS: terminated at Cloudflare; no cert-manager needed.
  • DNS: Cloudflare Tunnel public hostnames; no external-dns needed.

System

                    ┌────────────────────────────────────────────┐
  phones / laptop   │  Nuxt 4 SSR (Onyx, Chart.js, MapLibre)     │
  pulse.noelsabia   │  Node server, server-rendered shell        │
                    └───────────────┬────────────────────────────┘
                                    │ HTTPS
                    ┌───────────────▼────────────────────────────┐
                    │  Cloudflare edge + Tunnel                   │
                    │  pulse.noelsabia.dev     -> web:3000        │
                    │  api.pulse.noelsabia.dev -> api:8080        │
                    └───────────────┬────────────────────────────┘
                                    │ outbound from cloudflared pod
                    ┌───────────────▼────────────────────────────┐
                    │  cloudflared Deployment (namespace-scoped)  │
                    └───────┬───────────────────────┬────────────┘
                            │                       │
              ┌─────────────▼────────────┐  ┌───────▼────────────────┐
              │  Go API + SSE + cache    │  │  Nuxt SSR server       │
              │  scenario state (1 pod)  │  │  calls api:8080        │
              └─────────────┬────────────┘  └────────────────────────┘
                            │ internal HTTP
              ┌─────────────▼────────────┐
              │  FastAPI ML service      │
              │  /predict /simulate      │
              │  /healthz /meta          │
              └──────────────────────────┘

Build-time:

CSV ──► Python batch (pandas, sklearn) ──► model + JSON forecasts
                                              │
                    ┌─────────────────────────┴──────────────────┐
                    │ baked into ml image                        │
                    │ copied into Go image as fallback JSON      │
                    └────────────────────────────────────────────┘

Why this shape

Choice Why Cost
Nuxt SSR Fast first paint on phones; you know Nuxt; Onyx SSR was tested and works. A Node Deployment; hydration and SSE care.
Go API Bonus stack; small binary; SSE fan-out; scenario state; cache. One more service to deploy.
FastAPI Your requested Python service; model inference and scenario queries. Third Deployment; must not be a single point of failure.
Cloudflare Tunnel The only exposure path the namespace allows; free TLS; no Ingress, no cert-manager. One cloudflared pod and a tunnel token.
Baked model/data No PVC, no database, no runtime data fetch. Rebuild image to update data/model. Fine for 24 h.
Cached fallback in Go If FastAPI is down, the core screens still work from precomputed JSON. Slight duplication of data. Worth it.
One API replica SSE state is in memory; 120 clients fit. No HA. Documented.

Stack

  • Frontend: Nuxt 4.6.4, ssr: true, @sit-onyx/nuxt 1.2.0, sit-onyx 1.22.0, @sit-onyx/chartjs-plugin 1.0.1, Chart.js, MapLibre GL JS (static basemap fallback).
  • Backend: Go 1.27+, net/http; chi optional. SSE hub, scenario state, cache, health/ready.
  • ML: Python 3.12, pandas, scikit-learn (HistGradientBoostingRegressor), FastAPI/Uvicorn. Horizons: 15 min, 1 h, 48 h, 7 d. Weather from Open-Meteo; German/Baden-Württemberg public holidays as a feature and dashboard context.
  • Registry: Gitea at git.noelsabia.dev, Docker Registry v2, Let's Encrypt certificate. imagePullSecret in the namespace. Public Docker Hub pull also works, so cloudflare/cloudflared can be used directly.
  • K8s: three Deployments + cloudflared, four Services, ConfigMaps/Secrets, no PVC, no CRDs, no cluster-scoped resources.
  • TLS/DNS: Cloudflare Tunnel public hostnames. No cert-manager, no external-dns, no Ingress.

Cloudflare Tunnel setup

  1. Create a tunnel in the Cloudflare dashboard (Zero Trust → Networks → Tunnels).
  2. Add two public hostnames:
    • pulse.noelsabia.dev → http://web:3000
    • api.pulse.noelsabia.dev → http://api:8080 (If the tunnel UI supports path routing, pulse.noelsabia.dev/api → http://api:8080 is also possible; verify. Two hostnames are the reliable default.)
  3. Store the tunnel token as a Secret:
    kubectl -n fch26-onemanarmy create secret generic cloudflared \
      --from-literal=TUNNEL_TOKEN='<token>'
    
  4. Deploy cloudflare/cloudflared:latest with tunnel --no-autoupdate run --token $(TUNNEL_TOKEN).
  5. Verify https://pulse.noelsabia.dev and https://api.pulse.noelsabia.dev/api/healthz.
  6. Test SSE through Cloudflare; heartbeat every 15 s. If buffering appears, fall back to polling.

CORS: if you use two hostnames, Go must allow Origin: https://pulse.noelsabia.dev and the Authorization header for /control. The browser uses https://api.pulse.noelsabia.dev as the API base. If you use path routing, same-origin and no CORS.

Registry

docker login git.noelsabia.dev -u <user>
docker tag <image> git.noelsabia.dev/<owner>/<package>:<tag>
docker push git.noelsabia.dev/<owner>/<package>:<tag>

kubectl -n fch26-onemanarmy create secret docker-registry gitea-regcred \
  --docker-server=git.noelsabia.dev \
  --docker-username=<user> \
  --docker-password=<read-only-package-token>

Reference gitea-regcred in each Deployment or the default ServiceAccount. Verify with a hello-world before building the real images. Public Docker Hub pull works as a fallback for cloudflared.

Quotas and Pod Security

  • No ResourceQuota or LimitRange was visible; three small pods fit.
  • Still set requests/limits: web 100m/256Mi, api 100m/128Mi, ml 200m/512Mi, cloudflared 50m/64Mi.
  • Run as non-root; drop all capabilities; allowPrivilegeEscalation: false; read-only root filesystem with an emptyDir at /tmp.
  • The namespace has no explicit PSA labels; a non-root nginx-unprivileged pod ran. Design for restricted anyway.
  • No NetworkPolicy permission; assume default allow. Test web → api and api → ml after deployment. If a default-deny policy blocks traffic, escalate; you cannot create NetworkPolicies.

API contract

Go API

GET  /api/healthz                  -> 200 {"ok":true}
GET  /api/readyz                   -> 200 if data loaded
GET  /api/meta                     -> data/model versions, coverage, metrics
GET  /api/sensors                  -> [{id,name,lat,lon,mean,p95,battery}]
GET  /api/series?sensor=&from=&to= -> [{t,v}]
GET  /api/context                  -> {weather:{now,forecast[]}, holidays:[], updated_at}
GET  /api/forecast?sensor=&horizon=15m|1h|48h|7d&scenario= -> [{t,v,lo,hi}]
GET  /api/staffing?sensor=&scenario=&standard= -> {hours:[...],summary:{...}}
POST /api/scenario                 -> 202 {version, scenario}
GET  /api/stream                   -> text/event-stream; events: scenario, pulse
  • scenario enum: base, sunny-saturday-market, rainy-saturday, weekday-event; standard: relaxed|standard|premium.
  • Go never computes the model. It calls FastAPI or serves the cached JSON.
  • Every response carries data_version, model_version and context_version; the UI shows "last updated" for each.
  • SSE: event: scenario\ndata: {...}\n\n, heartbeat 15 s, retry: 3000.
  • Plan generation: GET /api/staffing returns a complete next-week plan in under 2 seconds; the heuristic is deterministic and cached by sensor|scenario|standard.

FastAPI ML service

GET  /healthz                      -> 200
GET  /readyz                       -> 200 if model loaded
GET  /meta                         -> model version, metrics, training window, context version
POST /predict                      -> {sensor, horizon, scenario, weather, is_holiday} -> forecast
POST /simulate                     -> {scenario, weather, is_holiday} -> forecasts for all sensors/horizons
  • Go calls /simulate on a scenario change; caches by scenario|standard|weather|holiday.
  • If FastAPI times out (> 1.5 s) or is unhealthy, Go serves the cached JSON and marks the UI "cached forecast."
  • The batch job is build-time; there is no Job/CronJob permission. A production path can use a CronJob in a less restricted namespace; document it, do not depend on it.

Context features: weather and holidays

The dashboard shows the latest Heilbronn weather and the German/Baden-Württemberg public holidays from the past. The ML service uses the same context as features, and the what-if scenarios can change the weather.

Open-Meteo historical ──► training/validation features
Open-Meteo current/7d  ──► /api/context + dashboard cards + scenario input
BW holiday calendar    ──► is_holiday feature + timeline badges
                                      │
                                      ▼
                         FastAPI /predict + /simulate
                                      │
                                      ▼
                         Go cache ──► Nuxt SSR dashboard
  • Weather source: Open-Meteo archive for training; forecast API for the dashboard. No API key.
  • Weather fields: temperature, precipitation, cloud cover, wind, humidity. Show all; use temperature/rain as model features (cloud/wind did not improve the short test).
  • Holidays: 2025–2026 Baden-Württemberg public holidays; is_holiday feature plus holiday_name for display.
  • Dashboard: current conditions, 7-day weather strip, holiday badges on the timeline, and a "context" card that explains spikes.
  • Reliability: Go caches the last /api/context payload. If Open-Meteo is unreachable, the dashboard shows "last updated" and the forecast still renders. No runtime weather call is required for the demo. If the context is newer than the forecast, the UI labels it "context updated; forecast cached."
  • Honesty: weather is the main driver; holidays are context and a small feature. Do not claim a large holiday accuracy lift.

Horizon semantics

Horizon Use Data reality
15 min "Now" / replay pulse One-step nowcast; needs a live/replay clock.
1 h "Now" / short operational 4 steps; useful for immediate staffing.
48 h Plan Day-ahead; weather-sensitive.
7 d Plan Weekly shift plan; the hero.

The CSV is static. Without a live feed, 15 min/1 h are replay/nowcast features, not live sensor data. Label them accordingly.

Failure modes

Failure Detection Fallback
No SKE access No kubeconfig by Sat 09:00 Escalate to contact; build locally; manifests ready.
Gitea pull fails ImagePullBackOff Check gitea-regcred, DNS, NetworkPolicy; public Docker Hub pull works as fallback.
Cloudflare Tunnel down URL 502 / no tunnel Restart cloudflared; use a temporary kubectl port-forward for the demo? No — use a backup tunnel or local fallback.
SSE buffered No events on phone Poll /api/scenario every 5 s; keep UI.
FastAPI down /readyz fails Go serves cached JSON; banner "cached forecast."
Open-Meteo down /api/context stale Go serves cached context; show "last updated"; forecast unaffected.
Nuxt SSR down 502 on / Restart pod; local fallback; recorded video.
No Jobs/CronJobs Batch cannot run in-cluster Build-time pipeline; document production path.
No NetworkPolicy permission Cannot fix a default-deny Test early; escalate to the platform.
Venue network Phones cannot load Mobile data; laptop local build; video.
Single API pod restarts Brief 502 Probes, small image, Recreate; SSE clients reconnect.

Security and privacy

  • No PII, no MAC addresses, no geolocation permission; audience view uses a random session ID.
  • /control uses basic auth (Go middleware) or a token; public dashboard and /live stay open. No SSO.
  • All images run as non-root. No privileged containers.
  • Secrets: gitea-regcred, cloudflared token, control credentials. Never in git.
  • Same-origin or explicit CORS; no third-party trackers; bundle fonts/assets.
  • UI states the data limitation: "estimated devices, not people; 8 sensors; 5 months."
  • Cluster credential: the provided kubeconfig is a secret. It is git-ignored and chmod 600. Rotate the token after the event; it was shared in chat.

Degradation ladder

  1. Full: Nuxt SSR + Go + FastAPI + cloudflared.
  2. Two services: Nuxt SPA served by Go + FastAPI; drop SSR if resources/time are tight.
  3. One service: Go serves SPA, API, SSE and cached forecasts; Python is build-time only.
  4. No tunnel: temporary kubectl port-forward for local verification only; not a public demo.
  5. No SKE: local kind + recorded demo; escalate; do not pretend it meets the requirement.

Open decisions

  1. Q1: settled — staffing copilot.
  2. SSR: settled — yes.
  3. Replay vs static for 15 min/1 h — default: replay P1, static P0.
  4. Paper: settled — hackathon deliverable, 4–5 pages, AI draft + human verification.
  5. Herdr: settled — four workstreams/worktrees.
  6. Basic auth scope: /control only (default) vs whole app (would hurt the 120-phone demo).
  7. Weather: use Open-Meteo/holidays (default) or only the provided CSV. See paper/data implications.