diff --git a/architecture/chapter-service.md b/architecture/chapter-service.md index 7f3c210..5cc9c81 100644 --- a/architecture/chapter-service.md +++ b/architecture/chapter-service.md @@ -30,7 +30,7 @@ generated: { by: claude-code/claude-opus-5-5, at: 2026-10-11T12:15:30Z } - Services set `publishNotReadyAddresses: true`, otherwise a NotReady chapter has no endpoints and the gateway cannot push it. - Label every pod `cv.noelsabia.dev/chapter: {name}`. The gateway uses it to map pods to chapters. -- Tight resource limits on every chapter (for example 10m CPU, 16Mi memory). +- Limits per chapter: 10m CPU, 16Mi memory, `GOMEMLIMIT=12MiB`. See [server budget](/architecture/server-budget.md). Crashing chapters back off on Kubernetes' default schedule, which caps at 5 minutes. - Completed Jobs keep no `ttlSecondsAfterFinished`, so they stay visible as Completed. - A CrashLoopBackOff chapter cannot be pushed; it is already down. The UI shows its lesson card instead. diff --git a/architecture/delivery.md b/architecture/delivery.md index 423ab66..032d3e6 100644 --- a/architecture/delivery.md +++ b/architecture/delivery.md @@ -31,6 +31,7 @@ push to main ─► Gitea Actions (runner on the home server) - **Jobs are immutable:** changing a completed chapter's Job spec needs a delete before apply. Rare, since finished chapters do not change. - **Snyk:** needs a `SNYK_TOKEN` secret in Gitea. +- **Memory:** builds run on the same 4 GB server. Runner capacity 1, Node capped at 512 MB, swap as a safety net. See [server budget](/architecture/server-budget.md). # Later diff --git a/architecture/gateway.md b/architecture/gateway.md index 8866c5a..0b2c4a2 100644 --- a/architecture/gateway.md +++ b/architecture/gateway.md @@ -42,6 +42,10 @@ The same stream feeds the life map and the developer view. - Send `Content-Type: text/event-stream` and flush after every message. - Send a comment line (`: ping`) every 30 seconds so idle connections are not closed. +# Resources + +64Mi memory limit with `GOMEMLIMIT=50MiB`, 100m CPU. See [server budget](/architecture/server-budget.md). + # RBAC ServiceAccount `gateway` in `cv`, Role with `get/list/watch` on `pods`, diff --git a/architecture/index.md b/architecture/index.md index 3c70678..4353fd8 100644 --- a/architecture/index.md +++ b/architecture/index.md @@ -4,5 +4,6 @@ * [Chapter service](chapter-service.md) - One small Go binary whose health behavior is set by the MODE env var; every life chapter runs it. * [Gateway](gateway.md) - The only public service; serves the static site, streams chapter states over SSE, and forwards rate-limited knockdowns. * [Frontend](frontend.md) - Nuxt site generated to static files and served by the gateway, with three audience layers fed by one SSE stream. +* [Server and memory budget](server-budget.md) - The CV shares a 2 vCPU, 4 GB server with Gitea and a small webapp, so every workload gets a hard memory cap. * [Delivery](delivery.md) - Gitea Actions on the home server scan, build, push to the Gitea registry, deploy with kubectl, and version with release-please. * [In-browser language model](language-model.md) - A tiny language model trained by Noel that runs entirely in the visitor's browser and shows its internals live. diff --git a/architecture/language-model.md b/architecture/language-model.md index 482826b..6357afe 100644 --- a/architecture/language-model.md +++ b/architecture/language-model.md @@ -37,4 +37,5 @@ statically by the [gateway](/architecture/gateway.md). It adds nothing to the cl - **Download size:** 5M parameters in fp32 is about 20 MB, 30M is about 120 MB. Ship int8 weights (about 4× smaller) and load them lazily. +- **Training happens on Noel's own computer.** The [server](/architecture/server-budget.md) only serves the weights file. - **PyTorch is new tooling** and needs Noel's approval under the stack rule. diff --git a/architecture/server-budget.md b/architecture/server-budget.md new file mode 100644 index 0000000..f98051e --- /dev/null +++ b/architecture/server-budget.md @@ -0,0 +1,98 @@ +--- +type: Reference +title: Server and memory budget +description: The CV shares a 2 vCPU, 4 GB server with Gitea and a small webapp, so every workload gets a hard memory cap. +tags: [architecture, k3s, resources] +status: stable +generated: { by: claude-code/claude-opus-5-5, at: 2026-10-11T12:15:30Z } +sources: + - id: free-m + resource: "`free -m` on server0, before k3s, with Gitea, runner and webapp running" + author: human:noel + last_modified: 2026-10-11T12:15:30Z +--- + +# Server + +`server0`: 2 vCPU, 3805 MB RAM, no swap yet. Shared with Gitea, its Actions +runner, and a small webapp with one user. + +# Budget + +| What | Memory | Basis | +|---|---|---| +| OS, Gitea, runner (idle), webapp | 1143 MB | Measured[^free-m] | +| k3s, trimmed | ~500 MB | Estimate, measure after install | +| Namespace `cv`, hard quota | 384 MB | ResourceQuota | +| Left for CI builds and page cache | ~1.7 GB | | +| Swap | 2 GB | Safety net for build spikes | + +[^free-m]: `free -m` on server0 + +Per workload inside `cv`: + +| Workload | Memory limit | CPU limit | +|---|---|---| +| Gateway | 64Mi, `GOMEMLIMIT=50MiB` | 100m | +| Each chapter (~10) | 16Mi, `GOMEMLIMIT=12MiB` | 10m | +| Analytics ([GoatCounter](/decisions/goatcounter-over-umami.md)) | 64Mi | 50m | + +Completed Jobs do not count against the quota. + +# Namespace caps + +```yaml +apiVersion: v1 +kind: ResourceQuota +metadata: { name: cv-budget, namespace: cv } +spec: + hard: { limits.memory: 384Mi } +--- +apiVersion: v1 +kind: LimitRange +metadata: { name: cv-defaults, namespace: cv } +spec: + limits: + - type: Container + default: { memory: 16Mi, cpu: 50m } + defaultRequest: { memory: 8Mi, cpu: 5m } +``` + +# k3s config + +`/etc/rancher/k3s/config.yaml`: + +```yaml +disable: [traefik, servicelb] # Cloudflare Tunnel replaces both +kubelet-arg: + - "system-reserved=memory=1300Mi" # measured 1143 MB plus margin + - "eviction-hard=memory.available<200Mi" +``` + +Keep `metrics-server` and `local-storage`: the HPA demo needs the first, +analytics storage the second. Single-node k3s uses SQLite, not etcd; do not +pass `--cluster-init`. + +# Swap + +```sh +sudo fallocate -l 2G /swapfile && sudo chmod 600 /swapfile +sudo mkswap /swapfile && sudo swapon /swapfile +echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab +``` + +k3s starts with swap enabled. Pods do not swap; host processes such as +Gitea and builds can. + +# CI builds + +The biggest memory risk. A Nuxt build can take over 1 GB. + +- Runner capacity 1: one job at a time. +- `NODE_OPTIONS=--max-old-space-size=512` for the Nuxt build. +- Swap catches the rest. + +# What does not run here + +- **Language model training:** on Noel's own computer. The server only serves the weights file. +- **Large load tests:** k6 and the HPA demo stay tiny (max 3 replicas, a few virtual users), or are cut. diff --git a/architecture/system.md b/architecture/system.md index 14c60de..800385b 100644 --- a/architecture/system.md +++ b/architecture/system.md @@ -17,7 +17,10 @@ generated: { by: claude-code/claude-opus-5-5, at: 2026-10-11T12:15:30Z } | [Chapter service](/architecture/chapter-service.md) | One Deployment or Job per chapter | Behaves healthy, not-ready, crashing, or completed, set by env. | | [Delivery](/architecture/delivery.md) | Gitea Actions on the home server | Test, scan, build, push, deploy, version. | -Everything runs in one namespace: `cv`. +| Analytics | 1 Deployment, GoatCounter (proposed) | Page views, PDF downloads, falls per chapter. See [decision](/decisions/goatcounter-over-umami.md). | + +Everything runs in one namespace: `cv`, capped at 384Mi memory in total, on a +2 vCPU, 4 GB server shared with Gitea. See [server budget](/architecture/server-budget.md). # Images diff --git a/decisions/goatcounter-over-umami.md b/decisions/goatcounter-over-umami.md new file mode 100644 index 0000000..296f239 --- /dev/null +++ b/decisions/goatcounter-over-umami.md @@ -0,0 +1,38 @@ +--- +type: Decision +title: GoatCounter over Umami for analytics +description: Proposed; GoatCounter gives cookie-free open source analytics in about 30 MB, where Umami needs Postgres and about 300-400 MB. +tags: [decision, analytics, resources] +status: draft +generated: { by: claude-code/claude-opus-5-5, at: 2026-10-11T12:15:30Z } +--- + +# Proposal + +Self-host GoatCounter in namespace `cv` instead of Umami. Needs Noel's +approval as new tooling. + +# Why + +| | Umami | GoatCounter | +|---|---|---| +| Runtime | Node app | Single Go binary | +| Database | Postgres | SQLite file | +| Memory | ~300-400 MB with Postgres | ~30 MB | +| Cookie-free, no consent banner | Yes | Yes | +| Custom events | Yes | Yes | +| License | Open source | Open source | + +On a [4 GB shared server](/architecture/server-budget.md), Umami alone would +match the whole CV budget. + +# Track + +- Page views. +- PDF CV downloads. +- Falls per chapter. + +# Details + +One Deployment, 64Mi limit, SQLite on a small `local-path` volume. No backup: +losing analytics hurts nobody. diff --git a/decisions/index.md b/decisions/index.md index f0dd6bd..d09ba62 100644 --- a/decisions/index.md +++ b/decisions/index.md @@ -6,6 +6,10 @@ * [CI/CD on Gitea Actions](ci-on-gitea.md) - The existing Gitea instance and runner on the home server build, scan, deploy and version the project. * [Knowledge bundle lives at the repository root](bundle-at-root.md) - The repository root is the OKF bundle root; any other markdown file there needs frontmatter to stay conformant. +# Proposed + +* [GoatCounter over Umami for analytics](goatcounter-over-umami.md) - Proposed; GoatCounter gives cookie-free open source analytics in about 30 MB, where Umami needs Postgres and about 300-400 MB. + # Open * [Open questions](open-questions.md) - Decisions still to make, with the release that needs each answer. diff --git a/decisions/open-questions.md b/decisions/open-questions.md index 9c7dbaa..0d7a5f9 100644 --- a/decisions/open-questions.md +++ b/decisions/open-questions.md @@ -15,5 +15,6 @@ generated: { by: claude-code/claude-opus-5-5, at: 2026-10-11T12:15:30Z } | Which Gitea release-please action, exact name | 0.1.0 | None, Noel names it | | Staging for k6 in CI: namespace or separate cluster | 1.2.0 | Namespace `cv-staging` on the same cluster | | MCP scope: read-only or read plus `/fall` | 1.1.0 | Read plus `/fall`, same limits as visitors | +| Approve GoatCounter for analytics | 0.9.0 | None, needs Noel | | Approve PyTorch for the language model | 1.5.0 | None, needs Noel | | Approve NumPy (or similar) for the own autograd engine | 1.13.0 | None, needs Noel | diff --git a/log.md b/log.md index 46ab92d..958ed32 100644 --- a/log.md +++ b/log.md @@ -1,6 +1,7 @@ # Update Log ## 2026-10-11 +* **Creation**: Added the [server and memory budget](/architecture/server-budget.md) from measured usage, and proposed [GoatCounter over Umami](/decisions/goatcounter-over-umami.md). * **Update**: Moved the bundle from `knowledge/` to the repository root and replaced the planning brief with this bundle. See [bundle at root](/decisions/bundle-at-root.md). * **Creation**: Wrote the [overview](/overview.md), [architecture](/architecture/), [decisions](/decisions/), [chapter states](/chapters/state-mapping.md) and [roadmap](/roadmap.md) from the planning conversation. * **Initialization**: Created the bundle [index](/index.md). diff --git a/overview.md b/overview.md index a17f14d..4f70c6c 100644 --- a/overview.md +++ b/overview.md @@ -59,6 +59,7 @@ page shows the real recovery time next to the story of how Noel recovered. - **Learning project, ship small.** Each release is a few days of work. - **No tooling beyond the stack without Noel's approval.** The stack is Nuxt, Go, k3s, Snyk, Gitea Actions, release-please, Cloudflare Tunnel, and k6. +- **Small server.** 2 vCPU, 4 GB, shared with Gitea. The whole CV is capped at 384Mi. See [server budget](/architecture/server-budget.md). - **Nobody gets Kubernetes API access.** Visitors and agents only talk to the gateway. - **Failures are lessons, never bugs.** diff --git a/roadmap.md b/roadmap.md index 01f7213..6d12cc6 100644 --- a/roadmap.md +++ b/roadmap.md @@ -13,18 +13,19 @@ generated: { by: claude-code/claude-opus-5-5, at: 2026-10-11T12:15:30Z } |---|---|---| | 1: Ship the CV | 0.1.0 - 1.0.0 | Public interactive CV on the apex domain | | 2: Depth | 1.1.0 - 1.7.0 | MCP, k6 release gate, full dev terminal, in-browser language model | -| 3: Pressure and from scratch | 1.8.0 - 1.13.0 | Interactive load test with HPA, chat model, own autograd engine | +| 3: Pressure and from scratch | 1.8.0 - 1.13.0 | Tiny interactive load test with HPA, chat model, own autograd engine | | Release | Name | Days | |---|---|---| | 0.1.0 | Chapter service | 1-2 | -| 0.2.0 | Chapters on k3s | 1-2 | +| 0.2.0 | Chapters on k3s | 2 | | 0.3.0 | Pipeline | 2 | | 0.4.0 | Live states | 2-3 | | 0.5.0 | Knockdown | 1-2 | | 0.6.0 | First screen | 2 | | 0.7.0 | Life map | 3 | | 0.8.0 | Story cards and dev view | 2 | +| 0.9.0 | Analytics | 1 | | 1.0.0 | Real content, launch | 2-3 | | 1.1.0 | MCP endpoint | 1 | | 1.2.0 | k6 release gate | 2 | @@ -43,6 +44,9 @@ generated: { by: claude-code/claude-opus-5-5, at: 2026-10-11T12:15:30Z } Each release is cut by release-please from `feat:` commits. A release is done when its "done when" check passes on the real cluster, not locally. +Everything runs on a 2 vCPU, 4 GB server shared with Gitea. Every release +stays inside the [memory budget](/architecture/server-budget.md). + --- # Phase 1: Ship the CV @@ -58,16 +62,18 @@ when its "done when" check passes on the real cluster, not locally. ## 0.2.0 Chapters on k3s -- Namespace `cv`, four placeholder chapters: one Job (`complete`), one each of `healthy`, `not-ready`, `crash`. +- 2 GB swap file, k3s installed with traefik and servicelb disabled and kubelet reservations. See [server budget](/architecture/server-budget.md). +- ResourceQuota (384Mi) and LimitRange on namespace `cv`. +- Four placeholder chapters: one Job (`complete`), one each of `healthy`, `not-ready`, `crash`. - Services with `publishNotReadyAddresses: true`, resource limits, pod label `cv.noelsabia.dev/chapter`. - Registry pull from Gitea, `kubectl apply -k deploy/`. - **Done when:** `kubectl get pods -n cv` shows Completed, Running 1/1, Running 0/1 and CrashLoopBackOff, and a `/fall` from a debug pod raises `restartCount`. -- **Learn:** Deployments, Jobs, probes, Services, kustomize. +- **Learn:** Deployments, Jobs, probes, Services, kustomize, resource limits and quotas. ## 0.3.0 Pipeline - Gitea workflow: `go test`, `snyk test`, build and push images tagged with the git SHA, `kubectl apply -k`. -- Runner kubeconfig limited to namespace `cv`. +- Runner kubeconfig limited to namespace `cv`. Runner capacity 1, so builds never run in parallel. - **Done when:** a push to `main` deploys a new image without touching the server by hand. - **Learn:** Gitea Actions, RBAC for CI, Snyk. @@ -91,7 +97,7 @@ when its "done when" check passes on the real cluster, not locally. ## 0.6.0 First screen -- Nuxt project, `nuxt generate`, multi-stage Dockerfile that embeds the output in the gateway. +- Nuxt project, `nuxt generate` with `NODE_OPTIONS=--max-old-space-size=512`, multi-stage Dockerfile that embeds the output in the gateway. - Layer 0: name, role, contact, PDF CV download. Navy `#1A3D64` and white. - **Done when:** the page loads and the PDF downloads with JavaScript disabled. - **Learn:** Nuxt static generation, `go:embed`, multi-stage builds. @@ -111,6 +117,12 @@ when its "done when" check passes on the real cluster, not locally. - Dev view: the raw stream in a `
`.
 - **Done when:** two browsers see each other's knockdowns live.
 
+## 0.9.0 Analytics
+
+- GoatCounter in `cv`, 64Mi limit, SQLite on a `local-path` volume. Needs approval, see [decision](/decisions/goatcounter-over-umami.md).
+- Count page views, PDF downloads and falls per chapter.
+- **Done when:** a test visit, a PDF download and a fall all show up in GoatCounter.
+
 ## 1.0.0 Real content and launch
 
 - Final chapter list, one manifest per chapter, story text, final PDF CV.
@@ -147,7 +159,7 @@ when its "done when" check passes on the real cluster, not locally.
 ## 1.5.0 LM: model and training
 
 - About 5M-parameter transformer, layers written by hand, gradients from PyTorch (needs approval).
-- Training run on the home server CPU, checkpoint saved.
+- Training runs on Noel's own computer, not the server. Checkpoint saved.
 - **Done when:** samples read like short simple stories.
 
 ## 1.6.0 LM: browser inference
@@ -166,19 +178,20 @@ when its "done when" check passes on the real cluster, not locally.
 
 ## 1.8.0 Interactive k6
 
-- Fixed k6 script run as a Job with the `grafana/k6` image, capped virtual users and duration.
+- Fixed k6 script run as a Job with the `grafana/k6` image, a few virtual users, 30 seconds max, 64Mi limit.
+- The server is shared with Gitea: if a run slows Gitea noticeably, cut this release.
 - One run at a time across the site. No visitor-supplied parameters.
 - The gateway needs `create` on Jobs for this, the first write permission it gets. Limit it to this one Job template.
 - **Done when:** a second visitor sees "a test is already running".
 
 ## 1.9.0 HPA demo
 
-- HPA on one chapter using metrics-server (ships with k3s), live replica count and latency graph during a k6 run.
+- HPA on one chapter using metrics-server (ships with k3s), max 3 replicas, live replica count and latency graph during a k6 run.
 - **Done when:** replicas go up during a run and back down after.
 
 ## 1.10.0 LM: bigger base model
 
-- Retrain the base model at 15-30M parameters. Check training time on the CPU first.
+- Retrain the base model at 15-30M parameters on Noel's own computer. Check training time first.
 - **Done when:** samples are clearly better than the 5M model.
 
 ## 1.11.0 LM: chat fine-tune