docs(okf): add server memory budget and analytics proposal

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Noel SabiaandClaude Opus 5.5 committed 2026-10-11 14:39:54 +02:00
1 parent 64fef83314
commit 261cf17e52
13 files changed
+178 -12

No files matched your search

+1 -1
View File
@@ -30,7 +30,7 @@ generated: { by: claude-code/claude-opus-5-5, at: 2026-10-11T12:15:30Z }
- Services set `publishNotReadyAddresses: true`, otherwise a NotReady chapter - Services set `publishNotReadyAddresses: true`, otherwise a NotReady chapter
has no endpoints and the gateway cannot push it. has no endpoints and the gateway cannot push it.
- Label every pod `cv.noelsabia.dev/chapter: {name}`. The gateway uses it to map pods to chapters. - Label every pod `cv.noelsabia.dev/chapter: {name}`. The gateway uses it to map pods to chapters.
- Tight resource limits on every chapter (for example 10m CPU, 16Mi memory). - Limits per chapter: 10m CPU, 16Mi memory, `GOMEMLIMIT=12MiB`. See [server budget](/architecture/server-budget.md).
Crashing chapters back off on Kubernetes' default schedule, which caps at 5 minutes. Crashing chapters back off on Kubernetes' default schedule, which caps at 5 minutes.
- Completed Jobs keep no `ttlSecondsAfterFinished`, so they stay visible as Completed. - Completed Jobs keep no `ttlSecondsAfterFinished`, so they stay visible as Completed.
- A CrashLoopBackOff chapter cannot be pushed; it is already down. The UI shows its lesson card instead. - A CrashLoopBackOff chapter cannot be pushed; it is already down. The UI shows its lesson card instead.
+1
View File
@@ -31,6 +31,7 @@ push to main ─► Gitea Actions (runner on the home server)
- **Jobs are immutable:** changing a completed chapter's Job spec needs a - **Jobs are immutable:** changing a completed chapter's Job spec needs a
delete before apply. Rare, since finished chapters do not change. delete before apply. Rare, since finished chapters do not change.
- **Snyk:** needs a `SNYK_TOKEN` secret in Gitea. - **Snyk:** needs a `SNYK_TOKEN` secret in Gitea.
- **Memory:** builds run on the same 4 GB server. Runner capacity 1, Node capped at 512 MB, swap as a safety net. See [server budget](/architecture/server-budget.md).
# Later # Later
+4
View File
@@ -42,6 +42,10 @@ The same stream feeds the life map and the developer view.
- Send `Content-Type: text/event-stream` and flush after every message. - Send `Content-Type: text/event-stream` and flush after every message.
- Send a comment line (`: ping`) every 30 seconds so idle connections are not closed. - Send a comment line (`: ping`) every 30 seconds so idle connections are not closed.
# Resources
64Mi memory limit with `GOMEMLIMIT=50MiB`, 100m CPU. See [server budget](/architecture/server-budget.md).
# RBAC # RBAC
ServiceAccount `gateway` in `cv`, Role with `get/list/watch` on `pods`, ServiceAccount `gateway` in `cv`, Role with `get/list/watch` on `pods`,
+1
View File
@@ -4,5 +4,6 @@
* [Chapter service](chapter-service.md) - One small Go binary whose health behavior is set by the MODE env var; every life chapter runs it. * [Chapter service](chapter-service.md) - One small Go binary whose health behavior is set by the MODE env var; every life chapter runs it.
* [Gateway](gateway.md) - The only public service; serves the static site, streams chapter states over SSE, and forwards rate-limited knockdowns. * [Gateway](gateway.md) - The only public service; serves the static site, streams chapter states over SSE, and forwards rate-limited knockdowns.
* [Frontend](frontend.md) - Nuxt site generated to static files and served by the gateway, with three audience layers fed by one SSE stream. * [Frontend](frontend.md) - Nuxt site generated to static files and served by the gateway, with three audience layers fed by one SSE stream.
* [Server and memory budget](server-budget.md) - The CV shares a 2 vCPU, 4 GB server with Gitea and a small webapp, so every workload gets a hard memory cap.
* [Delivery](delivery.md) - Gitea Actions on the home server scan, build, push to the Gitea registry, deploy with kubectl, and version with release-please. * [Delivery](delivery.md) - Gitea Actions on the home server scan, build, push to the Gitea registry, deploy with kubectl, and version with release-please.
* [In-browser language model](language-model.md) - A tiny language model trained by Noel that runs entirely in the visitor's browser and shows its internals live. * [In-browser language model](language-model.md) - A tiny language model trained by Noel that runs entirely in the visitor's browser and shows its internals live.
+1
View File
@@ -37,4 +37,5 @@ statically by the [gateway](/architecture/gateway.md). It adds nothing to the cl
- **Download size:** 5M parameters in fp32 is about 20 MB, 30M is about 120 MB. - **Download size:** 5M parameters in fp32 is about 20 MB, 30M is about 120 MB.
Ship int8 weights (about 4× smaller) and load them lazily. Ship int8 weights (about 4× smaller) and load them lazily.
- **Training happens on Noel's own computer.** The [server](/architecture/server-budget.md) only serves the weights file.
- **PyTorch is new tooling** and needs Noel's approval under the stack rule. - **PyTorch is new tooling** and needs Noel's approval under the stack rule.
+98
View File
@@ -0,0 +1,98 @@
---
type: Reference
title: Server and memory budget
description: The CV shares a 2 vCPU, 4 GB server with Gitea and a small webapp, so every workload gets a hard memory cap.
tags: [architecture, k3s, resources]
status: stable
generated: { by: claude-code/claude-opus-5-5, at: 2026-10-11T12:15:30Z }
sources:
- id: free-m
resource: "`free -m` on server0, before k3s, with Gitea, runner and webapp running"
author: human:noel
last_modified: 2026-10-11T12:15:30Z
---
# Server
`server0`: 2 vCPU, 3805 MB RAM, no swap yet. Shared with Gitea, its Actions
runner, and a small webapp with one user.
# Budget
| What | Memory | Basis |
|---|---|---|
| OS, Gitea, runner (idle), webapp | 1143 MB | Measured[^free-m] |
| k3s, trimmed | ~500 MB | Estimate, measure after install |
| Namespace `cv`, hard quota | 384 MB | ResourceQuota |
| Left for CI builds and page cache | ~1.7 GB | |
| Swap | 2 GB | Safety net for build spikes |
[^free-m]: `free -m` on server0
Per workload inside `cv`:
| Workload | Memory limit | CPU limit |
|---|---|---|
| Gateway | 64Mi, `GOMEMLIMIT=50MiB` | 100m |
| Each chapter (~10) | 16Mi, `GOMEMLIMIT=12MiB` | 10m |
| Analytics ([GoatCounter](/decisions/goatcounter-over-umami.md)) | 64Mi | 50m |
Completed Jobs do not count against the quota.
# Namespace caps
```yaml
apiVersion: v1
kind: ResourceQuota
metadata: { name: cv-budget, namespace: cv }
spec:
hard: { limits.memory: 384Mi }
---
apiVersion: v1
kind: LimitRange
metadata: { name: cv-defaults, namespace: cv }
spec:
limits:
- type: Container
default: { memory: 16Mi, cpu: 50m }
defaultRequest: { memory: 8Mi, cpu: 5m }
```
# k3s config
`/etc/rancher/k3s/config.yaml`:
```yaml
disable: [traefik, servicelb] # Cloudflare Tunnel replaces both
kubelet-arg:
- "system-reserved=memory=1300Mi" # measured 1143 MB plus margin
- "eviction-hard=memory.available<200Mi"
```
Keep `metrics-server` and `local-storage`: the HPA demo needs the first,
analytics storage the second. Single-node k3s uses SQLite, not etcd; do not
pass `--cluster-init`.
# Swap
```sh
sudo fallocate -l 2G /swapfile && sudo chmod 600 /swapfile
sudo mkswap /swapfile && sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
```
k3s starts with swap enabled. Pods do not swap; host processes such as
Gitea and builds can.
# CI builds
The biggest memory risk. A Nuxt build can take over 1 GB.
- Runner capacity 1: one job at a time.
- `NODE_OPTIONS=--max-old-space-size=512` for the Nuxt build.
- Swap catches the rest.
# What does not run here
- **Language model training:** on Noel's own computer. The server only serves the weights file.
- **Large load tests:** k6 and the HPA demo stay tiny (max 3 replicas, a few virtual users), or are cut.
+4 -1
View File
@@ -17,7 +17,10 @@ generated: { by: claude-code/claude-opus-5-5, at: 2026-10-11T12:15:30Z }
| [Chapter service](/architecture/chapter-service.md) | One Deployment or Job per chapter | Behaves healthy, not-ready, crashing, or completed, set by env. | | [Chapter service](/architecture/chapter-service.md) | One Deployment or Job per chapter | Behaves healthy, not-ready, crashing, or completed, set by env. |
| [Delivery](/architecture/delivery.md) | Gitea Actions on the home server | Test, scan, build, push, deploy, version. | | [Delivery](/architecture/delivery.md) | Gitea Actions on the home server | Test, scan, build, push, deploy, version. |
Everything runs in one namespace: `cv`. | Analytics | 1 Deployment, GoatCounter (proposed) | Page views, PDF downloads, falls per chapter. See [decision](/decisions/goatcounter-over-umami.md). |
Everything runs in one namespace: `cv`, capped at 384Mi memory in total, on a
2 vCPU, 4 GB server shared with Gitea. See [server budget](/architecture/server-budget.md).
# Images # Images
+38
View File
@@ -0,0 +1,38 @@
---
type: Decision
title: GoatCounter over Umami for analytics
description: Proposed; GoatCounter gives cookie-free open source analytics in about 30 MB, where Umami needs Postgres and about 300-400 MB.
tags: [decision, analytics, resources]
status: draft
generated: { by: claude-code/claude-opus-5-5, at: 2026-10-11T12:15:30Z }
---
# Proposal
Self-host GoatCounter in namespace `cv` instead of Umami. Needs Noel's
approval as new tooling.
# Why
| | Umami | GoatCounter |
|---|---|---|
| Runtime | Node app | Single Go binary |
| Database | Postgres | SQLite file |
| Memory | ~300-400 MB with Postgres | ~30 MB |
| Cookie-free, no consent banner | Yes | Yes |
| Custom events | Yes | Yes |
| License | Open source | Open source |
On a [4 GB shared server](/architecture/server-budget.md), Umami alone would
match the whole CV budget.
# Track
- Page views.
- PDF CV downloads.
- Falls per chapter.
# Details
One Deployment, 64Mi limit, SQLite on a small `local-path` volume. No backup:
losing analytics hurts nobody.
+4
View File
@@ -6,6 +6,10 @@
* [CI/CD on Gitea Actions](ci-on-gitea.md) - The existing Gitea instance and runner on the home server build, scan, deploy and version the project. * [CI/CD on Gitea Actions](ci-on-gitea.md) - The existing Gitea instance and runner on the home server build, scan, deploy and version the project.
* [Knowledge bundle lives at the repository root](bundle-at-root.md) - The repository root is the OKF bundle root; any other markdown file there needs frontmatter to stay conformant. * [Knowledge bundle lives at the repository root](bundle-at-root.md) - The repository root is the OKF bundle root; any other markdown file there needs frontmatter to stay conformant.
# Proposed
* [GoatCounter over Umami for analytics](goatcounter-over-umami.md) - Proposed; GoatCounter gives cookie-free open source analytics in about 30 MB, where Umami needs Postgres and about 300-400 MB.
# Open # Open
* [Open questions](open-questions.md) - Decisions still to make, with the release that needs each answer. * [Open questions](open-questions.md) - Decisions still to make, with the release that needs each answer.
+1
View File
@@ -15,5 +15,6 @@ generated: { by: claude-code/claude-opus-5-5, at: 2026-10-11T12:15:30Z }
| Which Gitea release-please action, exact name | 0.1.0 | None, Noel names it | | Which Gitea release-please action, exact name | 0.1.0 | None, Noel names it |
| Staging for k6 in CI: namespace or separate cluster | 1.2.0 | Namespace `cv-staging` on the same cluster | | Staging for k6 in CI: namespace or separate cluster | 1.2.0 | Namespace `cv-staging` on the same cluster |
| MCP scope: read-only or read plus `/fall` | 1.1.0 | Read plus `/fall`, same limits as visitors | | MCP scope: read-only or read plus `/fall` | 1.1.0 | Read plus `/fall`, same limits as visitors |
| Approve GoatCounter for analytics | 0.9.0 | None, needs Noel |
| Approve PyTorch for the language model | 1.5.0 | None, needs Noel | | Approve PyTorch for the language model | 1.5.0 | None, needs Noel |
| Approve NumPy (or similar) for the own autograd engine | 1.13.0 | None, needs Noel | | Approve NumPy (or similar) for the own autograd engine | 1.13.0 | None, needs Noel |
+1
View File
@@ -1,6 +1,7 @@
# Update Log # Update Log
## 2026-10-11 ## 2026-10-11
* **Creation**: Added the [server and memory budget](/architecture/server-budget.md) from measured usage, and proposed [GoatCounter over Umami](/decisions/goatcounter-over-umami.md).
* **Update**: Moved the bundle from `knowledge/` to the repository root and replaced the planning brief with this bundle. See [bundle at root](/decisions/bundle-at-root.md). * **Update**: Moved the bundle from `knowledge/` to the repository root and replaced the planning brief with this bundle. See [bundle at root](/decisions/bundle-at-root.md).
* **Creation**: Wrote the [overview](/overview.md), [architecture](/architecture/), [decisions](/decisions/), [chapter states](/chapters/state-mapping.md) and [roadmap](/roadmap.md) from the planning conversation. * **Creation**: Wrote the [overview](/overview.md), [architecture](/architecture/), [decisions](/decisions/), [chapter states](/chapters/state-mapping.md) and [roadmap](/roadmap.md) from the planning conversation.
* **Initialization**: Created the bundle [index](/index.md). * **Initialization**: Created the bundle [index](/index.md).
+1
View File
@@ -59,6 +59,7 @@ page shows the real recovery time next to the story of how Noel recovered.
- **Learning project, ship small.** Each release is a few days of work. - **Learning project, ship small.** Each release is a few days of work.
- **No tooling beyond the stack without Noel's approval.** The stack is Nuxt, - **No tooling beyond the stack without Noel's approval.** The stack is Nuxt,
Go, k3s, Snyk, Gitea Actions, release-please, Cloudflare Tunnel, and k6. Go, k3s, Snyk, Gitea Actions, release-please, Cloudflare Tunnel, and k6.
- **Small server.** 2 vCPU, 4 GB, shared with Gitea. The whole CV is capped at 384Mi. See [server budget](/architecture/server-budget.md).
- **Nobody gets Kubernetes API access.** Visitors and agents only talk to the gateway. - **Nobody gets Kubernetes API access.** Visitors and agents only talk to the gateway.
- **Failures are lessons, never bugs.** - **Failures are lessons, never bugs.**
+23 -10
View File
@@ -13,18 +13,19 @@ generated: { by: claude-code/claude-opus-5-5, at: 2026-10-11T12:15:30Z }
|---|---|---| |---|---|---|
| 1: Ship the CV | 0.1.0 - 1.0.0 | Public interactive CV on the apex domain | | 1: Ship the CV | 0.1.0 - 1.0.0 | Public interactive CV on the apex domain |
| 2: Depth | 1.1.0 - 1.7.0 | MCP, k6 release gate, full dev terminal, in-browser language model | | 2: Depth | 1.1.0 - 1.7.0 | MCP, k6 release gate, full dev terminal, in-browser language model |
| 3: Pressure and from scratch | 1.8.0 - 1.13.0 | Interactive load test with HPA, chat model, own autograd engine | | 3: Pressure and from scratch | 1.8.0 - 1.13.0 | Tiny interactive load test with HPA, chat model, own autograd engine |
| Release | Name | Days | | Release | Name | Days |
|---|---|---| |---|---|---|
| 0.1.0 | Chapter service | 1-2 | | 0.1.0 | Chapter service | 1-2 |
| 0.2.0 | Chapters on k3s | 1-2 | | 0.2.0 | Chapters on k3s | 2 |
| 0.3.0 | Pipeline | 2 | | 0.3.0 | Pipeline | 2 |
| 0.4.0 | Live states | 2-3 | | 0.4.0 | Live states | 2-3 |
| 0.5.0 | Knockdown | 1-2 | | 0.5.0 | Knockdown | 1-2 |
| 0.6.0 | First screen | 2 | | 0.6.0 | First screen | 2 |
| 0.7.0 | Life map | 3 | | 0.7.0 | Life map | 3 |
| 0.8.0 | Story cards and dev view | 2 | | 0.8.0 | Story cards and dev view | 2 |
| 0.9.0 | Analytics | 1 |
| 1.0.0 | Real content, launch | 2-3 | | 1.0.0 | Real content, launch | 2-3 |
| 1.1.0 | MCP endpoint | 1 | | 1.1.0 | MCP endpoint | 1 |
| 1.2.0 | k6 release gate | 2 | | 1.2.0 | k6 release gate | 2 |
@@ -43,6 +44,9 @@ generated: { by: claude-code/claude-opus-5-5, at: 2026-10-11T12:15:30Z }
Each release is cut by release-please from `feat:` commits. A release is done Each release is cut by release-please from `feat:` commits. A release is done
when its "done when" check passes on the real cluster, not locally. when its "done when" check passes on the real cluster, not locally.
Everything runs on a 2 vCPU, 4 GB server shared with Gitea. Every release
stays inside the [memory budget](/architecture/server-budget.md).
--- ---
# Phase 1: Ship the CV # Phase 1: Ship the CV
@@ -58,16 +62,18 @@ when its "done when" check passes on the real cluster, not locally.
## 0.2.0 Chapters on k3s ## 0.2.0 Chapters on k3s
- Namespace `cv`, four placeholder chapters: one Job (`complete`), one each of `healthy`, `not-ready`, `crash`. - 2 GB swap file, k3s installed with traefik and servicelb disabled and kubelet reservations. See [server budget](/architecture/server-budget.md).
- ResourceQuota (384Mi) and LimitRange on namespace `cv`.
- Four placeholder chapters: one Job (`complete`), one each of `healthy`, `not-ready`, `crash`.
- Services with `publishNotReadyAddresses: true`, resource limits, pod label `cv.noelsabia.dev/chapter`. - Services with `publishNotReadyAddresses: true`, resource limits, pod label `cv.noelsabia.dev/chapter`.
- Registry pull from Gitea, `kubectl apply -k deploy/`. - Registry pull from Gitea, `kubectl apply -k deploy/`.
- **Done when:** `kubectl get pods -n cv` shows Completed, Running 1/1, Running 0/1 and CrashLoopBackOff, and a `/fall` from a debug pod raises `restartCount`. - **Done when:** `kubectl get pods -n cv` shows Completed, Running 1/1, Running 0/1 and CrashLoopBackOff, and a `/fall` from a debug pod raises `restartCount`.
- **Learn:** Deployments, Jobs, probes, Services, kustomize. - **Learn:** Deployments, Jobs, probes, Services, kustomize, resource limits and quotas.
## 0.3.0 Pipeline ## 0.3.0 Pipeline
- Gitea workflow: `go test`, `snyk test`, build and push images tagged with the git SHA, `kubectl apply -k`. - Gitea workflow: `go test`, `snyk test`, build and push images tagged with the git SHA, `kubectl apply -k`.
- Runner kubeconfig limited to namespace `cv`. - Runner kubeconfig limited to namespace `cv`. Runner capacity 1, so builds never run in parallel.
- **Done when:** a push to `main` deploys a new image without touching the server by hand. - **Done when:** a push to `main` deploys a new image without touching the server by hand.
- **Learn:** Gitea Actions, RBAC for CI, Snyk. - **Learn:** Gitea Actions, RBAC for CI, Snyk.
@@ -91,7 +97,7 @@ when its "done when" check passes on the real cluster, not locally.
## 0.6.0 First screen ## 0.6.0 First screen
- Nuxt project, `nuxt generate`, multi-stage Dockerfile that embeds the output in the gateway. - Nuxt project, `nuxt generate` with `NODE_OPTIONS=--max-old-space-size=512`, multi-stage Dockerfile that embeds the output in the gateway.
- Layer 0: name, role, contact, PDF CV download. Navy `#1A3D64` and white. - Layer 0: name, role, contact, PDF CV download. Navy `#1A3D64` and white.
- **Done when:** the page loads and the PDF downloads with JavaScript disabled. - **Done when:** the page loads and the PDF downloads with JavaScript disabled.
- **Learn:** Nuxt static generation, `go:embed`, multi-stage builds. - **Learn:** Nuxt static generation, `go:embed`, multi-stage builds.
@@ -111,6 +117,12 @@ when its "done when" check passes on the real cluster, not locally.
- Dev view: the raw stream in a `<pre>`. - Dev view: the raw stream in a `<pre>`.
- **Done when:** two browsers see each other's knockdowns live. - **Done when:** two browsers see each other's knockdowns live.
## 0.9.0 Analytics
- GoatCounter in `cv`, 64Mi limit, SQLite on a `local-path` volume. Needs approval, see [decision](/decisions/goatcounter-over-umami.md).
- Count page views, PDF downloads and falls per chapter.
- **Done when:** a test visit, a PDF download and a fall all show up in GoatCounter.
## 1.0.0 Real content and launch ## 1.0.0 Real content and launch
- Final chapter list, one manifest per chapter, story text, final PDF CV. - Final chapter list, one manifest per chapter, story text, final PDF CV.
@@ -147,7 +159,7 @@ when its "done when" check passes on the real cluster, not locally.
## 1.5.0 LM: model and training ## 1.5.0 LM: model and training
- About 5M-parameter transformer, layers written by hand, gradients from PyTorch (needs approval). - About 5M-parameter transformer, layers written by hand, gradients from PyTorch (needs approval).
- Training run on the home server CPU, checkpoint saved. - Training runs on Noel's own computer, not the server. Checkpoint saved.
- **Done when:** samples read like short simple stories. - **Done when:** samples read like short simple stories.
## 1.6.0 LM: browser inference ## 1.6.0 LM: browser inference
@@ -166,19 +178,20 @@ when its "done when" check passes on the real cluster, not locally.
## 1.8.0 Interactive k6 ## 1.8.0 Interactive k6
- Fixed k6 script run as a Job with the `grafana/k6` image, capped virtual users and duration. - Fixed k6 script run as a Job with the `grafana/k6` image, a few virtual users, 30 seconds max, 64Mi limit.
- The server is shared with Gitea: if a run slows Gitea noticeably, cut this release.
- One run at a time across the site. No visitor-supplied parameters. - One run at a time across the site. No visitor-supplied parameters.
- The gateway needs `create` on Jobs for this, the first write permission it gets. Limit it to this one Job template. - The gateway needs `create` on Jobs for this, the first write permission it gets. Limit it to this one Job template.
- **Done when:** a second visitor sees "a test is already running". - **Done when:** a second visitor sees "a test is already running".
## 1.9.0 HPA demo ## 1.9.0 HPA demo
- HPA on one chapter using metrics-server (ships with k3s), live replica count and latency graph during a k6 run. - HPA on one chapter using metrics-server (ships with k3s), max 3 replicas, live replica count and latency graph during a k6 run.
- **Done when:** replicas go up during a run and back down after. - **Done when:** replicas go up during a run and back down after.
## 1.10.0 LM: bigger base model ## 1.10.0 LM: bigger base model
- Retrain the base model at 15-30M parameters. Check training time on the CPU first. - Retrain the base model at 15-30M parameters on Noel's own computer. Check training time first.
- **Done when:** samples are clearly better than the 5M model. - **Done when:** samples are clearly better than the 5M model.
## 1.11.0 LM: chat fine-tune ## 1.11.0 LM: chat fine-tune