Files
OKF_CV_Website/architecture/server-budget.md
T
Noel Sabia e022c30527 feat(okf): run on full Kubernetes with FreeLens for admin
k3s is out. The cluster is upstream Kubernetes installed with kubeadm:
containerd, single control-plane etcd, Calico for NetworkPolicy.

- add the cluster decision and the FreeLens decision
- move Terraform from the k3s service to kubeadm, Calico, metrics-server
  and the local-path provisioner
- budget ~900 MB for the control plane and plan a bigger machine before 0.3.0
- note kubelet failSwapOn and containerd hosts.toml
- rename 0.3.0 and commit the 0.2.0 done-when to the new stack
- update the German learning plan
2026-10-24 19:02:17 +02:00

3.7 KiB

type, title, description, tags, status, generated, sources
type title description tags status generated sources
Reference Server and memory budget The CV shares a small home server with Gitea, so every workload gets a hard memory cap; a kubeadm control plane needs more room than k3s did.
architecture
kubernetes
resources
revised
by at
opencode-go/deepseek-v4.1-flash 2026-10-24T00:00:00Z
id resource author last_modified
free-m `free -m` on server0, before the cluster, with Gitea, runner and webapp running human:noel 2026-10-11T12:15:30Z

Server

server0: 2 vCPU, 3805 MB RAM, 2 GB swap. Shared with Gitea, its Actions runner, and a small webapp with one user.

A control plane with etcd costs about twice what k3s' single binary cost. The plan is a stronger machine (8 GB or more, or a second server that carries Gitea) before release 0.3.0. The numbers below are the floor that must fit until then; the cv quota can rise once the bigger machine exists.

Budget

What Memory Basis
OS, Gitea, runner (idle), webapp 1143 MB Measured1
etcd, kube-apiserver, scheduler, controller-manager, kubelet ~900 MB Estimate, measure after install
Calico, Caddy, metrics-server ~150 MB Estimate
Namespace cv, hard quota 384 MB ResourceQuota
Left for CI builds and page cache ~1.2 GB (4 GB) / ~5 GB (8 GB)
Swap 2 GB Safety net for build spikes

Per workload inside cv:

Workload Memory limit CPU limit
Gateway, including the tracker 64Mi, GOMEMLIMIT=50MiB 100m
Each chapter (~10) 16Mi, GOMEMLIMIT=12MiB 10m

Completed Jobs do not count against the quota.

Namespace caps

apiVersion: v1
kind: ResourceQuota
metadata: { name: cv-budget, namespace: cv }
spec:
  hard: { limits.memory: 384Mi }
---
apiVersion: v1
kind: LimitRange
metadata: { name: cv-defaults, namespace: cv }
spec:
  limits:
    - type: Container
      default:        { memory: 16Mi, cpu: 50m }
      defaultRequest: { memory: 8Mi,  cpu: 5m }

Kubelet config

kubeadm writes /var/lib/kubelet/config.yaml from the KubeletConfiguration in its config file. The values that matter:

failSwapOn: false                     # node has swap; pods still never swap
systemReserved: { memory: 1300Mi }    # measured 1143 MB plus margin
evictionHard: { memory.available: 200Mi }

Full Kubernetes installs no addons. Two are required and are applied by the Terraform module after kubeadm init: metrics-server for the HPA demo in 1.9.0, and a local-path provisioner for the tracker's SQLite volume. Caddy replaces the ingress controller. Calico provides NetworkPolicy.

Do not use --pod-network-cidr values that collide with the home LAN; Calico defaults to 192.168.0.0/16, which on a typical 192.168.x.x home network is a conflict. Pick 10.244.0.0/16.

Swap

sudo fallocate -l 2G /swapfile && sudo chmod 600 /swapfile
sudo mkswap /swapfile && sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

kubeadm and the kubelet refuse to start on a swapped host unless failSwapOn: false is set. With that flag the node runs, pods still get no swap, and host processes such as Gitea and builds can use it. Enabling swap for pods needs the NodeSwap feature gate and is not worth it here.

CI builds

The biggest memory risk. A Nuxt build can take over 1 GB.

  • Runner capacity 1: one job at a time.
  • NODE_OPTIONS=--max-old-space-size=512 for the Nuxt build.
  • Swap catches the rest.

What does not run here

  • Language model training: on Noel's own computer. The server only serves the weights file.
  • Large load tests: k6 and the HPA demo stay tiny (max 3 replicas, a few virtual users), or are cut.

  1. free -m on server0 ↩︎