Files
OKF_CV_Website/architecture/server-budget.md
T
Noel Sabia e022c30527 feat(okf): run on full Kubernetes with FreeLens for admin
k3s is out. The cluster is upstream Kubernetes installed with kubeadm:
containerd, single control-plane etcd, Calico for NetworkPolicy.

- add the cluster decision and the FreeLens decision
- move Terraform from the k3s service to kubeadm, Calico, metrics-server
  and the local-path provisioner
- budget ~900 MB for the control plane and plan a bigger machine before 0.3.0
- note kubelet failSwapOn and containerd hosts.toml
- rename 0.3.0 and commit the 0.2.0 done-when to the new stack
- update the German learning plan
2026-10-24 19:02:17 +02:00

111 lines
3.7 KiB
Markdown

---
type: Reference
title: Server and memory budget
description: The CV shares a small home server with Gitea, so every workload gets a hard memory cap; a kubeadm control plane needs more room than k3s did.
tags: [architecture, kubernetes, resources]
status: revised
generated: { by: opencode-go/deepseek-v4.1-flash, at: 2026-10-24T00:00:00Z }
sources:
- id: free-m
resource: "`free -m` on server0, before the cluster, with Gitea, runner and webapp running"
author: human:noel
last_modified: 2026-10-11T12:15:30Z
---
# Server
`server0`: 2 vCPU, 3805 MB RAM, 2 GB swap. Shared with Gitea, its Actions
runner, and a small webapp with one user.
A control plane with etcd costs about twice what k3s' single binary cost. The
plan is a stronger machine (8 GB or more, or a second server that carries
Gitea) before release 0.3.0. The numbers below are the floor that must fit
until then; the `cv` quota can rise once the bigger machine exists.
# Budget
| What | Memory | Basis |
|---|---|---|
| OS, Gitea, runner (idle), webapp | 1143 MB | Measured[^free-m] |
| etcd, kube-apiserver, scheduler, controller-manager, kubelet | ~900 MB | Estimate, measure after install |
| Calico, Caddy, metrics-server | ~150 MB | Estimate |
| Namespace `cv`, hard quota | 384 MB | ResourceQuota |
| Left for CI builds and page cache | ~1.2 GB (4 GB) / ~5 GB (8 GB) | |
| Swap | 2 GB | Safety net for build spikes |
[^free-m]: `free -m` on server0
Per workload inside `cv`:
| Workload | Memory limit | CPU limit |
|---|---|---|
| Gateway, including the [tracker](/decisions/own-tracker.md) | 64Mi, `GOMEMLIMIT=50MiB` | 100m |
| Each chapter (~10) | 16Mi, `GOMEMLIMIT=12MiB` | 10m |
Completed Jobs do not count against the quota.
# Namespace caps
```yaml
apiVersion: v1
kind: ResourceQuota
metadata: { name: cv-budget, namespace: cv }
spec:
hard: { limits.memory: 384Mi }
---
apiVersion: v1
kind: LimitRange
metadata: { name: cv-defaults, namespace: cv }
spec:
limits:
- type: Container
default: { memory: 16Mi, cpu: 50m }
defaultRequest: { memory: 8Mi, cpu: 5m }
```
# Kubelet config
kubeadm writes `/var/lib/kubelet/config.yaml` from the
`KubeletConfiguration` in its config file. The values that matter:
```yaml
failSwapOn: false # node has swap; pods still never swap
systemReserved: { memory: 1300Mi } # measured 1143 MB plus margin
evictionHard: { memory.available: 200Mi }
```
Full Kubernetes installs no addons. Two are required and are applied by the
Terraform module after `kubeadm init`: `metrics-server` for the HPA demo in
1.9.0, and a `local-path` provisioner for the tracker's SQLite volume.
Caddy replaces the ingress controller. Calico provides NetworkPolicy.
Do not use `--pod-network-cidr` values that collide with the home LAN; Calico
defaults to `192.168.0.0/16`, which on a typical `192.168.x.x` home network is
a conflict. Pick `10.244.0.0/16`.
# Swap
```sh
sudo fallocate -l 2G /swapfile && sudo chmod 600 /swapfile
sudo mkswap /swapfile && sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
```
kubeadm and the kubelet refuse to start on a swapped host unless
`failSwapOn: false` is set. With that flag the node runs, pods still get no
swap, and host processes such as Gitea and builds can use it. Enabling swap
for pods needs the `NodeSwap` feature gate and is not worth it here.
# CI builds
The biggest memory risk. A Nuxt build can take over 1 GB.
- Runner capacity 1: one job at a time.
- `NODE_OPTIONS=--max-old-space-size=512` for the Nuxt build.
- Swap catches the rest.
# What does not run here
- **Language model training:** on Noel's own computer. The server only serves the weights file.
- **Large load tests:** k6 and the HPA demo stay tiny (max 3 replicas, a few virtual users), or are cut.