Files
OKF_CV_Website/architecture/terraform.md
T
Noel Sabia e022c30527 feat(okf): run on full Kubernetes with FreeLens for admin
k3s is out. The cluster is upstream Kubernetes installed with kubeadm:
containerd, single control-plane etcd, Calico for NetworkPolicy.

- add the cluster decision and the FreeLens decision
- move Terraform from the k3s service to kubeadm, Calico, metrics-server
  and the local-path provisioner
- budget ~900 MB for the control plane and plan a bigger machine before 0.3.0
- note kubelet failSwapOn and containerd hosts.toml
- rename 0.3.0 and commit the 0.2.0 done-when to the new stack
- update the German learning plan
2026-10-24 19:02:17 +02:00

3.2 KiB

Server provisioning

Terraform owns everything outside the cluster: packages, swap, containerd, the kubeadm control plane, the reverse proxy, and DNS records. Rebuilding the server from nothing is two commands.

Scope

Layer Managed by Examples
Server, packages, swap, runtime, kubeadm install Terraform (SSH provisioner, cloud-init or shell) fallocate, mkswap, containerd, kubeadm init, Calico
Cluster addons Terraform (kubectl provisioner on the kubeadm output) metrics-server, local-path provisioner
DNS records Terraform provider for the DNS zone A/AAAA record for the apex domain
Reverse proxy Terraform-managed Caddy, Caddyfile templated reverse_proxy block to the gateway
Workloads inside the cluster kustomize, applied by CI or hand Namespace cv, Deployments, Jobs, Services, quota

Terraform does not manage anything inside the cluster. Kubernetes already describes its own state; Terraform and kubectl would fight over it. The one exception is bootstrap: the namespace, the quota and the addons are created once, after kubeadm init, and never reconciled again.

Files

terraform/
  main.tf          server: packages, swap, containerd, kubeadm, addons, Caddy
  dns.tf           A record for the apex domain
  variables.tf     server IP, SSH key, domain
  outputs.tf       server IP, kubeconfig path, proxy URL

Plan output for one run:

+ swapfile 2 GB
+ containerd 2.x, kubeadm, kubelet, kubectl
+ control plane on 127.0.0.1:6443 (etcd, apiserver, scheduler, controller-manager)
+ Calico (NetworkPolicy)
+ metrics-server and local-path provisioner
+ Caddy reverse_proxy block -> http://127.0.0.1:8080 (gateway NodePort)
+  certificate for noelsabia.dev

Why

  • The server became disposable: terraform destroy && terraform apply plus kubectl apply -k deploy/ returns the whole system. That is the fastest possible disaster recovery and the clearest possible demo.
  • kubeadm is the upstream installer. The same steps on a bigger machine or a cloud VM behave identically, which is what the cluster decision asks for.
  • Proxmox and cloud names change; code does not. The same Terraform targets the home server now and a VM later.
  • Home IP changes: the DNS provider has an A record the server can update. Terraform manages the record; a small script re-sets it if the IP moves.

Constraints

  • Terraform is new tooling for Noel and needs approval under the stack rule. Worth it: infrastructure as code is a core work skill.
  • Control plane memory: allow about 900 MB to 1.2 GB for etcd, apiserver, scheduler, controller-manager and kubelet. On a 4 GB server this is tight; the machine is enlarged or replaced before 0.3.0. See server budget.
  • No bundled addons: full Kubernetes ships no ingress, load balancer, metrics-server or storage provisioner. Caddy, metrics-server and the local-path provisioner are explicit steps.
  • State file: local, not in Git. Nothing secret in the config itself; the kubeconfig it produces stays on Noel's machine.
  • Changing a completed chapter or a Job is a delete plus apply, documented.