feat(okf): run on full Kubernetes with FreeLens for admin
k3s is out. The cluster is upstream Kubernetes installed with kubeadm: containerd, single control-plane etcd, Calico for NetworkPolicy. - add the cluster decision and the FreeLens decision - move Terraform from the k3s service to kubeadm, Calico, metrics-server and the local-path provisioner - budget ~900 MB for the control plane and plan a bigger machine before 0.3.0 - note kubelet failSwapOn and containerd hosts.toml - rename 0.3.0 and commit the 0.2.0 done-when to the new stack - update the German learning plan
This commit is contained in:
1 parent
6ab76c2a36
commit
e022c30527
17 files changed
+214
-64
No files matched your search
@@ -2,7 +2,7 @@
|
||||
type: Service
|
||||
title: Chapter service
|
||||
description: One small Go binary whose health behavior is set by the MODE env var; every life chapter runs it.
|
||||
tags: [architecture, go, k3s]
|
||||
tags: [architecture, go, kubernetes]
|
||||
status: stable
|
||||
generated: { by: claude-code/claude-opus-5-5, at: 2026-10-11T12:15:30Z }
|
||||
---
|
||||
|
||||
@@ -24,8 +24,9 @@ push to main ─► Gitea Actions (runner on the home server)
|
||||
- **Commits:** Conventional Commits. `feat:` bumps minor, `fix:` bumps patch,
|
||||
`docs:` and `chore:` do not release.
|
||||
- **Image tags:** the git SHA, so every deploy is traceable.
|
||||
- **Registry pull:** k3s needs an `imagePullSecret` for the Gitea registry,
|
||||
or `/etc/rancher/k3s/registries.yaml` if the registry is plain HTTP.
|
||||
- **Registry pull:** the cluster needs an `imagePullSecret` for the Gitea
|
||||
registry, or `/etc/containerd/certs.d/<registry>/hosts.toml` if the registry
|
||||
is plain HTTP.
|
||||
- **Runner permissions:** a kubeconfig for a ServiceAccount that can only
|
||||
manage resources in `cv`.
|
||||
- **Jobs are immutable:** changing a completed chapter's Job spec needs a
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
* [Chapter service](chapter-service.md) - One small Go binary whose health behavior is set by the MODE env var; every life chapter runs it.
|
||||
* [Gateway](gateway.md) - The only public service; serves the static site, streams chapter states over SSE, and forwards rate-limited knockdowns.
|
||||
* [Frontend](frontend.md) - Nuxt site generated to static files and served by the gateway, with three audience layers fed by one SSE stream.
|
||||
* [Server and memory budget](server-budget.md) - The CV shares a 2 vCPU, 4 GB server with Gitea and a small webapp, so every workload gets a hard memory cap.
|
||||
* [Server and memory budget](server-budget.md) - The CV shares a small home server with Gitea, so every workload gets a hard memory cap; the kubeadm control plane is the biggest new cost.
|
||||
* [Delivery](delivery.md) - Gitea Actions on the home server scan, build, push to the Gitea registry, deploy with kubectl, and version with release-please.
|
||||
* [Terraform provisioning](terraform.md) - Terraform rebuilds the server, swap, k3s, Caddy and DNS from nothing; nothing inside the cluster.
|
||||
* [Terraform provisioning](terraform.md) - Terraform rebuilds the server, containerd, the kubeadm control plane, Caddy and DNS from nothing; nothing inside the cluster.
|
||||
* [In-browser language model](language-model.md) - A tiny language model trained by Noel that runs entirely in the visitor's browser and shows its internals live.
|
||||
@@ -1,30 +1,36 @@
|
||||
---
|
||||
type: Reference
|
||||
title: Server and memory budget
|
||||
description: The CV shares a 2 vCPU, 4 GB server with Gitea and a small webapp, so every workload gets a hard memory cap.
|
||||
tags: [architecture, k3s, resources]
|
||||
status: stable
|
||||
generated: { by: claude-code/claude-opus-5-5, at: 2026-10-11T12:15:30Z }
|
||||
description: The CV shares a small home server with Gitea, so every workload gets a hard memory cap; a kubeadm control plane needs more room than k3s did.
|
||||
tags: [architecture, kubernetes, resources]
|
||||
status: revised
|
||||
generated: { by: opencode-go/deepseek-v4.1-flash, at: 2026-10-24T00:00:00Z }
|
||||
sources:
|
||||
- id: free-m
|
||||
resource: "`free -m` on server0, before k3s, with Gitea, runner and webapp running"
|
||||
resource: "`free -m` on server0, before the cluster, with Gitea, runner and webapp running"
|
||||
author: human:noel
|
||||
last_modified: 2026-10-11T12:15:30Z
|
||||
---
|
||||
|
||||
# Server
|
||||
|
||||
`server0`: 2 vCPU, 3805 MB RAM, no swap yet. Shared with Gitea, its Actions
|
||||
`server0`: 2 vCPU, 3805 MB RAM, 2 GB swap. Shared with Gitea, its Actions
|
||||
runner, and a small webapp with one user.
|
||||
|
||||
A control plane with etcd costs about twice what k3s' single binary cost. The
|
||||
plan is a stronger machine (8 GB or more, or a second server that carries
|
||||
Gitea) before release 0.3.0. The numbers below are the floor that must fit
|
||||
until then; the `cv` quota can rise once the bigger machine exists.
|
||||
|
||||
# Budget
|
||||
|
||||
| What | Memory | Basis |
|
||||
|---|---|---|
|
||||
| OS, Gitea, runner (idle), webapp | 1143 MB | Measured[^free-m] |
|
||||
| k3s, trimmed | ~500 MB | Estimate, measure after install |
|
||||
| etcd, kube-apiserver, scheduler, controller-manager, kubelet | ~900 MB | Estimate, measure after install |
|
||||
| Calico, Caddy, metrics-server | ~150 MB | Estimate |
|
||||
| Namespace `cv`, hard quota | 384 MB | ResourceQuota |
|
||||
| Left for CI builds and page cache | ~1.7 GB | |
|
||||
| Left for CI builds and page cache | ~1.2 GB (4 GB) / ~5 GB (8 GB) | |
|
||||
| Swap | 2 GB | Safety net for build spikes |
|
||||
|
||||
[^free-m]: `free -m` on server0
|
||||
@@ -57,20 +63,25 @@ spec:
|
||||
defaultRequest: { memory: 8Mi, cpu: 5m }
|
||||
```
|
||||
|
||||
# k3s config
|
||||
# Kubelet config
|
||||
|
||||
`/etc/rancher/k3s/config.yaml`:
|
||||
kubeadm writes `/var/lib/kubelet/config.yaml` from the
|
||||
`KubeletConfiguration` in its config file. The values that matter:
|
||||
|
||||
```yaml
|
||||
disable: [traefik, servicelb] # Caddy (Terraform) replaces both
|
||||
kubelet-arg:
|
||||
- "system-reserved=memory=1300Mi" # measured 1143 MB plus margin
|
||||
- "eviction-hard=memory.available<200Mi"
|
||||
failSwapOn: false # node has swap; pods still never swap
|
||||
systemReserved: { memory: 1300Mi } # measured 1143 MB plus margin
|
||||
evictionHard: { memory.available: 200Mi }
|
||||
```
|
||||
|
||||
Keep `metrics-server` and `local-storage`: the HPA demo needs the first,
|
||||
the tracker's SQLite volume the second. Single-node k3s uses SQLite, not etcd; do not
|
||||
pass `--cluster-init`.
|
||||
Full Kubernetes installs no addons. Two are required and are applied by the
|
||||
Terraform module after `kubeadm init`: `metrics-server` for the HPA demo in
|
||||
1.9.0, and a `local-path` provisioner for the tracker's SQLite volume.
|
||||
Caddy replaces the ingress controller. Calico provides NetworkPolicy.
|
||||
|
||||
Do not use `--pod-network-cidr` values that collide with the home LAN; Calico
|
||||
defaults to `192.168.0.0/16`, which on a typical `192.168.x.x` home network is
|
||||
a conflict. Pick `10.244.0.0/16`.
|
||||
|
||||
# Swap
|
||||
|
||||
@@ -80,8 +91,10 @@ sudo mkswap /swapfile && sudo swapon /swapfile
|
||||
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
|
||||
```
|
||||
|
||||
k3s starts with swap enabled. Pods do not swap; host processes such as
|
||||
Gitea and builds can.
|
||||
kubeadm and the kubelet refuse to start on a swapped host unless
|
||||
`failSwapOn: false` is set. With that flag the node runs, pods still get no
|
||||
swap, and host processes such as Gitea and builds can use it. Enabling swap
|
||||
for pods needs the `NodeSwap` feature gate and is not worth it here.
|
||||
|
||||
# CI builds
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
type: Architecture
|
||||
title: System architecture
|
||||
description: All runtime components of the v1 site and how a request and a knockdown flow through them.
|
||||
tags: [architecture, k3s, go, nuxt]
|
||||
tags: [architecture, kubernetes, go, nuxt]
|
||||
status: stable
|
||||
generated: { by: claude-code/claude-opus-5-5, at: 2026-10-11T12:15:30Z }
|
||||
---
|
||||
@@ -16,10 +16,13 @@ generated: { by: claude-code/claude-opus-5-5, at: 2026-10-11T12:15:30Z }
|
||||
| [Frontend](/architecture/frontend.md) | Static files inside the gateway image | Nuxt, built with `nuxt generate`. |
|
||||
| [Chapter service](/architecture/chapter-service.md) | One Deployment or Job per chapter | Behaves healthy, not-ready, crashing, or completed, set by env. |
|
||||
| [Delivery](/architecture/delivery.md) | Gitea Actions on the home server | Test, scan, build, push, deploy, version. |
|
||||
| FreeLens | Noel's laptop | Admin UI for the cluster over an SSH tunnel. Not part of the public site. |
|
||||
|
||||
|
||||
Everything runs in one namespace: `cv`, capped at 384Mi memory in total, on a
|
||||
2 vCPU, 4 GB server shared with Gitea. See [server budget](/architecture/server-budget.md).
|
||||
2 vCPU, 4 GB server shared with Gitea. A machine with more RAM is planned
|
||||
before 0.3.0, because the kubeadm control plane costs more than k3s did. See
|
||||
[server budget](/architecture/server-budget.md).
|
||||
|
||||
# Images
|
||||
|
||||
@@ -46,14 +49,15 @@ cmd/chapter → image `chapter`
|
||||
6. Recovery time = Ready condition `lastTransitionTime` minus the fall time.
|
||||
7. The page shows the story card with the measured seconds.
|
||||
|
||||
No step needs write access to the Kubernetes API.
|
||||
No visitor-facing step needs write access to the Kubernetes API.
|
||||
|
||||
# Security boundaries
|
||||
|
||||
- Only the gateway is reachable from outside.
|
||||
- Gateway ServiceAccount: `get/list/watch` on pods, events and jobs in `cv`. Nothing else.
|
||||
- `/fall` is unauthenticated inside the cluster. A NetworkPolicy (supported natively by k3s) restricts it to the gateway.
|
||||
- `/fall` is unauthenticated inside the cluster. A NetworkPolicy (Calico) restricts it to the gateway.
|
||||
- No visitor-supplied input reaches the cluster except a chapter name checked against an allowlist.
|
||||
- Only Noel holds a kubeconfig. The API server listens on localhost; FreeLens reaches it through an SSH tunnel. See [FreeLens](/decisions/freelens-admin-ui.md).
|
||||
|
||||
# Accepted risk
|
||||
|
||||
|
||||
+25
-10
@@ -1,36 +1,42 @@
|
||||
# Server provisioning
|
||||
|
||||
Terraform owns everything outside the cluster: packages, swap, k3s itself,
|
||||
the reverse proxy, and DNS records. Rebuilding the server from nothing is two
|
||||
commands.
|
||||
Terraform owns everything outside the cluster: packages, swap, containerd,
|
||||
the kubeadm control plane, the reverse proxy, and DNS records. Rebuilding the
|
||||
server from nothing is two commands.
|
||||
|
||||
## Scope
|
||||
|
||||
| Layer | Managed by | Examples |
|
||||
|---|---|---|
|
||||
| Server, packages, swap, k3s install | Terraform (SSH provisioner, cloud-init or shell) | `fallocate`, `mkswap`, k3s service with trimmed config |
|
||||
| Server, packages, swap, runtime, kubeadm install | Terraform (SSH provisioner, cloud-init or shell) | `fallocate`, `mkswap`, containerd, `kubeadm init`, Calico |
|
||||
| Cluster addons | Terraform (`kubectl` provisioner on the kubeadm output) | metrics-server, local-path provisioner |
|
||||
| DNS records | Terraform provider for the DNS zone | A/AAAA record for the apex domain |
|
||||
| Reverse proxy | Terraform-managed Caddy, Caddyfile templated | reverse_proxy block to the gateway |
|
||||
| Workloads inside k3s | kustomize, applied by CI or hand | Namespace `cv`, Deployments, Jobs, Services, quota |
|
||||
| Workloads inside the cluster | kustomize, applied by CI or hand | Namespace `cv`, Deployments, Jobs, Services, quota |
|
||||
|
||||
Terraform does not manage anything inside the cluster. Kubernetes already
|
||||
describes its own state; Terraform and kubectl would fight over it.
|
||||
describes its own state; Terraform and kubectl would fight over it. The one
|
||||
exception is bootstrap: the namespace, the quota and the addons are created
|
||||
once, after `kubeadm init`, and never reconciled again.
|
||||
|
||||
## Files
|
||||
|
||||
```
|
||||
terraform/
|
||||
main.tf server: packages, swap, k3s, Caddy
|
||||
main.tf server: packages, swap, containerd, kubeadm, addons, Caddy
|
||||
dns.tf A record for the apex domain
|
||||
variables.tf server IP, SSH key, domain
|
||||
outputs.tf server IP, proxy URL
|
||||
outputs.tf server IP, kubeconfig path, proxy URL
|
||||
```
|
||||
|
||||
Plan output for one run:
|
||||
|
||||
```
|
||||
+ swapfile 2 GB
|
||||
+ k3s service (traefik, servicelb disabled)
|
||||
+ containerd 2.x, kubeadm, kubelet, kubectl
|
||||
+ control plane on 127.0.0.1:6443 (etcd, apiserver, scheduler, controller-manager)
|
||||
+ Calico (NetworkPolicy)
|
||||
+ metrics-server and local-path provisioner
|
||||
+ Caddy reverse_proxy block -> http://127.0.0.1:8080 (gateway NodePort)
|
||||
+ certificate for noelsabia.dev
|
||||
```
|
||||
@@ -40,6 +46,8 @@ Plan output for one run:
|
||||
- The server became disposable: `terraform destroy && terraform apply` plus
|
||||
`kubectl apply -k deploy/` returns the whole system. That is the fastest
|
||||
possible disaster recovery and the clearest possible demo.
|
||||
- kubeadm is the upstream installer. The same steps on a bigger machine or a
|
||||
cloud VM behave identically, which is what the [cluster decision](/decisions/full-kubernetes.md) asks for.
|
||||
- Proxmox and cloud names change; code does not. The same Terraform targets the
|
||||
home server now and a VM later.
|
||||
- Home IP changes: the DNS provider has an A record the server can update.
|
||||
@@ -49,5 +57,12 @@ Plan output for one run:
|
||||
|
||||
- **Terraform is new tooling** for Noel and needs approval under the stack
|
||||
rule. Worth it: infrastructure as code is a core work skill.
|
||||
- State file: local, not in Git. Nothing secret in the config itself.
|
||||
- **Control plane memory:** allow about 900 MB to 1.2 GB for etcd, apiserver,
|
||||
scheduler, controller-manager and kubelet. On a 4 GB server this is tight;
|
||||
the machine is enlarged or replaced before 0.3.0. See [server budget](/architecture/server-budget.md).
|
||||
- **No bundled addons:** full Kubernetes ships no ingress, load balancer,
|
||||
metrics-server or storage provisioner. Caddy, metrics-server and the
|
||||
local-path provisioner are explicit steps.
|
||||
- State file: local, not in Git. Nothing secret in the config itself; the
|
||||
kubeconfig it produces stays on Noel's machine.
|
||||
- Changing a completed chapter or a Job is a delete plus apply, documented.
|
||||
Reference in new issue
Block a user