docs(okf): write overview, architecture, decisions and roadmap

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Noel SabiaandClaude Opus 5.5 committed 2026-10-11 14:18:32 +02:00
1 parent 987a5efba1
commit 557ac1985b
20 files changed
+750 -16

No files matched your search

+40
View File
@@ -0,0 +1,40 @@
---
type: Project
title: In-browser language model
description: A tiny language model trained by Noel that runs entirely in the visitor's browser and shows its internals live.
tags: [architecture, ml, typescript, learning]
status: draft
generated: { by: claude-code/claude-opus-5-5, at: 2026-10-11T12:15:30Z }
---
# Goal
A from-scratch learning project on the same domain: a small transformer whose
attention patterns, next-token probabilities and temperature are visible and
adjustable in the browser.
# Architecture
No backend. A `/lm` page in the Nuxt build plus a weights file, both served
statically by the [gateway](/architecture/gateway.md). It adds nothing to the cluster.
# Plan
| Step | What | Guide |
|---|---|---|
| Base model | About 5M parameters, layers written by hand, trained with PyTorch on TinyStories with word-piece tokens | Karpathy "Zero to Hero", llama2.c |
| Browser inference | Forward pass in plain TypeScript | llama2.c |
| Visuals | Attention heatmaps, next-token probabilities, temperature slider | |
| Chat | Fine-tune at 15-30M parameters on simple dialogues in a fixed user/bot format | |
| Own engine | Replace PyTorch gradients with Noel's own autograd | micrograd |
# Expectations
- TinyStories gives coherent short stories. Grimm gives stylish nonsense.
- The chat handles greetings and small talk, not knowledge questions or long conversations.
# Constraints
- **Download size:** 5M parameters in fp32 is about 20 MB, 30M is about 120 MB.
Ship int8 weights (about 4× smaller) and load them lazily.
- **PyTorch is new tooling** and needs Noel's approval under the stack rule.