docs(okf): write overview, architecture, decisions and roadmap
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
987a5efba1
commit
557ac1985b
20 files changed
+750
-16
No files matched your search
@@ -0,0 +1,40 @@
|
||||
---
|
||||
type: Project
|
||||
title: In-browser language model
|
||||
description: A tiny language model trained by Noel that runs entirely in the visitor's browser and shows its internals live.
|
||||
tags: [architecture, ml, typescript, learning]
|
||||
status: draft
|
||||
generated: { by: claude-code/claude-opus-5-5, at: 2026-10-11T12:15:30Z }
|
||||
---
|
||||
|
||||
# Goal
|
||||
|
||||
A from-scratch learning project on the same domain: a small transformer whose
|
||||
attention patterns, next-token probabilities and temperature are visible and
|
||||
adjustable in the browser.
|
||||
|
||||
# Architecture
|
||||
|
||||
No backend. A `/lm` page in the Nuxt build plus a weights file, both served
|
||||
statically by the [gateway](/architecture/gateway.md). It adds nothing to the cluster.
|
||||
|
||||
# Plan
|
||||
|
||||
| Step | What | Guide |
|
||||
|---|---|---|
|
||||
| Base model | About 5M parameters, layers written by hand, trained with PyTorch on TinyStories with word-piece tokens | Karpathy "Zero to Hero", llama2.c |
|
||||
| Browser inference | Forward pass in plain TypeScript | llama2.c |
|
||||
| Visuals | Attention heatmaps, next-token probabilities, temperature slider | |
|
||||
| Chat | Fine-tune at 15-30M parameters on simple dialogues in a fixed user/bot format | |
|
||||
| Own engine | Replace PyTorch gradients with Noel's own autograd | micrograd |
|
||||
|
||||
# Expectations
|
||||
|
||||
- TinyStories gives coherent short stories. Grimm gives stylish nonsense.
|
||||
- The chat handles greetings and small talk, not knowledge questions or long conversations.
|
||||
|
||||
# Constraints
|
||||
|
||||
- **Download size:** 5M parameters in fp32 is about 20 MB, 30M is about 120 MB.
|
||||
Ship int8 weights (about 4× smaller) and load them lazily.
|
||||
- **PyTorch is new tooling** and needs Noel's approval under the stack rule.
|
||||
Reference in new issue
Block a user