Files
OKF_CV_Website/architecture/language-model.md
T
Noel SabiaandClaude Opus 5.5 64fef83314 docs(okf): move bundle to repo root and retire brief
The brief is fully captured in the bundle; it stays in git history.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 14:22:03 +02:00

1.6 KiB
Raw Blame History

type, title, description, tags, status, generated
type title description tags status generated
Project In-browser language model A tiny language model trained by Noel that runs entirely in the visitor's browser and shows its internals live.
architecture
ml
typescript
learning
draft
by at
claude-code/claude-opus-5-5 2026-10-11T12:15:30Z

Goal

A from-scratch learning project on the same domain: a small transformer whose attention patterns, next-token probabilities and temperature are visible and adjustable in the browser.

Architecture

No backend. A /lm page in the Nuxt build plus a weights file, both served statically by the gateway. It adds nothing to the cluster.

Plan

Step What Guide
Base model About 5M parameters, layers written by hand, trained with PyTorch on TinyStories with word-piece tokens Karpathy "Zero to Hero", llama2.c
Browser inference Forward pass in plain TypeScript llama2.c
Visuals Attention heatmaps, next-token probabilities, temperature slider
Chat Fine-tune at 15-30M parameters on simple dialogues in a fixed user/bot format
Own engine Replace PyTorch gradients with Noel's own autograd micrograd

Expectations

  • TinyStories gives coherent short stories. Grimm gives stylish nonsense.
  • The chat handles greetings and small talk, not knowledge questions or long conversations.

Constraints

  • Download size: 5M parameters in fp32 is about 20 MB, 30M is about 120 MB. Ship int8 weights (about 4× smaller) and load them lazily.
  • PyTorch is new tooling and needs Noel's approval under the stack rule.