Files
OKF_CV_Website/architecture/language-model.md
T

42 lines
1.7 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
type: Project
title: In-browser language model
description: A tiny language model trained by Noel that runs entirely in the visitor's browser and shows its internals live.
tags: [architecture, ml, typescript, learning]
status: draft
generated: { by: claude-code/claude-opus-5-5, at: 2026-10-11T12:15:30Z }
---
# Goal
A from-scratch learning project on the same domain: a small transformer whose
attention patterns, next-token probabilities and temperature are visible and
adjustable in the browser.
# Architecture
No backend. A `/lm` page in the Nuxt build plus a weights file, both served
statically by the [gateway](/architecture/gateway.md). It adds nothing to the cluster.
# Plan
| Step | What | Guide |
|---|---|---|
| Base model | About 5M parameters, layers written by hand, trained with PyTorch on TinyStories with word-piece tokens | Karpathy "Zero to Hero", llama2.c |
| Browser inference | Forward pass in plain TypeScript | llama2.c |
| Visuals | Attention heatmaps, next-token probabilities, temperature slider | |
| Chat | Fine-tune at 15-30M parameters on simple dialogues in a fixed user/bot format | |
| Own engine | Replace PyTorch gradients with Noel's own autograd | micrograd |
# Expectations
- TinyStories gives coherent short stories. Grimm gives stylish nonsense.
- The chat handles greetings and small talk, not knowledge questions or long conversations.
# Constraints
- **Download size:** 5M parameters in fp32 is about 20 MB, 30M is about 120 MB.
Ship int8 weights (about 4× smaller) and load them lazily.
- **Training happens on Noel's own computer.** The [server](/architecture/server-budget.md) only serves the weights file.
- **PyTorch is new tooling** and needs Noel's approval under the stack rule.