Aditya Sridhar

Model · Oct 2026 · runs in your browser

Mini-GPT

The same architecture as GPT, shrunk until it runs in a browser tab: 4 layers, 4 attention heads, a 128-character context, trained from scratch on a million characters of Shakespeare's plays, one character at a time. It has never seen a word list; spelling, line breaks and speaker names are all learned from the characters.

Validation loss
2.11 bits/char
Parameters
818,048
Model file
3.3 MB, ONNX
Training time
2.8 min, GPU

Try it

Generated text appears here. Afterwards, click any character to see what the model attended to when it chose it.

Runs in your browser. Nothing is uploaded.

Each character is one full pass of the network over up to 128 characters of context. Lower temperature picks the likeliest character more often; higher temperature takes more risks.

Training run

Cross-entropy, nats per charactertrainvalidation
1.002.003.004.001100020003000400050006000

Estimated every 250 steps on 40 random batches; the x axis is the optimiser step.

Results

MetricValueSource
Validation loss, all 111,540 held-out characters1.465 nats · 2.11 bitspublic/models/mini-gpt/meta.json
Bigram model on the same split, for comparison2.482 nats · 3.58 bitspublic/models/mini-gpt/meta.json
Unigram (character frequency) model3.347 nats · 4.83 bitspublic/models/mini-gpt/meta.json
Largest logit difference, exported ONNX vs PyTorch9.2e-6public/models/mini-gpt/meta.json
A sample from the exported model at temperature 0.8, written by the training script:
ROMEO:
Peace, sir.

AUFIDIUS:
Sirs; let them thee discontinual lord.

MERCUTIO:
What, my lord? Just order? have we an arch, I am the
that none sound that we are discontented without thy presence?

CLARENCE:
I think thee that with all grave the sovereign.

QUEEN ELIZABETH:
Nay that your household in no friend soulship the reasons
That his action and part your clouds, for this happy
May breathe to the quee

How it works

Tokens. Every one of the 65 distinct characters in the text is a token. Each gets a learned 128-dimensional embedding, plus a learned embedding for its position.

Blocks. Each of the 4 blocks is causal self-attention with 4 heads, then a two-layer MLP, each wrapped in layer norm and a residual connection. A position can only attend to itself and earlier positions, so training on a whole sequence teaches it to predict every next character at once. The output layer shares its weights with the input embedding.

Training. AdamW, batch 64 × 128 characters, 6,000 steps, learning rate 0.002 with warm-up and cosine decay, weight decay 0.1, dropout 0.1, bf16 autocast. The last 10% of the text is held out for validation.

Reproduce

Hardware
NVIDIA GeForce RTX 3050 Laptop GPU, PyTorch 2.6.0+cu124
Commit
534648f
Data
Tiny Shakespeare, 1,115,394 characters, MD5-checked by the script

Environment

pip install torch numpy onnx onnxruntime

Train, export, check, sample

python training/mini_gpt/train.py