Explainer 4 of 6 · Sequence · interactive
Attention
How does a transformer decide which earlier words matter?
Watch
The idea
Each position turns its current meaning into three vectors: a query (what am I looking for), a key (what do I offer) and a value (what I pass on).
A word's query is compared with every earlier key by a dot product. Softmax turns those scores into weights that add up to one, and the word's new representation is the weighted mix of the values.
Nothing about position is built in: the model learns which relations matter, a pronoun to its noun, a closing bracket to its opening one. Several heads do this in parallel, each free to look for something different.
Try it
Score = query · key, then softmax
Drag the amber query. A key pointing the same way scores high; softmax turns the scores into weights that add up to 100%, and the word's new meaning is that weighted mix. Low sharpness spreads attention evenly; high sharpness picks one word.