Multi-Head Attention
An extension of attention where multiple attention mechanisms (heads) run in parallel, each learning to focus on different types of relationships in the data. The outputs are then combined.
Why It Matters
Multi-head attention lets the model simultaneously attend to different aspects — syntax, semantics, long-range dependencies — rather than being limited to one type of pattern.
Example
One head might learn to track subject-verb agreement, another tracks pronoun references, and a third captures topical relationships — all working simultaneously.
Think of it like...
Like having a panel of experts review a document simultaneously — one focuses on grammar, another on logic, another on factual accuracy, and their insights are combined.
Related Terms
Self-Attention
A mechanism where each element in a sequence attends to all other elements to compute a representation, determining how much focus to place on each part of the input. It is the core innovation of the transformer.
Attention Mechanism
A component in neural networks that allows the model to focus on the most relevant parts of the input when producing each part of the output. It assigns different weights to different input elements based on their relevance.
Transformer
A neural network architecture introduced in 2017 that uses self-attention mechanisms to process sequential data in parallel rather than sequentially. Transformers are the foundation of modern LLMs like GPT, Claude, and Gemini.
Positional Encoding
A technique used in transformers to inject information about the position of each token in a sequence. Since transformers process all tokens in parallel, they need explicit position information.