Multi-head attention

A single round of self-attention captures one type of relationship between words. But words relate in many ways at once: grammar, meaning, tone, cause and effect. Multi-head attention runs several attention operations in parallel, each called a head, and each free to focus on a different pattern.

One head might learn to link pronouns to the nouns they stand for, another might track verb tense, another might connect adjectives to their nouns. Their findings are then combined. Think of it as a committee of specialists all reading the same sentence and pooling their notes. The original Transformer used eight heads.