Neural Networks for Language

Attention Mechanisms

---

The remaining problem

So far in this part, we’ve learned that:

• Embeddings represent words with numerical vectors that can reflect patterns of use and semantic relationships
• RNNs and LSTMs process sequences step by step, maintaining memory of past information through hidden states

Together, these innovations represented substantial progress toward effective language modeling. However, even LSTMs face limitations. They process information sequentially, which means that information from early in the sequence must pass through many intermediate steps to influence later processing; this can take significant computational resources and the information can degrade or be lost along the way.

The limitation suggested an approach where a word could directly connect to relevant positions without passing information through every intervening step.

---

The core idea of attention

Consider how humans read and understand sentences. You don’t treat every word with equal importance or process them in a purely linear fashion. Instead, you focus more attention on certain words to extract meaning, and you automatically connect related words even when they appear far apart.

Consider the sentence:

The book that the student borrowed from the library was fascinating.

When you reach was fascinating, you immediately understand it describes the book, not the library or the student. You mentally connect these words despite seven intervening words. This selective focusing — paying more attention to relevant words while largely ignoring irrelevant ones — is what attention mechanisms provide to neural networks.

The attention mechanism allows the model to learn which words are most relevant for understanding each word in context, and to weight their contributions accordingly.

---

How attention actually works

The Transformer’s self-attention mechanism uses three learned transformations of each word’s current representation. This may be its input embedding or a representation from a preceding layer:

• Query: Represents what this word is “looking for” in other words — what information it needs to understand itself in context

• Key: Represents what this word “offers” to other words — what information it can provide to help understand them

• Value: Represents the actual information this word carries that will be passed along

To compute attention for a given word:

1. Compute the query vector for that word
2. Compute dot products with the key vectors and divide by the square root of the query/key dimension to obtain scaled compatibility scores
3. Exclude positions that the model is not allowed to use, then apply softmax to obtain weights that sum to 1.0
4. Use these weights to compute a weighted sum of the value vectors

The output mixes information according to the learned query-key compatibility. For a supplied sequence, these matrix operations can run efficiently in parallel across positions on modern hardware, especially GPUs.

Intuitively, you could think of each word “asking questions” through its query, comparing the keys at allowed positions, and blending their values using the resulting weights. These weights describe how the model mixes information; they are not guarantees of relevance or correctness.

---

Self-attention: Applying attention everywhere

In self-attention, the queries, keys, and values come from the same sequence — the sequence attends to itself:

• Each position compares its query with the keys at allowed positions
• Its output combines their values using the attention weights

Bidirectional self-attention, used in a typical encoder, allows each position to attend to all positions, including itself. Causal self-attention, used in an autoregressive decoder, masks later positions: a position can attend only to itself and earlier positions. This prevents a next-token predictor from using future tokens during training.

An allowed connection is direct regardless of the distance between positions. This shortens the path information and gradients must follow, but does not guarantee that the model will preserve or use all relevant information.

The resulting representation is contextualized by the positions it can attend to. A bidirectional encoder can use surrounding words; a causal decoder uses the available prefix.

---

Why attention was key

Attention produces context-specific representations by directly mixing information from allowed positions. For example, bank can have different representations when its context concerns a river or a financial institution. RNNs also produce context-dependent states; attention changes how information from different positions is combined.

This contextual sensitivity allows models to capture the nuanced, context-dependent nature of language where the same word can mean different things in different contexts.

Attention can calculate outputs for the supplied training sequence in parallel, including when a causal mask is used. Autoregressive generation still produces new tokens sequentially. Direct connections make distant relationships easier to learn, but full attention has a cost that grows quadratically with sequence length, and models can still struggle to use long contexts reliably. Experiments on long-context language models demonstrate that access to information does not ensure it will be used successfully.

Parallel computation over supplied positions and shorter paths between them made attention central to the Transformer architecture, which we discuss next.