Neural Networks for Language

Transformers

---

From sequential processing to pure attention

Earlier chapters showed that RNNs and LSTMs process sequences one word at a time:

• They maintain hidden states that carry information forward through the sequence
• This approach works reasonably well for shorter texts but becomes slow and inefficient for long sequences
• Important information can degrade as it passes through many sequential steps

Self-attention helps address these problems by connecting positions directly wherever the attention mask allows. Self-attention alone does not constitute a complete model; the Transformer combines it with other components.

---

The transformer architecture

In 2017, researchers at Google introduced the Transformer in their paper Attention Is All You Need.

The title reveals the central insight: instead of relying on recurrence (RNNs/LSTMs) to process sequences, transformers rely on attention to model relationships between positions in sequences.

Core components

Transformers combine several important features:

• Self-attention layers mix information from allowed positions through direct connections. Bidirectional attention can use the entire supplied sequence; causal attention excludes later positions. Multiple heads learn different query, key, and value projections, and their outputs are combined.

• Stacked layers repeat the attention and processing multiple times. Each layer refines the representations further, learning increasingly abstract patterns and relationships. Modern transformers typically stack 12, 24, or even 96+ layers, with each layer building on the representations from the previous layer.

• During training, computations for all supplied positions within a layer can run in parallel. Each layer still depends on the previous layer. This supports efficient training on parallel hardware such as GPUs.

The architecture also includes position-wise feed-forward layers (processing each position’s representation through the same neural network), layer normalization, and residual connections. Position information lets the model distinguish where tokens occur. The original Transformer adds positional encodings to its input embeddings; self-attention alone does not specify token order.

---

Two components: encoder and decoder

The original transformer architecture had two main components serving different purposes:

• Encoder uses bidirectional self-attention to build representations of the input sequence
• Decoder uses causal self-attention over its output prefix and cross-attention to the encoder’s representations; during generation, it produces output tokens one at a time

This encoder-decoder design proved powerful for sequence-to-sequence tasks like machine translation, where you need to both understand the input and generate appropriate output.

However, modern applications typically use just one of the components, depending on the task:

• Encoder-only models excel at understanding and analysis tasks: text classification, question answering, information extraction, and sentiment analysis
• Decoder-only models excel at generation tasks: text completion, dialogue, creative writing, and code generation

Many generative large language models, including GPT models, are decoder-only. They use causal self-attention without the original architecture’s encoder or encoder-decoder cross-attention. Parallel training on known text does not remove the sequential dependency when generating new tokens.

The foundation for LLMs

This architectural foundation enabled the development of large language models that have transformed natural language processing:

• BERT (2018) with 340 million parameters demonstrated the power of large-scale pre-training on masked language modeling, achieving state-of-the-art results across many understanding tasks.

• GPT-2 (2019) with 1.5 billion parameters showed impressive text generation capabilities and the beginnings of few-shot learning — performing tasks from just a few examples.

• GPT-3 (2020) with 175 billion parameters exhibited emergent capabilities like few-shot and zero-shot learning, where the model could perform many tasks without task-specific training.

• Modern models continue scaling to hundreds of billions or even trillions of parameters, with capabilities continuing to expand.

In the next part of this course, we’ll explore how these models are trained at scale, what capabilities emerge from that training, the techniques used to align them with human intentions, and the recent trends in the area.