Summary
This part explored how neural networks evolved to handle language, from basic architectures through embeddings and sequential models to transformers powering modern large language models.
We studied neural network basics: layers of nodes with weights and biases, nonlinear activation functions, and training. Backpropagation calculates loss gradients; an optimizer such as gradient descent uses them to update parameters.
We examined word embeddings — dense vectors whose geometry can reflect patterns of use and semantic relationships. Training methods like Word2Vec often give words used in similar contexts similar representations. Nearby words need not be synonyms, and vector analogies such as king - man + woman ≈ queen are learned tendencies rather than guarantees.
We explored sequential models. Recurrent neural networks reuse weights to update a hidden state through a sequence. Training can encounter vanishing gradients when backward signals shrink over many steps. LSTM gates can preserve a separate cell state and reduce this problem, but do not guarantee perfect memory. Their state updates remain sequential.
Attention provides direct connections between allowed positions and can produce context-dependent representations. Bidirectional self-attention uses the supplied sequence; causal self-attention excludes later positions. Shorter paths help with distant dependencies, but access to a long context does not guarantee that the model will use it reliably.
Finally, we studied transformers, introduced in “Attention Is All You Need” (2017). They combine multi-head attention, feed-forward processing, position information, layer normalization, and residual connections without recurrent state updates. Encoder-only models such as BERT and decoder-only models such as GPT use this architecture differently. Computations across supplied token positions can run in parallel during training; autoregressive generation still produces new tokens sequentially.
Key Takeaways
- Backpropagation calculates gradients; the optimizer updates weights and biases.
- Word embeddings can reflect semantic relationships, but similarity does not establish synonymy.
- RNNs carry hidden states through a sequence and can encounter vanishing gradients.
- LSTM gates control memory and help with longer-range learning.
- Attention directly connects positions permitted by the attention mask.
- Self-attention contextualizes representations using the context available to each position.
- Transformers support parallel training over supplied positions; autoregressive generation remains sequential.