Neural Networks for Language

Sequential Models (RNNs, LSTMs)

---

Why sequential data is challenging

Many types of data are fundamentally sequential — they consist of ordered elements where position and order carry crucial meaning. Examples include text (sentences and documents), speech (sequences of sounds), music (notes over time), and time series data like stock prices or weather measurements. In these cases, order is essential to interpretation:

• “The cat chased the dog” means something completely different from “The dog chased the cat” — reversing the order changes who is doing the chasing and who is being chased
• Predicting tomorrow’s stock price requires understanding trends over recent days or weeks, not just today’s isolated value
• Understanding spoken language requires processing sounds in the order they occur — hearing phonemes out of order would make speech incomprehensible

A basic feed-forward neural network has no state carried from one invocation to the next. We can give it several observations at once by assigning input positions to successive time steps. This preserves their order because different input positions have different weights:

• For 2 rock-paper-scissors moves: 6 input nodes (2 moves × 3 options each)
• For 3 moves: 9 input nodes
• For 10 moves: 30 input nodes

This is a valid approach used in fixed-context feed-forward language models. Its limitation is a fixed-width input: each prediction uses a fixed number of preceding observations, much like the context window of an n-gram model. Increasing that window requires a larger input representation.

---

Recurrent neural networks (RNNs)

Recurrent neural networks (RNNs) introduced a different approach to handling sequential data. A recurrent connection carries a hidden state from one time step into the next. In a typical RNN, this state is a vector: each new hidden value combines the current input with weighted values from the previous hidden state. The same weights are reused at every step.

This creates a form of memory, allowing information to persist across time steps. Effectively, the network maintains a hidden state that summarizes what it has seen so far in the sequence.

Think of reading a story word by word. As you encounter each new word, you don’t process it in isolation — you carry forward your evolving understanding of what came before. An RNN works similarly: it processes one element of the sequence at a time while maintaining a “hidden state” that summarizes what it has observed so far.

How RNNs process sequences

When processing the sentence “the cat sat,” an RNN operates as follows:

1. Process the word “the” → produce hidden state h₁ (encoding information about seeing “the”)
2. Process the word “cat” along with h₁ → produce hidden state h₂ (encoding information about “the cat”)
3. Process the word “sat” along with h₂ → produce hidden state h₃ (encoding information about “the cat sat”)

Each hidden state serves as both an output and an input for the next time step, creating a chain of processing where earlier information influences later processing.

To train an RNN, we conceptually “unfold” it through time into multiple copies — one for each time step in the sequence. The same weights are shared across all time steps (the network applies the same transformation at each step), and we can apply backpropagation through this unfolded network, a process called “backpropagation through time.”

---

The vanishing gradient problem

RNNs introduced an important capability for handling sequential data, but training them can encounter the vanishing gradient problem.

During training, loss gradients pass backward through many time steps. They are repeatedly transformed by weights and activation derivatives. If these transformations repeatedly shrink the signal, the effect can be exponential. For example, suppose a scalar backward signal is multiplied by 0.8 at every step:

• After 10 steps, 0.8^10 leaves about 10.7% of the original signal
• After 20 steps, 0.8^20 leaves about 1.15%
• After 50 steps, 0.8^50 leaves about 0.00143%

These numbers illustrate one chosen multiplier, not universal time-step limits. Actual behavior depends on the weights, activations, and inputs; repeated amplification can instead produce exploding gradients. When gradients from distant steps become very small, they provide a weak learning signal for dependencies on those steps. See the analysis by Pascanu and colleagues.

This limitation restricts what RNNs can learn, particularly for tasks requiring long-term dependencies like understanding paragraph-level context or translating complex sentences.

---

Long short-term memory (LSTM) networks

To address the vanishing gradient problem, researchers developed Long Short-Term Memory (LSTM) networks in 1997.

LSTMs are specialized RNNs with a more sophisticated internal structure designed specifically to maintain information over long sequences. Instead of a simple recurrent connection passing a hidden state forward, each LSTM unit contains:

• A memory cell with a separate cell state that can retain information across many time steps when its gates preserve that information.

• Three learned gates that control information flow:

  • Forget gate: Decides what information from the memory cell should be discarded (which memories to let fade)
  • Input gate: Decides what new information should be stored in the memory cell (which new memories to form)
  • Output gate: Decides what information from the memory cell should be exposed as output (which memories to use right now)

These gates are themselves small neural networks (typically single layers with sigmoid activations producing values between 0 and 1) that learn appropriate behavior during training. A gate value near 0 means “block this information,” while a value near 1 means “pass this information through.”

Why LSTMs work better

The cell state has a direct path from one step to the next. Along this path, backward signals are multiplied by forget-gate values. Keeping those values near 1 can reduce shrinking; values below 1 can still cause the signal to fade.

The gates learn when to preserve information (keep the forget gate open and the input gate closed), when to incorporate new information (open the input gate), and when to expose stored information through the hidden output (open the output gate). Closing the output gate does not erase the cell state. This learned control allows selective retention and updating.

This architecture can make long-term dependencies easier to learn than with a basic RNN. It does not guarantee perfect retention or eliminate every gradient problem.

---

Limitations that remain

LSTMs represented a major advance and dominated sequence modeling from the late 1990s through the mid-2010s, enabling many applications that standard RNNs could not handle. However, they still face significant limitations: LSTMs process sequences one step at a time in order, making training computationally slow. The hidden state at step 100 depends on step 99, which depends on step 98, and so on — this sequential dependency cannot be eliminated.

Within one sequence, each recurrent state update depends on the previous one. Work across different examples and many operations within a step can still run in parallel. Feed-forward networks can also parallelize operations within a layer, while successive layers still depend on earlier layers.

Furthermore, the gates and memory cell make each LSTM unit more complex than a standard RNN unit, requiring more computation per time step, and they still struggle with sequences of hundreds or thousands of time steps.

These remaining limitations motivated development of attention mechanisms and transformer architectures, which we’ll explore in subsequent chapters.