Neural Networks for Language

Training a Neural Network

---

A neural network with randomly initialized weights has not yet learned the desired task. Training adjusts its weights and biases to improve its predictions on training examples. We also need to evaluate it on separate examples to check whether that improvement generalizes.

Training repeats a forward pass, a backward pass, and a parameter update over the training data:

Forward propagation

In the forward pass:

• Input data flows through the network layer by layer.
• Each hidden or output node computes its weighted sum of inputs, adds its bias, and applies its activation function.
• Eventually, an output is produced at the final layer.

Example: Input value 5 flows through the network → produces output 0.15.

This forward pass generates a prediction that can be compared to the correct answer.

Backpropagation

After the forward pass, the network evaluates how accurate its prediction was:

The network compares its prediction (the output) to the correct answer (called the label or target). A loss function measures how well the prediction matches that target.

Using backpropagation, the network calculates the loss gradient with respect to each weight and bias. These partial derivatives measure local sensitivity: how a small change in a parameter would affect the loss at the current parameter values. Calculating them leaves the parameters unchanged.

A separate optimizer uses these gradients to update the parameters. With gradient descent, each new parameter equals its old value minus learning rate × gradient. The negative gradient gives a direction of local decrease; a finite update is not guaranteed to reduce the loss.

The learning rate scales each update. A rate that is too large can overshoot good solutions or make training unstable; a rate that is too small can make learning slow. A suitable value depends on the model, data, and optimizer. There is no universally safe numerical range.

This update process repeats for many training examples — often millions or billions of times across multiple passes through the training data (called epochs). Gradually, through this iterative adjustment, the network may learn weight values that allow it to perform its task effectively.

---

An intuition for backpropagation

Consider teaching someone to throw darts at a target:

• The student throws and misses the bullseye by some distance and direction.
• You measure the error (how far off they were and in which direction).
• You provide specific feedback: “throw a bit higher,” “use less force,” “adjust your angle slightly right.”
• Each piece of feedback addresses a different aspect of their technique (analogous to different weights in different parts of the network).
• After many rounds of throws and feedback, the student’s aim improves gradually.

Backpropagation supplies the feedback: gradients describe how the loss responds to small changes in different parameters. The optimizer decides how to use that feedback to change the parameters, including the size of the update.

Backpropagation applies the chain rule from calculus in reverse through the network, reusing intermediate results to calculate gradients efficiently for millions or billions of parameters. This efficiency makes it central to training large neural networks. For a worked mathematical treatment, see forward and backward propagation in Dive into Deep Learning.