Neural Networks for Language

Embeddings and Word Representations

---

Why embeddings?

Neural networks operate exclusively on numbers — they perform mathematical operations like multiplication, addition, and applying activation functions to numerical values. However, much of the data we want to process — text, images, categorical variables — is not inherently numerical. Words like “cat” or “understand” are symbols, not quantities that can be directly added or multiplied.

We need a way to convert these non-numeric entities into numerical representations that computers can process mathematically. This conversion process is called creating an embedding: a numerical representation that captures important properties of the original data in a form that neural networks can work with.

The challenge is not merely to assign numbers arbitrarily, but to create representations that preserve meaningful relationships from the original domain. A good embedding should encode relevant information in a way that the neural network can exploit for its task — similar items should have similar representations, and important differences should be reflected in the numerical structure.

---

A simple example: Rock-paper-scissors

To understand the challenge of creating good representations, consider how we might encode the three moves in rock-paper-scissors as numbers.

Naive approach: Assign simple integers: rock = 0, paper = 1, scissors = 2.

Problem: This representation implies an artificial ordering, as if scissors (2) is “greater than” paper (1), which is “greater than” rock (0). This ordering doesn’t reflect the actual game relationships where each move beats one option and loses to another in a circular pattern. A neural network might incorrectly learn patterns based on this false numerical ordering, potentially treating the difference between scissors and rock as larger than the difference between rock and paper simply because 2 - 0 = 2 while 1 - 0 = 1.

Better approach: Use a vector with one position for each possible move, placing a 1 in the position corresponding to the chosen move and 0 everywhere else:

• Rock = [1, 0, 0]
• Paper = [0, 1, 0]
• Scissors = [0, 0, 1]

This representation is called one-hot encoding. Each move is represented by a vector of length 3 (the number of possible moves), with exactly one “hot” (set to 1) position. This avoids creating false orderings — rock, paper, and scissors are all equally distant from each other in this representation (the distance between any two different one-hot vectors is the same), with no move privileged over others.

If we built a neural network to predict rock-paper-scissors moves, the input layer would have three nodes (one for each position in the input vector), and the output layer would also have three nodes (producing probabilities for each possible next move). Figure 1 below shows such a network with one hidden layer with two nodes.

Figure 1 — A simple neural network for predicting rock-paper-scissors moves using one-hot encoded inputs and probability outputs.

Depending on the data, weights, biases, and activation functions, the above network could learn to predict the next move based on the current move. For example, if the data has many instances where rock [1, 0, 0] is followed by paper [0, 1, 0], the network might learn to associate rock with a high probability of paper. For input [1, 0, 0], a possible probability output is [0.05, 0.90, 0.05]. The one-hot vector [0, 1, 0] is the training target when the actual next move is paper.

Predicting the opponent’s next move and choosing our own response are separate tasks. If the opponent’s last move was rock and the model predicts paper next, we could choose scissors to beat that predicted move. The outcome depends on what the opponent actually plays next.

---

Word embeddings

Representing words as numbers presents a far more complex challenge than rock-paper-scissors. Unlike game moves, words have rich semantic relationships that we would like to capture:

• Words like “understand” and “comprehend” are nearly synonymous — they have very similar meanings and can often substitute for each other in sentences.

• Words like “understand” and “eat” are semantically distant — they refer to completely different concepts with no obvious relationship.

• Some words share partial meaning: “cat” and “kitten” both refer to felines, but carry different connotations about age and size.

If we used one-hot encoding for words (as we did for rock-paper-scissors), we would create vectors like:

• “understand” = [0, 0, 0, 1, 0, 0, ...] (assuming it’s the 4th word in our vocabulary)
• “comprehend” = [0, 0, 0, 0, 0, 1, ...] (perhaps the 6th word)
• “eat” = [1, 0, 0, 0, 0, 0, ...] (perhaps the 1st word)

With one-hot encoding, every word is represented by a vector that is equally distant from every other word — “understand” is just as different from “comprehend” as it is from “eat” in terms of vector distance. The representation completely ignores semantic relationships between words. For a vocabulary of 50,000 words, we would need 50,000-dimensional vectors where each word differs from every other word in exactly two positions (its own position and the other word’s position), making all pairwise distances identical.

To address this limitation, researchers developed word embeddings: dense, lower-dimensional numerical representations of words that encode information about their meanings and relationships.

Word embeddings are typically dense vectors with continuous values like [0.23, -0.11, 0.76, 0.45, -0.33, ...] rather than sparse one-hot vectors containing mostly zeros. Training on text often gives words with similar patterns of use similar representations. For example, “understand” and “comprehend” may be closer than “understand” and “eat”, depending on the training data, task, and similarity measure.

---

How embeddings capture meaning

One way to learn word embeddings is to train neural networks on language tasks. The key insight underlying this approach is that words used in similar contexts often have related meanings — an idea rooted in the linguistic principle we mentioned earlier: “You shall know a word by the company it keeps.”

Common training tasks include:

• Predicting a word from its context: Given surrounding words with a blank like “I ??? the meaning of the word”, the task would be to predict the word for the position ???. This would allow learning that “understand” is a good fit. At the same time, words like “comprehend,” “grasp,” or “know” would also fit well — during training, the network learns to assign similar representations to words used in similar contexts.

• Predicting the next word in a sequence (language modeling): Given “I understand the meaning of the ???” predict what word comes to ??? — likely “word,” “sentence,” “text,” or similar completions.

During training on text, an optimizer updates the network’s weights using gradients calculated by backpropagation. Words used in similar contexts — like “understand” and “comprehend” — often develop similar representations that support similar predictions.

There is no need to explicitly (or manually) state that “these words mean similar things” — similarity emerges naturally from training on huge amounts of text data.

The resulting embedding space can exhibit patterns such as:

• Related words appearing nearby: “cat,” “kitten,” and “feline” may occupy nearby regions
• Relationships reflected in vector directions: The offset from “king” to “queen” may resemble the offset from “man” to “woman”
• Different usage reflected in different vectors: Words used in different contexts may be farther apart

These are learned tendencies, not guarantees about meaning. Opposites such as “hot” and “cold” can be close because they occur in similar contexts. A single similarity score cannot describe every relationship between two words, as the GloVe authors explain.

---

Learning word embeddings with neural networks

A typical architecture for learning word embeddings looks like this:

Input layer: One node for each word in the vocabulary. For a vocabulary of 50,000 words, this layer has 50,000 nodes. The input is represented as a one-hot vector indicating which word we’re currently processing.

Hidden layer (embedding layer): A much smaller set of nodes — typically a few hundred in classic word-embedding models, with larger dimensions also common. This sets the width of each word’s representation. The training task encourages useful patterns in these vectors; a smaller width alone does not guarantee meaningful representations or prevent memorization.

Output layer: Predicts the target word(s) for the training task. For example, this might output probabilities for each word in the vocabulary appearing in the predicted position.

During training:

• The network processes thousands or millions of examples from text corpora.
• An optimizer updates the weights connecting the input and hidden layers using gradients calculated by backpropagation.
• These learned weights become the word embeddings. Each word’s embedding is the set of weights from its one-hot input node to all the hidden layer nodes.

For example, if the hidden layer has 300 nodes, each word is represented by a 300-dimensional vector — the 300 weights connecting that word’s input node to the hidden layer nodes. These vectors are dense (most values are non-zero) and continuous (values can be any real number, not just 0 or 1).

That is, a single word embedding vector might have consisted of just 300 numbers (the weights from the input node representing the single word to the hidden layer nodes), rather than a sparse vector of 50,000 dimensions with only one non-zero entry.

After training, we can compare word vectors using measures such as cosine similarity, the cosine of the angle between two nonzero vectors. Vectors pointing in similar directions have high cosine similarity. This can indicate related patterns of use, but does not establish that the words are synonyms.

---