Probabilities and Language

Towards Language Models

---

Probabilistic models of natural language

Language models are probabilistic models of natural language. They use probability theory to capture patterns in how language works by analyzing large amounts of text data. Specifically, they estimate quantities such as:

• The probability of a sequence of words: — “How likely is this particular sentence to occur?”
• The probability of the next word given previous words: — “Given the words we’ve seen so far, what word is likely to come next?”

For example, after observing the words “The cat sat on the,” a language model would assign high probability to continuation words like “mat,” “floor,” or “chair,” and very low probability to words like “jumped” or “telescope.” This reflects learned statistical patterns about which words typically follow others in natural language.

Linguist J. R. Firth captured the fundamental intuition underlying this approach:

You shall know a word by the company it keeps.

In other words, a word’s meaning and usage can be understood by examining the contexts in which it typically appears. Language models operationalize this principle by learning statistical patterns from large amounts of text — discovering which words typically occur together, which sequences are common, and which are rare.

---

Experiments with letters

Some of the earliest ideas behind language models trace to Andrey Markov (early 1900s) and Claude Shannon (1940s-1950s). Both studied how letters in text are not independent but instead depend on what preceded them. This represented a crucial insight: you cannot treat each letter as a random, isolated event if you want to understand or generate realistic text.

In English, the letter “q” almost always precedes “u,” and “h” frequently follows “t” but rarely follows “x.” By counting how often one letter follows another in actual text, clear patterns emerge. The table below uses the first 50,000 characters of a cleaned version of Shakespeare’s Romeo and Juliet, with letters lowercased and punctuation and whitespace replaced by spaces. Rows represent the current letter, and columns represent the next letter. Each cell estimates the conditional probability of the column letter following the row letter: the count of that pair divided by the count of all observed transitions from the row letter.

Only five common letters are shown as rows and columns. The denominator includes every observed following character, including spaces and letters not displayed. The five visible probabilities in a row therefore need not add up to one.

Current / next	e	t	o	a	i
e	0.0390	0.0563	0.0274	0.0685	0.0126
t	0.0652	0.0121	0.0814	0.0349	0.0470
o	0.0050	0.0533	0.0459	0.0053	0.0059
a	0.0000	0.1150	0.0003	0.0000	0.0493
i	0.0613	0.1234	0.0485	0.0158	0.0094

Compare the probabilities within a row. For example, after “a”, is “t” more likely than “a” or “e” in this sample? The crucial point is that meaningful structure emerges simply from counting patterns in data. You don’t need to encode grammar rules or semantic meanings to capture these patterns.

To further illustrate this, using the search function of your browser, try first to search “et” on this page. Then, try to search for “ae”. Certain letter combinations are much more common than others!

Claude Shannon’s experiments demonstrated the power of statistical patterns in language through influential text generation experiments. Shannon explored generating text by:

1. First, selecting letters completely at random, ignoring all structure and producing pure nonsense.
2. Then, choosing letters based on their observed likelihood of following previous letters, using statistics like those shown in the visualization above.

Random letters with no structure produce output like:

XFOML RXKHRJFFJUJ ZLPWCFWKCYJ FFJEYVKCQSGHYD QPAAMKBZAACIBZLHJQD

This is completely unreadable and resembles no natural language.

In the following example from Shannon’s 1948 paper, each letter is chosen using the preceding two letters. Shannon calls this a third-order approximation because it preserves three-letter, or trigram, statistics. The results begin resembling English, even though most of the “words” remain meaningless:

IN NO IST LAT WHEY CRATICT FROURE BIRS GROCID PONDENOME OF DEMONSTURES OF THE REPTAGIN IS REGOACTIONA OF CRE

Notice this second example contains more plausible letter combinations. You can almost perceive English words forming, even though most are nonsensical. This demonstrates that capturing even simple statistical dependencies between letters produces text resembling natural language in its surface structure.

Shannon also proposed a guessing game to demonstrate how humans use implicit knowledge of language structure to predict what comes next. In this game, you attempt to guess each successive letter in hidden text. Your ability to make accurate guesses — often predicting the correct letter on the first or second attempt — reveals how much statistical structure exists in language and how thoroughly humans internalize these patterns. Try it:

Shannon's Game

Clicking the "Make Guess" button opens up a prompt that asks for a character. Type in a character and press submit. If the guess is correct, you move to guessing the next character.

Correct guesses: 0 / 0

Passage so far: L - - - - - - -

Make Guess

As you play, notice certain letters prove much easier to predict than others. After “th,” you might correctly guess “e” (forming “the”), but after an uncommon combination, the next letter becomes considerably harder to predict. When context provides strong constraints, prediction becomes easier; when context is less informative, uncertainty increases.

---

From letters to language models

These early experiments revealed a profound insight: even simple statistical rules based on letter frequencies can produce text that resembles natural language in its surface structure. The key principle is that language contains statistical regularities that can be captured and exploited for both prediction and generation.

This approach scales. If counting letter pairs produces somewhat English-like text, then counting word pairs, word triples, or longer sequences should produce increasingly realistic results. This progression — from letters to words to longer sequences — traces the path that led from early experiments to n-gram models and eventually to modern neural language models.

The fundamental idea remains consistent across this evolution: language exhibits statistical patterns, and models that capture these patterns can predict and generate language-like sequences. What has changed is the sophistication of the models and the scale of data they learn from.