Large Language Models

Pre-training and Fine-tuning

---

Many large language models are developed through pre-training followed by further adaptation. Pre-training builds broad language capabilities, while subsequent training can adapt them to particular tasks and desired behaviors. A pre-trained model can also be used directly, and adaptation may involve several stages rather than one fixed second phase.

Phase 1: Pre-training

Pre-training is the phase where a model learns structure and patterns of language. Large models can train on hundreds of billions to trillions of tokens drawn from sources such as books, websites, and scientific papers. The amount of data varies by model.

As in learning word representations, training learns useful representations. Transformer layers produce context-dependent representations of tokens. In a causal next-token model, each position can use only that token and preceding tokens; masked-language models such as BERT can also use later visible tokens.

During pre-training, the model learns to perform simple prediction tasks such as predict the next token in a sequence, or predict missing tokens within sequences. For example:

• Given “The cat sat on the”, the model learns to predict “mat” (or other plausible completions like “floor” or “chair”)
• Given “In summary, the main findings”, the model learns to predict tokens like “are”, “were”, “suggest”, or “indicate”

By repeating prediction tasks across large amounts of text, the model gradually learns several types of knowledge:

• Grammar rules, syntax patterns, and how sentences are properly constructed across different registers and styles.

• How words and concepts relate to each other in meaning, common phrases and idioms, typical ways concepts are expressed, and relationships between related terms.

• Information that appears frequently in training data, such as historical facts, scientific concepts, geographic information, and general world knowledge — though this knowledge reflects patterns in the training data rather than verified facts.

• How arguments are constructed, how problems are solved step-by-step in text, how logical relationships are expressed, and patterns of explanatory discourse.

A key part of pre-training is that it does not require human annotation or labeled examples. The text itself provides supervision: every next token in the corpus serves as the correct answer for that prediction. This self-supervised learning approach allows using virtually unlimited amounts of text data without the prohibitive cost of manual labeling.

After pre-training, a model can already perform tasks such as completion, translation, or question answering when given suitable prompts. It is not necessarily optimized to follow instructions reliably or meet a particular application’s requirements. For example, GPT-3 demonstrated task performance from examples in the prompt without further parameter updates.

---

Phase 2: Fine-tuning

A pre-trained model provides a starting point for further adaptation. Fine-tuning means continuing to train a pre-trained model to improve its performance on particular tasks, domains, or behaviors.

Fine-tuning often uses a smaller, more focused dataset. In supervised fine-tuning, the model receives examples of inputs paired with desired outputs. Other adaptation approaches use self-supervised training on domain-specific text or optimize responses using feedback. The common feature is further parameter training, not one required dataset format.

Common fine-tuning objectives include:

• Task-specific applications: Training the model to excel at particular tasks like summarization (condensing long documents while preserving key information), translation (converting between languages while maintaining meaning and style), or question answering (providing accurate, relevant responses to queries).

• Domain adaptation: Specializing the model for specific fields like medicine, law, or scientific research by training on domain-specific texts and expert-curated examples that reflect terminology and reasoning patterns particular to those domains.

• Behavioral alignment: Teaching the model to follow instructions, engage in helpful conversation, maintain appropriate tone, and avoid problematic outputs. We’ll explore this crucial application in detail in the next chapter.

Fine-tuning datasets are much smaller than pre-training corpora — typically thousands to millions of examples rather than billions of tokens. For instance, a fine-tuning dataset for summarization might include:

• Input: [Full news article about climate policy]
• Target output: “New legislation aims to reduce carbon emissions by 40% by 2030 through renewable energy incentives and stricter industrial regulations.”

Or for question answering:

• Input: “What is the capital of France?”
• Target output: “The capital of France is Paris.”

By training on such examples, the model learns to apply its general language understanding to the specific format and requirements of the target application.

The two-phase paradigm has proven to be highly efficient. Pre-training requires enormous computational resources — often millions of dollars in computing costs and weeks or months of continuous training on specialized hardware. However, this expensive phase only needs to be done once. The resulting pre-trained model serves as a foundation that can be fine-tuned many times for different applications, each time with modest computational cost (often hours or days rather than weeks). This allows a single pre-training effort to support dozens or hundreds of specialized applications.

---

Historical milestones

Several influential papers in 2018 demonstrated the benefits of language-model pre-training followed by task-specific fine-tuning:

ULMFiT (Howard and Ruder, first published in January 2018) demonstrated that a pre-trained language model could be adapted effectively to multiple text-classification datasets.

GPT-1 (OpenAI, 2018) demonstrated this approach using a generative transformer. It showed that pre-training followed by task-specific fine-tuning could outperform strong task-specific systems on several language-understanding benchmarks. Researchers could reuse a pre-trained model across tasks instead of training each model from scratch.

BERT (Google, 2018) introduced a complementary pre-training approach based on “masked language modeling” — predicting randomly hidden tokens throughout sequences rather than only predicting the next token. BERT demonstrated that benefits of large-scale pre-training extended beyond generative tasks to understanding-focused applications.

These papers helped establish pre-training and adaptation as a widely used approach, with strong results across the tasks they evaluated.

For reference:

• Universal Language Model Fine-tuning for Text Classification (Howard and Ruder, 2018)
• Improving Language Understanding by Generative Pre-training (OpenAI, 2018)
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Google, 2018)

---