Large Language Models

Instruction Tuning and RLHF

---

From general models to conversational assistants

In the previous chapter, we examined the two-phase training paradigm: pre-training followed by fine-tuning. Early work from 2018-2020 demonstrated that this approach could produce models capable of impressive feats — generating coherent text, answering questions, and performing various language tasks.

However, these early models had a fundamental limitation. When you interacted with them, they didn’t behave like helpful assistants. Instead, they acted like sophisticated text completion engines. As an example, a user might type a question, and — instead of answering the question — the model could generate text that plausibly continued from that input, such as more similar questions.

The model had learned language patterns but not how to be helpful. It could generate fluent text but didn’t understand that it should function as an assistant responding to user requests rather than a system predicting likely text continuations.

During 2021 and 2022, instruction tuning and reinforcement learning from human feedback (RLHF) became prominent approaches to adapting language models into conversational assistants. They build on earlier work on learning from demonstrations and human preferences.

---

Instruction tuning: Learning to follow instructions

Instruction tuning is a specific type of fine-tuning that trains models on examples of helpful responses to instructions. A generative language model still predicts subsequent tokens, but training changes which continuations it favors.

During instruction tuning, the model undergoes fine-tuning on datasets specifically constructed to demonstrate instruction-following behavior. These datasets contain thousands to hundreds of thousands of examples, each consisting of an instruction-response pair. The examples cover diverse tasks and interaction styles to develop general instruction-following capabilities.

For instance:

• Instruction: “Translate the following sentence to Spanish: ‘The library closes at 8 PM.’”

• Response: “La biblioteca cierra a las 8 PM.”

• Instruction: “Explain what causes ocean tides in 2-3 sentences.”

• Response: “Ocean tides are caused primarily by the gravitational pull of the Moon on Earth’s oceans. As the Moon orbits Earth, its gravity creates bulges in the ocean water on the side facing the Moon and the opposite side. As Earth rotates, different locations pass through these bulges, experiencing the cycle of high and low tides.”

• Instruction: “Write a haiku about autumn.”

• Response: “Crisp leaves underfoot / Golden sunlight fading fast / Cool winds whisper change”

• Instruction: “List three renewable energy sources and briefly describe each.”

• Response: “1. Solar power captures energy from sunlight using photovoltaic panels or thermal collectors. 2. Wind power uses turbines to convert wind’s kinetic energy into electricity. 3. Hydroelectric power generates electricity from flowing water, typically through dams on rivers.”

Through exposure to many such examples, the model learns relevant behaviors, including recognizing instructions, structuring responses appropriately, and adapting tone and detail level based on the request.

Instruction tuning makes useful instruction-following responses more likely. It changes learned behavior while retaining the next-token generation process.

---

Reinforcement learning from human feedback

Instruction tuning makes models responsive to instructions, but responsiveness alone doesn’t ensure quality. A model might technically follow an instruction but still produce responses that are unhelpful, misleading, inappropriate, unnecessarily verbose, or poorly structured. We need models that not only understand what the user asked but also produce responses that humans would judge as genuinely useful and appropriate.

Reinforcement learning from human feedback (RLHF) is a training technique that teaches models to prefer responses aligning with human preferences. Unlike instruction tuning (which uses supervised learning on instruction-response pairs), RLHF employs reinforcement learning guided by human judgments of quality.

The three stages of RLHF

A common RLHF pipeline involves three stages, each building on the previous one:

Stage 1: Collecting human preference data

Human annotators compare multiple model outputs and rank them by quality. The process typically works as follows:

• Present the same prompt to the model multiple times, generating different responses (models can produce varied outputs due to sampling randomness in the generation process)
• Show these responses to human annotators
• Ask annotators to rank the responses according to criteria like helpfulness, accuracy, harmlessness, and clarity

For example, given the prompt “Explain how vaccines work,” the model might generate three responses. Annotators would rank them, typically placing the accessible and comprehensive response first.

This process creates a dataset of human preferences — not explicit instruction-response pairs, but comparative judgments showing which outputs humans prefer over others.

Stage 2: Training a reward model

Using the preference data, researchers train a separate neural network called a reward model (sometimes called a preference model). This model learns to predict human preferences: given a prompt and a potential response, it outputs a numerical score estimating how much humans would prefer that response.

The reward model essentially learns to approximate human judgment. If trained on sufficient preference data covering diverse scenarios, it can evaluate new responses without requiring human review of each one. This is crucial because reinforcement learning requires evaluating thousands or millions of candidate responses — far too many for humans to review directly.

The reward model doesn’t generate text responses — it is only used to evaluate responses generated by the language model, assigning higher scores to responses that match patterns humans preferred in the training data.

Stage 3: Optimizing the language model with reinforcement learning

When a reward model has been trained, a language model can be fine-tuned using reinforcement learning. The goal is to adjust the language model’s parameters so that it generates responses that receive higher scores from the reward model. The process works as follows:

1. The language model generates a response to a prompt
2. The reward model evaluates this response and assigns a numerical score
3. The language model’s parameters are adjusted using reinforcement learning algorithms to make higher-scored responses more likely in future generations
4. This process repeats across many prompts and generated responses

Through many iterations, training increases the likelihood of outputs that receive higher reward scores. Those scores approximate preferences in the feedback data; optimizing them does not guarantee helpfulness or truthfulness. Unlike supervised imitation of target responses, this stage uses generated responses and reward feedback to guide parameter updates.

What RLHF teaches the model

Through RLHF, models learn preferences that are difficult to capture through instruction tuning alone:

• The model learns when to be concise and when to elaborate. For a straightforward factual question, brevity is typically preferred. For a complex topic where the user is seeking deep understanding, thorough explanation with examples is better.

• Training can encourage the model to acknowledge uncertainty when evidence is missing or conflicting. This is a goal, not a guarantee: preference judgments and reward models can favor convincing answers even when they are wrong. Research on sycophancy shows that preference optimization can also encourage agreement with a user’s beliefs at the expense of accuracy.

• The model learns to decline inappropriate requests politely and to avoid generating harmful content, even when the request seems superficially reasonable or the harm isn’t immediately obvious.

• The model learns subtle aspects of effective communication — appropriate tone for different contexts, clear structure, natural language patterns, and helpful framing of information that aids understanding.

These preferences emerge from patterns in aggregated human feedback rather than explicit programmed rules, allowing the model to generalize to novel situations it wasn’t explicitly trained on.

---