Rise of GPT and Open-Weight Models

The early wave: 2018-2019

Following the foundational work of GPT-1 and BERT in 2018, researchers quickly began exploring how to scale and improve transformer-based models. The year 2019 saw several influential developments:

GPT-2 (OpenAI, 2019) scaled to 1.5 billion parameters — approximately 10 times larger than GPT-1. It demonstrated that larger models could generate remarkably coherent text across diverse topics and styles, often maintaining consistency over multiple paragraphs. The model could continue stories, answer questions, and adapt to various writing styles with greater fluency than its predecessors.

RoBERTa (Meta AI, 2019) showed that careful optimization of BERT-style training — using more data, longer training duration, larger batches, and refined techniques — could substantially improve performance on language understanding benchmarks without architectural changes.

T5 (Google, 2019) introduced a unified “text-to-text” framework where every language task — translation, summarization, question answering, classification — was treated as converting input text to output text. This elegant simplification influenced how researchers thought about task formulation and model design.

During this period, many models were primarily research tools, but access was not limited to APIs or selected partners. For example, Google released BERT’s code and pre-trained models, and GPT-2 weights were released in stages. Running and adapting these releases still required technical knowledge and suitable computing resources.

The GPT-2 controversy

GPT-2’s release drew attention because OpenAI initially withheld the largest model, citing potential misuse such as automated disinformation and spam. In February 2019, it released only the smallest version, then described as having 117 million parameters. Larger versions followed in May and August 2019.

This cautious approach generated significant media coverage, including articles like The AI Text Generator That’s Too Dangerous to Make Public from Wired. The debate marked one of the first times language models entered mainstream public discourse, raising questions about responsible AI development that continue today.

The 1.5 billion parameter version was released on November 5, 2019. At that time, OpenAI reported that it had not seen strong evidence of misuse, while acknowledging that its observations could not cover all threats. This was a dated assessment, not proof that misuse was impossible. The staged release became a case study in responsible publication.

Scaling up: 2020-2021

The next phase saw dramatic increases in model scale and growing public awareness of language model capabilities, alongside increasing recognition of their limitations.

GPT-3 and the emergent behavior

GPT-3 (OpenAI, 2020) represented a major leap, scaling to 175 billion parameters — more than 100 times larger than GPT-2. Beyond its size, GPT-3 demonstrated a surprising capability: given just a few examples of a task within the prompt, the model could often perform that task reasonably well without any fine-tuning or parameter updates. For instance:

Translate English to French:
sea otter => loutre de mer
peppermint => menthe poivrée
plush giraffe => girafe peluche
cheese =>

Given this pattern, GPT-3 would correctly respond “fromage.” This ability to adapt to new tasks from examples alone suggested that sufficiently large models develop flexible, general-purpose capabilities during pre-training that can be directed through prompting.

This emergent behavior — capabilities not explicitly programmed or trained for — sparked widespread interest and effectively led to a surge in experimentation with prompt engineering, where users crafted specific prompts to elicit desired behaviors from the model.

GPT-3’s release coincided with growing public awareness. Mainstream publications began covering language models more seriously. The New York Times published How Do You Know a Human Wrote This?, and demonstrations of GPT-3 writing articles, poetry, code, and engaging in conversation circulated widely online, generating both enthusiasm and concern.

Other developments in 2020-2021

Microsoft introduced Turing-NLG with 17 billion parameters, and other organizations explored large-scale models. Access varied: some models were available as downloadable weights, others through APIs, and others only through research collaborations. Using them generally involved more setup than a consumer chat interface.

During this period, researchers also focused on understanding and improving model behavior — reducing factual errors (often called “hallucinations”), mitigating biases present in training data, improving factual grounding, and developing techniques to make models more controllable and aligned with user intentions. This work laid the groundwork for the instruction tuning and RLHF techniques discussed in the previous chapter.

The breakthrough: ChatGPT and mainstream adoption

In late 2022, the landscape changed dramatically. OpenAI released ChatGPT on November 30, 2022, a conversational interface powered by GPT-3.5 (an improved version of GPT-3 enhanced with instruction tuning and RLHF, as discussed in the previous chapter).

ChatGPT brought conversational language models to a broad audience through a familiar web-based chat interface. Users could interact with the model without setting up model weights or writing API code.

The Google Trends comparison illustrates changes in search interest around the release. Search interest is not a count of registered or active users, and the comparison depends on the selected terms, region, and time period.

ChatGPT demonstrated that instruction tuning and RLHF had successfully transformed language models from impressive but difficult-to-use research systems into practical tools for everyday tasks that non-technical users could interact with naturally.

The current landscape: 2023-present

Since early 2023, the field has seen explosive growth in both model development and deployment across diverse organizations and contexts.

Proliferation of models

New models now appear regularly from diverse sources:

• Proprietary models from major technology companies continue to advance.

• Open-source and open-weight models have democratized access to language model technology. Platforms like Hugging Face host hundreds of thousands of models contributed by researchers and developers worldwide. These models vary in size, architecture, training data, and intended use cases.

  Open-weight means the trained parameters are available, subject to their license. This does not by itself make a model open source. For example, Llama licenses impose restrictions that the Open Source Initiative identifies as incompatible with open-source requirements. Check the particular model’s license before assuming how it may be used or redistributed.

• Specialized models targeting specific languages, domains, or applications have also started to emerge. Medical language models for clinical documentation and decision support, legal assistants for contract analysis, coding-focused models optimized for software development, and models designed specifically for languages beyond English have all emerged to meet particular needs.

This ecosystem diversity means different users and organizations can choose models appropriate for their specific needs, computational resources, privacy requirements, and deployment contexts.

The scale explosion

The published parameter counts for the first three GPT generations show a substantial increase in scale:

• GPT-1 (2018): 117 million parameters
• GPT-2 (2019): 1.5 billion parameters
• GPT-3 (2020): 175 billion parameters

These published counts should not be extended using estimates for models whose sizes have not been disclosed. The GPT-4 technical report, for example, withholds model-size details. Parameter count alone also does not establish performance: architecture, training data, and the computation used during training and inference matter.