Infrastructure and Environment
Learning Objectives
- You understand why training large language models requires massive computational infrastructure.
- You can identify the major cost factors: hardware, energy, and engineering effort.
- You know the environmental impacts of AI training and deployment.
The infrastructure challenge
Training large language models combines algorithm design, data preparation, and infrastructure engineering. Scale is a major challenge for frontier models, while improvements in algorithms and data can change how much computation is needed.
Training frontier models can involve trillions of tokens, thousands of specialized processors, high-speed networks, and large storage systems. Requirements vary substantially with model size, training data, and methods; smaller models and adaptation of existing models can require much less.
The following discussion focuses on the infrastructure demands of large training runs and deployments.
Infrastructure challenge
0 / 5 points
What makes training large language models primarily an infrastructure challenge rather than an algorithmic one? (Select all that apply)
Hardware and infrastructure requirements
Large language models are trained on specialized hardware designed for high-throughput numerical computation. These include GPUs (Graphics Processing Units) and TPUs (Tensor Processing Units), which can perform the massive number of matrix multiplications required for training neural networks far more efficiently than general-purpose CPUs.
For example, NVIDIA’s account of GPT-3 training states that it used V100 GPUs on part of a Microsoft cluster. The article separately describes a Microsoft supercomputer containing 10,000 GPUs; that is not a reported allocation of all 10,000 GPUs to the GPT-3 training run. CPUs, memory, storage, and networking also have to support the required data throughput and parallelism.
At the moment, the main provider of AI training hardware is NVIDIA, whose GPUs dominate the market. Visit NVIDIA’s stock price page to see how the market values this dominance — note that NVIDIA has done multiple stock splits, so the price per share is not directly comparable over time.
The data center must also supply power and cooling. Energy use depends on the hardware, utilization, duration of the workload, and facility efficiency. Comparisons need a stated time period and measurement boundary: power at a moment and energy consumed over a training run are different quantities. The IEA’s Energy and AI report explains these sources of variation in data-center electricity demand.
For more details on energy consumption, see e.g. How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inference.
Hardware requirements
0 / 5 points
Which statements accurately describe the hardware requirements for training LLMs? (Select all that apply)
Environmental impacts
In addition to the direct energy consumption and the associated carbon emissions (which depend on the energy source), there are also broader environmental impacts to consider. In particular, the hardware supply chain and water usage.
First, AI systems depend on hardware whose materials must be extracted and refined, manufactured into components, and transported through global supply chains. These processes use energy and can cause pollution, habitat damage, and greenhouse gas emissions. The impact depends on the materials, manufacturing processes, and energy sources involved.
The costs of manufacturing and distributing the hardware are often overlooked compared to electricity but are significant.
Furthermore, as AI data centers require substantial cooling to prevent overheating, they often rely on water-intensive cooling methods. In water-scarce regions, this can strain local water supplies and impact ecosystems.
Energy and environment
0 / 5 points
Which factors contribute to the environmental impact of training LLMs? (Select all that apply)
Broader implications
As the costs of training LLMs is high and rising, several broader implications arise:
-
Only well-resourced organizations can afford to train the largest models from scratch, concentrating capabilities and raising questions about equitable access to AI technology. Currently, companies like OpenAI, Google, Microsoft, and Meta invest more in AI infrastructure than many countries spend on research and development.
-
As AI deployment and use grows, the cumulative energy consumption and environmental impact become significant, prompting calls for more sustainable practices in AI development. Currently, many search engines are also integrating AI features, leading to increased computational load and energy use even for routine queries.
-
As access to advanced semiconductors has become a strategic consideration for nations seeking to develop AI capabilities, there are geopolitical implications on supply chain, including minerals. For instance, countries have been restricting exports of advanced chips to competitors.
Geopolitical implications
0 / 5 points
Which geopolitical factors are linked to AI infrastructure and environment? (Select all that apply)