Four years after OpenAI’s GPT-3 achieved 43.9 percent on a knowledge-and-reasoning test in 2020, its successor, GPT-4o, has reached 88.7 percent on the same exam. The models grew much more intelligent at a rapid pace, and today the business of operating them is consuming the entire industry.
The shift is simple to state and hard to ignore. Training ever-larger models remains the headline, but the actual work of inference — the moment a model produces a response to a prompt — has become the daily job. Matt Kimball, principal data-center analyst at Moor Insights & Strategy, put it bluntly: “It’s like training is yesterday’s news.” Nvidia CEO Jensen Huang framed the same point at the company’s GTC 2026 conference, calling the change the “inflection point of inference.”
Why Inference Took Over
People now actually find value in LLMs, which is what’s driving their use. These systems answer questions, compose code, and produce images of users rendered as elves. That value builds demand, and demand builds adoption. The change goes beyond mere usefulness, though. Today, many models operate as reasoning models, running inference repeatedly — prompting themselves again and again in a process known as chain of thought. This approach produces longer outputs, and some models with high reasoning effort can generate up to 20 times as much text as models with low or no effort.
Agentic AI systems don’t just process queries when a person asks something; they keep running inference around the clock, working toward a user-defined goal. This constant operation adds more processing on top of what would otherwise be the standard amount.
Chips Change Sides
Unexpected partnerships have come about because of the demand. OpenAI and Amazon have used dinner-plate-sized chips created by Cerebras, despite Amazon making its own Trainium chips. Nvidia secured key talent and intellectual property from AI-inference startup Groq through a deal valued at $20 billion. Anthropic is leasing spare compute from LLM competitor SpaceXAI for more than a billion dollars per month.
These agreements point toward a major change in how reasoning is carried out. The combination of machines required to back it up now diverges from what experts predicted even a couple of years ago.
Training vs. Inference
Consider the distinction between training and inference through a comparison. A language model that has not yet been trained resembles a collection of Scrabble tiles scattered upon a table. Among these tiles, which represent individual pieces of words known as tokens, all the building blocks for nearly any composition exist. Yet none of it coheres into anything that can be read or understood.
Training arranges that jumble through a guessing game played on a large scale. The model encounters real text with the next token concealed and makes a guess about what follows. Once the guess is made, the correct token is shown, the discrepancy is measured, and the model adjusts its parameters using backpropagation — a method that works again and again to determine how each of a model’s billions or trillions of parameters ought to change to improve the next prediction.
The process works in reverse compared to training. Once the model has learned from its data, its numbers stay fixed. It takes in a prompt, weighs how each word connects to every other word, and returns an answer. The calculations needed to adjust those numbers are no longer part of the process. Still, the task does not stop at that point.
The Chain-of-Thought Problem
Sudeep Bhoja, founder and CTO of the inference-hardware company d-Matrix, explains that inference adds new challenges. Models are autoregressive — the next output depends on the previous one. “So to generate the next token, you have to read all of the weights and all of the context from the previous token,” he says. The context includes all prompts, all of the LLM’s replies, and all uploaded files. It’s a lot of data and a lot of processing.
An LLM generates its reply in two phases:
- Prefill is the model reading a prompt. It processes every token at once, computing how each token relates to all the others. This operation is called attention, a defining characteristic of the transformer architecture behind modern LLMs. Attention allows the model to respond to a word in its sentence, paragraph, and larger context rather than on its own.
The Context Problem
A vast amount of information has to be kept track of for the system to work properly. Every piece of text produced relies on the details of what came before it — every question, every answer, every document added. So the model needs to hold onto and go through the entire record of the exchange.
What the Deals Mean
Nvidia’s Groq 3 language-processing unit is built to minimize data movement, with on-chip SRAM memory and computational blocks arranged in the sequence they are required. SRAM is a fast, low-latency memory type that reduces the time spent moving data between operations. Amazon has divided AI inference into two parts, assigning Trainium to handle the more computationally complex portion and leaving the more memory-intensive portion to Cerebras’s wafer-scale engine.
What Comes Next
The inference hardware revolution of 2026 is real, and it’s accelerating. The demand curve is steep, and the alliances keep forming.
This summary draws on a broader movement, and its weight rests upon one individual’s remark and a single conference quote. The sense behind it is real: reasoning has come to rule.
Get the Notebook.
The day's best stories and every fresh verdict, in plain English, in your inbox by seven. One email a day, no more.