OpenAI inference costs cut by half with software: GPU count drops to hundreds for guest traffic

OpenAI has achieved a software-only optimization that reportedly cuts inference costs by more than half for the ChatGPT guest tier, reducing the required Nvidia GPUs to a couple hundred. The gain comes from better utilization of existing infrastructure rather than new hardware, though the exact technique remains undisclosed. This development matters to AI builders because inference costs are the primary barrier to AI profitability and API pricing. If the gains generalize beyond guest traffic, they could lead to lower API prices and higher usage limits.

What happened

In June 2026, OpenAI engineers demonstrated a software optimization that, when applied to logged-out ChatGPT traffic, cut the number of Nvidia GPUs needed from an estimated tens of thousands to roughly a couple hundred, according to The Information citing an internal source. The gain comes entirely from improved utilization of existing server infrastructure, with no new hardware deployed. OpenAI has not publicly disclosed the technique.

Separately, on June 24, 2026, OpenAI and Broadcom unveiled Jalapeño, a custom inference accelerator manufactured by TSMC that targets roughly 50% lower cost per token in early testing. Full production is expected by 2027-2028. The software optimization and hardware roadmap form two phases of OpenAI’s inference strategy.

Why AI builders should care

Inference cost is the ongoing operational expense of serving every query. OpenAI spent $5.02 billion on Azure inference alone in the first half of 2025, making it the central obstacle to profitability. A software-only reduction of this magnitude shifts the competitive landscape from who has the most GPUs to who can make the same chips produce more output per dollar.

If the optimization holds, it creates room for lower API prices, higher usage limits, or improved margins. The techniques analysts suspect - KV cache reuse, quantization, in-flight batching, and query routing - are established methods that compound when tuned to a specific traffic profile. Any AI builder hosting models at scale should watch whether these gains reach the API tier.

Practical implications

The software optimization currently applies only to the guest tier, which produces simpler, more predictable traffic than free-tier, paid, or API workloads. If the same technique generalizes, API pricing could face downward pressure, and usage caps could rise for multiple tiers. OpenAI’s custom Jalapeño hardware, if it delivers the claimed 50% cost reduction, could compound software gains and further reduce per-token costs.

For builders relying on OpenAI’s API, the immediate impact is uncertain. No pricing changes have been announced. But the direction is clear: serving-stack engineering is becoming as strategically important as model architecture, and those who can optimize inference will hold a cost advantage.

Caveats

Several important unknowns limit the practical takeaway. OpenAI has not disclosed the exact technique - the four candidate methods (KV cache reuse, quantization, in-flight batching, query routing) are analyst speculation, not confirmed by the company. The confirmed results apply only to anonymous guest traffic; whether the gains transfer to free-tier, paid, or API users is completely unconfirmed. OpenAI has made no commitments about extending the optimization, and the durability of the gains at production scale remains to be seen.

Additionally, Jalapeño’s 50% cost-per-token claim comes from early testing and lacks independent benchmarks. Full production is not expected until 2027-2028. Until the software optimization is proven beyond the guest tier and the hardware claims are validated, builders should treat this as a promising signal rather than a concrete change to their cost structure.

FAQs

What caused OpenAI halving inference costs?

The Information reported that OpenAI engineers developed a software-only optimization in June 2026 that reduced GPU requirements for ChatGPT guest traffic from tens of thousands to roughly a couple hundred. The exact technique has not been publicly disclosed. The gain comes from better utilization of existing server infrastructure rather than new hardware, according to the report.

Will the software optimizations apply to paid ChatGPT tiers or only guest traffic?

OpenAI has publicly disclosed the optimization only for the guest tier (anonymous visitor traffic). No commitment has been made regarding extension to free-tier, paid, or API workloads. The guest tier produces more homogeneous and lower-complexity traffic, making it an easier test bed for the optimization. Whether the same gains apply to more complex paid or API workloads remains the central open question.

What is the Jalapeño chip and when is production-ready?

Jalapeño is OpenAI’s first custom-designed inference accelerator, announced on June 24, 2026, and manufactured by Broadcom and TSMC. Broadcom CEO Hock Tan stated early testing shows roughly 50% lower inference cost per token compared with current GPUs, though no independent benchmarks have been published. Initial prototype deployments are expected by end of 2026, with full production scale in 2027-2028.

How many GPUs were required before and after the optimization?

Before the optimization, industry estimates placed the number of Nvidia GPUs needed to serve ChatGPT guest traffic in the tens of thousands. After applying the software optimization, the number dropped to roughly a couple hundred, according to the internal source cited by The Information. This represents a reduction of more than 99% in the GPU count for that specific segment.