OpenAI's first inference chip delivers: why Jalapeño boosts both speed and efficiency

Initial results clear both throughput and latency bars

The first public results can be broken down into three main takeaways:

This overview only summarizes the headline results from public benchmarks; it does not demonstrate full cost, production readiness, or large-scale deployment viability. Below, we explain how to read these numbers, why the chip achieves them, and which conclusions remain unsupported.

The tests used the public InferenceX framework, comparing systems under nominal conditions of 8,000 tokens of input and 1,000 tokens of output. GPT-OSS 120B was compared against a GB200 STP; DeepSeek R1 and Kimi K2.5, using MXFP4, were compared against a GB300 STP. The full hardware configuration of the STP systems was not disclosed. All per-watt figures are normalized to the chip package's rated power, so these results should not be extrapolated to other precision formats, software stacks, or service configurations.

The three overview charts compare peak per-watt throughput, maximum single-user decode speed, and per-watt throughput when both systems operate at the same decode speed. Green represents Jalapeño, and blue-gray represents the reference systems.

Model data summary

• GPT-OSS 120B / GB200 STP: Peak per-watt throughput 85,448 vs. 44,960; end-to-end latency 1.03 vs. 1.80 seconds; minimum TBT 0.69 vs. 1.87 ms.
• DeepSeek R1 / GB300 STP: Peak per-watt throughput 19,641 vs. 11,781; end-to-end latency 1.65 vs. 5.99 seconds; minimum TBT 1.43 vs. 5.90 ms.
• Kimi K2.5 / GB300 STP: Peak per-watt throughput 18,195 vs. 11,862; end-to-end latency 1.56 vs. 5.31 seconds; minimum TBT 1.44 vs. 5.48 ms.

These three charts cannot be combined into a single 'overall speedup.' Peak per-watt throughput speaks to system capacity; tok/s/user reflects the speed at which an individual user sees continuous output; and throughput at equal decode speed indicates how much work the system can handle while maintaining the same interactive experience.

End-to-end latency is another measurement. Under the same nominal 8,000 token input / 1,000 token output, GPT-OSS dropped from 1.80 to 1.03 seconds, DeepSeek R1 from 5.99 to 1.65 seconds, and Kimi K2.5 from 5.31 to 1.56 seconds. How would these per-request savings accumulate across a sequence of tasks?

The default 'coding task' example assumes 10 sequential steps. It simply adds the three end-to-end latency differences from above to illustrate the concept of cumulative delay in multi-step workflows. Real agents don't have uniform input/output lengths per step, can execute in parallel, and incur overhead from tools, network calls, and retries. This is therefore not an actual agent task benchmark.

A single model's results can be swayed by quantization format, kernel maturity, or test setup. The more consistent signal here is that three models with different architectures and sizes all show both higher throughput per watt and lower per-user wait times: Jalapeño shifts the entire throughput-latency curve, not just a single peak. The company also mentions 'further gains' on a frontier proprietary model, but without model details, configuration, or numbers, that claim cannot be weighed equally with the three public results.

The 53.7-fold and 104.3-fold gains apply only at specific latency points

An inference system doesn't have a single 'maximum speed' detached from user experience. Cranking up concurrency lets the system process more tokens per second overall, but often makes each user wait longer. Dropping concurrency can lower time-between-tokens (TBT), but may leave chip capacity idle. When you plot throughput against latency at different concurrency levels, the points where you can't improve one without hurting the other form the Pareto frontier.

These four metrics each answer a different question. Peak mixed TPS/kW measures the highest system-level mixed token throughput per watt of rated power during the test window. End-to-end latency tracks how long a fixed request takes from start to finish. Minimum TBT shows the fastest speed at which a single user sees the next token. Throughput at the reference system's best TBT locks both systems to the same interaction speed, then compares how much more work one can handle. Mixed TPS/kW and tok/s/user are not the same metric; the former is a system-level mixed throughput, the latter a single-user decode speed, and they should not be compared directly.

The 104.3-fold result comes from that fourth comparison. When running DeepSeek R1, the GB300's best TBT was 5.90 ms. At that same interaction speed, Jalapeño hit 12,258 mixed TPS/kW while the GB300 managed 118. Dividing those gives 104.3. The 53.7-fold result for GPT-OSS and the 56.1-fold result for Kimi are read the same way. These numbers show that Jalapeño's curve is far ahead at specific latency points; they don't mean 'the model is 104.3 times faster overall.'

Power measurements also need careful reading. Per-watt metrics are normalized to Jalapeño's 700W rated power, the GB200's 1,200W, and the GB300's 1,400W. During testing, Jalapeño's sustained power draw was no higher than 550W. This makes the comparison more complete than reporting raw throughput alone, but it's still package-level power, not the total energy draw of the full rack, networking, cooling, and data center.

Less model state movement cuts prefill, decode, and communication stalls

Large-model inference involves at least three distinct bottlenecks. Prefill processes the entire input at once and is usually compute-bound. Decode generates just a few new tokens at a time but repeatedly reads model weights and KV cache, making it typically memory-bandwidth-bound. And when a model spans multiple chips, communication creates additional waiting.

Moving data between compute units, various memory tiers, and the network consumes time and energy without generating any new tokens. Jalapeño's approach starts from the language model workload and works backward: the chip, memory, network, software, and rack are co-designed, with explicit decisions about where model state resides, keeping state like the KV cache as close as possible to the compute that will need it next.

The chain of mechanisms is straightforward: moving state less often means compute units wait less; less waiting means lower latency; handling more requests in the same time means higher throughput; and fewer memory/network transfers also mean less energy consumed. OpenAI's emphasis on a large connectivity domain serves the same goal. When a model scales across chips, the network becomes part of the inference machine itself.

So 'full-stack' isn't a vague systems slogan. It points to a specific claim: Jalapeño's advantage comes from compute, memory, interconnect, programming model, and serving software working together—not just from reduced data movement, and not just from a faster arithmetic unit. OpenAI describes it as a balanced accelerator that can reallocate resources, handling both prefill and decode, and adapting as model architectures and the ratio between the two phases change.

AI helped design the chip and helps port new models to it

There's another easily conflated result here: AI was used in both chip development and chip programming, but the three figures belong to different stages.

First, the first-generation chip went from design start to tapeout in 9 months. AI was used in the design, measurement, and verification loops, helping the team search for solutions, identify problems, and optimize the chip's arithmetic circuits to pack more compute performance into the given schedule. Tapeout only means the design data was handed to manufacturing; it doesn't mean production or qualification are complete.

Second, Jalapeño has clear and predictable programming targets. Engineers describe tasks using local tensors, explicit communication, and predictable synchronization, and then AI optimizes how those tasks are mapped, placed, scheduled, and coordinated within the system. Codex, working with GPT-Astra, brought three open-weight models that weren't originally planned to high performance within two months. This shows the software stack can keep up with model changes reasonably quickly, but each new model family still needs new kernels and model-specific optimizations—it's not 'any model auto-adapts in two months.'

Third, on some attention and MoE modules in GPT-OSS, AI-generated implementations ran at 1.5–1.8 times the speed of implementations written by human experts. The comparison covers selected modules, not the entire GPT-OSS model, and certainly not the whole-system performance of Jalapeño versus NVIDIA.

Going forward, it's worth tracking the speed of this feedback loop: real models expose bottlenecks, AI helps search for kernel placement and scheduling, and the measurement results feed into the next round of hardware and software design. Having its own chip lets OpenAI feed product workloads more directly into the next design cycle.

Jalapeño establishes a first-party compute path; production scale and full costs remain unproven

For OpenAI, Jalapeño's strategic value is first and foremost an additional first-party inference path. The company can decide on memory, networking, and serving software around its own models and agent workloads. On the supply side, this could increase procurement options and reduce dependence on any single path, but procurement volumes and negotiating leverage haven't been disclosed. OpenAI will still deploy NVIDIA and other partners' accelerators broadly, so a more accurate description right now is 'adding a dedicated supply option,' not 'replacing NVIDIA.'

Service tiers can be thought of in three levels: Jalapeño aims to get what used to be 'fast' mode efficiency at 'ultra-fast' mode, what used to be 'batched' mode efficiency at 'fast' mode, while continuing to improve 'batched' mode efficiency. This means the entire spectrum of speed-versus-efficiency options has shifted forward, not a set of independent speedups.

Related readingCodex lead Tibo, the “God of Resets”: Codex is only a transitional form—the real race is expanding from individual models to entire systemsCodex lead Tibo lays out a roadmap that matters more than model upgrades: fold Skills, memory, sub-agents, tools, and compute management back into the system, and let the same AI actively adapt to each person.→

Higher per-watt throughput also doesn't translate directly into a proportional drop in total cost per token. Full TCO includes chip and rack pricing, utilization, interconnect, memory, cooling, operations, yield, software development, and depreciation. InferenceX's public methodology makes throughput, latency, and energy easier to audit; but chip pricing, utilization rates, and production data are still missing to answer economic questions.

What can be confirmed today: there are initial measurements on three public models, and the benchmark name, some test conditions, and summary results are public, so the calculations can be checked. But the Jalapeño chip itself, the full runtime environment, and the complete test results haven't been released, so these numbers can't yet be independently reproduced by third parties. The roadmap includes starting deployment of Jalapeño into OpenAI's own compute infrastructure before the end of 2026, continuing to mature the software stack, and pushing forward on Gen2 and Gen3. What this evidence doesn't yet prove: large-scale production qualification, long-term availability, utilization under real product traffic, full TCO, and a supply volume sufficient to support a conclusion of NVIDIA replacement.

Jalapeño has already cleared the bar of 'can OpenAI build a working inference chip?' The next bar is higher and more commercially significant: can it turn the benchmark curve advantages into stable user experiences and unit economics under sustained production, real traffic, and full cost accounting?