OpenAI has published the first detailed performance results for Jalapeño, the custom AI inference chip it unveiled with Broadcom in June 2026. The headline numbers are substantial: OpenAI says Jalapeño delivered 1.5 to 1.9 times more AI work per watt at peak throughput and 1.7 to 3.6 times lower end-to-end latency than the Nvidia-based systems it tested.
The interesting part is not simply that OpenAI has built a fast chip. Jalapeño is designed to improve throughput and reduce latency simultaneously, rather than increasing efficiency by batching more requests and making individual users wait longer.
There is an important limit to the announcement, however. These are inference benchmark results, not evidence that every ChatGPT, Codex or OpenAI API request will suddenly become several times faster. Jalapeño is still being prepared for production deployment within OpenAI’s infrastructure, and the company has not announced a customer-facing Jalapeño instance, separate hardware pricing, or a corresponding API price cut.
What OpenAI actually measured
According to OpenAI’s published Jalapeño benchmark results, the company tested three large public models using the InferenceX benchmark: GPT-OSS 120B, DeepSeek R1 670B and Kimi K2.5 1T.
The tests used nominal 8K input and 1K output workloads and compared Jalapeño against Nvidia GB200 or GB300 systems, depending on the model.
| Model | Comparison | Peak throughput per kW | End-to-end latency | Minimum time between tokens |
|---|---|---|---|---|
| GPT-OSS 120B | Jalapeño vs Nvidia GB200 | About 1.9x higher | 1.03s vs 1.80s | 0.69ms vs 1.87ms |
| DeepSeek R1 670B | Jalapeño vs Nvidia GB300 | About 1.7x higher | 1.65s vs 5.99s | 1.43ms vs 5.90ms |
| Kimi K2.5 1T | Jalapeño vs Nvidia GB300 | About 1.5x higher | 1.56s vs 5.31s | 1.44ms vs 5.48ms |
OpenAI also says Jalapeño achieved 2.1 to 4.1 times higher performance for highly interactive workloads. The chip has a published rating of 700 watts, while OpenAI says measured sustained consumption remained at or below 550 watts during the workloads it tested.
The bigger claim is beating the latency-throughput trade-off
Raw tokens per second can be a poor metric for evaluating an inference platform. A server can increase total throughput by batching requests, yet individual users may wait longer for their responses to start or see tokens arrive more slowly.
This creates an operating curve. At one end, the infrastructure prioritises maximising the amount of work completed by the hardware. On the other hand, it gives individual requests more resources to minimise delays.
OpenAI’s more consequential claim is that Jalapeño shifts that curve. It says the same architecture can sustain strong throughput per unit of power without paying the usual latency penalty. In benchmarking terms, OpenAI describes Jalapeño as sitting on the Pareto frontier: there was no tested alternative that could improve one of the measured dimensions without giving up ground elsewhere.
That is particularly useful for agents. A normal chatbot request might tolerate a small delay. An agent can make many dependent model calls, tool calls and follow-up decisions before completing one task. A few hundred milliseconds added repeatedly between stages can result in a noticeable delay across the entire workflow.
Jalapeño is an inference ASIC, not a replacement for every Nvidia GPU
Jalapeño is an application-specific integrated circuit built around language-model inference. It is not positioned as a general-purpose accelerator for every AI workload, and OpenAI has not presented it as its new training platform.
The design focuses on the parts of LLM serving that become bottlenecks at scale. Processing the initial prompt is relatively compute-heavy. Generating tokens after that can become constrained by memory bandwidth. Moving model state and KV cache data between components creates another source of delay.
OpenAI says Jalapeño was designed across the chip, memory, networking, software, and rack system so that more of that state can stay close to the resources using it. This is where custom silicon can deliver an advantage that is difficult to replicate by replacing a single accelerator while leaving the rest of the serving stack unchanged.
It also explains why the chip is not simply “an OpenAI model baked into hardware”. The public tests include DeepSeek and Kimi models, as well as GPT-OSS, suggesting that OpenAI wants a specialised LLM inference platform rather than silicon locked to a single model architecture.
The benchmarks are strong, but they are not a production SLA
The numbers justify taking Jalapeño seriously, but infrastructure buyers should resist the urge to turn a benchmark multiplier into a user-facing performance promise.
First, these are controlled 8K/1K inference workloads. Real applications produce messy traffic: short prompts, very long contexts, cached conversations, bursts of concurrent users, tool calls and outputs whose length cannot be known in advance.
Second, OpenAI’s headline appendix reports end-to-end latency and minimum time between tokens, but does not provide p95 or p99 latency, which operators normally inspect for user-facing services. A system can have an excellent average while a smaller group of requests waits much longer during congestion.
Third, the power comparison is normalised using published accelerator power ratings. That makes different chips easier to compare, but it is not the same as measuring complete rack-level power consumption under identical production traffic. Networking, host CPUs, memory, cooling and under-utilised capacity still affect the real cost of serving a request.
OpenAI also has to prove that the system behaves the same way once it moves from benchmark runs to a large production fleet. Software maturity, scheduler behaviour, failure rates, hardware availability and fallback capacity become much more important at that stage.
Faster silicon does not automatically mean cheaper OpenAI API prices
Efficiency gives OpenAI more options. If a rack can complete more useful inference work for the same power budget, OpenAI can serve more demand, reduce infrastructure cost per completed task, improve margins or use the spare capacity for more compute-intensive models.
There is no guarantee that those savings are passed directly to API customers. Model pricing is influenced by far more than accelerator electricity consumption, including data-centre capacity, memory, networking, redundancy, model development and the commercial value of the service.
This is why developers comparing providers should keep model economics separate from chip economics. DIY AI’s analysis of the cheapest AI models for inference focuses on what an application actually pays for and receives, rather than assuming that the most efficient underlying hardware yields the cheapest API.
Most developers cannot choose Jalapeño anyway
For infrastructure teams, this is an easy point to miss. Jalapeño is currently an OpenAI infrastructure story rather than a new accelerator you can rent alongside an H100, B200 or GB300.
If you use the OpenAI API, OpenAI chooses the serving infrastructure behind the endpoint. If you self-host an open model, Jalapeño is not currently among the standard hardware options available from GPU clouds.
That means a developer should not redesign a deployment simply because Jalapeño won a benchmark. The practical comparison is still between hosted API, serverless inference, and rented or owned accelerators. Our AI hosting cost ranges by infrastructure route show why utilisation, idle capacity and operational overhead can outweigh the accelerator’s theoretical efficiency.
What developers should measure if Jalapeño reaches their workloads
If OpenAI begins routing more customer workloads through Jalapeño, the useful evidence will come from application-level measurements rather than the chip specification.
| Workload | Metric worth watching | Why |
|---|---|---|
| Interactive chat | p50, p95 and p99 time to first token | Shows how quickly users actually see a response begin |
| Streaming responses | Inter-token latency | Determines whether generation feels fluid after it starts |
| Agents | Total task completion time | Captures latency compounded across many dependent calls |
| High-volume APIs | Throughput at a fixed latency target | Avoids hiding slower user experience behind larger batches |
| Cost-sensitive applications | Cost per accepted result | Includes retries, failures and unnecessary output rather than token price alone |
| Production services | Error rate, regional capacity and fallback behaviour | Fast hardware is of limited value if capacity becomes unreliable under load |
Jalapeño could matter most by giving OpenAI control over the complete inference stack
A recurring concern around custom AI chips is whether replacing an external accelerator actually saves enough money to justify designing silicon. That misses some of the value.
Owning the processor gives OpenAI another component it can optimise alongside model architecture, kernels, memory placement, networking and scheduling. Improvements in one layer can be designed around assumptions made in another. At OpenAI’s scale, small efficiency gains can also become meaningful when repeated across an enormous number of inference requests.
It does not mean OpenAI is abandoning Nvidia. OpenAI says it will continue to deploy Nvidia accelerators and hardware from other partners for both training and inference while developing the second and third generations of Jalapeño.
The first Jalapeño systems are due to begin deployment inside OpenAI’s compute infrastructure by the end of 2026. Production qualification, software maturity and performance across additional models are still being worked through.
DIY AI view: the production rollout is now more important than another benchmark
Jalapeño has moved beyond the vague “custom chip” announcement stage. OpenAI now has functional first-party inference silicon, with public benchmark results showing a credible advantage over GB200 and GB300 systems across several large models.
The strongest result is not the single 3.6x latency figure. The claim is that Jalapeño can preserve low latency while increasing throughput per unit of power. If that survives production traffic, it could make long-running agents and other sequential AI applications noticeably more responsive without forcing OpenAI to sacrifice serving capacity.
The next evidence should come from the fleet rather than the lab: tail latency, reliability, utilisation, deployment volume and any measurable change in OpenAI product or API performance. Until then, Jalapeño is a significant infrastructure result, but not yet a reason for developers to rewrite their own hardware strategy.
Reporting basis
Sources
Primary source: openai.com Media
