Nvidia Built a Whole Rack for the Decode Phase: Groq 3 LPX Enters Full Production, Trading HBM Bandwidth for On-Chip SRAM

During Hot Chips on August 24, Nvidia announced that its Groq 3 LPX inference accelerator has entered full production. It is the first product out of the roughly $20 billion deal that took Groq's technology and team last year, and Nvidia's first rack-scale system built around non-GPU silicon: 256 LP30 chips per rack, 512MB of on-chip SRAM each for 128GB in total, using on-chip memory to route around the HBM bandwidth wall and specialize in token generation at long context. It does not replace GPUs — Vera Rubin NVL72 still handles long-context prefill while LPX takes decode, with Dynamo scheduling across the two. Nvidia's headline figure is 3,431 output tokens per second for Gemma 4 31B at 100K context. Neocloud provider Nebius is the first customer, with racks online later this year.

Why Split It: The Two Phases Choke on Different Things

In a single inference pass, prefill reads the entire context and computes KV — compute-bound work, which is what GPUs are for. Decode emits one token at a time, hauling weights and KV out of memory on every step, and stalls on memory bandwidth. Putting both on the same silicon means one of them is always waiting on the other. LPX takes the other route: no HBM, on-chip SRAM instead. A single LP30 carries 512MB, a full rack of 256 adds up to 128GB, and the chips interconnect over 96 chip-to-chip links at 112 Gbps each, fully liquid-cooled on MGX infrastructure. Nvidia's technical blog describes three ways it pairs with Vera Rubin NVL72: standard prefill/decode disaggregation, splitting attention and FFN layers across the two systems, and external-drafter speculative decoding with dual KV caches. Nvidia senior director Dion Harris framed it as "not about replacing GPUs" but about running each part of the workload on the right processor.

Read the Numbers as Vendor Numbers

The cited results: on Artificial Analysis, Gemma 4 31B produces 3,431 output tokens per second at 100K context and 3,382 at 10K; on SPEED-Bench coding tasks, a median of 4,767 tokens per second and a P80 of 5,520. Nvidia also claims an LPX rack paired with Vera Rubin NVL72 delivers 35 times the throughput per megawatt of Blackwell NVL72 alone on trillion-parameter models. All of it is benchmark data the vendor selected and cited. Change the model, the batch size or the context distribution and the advantage moves with it; comparable numbers will have to come from third parties running the same workload. Jensen Huang's line in the release is that the company is "advancing the performance frontier with LPX for ultra-fast token generation." Nebius CTO Danila Shtan was more specific: generation is the phase that determines responsiveness, and that is the phase LPX accelerates — Nebius plans to run it inside its Token Factory inference platform.

What It Means for Developers

You will not see one of these in your own rack any time soon. Systems like this land at inference clouds such as Nebius first, and what an ordinary team notices is that inter-token latency at long context suddenly drops on somebody's platform. The trend is the part worth keeping: inference is being pulled apart into phases that get optimized separately, and the unit of procurement shifts from "how many cards" to "which stage runs on which silicon." That changes the question you ask a vendor. Not "do you have the newest GPUs," but "what path does a long-context, latency-sensitive workload actually take on your infrastructure."

via: NVIDIA's technical blog, "How NVIDIA Groq 3 LPX Unlocks Ultrafast Interactivity at Long Context", NVIDIA's press release, SiliconANGLE