The Narrowness Is the Performance Argument
This site covered Perplexity's Hybrid Compute on Mac on September 2 — cloud-first, handing sensitive steps down to a local model. Lily is the engine on the local end of that chain, and it is now open source. Its design stance is unambiguous: not general purpose. Neither PyTorch nor MLX sits in the execution path. Rust handles loading and the generation loop, hand-written Metal kernels do the actual compute, and the external surface is a minimal OpenAI-compatible HTTP API doing greedy generation. There is exactly one target model — Qwen3.6-35B-A3B, a sparse hybrid using mixture-of-experts routing with fixed-size recurrent states and full attention. Much of the speed comes from treating prefill and decode as fundamentally different workloads. Prefill processes many prompt tokens at once and can reuse weights across them; decode generates one token at a time with far less reuse, so memory traffic and bandwidth dominate. The largest published gains bear that out: prefill improved most from GPU-resident expert routing (+89%) and from fusing dequantization into the grouped GEMM (+77.4%).
The Numbers and Their Test Conditions
The machine is a 40-core, 128 GB M5 Max at batch 1, across ten lengths from 256 to 128K tokens. Results: prefill averaging 4,156 tokens/s against MLX-LM's 3,388 (1.23×), decode 170.0 against 126.4 (1.35×). Broken out, prefill ran 1.12–1.42× and decode 1.31–1.37×, faster at every recorded point. Quality was checked too: a teacher-forced comparison across 192 positions put Lily's perplexity 0.04% higher, with the same top-ranked token 96.35% of the time. This is not speed bought with accuracy. One practice here is worth noting: Perplexity also published negative results — its own testing showed not every optimization helped, with some changes improving individual steps without meaningfully speeding up the full response, speculative decoding named specifically. Engineering blogs willing to include the "we tried this and it did not work" section are rare, and for anyone attempting to reproduce this, that section is worth more than the benchmark table.
Who Should Look, and Who Should Not
The requirements first: the 4-bit checkpoint is 19.4 GB, so the realistic floor is a Mac with 32 GB or more of unified memory. The shipping Hybrid Compute product lists macOS 15+, 24 GB minimum and 32 GB for best results. It is not a drop-in replacement for Ollama or LM Studio — their value is that you can load anything into them, while Lily's value is precisely that it serves one model on one hardware family. The right readers are inference-framework engineers, people studying Apple GPU optimization, and developers who want to know how much specialization is actually worth. If what you want is to run a handful of open models on a Mac, this repo will not help.
via: Perplexity engineering blog, GitHub: pplx-garden, MarkTechPost