Don't Fill NVL72 Racks: At a 2% Node Unavailability Rate, SkyPilot Calculates That Using 16 Nodes per Rack With 2 Spares Gives 27% More Usable Compute Than Filling All 18

Why jobs need to stay inside one rack

The two most communication-heavy kinds of parallelism in large model training, tensor parallelism's all-reduce and expert parallelism's all-to-all, sit on the critical path and can't be fully overlapped with computation. On standard 8-GPU servers, that traffic has to go over InfiniBand as soon as it crosses nodes; Hugging Face measured a 43% drop in training throughput when tensor parallelism moved from NVLink to InfiniBand. NVIDIA's pitch for NVL72 is putting all 72 GPUs in a rack on one NVLink fabric: 18 NVLink ports per GPU for 1.8 TB/s of bidirectional bandwidth, with 18 NVSwitch chips forming a non-blocking backplane. But in an October 29 post, GPU scheduling platform SkyPilot points out that the benefit only applies to jobs that stay within one rack. Anything spanning racks is still bottlenecked by InfiniBand, and as failed nodes go offline and jobs finish at different times, free capacity scatters across racks, making it harder and harder to keep a rack whole.

Where the 27% comes from

Once the rack becomes the unit of compute, failures hit harder too. Using Meta's figure of roughly 490 days mean time between interruptions per 4-GPU node, a rack of 18 such nodes is interrupted about every 27 days; on a 128-rack cluster where jobs are allocated whole racks, roughly every 5 hours some rack becomes unusable for full-rack jobs because of a single node failure.

SkyPilot says it has measured about 2% node unavailability on production NVL72 clusters. Applying NVIDIA's model of a 128-rack (2,304-node) cluster at that rate, 16-node slices with two spare nodes per rack give 2,037 usable nodes against 1,602 for full 18-node slices, 27% more. They also beat 9-node slices by 4%, because with 9-node slices every failure costs half a rack, while 16-node slices fully absorb the first two failures in each rack. Sixteen nodes means 64 GPUs, which is also the power of two that parallelism groups are usually sized to.

Measurements and tooling

NVIDIA's GR00T team compared two scheduling policies while pre-training GR00T N1.7, a foundation model for humanoid robots, on 512 GB300 GPUs across 128 nodes: nodes spread across arbitrary racks, versus rack-aware slices placed on exactly 8 racks. The latter raised throughput 14.4%, cutting each iteration from 1.59 s to 1.39 s. SkyPilot extrapolates that a 10,000-GPU cluster would save about $400,000 per training day.

The second half of the post covers three of SkyPilot's own features: topology-aware scheduling that places each slice on a single rack; hot-swapping a failed node with a spare in the same rack while the other nodes keep running; and automatic defragmentation that, when a job is blocked by topology rather than capacity, preempts jobs that can resume from checkpoints to free a whole rack, leaving interactive development environments alone by default. The post is itself product marketing, it says little about how the scheduler is implemented, and the GR00T numbers come from NVIDIA's own comparison. Still, the finding that keeping spare nodes yields more usable compute, and the method behind the numbers, are directly useful to teams planning NVL72 clusters.