A training cluster only moves as fast as its slowest GPU.
Why it is hard to catch
Nothing crashes. The job runs, the loss goes down, the dashboards look healthy. A straggler does not announce itself as a failure, it just shows up as a training run that is quietly costing a third more than it should, for weeks at a time.
The usual way you find out is after the fact, reading timings once the run is already paid for, and even then a job spans so many nodes that working out which one was slow is its own investigation. Detecting it while the job is live is what makes it actionable: you can drain that machine and reschedule mid-run instead of eating the tax to the end.
Watching the collectives
Distributed training synchronizes at collective operations, like the all-reduce that averages gradients every step. When one rank is slow, every other rank reaches the collective early and waits for it.
The tool monitors that collective communication through NCCL on NVIDIA and RCCL on AMD, so the rank everyone is waiting on stands out at runtime. For stock jobs, a profiler plugin written in C records device-side timing for every collective into a shared-memory ring, so the rank that arrives last is found even when only its forward or backward pass is slow.
The mixed-vendor problem
These clusters are not uniform, so the tool works across both vendors instead of locking into one. NVIDIA and AMD expose performance counters differently and have different baseline timing behavior, so a naive cross-vendor comparison reads an ordinary hardware difference as a fault and flags healthy machines.
The signal has to be normalized per vendor before a real straggler is separable from routine variance. Each rank is scored against its peers on the same vendor and against its own baseline, so a vendor that is slower by design reads as load imbalance rather than as bad GPUs, and hardware classes keep mixed generations (V100, A100, H200, MI50) from misflagging. Getting that boundary right is most of the work.
Fast enough to run alongside training
Cross-rank summary exchange runs on a background thread over the job's own store, so training never waits for a missing rank's summary. In a CPU simulation with 10 ms steps, the direct detector cost was about 0.2% of step time; this is not a measured overhead guarantee for GPU training.
The complete stateful scoring update runs in Rust, with one native boundary call per window. The validation corpus matched the Python reference bit for bit across 3,000 seeded jobs and over 4.1 million floating-point comparisons.
From finding to fix
Every finding carries its cost in milliseconds per step and, through NVML and AMD SMI telemetry, a likely cause such as thermal throttling or a clock drop. Hangs are reported with a flight recorder that names the stuck rank and every rank's collective phase.
Imbalance turns into per-vendor batch sizes with exact gradient averaging: with AMD 20% slower, simulated step time falls 5.5%. One command attaches the detector to an unchanged training script, with a Prometheus exporter and a Grafana dashboard.
Where it stands
In calibrated simulations with 100-step windows, it detected 2% slowdowns under 5% timing jitter, with no false flags across 250,560 healthy rank-windows. The next validation stage exercises the integrated Rust RDMA transport and NIC diagnosis on allocated hardware; mock and CPU results are kept separate from real GPU measurements.
Where it comes from
It started with following the work Meta and AMD have been doing on distributed training reliability. Hyperscalers are tackling this problem internally, and it matters most at supercluster scale: the more GPUs a job spans, the more places a straggler can hide, and the more every slow step costs.
