AI INFRASTRUCTURE RESEARCH · MIXED-VENDOR GPU TRAININGGPU-TESTED COLLECTIVES · RDMA IN PROGRESS
gpubridge + straggler detector
AUG 2026 – PRESENT
Two complementary systems: an open-source PyTorch backend that joins NVIDIA and AMD GPUs in one training job, and a runtime detector and workload rebalancer that identifies what slows that job down.
- Designed hierarchical collectives that keep NCCL on NVIDIA and RCCL on AMD while bridging vendor-specific GPU groups. Implemented asynchronous collectives, GPU-stream synchronization, DDP/FSDP2 integration and Triton compression kernels across separate CUDA and ROCm software stacks. Hardware coverage spans six GPU models.
- Cut a 1 GB cross-vendor all-reduce 2.7x, from 1.67 s to 0.61 s, with fp8 compression on a V100 + MI50. In a separate LLaMA-style training benchmark, fp8 raised mixed data-parallel throughput from 4,549 to 9,523 tokens/s. These results used the TCP bridge.
- Built hardware-aware diagnosis that separates expected GPU differences from degradation, and slow computation from communication delays when they overlap. Calibrated simulations with 100-step windows detected 2% slowdowns under 5% timing jitter, with no false flags across 250,560 healthy rank-windows. Rebalancing preserves sample-weighted gradient averaging.
- Ported the stateful scoring engine to Rust and verified bit-exact numerical agreement across 3,000 seeded jobs and over 4.1 million floating-point comparisons. Differential tests, fault injection and hardware regressions check correctness across numerical, scheduling and failure boundaries.
- Current work: hardening a Rust/libibverbs RDMA prototype with NVIDIA GPUDirect and AMD GPU-memory registration, NIC-counter-based fabric diagnosis, and polling policies that trade latency against CPU use. Preparing a reproducible cluster-validation package; real GPU-direct transfers and their training benefit remain hardware-validation targets.
- faster 1 GB all-reduce across NVIDIA + AMD
- 2.7×FASTER 1 GB ALL-REDUCE ACROSS NVIDIA + AMD
- slowdown detected in calibrated simulation
- 2%SLOWDOWN DETECTED IN CALIBRATED SIMULATION
- bit-exact float comparisons, Rust vs Python
- 4.1M+BIT-EXACT FLOAT COMPARISONS, RUST VS PYTHON
- seeded jobs in the scoring validation corpus
- 3,000SEEDED JOBS IN THE SCORING VALIDATION CORPUS
PYTORCHNCCLRCCLTRITONCUDAROCMC / C++RUSTLIBIBVERBSSLURM
GITHUB ↗
