What we are. What we're building.

Performance engineering,
from signal to recovery.

Artemis is an autonomous performance engineer for distributed AI training. We're building a system that connects continuous detection, evidence-backed diagnosis, performance-regression testing, and permitted correction in one closed loop.

The goal: more useful training from the GPUs you already have, with less manual investigation. Built for teams running owned or rented clusters, across supported GPU vendors and generations.

An early product, with a clear destination.

The full autonomous workflow below is our product direction, not a claim that every capability is production-ready. Existing work includes GPUBridge and detector prototypes. Hardware results later on this page are historical, scoped experiments—not customer deployments or guarantees for the current build.

The product vision

An engineer in the loop.
Not an engineer on every alert.

01
Observe and detect
Build workload-aware baselines and find persistent slowdowns without confusing different hardware with faulty hardware.
02
Confirm the regression
Repeat suspicious measurements and compare matched workloads, hardware, topology, and software configurations.
03
Diagnose the cause
Connect evidence across the input pipeline, host, GPU, collectives, and network. Test an explanation before treating it as a cause.
04
Take permitted action
Recommend a targeted fix or apply a supported correction within operator policy. Keep workload adaptation mathematically correct.
05
Verify recovery
Measure whether useful throughput improved. Keep the evidence, support reversible rollback, and escalate what needs an engineer.

Cross-stack diagnosis and scheduled regression testing are planned work. A learned diagnosis model needs permitted incident data and held-out evaluation. The dedicated agent telemetry/control plane is separate from GPUBridge's training-data transport; RDMA support in one does not establish completion of the other.

Job 0x2a · 8 ranks · data parallel

r0NV
r1NV
r2NV
r3NV
r4AMD
r5AMD
r6AMD
r7AMD

Finding · window 37

kind: rank
state: flagged
rank: 2
island: nvidia
ratio: 1.19
excess_ms: +8.1 per step
loss_pct: 7.6
cause: clocks

Finding · window 37

kind: rank
state: flagged
rank: 2
island: nvidia
ratio: 1.19
excess_ms: +8.1 per step
loss_pct: 7.6
cause: clocks

Compute
Waiting at the barrier
All-reduce
The regressed rank

Illustration. Eight ranks in synchronous data-parallel training, four NVIDIA and four AMD. Rank 2 has regressed, so the other seven reach every all-reduce early and wait for it. Press Rebalance to apply the plan.

Explore the detector architecture

01The problem

Nothing crashes. Every step just gets slower.

Synchronous training moves in lockstep. Each step ends in a collective, such as the all-reduce that averages gradients, and a collective finishes only when the last rank arrives. One slow rank, often called a straggler, sets the pace for every GPU in the job while the rest sit idle at the barrier.

When that slowdown starts partway through a run, it's a performance regression. Throughput drops a few percent, the loss still goes down, utilization still looks busy, and nothing points at the GPU, link or host that changed.

Where regressions come from

Power, clocks, configuration
A power cap, clocks that stop boosting, a driver or setting that changed.
Thermal
A GPU that throttles as it heats up.
Communication
A degraded port, link or switch path, or one island's collectives slowing down.
Host and input
A data loader, CPU or storage stall that holds one rank back off the GPU.

Why step time alone can't find it

Every rank sees the same, slower step. What separates the culprit is its compute time against its peers.

Rank 2, regressed
The other 7 ranks
Suspect window
Flagged

The numbers as a table

Step time and compute ratio per window, synthetic data
Window	Step time	Rank 2	Peer median	State
0	100.1 ms	1.00×	1.00×	healthy
1	99.7 ms	1.01×	1.00×	healthy
2	100.4 ms	1.00×	1.00×	healthy
3	99.6 ms	1.00×	1.00×	healthy
4	99.9 ms	1.00×	1.01×	healthy
5	99.5 ms	1.01×	1.00×	healthy
6	99.5 ms	1.00×	1.00×	healthy
7	99.5 ms	1.01×	1.00×	healthy
8	99.9 ms	0.99×	1.00×	healthy
9	99.9 ms	1.00×	1.00×	healthy
10	100.5 ms	1.01×	1.00×	healthy
11	100.6 ms	1.00×	1.00×	healthy
12	100.3 ms	0.99×	1.00×	healthy
13	99.8 ms	0.99×	1.00×	healthy
14	99.9 ms	1.00×	1.00×	healthy
15	100.4 ms	1.01×	1.00×	healthy
16	110.1 ms	1.15×	1.01×	suspect
17	110.3 ms	1.14×	1.00×	suspect
18	110.5 ms	1.14×	1.00×	flagged
19	111.0 ms	1.14×	0.99×	flagged
20	110.1 ms	1.13×	1.01×	flagged
21	110.7 ms	1.13×	1.00×	flagged
22	110.0 ms	1.14×	1.00×	flagged
23	110.3 ms	1.15×	1.00×	flagged
24	110.5 ms	1.15×	1.00×	flagged
25	111.1 ms	1.13×	0.99×	flagged
26	110.2 ms	1.14×	1.00×	flagged
27	110.6 ms	1.14×	1.00×	flagged
28	110.1 ms	1.14×	0.99×	flagged
29	110.5 ms	1.13×	1.00×	flagged
30	110.1 ms	1.14×	1.00×	flagged
31	111.2 ms	1.13×	1.01×	flagged

Illustration with synthetic numbers. Windows are 50 steps. A rank is suspect when its compute runs about 10% over the median of its island's other ranks, and flagged after 3 suspect windows out of the last 4.

What it costs

s ÷ (1 + s)

If one rank's compute runs s slower, every step stretches by up to s, and up to s ÷ (1 + s) of every GPU-hour in the job goes to waiting: the bound when compute sets the step. One rank 8% slow in a 512-GPU job can idle the equivalent of 38 GPUs.

One slow rank, the whole job, at most
Rank slower by	GPU-hours waiting
2%	2.0%
5%	4.8%
10%	9.1%
20%	16.7%

02How it came together

One question led to three layers of infrastructure.

Which GPU is slowing this job down? Answering that across NVIDIA and AMD meant building the layer underneath first.

01The question
The detector idea
Find the GPU that's slowing a distributed job while the job is still running, on NVIDIA and AMD alike, instead of reading timings after the run is paid for.
02The prerequisite
One job, two vendors
Detecting across a mixed fleet needs a job that spans it, and NCCL and RCCL can't exchange data. PyTorch's "nccl" backend runs NCCL on CUDA builds and RCCL on ROCm builds, never both in one job.
03Open source · MIT
GPUBridge, in Python
A PyTorch backend that keeps each vendor on its native library and joins them over a Gloo bridge. It went on to train DDP, FSDP2 and pipeline-parallel jobs across an NVIDIA V100 and an AMD MI50.
Read more
04Rust · libibverbs
From Gloo to Rust and RDMA
Measurements put the bottleneck on the bridge: about 1.2 GB/s of TCP through host memory. Research into RDMA moved the transports to Rust. A multi-stream TCP prototype missed its own speed target and was dropped; a raw-verbs transport over libibverbs followed and has run on InfiniBand hardware, between two NVIDIA islands on one host.
Read more
05The detector
Artemis
With mixed-vendor jobs running, the detector followed. It flags regressions, estimates what each one costs, suggests a likely cause and, on opt-in, rebalances work at runtime. Its agent, coordinator and scoring core are Rust, with their own RDMA channel.
Read more
06In development
Learned diagnosis
Research into a model, trained on GPU incident and recovery data, that suggests likely causes, the checks that confirm them and the fixes.
Read more

03GPUBridge · open source · MIT

NVIDIA and AMD GPUs in one PyTorch job.

A PyTorch install is a CUDA build or a ROCm build, so each process speaks NCCL or RCCL, and an NCCL communicator can't exchange data with an RCCL one. GPUBridge is a torch.distributed backend that joins them.

Each vendor's GPUs form an island on their native library. For an all-reduce, each island reduces onto its leader, the leaders exchange over the bridge, and each leader broadcasts the result back. Everything inside an island stays on NCCL or RCCL; only the leaders' exchange crosses vendors.

It composes collectives PyTorch already provides, so it runs on stock CUDA and ROCm builds over any network, and moving a script over is one import and one string.

train.pyPython

import gpubridge  # registers the backendimport torch.distributed as distdist.init_process_group(backend="gpubridge")  # instead of "nccl"model = DistributedDataParallel(model)

Read the code Open an issue

DDP, FSDP2 and HSDP through torch.distributed
Async collectives and a DDP communication hook
Pipeline parallel across vendors, with automatic stage balancing
Optional bf16 and fp8 compression on the bridge, in Triton kernels
Swappable bridge transports: Gloo by default, Rust verbs opt-in
A simulation mode that runs two vendors on one CPU

2.75×faster 1 GB all-reduce between a V100 and an MI50 with lossy fp8 on the bridge, 1.67 s to 0.61 s
1.10×a V100's training throughput, with an MI50 added as a second pipeline stage (best hand-tuned split; automatic gave 1.07×)
6GPU models it has run on: V100, T4, L40S, A100, H200 and MI50

On one V100 and one MI50 over TCP, the pipeline beat the V100 alone and data parallelism didn't. Each data-parallel step all-reduces 536 MB of gradients over a link of about 1.2 GB/s; the pipeline passes 3 MB of activations per microbatch. Mixing vendors pays when the parallelism fits the link.

Collectives, step by step

dist.all_reduce(grad)

Every rank ends with the sum of all eight.

Training payload

Each number stands for one rank's gradient tensor.

With one vendor there's no bridge, and every call goes straight to NCCL or RCCL.

04Transport

Two channels, kept apart.

GPUBridge and Artemis both move data between processes, for different reasons. They run separate code over separate connections, and a result on one says nothing about the other.

Training payload
GPUBridge
Island leader to Island leader.
Collective payloads between vendor islands: gradients, parameters, activations.
Default
Gloo over TCP, staged through host memory
Opt-in
Rust over libibverbs: reliable-connected sends and RDMA writes into registered buffers
The verbs path has run on one host, between two NVIDIA islands, through one InfiniBand HCA, with GPU tensors staged through host memory.
Telemetry and control
Artemis
Node agent to Coordinator.
Window summaries up to the coordinator; control messages and acknowledgements back down. Never model tensors.
Transport
Rust over libibverbs, with its own versioned protocol
Has run end to end during real 4-rank DDP training on one host, over InfiniBand.

Next for both channels: across machines and on AMD. For training data, GPU memory straight to the NIC with GPUDirect RDMA.

The terms, precisely

NCCL and RCCL: NVIDIA's and AMD's collective communication libraries. Within one vendor they choose the fastest path themselves: NVLink or xGMI inside a node, InfiniBand or RoCE between nodes.
Gloo: PyTorch's CPU collective backend, over TCP. It's GPUBridge's default bridge because it runs on any network.
InfiniBand: The fabric: host channel adapters (HCAs) and switches built for low-latency, high-bandwidth traffic between machines.
RDMA and verbs: RDMA lets a network adapter read and write registered memory directly, with no copy through the remote CPU. libibverbs, from rdma-core, is the user-space API both Rust transports call.

05Artemis · the detector

From collective timings to a finding you can act on.

Every rank already checks in at the same place several times a step: the collectives. Artemis times them on each rank, scores each window of steps against the rank's peers and its own history, and reports what slowed down, by how much and why. Collection never blocks a rank, and scoring happens away from the training step.

Source private for now

Architecture · native pathDetector channel

Each training node

Training ranks

PyTorch DDP or FSDP on NCCL or RCCL. The script is unchanged.

Profiler plugin · C

Loaded through NCCL's profiler plugin API, which RCCL shares. Device-side timing of every collective.

Shared-memory rings

One per rank, written seqlock-style, so the reader never holds a rank up.

Node agent · Rust

Reads the rings and sends one summary per window.

Rust · libibverbs

Coordinator · Rust

Scoring core

Island peers and warmup baselines, window by window.

Findings

Slow rank, island, bridge link or vendor collective, and hangs.

Rebalance policy

Opt-in. Turns a finding into a bounded plan.

Out

Open one to see a sample.

JSONL findings

Every report, one JSON object per line, appended while the job runs. New flags also go to the straggler logger at WARNING.

straggler run ... --jsonl findings.jsonl

One line of findings.jsonl, wrapped for reading: rank 2 flagged at window 37. Real field names. The values are illustrative, from a simulated job.

{"kind": "rank","state": "flagged","window": 37,"steps": [1850, 1900],"island": "nvidia","rank": 2,"ranks": [2],"hosts": ["node-a"],"since": 35,"ratio": 1.19,"z": 14.2,"history": 1.18,"drift": null,"excess_ms": 8.1,"loss_pct": 7.6,"confidence": "peers","signal": "gap","basis": "peers","missing": [],"evidence": null,"cause": {"kind": "clocks","text": "clocks low (SM 1110 MHz vs 1530 MHz on its peers), no throttle reason: locked clocks or a misconfigured GPU","evidence": {"sm_mhz": 1110,"peers_sm_mhz": 1530,"temp_c": 61,"peers_temp_c": 63,"power_frac": 0.48,"throttle": {},"pcie_degraded": 0.0}}}

Prometheus and Grafana

Rank 0 serves straggler_* metrics off the training thread, and a Grafana dashboard for them comes with the detector.

Detector(exporter=straggler.PrometheusExporter(port=9464))

A mock of the dashboard's panels, drawn from simulated values. Not a Grafana screenshot. Real field names. The values are illustrative, from a simulated job.

Last window

Ranks missing

Detector active

Rank pace over its island

Rank 2
The other 7

Flagged now

kind	subject	value
rank	2	1

Median step time

98.7 ms 106.2 ms

What Prometheus scrapes, at window 52

# TYPE straggler_rank_pace_ratio gaugestraggler_rank_pace_ratio{rank="0",island="nvidia"} 1.0straggler_rank_pace_ratio{rank="2",island="nvidia"} 1.19...# TYPE straggler_flagged gaugestraggler_flagged{kind="rank",subject="2"} 1.0# TYPE straggler_step_ms gaugestraggler_step_ms 106.6# TYPE straggler_findings_total counterstraggler_findings_total{kind="rank",state="flagged"} 1.0straggler_findings_total{kind="rank",state="ongoing"} 15.0straggler_missing_ranks 0.0straggler_detector_active 1.0

No native agent? The same scoring runs in-process in Python, fed by GPUBridge's collective observer.

01
Observe
Each rank times its own compute gap between collectives and its wait inside them. The late rank has the longest gap and the shortest wait. Each rank measures only its own durations, so no clocks need syncing across nodes.
02
Score
Each window, every rank is compared with the other ranks in its island: about 10% over their median makes it suspect, and 3 suspect windows out of the last 4 flag it. Each island's warmup baseline catches a whole island slowing together. Opt-in hardware classes, tested in simulation so far, keep a slower GPU generation from reading as a fault.
03
Explain
Each finding carries its cost in milliseconds and percent of the step, and a likely cause from GPU telemetry: thermal, power, clocks, PCIe or host, tested against simulated telemetry so far. Missing telemetry is reported as missing. A hang names the ranks behind and dumps each rank's last 256 collectives.
04Opt-in
Act
Rebalancing moves batch work between the ranks the job already has, in proportion to measured speed and within set bounds. The global batch stays the same and each gradient is weighted by its samples, so the update is the mean over every sample, as before.

Every rebalancing plan

Open a step to see what it looks like.

01 · Finding

A plan starts from a finding of kind rank or degraded on the same rank in 3 windows in a row, and that rank must be the slowest per sample over the last 4.

findings.jsonl, three windows

{"kind": "rank", "state": "flagged", "window": 37, "rank": 2, "ratio": 1.19, "excess_ms": 8.1, ...}{"kind": "rank", "state": "ongoing", "window": 38, "rank": 2, "ratio": 1.20, "excess_ms": 8.4, ...}{"kind": "rank", "state": "ongoing", "window": 39, "rank": 2, "ratio": 1.19, "excess_ms": 8.2, ...}