GB10 cluster stuck at ~10 Gb/s through a MikroTik CRS804? It was the switch — in three places

How a 4× DGX Spark (GB10) cluster went from a hard ~10 Gb/s ceiling and a ~5 Gb/s single-flow RoCE collapse (NCCL all-reduce: 0.46 GB/s) to 109 Gb/s per rail and 23–26 GB/s NCCL busbw — by fixing three switch/host config bugs. None of them were the NIC, the cable, the firmware, or “GB10 GDR.”

This was debugged on MSI EdgeXpert units, but it applies to the whole GB10 family — NVIDIA DGX Spark, ASUS Ascent GX10, Gigabyte AI TOP ATOM, Dell Pro Max, and the rest — because the ConnectX-7 multi-host networking is identical across them. If your search history says “DGX Spark 10 Gbps ceiling”, “GB10 RoCE slow through switch”, or “CRS804 RDMA collapse”, you’re in the right place.

TL;DR

Four GB10 Sparks, each with a ConnectX-7 (2 cables → 4 ~100G “rails” per node), wired to a MikroTik CRS804-4DDQ via 400G→2×200G breakout cables. Every transport pinned around 10 Gb/s; single-flow RoCE collapsed to ~5 Gb/s; NCCL all-reduce sat at 0.46 GB/s — while mlxlink was clean on both ends, links negotiated 200G/RS-FEC, and the switch showed zero drops. Direct-connecting two Sparks gave full bandwidth, which pinned it on the switch path. Three independent bugs, all required:

The RoCE bridge wasn’t hardware-offloaded. Unused QSFP-DD breakout sub-ports were sitting in the management bridge, splitting each physical cage’s lanes across two bridges — the Marvell ASIC refuses to offload a bridge in that state. Traffic fell to the switch CPU → a hard ~10 Gb/s cap, QP-count-independent.
Switch QoS/PFC stalls RoCE on the offloaded path. With trust-l3=keep / pfc=… the connection sets up but data never flows. Fully off (lossy RoCE) is the correct end state on this platform.
Duplicate fabric IPs → ARP flux. Multiple virtual interfaces shared one IP; RDMA packets left the NIC and vanished before the switch. Fixed with 16 unique per-rail IPs, one subnet per rail.

Result: 109 Gb/s per rail (uniform, every node-pair, every rail), ~206 Gb/s full-duplex aggregate per rail, 23.33 GB/s 4-node NCCL busbw (26.58 for 2-node), switch CPU 0%, host CPUs ~94% idle.

Quick fix: a Claude Code prompt

If you just need your cluster running and you drive it with Claude Code (or any capable agent), paste this. It encodes the whole diagnosis in the order that works. Full explanations follow in the rest of the post.

MISSION: Diagnose/fix GB10 (DGX Spark family) RoCE collapse through a MikroTik
CRS804/CRS812-class switch. Known failure: ~10 Gb/s ceiling all transports /
single-flow RDMA ~5 Gb/s / NCCL busbw <1 GB/s despite clean 200G links.

SAFETY RAILS:
- /export file=pre-fix on the switch BEFORE any change; one change at a time.
- Confirm mgmt rides the copper ports (ether1/2), NOT the DDQ ports, before
  touching bridges.
- GROUND TRUTH = NCCL all_reduce/sendrecv. perftest/ib_write_bw has a documented
  GB10 false-negative mode (linux-rdma/perftest#394) — corroborate, never trust
  it alone.
- DRAIN RULE: any link transition (cable move, port bounce, switch reboot) can
  re-arm a CX7 low-throughput state that ONLY a full AC drain clears (power off,
  unplug 60s). A reboot is insufficient. Drain before judging any result.
- IGNORE the "27W insufficient power" dmesg on GB10 — confirmed cosmetic, fires
  on healthy systems.

CHECK 1 — SWITCH BRIDGE HARDWARE OFFLOAD (the ~10G cap):
Detect: during a transfer, switch CPU shows real load (/system/resource/cpu);
boot log lacks "hardware offloading activated on bridge <your-roce-bridge>";
bridge ports missing H flag; host MACs learned "D" (software) not "E" (hardware)
in /interface/bridge/host.
Cause: breakout sub-ports of one physical cage split across two bridges (unused
-2..-8 legs auto-membered into the default bridge).
Fix: /interface/bridge/port/remove [find bridge=bridge interface~"qsfp56-dd"]
Verify: log shows "hardware offloading activated on bridge <roce-bridge>",
ports show H, switch CPU ~0% under load.

CHECK 2 — SWITCH QoS/PFC (connection works, data hangs):
Detect: RDMA QP/GID exchange succeeds, sender tx_packets_phy climbs,
local_ack_timeout_err climbs, receiver sees nothing. Ping passes.
Fix (switch): /interface/ethernet/switch/qos/port/set [find name~"qsfp56-dd"]
trust-l3=ignore pfc=disabled
Fix (hosts): mlnx_qos -i <each rail netdev> --pfc 0,0,0,0,0,0,0,0
Lossy RoCE is the accepted end state. Do not re-enable PFC without re-proving
line rate afterward.

CHECK 3 — FABRIC ADDRESSING (packets transmitted but vanish):
Detect: ip -br addr across all nodes — ANY duplicate IP across the 4 rail
netdevs per host is fatal (ARP flux).
Fix: one subnet per rail, 4 unique IPs per host, 16 total on 4 nodes:
rail N netdev -> 10.9.N.<host-octet>/24, MTU 9000, IPv6 off on fabric ifaces,
sysctls arp_ignore=1 arp_announce=2 rp_filter=2. Map devices with ibdev2netdev
(names differ across GB10 vendors). Persist via netplan, not ip commands —
NetworkManager strips manual IPs.

VALIDATE (in order, expected numbers):
1. ib_write_bw single-QP, one rail: ~109 Gb/s (per ~100G rail).
2. Every node-pair, rail 0 + spot-check others: uniform ~109.
3. 2-node NCCL sendrecv: ~20-26 GB/s busbw. GDR shows "Disabled" — NORMAL on
   GB10 unified memory, not a bottleneck, do not chase it.
4. Full-cluster NCCL all_reduce, all rails (NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,
   mlx5_3, NCCL_IB_ROCE_VERSION_NUM=2, NCCL_SOCKET_IFNAME=<mgmt>, no
   NCCL_IB_TC): >= ~23 GB/s busbw on 4 nodes.
5. Switch CPU 0% and host CPUs idle during all of the above.
Then /export file=post-fix-golden and persist everything (netplan + systemd
oneshot for mlnx_qos/C-states + sysctl.d).

The setup

4× GB10 Sparks (MSI EdgeXpert; 128 GB unified LPDDR5X each), ConnectX-7 200GbE in multi-host mode — each 200G port presents as two ~100G logical NICs on two PCIe Gen5 x4 domains, so 2 cables per node = 4 rails per node.
MikroTik CRS804-4DDQ (Marvell 98DX7335), RouterOS 7.23.2, QSFP-DD 400G→2×QSFP56 200G breakout cables.
Goal: multi-node tensor parallelism for a 397B MoE model. TP moves an all-reduce across the fabric every layer — it lives or dies on this bandwidth.

The symptom, and why it was maddening

Everything looked healthy:

mlxlink clean on both ends: 200G, 4x width, RS-FEC (544,514), “No issue was observed.”
Zero drops on the switch, zero drops on the NIC, sub-millisecond latency.
ib_write_bw connected and ran — it just topped out around 9.5 Gb/s no matter how many QPs.
NCCL all-reduce: 0.46 GB/s busbw (≈3.7 Gb/s) — the “~5 Gb/s collapse.”

And nothing moved the number: RouterOS version, NIC firmware (28.45 vs 28.47), adaptive routing on/off, deep C-states, multi-QP, even a full lossless-PFC recipe. Every transport — RoCE, TCP over mgmt, TCP over the fabric — converged around 0.5–1 GB/s.

That invariance is the tell: if changing the endpoints doesn’t move the number, the bottleneck is a fixed structure in the path.

False trails (so you don’t repeat them)

NIC firmware. 28.45.4028 and 28.47.1088 behaved identically. Not it. (We later unified the fleet on 28.47.1088 via the official mlnx-fw-updater — fine, just not the fix.)
“GB10 GDR is unavailable.” NCCL logs cuMemGdrSupport 0 / “GPU Direct RDMA Disabled” on GB10 — that’s normal on a unified-memory part and is not a bottleneck. Chasing it (dmabuf symbols, nvidia_peermem, iommu.passthrough=1) was a multi-hour dead end.
The 27W “insufficient power” message. GB10 units log a PCIe slot-power complaint that looks damning. It’s cosmetic — a SlotPowerLimit=0 reporting artifact that fires on perfectly healthy systems. Ignore it.
The 13→196 Gbps firmware-throttle class. Real on some GB10 units (see chronara.io below), but not this case — firmware changes didn’t move our ceiling. Worth ruling out fast, then moving on.
ib_write_bw numbers on GB10. perftest has a documented GB10 false-negative mode. Use an NCCL all-reduce as ground truth and treat perftest as corroboration only.
The CX7 degraded-state trap. Separate from all of the above: any link transition — a cable move, a port bounce, a switch reboot — can re-arm a low-throughput state on the CX7 that survives reboots and only clears with a full AC drain (power off, unplug for 60 s). This poisons debugging: you change a config, bounce the link to apply it, measure garbage, and wrongly conclude the config failed. The rule that saved us: drain, then measure. Never judge a change whose application involved a link transition until the nodes have been drained after it.

The unlock was mundane: direct-connect two Sparks and you get full bandwidth. One cable move splits “NIC/host stack” from “switch path” definitively. Two community builds (a 2-Spark CRS804 write-up and a 4× GB10 + CRS804 thread hitting 192 Gbps) proved the exact hardware combination works — so it had to be config.

Root cause #1 — the RoCE bridge wasn’t hardware-offloaded

A QP-count-independent ~10 Gb/s cap through a 400G-class switch is the signature of software forwarding: traffic hair-pinning through the switch’s ARM CPU instead of the Marvell ASIC. Confirmed three ways:

switch CPU showed real work (ethernet, kernel processes) during transfers;
the boot log said “hardware offloading activated on bridge bridge“ (the mgmt bridge) — but never for the RoCE bridge;
host MACs were learned software-only (“D”) in /interface/bridge/host, not hardware (“E”).

Why the ASIC refused: each CRS804 DDQ cage breaks out into 8 sub-ports. The unused sub-ports (qsfp56-dd-*-2..-8) had been left in the default management bridge, while the active node legs lived in the RoCE bridge. That splits a single physical cage’s lanes across two bridges, and the Marvell ASIC will not offload a bridge in that state.

Fix (management rides ether1/ether2, so it’s untouched):

/interface/bridge/port/remove [find bridge=bridge interface~"qsfp56-dd"]

Instantly:

bridge,info hardware offloading activated on bridge "roce-br" ports: qsfp56-dd-1-1, 2-1, 3-1, 4-1, ...

Ports flipped to the H flag, host MACs became E entries, and switch CPU dropped to 0% under load. But single-flow still wasn’t right — two more bugs were hiding behind the first.

Root cause #2 — QoS/PFC stalls RoCE on the offloaded path

Once the ASIC actually enforced port config, the carefully applied “lossless RoCE” recipe (trust-l3=keep, pfc=pfctc3) killed the data path: QP/GID exchange completed, the sender’s tx_packets_phy climbed, local_ack_timeout_err climbed — and no packets reached any switch leg. Ping passed; bulk RoCE vanished.

This matches the wider pattern on Marvell-based CRS switches: RouterOS QoS/PFC has a history of breaking RDMA (see the CRS504 iSER thread below), and the working GB10 builds all run lossy RoCE at full rate. So:

# per active leg — QoS OFF
/interface/ethernet/switch/qos/port/set [find name~"qsfp56-dd"] trust-l3=ignore pfc=disabled
# host side, per rail netdev
mlnx_qos -i enp1s0f0np0 --pfc 0,0,0,0,0,0,0,0

Lossy RoCE is the accepted end state here. At 4-node scale it demonstrably runs at line rate. Do not re-enable PFC/lossless without re-proving the rail benchmark afterward — on this offloaded path it stalls the data plane.

Root cause #3 — duplicate IPs → ARP flux

With offload on and QoS off, single-flow still died in the strangest way yet: the NIC physically transmitted (tx_packets_phy +28k, zero discards) but the receiver’s PHY counters never moved and no switch leg counted the packets — while ping worked. Classic multi-NIC-same-subnet ARP flux.

The addressing was genuinely broken: several virtual interfaces shared one IP (two nodes had all four rails on a single address). The RoCE address handle resolved to a MAC that no longer mapped to a coherent path, and the frames went nowhere attributable.

Fix — 16 unique IPs, one subnet per rail (deterministic routing, zero ARP contention). Host octet mirrors the management octet:

enp1s0f0np0   -> rail0 -> 10.9.0.<oct>/24
enp1s0f1np1   -> rail1 -> 10.9.1.<oct>/24
enP2p1s0f0np0 -> rail2 -> 10.9.2.<oct>/24
enP2p1s0f1np1 -> rail3 -> 10.9.3.<oct>/24

MTU 9000, IPv6 off on fabric interfaces, plus:

sysctl -w net.ipv4.conf.all.arp_ignore=1 \
          net.ipv4.conf.all.arp_announce=2 \
          net.ipv4.conf.all.rp_filter=2

(Netdev names differ across GB10 vendors — map yours with ibdev2netdev.) The moment the IPs were unique per rail, single-QP ib_write_bw jumped to 109 Gb/s.

Results

Test	Before	After
`ib_write_bw` single-QP, every node-pair, every rail	~9.5 Gb/s cap	109 Gb/s (uniform)
Bidirectional (`-b`), one rail	—	~206 Gb/s aggregate (~103 each way — full-duplex saturation of a ~100G rail)
4-node NCCL all-reduce busbw	0.46 GB/s	23.33 GB/s (peak 24.63)
2-node NCCL busbw	0.46 GB/s	26.58 GB/s
Switch CPU / host CPU under load	busy / busy	0% / ~94% idle

All 16 fabric IPs, all 4 rails, all 4 nodes: 109 Gb/s each, nothing degraded. Exactly as the reference builds promised — at line rate over RDMA, the CPUs are asleep.

A note on that bidirectional figure: each rail is ~100G (the CX7 multi-host split puts two ~100G logical NICs on each 200G port), so 109 Gb/s unidirectional and ~206 Gb/s full-duplex is a saturated rail, not a 200G flow. Single-flow apps see ~100G per rail by design; the aggregate comes from driving rails in parallel, which NCCL does natively.

The reproducible recipe

Everything above, condensed into the state you actually want on the boxes.

Switch (RouterOS, CRS804/CRS812-class):

# 1. Bridge hygiene — active legs ONLY in the RoCE bridge; nothing DDQ in mgmt bridge
/interface/bridge/port/remove [find bridge=bridge interface~"qsfp56-dd"]

# 2. Active legs — jumbo; link mode: autoneg with the 200G advertise worked here
#    (some builds needed auto-negotiation=no speed=200G-baseCR4 fec-mode=fec91 —
#    verify with /interface/ethernet/monitor: you want 200Gbps + fec91 either way)
/interface/ethernet/set [find name~"qsfp56-dd" and running] l2mtu=9216 mtu=9000

# 3. QoS OFF on all legs
/interface/ethernet/switch/qos/port/set [find name~"qsfp56-dd"] trust-l3=ignore pfc=disabled

# 4. Golden backup once healthy
/export file=post-fix-golden

Hosts — netplan (/etc/netplan/98-roce-4rail.yaml, NetworkManager renderer; manual ip addr gets stripped):

network:
  version: 2
  renderer: NetworkManager
  ethernets:
    enp1s0f0np0:   { addresses: [10.9.0.OCT/24], mtu: 9000, dhcp4: no, dhcp6: no, link-local: [] }
    enp1s0f1np1:   { addresses: [10.9.1.OCT/24], mtu: 9000, dhcp4: no, dhcp6: no, link-local: [] }
    enP2p1s0f0np0: { addresses: [10.9.2.OCT/24], mtu: 9000, dhcp4: no, dhcp6: no, link-local: [] }
    enP2p1s0f1np1: { addresses: [10.9.3.OCT/24], mtu: 9000, dhcp4: no, dhcp6: no, link-local: [] }

Hosts — /etc/sysctl.d/98-roce.conf:

net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2
net.ipv4.conf.all.rp_filter=2

Hosts — systemd oneshot (after network-online.target): mlnx_qos -i <each rail netdev> --pfc 0,0,0,0,0,0,0,0 + deep C-state disable. Reboot one node and verify everything returns with zero manual steps before trusting it.

Canonical NCCL launch (the fabric-validation command):

mpirun -np 4 -H node1:1,node2:1,node3:1,node4:1 \
  -x NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3 \
  -x NCCL_IB_ROCE_VERSION_NUM=2 \
  -x NCCL_SOCKET_IFNAME=<mgmt-iface> \
  ./build/all_reduce_perf -b 256M -e 4G -f 2 -g 1 -n 30 -w 5

No NCCL_IB_TC (QoS is off). NCCL_IB_ROCE_VERSION_NUM=2 beats a hard-coded GID index because GID ordering differs per device. “GDR Disabled” in the logs is normal on GB10.

Expected numbers: ~109 Gb/s per rail single-QP; ~206 Gb/s full-duplex per rail; ≥23 GB/s busbw on 4 nodes; switch CPU 0%.

The payoff: TP=4 that was previously “impossible”

The old cluster notes said multi-node TP=4 was impossible (it ran pipeline-parallel over what turned out to be a 10G software path). On the healed fabric, TP=4 of a 397B MoE model came up and ran ~23 tok/s single-stream vs ~10 for PP=4 — a 2.3× win, with all four rails visibly carrying traffic in the PHY counters. The fabric now does exactly what tensor parallelism needs.

(One honest caveat, unrelated to the fabric: this vLLM build hits a sample_tokens RPC timeout in the TP multi-node executor under concurrent batching — a framework bug, not a network one, since single-stream is rock-solid and every rail benchmarks clean.)

Lessons

A clean mlxlink and a 200G link light tell you nothing about whether the fabric forwards at line rate. Both ends of ours reported textbook-perfect health throughout the collapse. Measure with an NCCL all-reduce.
Invariance is a clue. If firmware/version/QP-count don’t move the number, stop tuning endpoints and look for a fixed structure in the path — here, CPU forwarding on the switch.
On MikroTik CRS/Marvell, hardware bridge offload is fragile. Never split a breakout cage’s lanes across bridges. Verify the H flag actually appears and host MACs learn as hardware (E) entries — don’t assume.
RoCE lossless/PFC is not a freebie on these switches. The working configuration is lossy. Re-enable PFC only if you can re-prove line rate afterward.
One subnet per rail, unique IPs everywhere. Same-subnet multi-NIC + RoCE = ARP flux and vanished packets.
Drain, then measure. Any link transition can re-arm the CX7 degraded state; only a full AC power drain clears it. A config test that involved a link bounce is invalid until the nodes have been drained afterward.
Direct-connect is the fastest bisection you have. One cable move splits NIC/host from switch/config conclusively.

References

Primary (shaped the fix):

catharsis.net.au — DGX Spark benchmarking (2-Spark + CRS804) — RDMA proven working through this exact switch; port config + NCCL recipe we replicated.
NVIDIA forums — ConnectX-7 throughput drops after hot-plug on GB10 — 4× GB10 + CRS804, 192 Gbps proof and “clean mlxlink ≠ healthy fabric.”
NVIDIA forums — 200GbE link negotiates 200G but payload is 12 Gbps — the AC power-drain fix; a reboot is insufficient.
NVIDIA forums — CX-7 200GbE via MikroTik CRS812 DDQ breakout — 4-Spark breakout build; dual-half 196–198 Gb/s recipe (MTU 9000, IPv6 off).
MikroTik forum — CRS504 iSER breaks after RouterOS 7.20/7.21 upgrade — precedent: RouterOS QoS-era releases breaking RDMA on Marvell CRS switches.

Platform background:

ServeTheHome — the GB10 ConnectX-7 200GbE networking is really different — the multi-host architecture: 2× PCIe Gen5 x4 domains, four logical interfaces.
ServeTheHome — 8× NVIDIA GB10 cluster — 8× GB10 on MikroTik DDQ reference build.
chronara.io — GX10 ConnectX-7 200GbE — the 13→196 Gbps firmware-throttle fix class (which our case was not — stated for searchers).
linux-rdma/perftest #394 — perftest GB10 false-negative.
NVIDIA docs — DGX Spark clustering guide — official stacking/clustering reference.
MikroTik docs — Quality of Service — RouterOS RoCE/QoS documentation (the config that must stay off on this path).

False trails / related symptom threads (so searchers land here):

NVIDIA forums — inter-Spark link capped at 13 Gbps, “PCIe power throttling 27W” — the 27W message is cosmetic.
NVIDIA forums — direct QSFP only 13–16 Gbps instead of 200G
NVIDIA forums — low throughput on 200G QSFP56 DAC between two Sparks
wisp.net.au — 2× DGX Spark via ConnectX-7 400G breakout on CRS804-4DDQ — the original CRS804 pinning build.
dredyson.com — Kimi 2.6 / Qwen 3.5 397B on 8× GB10 + CRS804 — host-side mlnx_qos trust-state reference on 8× GB10 + CRS804.

Fabric healed 2026-07-22. Hardware: 4× GB10 (MSI EdgeXpert) + MikroTik CRS804-4DDQ. Applies to the GB10 family generally: DGX Spark, ASUS Ascent GX10, Gigabyte AI TOP ATOM, Dell Pro Max, and siblings.

dgx-spark-roce-crs804-fix