{"id":568,"date":"2026-07-22T19:18:50","date_gmt":"2026-07-22T12:18:50","guid":{"rendered":"https:\/\/itadmin.blog\/?p=568"},"modified":"2026-07-22T19:18:51","modified_gmt":"2026-07-22T12:18:51","slug":"dgx-spark-roce-crs804-fix","status":"publish","type":"post","link":"https:\/\/itadmin.blog\/?p=568","title":{"rendered":"dgx-spark-roce-crs804-fix"},"content":{"rendered":"\n<h1 class=\"wp-block-heading\">GB10 cluster stuck at ~10 Gb\/s through a MikroTik CRS804? It was the switch \u2014 in three places<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\"><em>How a 4\u00d7 DGX Spark (GB10) cluster went from a hard ~10 Gb\/s ceiling and a ~5 Gb\/s single-flow RoCE collapse (NCCL all-reduce: 0.46 GB\/s) to 109 Gb\/s per rail and 23\u201326 GB\/s NCCL busbw \u2014 by fixing three switch\/host config bugs. None of them were the NIC, the cable, the firmware, or &#8220;GB10 GDR.&#8221;<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This was debugged on <strong>MSI EdgeXpert<\/strong> units, but it applies to the whole <strong>GB10 family<\/strong> \u2014 NVIDIA DGX Spark, ASUS Ascent GX10, Gigabyte AI TOP ATOM, Dell Pro Max, and the rest \u2014 because the ConnectX-7 multi-host networking is identical across them. If your search history says <em>&#8220;DGX Spark 10 Gbps ceiling&#8221;<\/em>, <em>&#8220;GB10 RoCE slow through switch&#8221;<\/em>, or <em>&#8220;CRS804 RDMA collapse&#8221;<\/em>, you&#8217;re in the right place.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">TL;DR<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Four GB10 Sparks, each with a ConnectX-7 (2 cables \u2192 4 ~100G &#8220;rails&#8221; per node), wired to a <strong>MikroTik CRS804-4DDQ<\/strong> via 400G\u21922\u00d7200G breakout cables. Every transport pinned around 10 Gb\/s; single-flow RoCE collapsed to ~5 Gb\/s; NCCL all-reduce sat at <strong>0.46 GB\/s<\/strong> \u2014 while <code>mlxlink<\/code> was clean on both ends, links negotiated 200G\/RS-FEC, and the switch showed zero drops. Direct-connecting two Sparks gave full bandwidth, which pinned it on the <strong>switch path<\/strong>. Three independent bugs, all required:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>The RoCE bridge wasn&#8217;t hardware-offloaded.<\/strong> Unused QSFP-DD breakout sub-ports were sitting in the <em>management<\/em> bridge, splitting each physical cage&#8217;s lanes across two bridges \u2014 the Marvell ASIC refuses to offload a bridge in that state. Traffic fell to the <strong>switch CPU<\/strong> \u2192 a hard ~10 Gb\/s cap, QP-count-independent.<\/li>\n\n\n\n<li><strong>Switch QoS\/PFC stalls RoCE on the offloaded path.<\/strong> With <code>trust-l3=keep<\/code> \/ <code>pfc=\u2026<\/code> the connection sets up but <strong>data never flows<\/strong>. Fully <strong>off<\/strong> (lossy RoCE) is the correct end state on this platform.<\/li>\n\n\n\n<li><strong>Duplicate fabric IPs \u2192 ARP flux.<\/strong> Multiple virtual interfaces shared one IP; RDMA packets left the NIC and vanished before the switch. Fixed with <strong>16 unique per-rail IPs<\/strong>, one subnet per rail.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">Result: <strong>109 Gb\/s per rail<\/strong> (uniform, every node-pair, every rail), <strong>~206 Gb\/s full-duplex aggregate per rail<\/strong>, <strong>23.33 GB\/s 4-node NCCL busbw<\/strong> (26.58 for 2-node), <strong>switch CPU 0%, host CPUs ~94% idle<\/strong>.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Quick fix: a Claude Code prompt<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">If you just need your cluster running and you drive it with Claude Code (or any capable agent), paste this. It encodes the whole diagnosis in the order that works. Full explanations follow in the rest of the post.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>MISSION: Diagnose\/fix GB10 (DGX Spark family) RoCE collapse through a MikroTik\nCRS804\/CRS812-class switch. Known failure: ~10 Gb\/s ceiling all transports \/\nsingle-flow RDMA ~5 Gb\/s \/ NCCL busbw &lt;1 GB\/s despite clean 200G links.\n\nSAFETY RAILS:\n- \/export file=pre-fix on the switch BEFORE any change; one change at a time.\n- Confirm mgmt rides the copper ports (ether1\/2), NOT the DDQ ports, before\n  touching bridges.\n- GROUND TRUTH = NCCL all_reduce\/sendrecv. perftest\/ib_write_bw has a documented\n  GB10 false-negative mode (linux-rdma\/perftest#394) \u2014 corroborate, never trust\n  it alone.\n- DRAIN RULE: any link transition (cable move, port bounce, switch reboot) can\n  re-arm a CX7 low-throughput state that ONLY a full AC drain clears (power off,\n  unplug 60s). A reboot is insufficient. Drain before judging any result.\n- IGNORE the \"27W insufficient power\" dmesg on GB10 \u2014 confirmed cosmetic, fires\n  on healthy systems.\n\nCHECK 1 \u2014 SWITCH BRIDGE HARDWARE OFFLOAD (the ~10G cap):\nDetect: during a transfer, switch CPU shows real load (\/system\/resource\/cpu);\nboot log lacks \"hardware offloading activated on bridge &lt;your-roce-bridge&gt;\";\nbridge ports missing H flag; host MACs learned \"D\" (software) not \"E\" (hardware)\nin \/interface\/bridge\/host.\nCause: breakout sub-ports of one physical cage split across two bridges (unused\n-2..-8 legs auto-membered into the default bridge).\nFix: \/interface\/bridge\/port\/remove &#91;find bridge=bridge interface~\"qsfp56-dd\"]\nVerify: log shows \"hardware offloading activated on bridge &lt;roce-bridge&gt;\",\nports show H, switch CPU ~0% under load.\n\nCHECK 2 \u2014 SWITCH QoS\/PFC (connection works, data hangs):\nDetect: RDMA QP\/GID exchange succeeds, sender tx_packets_phy climbs,\nlocal_ack_timeout_err climbs, receiver sees nothing. Ping passes.\nFix (switch): \/interface\/ethernet\/switch\/qos\/port\/set &#91;find name~\"qsfp56-dd\"]\ntrust-l3=ignore pfc=disabled\nFix (hosts): mlnx_qos -i &lt;each rail netdev&gt; --pfc 0,0,0,0,0,0,0,0\nLossy RoCE is the accepted end state. Do not re-enable PFC without re-proving\nline rate afterward.\n\nCHECK 3 \u2014 FABRIC ADDRESSING (packets transmitted but vanish):\nDetect: ip -br addr across all nodes \u2014 ANY duplicate IP across the 4 rail\nnetdevs per host is fatal (ARP flux).\nFix: one subnet per rail, 4 unique IPs per host, 16 total on 4 nodes:\nrail N netdev -&gt; 10.9.N.&lt;host-octet&gt;\/24, MTU 9000, IPv6 off on fabric ifaces,\nsysctls arp_ignore=1 arp_announce=2 rp_filter=2. Map devices with ibdev2netdev\n(names differ across GB10 vendors). Persist via netplan, not ip commands \u2014\nNetworkManager strips manual IPs.\n\nVALIDATE (in order, expected numbers):\n1. ib_write_bw single-QP, one rail: ~109 Gb\/s (per ~100G rail).\n2. Every node-pair, rail 0 + spot-check others: uniform ~109.\n3. 2-node NCCL sendrecv: ~20-26 GB\/s busbw. GDR shows \"Disabled\" \u2014 NORMAL on\n   GB10 unified memory, not a bottleneck, do not chase it.\n4. Full-cluster NCCL all_reduce, all rails (NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,\n   mlx5_3, NCCL_IB_ROCE_VERSION_NUM=2, NCCL_SOCKET_IFNAME=&lt;mgmt&gt;, no\n   NCCL_IB_TC): &gt;= ~23 GB\/s busbw on 4 nodes.\n5. Switch CPU 0% and host CPUs idle during all of the above.\nThen \/export file=post-fix-golden and persist everything (netplan + systemd\noneshot for mlnx_qos\/C-states + sysctl.d).\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">The setup<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>4\u00d7 GB10 Sparks (MSI EdgeXpert; 128 GB unified LPDDR5X each), ConnectX-7 200GbE in multi-host mode \u2014 each 200G port presents as <strong>two ~100G logical NICs<\/strong> on two PCIe Gen5 x4 domains, so 2 cables per node = <strong>4 rails per node<\/strong>.<\/li>\n\n\n\n<li>MikroTik <strong>CRS804-4DDQ<\/strong> (Marvell 98DX7335), RouterOS 7.23.2, QSFP-DD 400G\u21922\u00d7QSFP56 200G breakout cables.<\/li>\n\n\n\n<li>Goal: multi-node tensor parallelism for a 397B MoE model. TP moves an all-reduce across the fabric every layer \u2014 it lives or dies on this bandwidth.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">The symptom, and why it was maddening<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Everything <em>looked<\/em> healthy:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>mlxlink<\/code> clean on <strong>both<\/strong> ends: 200G, 4x width, RS-FEC (544,514), &#8220;No issue was observed.&#8221;<\/li>\n\n\n\n<li>Zero drops on the switch, zero drops on the NIC, sub-millisecond latency.<\/li>\n\n\n\n<li><code>ib_write_bw<\/code> connected and ran \u2014 it just topped out around <strong>9.5 Gb\/s<\/strong> no matter how many QPs.<\/li>\n\n\n\n<li><strong>NCCL all-reduce: 0.46 GB\/s busbw<\/strong> (\u22483.7 Gb\/s) \u2014 the &#8220;~5 Gb\/s collapse.&#8221;<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">And <em>nothing<\/em> moved the number: RouterOS version, NIC firmware (28.45 vs 28.47), adaptive routing on\/off, deep C-states, multi-QP, even a full lossless-PFC recipe. Every transport \u2014 RoCE, TCP over mgmt, TCP over the fabric \u2014 converged around 0.5\u20131 GB\/s.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That invariance is the tell: <strong>if changing the endpoints doesn&#8217;t move the number, the bottleneck is a fixed structure in the path.<\/strong><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">False trails (so you don&#8217;t repeat them)<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>NIC firmware.<\/strong> 28.45.4028 and 28.47.1088 behaved identically. Not it. (We later unified the fleet on 28.47.1088 via the official <code>mlnx-fw-updater<\/code> \u2014 fine, just not the fix.)<\/li>\n\n\n\n<li><strong>&#8220;GB10 GDR is unavailable.&#8221;<\/strong> NCCL logs <code>cuMemGdrSupport 0<\/code> \/ &#8220;GPU Direct RDMA Disabled&#8221; on GB10 \u2014 that&#8217;s <strong>normal<\/strong> on a unified-memory part and is <strong>not<\/strong> a bottleneck. Chasing it (dmabuf symbols, <code>nvidia_peermem<\/code>, <code>iommu.passthrough=1<\/code>) was a multi-hour dead end.<\/li>\n\n\n\n<li><strong>The 27W &#8220;insufficient power&#8221; message.<\/strong> GB10 units log a PCIe slot-power complaint that looks damning. It&#8217;s <strong>cosmetic<\/strong> \u2014 a SlotPowerLimit=0 reporting artifact that fires on perfectly healthy systems. Ignore it.<\/li>\n\n\n\n<li><strong>The 13\u2192196 Gbps firmware-throttle class.<\/strong> Real on some GB10 units (see chronara.io below), but not this case \u2014 firmware changes didn&#8217;t move our ceiling. Worth ruling out fast, then moving on.<\/li>\n\n\n\n<li><strong><code>ib_write_bw<\/code> numbers on GB10.<\/strong> perftest has a documented GB10 false-negative mode. Use an NCCL all-reduce as ground truth and treat perftest as corroboration only.<\/li>\n\n\n\n<li><strong>The CX7 degraded-state trap.<\/strong> Separate from all of the above: <strong>any link transition<\/strong> \u2014 a cable move, a port bounce, a switch reboot \u2014 can re-arm a low-throughput state on the CX7 that survives reboots and <strong>only clears with a full AC drain<\/strong> (power off, unplug for 60 s). This poisons debugging: you change a config, bounce the link to apply it, measure garbage, and wrongly conclude the config failed. The rule that saved us: <strong>drain, then measure.<\/strong> Never judge a change whose application involved a link transition until the nodes have been drained after it.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The unlock was mundane: <strong>direct-connect two Sparks and you get full bandwidth.<\/strong> One cable move splits &#8220;NIC\/host stack&#8221; from &#8220;switch path&#8221; definitively. Two community builds (a 2-Spark CRS804 write-up and a 4\u00d7 GB10 + CRS804 thread hitting 192 Gbps) proved the <em>exact hardware combination<\/em> works \u2014 so it had to be config.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Root cause #1 \u2014 the RoCE bridge wasn&#8217;t hardware-offloaded<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A QP-count-independent ~10 Gb\/s cap through a 400G-class switch is the signature of <strong>software forwarding<\/strong>: traffic hair-pinning through the switch&#8217;s ARM CPU instead of the Marvell ASIC. Confirmed three ways:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>switch CPU showed real work (<code>ethernet<\/code>, <code>kernel<\/code> processes) during transfers;<\/li>\n\n\n\n<li>the boot log said <em>&#8220;hardware offloading activated on bridge <strong>bridge<\/strong>&#8220;<\/em> (the mgmt bridge) \u2014 but <strong>never for the RoCE bridge<\/strong>;<\/li>\n\n\n\n<li>host MACs were learned <strong>software-only<\/strong> (&#8220;D&#8221;) in <code>\/interface\/bridge\/host<\/code>, not hardware (&#8220;E&#8221;).<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Why the ASIC refused:<\/strong> each CRS804 DDQ cage breaks out into 8 sub-ports. The <em>unused<\/em> sub-ports (<code>qsfp56-dd-*-2..-8<\/code>) had been left in the default <strong>management bridge<\/strong>, while the active node legs lived in the RoCE bridge. That splits a single physical cage&#8217;s lanes across two bridges, and the Marvell ASIC will not offload a bridge in that state.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Fix<\/strong> (management rides <code>ether1\/ether2<\/code>, so it&#8217;s untouched):<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/interface\/bridge\/port\/remove &#91;find bridge=bridge interface~\"qsfp56-dd\"]\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Instantly:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>bridge,info hardware offloading activated on bridge \"roce-br\" ports: qsfp56-dd-1-1, 2-1, 3-1, 4-1, ...\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Ports flipped to the <code>H<\/code> flag, host MACs became <code>E<\/code> entries, and <strong>switch CPU dropped to 0%<\/strong> under load. But single-flow still wasn&#8217;t right \u2014 two more bugs were hiding behind the first.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Root cause #2 \u2014 QoS\/PFC stalls RoCE on the offloaded path<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Once the ASIC actually enforced port config, the carefully applied &#8220;lossless RoCE&#8221; recipe (<code>trust-l3=keep<\/code>, <code>pfc=pfctc3<\/code>) <strong>killed the data path<\/strong>: QP\/GID exchange completed, the sender&#8217;s <code>tx_packets_phy<\/code> climbed, <code>local_ack_timeout_err<\/code> climbed \u2014 and no packets reached any switch leg. Ping passed; bulk RoCE vanished.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This matches the wider pattern on Marvell-based CRS switches: <strong>RouterOS QoS\/PFC has a history of breaking RDMA<\/strong> (see the CRS504 iSER thread below), and the working GB10 builds all run <strong>lossy<\/strong> RoCE at full rate. So:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># per active leg \u2014 QoS OFF\n\/interface\/ethernet\/switch\/qos\/port\/set &#91;find name~\"qsfp56-dd\"] trust-l3=ignore pfc=disabled\n# host side, per rail netdev\nmlnx_qos -i enp1s0f0np0 --pfc 0,0,0,0,0,0,0,0\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Lossy RoCE is the accepted end state here.<\/strong> At 4-node scale it demonstrably runs at line rate. Do not re-enable PFC\/lossless without re-proving the rail benchmark afterward \u2014 on this offloaded path it stalls the data plane.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Root cause #3 \u2014 duplicate IPs \u2192 ARP flux<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">With offload on and QoS off, single-flow <em>still<\/em> died in the strangest way yet: the NIC physically transmitted (<code>tx_packets_phy<\/code> +28k, zero discards) but <strong>the receiver&#8217;s PHY counters never moved<\/strong> and no switch leg counted the packets \u2014 while ping worked. Classic multi-NIC-same-subnet ARP flux.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The addressing was genuinely broken: several virtual interfaces shared one IP (two nodes had <em>all four<\/em> rails on a single address). The RoCE address handle resolved to a MAC that no longer mapped to a coherent path, and the frames went nowhere attributable.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Fix \u2014 16 unique IPs, one subnet per rail<\/strong> (deterministic routing, zero ARP contention). Host octet mirrors the management octet:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>enp1s0f0np0   -&gt; rail0 -&gt; 10.9.0.&lt;oct&gt;\/24\nenp1s0f1np1   -&gt; rail1 -&gt; 10.9.1.&lt;oct&gt;\/24\nenP2p1s0f0np0 -&gt; rail2 -&gt; 10.9.2.&lt;oct&gt;\/24\nenP2p1s0f1np1 -&gt; rail3 -&gt; 10.9.3.&lt;oct&gt;\/24\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">MTU 9000, IPv6 off on fabric interfaces, plus:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sysctl -w net.ipv4.conf.all.arp_ignore=1 \\\n          net.ipv4.conf.all.arp_announce=2 \\\n          net.ipv4.conf.all.rp_filter=2\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">(Netdev names differ across GB10 vendors \u2014 map yours with <code>ibdev2netdev<\/code>.) The moment the IPs were unique per rail, single-QP <code>ib_write_bw<\/code> jumped to <strong>109 Gb\/s<\/strong>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Results<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><thead><tr><th>Test<\/th><th>Before<\/th><th>After<\/th><\/tr><\/thead><tbody><tr><td><code>ib_write_bw<\/code> single-QP, every node-pair, every rail<\/td><td>~9.5 Gb\/s cap<\/td><td><strong>109 Gb\/s<\/strong> (uniform)<\/td><\/tr><tr><td>Bidirectional (<code>-b<\/code>), one rail<\/td><td>\u2014<\/td><td><strong>~206 Gb\/s aggregate<\/strong> (~103 each way \u2014 full-duplex saturation of a ~100G rail)<\/td><\/tr><tr><td>4-node NCCL all-reduce busbw<\/td><td>0.46 GB\/s<\/td><td><strong>23.33 GB\/s<\/strong> (peak 24.63)<\/td><\/tr><tr><td>2-node NCCL busbw<\/td><td>0.46 GB\/s<\/td><td><strong>26.58 GB\/s<\/strong><\/td><\/tr><tr><td>Switch CPU \/ host CPU under load<\/td><td>busy \/ busy<\/td><td><strong>0% \/ ~94% idle<\/strong><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">All 16 fabric IPs, all 4 rails, all 4 nodes: <strong>109 Gb\/s each<\/strong>, nothing degraded. Exactly as the reference builds promised \u2014 at line rate over RDMA, the CPUs are asleep.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A note on that bidirectional figure: each <em>rail<\/em> is ~100G (the CX7 multi-host split puts two ~100G logical NICs on each 200G port), so 109 Gb\/s unidirectional and ~206 Gb\/s full-duplex is a <strong>saturated rail<\/strong>, not a 200G flow. Single-flow apps see ~100G per rail by design; the aggregate comes from driving rails in parallel, which NCCL does natively.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The reproducible recipe<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Everything above, condensed into the state you actually want on the boxes.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Switch (RouterOS, CRS804\/CRS812-class):<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># 1. Bridge hygiene \u2014 active legs ONLY in the RoCE bridge; nothing DDQ in mgmt bridge\n\/interface\/bridge\/port\/remove &#91;find bridge=bridge interface~\"qsfp56-dd\"]\n\n# 2. Active legs \u2014 jumbo; link mode: autoneg with the 200G advertise worked here\n#    (some builds needed auto-negotiation=no speed=200G-baseCR4 fec-mode=fec91 \u2014\n#    verify with \/interface\/ethernet\/monitor: you want 200Gbps + fec91 either way)\n\/interface\/ethernet\/set &#91;find name~\"qsfp56-dd\" and running] l2mtu=9216 mtu=9000\n\n# 3. QoS OFF on all legs\n\/interface\/ethernet\/switch\/qos\/port\/set &#91;find name~\"qsfp56-dd\"] trust-l3=ignore pfc=disabled\n\n# 4. Golden backup once healthy\n\/export file=post-fix-golden\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Hosts \u2014 netplan (<code>\/etc\/netplan\/98-roce-4rail.yaml<\/code>, NetworkManager renderer; manual <code>ip addr<\/code> gets stripped):<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>network:\n  version: 2\n  renderer: NetworkManager\n  ethernets:\n    enp1s0f0np0:   { addresses: &#91;10.9.0.OCT\/24], mtu: 9000, dhcp4: no, dhcp6: no, link-local: &#91;] }\n    enp1s0f1np1:   { addresses: &#91;10.9.1.OCT\/24], mtu: 9000, dhcp4: no, dhcp6: no, link-local: &#91;] }\n    enP2p1s0f0np0: { addresses: &#91;10.9.2.OCT\/24], mtu: 9000, dhcp4: no, dhcp6: no, link-local: &#91;] }\n    enP2p1s0f1np1: { addresses: &#91;10.9.3.OCT\/24], mtu: 9000, dhcp4: no, dhcp6: no, link-local: &#91;] }\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Hosts \u2014 <code>\/etc\/sysctl.d\/98-roce.conf<\/code>:<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>net.ipv4.conf.all.arp_ignore=1\nnet.ipv4.conf.all.arp_announce=2\nnet.ipv4.conf.all.rp_filter=2\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Hosts \u2014 systemd oneshot (after <code>network-online.target<\/code>):<\/strong> <code>mlnx_qos -i &lt;each rail netdev&gt; --pfc 0,0,0,0,0,0,0,0<\/code> + deep C-state disable. Reboot one node and verify everything returns with zero manual steps before trusting it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Canonical NCCL launch (the fabric-validation command):<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>mpirun -np 4 -H node1:1,node2:1,node3:1,node4:1 \\\n  -x NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3 \\\n  -x NCCL_IB_ROCE_VERSION_NUM=2 \\\n  -x NCCL_SOCKET_IFNAME=&lt;mgmt-iface&gt; \\\n  .\/build\/all_reduce_perf -b 256M -e 4G -f 2 -g 1 -n 30 -w 5\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">No <code>NCCL_IB_TC<\/code> (QoS is off). <code>NCCL_IB_ROCE_VERSION_NUM=2<\/code> beats a hard-coded GID index because GID ordering differs per device. &#8220;GDR Disabled&#8221; in the logs is normal on GB10.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Expected numbers:<\/strong> ~109 Gb\/s per rail single-QP; ~206 Gb\/s full-duplex per rail; \u226523 GB\/s busbw on 4 nodes; switch CPU 0%.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The payoff: TP=4 that was previously &#8220;impossible&#8221;<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The old cluster notes said multi-node TP=4 was impossible (it ran pipeline-parallel over what turned out to be a 10G software path). On the healed fabric, TP=4 of a 397B MoE model came up and ran <strong>~23 tok\/s single-stream vs ~10 for PP=4 \u2014 a 2.3\u00d7 win<\/strong>, with all four rails visibly carrying traffic in the PHY counters. The fabric now does exactly what tensor parallelism needs.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">(One honest caveat, unrelated to the fabric: this vLLM build hits a <code>sample_tokens<\/code> RPC timeout in the TP multi-node executor under concurrent batching \u2014 a framework bug, not a network one, since single-stream is rock-solid and every rail benchmarks clean.)<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Lessons<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>A clean <code>mlxlink<\/code> and a 200G link light tell you nothing about whether the fabric forwards at line rate.<\/strong> Both ends of ours reported textbook-perfect health throughout the collapse. Measure with an NCCL all-reduce.<\/li>\n\n\n\n<li><strong>Invariance is a clue.<\/strong> If firmware\/version\/QP-count don&#8217;t move the number, stop tuning endpoints and look for a fixed structure in the path \u2014 here, CPU forwarding on the switch.<\/li>\n\n\n\n<li><strong>On MikroTik CRS\/Marvell, hardware bridge offload is fragile.<\/strong> Never split a breakout cage&#8217;s lanes across bridges. Verify the <code>H<\/code> flag actually appears and host MACs learn as hardware (<code>E<\/code>) entries \u2014 don&#8217;t assume.<\/li>\n\n\n\n<li><strong>RoCE lossless\/PFC is not a freebie on these switches.<\/strong> The working configuration is lossy. Re-enable PFC only if you can re-prove line rate afterward.<\/li>\n\n\n\n<li><strong>One subnet per rail, unique IPs everywhere.<\/strong> Same-subnet multi-NIC + RoCE = ARP flux and vanished packets.<\/li>\n\n\n\n<li><strong>Drain, then measure.<\/strong> Any link transition can re-arm the CX7 degraded state; only a full AC power drain clears it. A config test that involved a link bounce is invalid until the nodes have been drained afterward.<\/li>\n\n\n\n<li><strong>Direct-connect is the fastest bisection you have.<\/strong> One cable move splits NIC\/host from switch\/config conclusively.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">References<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Primary (shaped the fix):<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/catharsis.net.au\/blog\/nvidea-dgx-spark-benchmarking\/\">catharsis.net.au \u2014 DGX Spark benchmarking (2-Spark + CRS804)<\/a> \u2014 RDMA proven working through this exact switch; port config + NCCL recipe we replicated.<\/li>\n\n\n\n<li><a href=\"https:\/\/forums.developer.nvidia.com\/t\/connectx-7-throughput-drops-to-25-gbps-after-hot-plug-cable-reconnection-on-gigabyte-gb10-ai-top-atom\/371031\">NVIDIA forums \u2014 ConnectX-7 throughput drops after hot-plug on GB10<\/a> \u2014 4\u00d7 GB10 + CRS804, 192 Gbps proof and &#8220;clean mlxlink \u2260 healthy fabric.&#8221;<\/li>\n\n\n\n<li><a href=\"https:\/\/forums.developer.nvidia.com\/t\/dgx-spark-200gbe-direct-qsfp-link-negotiates-200g-but-payload-is-12-gbps\/373538\">NVIDIA forums \u2014 200GbE link negotiates 200G but payload is 12 Gbps<\/a> \u2014 the AC power-drain fix; a reboot is insufficient.<\/li>\n\n\n\n<li><a href=\"https:\/\/forums.developer.nvidia.com\/t\/connectx-7-200gbe-via-mikrotik-crs812-qsfp-dd-400g-2xqsfp56-200g-breakout\/357162\">NVIDIA forums \u2014 CX-7 200GbE via MikroTik CRS812 DDQ breakout<\/a> \u2014 4-Spark breakout build; dual-half 196\u2013198 Gb\/s recipe (MTU 9000, IPv6 off).<\/li>\n\n\n\n<li><a href=\"https:\/\/forum.mikrotik.com\/t\/crs504-4xq-in-iscsi-over-rdma-iser-not-working-properly-after-upgrade-to-7-20-7-21\/265631\">MikroTik forum \u2014 CRS504 iSER breaks after RouterOS 7.20\/7.21 upgrade<\/a> \u2014 precedent: RouterOS QoS-era releases breaking RDMA on Marvell CRS switches.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Platform background:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/www.servethehome.com\/the-nvidia-gb10-connectx-7-200gbe-networking-is-really-different\/\">ServeTheHome \u2014 the GB10 ConnectX-7 200GbE networking is really different<\/a> \u2014 the multi-host architecture: 2\u00d7 PCIe Gen5 x4 domains, four logical interfaces.<\/li>\n\n\n\n<li><a href=\"https:\/\/www.servethehome.com\/big-cluster-little-power-the-8x-nvidia-gb10-cluster-marvell-cisco-ubiquiti-qnap-arm\/\">ServeTheHome \u2014 8\u00d7 NVIDIA GB10 cluster<\/a> \u2014 8\u00d7 GB10 on MikroTik DDQ reference build.<\/li>\n\n\n\n<li><a href=\"https:\/\/chronara.io\/dev-news\/gx10-connectx7-200gbe\">chronara.io \u2014 GX10 ConnectX-7 200GbE<\/a> \u2014 the 13\u2192196 Gbps firmware-throttle fix class (which our case was <strong>not<\/strong> \u2014 stated for searchers).<\/li>\n\n\n\n<li><a href=\"https:\/\/github.com\/linux-rdma\/perftest\/issues\/394\">linux-rdma\/perftest #394<\/a> \u2014 perftest GB10 false-negative.<\/li>\n\n\n\n<li><a href=\"https:\/\/docs.nvidia.com\/dgx\/dgx-spark\/spark-clustering.html\">NVIDIA docs \u2014 DGX Spark clustering guide<\/a> \u2014 official stacking\/clustering reference.<\/li>\n\n\n\n<li><a href=\"https:\/\/help.mikrotik.com\/docs\/spaces\/ROS\/pages\/189497483\/Quality+of+Service\">MikroTik docs \u2014 Quality of Service<\/a> \u2014 RouterOS RoCE\/QoS documentation (the config that must stay <strong>off<\/strong> on this path).<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>False trails \/ related symptom threads (so searchers land here):<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/forums.developer.nvidia.com\/t\/connectx-7-inter-spark-link-capped-at-13-gbps-expected-200-gbps-pcie-power-throttling-27w\/363461\">NVIDIA forums \u2014 inter-Spark link capped at 13 Gbps, &#8220;PCIe power throttling 27W&#8221;<\/a> \u2014 the 27W message is cosmetic.<\/li>\n\n\n\n<li><a href=\"https:\/\/forums.developer.nvidia.com\/t\/dgx-spark-direct-qsfp-connection-only-getting-13-16-gbps-instead-of-expected-200g-performance\/370035\">NVIDIA forums \u2014 direct QSFP only 13\u201316 Gbps instead of 200G<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/forums.developer.nvidia.com\/t\/low-throughput-on-200g-qsfp56-dac-link-between-two-nvidia-dgx-spark-systems\/368523\">NVIDIA forums \u2014 low throughput on 200G QSFP56 DAC between two Sparks<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/wisp.net.au\/blog\/news\/2-x-dgx-spark-cluster-connected-with-connectx-7-400gbps-breakout-cable-via-mikrotik-crs804-4ddq\">wisp.net.au \u2014 2\u00d7 DGX Spark via ConnectX-7 400G breakout on CRS804-4DDQ<\/a> \u2014 the original CRS804 pinning build.<\/li>\n\n\n\n<li><a href=\"https:\/\/dredyson.com\/fix-kimi-2-6-and-qwen-3-5-397b-fp8-on-8xgb10-cluster-in-under-5-minutes-actually-works-a-beginners-step-by-step-speed-optimization-and-deadlock-resolution-guide\/\">dredyson.com \u2014 Kimi 2.6 \/ Qwen 3.5 397B on 8\u00d7 GB10 + CRS804<\/a> \u2014 host-side <code>mlnx_qos<\/code> trust-state reference on 8\u00d7 GB10 + CRS804.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p class=\"wp-block-paragraph\"><em>Fabric healed 2026-07-22. Hardware: 4\u00d7 GB10 (MSI EdgeXpert) + MikroTik CRS804-4DDQ. Applies to the GB10 family generally: DGX Spark, ASUS Ascent GX10, Gigabyte AI TOP ATOM, Dell Pro Max, and siblings.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>GB10 cluster stuck at ~10 Gb\/s through a MikroTik CRS804? It was the switch \u2014 in three places How a 4\u00d7 DGX Spark (GB10) cluster went from a hard ~10 Gb\/s ceiling and a ~5 Gb\/s single-flow RoCE collapse (NCCL all-reduce: 0.46 GB\/s) to 109 Gb\/s per rail and 23\u201326 GB\/s NCCL busbw \u2014 by&hellip;&nbsp;<a href=\"https:\/\/itadmin.blog\/?p=568\" class=\"\" rel=\"bookmark\">Read More &raquo;<span class=\"screen-reader-text\">dgx-spark-roce-crs804-fix<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_import_markdown_pro_load_document_selector":0,"_import_markdown_pro_submit_text_textarea":"","neve_meta_sidebar":"","neve_meta_container":"","neve_meta_enable_content_width":"","neve_meta_content_width":0,"neve_meta_title_alignment":"","neve_meta_author_avatar":"","neve_post_elements_order":"[\"content\",\"tags\",\"comments\"]","neve_meta_disable_header":"","neve_meta_disable_footer":"","neve_meta_disable_title":"","_themeisle_gutenberg_block_has_review":false,"_ti_tpc_template_sync":false,"_ti_tpc_template_id":"","footnotes":""},"categories":[38,39],"tags":[],"class_list":["post-568","post","type-post","status-publish","format-standard","hentry","category-dgx-spark","category-mikrotik-crs504"],"_links":{"self":[{"href":"https:\/\/itadmin.blog\/index.php?rest_route=\/wp\/v2\/posts\/568","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/itadmin.blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/itadmin.blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/itadmin.blog\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/itadmin.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=568"}],"version-history":[{"count":1,"href":"https:\/\/itadmin.blog\/index.php?rest_route=\/wp\/v2\/posts\/568\/revisions"}],"predecessor-version":[{"id":569,"href":"https:\/\/itadmin.blog\/index.php?rest_route=\/wp\/v2\/posts\/568\/revisions\/569"}],"wp:attachment":[{"href":"https:\/\/itadmin.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=568"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/itadmin.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=568"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/itadmin.blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=568"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}