We test the leaf and spine switches of an AI or storage fabric on the CloudLAB bench with RoCEv2 traffic on 100G ports: PFC and ECN behaviour under incast, buffer headroom, ECMP balance across the uplinks, latency per flow at load, and a link failure with flows in flight. The scope follows the bench: what is on it today goes into the plan, nothing else. A run takes from 2 weeks and ends with a report that holds drops and pause time per queue at each incast ratio, p99 latency, link balance and failover time against your fabric design. Test plan and quote in 1–3 business days.
When the number is needed
A lossless Ethernet fabric is a set of thresholds on every switch. Each point below asks whether the thresholds on the switches match the traffic the cluster will produce, and each gets its own scope. The cases stay the same, so the shortlist number, the acceptance number and the post-upgrade number compare.
Every candidate runs the same incast, balance and failover cases with the same RoCEv2 endpoints. Vendors see the plan before the first packet and the results after the last.
Ranked table per KPI, buffer behaviour per model, 3–4 weeks for 3 models
The delivered leaf and spine with the production config: DSCP to queue mapping, PFC priority, ECN thresholds, buffer profile and hashing. Measured against your fabric design, before the GPUs arrive or before the first job.
Signed report with pass and fail per case, config exports, from 2 weeks
The acceptance cases run again on the new release or the new buffer profile. What moved, by how much, and whether the fabric still clears the design. The cluster keeps running; we test the lab pair or a spare leaf.
Diff per KPI against the last run, 3 business days
What we test
The bench is the same, the traffic and the pass criteria are not. The table is the starting point of the plan; your fabric design and the job profile set the thresholds.
| Piece under test | Numbers in the report | Traffic and config on the bench | Typical size |
|---|---|---|---|
| 01Leaf switch, one box | Drops and PFC pause time per queue at 2:1 to 16:1 incast, ECN marking against the configured thresholds, buffer headroom, latency per flow at each load step | RoCEv2 flows from emulated endpoints with a TCP background on the same ports; your DSCP map, PFC priority, ECN thresholds and buffer profile from the config export | 32–64 ports, 100G |
| 02Leaf and spine pod | ECMP balance across the uplinks, drops under cross-leaf incast, latency per hop, rehash time and drops when an uplink goes down | Flows between leaves through the spine, long and short flows mixed, hashing as configured, one uplink taken down on purpose with flows in flight | 2 leaves and 1–2 spines, 100G |
| 03Storage fabric | Lossless behaviour of the storage queue next to TCP traffic, pause propagation between ports, latency per I/O-sized flow | NVMe over RoCE sized flows and a TCP mix on the same leaf, storage priority as configured, headroom per port | 1 leaf, 25G and 100G |
| 04Congestion control on the NIC | Rate reduction and recovery on ECN marks, fairness between flows sharing a port, time to settle after a burst | Your NIC model behind the leaf under test, DCQCN or the vendor profile as configured; agreed per setup and written into the plan only when the bench has the card | 2–8 NICs, 100G |
Any switch vendor and NOS. 100G ports with emulated RoCEv2 endpoints on the bench today; 400G and GPU-level collective runs are agreed per setup. Named test tools are picked for the setup and listed in the report.
Around the switch
A lossless number only means something with the right things around the switch: the many-to-one traffic a collective produces, the thresholds your config sets, and the faults the fabric has to survive. All three are on the bench at once.
Emulated RoCEv2 endpoints send into one receiver port at 2:1, 4:1, 8:1 and 16:1, with a TCP background on the neighbouring ports. That is what the last step of an all-reduce looks like to a leaf.
DSCP to queue mapping, the PFC priority, ECN min and max, the buffer profile and the hash fields from your export. The report shows which line each drop or pause traces back to.
One uplink down, a spine reload, a port flap on the receiver. Rehash time, drops and pause time during each event, timed and logged.
Typical scenario
An integrator hands over the leaf and spine of a training pod for acceptance, two weeks before the GPUs. The datasheet says lossless Ethernet. With the delivered config an 8:1 incast drops 0.3% of RoCEv2 packets and p99 latency is 48 µs against a design of 20. The run log points at ECN switched off on the RoCE queue and PFC enabled on the wrong priority. The integrator and the vendor fix the thresholds and the buffer profile on day 7, the rerun gives 0 drops and 11 µs, ECMP imbalance across the uplinks goes from 38% to 6% after a hash change. The pod is accepted on a signed report, and the same cases run again after the next NOS release in 3 days.
Composite scenario from projects of this type. Numbers are realistic and not from one client.
What you take away
The report is built for the meeting after the test: the vendor, the integrator and your platform team read the same table, with the pass band from the fabric design next to the measured value
Each number comes with its evidence: generator statistics, switch counters per queue, captures and the config export of the run
| KPI | Pass band | Measured | Result |
|---|---|---|---|
| Drops at 8:1 incast, RoCE queue | 0 | 0 | Pass |
| PFC pause time per minute | ≤ 50 ms | 12 ms | Pass |
| p99 latency at 90% load | ≤ 20 µs | 11 µs | Pass |
| ECMP imbalance across 4 uplinks | ≤ 10% | 6% | Pass |
| Uplink down, flows in flight | ≤ 100 ms | 70 ms | Pass |
| ECN marks start | at 150 KB | from 180 KB | Pass with a note |
Where it runs
Switches ship to CloudLAB or we connect the bench to your lab, and you watch the run through the portal. The bench has 100G ports with emulated RoCEv2 endpoints, PFC and ECN today; it grows step by step and the plan lists only what is on it for your run. The frame-level base is RFC 2544 and RFC 2889; the lossless cases are written procedures you sign off.
How a load run goes, step by step RFC 2544: throughput, latency, loss RFC 2889: switch benchmarks Get the test plan written first
Before the brief
Any vendor and any NOS: leaf and spine boxes with 25G and 100G ports, as one switch or as a small pod of 2 leaves and 1–2 spines. We need the models, the port plan and an export of the config; the rest is in the plan.
No. The RoCEv2 flows come from emulated endpoints on the bench, which is what lets us run a 16:1 incast on demand and repeat it after a change. A collective run on real GPUs is a different test and is agreed per setup, with your servers or ours written into the plan.
With your config: DSCP to queue mapping, PFC priority, ECN min and max, buffer profile and hash fields from an anonymised export. A vendor-default run can be added as a second baseline so you see what each threshold changes.
The bench runs 100G ports today and grows step by step. A 400G leaf is agreed per setup and goes into the plan only when the bench has the ports for it; we say so before the quote, not after. Ultra Ethernet is on the blog and in our own development, not on the bench yet.
Yes. Boxes go into the CloudLAB rack with KVM and console access and return after the run. For a fabric that is already racked we connect the bench to your lab over a dedicated link and test a spare leaf or the lab pair.
Test plan and quote in 1–3 business days after the brief. A run takes from 2 weeks for one leaf and 3–4 weeks for a shortlist of 3 models or a pod. The report holds the result table, queue counters, latency histograms, captures and config exports, and the rerun of fixed items is part of the plan.
No prices on the site: scope first, then the quote. The vendor and the integrator can sit in on the brief.
Send the shortlist
Name the models and the KPIs that matter. An engineer replies within one business day with a test plan draft and a quote.
Prefer to chat? Message an engineer on WhatsApp
We use what you send only to answer you. See our Privacy Policy.