Meta at OCP 2026: No AI Fabric Gets Accepted Without Benchmarks on 1000+ GPUs

The key takeaway from Meta's post for OCP Global Summit 2026: the hyperscaler does not bring a single segment of its AI network online without a test run. Meta hands its cloud partners a single benchmark suite — point-to-point and collective operations, from dozens to more than 1000 GPUs — and only then accepts the capacity. For smaller companies this is a ready-made template: test before going live, not a post-mortem after an incident.

What Meta showed

IceCube and IcePack. 102.4 Tbit/s blades, 128 ports of 800G, two ASIC options — Broadcom Tomahawk6C and NVIDIA Spectrum-6 — in Meta's first liquid-cooled rack. Both run open FBOSS through OCP SAI. The hardware vendor can change; the software stack stays the same.

FPF — Flattened Planar Fabric (scale-out). Every GPU has eight independent 100G links, one into each of eight planes. A three-tier Clos is collapsed into a two-tier one: fewer hops, lower latency, lower cost. The NIC sprays packets across all planes in hardware, and the host can steer traffic away from a failed plane. Meta notes that the architecture was developed with NVIDIA with failure and planned-maintenance behavior in mind, not only peak load.

BAG — Backend Aggregation (scale-across). An Ethernet super-spine that links fabrics in different buildings and regions at petabit scale; between two regions, tens of Pbit/s. The Prometheus campus is around 1 GW in 2026; Hyperion is growing toward 5 GW. An engineering detail: buffer headroom for lossless Ethernet grows with cable length, so FBOSS measures the link length itself by timing probe traffic in an idle queue and sets headroom from the measured value. Inter-fabric latency is several times higher than intra-fabric latency, and collective libraries that are unaware of distance lose performance as a result.

Tahan (scale-up). An Ethernet switch for MTIA 300 accelerators built on Tomahawk 5 (51.2 Tbit/s, 64×800G). Four Tahan switches connect 16 accelerators in a non-blocking configuration, 6.4 Tbit/s to each. The transport is the same lossless RoCE as in scale-out: routable IPv6, PFC, AI-tuned QoS. The next generation is 216×400G on Tomahawk6C, eight switches for 72 accelerators per rack, with a move to ESUN.

ESUN and OSPA. ESUN 1.0 was released on March 10, 2026: PFC, Credit-Based Flow Control, Link-Level Retry, and a 4-byte header instead of 28–48 bytes of IP/UDP. The working group has more than 175 companies, and ESUN 1.1 is in preparation. OSPA v1.0 (Meta, Microsoft, Alibaba) standardizes the platform layer beneath SAI: power, sensors, firmware, diagnostics.

The tests behind it

ESUN and OSPA set requirements but not verification procedures. Independent analyses note that Meta publishes no data on tail latency or failure recovery. That verification therefore falls to the operator. Five classes of tests follow from the text:

1. Collective operations under load. All-Reduce, All-to-All, Reduce-Scatter on the real topology. The metric is job completion time (JCT) and tail latency, not raw throughput.
2. Lossless behavior. PFC without deadlocks, correct DCQCN/ECN operation, sufficient buffer headroom — especially on long links, where Meta measures the cable before configuring.
3. Degradation and failures. Loss of one of the eight FPF planes, of one link in an ECMP group, redistribution via eBGP link-bandwidth. Compare JCT before and after.
4. Interop. Two ASICs under one NOS, several NIC models, different firmware versions. The more vendors, the more combinations to qualify.
5. Scale. Meta runs benchmarks on 1000+ GPUs. On a test bench this stage is replaced with emulation: an 800GE traffic generator reproduces QP queues, collectives and the reaction to congestion notifications without purchasing GPUs.

The tools already exist. Keysight AI Data Center Builder on the AresONE platform emulates All-Reduce, All-Gather, All-to-All and other collectives on 4–16 ports of 100–800GE and reports JCT, tail latency, loss and congestion points. This is how Dell and Siemon validated an 800GE fabric on SONiC: All-Reduce under congestion, with completion-time spread measured in microseconds.

What this means for business

Meta builds gigawatt-scale campuses and still does not bring capacity online without tests. For a cluster of 32–256 GPUs the logic is the same: idle GPUs cost more than any test cycle. An unverified fabric shows up not as an outage but as slow training — JCT grows by tens of percent, and the root cause takes months to find. One round of emulation on a test bench before purchase, or before acceptance from the integrator, removes that risk and gives you a factual basis for discussions with the vendor.

What to do

Repeating Meta's framework at a smaller scale is realistic: take the target topology, emulate collectives on 400/800GE, capture JCT and tail latency, then take a link down and measure recovery. Run the same cycle on every firmware update and NIC change. If you need such a validation run for your topology, contact TEST4NET: we will discuss the task and propose a test program.

Sources

• Meta Engineering: OCP Summit 2026 — Powering AI with Open Innovations in Scale-Up, Scale-Out and Scale-Across Networks
• remio.ai: Meta OCP Summit 2026 puts open Ethernet against proprietary AI networks
• OCP: The OCP ESUN 1.0 Specification Has Been Released
• Keysight AI Data Center Builder
• Keysight success story: Dell + Siemon
• Keysight AI Fabric RoCEv2 Test Solution (data sheet)