Skip to content

Every feature checked. Every fail with a capture.

We take the feature list from your RFP or the vendor datasheet and the protocol rules from the RFC, turn them into a checklist of 80–150 items and run every item on the device in CloudLAB: BGP, OSPF, IS-IS, MPLS, EVPN and VXLAN against emulated peers on 1G to 100G ports, plus L2, QoS, ACL, high availability and management features. A run takes from 2 weeks. Each item ends as pass, fail or partial with the configuration, the CLI output and the packet capture behind it. The test plan and the quote come within 1–3 business days.

Request a test plan See the checklist areas
Checklist excerpt, EVPN-VXLAN leaf acceptance Items 41–47 of 138. Representative result.
  1. 41BGP EVPN route type 2, MAC and IP advertisementPass
  2. 42Route type 5, IP prefix with L3 VNIPass
  3. 43MAC mobility, sequence number per RFC 7432 §15Fail · pcap
  4. 44ARP suppression with 2,000 hostsPartial
  5. 45Multihoming, ESI and DF electionPass
  6. 46BUM traffic via ingress replicationPass
  7. 47Peer loss, reconvergence under 1 sFail · pcap

138 items 117 pass 14 fail 7 partial

  • 80–150items per checklist
  • 1G · 10G · 100Gports with emulated peers
  • From 2 weeksfor a run
  • 1–3 daystest plan and quote
One line in the datasheet Three conditions on the bench BFD, 50 ms timers 10 sessions pass 500 sessions 30 sessions flap 500 sessions + route churn fail

Where feature lists fail

Three ways a feature is on the list and not in the box

A line in a datasheet means a developer once made the feature work. It says nothing about scale, about the corner of the RFC you depend on, or about the other 20 features switched on next to it.

  1. It works at demo scale

    BFD holds 10 sessions at 50 ms timers. At 500 sessions the timers slip, and 30 of them flap during a routing churn. The datasheet line was true at 10.

  2. It reads the RFC differently

    RFC 4271 says a malformed UPDATE attribute ends the session with a NOTIFICATION. One implementation drops the route and keeps the session up. Both vendors call it BGP.

  3. It breaks in combination

    VXLAN works. The egress ACL works. QoS remarking works. All three on the same port lose the DSCP value on the way out, and no single-feature test catches it.

What goes into the checklist

Eight areas, one checklist, one verdict per item

Items come from 4 sources: your RFP, the vendor datasheet, the MUST and SHOULD clauses of the RFCs you rely on, and your own operating procedures such as failover and upgrade. We draft the list with a pass criterion for each item. You add, strike and sign off before the first test. After that nothing is added or dropped without a note in the report.

Your RFP or requirements Vendor datasheet RFC MUST and SHOULD clauses Your operating procedures Checklist 80–150 items, a pass criterion for each Pass Fail Partial
  • Your RFP or requirements
  • Vendor datasheet
  • RFC MUST and SHOULD clauses
  • Your operating procedures
AreaWhat we checkItemsEvidence in the report
01Routing: BGP, OSPF, IS-IS Session setup and authentication, route exchange and policy, graceful restart, BFD, convergence on peer and link loss 25–40 Emulated peers, captures, routing tables before and after
02MPLS and segment routing LDP, RSVP-TE or SR-MPLS label operations, L3VPN and L2VPN, FRR and TI-LFA switchover time 15–25 Label tables, failover timing with traffic
03EVPN and VXLAN Route types 2, 3 and 5, MAC mobility, multihoming, ARP suppression, BUM handling 15–30 Captures, MAC and IP tables, loss during moves
04L2 features VLAN and Q-in-Q, LACP, STP variants, LLDP, storm control, MAC table scale 10–20 Counters, captures
05QoS and ACL Classification, marking, queueing, policer accuracy, ACL hit logic and scale 10–20 Per-queue counters, measured rates
06High availability Redundant supervisors, ISSU, link and node failover, configuration sync 8–15 Failover time, frames lost per event
07Management and device security SSH, AAA with TACACS+ or RADIUS, SNMPv3, NETCONF and gNMI, syslog, RBAC, image signing 10–20 Session logs, API responses
08Negative tests and recovery Malformed messages, flapping peers, configuration rollback, reload under load 8–15 Captures, logs, recovery time

Item counts are typical for one device role. A list over 150 items is split into phases, and the phase 1 report comes out before phase 2 starts.

How a run goes

Five steps, the first blocker on the day it appears

  1. 1

    Brief 1 call

    The device, its role in your network and the documents the checklist will come from.

    Test plan and quote in 1–3 business days

  2. 2

    Checklist Days 1–3

    We draft 80–150 items, each with a pass criterion. You add, strike and sign off.

    Signed checklist, the only scope of the run

  3. 3

    Bench Days 3–5

    Your device in the lab or your VM on our hosts, emulated peers and topology per the plan, baseline configuration saved.

    Topology diagram and the baseline config

  4. 4

    Execution Days 5–12

    Every item run, captured and recorded. A blocker is reported the same day with the capture, so the vendor case opens while the run continues.

    Daily status, blockers as they appear

  5. 5

    Report and retest Days 12–14

    The report with a verdict and evidence per item. A retest of the failed items after the vendor fix is a short separate run, usually 2–3 days.

    Report, evidence archive, retest option

Typical scenario

Two leaf switches, one checklist, two different answers

An operator refreshes 40 racks and shortlists 2 vendors for the EVPN-VXLAN leaf. Both datasheets tick the same 30 features. The checklist has 138 items. After 2 weeks vendor A passes 129, vendor B 117. Among the 14 fails of vendor B are MAC mobility and reconvergence over 1 s on peer loss, 2 blockers for the planned design. The purchase order goes to vendor A, and vendor B gets the record for its next release.

138
items on the checklist
129 vs 117
items passed
2
blockers found before the PO

Composite scenario from projects of this type. Numbers are realistic and not from one client.

Vendor A Vendor B 100% Routing 33/34 31/34 EVPN and VXLAN 25/28 19/28 L2 features 16/16 15/16 QoS and ACL 17/18 15/18 High availability 12/12 9/12 Management 16/18 17/18 Negative tests 10/12 11/12

Evidence per item

What a fail looks like in the report

Each item in the report has the same 6 fields. A pass has them too. The vendor gets a record it can reproduce, and your team can rerun the item on the production box with the same configuration.

Verdicts follow 4 rules, agreed in the checklist before the run starts.

Pass
The pass criterion of the item is met in every trial.
Fail
The criterion is not met in at least one trial. Severity: blocker, major or minor against your design.
Partial
The feature works with a limitation outside the criterion, for example a lower scale or a workaround in the config.
Not applicable
The item cannot be run on this device or role. It stays in the report with the reason.
Item 43 EVPN and VXLAN Fail

MAC mobility, sequence number per RFC 7432 §15

Spec
RFC 7432 §15: the MAC Mobility extended community carries a sequence number that increments on every move.
Setup
One host moved between leaf-1 and leaf-3 every 10 s, 200 moves, 2,000 other hosts in the fabric.
Observed
The sequence number stays at 0 after the first move. Both leaves advertise the MAC for 40–60 s, and 3% of the frames to the host are black-holed in that window.
Verdict
Fail. Severity: blocker for the planned design with live migration.
Evidence
Capture 43-a (12 MB), route type 2 table before and after the move, configuration diff.
Vendor case
Opened on day 8 of the run, fix promised for the next maintenance release.

Where it runs

CloudLAB: emulated peers at scale, your device in the middle

The bench generators emulate the peers around your device: hundreds of BGP, OSPF, IS-IS and LDP sessions, thousands of routes and EVPN hosts, on 1G to 100G ports. Your device comes to the lab or your virtual appliance runs on our hosts. You watch the run in the browser, with KVM and console access to the device. The generator is picked for the setup and named in the report.

Bench access and capacity Open the CloudLAB portal Combine with load and performance testing

Questions before the brief

What buyers ask first

What is the difference between functional and conformance testing?

Functional testing checks that a feature does what the RFP or the datasheet says in your scenario. Conformance testing checks that the protocol behaves as the RFC requires, including when the peer misbehaves. One checklist covers both, and each item is marked as one or the other.

Where does the checklist come from?

From your RFP, the vendor datasheet, the MUST and SHOULD clauses of the RFCs you rely on and your operating procedures. We draft it, you sign it off. 80–150 items is typical for one device role; longer lists are split into phases.

Can you test a device we have not bought yet?

Yes. Vendors ship evaluation units to the lab under their own loan terms, and for virtual appliances you upload the image. Two or three candidates side by side on the same checklist is the most common form of this service.

Do you need our configuration?

The closer to production, the better. We take your configuration templates and build the topology around them. Where you have none, we configure per the vendor guide and note it in the report.

How long does a run take?

2–3 weeks for 80–150 items, including the checklist review and the report. The first blocker is reported on the day it appears. A retest after a vendor fix takes 2–3 days.

Who talks to the vendor about the fails?

You own the relationship, we provide the record and join the call when needed. Most vendors accept the capture and the configuration from the report as the reproduction, which saves weeks of back and forth.

Send the feature list. We send back the checklist.

Test plan and quote in 1–3 business days. No device on hand yet is fine: tell us the model and the role.

Request a test plan All services

Send the shortlist

You have the method. Now get the numbers.

Send the shortlist and the KPIs that matter. An engineer replies within one business day with a test plan draft and a quote.

Prefer to chat? Message an engineer on WhatsApp

We use what you send only to answer you. See our Privacy Policy.