How we automate network regression tests: every build, one diff

Release notes for a new firmware say "bug fixes and performance improvements". Nobody measures the second half. A manual regression takes an engineer 2 to 3 days per release, and every deadline cuts a few items from it, the slow ones first: full-table convergence, the frame-size sweep, the 40-item checklist. We turn that list into a regression suite that reruns on every firmware, config or release change and compares the result with the last build you approved.

The bench

On a CloudLAB subscription the device stays on the bench between sessions. Automatic patching connects it to the bench generators or to our own traffic generator, so a run needs no ticket and no engineer in the room. The device is configured through its API; a box without one is driven through the console or the web interface, which is slower and more fragile.

Each test case is code in your repository: topology, device configuration, traffic profile, load steps, duration, pass criterion. Your CI job calls the suite over an API or a CLI with the build number and gets the verdict next to the unit tests; a nightly schedule uses the same entry point. Why security platforms need a recheck after each update is in a separate post. This one is about the mechanics.

What we check

• Starting state: the firmware version the device reports, the configuration applied by script, counters before the first case and after the last
• Throughput and latency on 1G to 100G ports with your frame sizes and traffic profile, against the baseline
• Latency p99 at each load step, where a slower code path often shows before the maximum rate moves
• Convergence and failover with the full routing or session table: link loss, peer loss, HA switchover, the gap per flow in milliseconds
• The functional checklist, 40 to 150 items with a capture each; known issues stay marked as known
• Security performance with inspection on: throughput, connections per second, concurrent sessions
• Drift: every KPI over the last builds side by side

Each KPI gets a tolerance, agreed with your team before the first unattended run: a few percent for throughput, a fixed limit in milliseconds for convergence, zero for a functional item. The baseline is the first full run on a build you trust, stored with its configuration, traffic profile and bench layout, so later runs are compared like with like.

What usually breaks

The quiet regression. An example nightly run on a core router build: 37 of 38 checklist items pass, convergence is inside its limit, 1518-byte throughput is 6 % below the previous build and latency p99 at 70 % load is up 18 %. The release notes mention neither. The run takes 2 h 40 min unattended, and the red result is in the pipeline at 06:10. By hand, a 40-item checklist on a core router firmware takes 2 days.

Drift. A KPI that loses 1 % per release passes every single check and is 6 % down after a year. Each comparison with the previous build is green; the one with the approved baseline is not. The report keeps both.

A suite that cries wolf. In the first weeks a red run is often the suite's fault: a web form that moved a field, a baseline taken with another traffic profile, zero tolerance on a KPI that varies by 2 % between runs. Each case is fixed in the suite and logged, so that later a red bar points at the device.

What is in the report

One page your team reads in 5 minutes: what changed since the last approved build, pass, fail or known per item, the KPI diff to the previous build and to the baseline, the trend. Behind it: generator statistics, device counters before and after, the capture of each failed item, the console log. Scripts, configuration, thresholds and run history go to your repository with documentation, so the suite also runs in your lab. The first suite takes from 3 weeks, starting with a manual run that gives the test plan and the baseline.

Where to start

Send us the list you run before a release, the device model and how your pipeline starts a job. We answer with a draft test plan: the items, the bench layout and the dates. The scripts and the report are yours to keep and to show to the vendor. Write to us or look at what else the bench does.