Skip to content

A regression suite that runs on every build of your device

We turn the tests your team runs by hand before a release into a suite that runs on its own: throughput and latency with your traffic profile, convergence and failover, the functional checklist, service checks and the benchmark sets a tender asks for, started from your pipeline or on a nightly schedule, compared with the previous build against thresholds you set, with a report your team reads in 5 minutes. The scripts, configurations and run history stay yours. From 3 weeks for the first suite, in CloudLAB or in your lab.

Discuss a suite What the suite runs
One run, from commit to report
Commit Build Bench run Compare Report your repository your CI job CloudLAB or yours vs previous build pass, fail, trend YOUR SIDE OUR SIDE, OR YOURS AFTER HANDOVER API or CLI call from the pipeline Result back in the same pipeline
  • From 3 weeksto the first unattended run
  • Every buildor every night, your choice
  • 5 minutesto read the report
  • 100%of the scripts handed over

Who needs it

Three teams that test the same thing every few weeks

A manual regression takes an engineer 2 to 3 days per release and gets shorter with every deadline. The suite runs the full list every time, at night, and the engineer reads the diff in the morning.

  1. A vendor shipping a release every month

    The QA team covers the new features and reruns a third of the old performance tests, because a full pass with every customer profile takes a working week. A field unit on an older firmware has a throughput regression nobody measured. The suite reruns the whole set on every build; the regression shows up as a red bar the morning after the commit.

  2. An operator accepting a firmware before rollout

    The same 40-item checklist on every firmware of the core router: convergence with the full table, failover, QoS, the features the network actually uses. By hand it takes 2 days and the results live in a spreadsheet per engineer. Scripted, it runs over a night on the bench and the result is one page: what changed since the last approved firmware.

  3. An integrator with a repeating acceptance

    Ten sites, the same firewall model, the same acceptance plan. The first run is written as a test plan, the second is scripted, from the third on an engineer starts it from a laptop and attaches the report to the handover. Same plan, same thresholds, a number per site instead of a signature.

What the suite runs

Five families of tests, each with a trigger and a threshold

The suite is built from the tests your release process already needs. Each family has a trigger (every build, nightly, on demand), a threshold the result is checked against and an output that goes back to your pipeline.

FamilyWhat runsTrigger and thresholdOutput
01Throughput and latency Throughput, latency, loss and back-to-back on 1G to 100G ports with your frame sizes and traffic profile, the RFC 2544 set when a customer asks for it Every build; drop beyond the agreed percent of the baseline fails Curve per build, diff to the previous build
02Security performance Application mix from your traffic with inspection on, 200 Gbit/s and more, plus the RFC 9411 KPIs for the datasheet Nightly; throughput, CPS and sessions each against their own threshold Table of KPIs per build, trend over the last releases
03Ethernet services ITU-⁠T Y.1564 configuration and performance tests per service: CIR, EIR, loss, delay, jitter On demand or nightly; pass per service against the SLA values Pass or fail per service with the measured values
04Functional checklist Your 40 to 150 items: protocol behaviour, policy, NAT, HA, QoS, logging, evidence captured per item Every build; one failed item fails the run, items can be marked as known Checklist with pass, fail, known, and the capture per item
05Failure and convergence Link loss, peer loss, HA switchover with the full routing or session table loaded, timed per flow Nightly; gap in milliseconds against the agreed limit per scenario Gap table per scenario, worst flow and median

Families you do not need are left out. Families you need and we have not listed, such as Wi-⁠Fi throughput under TR-⁠398 or a TCP check under RFC 6349, are scripted the same way from the published methodology.

Throughput per build against the baseline
Throughput, % of baseline 1009080 Threshold: baseline minus 3% 100101991009894 Baseline7.1.27.1.47.1.57.2.07.2.1 Illustrative run history. Build numbers and values are examples.

Pass or fail

The result is a comparison, not a number

A single run says what the device did tonight. The suite stores every run, so each result is read against the previous build and against the baseline you approved. The threshold is written down once, with you, before the first automated run.

  1. A baseline you approve

    The first full run on a build you trust becomes the baseline. Every KPI in it is stored with the configuration, the traffic profile and the bench layout, so a later run is compared like with like.

  2. A tolerance per KPI

    Throughput may vary by a few percent between runs; convergence time and a functional item may not. Each KPI gets its own tolerance, agreed with your team, and the suite fails the run when a KPI crosses it.

  3. A trend, not just a flag

    A KPI that loses 1% per release passes every single check and is 6% down after a year. The report shows the last builds side by side, so the slow drift is visible before it crosses the line.

How the project goes

Four steps from a manual run to a suite your pipeline starts

  1. 1

    Manual run

    Week 1

    We run your current regression once by hand on the bench, with your configuration and your traffic profile, and write down every step, every KPI and every threshold as a test plan

    Test plan and the baseline run

  2. 2

    Scripts

    Weeks 2–3

    Each test case becomes a script against the bench generators or our own traffic generator, with the device configured through its API or console. Results land in one store with the run metadata.

    Suite that runs end to end from one command

  3. 3

    Pipeline

    Week 3

    Your CI job calls the suite over an API or a CLI, passes the build, waits for the result and shows it next to the unit tests. Nightly runs are scheduled on the bench with the same entry point.

    First unattended run, report in your pipeline

  4. 4

    Support

    Ongoing

    New features get new cases, thresholds are revised when the baseline changes, the bench is kept patched and calibrated. Or we hand the suite over and your team runs it in your lab.

    Monthly plan or a handover with documentation

What stays yours

The scripts, the history and the right to run them without us

Every script, configuration file and threshold table is delivered to your repository with documentation, written so that an engineer who has not seen the bench can run the suite. The run history is exported in full at any time. If you move the suite to your own lab, it runs on the generators you have or on our own traffic generator with a licence, and the pipeline call stays the same.

Every script
in your repository, documented
Every run
stored with its configuration and evidence
1 entry point
the same call in CloudLAB and in your lab

The run card shows what your pipeline gets back from one nightly run. Values are examples.

Run 214 Nightly, build 7.2.1 Fail

Performance set and 38 functional items on the core router build

Scope
Throughput and latency at 7 frame sizes on 8 ports, convergence with the full table, 38 checklist items
Result
37 items pass, 1 fails. Throughput at 94% of the baseline, below the 3% tolerance
What changed
Latency p99 at 70% load up 18% against build 7.2.0; 1518-byte throughput down 6%
Evidence
Generator statistics per run, device counters before and after, capture of the failed item, console log
Time
2 h 40 min on the bench, unattended, result in the pipeline at 06:10

Questions before the first call

What engineering teams ask first

Test plan and methodology design CloudLAB plans Our own traffic generator Methodologies we script

Whose tools does the suite run on?

On the generators of the CloudLAB bench, on our own traffic generator, or on the generators you already have in your lab. The scripts are written against the tool you will run them on. If a case needs a tool you do not have, the plan says so and that case runs in CloudLAB.

Whose code is it afterwards?

Yours. Scripts, configuration files, threshold tables and documentation are delivered to your repository and you may change and run them without us. The scripts are not tied to one generator; if the suite is to run on our own traffic generator in your lab, that part is agreed separately.

Can it run in our lab instead of CloudLAB?

Yes. The suite is built in CloudLAB and moved to your lab at handover, or built in your lab from the start if the generators are there. A load module on your site that connects out to CloudLAB is the third option when the device must not leave your building.

How long until the first unattended run?

From 3 weeks for a suite of one benchmark family and a checklist of about 40 items on one device model. Several families, several models or a device without an API add a week or two each. The manual run in week 1 is a result on its own: a test plan and a baseline.

How is the pass threshold set?

From the baseline run and your SLA. Each KPI gets a tolerance agreed with your team: a few percent for throughput, a fixed limit in milliseconds for convergence, zero for a functional item. The table is part of the deliverable and changes only with a logged decision.

What if the device has no API?

The scripts drive the console or the web interface the way an engineer would, and the configuration steps are recorded as part of the case. It is slower per run and more fragile on interface changes, so the plan says which cases depend on it.

Send the list you run before a release and the device model. The plan for the suite comes back in 3 business days

The first week is a manual run with a written plan and a baseline, useful on its own even if the suite never follows.

Discuss a suite All services

Send the shortlist

You have the method. Now get the numbers.

Name the models and the KPIs that matter. An engineer replies within one business day with a test plan draft and a quote.

Prefer to chat? Message an engineer on WhatsApp

We use what you send only to answer you. See our Privacy Policy.