How we test backup and replication over WAN: the window and RPO

The sizing sheet divides the data by the link: 20 TB over a 10G WAN is 4.4 hours, the window from 22:00 to 07:00 is 9, so the margin looks double. Over 30 ms of RTT with real provider loss, the backup product opens a few streams on its defaults, each held back by its TCP window, and in a typical project the same 20 TB took 13 hours. We measure backup and replication over your WAN profile: whether the job ends before the morning, how many minutes the replica lags, and how long a restore takes.

The bench

Your WAN goes onto the bench as a profile: delay, jitter, loss and a bandwidth cap with a queue, per direction, from the library or replayed from measurements of your own link. Our own emulator, WANemu, in beta on the 1G and 10G links of the bench, applies it, and the report lists the applied and the measured values side by side.

On one side sit the source hosts with a sample set built to your profile: file sizes, daily change rate, compressibility. The backup or replication product and its agents run on the bench. The target is a repository in the rack, software-defined storage on bench hosts, or your system over a dedicated link. Generator ports put the application mix on the same WAN link. The sample set is timed and the result extrapolated to your 10 to 200 TB.

What we check

• Throughput per stream at your RTT and loss, and the stream count where one more stops helping
• Full and incremental jobs, timed to the last byte committed at the target, against your deadline
• Dedup and compression: bytes on the wire against bytes protected for your data, and the proxy CPU
• Replication lag in minutes at the night and the peak change rate, and the time to drain it: your real RPO
• A job started at peak: its throughput, and the latency and loss the users get meanwhile
• WAN failover mid-job: whether the job resumes, retries from a checkpoint or starts over
• Restore of a sample set over the WAN, timed to the first usable file and to the last, extrapolated to your recovery set

The storage page lists backup over a WAN among 5 device classes. Why one TCP stream stalls over a long delay is in How we test TCP throughput; the network between the hosts and the array has its own post.

What usually breaks

Too few streams. In the project from the first paragraph the product opened 4 streams, a default chosen for a LAN. Throughput grew with the stream count up to 8 and flattened: the dedup proxy sat at 100 % CPU. With 2 more vCPUs it kept growing to 12 streams, and the same 20 TB crossed in 8.6 hours, inside the window. The link stayed the same.

An RPO that holds only at night. Asynchronous replication runs 1 to 2 minutes behind while the users sleep. At the morning peak the application writes more and the same link carries the users, and in a typical project the lag reached 35 minutes against an RPO of 15, then took 3 hours to drain. A dashboard with the daily average lag showed a healthy number all along.

A failover the replication does not survive. The primary WAN link goes down, and routing moves the traffic to the second one in 40 seconds. The replication session has a 30-second timeout: it drops and starts a full resync of a 2 TB volume over the slower link. The backup job on the same failover resumes from its checkpoint. Both are product settings, and the bench shows which one you have before a real outage does.

What is in the report

The window per job type against your deadline, with the throughput curve and the stream count. Throughput per stream count for each WAN profile. Bytes on the wire against bytes protected, and the proxy CPU. Replication lag over the day: the peak, the drain time, your RPO line. Restore time of the sample set, extrapolated to the full set. Every failover with the gap and what the job did. For every run: the WAN profile applied and measured, the product settings, host counters. After a product upgrade or a WAN change, a rerun and the diff against this report take 3 business days.

Where to start

Send us the data set size, the WAN between the sites and the window the job has to fit. We answer with a draft test plan: the items, the bench layout and the dates. The report is yours to keep and to show to the vendor. Write to us or look at what else the bench does.