Skip to content

Methodology · TCP throughput

RFC 6349: the path measured the way a file transfer sees it

RFC 2544 and Y.1564 push frames at a set rate and count what comes back. RFC 6349 opens real TCP connections, so the result depends on what TCP depends on: path MTU, round-trip time, window size, loss. Three steps: find the MTU, measure baseline RTT and bottleneck bandwidth, then run TCP with a window sized to the bandwidth-delay product, 1 to 16 connections, 30 seconds per run, each direction. Three metrics come out: TCP Transfer Time Ratio, TCP Efficiency and Buffer Delay Percentage. This page shows the method as we run it in CloudLAB and on live circuits, and what the report contains.

  • Window from the bandwidth-delay product: 1 Gbps at 20 ms RTT needs 2.5 MB in flight, and a 64 KB default window gives 26 Mbps
  • Three metrics per run: Transfer Time Ratio against the ideal, Efficiency as the share of bytes sent once, Buffer Delay as RTT growth under load
  • Report with the numbers per direction and connection count, the MTU and RTT they were taken at, and the capture behind each row
Request a test plan See the runs and metrics
TCP THROUGHPUT vs RTT 1 GbE · Mbps · RFC 6349 §3.3 64 KB WINDOW · HOST DEFAULT WINDOW = BANDWIDTH × RTT · RFC 6349 949 Mbps · MAX AT 1500 B MTU 50% 524 105 52 26 10 5 WINDOW ÷ RTT: 64 KB ÷ 20 ms = 26 Mbps 1 ms 5 ms 10 ms 20 ms 50 ms 100 ms The link is 1 Gbps. The transfer gets window ÷ RTT

One link, two windows. With the window set to the bandwidth-delay product the path carries 949 Mbps at any RTT. With a 64 KB default window the same path gives 524 Mbps at 1 ms and 26 Mbps at 20 ms. The circuit passed its acceptance test both times.

Where it fits

Layer 4 on top of the frame benchmarks

RFC 2544 and RFC 2889 benchmark a device with frames at a fixed rate. Y.1564 checks a service against its SLA. RFC 6349 answers the question that reaches the help desk: the link is fine, why is the transfer slow. It measures the end-to-end path with real TCP, under the RTT, window and loss that path has

Circuit acceptance, second pass

After Y.1564 passes, an RFC 6349 run in both directions shows what a server will get over the circuit at its real RTT. The CIR is the carrier's number; TCP throughput is the customer's.

Slow transfer investigation

Backups, replication, file sync that run at a fraction of the link. The three metrics separate the causes: a host window too small, loss on the path, or a queue in front of the bottleneck.

Devices that touch TCP

Firewalls, WAN optimizers, shapers, SD-WAN edges and proxies change what TCP sees. The device goes between the test hosts with an emulated RTT, and the metrics are taken with and without it.

Test bed

Two TCP hosts and the path between them

RFC 6349 needs a TCP sender and receiver that can set the window, log retransmissions and timestamp segments. In CloudLAB the hosts are ports of a traffic generator and analyser on the bench; on a live circuit, two test hosts at the customer's ends. The path is the device under test, the circuit, or a WAN emulator set to the RTT and loss of the target network

TCP TEST HOST · CLIENT SIDE GENERATOR PORT WITH A TCP STACK, OR A SERVER TCP SENDER, PER CONNECTION Window ≥ BDP, window scaling on MSS from the path MTU, no fragmentation 1, 4, 8 or 16 connections, same window total Bytes, retransmits, RTT per segment logged 30 s PER RUN · 3 RUNS · EACH DIRECTION 1G · 10G · 100G SEGMENTS + ACKS NETWORK UNDER TEST WHAT TCP SEES BOTTLENECK QUEUES DELAY 1G · 10G · 100G SHAPER · POLICER RTT 1 TO 300 ms DUT · CUSTOMER CIRCUIT · WAN EMULATOR loss 0 to 1% and jitter as agreed, logged per run BYTES IN FLIGHT NEEDED = BANDWIDTH × RTT 1500 OR 9000 B MTU WINDOW SCALING SACK ON IPv4 · IPv6 CUBIC OR BBR AS AGREED OFFLOADS STATED IN THE REPORT 1G · 10G · 100G COUNTED · TIMED TCP TEST HOST · SERVER SIDE SAME INSTRUMENT OR A SECOND HOST · SYNCED CLOCK TCP RECEIVER, PER CONNECTION Receive window ≥ BDP, advertised in every ACK Bytes in order, out of order and duplicated RTT from timestamps: baseline vs under load Capture kept per run, attached to the report SENT − RECEIVED = RETRANSMITTED HOSTS VERIFIED FIRST back to back the hosts fill the link rate with 0 retransmissions before the path goes in · a slow host would report its own limit as the network's

TCP test hosts

Hardware TCP stacks on the generator up to 100 GbE, software hosts with a tuned stack up to 10 GbE. Window scaling, SACK and offloads are set on purpose and listed in the report. The hosts are verified back to back before the path goes in.

Network under test

A device on the bench, a customer circuit between two sites, or the bench WAN emulator with RTT from 1 to 300 ms and loss from 0 to 1%. Link rates 1G, 10G and 100G; 25G and 40G on request.

What gets measured

Per connection and per run: bytes sent, bytes retransmitted, RTT per segment, bytes received in order. From these: throughput in Mbps, Transfer Time Ratio, Efficiency and Buffer Delay, in each direction and for each connection count.

The host must not be the bottleneck

A laptop with a 64 KB default window reports its own limit and calls it the network. Before the first run the two hosts are connected back to back and must fill the link rate with zero retransmissions. Only then does the path go in between.

Test conditions

Parameters the result depends on

RFC 6349 fixes the order: MTU first, then RTT and bottleneck bandwidth, then TCP with a window derived from them. Each value is written into the plan, measured, and printed next to the throughput it produced

Parameter RFC 6349 Typical range Why it matters
Path MTUFound by RFC 4821 probing, per direction1500 B; 9000 B on jumbo pathsThe MSS follows it. A black hole or fragmentation shows as low throughput with no loss
Baseline RTTMeasured on the idle path, before any load1 to 300 msSets the BDP and the window, and is the reference for Buffer Delay
Bottleneck bandwidthLowest link or shaper rate, verified with a layer 2/3 run10 Mbps to 100 Gbps; 1G and 10G most oftenSets the BDP and the maximum achievable TCP throughput
BDPBottleneck bandwidth × baseline RTT, in bytes25 KB at 10 Mbps, 2.5 MB at 1 Gbps, 250 MB at 100 Gbps, all at 20 msThe minimum bytes in flight to fill the link
TCP windowAt least the BDP, window scaling on64 KB to 256 MB, split across connectionsA window below the BDP caps throughput at window ÷ RTT, whatever the link
Connections1 first, then several whose windows add up to the BDP1, 4, 8, 16One shows a single flow and the host; several show the path
Run durationLong enough to pass slow start and fill buffers, repeated30 s, three runs; 60 s above 100 ms RTTSlow start and loss recovery must not dominate the average
DirectionEach direction alone, then both at onceUp, down, bothAsymmetric shapers and half-duplex faults show in one direction only

BDP = bottleneck bandwidth × RTT. A window below the BDP caps throughput at window ÷ RTT; a window far above it fills queues and shows as Buffer Delay. Maximum achievable TCP throughput at 1500 B MTU: 949 Mbps on 1 GbE, 9.49 Gbps on 10 GbE, 94.9 Gbps on 100 GbE

Always in our runs

  • Path MTU probed per RFC 4821, never assumed (§3.1)
  • Baseline RTT on the idle path, before load (§3.2)
  • Bottleneck bandwidth verified at layer 2/3 first (§3.2)
  • Window ≥ BDP, window scaling checked in the SYN (§3.3)
  • Each direction alone, then both at once (§3.3)
  • Hosts verified back to back before the path goes in

Window scaling checked in the SYN, MTU probed and never assumed, RTT taken before load, bottleneck verified at layer 2/3 first, each direction alone before both at once. This is what the RFC requires, and what separates a TCP number from a host artefact.

Agreed per project

  • Connection counts beyond 1: 4, 8, 16, our default steps; multi-connection per §5.1
  • Emulated RTT 1 to 300 ms, loss 0 to 1%
  • Congestion control: cubic, BBR, vendor stack
  • Runs of 60 s and more on long paths
  • IPv6, jumbo frames to 9000 B
  • A device between the hosts, with and without

Connection counts, emulated RTT and loss profiles, congestion control, IPv6 and jumbo frames are set in the plan. Defaults: 1 and 8 connections, cubic, IPv4, 1500 B, three 30-second runs per point.

Runs and metrics

Three steps, three metrics

The method is sequential: each step feeds the next. The metrics of section 4 are computed from the TCP runs and reported together, since one of them alone does not say what is wrong

  1. 3.1Preparation

    Path MTU discovery

    Packetization layer probing per RFC 4821: segments of growing size with DF set, until the largest that passes is found. ICMP is not trusted, because firewalls drop it. The MSS for every later run follows this value.

    Output: path MTU and MSS, in each direction
  2. 3.2Preparation

    Baseline RTT and bottleneck bandwidth

    RTT is sampled on the idle path for at least a minute: minimum, average, maximum. The bottleneck bandwidth is verified with a layer 2/3 throughput run at the MTU found, zero loss required. Together they give the BDP.

    Output: baseline RTT, bottleneck bandwidth, BDP in bytes
  3. 3.3TCP throughput

    Single connection, window at the BDP

    One TCP connection with the window set to the BDP, 30 seconds, three runs. Shows what one flow can get: a single backup stream, a database replica. A gap to the maximum achievable throughput here is the first thing to explain.

    Output: throughput, retransmitted bytes, RTT under load, per run
  4. 5.1TCP throughput

    Multiple connections, same total window

    4, 8 or 16 connections whose windows add up to the BDP. Applications that open several flows, and hosts whose per-flow limit is lower than the path, get a different number here. The spread between 1 and N is itself a result.

    Output: aggregate throughput per connection count
  5. 3.3TCP throughput

    Both directions at once

    Each direction is run alone first, then both together. Asymmetric shapers, half-duplex faults and ACK compression on the return path show up as a drop in one direction only when the other is loaded.

    Output: upstream, downstream and bidirectional throughput
  6. 4.1Metric

    TCP Transfer Time Ratio

    Actual transfer time divided by the ideal, where the ideal is the bytes moved at the maximum achievable TCP throughput for the link. 1.0 is perfect. 3.2 means a file took three times longer than the link allows.

    Example: 1 GB in 27 s on 1 GbE, ideal 8.4 s, ratio 3.2
  7. 4.2Metric

    TCP Efficiency

    Bytes transmitted minus bytes retransmitted, divided by bytes transmitted, in percent. 100% means no loss on the path. 99.6% looks small and is enough to cut throughput by two thirds at 20 ms RTT.

    Example: 1.00 GB sent, 4 MB retransmitted, efficiency 99.6%
  8. 4.3Metric

    Buffer Delay Percentage

    Average RTT during the transfer against the baseline RTT, as a percentage increase. It tells whether the path queues: a shaper, a policer with a deep buffer, or a router with too much memory in front of the bottleneck.

    Example: baseline 20 ms, 37 ms under load, buffer delay 85%

Procedure

How a run goes, from the path to the report

Half a day per path for both directions at two connection counts, three runs each. Longer on RTT above 100 ms, where each run is 60 seconds and TCP needs more of them to settle

  1. 00

    Agree the path and the target

    Where the hosts sit, link rate, expected RTT, which directions and connection counts, and the number to compare against: the SLA, the datasheet, or what the application needs

  2. 01

    Verify the bed

    Hosts back to back at the link rate with zero retransmissions, window scaling and SACK negotiated, offload settings recorded, capture on both ends, clocks synced

  3. 02

    MTU, RTT, bottleneck

    RFC 4821 probing for the MTU, a minute of RTT samples on the idle path, a layer 2/3 throughput run at the MTU found. The BDP and the windows for every later run come from these.

  4. 03

    TCP runs

    Window at the BDP, 1 connection then N, each direction then both. 30 seconds per run, three runs per point, spread checked before the point is accepted.

  5. 04

    Metrics and report

    Transfer Time Ratio, Efficiency and Buffer Delay per point, next to the throughput, the MTU and the RTT they were taken at. The captures stay attached.

READING THE THREE METRICS TOGETHER · RFC 6349 §5.2 1 GbE · RTT 20 ms · BDP 2.5 MB · 1500 B MTU CLEAN PATH throughput 905 Mbps of 949 TRANSFER TIME RATIO TCP EFFICIENCY BUFFER DELAY 1.05 100% 4% THE LINK IS THE LIMIT · ACCEPT LOSS ON THE PATH throughput 297 Mbps of 949 TRANSFER TIME RATIO TCP EFFICIENCY BUFFER DELAY 3.2 99.6% 8% 0.4% RETRANSMITTED · FIND THE HOP BUFFERS FILLING throughput 678 Mbps of 949 TRANSFER TIME RATIO TCP EFFICIENCY BUFFER DELAY 1.4 100% 85% RTT 20 → 37 ms · QUEUE AT THE SHAPER HOST LIMIT throughput 26 Mbps of 949 TRANSFER TIME RATIO TCP EFFICIENCY BUFFER DELAY 36 100% 0% 64 KB WINDOW · FIX THE HOST, RERUN EFFICIENCY BELOW 100% = LOSS SOMEWHERE ON THE PATH HIGH BUFFER DELAY WITHOUT LOSS = QUEUE AT THE BOTTLENECK RATIO NEAR 1, EFFICIENCY 100%, LOW DELAY = ACCEPT

The same link, four readings. Efficiency below 100% means loss somewhere on the path; a high Buffer Delay with no loss means a queue in front of the bottleneck; a Transfer Time Ratio in the tens with clean metrics means the host window. One metric alone cannot tell these apart.

4 checks a point must pass before it goes into the report

  • ≥ BDP

    Total window across connections, read from the capture, with window scaling negotiated. A run with a smaller window measures the host and is rerun.

  • 0

    Frames lost in the layer 2/3 run at the bottleneck bandwidth before any TCP run. Loss at that stage belongs to the device or the circuit, and TCP would only blur it.

  • 3 × 30 s

    Runs per point. Three results within 5% of each other make a point; a wider spread is itself reported, with the runs listed.

  • 60 s

    Run length on paths with RTT above 100 ms, where slow start and recovery take a larger share of a 30-second run. The plan says which length was used.

Anatomy of one run

  • Setup, secondsWindow and MSS set per the plan, capture started on both hosts, connection count and direction logged
  • Slow start, first RTTsCounted. RFC 6349 measures the whole transfer, so a path with a long RTT pays for its ramp in the Transfer Time Ratio
  • Run, 30 sBytes sent and retransmitted, RTT per segment, bytes received; throughput averaged over the run, never the peak
  • Close and repeatConnections closed, counters read, next run; three per point, then the next connection count or direction

Report

What the report contains

One table per direction with the throughput and the three metrics per connection count, the MTU and RTT they were taken at, and a capture behind each row. Written so the carrier, the vendor and the application team read the same numbers

Measured values

  • TCP throughput per direction and per connection count, in Mbps and as a share of the maximum achievable
  • TCP Transfer Time Ratio per point
  • TCP Efficiency per point, with the retransmitted byte count
  • Buffer Delay Percentage per point, with the baseline and the loaded RTT
  • Path MTU and the MSS used, per direction
  • Bottleneck bandwidth from the layer 2/3 run, with its loss count

Context, so the run can be repeated

  • Test hosts: type, TCP stack, window scaling, SACK, congestion control, offloads
  • Window per connection and in total, for every run
  • Run length, repeats and the spread between runs
  • Path description: endpoints, hops, shaper or policer settings, WAN emulator profile
  • Date, firmware and configuration exports of the device under test
  • Capture excerpts: retransmission timeline and RTT per segment for the worst run
RFC 6349 · TEST REPORT 1 GbE · RTT 20 ms · 1500 B · 30 s RUNS 3.1Path MTU, RFC 4821 probing 3.2Baseline RTT, idle path 3.2Bottleneck bandwidth, layer 2/3 3.3TCP throughput, 1 conn, down 3.3TCP throughput, 1 conn, up 5.1TCP throughput, 8 conn, down 4.1Transfer Time Ratio, up 4.3Buffer Delay, down 1500 B 20.3 ms 949 Mbps 912 Mbps 297 Mbps 941 Mbps 3.2 4% no black hole in either direction · MSS 1460 B min 20.1 · max 20.9 ms over 60 samples zero-loss run at 1500 B · shaper confirmed at 1 Gbps window 2.5 MB · 3 runs within 2% efficiency 99.6% · retransmissions start above 300 Mbps 8 × 320 KB · same total window as the single run 1 GB in 27 s against 8.4 s ideal · upstream policer RTT 21.1 ms under load against 20.3 baseline ATTACHED CONFIG EXPORT PCAP PER RUN RTT GRAPH PER-RUN TABLE Values are illustrative. Your report carries your numbers, per direction and per connection count.

What we add on top

  • The gap to the target: SLA, datasheet or the application's need, in Mbps and in Transfer Time Ratio
  • The diagnosis in one line per failed point: host window, loss on the path, or queueing, with the evidence
  • Settings for the hosts that would close the gap, when the host is the limit

From practice

A 1 Gbps circuit that moved backups at 28 Mbps

A typical case. An international 1 Gbps circuit passes Y.1564: CIR met, zero loss, jitter under 2 ms. The nightly backup over it runs at 28 Mbps, and the ticket travels between the carrier and the IT team for weeks. RFC 6349 on the path: MTU 1500, baseline RTT 92 ms, bottleneck 1 Gbps, so the BDP is 11.5 MB. The backup server's window was capped at 320 KB. 320 KB over 92 ms is 28 Mbps, within a megabit of the backup log.

With the window at the BDP and 8 connections the same circuit carried 905 Mbps, with 100% TCP Efficiency in every run. The circuit was never the problem. The fix was two lines in the server's TCP settings, and the carrier escalation closed the same day.

Throughput belongs to the path and the window together. Measure both.

SAME CIRCUIT · 1 Gbps · RTT 92 ms · TWO WINDOWS 320 KB WINDOW · THE BACKUP SERVER AS SHIPPED 28 Mbps of 949 · 320 KB ÷ 92 ms, within a megabit of the backup log 3% 11.5 MB WINDOW · 8 CONNECTIONS · RFC 6349 905 Mbps of 949 · window at the bandwidth-delay product 95% TCP EFFICIENCY, BOTH RUNS 100% Zero retransmissions in six runs. The circuit was never the problem.

What we run

RFC 6349 in CloudLAB and on your circuit

Four ways the method is used. Each ends with the same report: the three metrics per point, the captures, and the diagnosis where a point fails

Circuit acceptance with TCP

After the Y.1564 run, an RFC 6349 pass in both directions at the real RTT. Hosts on both ends are ours, so a host limit is ruled out before the first number.

Request a test plan

Slow transfer investigation

We reproduce the transfer on the path, or in the lab with the WAN emulator set to your RTT, and come back with which of the three causes it is and the evidence

Describe the transfer

A device under TCP load

Firewall, optimizer, shaper or SD-WAN edge between the hosts with an emulated RTT from 1 to 300 ms. Metrics with and without the device, per rate and connection count.

Request a test plan

Bench access in CloudLAB

Run RFC 6349 yourself on the bench with the WAN emulator and the generator's TCP stacks, 1G to 100G, with our engineer on the line when needed

Log in to CloudLAB
  • Independent: we sell the test, the carrier sells the circuit and the vendor sells the box
  • Both ends under our control, so a host limit is ruled out before the path is blamed
  • A report the carrier accepts: metrics per RFC 6349 section 4, with the capture behind each row

Read next

FAQ

Questions we get before an RFC 6349 run

How is RFC 6349 different from RFC 2544 and Y.1564?

RFC 2544 and Y.1564 send frames at a set rate and count loss, delay and jitter; they describe the device or the service. RFC 6349 runs real TCP across the path, so the result includes the window, the RTT and the reaction to loss. A circuit can pass Y.1564 and give a server 3% of the link rate. RFC 6349 shows that number and why.

Can it run on a live production circuit?

Yes. A run loads the path to its bottleneck for 30 seconds at a time, like a large file transfer. We schedule it in a maintenance window or at a low-traffic hour, run each direction alone and then both, and pause between runs. The test hosts sit at the two ends, or at one end and in the lab.

Which test hosts do you use, is iperf enough?

Up to 10 GbE a tuned software host with iperf3-class tools is fine, as long as its window, offloads and stack settings are set on purpose and recorded. Above that, and for repeatable RTT profiles from 1 to 300 ms, we use the TCP stacks of the traffic generator and analyser and the WAN emulator on the CloudLAB bench. The host type and its settings are in every report, and the hosts are verified back to back first.

Why run several connections?

One connection shows what a single flow gets, which is what a backup stream or a replica sees. Several connections with the same total window show what the path carries when the application opens many flows, which browsers, object storage and sync clients do. The gap between the two numbers points at a per-flow limit: a host window, a shaper per flow, or a middlebox.

What RTT do you test at?

The RTT of the real path when the hosts sit at the two ends. On the bench the WAN emulator is set to the RTT of the target network, with loss and jitter profiles when agreed: 20 ms for a national circuit, 90 ms intercontinental, 250 ms for satellite and remote sites. The baseline RTT is measured before every run and printed in the report.

How long does it take, and what do we get?

Half a day of testing per path for both directions at two connection counts, three runs each; a full day when several RTT or loss profiles are agreed. The report with the three metrics per point, the captures and the diagnosis follows within 3 business days. The test plan and the quote come back in 1 to 3 business days from your request.

Measure what the transfer will get

Send the path: the two endpoints, link rate, RTT if known, and what should move over it. You get an RFC 6349 test plan back in 1 to 3 business days: runs, windows, connection counts, lab days and the quote. Test. Compare. Decide.

Request a test plan Log in to CloudLAB

Send the shortlist

You have the method. Now get the numbers.

Name the models and the KPIs that matter. An engineer replies within one business day with a test plan draft and a quote.

Prefer to chat? Message an engineer on WhatsApp

We use what you send only to answer you. See our Privacy Policy.