How we test voice quality: every call scored, every second

A link passes its throughput test with room to spare, and the help desk still logs complaints: a robotic voice, words missing mid-sentence, one side hearing nothing. A throughput figure averages away the 40 ms of jitter and the burst of 6 lost packets that a single call cannot hide. We place calls through the device or link under test and score each one: MOS, setup time, calls that never connected.

The bench

The device under test is whatever the voice crosses: an SBC, the trunk side of an IP-PBX, an NGFW with the SIP ALG on, an SD-WAN edge, a Wi-Fi access point, a mobile core user plane. The bench emulates SIP endpoints on both sides that register, place and answer calls with RTP media in your codec mix, at the call rates of your traffic model. Data flows from our generator's application mix share the queues with voice.

For WAN and radio access paths, WANemu, our own emulator, sits on the access link and adds delay, jitter and burst loss per direction (1G and 10G links in the beta). Each call is scored on the bench: the E-model of ITU-T G.107 turns its measured loss, jitter and delay into an R-factor and a MOS, every second. Scoring against a reference recording is done per project.

What we check

• Call setup: time from INVITE to answer for every call at your busy-hour rate, the median next to p95
• Failed calls: each one with its SIP response code, so a 503 from an overloaded box and a 488 from a codec mismatch are counted apart
• Scale: concurrent calls held with media on all of them, and the calls per second at which setup time starts to climb
• Media quality: MOS for every call, with one-way delay, jitter, loss and loss bursts per direction
• Jitter buffer: jitter raised step by step until the receiving buffer discards late packets and the score drops
• Congestion: the link loaded 3:1 with data, QoS on, each call against the same run on an idle link
• Codecs and DTMF: G.711, G.729 and Opus as examples, transcoding on the SBC, digits in-band and as RFC 4733 events
• NAT and SIP ALG: one-way audio, SDP rewritten with the wrong address, pinholes that close mid-call

In a mobile core the same calls come from emulated subscribers through a loaded user plane. On SD-WAN each call is scored second by second through a brownout. Video has its own metrics: how we test video QoE.

What usually breaks

The SIP ALG. A typical project: an NGFW sits between the IP-PBX and the carrier trunk with the ALG on by default. Calls connect, setup time looks normal, and a share of them (1 in 20 in that run) carry audio in one direction only, because the ALG rewrote the SDP of the first INVITE and missed the re-INVITE. A throughput test places no calls, so nobody saw it.

Averages. Monitoring shows 8 ms of jitter on the access link, a healthy number. Per call, the jitter arrives in spikes of 60 to 80 ms whenever a backup job fills the queue. A 40 ms jitter buffer discards those packets as late, and calls overlapping the backup fall from MOS 4.2 to under 3.5 for those seconds. The fix was a voice queue on the egress, which the design had and the running config did not.

Call rate. An SBC holds its datasheet count of concurrent calls once they are up. At the busy-hour rate of new calls the signalling saturates first: in a typical run setup time grows from about 200 ms to 3 s, then new calls fail with 503 while the established ones sound fine. Plan with calls per second at a setup time your users accept.

What is in the report

MOS per call and per second, with the loss, jitter and delay behind each value. Setup time as a distribution per call rate. Failed calls by SIP response code. The call count and the call rate where the device stopped or slowed. The codec mix, the impairment profile and the device configuration, so the same calls replay on the next firmware. Every MOS is an E-model estimate, and the report says so.

Where to start

Send us the device model, the call flows, the codec list and the busy-hour call rate. We answer with a draft test plan: the items, the bench layout and the dates. The report is yours to keep and to show to the vendor. Write to us or look at what else the bench does.