Post 1 of 6: The Security Datasheet Lie — And Why It Is Structural
Alexander Zemskov
Every network security vendor publishes performance numbers that look compelling in a slide deck. The demo goes smoothly. The PoC delivers great results. Then the hardware lands in production — and within weeks, the operations team is raising tickets about latency spikes, throughput degradation, and security policies getting quietly bypassed under load.
This is not an accident. It is not unique to one vendor. It is a structural feature of how the entire security industry markets and tests its products — and understanding it is the first step toward making a procurement decision you will not regret.
Why Every Datasheet Number Is Technically True and Practically Misleading
Vendors do not fabricate the numbers in their datasheets. The device genuinely achieved that throughput — at that specific moment, under those specific conditions. The problem is that the conditions used to produce the best defensible number are chosen by the vendor, and they rarely resemble what happens in a real enterprise network.
Every security platform has a performance envelope. At one end: basic packet forwarding, no inspection, cleartext traffic, large steady-state sessions. At the other end: full security stack engaged — TLS inspection, DPI, IPS, AV, application control, URL filtering — under a realistic mixed enterprise traffic profile with short-lived sessions, encrypted traffic, and bursty connection rates.
Datasheets lead with the first number. They bury the second number in footnotes, if they include it at all.
The gap between the two ends of that envelope runs at 55–75% across every major vendor in the NGFW space, and similar ranges apply to WAF, IPS, and inline DLP platforms. That is not a bug. That is the cost of security processing. The issue is not the gap itself — it is the absence of honest disclosure.
The Features-Enabled Gap in Practice
Here is how this plays out with real product categories:
NGFW. A flagship appliance advertises its throughput number prominently. Enable the enterprise threat prevention stack — IPS, AV, application control, URL filtering, SSL inspection — and you will measure a fraction of that figure in production. The drop is not a flaw in the hardware; it reflects what deep packet inspection at line rate actually costs. The problem is that the datasheet number was never measured with those features on.
WAF. A WAF vendor publishes requests-per-second and throughput figures. Those numbers were typically measured with a uniform synthetic request profile, TLS offloaded or excluded, and a minimal rule set. Add your full CRS ruleset at production paranoia level, enable TLS inspection, and test with realistic application traffic diversity — the numbers look different.
DLP. Deep content inspection — reading and analyzing the payload of every file, every email attachment, every upload — is expensive. DLP throughput figures are almost always measured against simple, lightweight file types. A full inspection pass against a realistic enterprise document corpus including large Office files, PDFs, and compressed archives produces throughput that can be 20–30% of the datasheet figure.
SASE. A cloud-delivered security stack introduces a different kind of gap: the vendor's published latency and throughput figures were measured from their lab, to their infrastructure, at their chosen time. Your users are distributed across multiple geographies. Your applications are split between cloud and on-premises. Your SASE traffic transits shared PoP infrastructure. The number on the datasheet was not measured under those conditions.
The Configuration Variable Nobody Talks About
Beyond the feature set, there is a second variable that vendors consistently under-disclose: policy and rule complexity.
A firewall with 50 rules and a firewall with 5,000 rules are not the same device from a performance perspective. Policy lookup, rule matching, logging volume — all of these scale with configuration complexity. Vendors test with minimal or default configurations. Production environments accumulate rules over years of operation.
A WAF with the default CRS rule set and a WAF with the default CRS plus custom virtual patching policies plus application-specific rule exceptions operates at a different performance point. A DLP engine with 10 active policies and one with 200 fine-tuned policies for different data classifications behaves differently under load.
Testing in a configuration that does not reflect your production environment produces numbers that do not predict your production performance. This sounds obvious. It is almost universally ignored in vendor-led evaluations.
What This Series Covers
This post is the first in a five-part series on how to evaluate network security products with methodology that predicts production behavior.
The remaining posts cover:
- Post 2: NGFW-specific testing traps — fast paths, vendor-optimized test modes, and why the AppMix your test tool reports is not the AppMix that transited your device
- Post 3: WAF, IPS, and DLP — content inspection complexity, evasion testing, and the false positive problem
- Post 4: SASE — why cloud-delivered security is uniquely difficult to benchmark and what a meaningful evaluation looks like
- Post 5: How to structure a test that produces results you can trust — methodology, tooling, and the role of an independent laboratory
- Post 6: One test is not enough — why security platforms require continuous validation across their operational lifecycle, and how to build that process
The goal is not to indict any specific vendor. The goal is to give procurement teams, security architects, and network engineers the framework to ask the right questions — before signing a multi-year contract.
TEST4NET LLC is an independent network and security testing laboratory. We have no commercial relationship with any vendor whose products we evaluate. If you are planning a security product procurement and want methodology that produces defensible, production-representative results — contact TEST4NET.
