Post 3 of 6: WAF, IPS, and DLP — Why Content Inspection Testing Is Harder Than It Looks
Alexander Zemskov
This is part three of a six-part series on evaluating network security products. Part 1 covered the structural datasheet gap. Part 2 covered NGFW-specific testing traps.
WAF, IPS, and DLP platforms are frequently purchased on the strength of vendor demos and detection rate claims. They are also among the most frequently disappointing in production — not because the technology does not work, but because the testing methodology used to select them did not reflect the conditions they would operate under.
WAF: What "Requests Per Second" Does Not Tell You
Web Application Firewall performance figures — requests per second, throughput, latency — are almost universally measured under conditions that have little to do with production web application traffic.
The synthetic request problem. Most WAF test setups use a fixed set of HTTP/HTTPS requests, either synthetic or drawn from a limited PCAP capture. Every request follows the same structure: same headers, same cookie format, same URI pattern. Real web application traffic is the opposite: dynamic session tokens on every request, variable content lengths, authentication flows that span multiple requests with state tracked across them, redirects, varying TLS session characteristics. A WAF rule engine that processes a uniform synthetic request stream has a fundamentally easier job than one processing the actual output of a modern web application.
Attack isolation vs. attack-in-traffic. WAF detection accuracy figures are nearly always measured by sending attack payloads in isolation: a clean test session with one malicious request, response recorded, detection confirmed. What this does not test is whether the WAF maintains the same detection accuracy when attack payloads are embedded within a high-volume stream of legitimate application requests. Under load, with the rule engine processing thousands of concurrent sessions, detection accuracy for subtle injection attempts and low-signal evasions can degrade. Test both: isolated attack detection, and attack detection embedded at realistic rates within a background traffic load.
TLS inspection under concurrency. Many WAF deployments include TLS termination or full man-in-the-middle inspection. Vendors frequently report TLS and non-TLS performance separately and present the better number. The performance impact of TLS termination, certificate validation, session resumption handling, and re-origination at high concurrency is where the production number lives. Test with your realistic TLS connection rate, your typical session duration, and with TLS inspection active.
Rule set complexity matters. A WAF with the baseline rule set and a WAF with the baseline rules plus custom virtual patching for a dozen applications plus exception rules accumulated over two years of operation are running different workloads. Performance testing with a minimal rule set does not predict behavior under your production policy.
IPS: The Evasion Gap That Detection Rates Hide
Intrusion Prevention Systems are evaluated almost entirely on detection rate. This is a metric that is easy to measure, easy to present, and largely disconnected from the protection that matters.
Detection rate is a circular measurement when the test tool is known. An IPS that detects 99% of attacks drawn from a library the vendor prepared against will produce a 99% detection rate in a test using that library. The measurement tells you the device recognizes attacks it was trained to recognize. This is useful, but incomplete. The more important question is: what happens to attacks that use evasion techniques the vendor did not specifically train for?
Evasion coverage is where products diverge. Evasion techniques — payload fragmentation, protocol ambiguity exploitation, encoding variations, polymorphic shellcode, header manipulation — are how real attackers bypass signature-based detection. An IPS that blocks 99% of direct attacks but fails against 30% of evasion-wrapped variants of those same attacks provides substantially less protection than its headline number implies.
The only meaningful way to test evasion coverage is with an attack emulation platform that generates attacks dynamically with controlled variation in encoding, fragmentation strategy, and protocol handling — not a static replay of known payloads. Static PCAP-based attack testing gives the vendor advance knowledge of every payload. That is not a security test; it is a rehearsal.
Detection under load. At 20% utilization, an IPS engine can afford to be thorough. At 80% utilization under sustained attack alongside heavy legitimate traffic, the same engine may start making triage decisions: dropping packets, reducing inspection depth, or deferring classification. Testing IPS detection rate at idle tells you the ceiling. Testing it at 70–80% of rated throughput tells you what your users will experience under the conditions where an attacker is most likely to operate.
DLP: The Most Consistently Oversold Category in Security
Data Loss Prevention is where the gap between vendor claims and production reality tends to be widest, and where the consequences of a failed evaluation are most operationally costly.
The false positive problem at scale. DLP accuracy figures from vendor testing are measured against clean test datasets: purpose-built documents with clearly structured sensitive data, or small controlled corpora. False positive rates in this environment look acceptable — 1–3% is common in vendor materials.
Production environments do not have clean, controlled document corpora. They have years of accumulated files in dozens of formats, with complex embedded content, legacy templates that match data patterns without containing actual sensitive data, and business-specific terminology that overlaps with sensitive data definitions. Against a realistic enterprise content sample, false positive rates that appeared manageable in testing can reach levels that generate hundreds of incidents per day — making the system operationally unusable without months of tuning.
Ask vendors to run their DLP detection against a sample of your actual document repository during evaluation. The difference between controlled-test accuracy and production-realistic accuracy is the number that matters.
Throughput under real content inspection. DLP requires application-layer content analysis of file payloads — not packet inspection. This is computationally expensive, and the cost varies enormously by file type. A DLP engine processing a stream of plain text emails operates at much lower overhead than one processing a stream of large Excel workbooks with embedded macros, nested ZIP archives, or heavily formatted PDFs.
Vendor throughput figures are almost always measured against lightweight content. Test with a realistic file type distribution representative of what your organization actually moves across the network. The throughput number may look very different.
Protocol coverage: what "DLP" actually covers. Enterprise data exfiltration does not confine itself to SMTP and HTTP uploads. Modern environments involve Teams, Slack, Zoom file transfers, cloud sync clients (OneDrive, Google Drive, Dropbox), custom SaaS upload flows, and a long tail of proprietary protocols. Ask the vendor to enumerate which protocols receive full content inspection versus metadata-only monitoring versus no coverage. The distinction matters significantly for what the platform actually protects against.
Shared Principle: Test the Thing You Are Actually Buying
Across WAF, IPS, and DLP, the same principle applies: the test must reflect the conditions the product will operate under in your environment.
That means:
- The traffic or content used in testing must resemble your actual application traffic, your actual document corpus, your actual attack surface
- Security function testing must occur under realistic load, not at idle
- The feature configuration tested must match what will run in production — not a minimal default that produces favorable benchmark numbers
- Results must be granular enough to understand what drove them — not a single aggregate figure
None of this requires unusual methodology. It requires asking clearly what conditions produced each number in a vendor's test results — and independently verifying the answer.
Next in this series — Post 4: SASE — why cloud-delivered security is uniquely difficult to benchmark, and what a meaningful evaluation looks like for distributed architectures.
TEST4NET LLC is an independent network and security testing laboratory. We evaluate WAF, IPS, and DLP platforms using methodology designed to reflect production conditions, not vendor-optimized test scenarios. Contact us.
