This is part two of a six-part series on evaluating network security products.
Part 1 covered the structural datasheet gap that applies across the entire security industry.
Next-generation firewalls are the most purchased, most frequently benchmarked, and most poorly evaluated category in enterprise security. The testing traps here are well-known inside the industry — and consistently unknown to the buyers making procurement decisions.
The Fast Path: A Feature That Becomes a Testing Trap
Most enterprise NGFW platforms include custom ASICs or dedicated security processors. These chips can offload specific traffic categories to a hardware fast path that bypasses the software stack entirely. This is a legitimate and valuable architectural feature — under the right conditions, it delivers real performance gains with genuine security inspection.
The problem is the phrase "under the right conditions."
Hardware fast paths activate most effectively when traffic is uniform, predictable, and simple: long-lived sessions with large payloads, steady connection rates, cleartext or easily classified protocols. Under those conditions, the ASIC handles most of the work and the software stack is largely idle. Throughput approaches the theoretical hardware maximum.
Real enterprise traffic is structurally different: short sessions, many small objects, TLS 1.3 with encrypted SNI and ESNI, non-standard ports, proprietary application protocols, bursts of new connection establishment. Under real traffic, fast path coverage drops substantially and the software stack carries most of the load.
A poorly designed test — or a test designed to produce favorable results — generates traffic that looks like the first scenario. The device performs excellently. That performance does not survive contact with the production environment.
Vendor-Specific Test Modes: The Optimization That Nobody Announces
Beyond legitimate fast path behavior, there is a more deliberate category of optimization worth understanding.
Vendors know which traffic generators are commonly used for security product testing. They know the profiles, the patterns, the typical PCAP files used in evaluations. Some platforms implement optimizations specifically triggered by these known profiles: DPI result caching for sessions that are byte-for-byte identical to previously seen flows, parser shortcuts for exact payload patterns that appear in common test sets, fast-path rules keyed to characteristics of specific test traffic generators.
These optimizations produce impressive benchmark numbers. They provide little or no benefit against real diverse traffic — and in some cases introduce behavioral differences that only appear under realistic load.
The practical implication: a device that scores well against a familiar test tool may not score comparably well against a methodology it has not been prepared for. Independent testing with a tool the vendor cannot profile in advance is the only way to see past this.
The AppMix Problem: The Traffic Distribution You Configured Is Not the One That Transited Your Device
This is the most technically subtle problem in NGFW testing, and the most consistently overlooked.
When you configure a test with a target application mix — say 25% HTTPS, 20% HTTP, 15% DNS, 15% SMTP, 10% video, 10% database, 5% custom — you expect that distribution to actually move through the device under test. It does not.
Free and open-source traffic generators schedule new sessions according to configured weights. They do not account for how long each protocol type takes to process on the DUT. This matters because security processing is not uniform across protocol types.
Heavy protocols — TLS with mutual authentication, SMTP with large attachments, protocols requiring extended DPI classification — consume more processing time per session on the DUT. They queue. They slow down. The generator, unaware of this, keeps launching new sessions at the configured rate. Light protocols — plain HTTP, DNS queries, small UDP — process quickly and move through the device at full speed.
The result: over the course of the test, the actual traffic mix transiting the DUT shifts toward the lightest, fastest-processing protocols. Your device appears to handle 10 Gbps with full inspection. It genuinely is handling 10 Gbps — but 60% of it is plain HTTP and DNS, because the SMTP and TLS sessions you configured are stalled in queues or being dropped.
The throughput number is real. The traffic profile it was measured against is not what you intended.
Why this makes cross-vendor comparison meaningless without AppMix control. Device A handles heavy protocols efficiently — it processes TLS and complex DPI at moderate overhead. Under your test, Device A's actual AppMix at the DUT is close to what you configured: roughly the intended distribution. Device B drops heavy-protocol sessions faster — the overhead is too high and connections time out. Device B's actual AppMix is dominated by light traffic. Device B shows higher throughput.
You conclude Device B outperforms Device A. You are measuring them with different rulers and do not know it.
PCAP Replay: The Tool That Tells Vendors Exactly What Is Coming
TRex and many testing tools from China using pcap replay mode to reproduce pre-recorded traffic captures. Every session at L7 is byte-for-byte identical to every other session of the same type. Same headers. Same payloads. Same sequence numbers, same SNI, same URI patterns — repeated indefinitely.
The vendor testing your product knows which tools buyers use. They know the PCAP files. They know the payload patterns. It costs nothing to optimize for them.
Specific optimizations that work against deterministic PCAP replay traffic:
- DPI result caching: If two TCP sessions have identical payloads, classify the second one instantly from cache rather than running full DPI. This produces dramatically higher throughput against replay traffic and near-zero benefit against real diverse sessions.
- Fast-path rules for known-good patterns: A session that matches a known exact pattern from a test PCAP can be fast-pathed after minimal inspection. This does not apply to sessions where payload varies.
- Parser shortcuts: Some DPI engines implement shortcuts for payload patterns that appear frequently in test traffic. These shortcuts do not generalize to novel application data.
The argument for using free tools is cost. The true cost accounting should include: engineering time to build the test bench, time spent resolving scaling limitations, time spent designing the methodology — and then the cost of making a multi-year procurement decision based on results the vendor prepared for. That is not a cheap test.
What Good NGFW Testing Looks Like
The gap between a well-designed test and a poorly designed one comes down to three controllable variables.
Dynamic traffic generation. Every session must be generated dynamically during the test — unique headers, unique tokens, unique payloads at L7. No two sessions of the same type should be byte-for-byte identical. This prevents caching optimizations and ensures the device is classifying every flow from scratch, which is what it will do in production.
Closed-loop AppMix control. The traffic generator must measure the actual application mix transiting the DUT and feed that back into session scheduling. If the measured mix drifts from the target, scheduling weights are adjusted dynamically. Without this, the AppMix you report is the AppMix you configured, not the AppMix you measured — and those are not the same number.
Security effectiveness under load. Attack injection must occur within the background traffic stream, at realistic load levels — not in isolation with the device at idle. A device that blocks 99% of attacks at 5% utilization and 70% of attacks at 80% utilization is a fundamentally different security outcome than a headline detection rate suggests.
These are not exotic requirements. They are standard practice for infrastructure qualification at major carriers and financial institutions. They are achievable with the right tooling and methodology.
Next in this series — Post 3: WAF, IPS, and DLP — why content inspection testing is harder than it looks, and how evasion coverage separates products that protect you from products that protect you in demos.
TEST4NET LLC is an independent network and security testing laboratory operating with no commercial relationships to any security vendor. Contact us to discuss your NGFW evaluation.