Post 5 of 6: How to Run a Security Product Test You Can Actually Trust
Alexander Zemskov
This is the final part of a six-part series on evaluating network security products.
The previous posts in this series described what goes wrong in typical security product evaluations: vendor-optimized test conditions, PCAP replay tools the vendor can prepare against, AppMix drift that makes cross-vendor comparison meaningless, and demos that bear no relationship to production load. This post covers what a well-structured evaluation actually looks like — and who should be running it.
The Conflict of Interest Problem
Before discussing methodology, it is worth being direct about who typically runs security product evaluations and what their incentives are.
Vendors conduct proof-of-concept engagements to sell their products. Their engineers are skilled, professional, and knowledgeable — and they are optimizing for a favorable result. The traffic profile, the DUT configuration, the specific scenarios selected for the PoC, and the metrics highlighted in the results report all reflect this. This is not dishonesty; it is rational commercial behavior. It does not produce a neutral assessment.
Integrators have a structural conflict that is less frequently acknowledged. An integrator's margin on a project depends on which product is selected, at what price point, and with what scope of implementation services. An integrator who runs a PoC comparing three vendors and recommends the one with the lowest margin is acting against their financial interest. Most integrators are professional and act in good faith — but the incentive structure is real and should be accounted for, particularly in high-value decisions.
Internal teams conducting PoCs with open-source tools face a different problem: they typically lack the methodology depth to design a test that accounts for fast path optimization, AppMix drift, and evasion coverage. They get results that feel meaningful and are technically real — but reflect conditions the vendor was prepared for, not the conditions that matter.
An independent laboratory eliminates the structural conflict. Its fee is paid by the buyer, fixed regardless of outcome, and not contingent on any vendor being selected. Its value depends entirely on the credibility and defensibility of its results.
The Non-Negotiable Elements of a Trustworthy Test Plan
A test plan that produces decision-quality results must address five elements consistently missed in typical evaluations.
1. Locked Configuration Before Testing Begins
The DUT configuration — firmware version, enabled features, security policies, logging levels, all modules active or inactive — must be documented and locked before the first packet is sent. No changes during the test window. No "tuning" between runs. No post-hoc adjustments before re-running a scenario.
This matters because configuration changes during testing are how results get managed. A device that underperforms in early runs with feature X enabled and then has feature X silently disabled for the final run that gets reported is not a fair comparison. Locking configuration in advance eliminates this variable.
The locked configuration should reflect your intended production configuration — your rule count, your enabled policies, your logging volume, your inspection settings. Not the vendor's default. Not a minimal configuration that produces clean numbers.
2. Dynamic Traffic Generation With L7 Session Diversity
Every session must be generated dynamically during the test. No two flows of the same type should share identical L7 payloads. This means:
- Unique HTTP transactions with varied headers, tokens, URI patterns, and content lengths
- TLS sessions with varied cipher suite negotiation, certificate handling, and SNI values
- SMTP with varied sender/recipient combinations, message structures, and attachment profiles
- DNS with varied query types and response handling
This prevents the caching and fast-path optimizations that make PCAP replay results unreliable. It forces the device to classify and inspect each flow independently — which is exactly what it will do in production.
Dynamic generation does not mean random and uncontrolled. The traffic profile must be defined, parameterized, and reproducible. The same seed, the same parameters, the same test — identical traffic characteristics every time. Reproducibility is what makes results defensible and enables re-testing after configuration changes.
3. Closed-Loop AppMix Control With DUT Verification
The application mix that transits the DUT must be measured, reported, and controlled — not assumed to match what the generator was configured to send.
A test report that states "traffic profile: 30% HTTPS, 20% HTTP, 15% DNS..." should include a verified measurement of what actually transited the DUT at the protocol level, taken from traffic counters on the DUT itself or from inline measurement at the test ports. If the measured mix differs from the configured mix by more than an agreed tolerance, the test is invalid.
Without this, cross-vendor comparison is mathematically unsound. You cannot compare results from Device A and Device B if the actual AppMix each device processed was different — even if you configured the same target distribution.
4. Security Effectiveness Under Load, Not in Isolation
Attack detection rates and blocking effectiveness must be measured during, not separately from, performance testing.
The test scenario must inject attack traffic within a background stream of legitimate application traffic, at load levels representative of production utilization — targeting 60–80% of the device's rated throughput. Detection rate, block rate, false positive rate, and any change in latency or throughput caused by the security processing under attack conditions must all be captured.
A device that detects 99% of attacks at idle and 65% at 75% utilization is providing a different level of protection than a headline detection rate suggests. Testing in isolation hides this.
5. Granular Per-Protocol Reporting
Aggregate throughput, aggregate latency, and an overall attack detection rate are not sufficient to understand a result. The test report must include:
- Per-protocol throughput and transaction success rate (HTTPS separate from HTTP, DNS, SMTP, etc.)
- Per-protocol latency distributions (not just averages — 95th and 99th percentile matter)
- Verified AppMix distribution at the DUT
- Attack detection and block rates by attack category
- Failed transaction counts by protocol type
- Device resource utilization over the test window (CPU, memory, concurrent sessions)
Granular reporting is what allows you to understand why a device produced a given result — and to predict whether that result will hold under variations in your production traffic profile.
The Right Tooling: Why It Matters
Methodology alone is not sufficient. The traffic generation and measurement platform must be capable of implementing the methodology accurately.
Commercial professional test platforms — including Keysight BreakingPoint and CyPerf — were built specifically for this class of testing. They generate traffic dynamically, maintain closed-loop AppMix control, inject attacks within legitimate traffic streams, and produce the granular per-protocol statistics required for a complete test report. They are used by the world's largest carriers, financial institutions, and government organizations for infrastructure qualification precisely because their results are reproducible and auditable.
Open-source tools are packet generators and basic load tools. They are useful for what they do. What they do is not sufficient for security product evaluation: no dynamic L7 generation, no AppMix feedback loop, no per-protocol statistics at the DUT level, no integrated attack emulation with evasion variation. They produce numbers. They do not produce understanding.
The cost differential between professional tooling and open-source is real. It is also dwarfed by the cost of a procurement error on a multi-year security infrastructure contract.
What TEST4NET Does
TEST4NET is an independent network and security testing laboratory. Engagements are fixed-fee, paid by the organization making the procurement decision, with no commercial relationship to any vendor under evaluation.
Testing is conducted using Keysight BreakingPoint and CyPerf, providing dynamic traffic generation, closed-loop AppMix control, and full granular reporting across all protocol types. Attack emulation uses dynamically generated payloads with controlled evasion variation — not static PCAP replay.
Every engagement begins with a test plan agreed and locked before testing starts, covering DUT configuration, traffic profile parameters, AppMix targets with verification methodology, pass/fail criteria, and reporting format. The test plan is a document the buyer owns. If a vendor or integrator challenges the methodology, the buyer has a defensible record of what was tested and how.
Deliverables include:
- Pre-test methodology document (locked before testing begins)
- Raw test data with per-protocol granularity
- Verified AppMix measurements from the DUT
- Side-by-side comparison across evaluated vendors under identical conditions
- Final report with findings, analysis, and a procurement recommendation grounded in measured data
TEST4NET brings one thing that vendors and integrators structurally cannot: no stake in the outcome.
Before You Sign the Contract
The investment in independent testing is a fraction of the cost of a three-to-five year security infrastructure commitment based on vendor-optimized results. The question is not whether you can afford independent testing. The question is whether you can afford to skip it.
If your organization is planning an NGFW, WAF, DLP, IPS, or SASE evaluation and wants methodology that produces results you can defend to your board, your auditors, and your operations team — talk to TEST4NET before the PoC begins.
Next in this series — Post 6: Why a single evaluation is not enough — continuous validation across the security platform lifecycle.
TEST4NET LLC is an independent network and security testing laboratory. We design and execute SASE evaluations with agents deployed at your actual user locations, against your actual application endpoints. Contact us.
