TEST4NET blog

Post 6 of 6: One Test Is Not Enough — Why Security Infrastructure Needs Continuous Validation

az@test4net.com (Alexander Zemskov) — Fri, 10 Jul 2026 17:24:48 GMT

This is the sixth post in a series on evaluating network security products. Part 1: the datasheet gap. Part 2: NGFW traps. Part 3: WAF, IPS, DLP. Part 4: SASE. Part 5: methodology and independent testing.

The previous five posts focused on the procurement stage: how to design a test that produces results you can trust before signing a contract. This post addresses what happens after the contract is signed — and why treating the initial evaluation as a one-time event is one of the most common and costly mistakes in security operations.

The Platform You Bought Is Not the Platform You Will Be Running in Two Years

Every enterprise security platform — NGFW, WAF, IPS, DLP, SASE — receives regular software updates. Firmware releases, security patches, threat intelligence database updates, new feature rollouts, algorithm changes, performance optimizations. Vendors push these on a continuous schedule. In the NGFW space alone, major vendors release firmware updates multiple times per year, with hotfixes and security patches arriving far more frequently.

Each of these updates changes the platform. Sometimes the changes are minor. Sometimes they are not.

A firmware update that introduces a new DPI algorithm — better detection accuracy, broader protocol coverage, more sophisticated evasion resistance — also introduces new processing overhead per inspected session. A threat intelligence database update that adds 50,000 new signatures improves security posture and increases the lookup cost per packet. A new feature that provides genuine operational value — encrypted DNS inspection, improved ML-based anomaly detection, tighter integration with a cloud sandbox — consumes memory, CPU cycles, and potentially dedicated ASIC capacity.

The platform you validated at procurement, at a specific firmware version, with a specific feature set and database state, is not the platform that will be running after 18 months of regular updates. The performance envelope shifts with every significant change. The question is whether you are tracking that shift — or discovering it when a production incident forces the conversation.

Hardware Is a Fixed Constraint. Software Is Not.

This is where the gap between cloud-native and hardware-based security platforms becomes operationally significant in a way that procurement decisions rarely fully account for.

A cloud-delivered SASE platform or a virtual NGFW running on elastic infrastructure has a degree of flexibility when a software update increases resource consumption: the underlying compute can be scaled, additional instances can be spun up, memory and CPU headroom can be added without a hardware procurement cycle. The operational friction is real, and cost increases, but the ceiling is not fixed.

A hardware appliance is a closed resource envelope. The RAM is soldered. The CPU is fixed. The network processor and security ASIC are what they are. When a firmware update increases per-session memory consumption, there is no way to add memory. When a new inspection algorithm adds 15% CPU overhead, there is no way to add cores. When the combination of new features and an updated signature database pushes concurrent session memory usage to 90% of installed RAM, the margin for traffic spikes disappears.

Vendors announce new firmware capabilities with marketing materials that lead with the security and functionality improvements. Resource consumption implications are documented in release notes, if at all, in technical language that rarely translates into a clear statement of "this update will reduce your effective throughput by N% on the following hardware models." The customer sees the feature announcement. The performance impact arrives in production.

This is not hypothetical. In practice, organizations running hardware security appliances at 60–70% rated capacity at procurement routinely find themselves at 85–90% rated capacity two or three years into the deployment lifecycle — with no hardware change, no significant traffic growth, and a list of firmware updates that each seemed individually minor.

New Features as a Trojan Horse

There is a specific pattern worth naming directly, because it recurs across vendors and product categories.

A vendor releases a major firmware version that introduces capabilities the customer genuinely wants: better visibility into encrypted traffic, improved application identification, a new threat detection engine with demonstrably higher catch rates. The security team evaluates the new capabilities, confirms they address real gaps in the current posture, and schedules the upgrade.

What does not get evaluated — because no structured test exists for it — is the performance impact of enabling those new capabilities on the existing hardware, under the production traffic profile, at production load levels.

The upgrade is deployed. The new features are enabled. For the first few days, everything appears normal. Then, gradually or suddenly depending on the nature of the change, the symptoms appear: latency increases under peak load, connection setup rates degrade, memory utilization trends upward and does not come back down, and the first tickets arrive from users reporting slowness in applications that were fast before.

The incident response process begins. The conclusion, eventually, is that the new firmware on the existing hardware at the current traffic volume no longer has sufficient headroom. The options are to roll back the firmware (losing the security improvements), disable the new features (losing the reason for upgrading), or accelerate hardware refresh (unplanned capital expenditure).

All three outcomes could have been identified before the production deployment. None of them require expensive or elaborate testing. They require a structured pre-deployment validation process.

Pre-Deployment Validation: What It Actually Requires

Validating a security platform update before production deployment does not require reproducing the full procurement PoC. It requires answering a more targeted set of questions.

Throughput regression. Does the updated firmware maintain throughput within an acceptable tolerance of the previous version, under the same traffic profile? A 5% regression may be acceptable. A 20% regression is a capacity planning event.

Resource consumption at production load. What are CPU utilization, memory consumption, and concurrent session counts at representative production load under the new firmware? How does this compare to the previous version? What is the new headroom margin?

Correctness of new features. Do the new capabilities function as documented? Do they interact correctly with existing policies? Do they produce unexpected changes in traffic handling behavior?

Security regression. Does the update maintain or improve detection effectiveness? Firmware changes that touch the DPI engine, the rule matching logic, or the protocol parsers can introduce regressions in detection coverage — not because the vendor intended it, but because complex systems change in non-obvious ways. This should be verified, not assumed.

Rollback viability. If the update produces an unexpected problem in production, can it be reversed cleanly? Testing the rollback path before deploying the update is the difference between a planned rollback taking 20 minutes and an emergency rollback taking four hours while a production system is degraded.

The Case for a Permanent Testing Capability

The cumulative argument of this post is the case for continuous testing as an operational discipline — not testing as an episodic activity triggered by procurement events.

At the point where an organization is managing multiple security platforms, each with its own update cadence, operating across a production environment where performance and security baselines matter, the operational model shifts. Ad hoc testing before major updates is a start. A structured, repeatable validation process with defined baselines, regression thresholds, and a clear workflow from "update released" to "approved for production" is the operational maturity level that eliminates the class of incidents described above.

Building this capability in-house is the right answer at a certain scale — when the volume of testing, the frequency of updates requiring validation, and the internal staff available to operate the process justify the investment in permanent test infrastructure and tooling. At that point, owning a dedicated test environment with commercial-grade traffic generation gives the security team a capability that pays dividends across procurement, update validation, incident response, and capacity planning.

For organizations that are not yet at that scale — or that need the capability immediately without the lead time of building internal infrastructure — the operational model is an ongoing relationship with an independent laboratory. Not a one-time procurement engagement, but a repeatable process: update is released, validation test is run against the agreed baseline, results are compared, go/no-go decision is made before production deployment.

The cost of that process per update cycle is predictable and bounded. The cost of an unvalidated update causing a production degradation — engineering time, incident response, potential emergency hardware procurement, business impact — is not.

A Practical Framework for Update Validation

Regardless of whether testing is conducted internally or through an independent laboratory, the process requires three elements to be consistent and defensible.

A frozen baseline. At procurement validation (or at the point of establishing the testing program), a performance and security baseline is captured against the agreed traffic profile. This is the reference point against which all subsequent updates are compared. Throughput at defined AppMix, resource utilization at defined load levels, security detection rates against the agreed attack profile. These numbers are locked and do not change until a deliberate decision is made to update the baseline.

A defined regression tolerance. What magnitude of change in any measured parameter requires escalation before production deployment? The thresholds should be defined in advance and agreed between the security team and operations — not determined retrospectively during an incident review. Typical parameters: throughput regression greater than X%, memory utilization increase greater than Y percentage points, concurrent session capacity reduction greater than Z%.

A repeatable test execution process. The test must be executable by whoever runs it — internal team, external lab — and produce comparable results every time. Same traffic profile, same load levels, same measurement methodology. Variability in execution methodology produces variability in results that makes trend analysis impossible.

What Does Not Change

One thing remains constant regardless of how mature the continuous testing program becomes: the validity of the test depends on the quality of the traffic profile and the accuracy of the measurement tools.

All the problems described in earlier posts in this series — AppMix drift, PCAP replay predictability, security effectiveness testing in isolation — apply equally to update validation testing as they do to procurement testing. A baseline captured with unreliable tooling is not a baseline. A regression comparison made with a tool that produces inconsistent results does not produce defensible conclusions.

The investment in proper methodology and proper tooling at the procurement stage pays forward into every subsequent test run across the lifecycle of the platform.

Summary: Testing Is Operational Infrastructure, Not a Project

The procurement test is the beginning of a testing relationship with a security platform, not the end of it. Hardware constraints are fixed; software complexity grows with every update cycle. New features and algorithms that vendors market as improvements can silently consume the performance headroom that validated the hardware selection in the first place.

The organizations that avoid the class of incidents described in this post share one characteristic: they treat testing as ongoing operational infrastructure, not as a one-time project cost. They have a baseline. They have a process. They run it.
For organizations building toward that maturity — or needing the validation capability now, before building internal infrastructure — TEST4NET LLC provides both the initial procurement evaluation and the ongoing update validation framework.

One engagement model, consistent methodology, results that are comparable across every point in the platform lifecycle.

Post 5 of 6: How to Run a Security Product Test You Can Actually Trust

az@test4net.com (Alexander Zemskov) — Fri, 10 Jul 2026 17:06:27 GMT

This is the final part of a six-part series on evaluating network security products.

Part 1: the datasheet gap. Part 2: NGFW traps. Part 3: WAF, IPS, DLP. Part 4: SASE.

The previous posts in this series described what goes wrong in typical security product evaluations: vendor-optimized test conditions, PCAP replay tools the vendor can prepare against, AppMix drift that makes cross-vendor comparison meaningless, and demos that bear no relationship to production load. This post covers what a well-structured evaluation actually looks like — and who should be running it.

The Conflict of Interest Problem

Before discussing methodology, it is worth being direct about who typically runs security product evaluations and what their incentives are.

Vendors conduct proof-of-concept engagements to sell their products. Their engineers are skilled, professional, and knowledgeable — and they are optimizing for a favorable result. The traffic profile, the DUT configuration, the specific scenarios selected for the PoC, and the metrics highlighted in the results report all reflect this. This is not dishonesty; it is rational commercial behavior. It does not produce a neutral assessment.

Integrators have a structural conflict that is less frequently acknowledged. An integrator's margin on a project depends on which product is selected, at what price point, and with what scope of implementation services. An integrator who runs a PoC comparing three vendors and recommends the one with the lowest margin is acting against their financial interest. Most integrators are professional and act in good faith — but the incentive structure is real and should be accounted for, particularly in high-value decisions.

Internal teams conducting PoCs with open-source tools face a different problem: they typically lack the methodology depth to design a test that accounts for fast path optimization, AppMix drift, and evasion coverage. They get results that feel meaningful and are technically real — but reflect conditions the vendor was prepared for, not the conditions that matter.

An independent laboratory eliminates the structural conflict. Its fee is paid by the buyer, fixed regardless of outcome, and not contingent on any vendor being selected. Its value depends entirely on the credibility and defensibility of its results.

The Non-Negotiable Elements of a Trustworthy Test Plan

A test plan that produces decision-quality results must address five elements consistently missed in typical evaluations.

1. Locked Configuration Before Testing Begins

The DUT configuration — firmware version, enabled features, security policies, logging levels, all modules active or inactive — must be documented and locked before the first packet is sent. No changes during the test window. No "tuning" between runs. No post-hoc adjustments before re-running a scenario.

This matters because configuration changes during testing are how results get managed. A device that underperforms in early runs with feature X enabled and then has feature X silently disabled for the final run that gets reported is not a fair comparison. Locking configuration in advance eliminates this variable.

The locked configuration should reflect your intended production configuration — your rule count, your enabled policies, your logging volume, your inspection settings. Not the vendor's default. Not a minimal configuration that produces clean numbers.

2. Dynamic Traffic Generation With L7 Session Diversity

Every session must be generated dynamically during the test. No two flows of the same type should share identical L7 payloads. This means:

Unique HTTP transactions with varied headers, tokens, URI patterns, and content lengths
TLS sessions with varied cipher suite negotiation, certificate handling, and SNI values
SMTP with varied sender/recipient combinations, message structures, and attachment profiles
DNS with varied query types and response handling

This prevents the caching and fast-path optimizations that make PCAP replay results unreliable. It forces the device to classify and inspect each flow independently — which is exactly what it will do in production.

Dynamic generation does not mean random and uncontrolled. The traffic profile must be defined, parameterized, and reproducible. The same seed, the same parameters, the same test — identical traffic characteristics every time. Reproducibility is what makes results defensible and enables re-testing after configuration changes.

3. Closed-Loop AppMix Control With DUT Verification

The application mix that transits the DUT must be measured, reported, and controlled — not assumed to match what the generator was configured to send.

A test report that states "traffic profile: 30% HTTPS, 20% HTTP, 15% DNS..." should include a verified measurement of what actually transited the DUT at the protocol level, taken from traffic counters on the DUT itself or from inline measurement at the test ports. If the measured mix differs from the configured mix by more than an agreed tolerance, the test is invalid.

Without this, cross-vendor comparison is mathematically unsound. You cannot compare results from Device A and Device B if the actual AppMix each device processed was different — even if you configured the same target distribution.

4. Security Effectiveness Under Load, Not in Isolation

Attack detection rates and blocking effectiveness must be measured during, not separately from, performance testing.

The test scenario must inject attack traffic within a background stream of legitimate application traffic, at load levels representative of production utilization — targeting 60–80% of the device's rated throughput. Detection rate, block rate, false positive rate, and any change in latency or throughput caused by the security processing under attack conditions must all be captured.

A device that detects 99% of attacks at idle and 65% at 75% utilization is providing a different level of protection than a headline detection rate suggests. Testing in isolation hides this.

5. Granular Per-Protocol Reporting

Aggregate throughput, aggregate latency, and an overall attack detection rate are not sufficient to understand a result. The test report must include:

Per-protocol throughput and transaction success rate (HTTPS separate from HTTP, DNS, SMTP, etc.)
Per-protocol latency distributions (not just averages — 95th and 99th percentile matter)
Verified AppMix distribution at the DUT
Attack detection and block rates by attack category
Failed transaction counts by protocol type
Device resource utilization over the test window (CPU, memory, concurrent sessions)

Granular reporting is what allows you to understand why a device produced a given result — and to predict whether that result will hold under variations in your production traffic profile.

The Right Tooling: Why It Matters

Methodology alone is not sufficient. The traffic generation and measurement platform must be capable of implementing the methodology accurately.

Commercial professional test platforms — including Keysight BreakingPoint and CyPerf — were built specifically for this class of testing. They generate traffic dynamically, maintain closed-loop AppMix control, inject attacks within legitimate traffic streams, and produce the granular per-protocol statistics required for a complete test report. They are used by the world's largest carriers, financial institutions, and government organizations for infrastructure qualification precisely because their results are reproducible and auditable.

Open-source tools are packet generators and basic load tools. They are useful for what they do. What they do is not sufficient for security product evaluation: no dynamic L7 generation, no AppMix feedback loop, no per-protocol statistics at the DUT level, no integrated attack emulation with evasion variation. They produce numbers. They do not produce understanding.

The cost differential between professional tooling and open-source is real. It is also dwarfed by the cost of a procurement error on a multi-year security infrastructure contract.

What TEST4NET Does

TEST4NET is an independent network and security testing laboratory. Engagements are fixed-fee, paid by the organization making the procurement decision, with no commercial relationship to any vendor under evaluation.

Testing is conducted using Keysight BreakingPoint and CyPerf, providing dynamic traffic generation, closed-loop AppMix control, and full granular reporting across all protocol types. Attack emulation uses dynamically generated payloads with controlled evasion variation — not static PCAP replay.

Every engagement begins with a test plan agreed and locked before testing starts, covering DUT configuration, traffic profile parameters, AppMix targets with verification methodology, pass/fail criteria, and reporting format. The test plan is a document the buyer owns. If a vendor or integrator challenges the methodology, the buyer has a defensible record of what was tested and how.

Deliverables include:

Pre-test methodology document (locked before testing begins)
Raw test data with per-protocol granularity
Verified AppMix measurements from the DUT
Side-by-side comparison across evaluated vendors under identical conditions
Final report with findings, analysis, and a procurement recommendation grounded in measured data

TEST4NET brings one thing that vendors and integrators structurally cannot: no stake in the outcome.

Before You Sign the Contract

The investment in independent testing is a fraction of the cost of a three-to-five year security infrastructure commitment based on vendor-optimized results. The question is not whether you can afford independent testing. The question is whether you can afford to skip it.

If your organization is planning an NGFW, WAF, DLP, IPS, or SASE evaluation and wants methodology that produces results you can defend to your board, your auditors, and your operations team — talk to TEST4NET before the PoC begins.

Next in this series — Post 6: Why a single evaluation is not enough — continuous validation across the security platform lifecycle.

TEST4NET LLC is an independent network and security testing laboratory. We design and execute SASE evaluations with agents deployed at your actual user locations, against your actual application endpoints. Contact us.

Post 4 of 6: SASE — Why Cloud Security Benchmarks Are Almost Always Irrelevant to Your Deployment

az@test4net.com (Alexander Zemskov) — Fri, 10 Jul 2026 16:47:28 GMT

This is part four of a six-part series on evaluating network security products. Part 1 covered the structural datasheet gap. Part 2 covered NGFW testing traps. Part 3 covered WAF, IPS, and DLP.

Secure Access Service Edge introduces a different class of evaluation challenge compared to on-premises hardware. The performance of a SASE deployment is not a fixed property of a product — it is a function of architecture, geography, shared infrastructure load, and the specific routing path between your users and your applications at any given moment. That makes the numbers in a vendor's marketing materials uniquely unreliable as a predictor of what you will experience.

The Shared Infrastructure Problem

An on-premises NGFW appliance is dedicated hardware sitting in your rack. Its performance envelope is deterministic: the same traffic profile produces the same throughput and latency consistently, because no one else is competing for its resources.

A SASE Point of Presence is shared infrastructure. The same PoP handles traffic from dozens or hundreds of enterprise customers simultaneously. Its capacity is pooled. Its utilization varies by time of day, by day of week, and by regional traffic patterns that have nothing to do with your deployment.

A vendor demo conducted from a controlled location, during business hours in a chosen time zone, against a vendor-controlled application endpoint, tells you what the platform can achieve under ideal conditions with dedicated attention. It does not tell you what your remote users in different offices will experience during Monday morning peak hours when the regional PoP is under concurrent load from multiple tenants.

This is not a hypothetical concern. Performance variability across time and PoP location is a structural feature of shared cloud infrastructure. Evaluating a SASE platform requires measuring performance from your actual user locations, to your actual application endpoints, across multiple time windows — including peak load periods.

Latency Amplification Under Full Stack

SASE architecture stacks multiple security functions in the traffic path. Each function adds latency. The vendor's architecture diagram presents these as discrete, additive components. The reality of how latency compounds under concurrent load is more complex.

Functions commonly in the SASE inline path: ZTNA broker, SWG (Secure Web Gateway), CASB, URL filtering, DLP inspection, remote browser isolation, SSL inspection. Each of these processes traffic before forwarding it. Under low load, the latency of each is small and the total remains manageable. Under high concurrency — many users, many simultaneous sessions, peak traffic periods — queuing effects and processing interaction between components produce non-linear latency growth.

Vendors typically present per-function latency figures measured in isolation. They rarely publish end-to-end latency with all functions active under realistic concurrent load. These are not the same number, and the gap between them is what your users will notice.

When evaluating SASE, insist on end-to-end latency measurements — from a client agent at a representative user location to a representative application — with the full intended security stack active, under load representative of your peak concurrent session count.

TLS Inspection at Enterprise Scale

SASE platforms that perform SSL/TLS inspection are terminating and re-originating every TLS session for every user, at the PoP level. The security benefit is real: without inspection, encrypted traffic transits the security stack without content analysis. With inspection, full DPI and policy enforcement apply.

The operational and performance implications are frequently understated.

Certificate management. TLS inspection requires the PoP to present a certificate trusted by client endpoints. Managing this across a distributed enterprise — including BYOD devices, guest endpoints, and applications that use certificate pinning — has operational complexity that becomes visible during deployment, not during the demo.

Inspection accuracy under high-entropy encrypted traffic. Modern TLS 1.3 with encrypted SNI and ESNI limits the metadata available before decryption. Classification accuracy and policy enforcement correctness under these conditions varies across vendors and is rarely tested with realistic modern TLS traffic during evaluations.

Throughput under concurrent TLS sessions. A vendor PoP handling TLS termination for thousands of concurrent enterprise sessions is performing continuous asymmetric cryptographic operations. Throughput figures measured against a handful of test clients do not predict behavior at enterprise scale. If your deployment involves tens of thousands of remote users transiting a shared PoP, test the platform under a load level that approximates that concurrency — not a sanitized lab scenario.

Geographic Reality: The Benchmark Was Not Measured Under Your Conditions

This point is worth stating directly: no vendor's published SASE benchmark was measured under the conditions of your specific deployment.

Published benchmarks are produced from vendor-controlled infrastructure, between vendor-chosen endpoints, at vendor-chosen times. They optimize for the best number achievable. Your deployment involves:

Users distributed across your actual office locations and home offices
Applications split between specific cloud regions and specific on-premises data centers
Traffic routing through PoPs that may or may not be geographically optimal for your topology
Concurrent load from other tenants sharing the same PoP infrastructure

The latency and throughput you experience will be determined by these actual conditions. The only way to predict them is to measure them — with real agents at real user locations, against real application endpoints, across the time windows that matter to your business operations.

Proof of Concept evaluations for SASE that run from a single test location, against a vendor-hosted demo application, for a few hours during a scheduled window, do not produce useful data. They produce marketing-quality numbers that look good in a selection committee presentation.

What a Meaningful SASE Evaluation Looks Like

A SASE evaluation that produces decision-quality results requires a different approach from appliance testing.

Deploy agents at representative user locations. If your users are in Yerevan, London, Singapore, and São Paulo, those are the measurement points. Traffic must originate from those locations through the SASE stack to reach your actual applications.

Measure against real application endpoints. Your business-critical applications — ERP, collaboration tools, cloud storage — are the target. Vendor-hosted demo endpoints do not represent the routing paths, latency characteristics, or TLS profiles of your actual applications.

Test at multiple time windows. Monday 9 AM, Wednesday 2 PM, and Friday 5 PM represent different load conditions at shared PoPs. Measure across the full operational range.

Enable the full intended security stack. ZTNA, CASB, SWG, DLP, URL filtering — whatever combination you intend to operate. Per-function benchmarks are not additive and do not predict full-stack performance.

Measure end-to-end latency, not hop latency. What users experience is the total round-trip time from their device to the application. Measure that directly, not the latency to the PoP edge.

Compare against your current baseline. The question is not whether SASE meets an absolute threshold; it is whether it meets or improves on what you have today, under the same conditions. Measure your current architecture and the SASE candidate under identical conditions.

Next in this series — Post 5: How to structure a test that produces results you can defend — methodology, tooling, and why independent evaluation pays for itself.

Post 3 of 6: WAF, IPS, and DLP — Why Content Inspection Testing Is Harder Than It Looks

az@test4net.com (Alexander Zemskov) — Fri, 10 Jul 2026 16:38:58 GMT

This is part three of a six-part series on evaluating network security products. Part 1 covered the structural datasheet gap. Part 2 covered NGFW-specific testing traps.

WAF, IPS, and DLP platforms are frequently purchased on the strength of vendor demos and detection rate claims. They are also among the most frequently disappointing in production — not because the technology does not work, but because the testing methodology used to select them did not reflect the conditions they would operate under.

WAF: What "Requests Per Second" Does Not Tell You

Web Application Firewall performance figures — requests per second, throughput, latency — are almost universally measured under conditions that have little to do with production web application traffic.

The synthetic request problem. Most WAF test setups use a fixed set of HTTP/HTTPS requests, either synthetic or drawn from a limited PCAP capture. Every request follows the same structure: same headers, same cookie format, same URI pattern. Real web application traffic is the opposite: dynamic session tokens on every request, variable content lengths, authentication flows that span multiple requests with state tracked across them, redirects, varying TLS session characteristics. A WAF rule engine that processes a uniform synthetic request stream has a fundamentally easier job than one processing the actual output of a modern web application.

Attack isolation vs. attack-in-traffic. WAF detection accuracy figures are nearly always measured by sending attack payloads in isolation: a clean test session with one malicious request, response recorded, detection confirmed. What this does not test is whether the WAF maintains the same detection accuracy when attack payloads are embedded within a high-volume stream of legitimate application requests. Under load, with the rule engine processing thousands of concurrent sessions, detection accuracy for subtle injection attempts and low-signal evasions can degrade. Test both: isolated attack detection, and attack detection embedded at realistic rates within a background traffic load.

TLS inspection under concurrency. Many WAF deployments include TLS termination or full man-in-the-middle inspection. Vendors frequently report TLS and non-TLS performance separately and present the better number. The performance impact of TLS termination, certificate validation, session resumption handling, and re-origination at high concurrency is where the production number lives. Test with your realistic TLS connection rate, your typical session duration, and with TLS inspection active.

Rule set complexity matters. A WAF with the baseline rule set and a WAF with the baseline rules plus custom virtual patching for a dozen applications plus exception rules accumulated over two years of operation are running different workloads. Performance testing with a minimal rule set does not predict behavior under your production policy.

IPS: The Evasion Gap That Detection Rates Hide

Intrusion Prevention Systems are evaluated almost entirely on detection rate. This is a metric that is easy to measure, easy to present, and largely disconnected from the protection that matters.

Detection rate is a circular measurement when the test tool is known. An IPS that detects 99% of attacks drawn from a library the vendor prepared against will produce a 99% detection rate in a test using that library. The measurement tells you the device recognizes attacks it was trained to recognize. This is useful, but incomplete. The more important question is: what happens to attacks that use evasion techniques the vendor did not specifically train for?

Evasion coverage is where products diverge. Evasion techniques — payload fragmentation, protocol ambiguity exploitation, encoding variations, polymorphic shellcode, header manipulation — are how real attackers bypass signature-based detection. An IPS that blocks 99% of direct attacks but fails against 30% of evasion-wrapped variants of those same attacks provides substantially less protection than its headline number implies.

The only meaningful way to test evasion coverage is with an attack emulation platform that generates attacks dynamically with controlled variation in encoding, fragmentation strategy, and protocol handling — not a static replay of known payloads. Static PCAP-based attack testing gives the vendor advance knowledge of every payload. That is not a security test; it is a rehearsal.

Detection under load. At 20% utilization, an IPS engine can afford to be thorough. At 80% utilization under sustained attack alongside heavy legitimate traffic, the same engine may start making triage decisions: dropping packets, reducing inspection depth, or deferring classification. Testing IPS detection rate at idle tells you the ceiling. Testing it at 70–80% of rated throughput tells you what your users will experience under the conditions where an attacker is most likely to operate.

DLP: The Most Consistently Oversold Category in Security

Data Loss Prevention is where the gap between vendor claims and production reality tends to be widest, and where the consequences of a failed evaluation are most operationally costly.

The false positive problem at scale. DLP accuracy figures from vendor testing are measured against clean test datasets: purpose-built documents with clearly structured sensitive data, or small controlled corpora. False positive rates in this environment look acceptable — 1–3% is common in vendor materials.

Production environments do not have clean, controlled document corpora. They have years of accumulated files in dozens of formats, with complex embedded content, legacy templates that match data patterns without containing actual sensitive data, and business-specific terminology that overlaps with sensitive data definitions. Against a realistic enterprise content sample, false positive rates that appeared manageable in testing can reach levels that generate hundreds of incidents per day — making the system operationally unusable without months of tuning.

Ask vendors to run their DLP detection against a sample of your actual document repository during evaluation. The difference between controlled-test accuracy and production-realistic accuracy is the number that matters.

Throughput under real content inspection. DLP requires application-layer content analysis of file payloads — not packet inspection. This is computationally expensive, and the cost varies enormously by file type. A DLP engine processing a stream of plain text emails operates at much lower overhead than one processing a stream of large Excel workbooks with embedded macros, nested ZIP archives, or heavily formatted PDFs.

Vendor throughput figures are almost always measured against lightweight content. Test with a realistic file type distribution representative of what your organization actually moves across the network. The throughput number may look very different.

Protocol coverage: what "DLP" actually covers. Enterprise data exfiltration does not confine itself to SMTP and HTTP uploads. Modern environments involve Teams, Slack, Zoom file transfers, cloud sync clients (OneDrive, Google Drive, Dropbox), custom SaaS upload flows, and a long tail of proprietary protocols. Ask the vendor to enumerate which protocols receive full content inspection versus metadata-only monitoring versus no coverage. The distinction matters significantly for what the platform actually protects against.

Shared Principle: Test the Thing You Are Actually Buying

Across WAF, IPS, and DLP, the same principle applies: the test must reflect the conditions the product will operate under in your environment.

That means:

The traffic or content used in testing must resemble your actual application traffic, your actual document corpus, your actual attack surface
Security function testing must occur under realistic load, not at idle
The feature configuration tested must match what will run in production — not a minimal default that produces favorable benchmark numbers
Results must be granular enough to understand what drove them — not a single aggregate figure

None of this requires unusual methodology. It requires asking clearly what conditions produced each number in a vendor's test results — and independently verifying the answer.

Next in this series — Post 4: SASE — why cloud-delivered security is uniquely difficult to benchmark, and what a meaningful evaluation looks like for distributed architectures.

TEST4NET LLC is an independent network and security testing laboratory. We evaluate WAF, IPS, and DLP platforms using methodology designed to reflect production conditions, not vendor-optimized test scenarios. Contact us.

Post 2 of 6: NGFW Testing — Fast Paths, Fake AppMix, and Why PCAP Replay Helps Vendors More Than It Helps You

az@test4net.com (Alexander Zemskov) — Fri, 10 Jul 2026 16:32:09 GMT

This is part two of a six-part series on evaluating network security products.

Part 1 covered the structural datasheet gap that applies across the entire security industry.

Next-generation firewalls are the most purchased, most frequently benchmarked, and most poorly evaluated category in enterprise security. The testing traps here are well-known inside the industry — and consistently unknown to the buyers making procurement decisions.

The Fast Path: A Feature That Becomes a Testing Trap

Most enterprise NGFW platforms include custom ASICs or dedicated security processors. These chips can offload specific traffic categories to a hardware fast path that bypasses the software stack entirely. This is a legitimate and valuable architectural feature — under the right conditions, it delivers real performance gains with genuine security inspection.

The problem is the phrase "under the right conditions."

Hardware fast paths activate most effectively when traffic is uniform, predictable, and simple: long-lived sessions with large payloads, steady connection rates, cleartext or easily classified protocols. Under those conditions, the ASIC handles most of the work and the software stack is largely idle. Throughput approaches the theoretical hardware maximum.

Real enterprise traffic is structurally different: short sessions, many small objects, TLS 1.3 with encrypted SNI and ESNI, non-standard ports, proprietary application protocols, bursts of new connection establishment. Under real traffic, fast path coverage drops substantially and the software stack carries most of the load.

A poorly designed test — or a test designed to produce favorable results — generates traffic that looks like the first scenario. The device performs excellently. That performance does not survive contact with the production environment.

Vendor-Specific Test Modes: The Optimization That Nobody Announces

Beyond legitimate fast path behavior, there is a more deliberate category of optimization worth understanding.

Vendors know which traffic generators are commonly used for security product testing. They know the profiles, the patterns, the typical PCAP files used in evaluations. Some platforms implement optimizations specifically triggered by these known profiles: DPI result caching for sessions that are byte-for-byte identical to previously seen flows, parser shortcuts for exact payload patterns that appear in common test sets, fast-path rules keyed to characteristics of specific test traffic generators.

These optimizations produce impressive benchmark numbers. They provide little or no benefit against real diverse traffic — and in some cases introduce behavioral differences that only appear under realistic load.

The practical implication: a device that scores well against a familiar test tool may not score comparably well against a methodology it has not been prepared for. Independent testing with a tool the vendor cannot profile in advance is the only way to see past this.

The AppMix Problem: The Traffic Distribution You Configured Is Not the One That Transited Your Device

This is the most technically subtle problem in NGFW testing, and the most consistently overlooked.

When you configure a test with a target application mix — say 25% HTTPS, 20% HTTP, 15% DNS, 15% SMTP, 10% video, 10% database, 5% custom — you expect that distribution to actually move through the device under test. It does not.

Free and open-source traffic generators schedule new sessions according to configured weights. They do not account for how long each protocol type takes to process on the DUT. This matters because security processing is not uniform across protocol types.

Heavy protocols — TLS with mutual authentication, SMTP with large attachments, protocols requiring extended DPI classification — consume more processing time per session on the DUT. They queue. They slow down. The generator, unaware of this, keeps launching new sessions at the configured rate. Light protocols — plain HTTP, DNS queries, small UDP — process quickly and move through the device at full speed.

The result: over the course of the test, the actual traffic mix transiting the DUT shifts toward the lightest, fastest-processing protocols. Your device appears to handle 10 Gbps with full inspection. It genuinely is handling 10 Gbps — but 60% of it is plain HTTP and DNS, because the SMTP and TLS sessions you configured are stalled in queues or being dropped.

The throughput number is real. The traffic profile it was measured against is not what you intended.

Why this makes cross-vendor comparison meaningless without AppMix control. Device A handles heavy protocols efficiently — it processes TLS and complex DPI at moderate overhead. Under your test, Device A's actual AppMix at the DUT is close to what you configured: roughly the intended distribution. Device B drops heavy-protocol sessions faster — the overhead is too high and connections time out. Device B's actual AppMix is dominated by light traffic. Device B shows higher throughput.

You conclude Device B outperforms Device A. You are measuring them with different rulers and do not know it.

PCAP Replay: The Tool That Tells Vendors Exactly What Is Coming

TRex and many testing tools from China using pcap replay mode to reproduce pre-recorded traffic captures. Every session at L7 is byte-for-byte identical to every other session of the same type. Same headers. Same payloads. Same sequence numbers, same SNI, same URI patterns — repeated indefinitely.

The vendor testing your product knows which tools buyers use. They know the PCAP files. They know the payload patterns. It costs nothing to optimize for them.

Specific optimizations that work against deterministic PCAP replay traffic:

DPI result caching: If two TCP sessions have identical payloads, classify the second one instantly from cache rather than running full DPI. This produces dramatically higher throughput against replay traffic and near-zero benefit against real diverse sessions.
Fast-path rules for known-good patterns: A session that matches a known exact pattern from a test PCAP can be fast-pathed after minimal inspection. This does not apply to sessions where payload varies.
Parser shortcuts: Some DPI engines implement shortcuts for payload patterns that appear frequently in test traffic. These shortcuts do not generalize to novel application data.

The argument for using free tools is cost. The true cost accounting should include: engineering time to build the test bench, time spent resolving scaling limitations, time spent designing the methodology — and then the cost of making a multi-year procurement decision based on results the vendor prepared for. That is not a cheap test.

What Good NGFW Testing Looks Like

The gap between a well-designed test and a poorly designed one comes down to three controllable variables.

Dynamic traffic generation. Every session must be generated dynamically during the test — unique headers, unique tokens, unique payloads at L7. No two sessions of the same type should be byte-for-byte identical. This prevents caching optimizations and ensures the device is classifying every flow from scratch, which is what it will do in production.

Closed-loop AppMix control. The traffic generator must measure the actual application mix transiting the DUT and feed that back into session scheduling. If the measured mix drifts from the target, scheduling weights are adjusted dynamically. Without this, the AppMix you report is the AppMix you configured, not the AppMix you measured — and those are not the same number.

Security effectiveness under load. Attack injection must occur within the background traffic stream, at realistic load levels — not in isolation with the device at idle. A device that blocks 99% of attacks at 5% utilization and 70% of attacks at 80% utilization is a fundamentally different security outcome than a headline detection rate suggests.

These are not exotic requirements. They are standard practice for infrastructure qualification at major carriers and financial institutions. They are achievable with the right tooling and methodology.

Next in this series — Post 3: WAF, IPS, and DLP — why content inspection testing is harder than it looks, and how evasion coverage separates products that protect you from products that protect you in demos.

TEST4NET LLC is an independent network and security testing laboratory operating with no commercial relationships to any security vendor. Contact us to discuss your NGFW evaluation.

Post 1 of 6: The Security Datasheet Lie — And Why It Is Structural

az@test4net.com (Alexander Zemskov) — Fri, 10 Jul 2026 16:03:13 GMT

Every network security vendor publishes performance numbers that look compelling in a slide deck. The demo goes smoothly. The PoC delivers great results. Then the hardware lands in production — and within weeks, the operations team is raising tickets about latency spikes, throughput degradation, and security policies getting quietly bypassed under load.

This is not an accident. It is not unique to one vendor. It is a structural feature of how the entire security industry markets and tests its products — and understanding it is the first step toward making a procurement decision you will not regret.

Why Every Datasheet Number Is Technically True and Practically Misleading

Vendors do not fabricate the numbers in their datasheets. The device genuinely achieved that throughput — at that specific moment, under those specific conditions. The problem is that the conditions used to produce the best defensible number are chosen by the vendor, and they rarely resemble what happens in a real enterprise network.

Every security platform has a performance envelope. At one end: basic packet forwarding, no inspection, cleartext traffic, large steady-state sessions. At the other end: full security stack engaged — TLS inspection, DPI, IPS, AV, application control, URL filtering — under a realistic mixed enterprise traffic profile with short-lived sessions, encrypted traffic, and bursty connection rates.

Datasheets lead with the first number. They bury the second number in footnotes, if they include it at all.

The gap between the two ends of that envelope runs at 55–75% across every major vendor in the NGFW space, and similar ranges apply to WAF, IPS, and inline DLP platforms. That is not a bug. That is the cost of security processing. The issue is not the gap itself — it is the absence of honest disclosure.

The Features-Enabled Gap in Practice

Here is how this plays out with real product categories:

NGFW. A flagship appliance advertises its throughput number prominently. Enable the enterprise threat prevention stack — IPS, AV, application control, URL filtering, SSL inspection — and you will measure a fraction of that figure in production. The drop is not a flaw in the hardware; it reflects what deep packet inspection at line rate actually costs. The problem is that the datasheet number was never measured with those features on.

WAF. A WAF vendor publishes requests-per-second and throughput figures. Those numbers were typically measured with a uniform synthetic request profile, TLS offloaded or excluded, and a minimal rule set. Add your full CRS ruleset at production paranoia level, enable TLS inspection, and test with realistic application traffic diversity — the numbers look different.

DLP. Deep content inspection — reading and analyzing the payload of every file, every email attachment, every upload — is expensive. DLP throughput figures are almost always measured against simple, lightweight file types. A full inspection pass against a realistic enterprise document corpus including large Office files, PDFs, and compressed archives produces throughput that can be 20–30% of the datasheet figure.

SASE. A cloud-delivered security stack introduces a different kind of gap: the vendor's published latency and throughput figures were measured from their lab, to their infrastructure, at their chosen time. Your users are distributed across multiple geographies. Your applications are split between cloud and on-premises. Your SASE traffic transits shared PoP infrastructure. The number on the datasheet was not measured under those conditions.

The Configuration Variable Nobody Talks About

Beyond the feature set, there is a second variable that vendors consistently under-disclose: policy and rule complexity.

A firewall with 50 rules and a firewall with 5,000 rules are not the same device from a performance perspective. Policy lookup, rule matching, logging volume — all of these scale with configuration complexity. Vendors test with minimal or default configurations. Production environments accumulate rules over years of operation.

A WAF with the default CRS rule set and a WAF with the default CRS plus custom virtual patching policies plus application-specific rule exceptions operates at a different performance point. A DLP engine with 10 active policies and one with 200 fine-tuned policies for different data classifications behaves differently under load.

Testing in a configuration that does not reflect your production environment produces numbers that do not predict your production performance. This sounds obvious. It is almost universally ignored in vendor-led evaluations.

What This Series Covers

This post is the first in a five-part series on how to evaluate network security products with methodology that predicts production behavior.

The remaining posts cover:

Post 2: NGFW-specific testing traps — fast paths, vendor-optimized test modes, and why the AppMix your test tool reports is not the AppMix that transited your device
Post 3: WAF, IPS, and DLP — content inspection complexity, evasion testing, and the false positive problem
Post 4: SASE — why cloud-delivered security is uniquely difficult to benchmark and what a meaningful evaluation looks like
Post 5: How to structure a test that produces results you can trust — methodology, tooling, and the role of an independent laboratory
Post 6: One test is not enough — why security platforms require continuous validation across their operational lifecycle, and how to build that process

The goal is not to indict any specific vendor. The goal is to give procurement teams, security architects, and network engineers the framework to ask the right questions — before signing a multi-year contract.

TEST4NET LLC is an independent network and security testing laboratory. We have no commercial relationship with any vendor whose products we evaluate. If you are planning a security product procurement and want methodology that produces defensible, production-representative results — contact TEST4NET.

Why Network and Device Testing Is a Business Requirement

az@test4net.com (Alexander Zemskov) — Fri, 10 Jul 2026 13:12:11 GMT

Network and security infrastructure testing is often treated as a technical exercise near the end of deployment. In reality, it is a business requirement.

When environments fail under load, the issue is rarely that production behaved unpredictably. More often, the infrastructure was never fully validated under realistic traffic, session scale, or failure conditions. What looks stable in a basic lab or maintenance window can behave very differently under real operational pressure.

Performance Problems Rarely Start in Production

Throughput bottlenecks, latency spikes, packet loss, unstable failover, session exhaustion, and inconsistent policy behavior do not begin in production. They are usually present earlier, but only become visible when the environment is exposed to realistic traffic and stress.

That is why traffic-based validation matters. Networks and security controls need to be tested not only for nominal operation, but for how they behave under peak load, mixed application traffic, burst conditions, degraded links, path changes, route convergence events, and recovery scenarios. Without that level of validation, deployment decisions are based on assumptions rather than evidence.

Why Realistic Testing Matters

Modern infrastructure is expected to support uninterrupted services, distributed users, cloud-connected applications, and increasingly complex policy enforcement. Under these conditions, a functional check is not enough.

A firewall may appear healthy until concurrent sessions rise sharply. A router may look stable until failover occurs under sustained traffic. A policy change may seem safe in isolation, but create unexpected performance degradation at scale. These are exactly the conditions that realistic lab testing is designed to expose.

Testing with production-like traffic profiles helps answer the questions that matter:

Can the environment sustain expected throughput and session volume?
How does latency behave under stress?
What happens when a link, node, or path fails?
How quickly does traffic recover?
Does performance degrade gradually or collapse abruptly?
Do policies, queues, and control functions behave consistently at scale?

Other Industries Treat Testing as Mandatory

In many industries, testing is not considered optional. Manufacturers rely on quality control before release. Aviation systems must demonstrate reliability before operational use. Automotive platforms are validated under failure and stress conditions. Pharmaceutical production depends on process validation before it is trusted at scale.

Network and security infrastructure should be held to the same standard, yet realistic pre-production validation is still not applied consistently enough. One reason is that teams often rely too heavily on vendor specifications, reference designs, or limited functional testing. Another is that realistic traffic, failover, and resilience validation requires dedicated tooling, time, and process ownership, which are still too often viewed as secondary to delivery speed.

That gap is increasingly difficult to justify. The business dependency on network and security infrastructure is already high, and the cost of failure is immediate. If anything, the stakes are higher here, not lower.

Why the Gap Still Exists

Many organizations assume that if a design is vendor-approved and the change window is controlled, the risk is manageable. In practice, that confidence can be misplaced.

Infrastructure behavior under real load is influenced by traffic composition, session growth, policy enforcement, route changes, retry storms, and recovery timing. These variables are difficult to assess without structured validation. Yet in many projects, the final decision to deploy is still based on configuration review, limited lab checks, or confidence in past experience rather than measured performance under realistic conditions.

This is one of the reasons production incidents remain so costly. The environment is not failing for the first time. It is being tested meaningfully for the first time.

The Business Cost of Untested Behavior

When realistic validation is skipped, the business absorbs the risk.

Performance-related failures affect far more than the network team. They delay transactions, interrupt services, reduce user confidence, and increase the cost of recovery. Even short periods of degraded performance can create lost revenue, operational disruption, reputational damage, and downstream pressure on support and engineering teams.

The cost of testing is planned and measurable. The cost of production failure is not.

What Effective Validation Should Include

Effective pre-production validation should go beyond basic acceptance testing. It should include:

throughput and latency testing under realistic traffic mixes
scale testing for sessions, flows, users, and application behavior
failover and resilience testing during link, path, and node loss
route convergence and recovery validation
stability testing during upgrades, policy changes, and architecture transitions
verification of policy behavior under load

This is where IXIA-, Spirent-, and similar traffic-driven methodologies provide real value. They help organizations measure how infrastructure behaves before customers, users, or operations teams are forced to discover the answer themselves.

If your business depends on uptime, performance, and resilience, validate your network before production does it for you.

Why Vendor Benchmarks Are Not Enough for Infrastructure Decisions

az@test4net.com (Alexander Zemskov) — Fri, 10 Jul 2026 08:19:11 GMT

Your infrastructure plan has a shelf life of 18 months.

AI, 5G, SD-WAN and cloud-native security — every cycle arrives faster than the last.

GPUs went from gaming cards to enterprise AI backbone in under two years
SD-WAN quietly killed MPLS refresh cycles across thousands of networks
5G Core hit production before most operators finished reading the spec
GenAI rewrote security perimeters faster than vendors could ship patches

The pace isn't slowing. But most procurement processes haven't noticed.

When you're moving fast, you rely on what vendors give you: presentations, datasheets and competitive benchmarks. That's the standard procurement playbook. Pick the product that fits the requirements on paper, sign the PO, deploy. The problem? Vendor requirements documents are written to win deals, not to guarantee performance.

Specs are measured under ideal conditions. Performance numbers reflect best-case scenarios. And the fine print about what happens when you actually turn on all the features you paid for? It's usually not there.

Here's the pattern we see again and again:

A company rolls out new infrastructure. The first few months look fine. Then — gradually or suddenly — things start breaking. Bottlenecks appear. Latency spikes. Specific modules hit 100% CPU the moment real traffic touches them.

The root cause is almost always the same: the solution was sized for the spec sheet, not for the real-world workload. The moment advanced features get enabled — DPI, SSL inspection, threat prevention, application control — the headroom evaporates.

By the time this becomes visible, the budget is spent. Building a new procurement case takes at least 12 months. So what do IT teams actually do?

Option A: Keep running degraded. Users suffer. Service quality drops. Competitors notice.
Option B: Make an emergency purchase. Spend the budget twice. Explain that to the CFO.
Option C (most common): Quietly disable features to reduce load. Hide the problem. Ship a product you're using at 30% of its intended capability.

All three paths lead to the same place: subscriber churn, brand damage, missed deadlines and eroded trust.

This is a solvable problem. It just needs to happen earlier.

At TEST4NET, we work with companies that want to get ahead of this — before the contract is signed, before the deployment, before the crisis.

Three words. One methodology: Test. Compare. Decide.

We run deep, independent validation of network solutions using professional-grade traffic generation equipment — testing real-world scenarios, not vendor benchmarks. Stateful traffic. Real application mixes. SSL at full cipher strength. Edge cases that datasheets don't mention.

The typical market standard for device performance validation is around one month per device.

We've cut that significantly — through test automation, a ready-to-run scenario library covering the most common network architectures, and 20+ years of hands-on engineering experience across security, routing, switching and carrier-grade infrastructure testing.

The result: faster answers, deeper insight and decisions backed by evidence — not by a vendor's marketing deck.

If your company is evaluating new network infrastructure, upgrading a security stack or preparing for a major deployment — and you want to know what you're actually buying before you buy it — let’s talk.