Post 4 of 6: SASE — Why Cloud Security Benchmarks Are Almost Always Irrelevant to Your Deployment
Alexander Zemskov
This is part four of a six-part series on evaluating network security products. Part 1 covered the structural datasheet gap. Part 2 covered NGFW testing traps. Part 3 covered WAF, IPS, and DLP.
Secure Access Service Edge introduces a different class of evaluation challenge compared to on-premises hardware. The performance of a SASE deployment is not a fixed property of a product — it is a function of architecture, geography, shared infrastructure load, and the specific routing path between your users and your applications at any given moment. That makes the numbers in a vendor's marketing materials uniquely unreliable as a predictor of what you will experience.
The Shared Infrastructure Problem
An on-premises NGFW appliance is dedicated hardware sitting in your rack. Its performance envelope is deterministic: the same traffic profile produces the same throughput and latency consistently, because no one else is competing for its resources.
A SASE Point of Presence is shared infrastructure. The same PoP handles traffic from dozens or hundreds of enterprise customers simultaneously. Its capacity is pooled. Its utilization varies by time of day, by day of week, and by regional traffic patterns that have nothing to do with your deployment.
A vendor demo conducted from a controlled location, during business hours in a chosen time zone, against a vendor-controlled application endpoint, tells you what the platform can achieve under ideal conditions with dedicated attention. It does not tell you what your remote users in different offices will experience during Monday morning peak hours when the regional PoP is under concurrent load from multiple tenants.
This is not a hypothetical concern. Performance variability across time and PoP location is a structural feature of shared cloud infrastructure. Evaluating a SASE platform requires measuring performance from your actual user locations, to your actual application endpoints, across multiple time windows — including peak load periods.
Latency Amplification Under Full Stack
SASE architecture stacks multiple security functions in the traffic path. Each function adds latency. The vendor's architecture diagram presents these as discrete, additive components. The reality of how latency compounds under concurrent load is more complex.
Functions commonly in the SASE inline path: ZTNA broker, SWG (Secure Web Gateway), CASB, URL filtering, DLP inspection, remote browser isolation, SSL inspection. Each of these processes traffic before forwarding it. Under low load, the latency of each is small and the total remains manageable. Under high concurrency — many users, many simultaneous sessions, peak traffic periods — queuing effects and processing interaction between components produce non-linear latency growth.
Vendors typically present per-function latency figures measured in isolation. They rarely publish end-to-end latency with all functions active under realistic concurrent load. These are not the same number, and the gap between them is what your users will notice.
When evaluating SASE, insist on end-to-end latency measurements — from a client agent at a representative user location to a representative application — with the full intended security stack active, under load representative of your peak concurrent session count.
TLS Inspection at Enterprise Scale
SASE platforms that perform SSL/TLS inspection are terminating and re-originating every TLS session for every user, at the PoP level. The security benefit is real: without inspection, encrypted traffic transits the security stack without content analysis. With inspection, full DPI and policy enforcement apply.
The operational and performance implications are frequently understated.
Certificate management. TLS inspection requires the PoP to present a certificate trusted by client endpoints. Managing this across a distributed enterprise — including BYOD devices, guest endpoints, and applications that use certificate pinning — has operational complexity that becomes visible during deployment, not during the demo.
Inspection accuracy under high-entropy encrypted traffic. Modern TLS 1.3 with encrypted SNI and ESNI limits the metadata available before decryption. Classification accuracy and policy enforcement correctness under these conditions varies across vendors and is rarely tested with realistic modern TLS traffic during evaluations.
Throughput under concurrent TLS sessions. A vendor PoP handling TLS termination for thousands of concurrent enterprise sessions is performing continuous asymmetric cryptographic operations. Throughput figures measured against a handful of test clients do not predict behavior at enterprise scale. If your deployment involves tens of thousands of remote users transiting a shared PoP, test the platform under a load level that approximates that concurrency — not a sanitized lab scenario.
Geographic Reality: The Benchmark Was Not Measured Under Your Conditions
This point is worth stating directly: no vendor's published SASE benchmark was measured under the conditions of your specific deployment.
Published benchmarks are produced from vendor-controlled infrastructure, between vendor-chosen endpoints, at vendor-chosen times. They optimize for the best number achievable. Your deployment involves:
- Users distributed across your actual office locations and home offices
- Applications split between specific cloud regions and specific on-premises data centers
- Traffic routing through PoPs that may or may not be geographically optimal for your topology
- Concurrent load from other tenants sharing the same PoP infrastructure
The latency and throughput you experience will be determined by these actual conditions. The only way to predict them is to measure them — with real agents at real user locations, against real application endpoints, across the time windows that matter to your business operations.
Proof of Concept evaluations for SASE that run from a single test location, against a vendor-hosted demo application, for a few hours during a scheduled window, do not produce useful data. They produce marketing-quality numbers that look good in a selection committee presentation.
What a Meaningful SASE Evaluation Looks Like
A SASE evaluation that produces decision-quality results requires a different approach from appliance testing.
Deploy agents at representative user locations. If your users are in Yerevan, London, Singapore, and São Paulo, those are the measurement points. Traffic must originate from those locations through the SASE stack to reach your actual applications.
Measure against real application endpoints. Your business-critical applications — ERP, collaboration tools, cloud storage — are the target. Vendor-hosted demo endpoints do not represent the routing paths, latency characteristics, or TLS profiles of your actual applications.
Test at multiple time windows. Monday 9 AM, Wednesday 2 PM, and Friday 5 PM represent different load conditions at shared PoPs. Measure across the full operational range.
Enable the full intended security stack. ZTNA, CASB, SWG, DLP, URL filtering — whatever combination you intend to operate. Per-function benchmarks are not additive and do not predict full-stack performance.
Measure end-to-end latency, not hop latency. What users experience is the total round-trip time from their device to the application. Measure that directly, not the latency to the PoP edge.
Compare against your current baseline. The question is not whether SASE meets an absolute threshold; it is whether it meets or improves on what you have today, under the same conditions. Measure your current architecture and the SASE candidate under identical conditions.
Next in this series — Post 5: How to structure a test that produces results you can defend — methodology, tooling, and why independent evaluation pays for itself.
TEST4NET LLC is an independent network and security testing laboratory. We design and execute SASE evaluations with agents deployed at your actual user locations, against your actual application endpoints. Contact us.
