Computer Science and Engineering, University of California, Riverside SyFi: A Systematic Approach for Estimating Stateful Firewall Performance Yordanos Beyene Michalis Faloutsos Harsha V. Madhyastha Computer Science and Engineering, University of California, Riverside

Which firewall will meet my throughput needs? Problem spec: Input: traffic workload Output: select the right firewall Solution requirements: systematic, accurate, cost effective Can't run each workload on each firewall Add picture: Network (cloud) –Firewall (box)- a questionmark on the box?

SyFi: Estimating firewall performance without tears Key novelty: - Identify what really affects firewall performance - Develop SyFi predictive model for any workload -- We only need to measure 4 parameters once! SyFi is highly accurate ~94% accuracy - Validated through experiments with real firewalls

Motivation: It is the wild west out there.... Performance is workload specific There is no systematic methodology Buyers fliers "lie" by presenting best case numbers Relative order of which is best firewall varies by workload!

Roadmap Part I: Measurement: what affects FWall performance Part II: The SyFi Predictive Model Validation of Model Previous work Conclusion

Part I: What really affects FWall performance? Experiments conducted on two commercial firewalls SonicWall E5500, and Fortinet Fortigate-ONE We used a third commercial firewall to validate model: HP TMSzl firewall Traffic generating tool: BreakingPoint Systems

#Concurrent Sessions: no effect The number of active sessions on firewalls has negligible impact on maximum packet rate of firewalls. Similar results observed with UDP packets

Packet Size: no effect Packet size does not affect maximum packet rate! THUS: thruput should be reported in packets/sec Similar results observed with UDP packets

The first packet takes longer: #sessions matters Packets that create sessions on Firewalls impose significantly higher cost that subsequent packets. Session rate: Sessions of 1 packet Packet rate: Session of 10K packets The overall idea for figure:

TCP is costlier than UDP TCP packets impose higher cost than UDP packets. Similar results are observed with UDP versus TCP session packets.

SyFi Measurement: ACL Size Access Control List (ACL) size no impact on data packets that belong to an existing sessions, but affects session rate significantly. Similar results are observed with the other firewall devices. This slide is not critical it can be shown just to show that we thought about this, but it can be skipped completely too…

SyFi Measurement: Key Findings Identified four types of packets that generate different load TCP SYN packets – trigger session on firewall TCP data packets UDP flow first packets - trigger session on firewall UDP flow subsequent packets Number of concurrent flows doesn’t matter Packet size has negligible impact on packet rate The size of ACL has significant impact on session rate

Part II: The SyFi Predictive Model Measure once, for each firewall, cost Ct: C1: TCP session start C2: TCP subsequent packet C3: UDP flow first packet C4: UDP flow subsequent packet Cost is expressed relatively to 100% utilization Transform traffic load to, percentage of packets Pt #sessions, and length for UDP and TCP Calculate expected thruput Note: model does not consider ACL effect

Step 1: How to measure packet cost in practice For a given firewall, do Maximum Packet Rate Procedure (MPRP) Each flow sends 10,000 packets per second Initialize: start with 1 flow Repeat: add 1 flow that sends 10,000 packets per second every 60 seconds Stop: when packets start to drop. MPRP is used to measure maximum packet rate for both TCP (MPR_TCP) and UDP(MPR_UDP) C2 = 1/(MPR_TCP) ; C4= 1/(MPR_UDP) The limitation of 10,000 packets /second/flow is test tool limitation

Step I: How to measure session setup cost For each firewall, do Maximum Session Rate Procedure (MSRP) Each flow has ``one” packet Every second start S new flows Initialize: start S=5k new flows every second Repeat: Increase S by 1K flows every 60 seconds Stop: when packets start to drop MSRP is used to measure maximum packet rate for both TCP (MSR_TCP) and UDP(MSR_UDP) C1 = 1/(MSR_TCP) ; C3= 1/(MSR_UDP) For both TCP and UDP sessions Tip: firewalls configured with low session time-out to avoid session table overflow. MSRP is used to measure maximum TCP and UDP session rate with flows that send packets that create new sessions on the firewall but don’t send subsequent data packets.

Step II: Transform expected workload to numbers Given that only the four packet types matter We only need to know Percentage of packets Pt are in each type!

Step III: The Predictive Part Given workload type: Pt : percentage of packet type t Measured (once per firewall): Ct : cost of packet type t Two Outcomes: For a given traffic intensity, N Predict total system utilization U where N is the total number of packets/sec) For a 100% system utilization, U = 100% Predict max N for a given workload N: total number of packet/sec Given average packet size , throughput can be computed in bytes per second( bps) Sidenote: CPU is the bottleneck: Cost reflects CPU utilization Ct Maximum N computed when CPU utilization is 100% i.e. c=1

Prediction test case Take home message: it is very simple! Sample workload: 20 % tcp flows, 80% udp flows tcp flow packet count = 10 ; tcp average packet size=512 bytes udp flow packet count = 100; udp average packet size=64 bytes; Firewall measured Cost: C1=1/10,000, c2=1/200,000, c3=1/30,000, c4=1/400,000 Calculate Pi: P1 = 0.2(1/10) , P2 = 0.2(9/10), P3 = 0.8(1/100), P4 = 0.8(99/100) Calculate maximum packets per second(pps) N= (0.2(1/10)*1/10,000 + 0.2(9/10)*1/200,000 + 0.8(1/100)*30,000 + 0.8(99/100)*1/400,000) Calculate Throughput in Bytes (bps) Throughput in bytes = o.2*N *512 bytes + 0.8*N*64 bytes

The model is >94% accurate Workload: TP1, TP2, TP3, TP4, details in the paper Compares measured with model results using the third firewall

Previous work Improving Firewall architecture Research so far has not focused on this problem, but focuses more on: Detecting Firewall Rules conflicts Al-Shaer, E., Hamed, H., Boutaba, R., Hasan, M.: Conflict classification and analysis of distributed firewall policies. In: IEEE JSAC (2005) Baboescu, F., Varghese, G.: Fast and scalable conflict detection for packet classifiers. In: IEEE ICNP (2002) Hari, A., Suri, S., Parulkar, G.: Detecting and resolving packet filter conflicts. In: IEEE INFOCOM (2000) Optimizing firewall rule sets Acharya, S., Wang, J., Ge, Z., Zane, T.F., Greenberg, A.: Traffic-aware firewall optimization strategies. In: ICC (2006) Cohen, E., Lund, C.: Packet classification in large ISPs: Design and evaluation of decision tree classifiers. In: ACM SIGMETRICS (2005) Hamed, H., Al-Shaer, E.: Dynamic rule-ordering optimization for high-speed firewall filtering. In: ASIACCS (2006) Improving Firewall architecture Gouda, M.G., Liu, A., Jafry, M.: Verification of distributed firewalls. In: IEEE GLOBECOM (2008) Gouda, M.G., Liu, A.X.: Structured firewall design. Computer Networks (2007) Liu, A.X.: Firewall policy verification and troubleshooting. In: ICC (2008)

Conclusion Currently assessing firewall performance is “chaotic” Leaves room for manipulation by vendors! Key contribution: - We identify what really affects firewall performance - Develop SyFi predictive model for any workload - We only need to measure 4 parameters once! SyFi is highly accurate ~94% accuracy - Validated through experiments with real firewalls

Future Work Our model was focused on stateful firewalls which inspect packet headers only. We are working on expanding our model to network security devices that inspect payload. We expect packet size to have a significant impact on performance when packet payload inspection is involved.

Thank you!