1. Technical University of Crete Packet Pre-filtering for Network Intrusion Detection Ioannis Sourdis, Vasilis Dimopoulos, Dionisios Pnevmatikatos and Stamatis Vassiliadis CE, TU Delft, the Netherlands ECE, TU Crete, Greece ICS-FORTH, Greece

2. TU Crete 5.12.2006 © Ioannis Sourdis 2 Introduction Intrusion Detection System (IDS): Packet Classification (Header Matching) Payload Scan (Pattern Matching) Related work on Pattern matching & Packet Classification Is that enough for the core of a high-speed IDS???

3. TU Crete 5.12.2006 © Ioannis Sourdis 3 Motivation IDS rulesets contain thousands of rules (SNORT IDS: 3,000-4,000) IDS rule syntax becomes more complicated: Significant Cost Performance limitation The high-speed Pattern matching techniques are required but are not adequate to accommodate the new IDS syntax features HEADERHEADER P1 payload depth HEADERHEADER P1P2 min Distance offset within

4. TU Crete 5.12.2006 © Ioannis Sourdis 4 What do we need? Need for different levels of processing packets 1 st Level: use a simplified version of the rules It’s a Fact: Not ALL rules will match every single packet Exclude the majority of the rules 2 nd Level: use advanced attack descriptions for the remaining rules Reduce the cost and maintain high performance

5. TU Crete 5.12.2006 © Ioannis Sourdis 5 How to accomplish this? We are looking for a fast-inexpensive way to exclude/ “filter-out” for each incoming packet most of the IDS rules + point out a small subset of possibly matching rules? Partially match each rule of the set Considering only header info per rule  groups of hundreds of rules. Many rules may have the same header description! Suggestion: header + a small portion of payload pattern in the filter Rule #1 Rule #2 Rule #3.............. Rule #N Rule #N+1.... Rule #M IDS ruleset Possibly matching rules

6. TU Crete 5.12.2006 © Ioannis Sourdis 6 Packet Pre-filtering How: Put a small FIRST part of each rule in the pre-filtering Match (if any) the header description of the rule (Source and Destination Address, protocol) Match a prefix of the first payload pattern of the rule (constant number of bytes) IF for a packet, this part of the rule matches  the rule needs to be fully matched ELSE  the rule is excluded/filtered-out

7. TU Crete 5.12.2006 © Ioannis Sourdis 7 Example IDS Ruleset: 1. Rule(1): header: H(1) payload: P(1)+[pattern_suffix] 2. Rule(2): header: H(1) payload: P(2)+[pattern_suffix] … N. Rule(N): header: H(1) payload: P(N)+[pattern_suffix] P(1)…P(N): prefixes of payload patterns (i.e. 4-10 characters long) The pre-filtering won’t be efficient if incoming packets look like this: H(1)P(1)…P(2)…P(3)…P(N)… Candidate: Rule(1)Rule(3)……..Rule(N)Rule(2)

8. TU Crete 5.12.2006 © Ioannis Sourdis 8 Example IDS Ruleset: 1. Rule(1): header: H(1) payload: P(1)+[pattern_suffix] 2. Rule(2): header: H(1) payload: P(2)+[pattern_suffix] … N. Rule(N): header: H(1) payload: P(N)+[pattern_suffix] P(1)…P(N): prefixes of payload patterns (i.e. 4-10 characters long) The pre-filtering will be efficient if incoming packets look like this: The less rules activated per single packet the most efficient the pre-filtering H(1)P(1)…P(2)… Candidate: Rule(1)Rule(2)

9. TU Crete 5.12.2006 © Ioannis Sourdis 9 HW Implementation Field extractor: Extract header and payload Payload Matching: Prefix of the 1 st payload pattern (i.e. 2-10 chars) Implementation  DCAM Regular Expressions may also be used (matching again Reg. Expr. prefixes) Header matching: Src/Dest IP+Port, Protocol Implementation  simple comparators Bitmask, each bit corresponds to a rule Priority Encoder: Pipelined, encodes/outputs every SET bit of the bitmask.

10. TU Crete 5.12.2006 © Ioannis Sourdis 10 2 nd Level: Complete Match Engine HW or SW? SW  assign in multiple threads to match the candidate rules Network processor  multiple processing engines... Fully customized HW system We propose (not excluding other solutions): Reconfigurable Hardware: HW performance, flexibility, reconfiguration to update the ruleset

11. TU Crete 5.12.2006 © Ioannis Sourdis 11 Reconfigurable IDS core Organization Pre-filtering points out the rules to be fully matched Specialized Engines: For each candidate rule: A PE is reserved A firmware is transferred to the PE PE released  EoP or rule mismatch Coprocessors (Static patterns & Regular expression matching) perform payload scan PEs select the coprocessor info and decide whether a rule matches or not

12. TU Crete 5.12.2006 © Ioannis Sourdis 12 What if (Candidate rules > PEs) ? What does (Candidate rules>PEs) mean? A single packet partially matches more than x rules (e.g. x=32 or 64) Can such a packet be a normal packet? What happens when (Candidate rules>PEs)? In order to guarantee performance, the packet is reported, Admin policies determine the next step (i.e. drop)

13. TU Crete 5.12.2006 © Ioannis Sourdis 13 Experimental Results Defcon11 traces 9 trace files ~10 millions packets 4.6 million packets have payload payload length: Mean 698 bytes Max 1460 bytes SNORT v2.4 3,191 rules 2,271 rules with payload description 920 only header

14. TU Crete 5.12.2006 © Ioannis Sourdis 14 Simulation Results Pre-Filtering setup: Header matching  Scr/dest IP+Port, Protocol Payload Pattern match  2-10 chars prefix match For prefix>2 chars: Average Candidate rules per packet=[1,3] per trace Overall average: 1.8 rules per packet Only header match  ~45 rules per packet

15. TU Crete 5.12.2006 © Ioannis Sourdis 15 Simulation Results Payload prefix match= 2 chars: max 63 candidate rules per packets Payload prefix match>=4 chars: max 32 candidate rules per packets What does this mean: Max number of rules for further processing  1% or 32 out of 3,200 rules The Max degree of parallelism needed (processing engines, threads etc.)

16. TU Crete 5.12.2006 © Ioannis Sourdis 16 Implementation Results Datapath 8 bits/cycle: Virtex2: 2.7 Gbps Virtex4: 4 Gbps Area 11K slices (medium-small FPGA) Datapath 32 bits/cycle: Virtex2: 9.7 Gbps Virtex4: 14 Gbps Area 15K slices (medium-small FPGA) Priority encoder takes most of the area

17. TU Crete 5.12.2006 © Ioannis Sourdis 17 Conclusions Packet Pre-filtering Points out a small rule subset per incoming packet for further processing Offloads an IDS core Allows to utilize more sophisticated attack descriptions on the 2 nd phase of the system (for a few rules) We include in the filter (per rule) Header matching (Source/Destination Address & Protocol) 4-8 characters payload pattern match

18. TU Crete 5.12.2006 © Ioannis Sourdis 18 Conclusions Performance: 99% of the IDS rules per incoming packet do not need further processing (in Defcon11 traces), without loosing detection precision. Throughput: 2.7-10 Gbps (Virtex2) or 4-14 Gbps (Virtex4), 8 or 32 bits datapaths Requirements: Lightweight system, requires 10-15K slides, can fit in a medium-sized FPGA Can be integrated in both HW or SW based systems

19. TU Crete 5.12.2006 © Ioannis Sourdis 19 Questions?

20. TU Crete 5.12.2006 © Ioannis Sourdis 20 Complete Intrusion Detection Engine IDS core: We’ve implemented: Packet Pre-filtering Coprocessors (Regular Expression + Static Patterns)  Specialized engines  Glue logic/interfacing Coprocessors more area consuming. Pre-filtering 15-25% of the area. Preliminary results: 8-bit datapath: can fit in a medium FPGA 32-bit datapath: can fit in a the largest existing FPGA.

21. TU Crete 5.12.2006 © Ioannis Sourdis 21 Priority Encoder Fixed Priority Pipelined  scales well as the #inputs increases Encodes/outputs every SET bit of the bitmask Binary tree like structure Bitmask  leafs of the tree Each time an input is selected over another to move from stage N  stage N+1, is partially encoded in N+1 and deleted from stage N Stage NStage N+1