Technical University of Crete Packet Pre-filtering for Network Intrusion Detection Ioannis Sourdis, Vasilis Dimopoulos, Dionisios Pnevmatikatos and Stamatis Vassiliadis CE, TU Delft, the Netherlands ECE, TU Crete, Greece ICS-FORTH, Greece

TU Crete © Ioannis Sourdis 2 Introduction Intrusion Detection System (IDS): Packet Classification (Header Matching) Payload Scan (Pattern Matching) Related work on Pattern matching & Packet Classification Is that enough for the core of a high-speed IDS???

TU Crete © Ioannis Sourdis 3 Motivation IDS rulesets contain thousands of rules (SNORT IDS: 3,000-4,000) IDS rule syntax becomes more complicated: Significant Cost Performance limitation The high-speed Pattern matching techniques are required but are not adequate to accommodate the new IDS syntax features HEADERHEADER P1 payload depth HEADERHEADER P1P2 min Distance offset within

TU Crete © Ioannis Sourdis 4 What do we need? Need for different levels of processing packets 1 st Level: use a simplified version of the rules It’s a Fact: Not ALL rules will match every single packet Exclude the majority of the rules 2 nd Level: use advanced attack descriptions for the remaining rules Reduce the cost and maintain high performance

TU Crete © Ioannis Sourdis 5 How to accomplish this? We are looking for a fast-inexpensive way to exclude/ “filter-out” for each incoming packet most of the IDS rules + point out a small subset of possibly matching rules? Partially match each rule of the set Considering only header info per rule  groups of hundreds of rules. Many rules may have the same header description! Suggestion: header + a small portion of payload pattern in the filter Rule #1 Rule #2 Rule # Rule #N Rule #N Rule #M IDS ruleset Possibly matching rules

TU Crete © Ioannis Sourdis 6 Packet Pre-filtering How: Put a small FIRST part of each rule in the pre-filtering Match (if any) the header description of the rule (Source and Destination Address, protocol) Match a prefix of the first payload pattern of the rule (constant number of bytes) IF for a packet, this part of the rule matches  the rule needs to be fully matched ELSE  the rule is excluded/filtered-out

TU Crete © Ioannis Sourdis 7 Example IDS Ruleset: 1. Rule(1): header: H(1) payload: P(1)+[pattern_suffix] 2. Rule(2): header: H(1) payload: P(2)+[pattern_suffix] … N. Rule(N): header: H(1) payload: P(N)+[pattern_suffix] P(1)…P(N): prefixes of payload patterns (i.e characters long) The pre-filtering won’t be efficient if incoming packets look like this: H(1)P(1)…P(2)…P(3)…P(N)… Candidate: Rule(1)Rule(3)……..Rule(N)Rule(2)

TU Crete © Ioannis Sourdis 8 Example IDS Ruleset: 1. Rule(1): header: H(1) payload: P(1)+[pattern_suffix] 2. Rule(2): header: H(1) payload: P(2)+[pattern_suffix] … N. Rule(N): header: H(1) payload: P(N)+[pattern_suffix] P(1)…P(N): prefixes of payload patterns (i.e characters long) The pre-filtering will be efficient if incoming packets look like this: The less rules activated per single packet the most efficient the pre-filtering H(1)P(1)…P(2)… Candidate: Rule(1)Rule(2)

TU Crete © Ioannis Sourdis 9 HW Implementation Field extractor: Extract header and payload Payload Matching: Prefix of the 1 st payload pattern (i.e chars) Implementation  DCAM Regular Expressions may also be used (matching again Reg. Expr. prefixes) Header matching: Src/Dest IP+Port, Protocol Implementation  simple comparators Bitmask, each bit corresponds to a rule Priority Encoder: Pipelined, encodes/outputs every SET bit of the bitmask.

TU Crete © Ioannis Sourdis 10 2 nd Level: Complete Match Engine HW or SW? SW  assign in multiple threads to match the candidate rules Network processor  multiple processing engines... Fully customized HW system We propose (not excluding other solutions): Reconfigurable Hardware: HW performance, flexibility, reconfiguration to update the ruleset

TU Crete © Ioannis Sourdis 11 Reconfigurable IDS core Organization Pre-filtering points out the rules to be fully matched Specialized Engines: For each candidate rule: A PE is reserved A firmware is transferred to the PE PE released  EoP or rule mismatch Coprocessors (Static patterns & Regular expression matching) perform payload scan PEs select the coprocessor info and decide whether a rule matches or not

TU Crete © Ioannis Sourdis 12 What if (Candidate rules > PEs) ? What does (Candidate rules>PEs) mean? A single packet partially matches more than x rules (e.g. x=32 or 64) Can such a packet be a normal packet? What happens when (Candidate rules>PEs)? In order to guarantee performance, the packet is reported, Admin policies determine the next step (i.e. drop)

TU Crete © Ioannis Sourdis 13 Experimental Results Defcon11 traces 9 trace files ~10 millions packets 4.6 million packets have payload payload length: Mean 698 bytes Max 1460 bytes SNORT v2.4 3,191 rules 2,271 rules with payload description 920 only header

TU Crete © Ioannis Sourdis 14 Simulation Results Pre-Filtering setup: Header matching  Scr/dest IP+Port, Protocol Payload Pattern match  2-10 chars prefix match For prefix>2 chars: Average Candidate rules per packet=[1,3] per trace Overall average: 1.8 rules per packet Only header match  ~45 rules per packet

TU Crete © Ioannis Sourdis 15 Simulation Results Payload prefix match= 2 chars: max 63 candidate rules per packets Payload prefix match>=4 chars: max 32 candidate rules per packets What does this mean: Max number of rules for further processing  1% or 32 out of 3,200 rules The Max degree of parallelism needed (processing engines, threads etc.)

TU Crete © Ioannis Sourdis 16 Implementation Results Datapath 8 bits/cycle: Virtex2: 2.7 Gbps Virtex4: 4 Gbps Area 11K slices (medium-small FPGA) Datapath 32 bits/cycle: Virtex2: 9.7 Gbps Virtex4: 14 Gbps Area 15K slices (medium-small FPGA) Priority encoder takes most of the area

TU Crete © Ioannis Sourdis 17 Conclusions Packet Pre-filtering Points out a small rule subset per incoming packet for further processing Offloads an IDS core Allows to utilize more sophisticated attack descriptions on the 2 nd phase of the system (for a few rules) We include in the filter (per rule) Header matching (Source/Destination Address & Protocol) 4-8 characters payload pattern match

TU Crete © Ioannis Sourdis 18 Conclusions Performance: 99% of the IDS rules per incoming packet do not need further processing (in Defcon11 traces), without loosing detection precision. Throughput: Gbps (Virtex2) or 4-14 Gbps (Virtex4), 8 or 32 bits datapaths Requirements: Lightweight system, requires 10-15K slides, can fit in a medium-sized FPGA Can be integrated in both HW or SW based systems

TU Crete © Ioannis Sourdis 19 Questions?

TU Crete © Ioannis Sourdis 20 Complete Intrusion Detection Engine IDS core: We’ve implemented: Packet Pre-filtering Coprocessors (Regular Expression + Static Patterns)  Specialized engines  Glue logic/interfacing Coprocessors more area consuming. Pre-filtering 15-25% of the area. Preliminary results: 8-bit datapath: can fit in a medium FPGA 32-bit datapath: can fit in a the largest existing FPGA.

TU Crete © Ioannis Sourdis 21 Priority Encoder Fixed Priority Pipelined  scales well as the #inputs increases Encodes/outputs every SET bit of the bitmask Binary tree like structure Bitmask  leafs of the tree Each time an input is selected over another to move from stage N  stage N+1, is partially encoded in N+1 and deleted from stage N Stage NStage N+1