Network-based and Attack-resilient Length Signature Generation for Zero-day Polymorphic Worms Zhichun Li 1, Lanjia Wang 2, Yan Chen 1 and Judy Fu 3 1 Lab for Internet and Security Technology (LIST), Northwestern Univ. 2 Tsinghua University, China 3 Motorola Labs, USA

2 The Spread of Sapphire/Slammer Worms

3 Limitations of Exploit Based Signature Our network Traffic Filtering Internet Signature: 10.*01 X X Polymorphic worm might not have exact exploit based signature Polymorphism!

4 Vulnerability Signature Work for polymorphic worms Work for all the worms which target the same vulnerability Vulnerability signature traffic filtering Internet X X Our network Unknown Vulnerability X X Better!

5 Benefits of Network Based Detection At the early stage of the worm, only limited worm samples. Host based sensors can only cover limited IP space, which might have scalability issues. Gateway routers Internet Our network Host based detection Early Detection!

6 Design Space and Related Work Most host approaches depend on lots of host information, such as source/binary code of the vulnerable program, vulnerability condition, execution traces, etc. [Polygraph-SSP05] [Hamsa-SSP06] [PADS-INFOCOM05] [CFG-RAID05] [Nemean-Security05] [DOCODA-CCS05] [TaintCheck-NDSS05] LESG (this paper) [Vulsig-SSP06] [Vigilante-SOSP05] [COVERS-CCS05] [ShieldGen-SSP07] Vulnerability Based Exploit Based Network BasedHost Based

7 Outline Motivation and Related Work Design of LESG Problem Statement Three Stage Algorithm Attack Resilience Analysis Evaluation Conclusions

8 Basic Ideas At least 75% vulnerabilities are due to buffer overflow Intrinsic to buffer overflow vulnerability and hard to evade However, there could be thousands of fields to select the optimal field set is hard Vulnerable buffer Protocol message Overflow!

9 Framework ICDCS06, INFOCOM 06, TON

10 LESG Signature Generator

11 Outline Motivation and Related Work Design of LESG Problem Statement Three Stage Algorithm Attack Resilience Analysis Evaluation Conclusions

12 Field Hierarchies DNS PDU

13 Length-based Signature Definition NameTypeClassTTLRDlengthRDATA Length Signature (Name,100) 100 Length Signature Signature Set {(Name,100), (Class,50), (RDATA,300)} “OR” relationship Ground truth signature (RDATA,315) Buffer length! Vulnerable RDATA

14 Problem Formulation LESG  Worms which are not covered in the suspicious pool are at most  Minimize the false positives in the normal pool Suspicious pool Normal pool Signature With noise NP-Hard!

15 Outline Motivation and Related Work Design of LESG Problem Statement Three Stage Algorithm Attack Resilience Analysis Evaluation Conclusions

16 Stages I and II Stage I: Field Filtering Stage II: Length Optimization COV≥1% FP≤0.1% Trade off between specificity and sensitivity Score function Score(COV,FP)

17 Stage III Find the optimal set of fields as the signature with high coverage and low false positive Separate the fields to two sets, FP=0 and FP>0 –Opportunistic step (FP=0) –Attack Resilience step (FP>0) The similar greedy algorithm for each step

18 Stage III (cont.) suspiciousnormal NameTypeClassTTLCommentsRDATA (Name,100) [40%,0.03%] (Class,50) [35%,0.09%] (RDATA,300) [50%,0.05%] (Comments,2000) [10%,0.1%] Stage I COV 0 ≥1% FP 0 ≤0.1% 50%0.05% Residual coverage ≥5%

19 Stage III (cont.) suspiciousnormal NameTypeClassTTLCommentsRDATA (Class,50) [25%,0.02%] (Name,100) [3%,0.08%] {(RDATA,300)} (Comments,2000) [1%,0.05%] Stage I COV 0 ≥1% FP 0 ≤0.1% 50%0.05% Residual coverage ≥5%

20 Stage III (cont.) suspiciousnormal NameTypeClassTTLCommentsRDATA (Class,50) [25%,0.02%] (Name,100) [3%,0.08%] {(RDATA,300),(Class,50)} (Comments,2000) [1%,0.05%] Stage I COV 0 ≥1% FP 0 ≤0.1% (50+25)%( )% Residual coverage γ≥5%

21 Attack Resilience Bounds Depend on whether deliberated noise injection (DNI) exists, we get different bounds With 50% noise in the suspicious pool, we can get the worse case bound FN<2% and FP<1% In practice, the DNI attack can only achieve FP<0.2% Resilient to most proposed attacks (proposed in other papers)

22 Outline Motivation and Related Work Design of LESG Problem Statement Three Stage Algorithm Attack Resilience Analysis Evaluation Conclusions

23 Methodology Protocol parsing with Bro and BINPAC (IMC2006) Worm workload –Eight polymorphic worms created based on real world vulnerabilities including CodeRed II and Lion worms. –DNS, SNMP, FTP, SMTP Normal traffic data –27GB from a university gateway and 123GB log

24 Results Single/Multiple worms with noise –Noise ratio: 0~80% –False negative: 0~1% (mostly 0) –False positive: 0~0.01% (mostly 0) Pool size requirement –10 or 20 flows are enough even with 20% noises Speed results –With 500 samples in suspicious pool and 320K samples in normal pool, For DNS, parsing 58 secs, LESG 18 secs

25 Conclusions A novel network-based automated worm signature generation approach –Work for zero day polymorphic worms with unknown vulnerabilities –First work which is both Vulnerability based and Network based using length signature for buffer overflow vulnerabilities –Provable attack resilience –Fast and accurate through experiments