Connect. Communicate. Collaborate Experiences with tools for network anomaly detection in the GÉANT2 core Maurizio Molina, DANTE COST TMA tech. Seminar Samos, 23 rd Sep 2008

Connect. Communicate. Collaborate The GÉANT Network DANTE operates GÉANT2 Backbone network for National Research and Education Networks in Europe 30+ NRENs, 2 global connectivity providers (Telia and GCrossing), peerings with other research networks (Abilene, Canarie, Clara, TEIN2, SINET…)

Connect. Communicate. Collaborate The GÉANT Network (IP layer) 20 Juniper routers tenths of GBit/s of aggregated traffic Main accesses and the backbone 10Gbit/s Pls see

Connect. Communicate. Collaborate The Services So…. Just a big pipe? No! Services –Dedicated L1-L2 circuits via multiple technologies –Performance Monitoring services (perfSONAR) –Support for federation of National AA Infrastructures (eduGAIN) and wireless roaming (eduROAM) –Security Service Very NEW! NEW!

Connect. Communicate. Collaborate The vision: enhance NRENs security NRENs have their (+ - evolved…) CERTs to deal with security and DANTE can filter traffic on GÉANT upon NRENs request…. ! BUT ! Can we be more proactive to NREN CERTs exploiting the visibility of the GN2 core?

Connect. Communicate. Collaborate The vision (cont.): enhance NRENs security Approach: NetFlow (+ Routing data) & good processing tools NetFlow v5 collector Netflow collected on all peering interfaces 1 / 1,000 Sampling ~3k flows/s

Connect. Communicate. Collaborate Proof of concept: Can we identify anomalies in the core? Anomalies are often “hidden” Requirements:  High detection rate  Low false positives  Anomaly classification  Evidence collection NfSen

Connect. Communicate. Collaborate From “volume” to “IP feature entropies” Connect. Communicate. Collaborate “IP features entropies” Simple linear filter

Connect. Communicate. Collaborate Drilling down on peaks Connect. Communicate. Collaborate -Concentration of DST IPs and DST ports receiving flows -Dispersion of SRC IPs and SRC ports IRC server in Slovenia, receiving a lot of 60 bytes syn pkts on port 6667, mainly from a /16 Subnetwork of an University in the Netherlands. Likely a “BotNet war”?

Connect. Communicate. Collaborate Drilling down on peaks (cont.) Connect. Communicate. Collaborate - Concentration of SRC and DST IPs and SRC ports - Dispersion of DST ports Portscan of host in CARNET, from 4 hosts, 29 bytes packets

Connect. Communicate. Collaborate Open source tools Results: –anomalies are observable in the GÉANT2 core –Novel methodologies (IP Features entropy) for their classifications are applicable Limits: –NfSen does not fuse NetFlow and Routing data –Extensions would need to be run (and tuned) on all ingress/egress points –No support, no guaranteed development

Connect. Communicate. Collaborate Commercial tools Test started Jun 08 (3 tools) –Tool 1 PCA, entropy –Tool 2 Large scale DDoS and Worm spread –Tool 3 Per host behaviour

Connect. Communicate. Collaborate Tool 1 (as a security tool…) Two main novel elements –Principal Component Analysis (PCA) –Both Volume and IP features Entropy anomaly detection Address what makes anomaly detection a complex task –PCA: single parameter to control detection sensitivity, even if anomalies are attributed to specific OD pairs –Entropy: Detection of both low volume (scans) and high volume (DoS) anomalies

Connect. Communicate. Collaborate Demo…. …. Or Screenshots….

Connect. Communicate. Collaborate

Tool 2 Well-established (and expensive!) solution for detecting “large” events Originally based on large volume shifts only Now enhanced to give alerts on “fingerprints” (e.g. communication with C&C servers) –Shared by (part) of the user community (50 out of 120) No usage of routing data –though “zones” can be manually created via BGP prefixes lists Traditional threshold based detection (although adaptive)

Connect. Communicate. Collaborate Tool 3 Per host behavioural analysis rather complex “scoring” system to distinguish normal from abnormal behaviour. Proprietary algorithms Doesn’t use routing info –though “zones” can be manually created via BGP prefixes lists Potentially attractive methodology Concerns on scalability and accuracy with 1,000 sampling

Connect. Communicate. Collaborate lessons learnt and directions for research Manual validation is required to confirm/correct anomalies –More automatic intelligence to help this process –Fusion with other data sources (router logs? Honeynets?) Detection space of 3 tools often disjoint –(Standard) anomaly injection Operations need supported tools to support services If choice is among published but “not a tool” or “secret but supported and (claiming to) work” => risk to stick to those! –Fill the gap towards TOOLS!

Connect. Communicate. Collaborate Thank you!