Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap Algorithms for Counting Active Flows on High Speed Links Cristian Estan, George Varghese, Mike Fisk Computer Science and Engineering Department, University of California, San Diego

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Why count flows? Detect port/IP scans Identify DoS attacks Estimate spreading rate of a worm Packet scheduling Dave Plonkas FlowScan

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Existing flow counting solutions Server NetFlow data Analysis Traffic reports Network Operations Center Router Fast link Memory Network Memory size & bandwidth Networkbandwidth

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Motivating question Can we count flows at line speeds at the router? –Wrong solution – counters –Naïve solution – use hash tables (like NetFlow) –Our approach – use bitmaps

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting algorithms A family of algorithms that can be used as building blocks in various systems Algorithms can be adapted to application Low memory and per packet processing Generalize flows to distinct header patterns –Count flows or source addresses to detect attack –Count destination address+port pairs to detect scan

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(green)= Set bits in the bitmap using hash of the flow ID of incoming packets

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(blue)= Different flows have different hash values

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(green)= Packets from the same flow always hash to the same bit

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(violet)= Collisions OK, estimates compensate for them

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(orange)=

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(pink)=

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(yellow)= As the bitmap fills up, estimates get inaccurate

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap Solution: use more bits HASH(green)=

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap Solution: use more bits Problem: memory scales with the number of flows HASH(blue)=

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – virtual bitmap Solution: a) store only a portion of the bitmap b) multiply estimate by scaling factor

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – virtual bitmap HASH(pink)=

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – virtual bitmap HASH(yellow)= Problem: estimate inaccurate when few flows active

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multiple bmps Solution: use many bitmaps, each accurate for a different range

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multiple bmps HASH(pink)=

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multiple bmps HASH(yellow)=

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multiple bmps Use this bitmap to estimate number of flows

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multiple bmps Use this bitmap to estimate number of flows

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multires. bmp Problem: must update up to three bitmaps per packet Solution: combine bitmaps into one OR

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 HASH(pink)= Bitmap counting – multires. bmp

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multires. bmp HASH(yellow)=

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Basic estimates Direct bitmap Virtual bitmap

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Multiresolution bitmap estimate Find most accurate component Estimate number of flows hashing to it Apply scaling factor

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Relative error in estimates Direct bitmap Virtual bitmap Multiresolution bitmap

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Error of virtual bitmap Flow density (flows/bit) Average (relative) error

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Memory requirements Direct bitmap< N / ln (Nε 2 +1) Virtual bitmap1.5441/ ε 2 Multiresolution bitmap ln (Nε 2 ) / ε 2 +ct.

Bitmap algorithms for flow counting – Internet Measurement Conference, October million flows, error 1% Hash table*1.21 Gbytes Direct bitmap1.29 Mbytes Virtual bitmap*1.88 Kbytes Multiresolution bitmap10.33 Kbytes

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Triggered bitmap Need multiple instances of counting algorithm (e.g. port scan detection) Many instances count few flows Triggered bitmap –Allocate small direct bitmap to new sources –If number of bits set exceeds trigger value, allocate large multiresolution bitmap

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Related work Flajolet, Martin (1985) probabilistic counting –Memory use similar to multiresolution bitmap Whang et al (1990) introduce direct bitmap You, Chang (1996) use virtual bitmap Chauduri, Motwani, Narasayya (1998) –Counting flows without bias impossible from sampled data Duffield, Lund, Thorup (2002) –Accurate solutions based on counting TCP SYN flags

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Multires. bmp. vs. prob. counting Number of flows (log scale) Average (relative) error

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Scan detection memory usage Interval length Snort (naïve) Probabilistic counting Triggered bitmap 12 seconds1.94 M2.42 M0.37 M 600 seconds49.60 M22,34 M5.59 M

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 A family of counting algorithms SettingAlgorithmApplications General countingMultiresolution bmp.Track infections Narrow rangeVirtual bitmapTriggers (e.g. DoS) Small counts commonTriggered bitmapPort scans StationarityAdaptive bitmapMeasurement Add and deleteIncrement-decrementScheduling

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting algorithms A family of algorithms that can be used as building blocks in various systems Algorithms can be adapted to application Low memory and per packet processing –With 2Kbytes error around 1%

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 The end Bitmap algorithms will be available at: Any questions? Acknowledgements: Vern Paxson, David Moore, Philippe Flajolet, Marianne Durand, Alex Snoeren, K Claffy, Stefan Savage, Florin Baboescu, NIST,NSF

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Adaptive bitmap Virtual bitmap measures accurately number of flows if range known in advance Often number of flows does not change rapidly Measurement repeated Can use previous measurement to tune virtual bitmap Combine a large virtual bitmap with a small multiresolution bitmap used for tuning

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Adaptive bitmap accuracy Number of flows (log scale) Average (relative) error

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 With 2 kilobytes of memory With 2 kilobytes of memory Adaptive bitmap (min avg max) Probabilistic counting (min avg max) Trace1-4.4% 1.1% 4.7%-9.5% 2.8% 13.3% Trace2-1.9% 0.7% 2.0%-6.9% 2.8% 7.6% Trace3-1.8% 0.6% 1.8%2.4% 10.2% 17.7%

Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Increment-decrement algorithms Active flow defined as flow with packets in queue Must support additions and deletions Replace bits of bitmap with counters –Increment when packet arrives –Decrement when packet leaves –Estimate number of flows based on zero counters