Download presentation
Presentation is loading. Please wait.
Published byAdam Power Modified over 11 years ago
1
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap Algorithms for Counting Active Flows on High Speed Links Cristian Estan, George Varghese, Mike Fisk Computer Science and Engineering Department, University of California, San Diego
2
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Why count flows? Detect port/IP scans Identify DoS attacks Estimate spreading rate of a worm Packet scheduling Dave Plonkas FlowScan
3
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Existing flow counting solutions Server NetFlow data Analysis Traffic reports Network Operations Center Router Fast link Memory Network Memory size & bandwidth Networkbandwidth
4
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Motivating question Can we count flows at line speeds at the router? –Wrong solution – counters –Naïve solution – use hash tables (like NetFlow) –Our approach – use bitmaps
5
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting algorithms A family of algorithms that can be used as building blocks in various systems Algorithms can be adapted to application Low memory and per packet processing Generalize flows to distinct header patterns –Count flows or source addresses to detect attack –Count destination address+port pairs to detect scan
6
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions
7
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(green)=10001001 Set bits in the bitmap using hash of the flow ID of incoming packets
8
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(blue)=00100100 Different flows have different hash values
9
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(green)=10001001 Packets from the same flow always hash to the same bit
10
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(violet)=10010101 Collisions OK, estimates compensate for them
11
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(orange)=11110011
12
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(pink)=11100000
13
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap HASH(yellow)=01100011 As the bitmap fills up, estimates get inaccurate
14
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap Solution: use more bits HASH(green)=10001001
15
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – direct bitmap Solution: use more bits Problem: memory scales with the number of flows HASH(blue)=00100100
16
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – virtual bitmap Solution: a) store only a portion of the bitmap b) multiply estimate by scaling factor
17
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – virtual bitmap HASH(pink)=11100000
18
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – virtual bitmap HASH(yellow)=01100011 Problem: estimate inaccurate when few flows active
19
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multiple bmps Solution: use many bitmaps, each accurate for a different range
20
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multiple bmps HASH(pink)=11100000
21
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multiple bmps HASH(yellow)=01100011
22
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multiple bmps Use this bitmap to estimate number of flows
23
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multiple bmps Use this bitmap to estimate number of flows
24
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multires. bmp Problem: must update up to three bitmaps per packet Solution: combine bitmaps into one OR
25
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 HASH(pink)=11100000 Bitmap counting – multires. bmp
26
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting – multires. bmp HASH(yellow)=01100011
27
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions
28
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Basic estimates Direct bitmap Virtual bitmap
29
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Multiresolution bitmap estimate Find most accurate component Estimate number of flows hashing to it Apply scaling factor
30
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions
31
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Relative error in estimates Direct bitmap Virtual bitmap Multiresolution bitmap
32
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Error of virtual bitmap Flow density (flows/bit) Average (relative) error
33
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Memory requirements Direct bitmap< N / ln (Nε 2 +1) Virtual bitmap1.5441/ ε 2 Multiresolution bitmap0.9186 ln (Nε 2 ) / ε 2 +ct.
34
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 100 million flows, error 1% Hash table*1.21 Gbytes Direct bitmap1.29 Mbytes Virtual bitmap*1.88 Kbytes Multiresolution bitmap10.33 Kbytes
35
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions
36
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Triggered bitmap Need multiple instances of counting algorithm (e.g. port scan detection) Many instances count few flows Triggered bitmap –Allocate small direct bitmap to new sources –If number of bits set exceeds trigger value, allocate large multiresolution bitmap
37
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions
38
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Related work Flajolet, Martin (1985) probabilistic counting –Memory use similar to multiresolution bitmap Whang et al (1990) introduce direct bitmap You, Chang (1996) use virtual bitmap Chauduri, Motwani, Narasayya (1998) –Counting flows without bias impossible from sampled data Duffield, Lund, Thorup (2002) –Accurate solutions based on counting TCP SYN flags
39
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions
40
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Multires. bmp. vs. prob. counting Number of flows (log scale) Average (relative) error
41
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Scan detection memory usage Interval length Snort (naïve) Probabilistic counting Triggered bitmap 12 seconds1.94 M2.42 M0.37 M 600 seconds49.60 M22,34 M5.59 M
42
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Talk structure Per packet processing for bitmap algorithms Computing flow count estimates from bitmaps Variance analysis of estimates Derived algorithms Related work Measurements Conclusions
43
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 A family of counting algorithms SettingAlgorithmApplications General countingMultiresolution bmp.Track infections Narrow rangeVirtual bitmapTriggers (e.g. DoS) Small counts commonTriggered bitmapPort scans StationarityAdaptive bitmapMeasurement Add and deleteIncrement-decrementScheduling
44
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Bitmap counting algorithms A family of algorithms that can be used as building blocks in various systems Algorithms can be adapted to application Low memory and per packet processing –With 2Kbytes error around 1%
45
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 The end Bitmap algorithms will be available at: http://ial.ucsd.edu/bitmaps/ Any questions? Acknowledgements: Vern Paxson, David Moore, Philippe Flajolet, Marianne Durand, Alex Snoeren, K Claffy, Stefan Savage, Florin Baboescu, NIST,NSF
46
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Adaptive bitmap Virtual bitmap measures accurately number of flows if range known in advance Often number of flows does not change rapidly Measurement repeated Can use previous measurement to tune virtual bitmap Combine a large virtual bitmap with a small multiresolution bitmap used for tuning
47
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Adaptive bitmap accuracy Number of flows (log scale) Average (relative) error
48
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 With 2 kilobytes of memory With 2 kilobytes of memory Adaptive bitmap (min avg max) Probabilistic counting (min avg max) Trace1-4.4% 1.1% 4.7%-9.5% 2.8% 13.3% Trace2-1.9% 0.7% 2.0%-6.9% 2.8% 7.6% Trace3-1.8% 0.6% 1.8%2.4% 10.2% 17.7%
49
Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003 Increment-decrement algorithms Active flow defined as flow with packets in queue Must support additions and deletions Replace bits of bitmap with counters –Increment when packet arrives –Decrement when packet leaves –Estimate number of flows based on zero counters
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.