1. 1 (Ab-) using Round Robin Databases with Perl Mike Schilli, Yahoo! 06/18/2008

2. 2 Overview What are Round Robin Databases? RRD and Perl – which module to use? Practical Examples

3. 3 Round Robin Databases Archives of fixed size for unlimited data Overwrite old spots if full

4. 4 Round Robin Archives start

5. 5 Round Robin Archives Overwrite if full

6. 6 Round Robin Archives RRDs with multiple archives with different granularity

7. 7 Data Flow with rrdtool Data Source Archive

8. 8 Data Source Conversions

9. 9 Steps and Heartbeat

10. 10 Steps and Heartbeat

11. 11 Steps and Heartbeat

12. 12 Generous Heartbeat

13. 13 Generous Heartbeat

14. 14 Steps and Heartbeat Long story short: Heartbeat > Step if you’re roughly feeding in “step” intervals. Example: Step=300 (5 min) Heartbeat=600

15. 15 Feeding PDPs to the Archive

16. 16 Feeding PDPs to the Archive Consolidation Functions (MAX, AVERAGE, etc.) xfiles factor (allowed percentage of missing values)

17. 17

18. 18 Simple Example Web Orders: 0:00100 1:00110 2:00120 3:00130 4:00140 5:00150

19. 19 Simple Example

20. 20 Simple Example rrdtool 'create' 'myrrdfile.rrd' '--start' '1211871600' '--step' '3600' 'DS:orders:GAUGE:7200:U:U' 'RRA:MAX:0.5:1:100' rrdtool 'update' 'myrrdfile.rrd' '1211875200:100' rrdtool 'update' 'myrrdfile.rrd' '1211878800:110' rrdtool 'update' 'myrrdfile.rrd' '1211882400:120' rrdtool 'update' 'myrrdfile.rrd' '1211886000:130' rrdtool 'update' 'myrrdfile.rrd' '1211889600:140' rrdtool 'update' 'myrrdfile.rrd' '1211893200:150' rrdtool 'graph' 'mygraph.png' '--vertical-label' 'Orders per Hour' '--end' '1211893200' '--start' '1211871600' 'DEF:draw1=myrrdfile.rrd:orders:MAX' 'AREA:draw1#0000FF:Web Orders

21. 21 RRDTool Arcane syntax Perl API RRDs.pm Pretty close to RRDTool syntax Calls rrdtool under the hood

22. 22 To the Rescue: RRDTool::OO CPAN Module RRDTool::OO Takes the drudgery out of the RRD syntax Does what you want by default Translates between Perl and RRD (watch it with debug on) Named parameters instead of fixed order

23. 23 Simple Example #!/usr/local/bin/perl -w use strict; use RRDTool::OO; use DateTime; my $rrd = RRDTool::OO->new( file => "myrrdfile.rrd" );

24. 24 Simple Example my $start_dt = DateTime->today( time_zone => "local" ); my $dt = $start_dt->clone();

25. 25 Simple Example $rrd->create( step => 3600, start => $start_dt->epoch(), data_source => { name => "orders", type => "GAUGE", }, archive => { rows => 100 } );

26. 26 Simple Example for my $hour (0..5) { $dt->add(hours => 1); $rrd->update( time => $dt->epoch(), value => 100 + $hour*10, ); }

27. 27 Simple Example $rrd->graph( image => "mygraph.png", vertical_label => 'Orders per Hour', start => $start_dt->epoch(), end => $dt->epoch(), draw => { type => "area", color => '0000FF', legend => "Web Orders", } );

28. 28 Simple Example

29. 29 RRDTool Commands rrdtool 'create' 'myrrdfile.rrd' '--start' '1211871600' '--step' '3600' 'DS:orders:GAUGE:7200:U:U' 'RRA:MAX:0.5:1:100' rrdtool 'update' 'myrrdfile.rrd' '1211875200:100' rrdtool 'update' 'myrrdfile.rrd' '1211878800:110' rrdtool 'update' 'myrrdfile.rrd' '1211882400:120' rrdtool 'update' 'myrrdfile.rrd' '1211886000:130' rrdtool 'update' 'myrrdfile.rrd' '1211889600:140' rrdtool 'update' 'myrrdfile.rrd' '1211893200:150' rrdtool 'graph' 'mygraph.png' '--vertical-label' 'Orders per Hour' '--end' '1211893200' '--start' '1211871600' 'DEF:draw1=myrrdfile.rrd:orders:MAX' 'AREA:draw1#0000FF:Web Orders

30. 30 Where RRDtool shines Graph time-based data Don’t worry about scaling or axis text Stacked graphs

31. 31 Where RRDtool sucks Can’t add data for past events Can’t add data twice at the same timestamp

32. 32 Pitfalls Make sure that the first update happens after RRD “start” time Updates need to happen at increasing timestamps On large step sizes (1 year), limit the number of rows to fit within < 2030.

33. 33 Debugging with Log4perl use Log::Log4perl qw(:easy) Log::Log4perl-> easy_init( $DEBUG );

34. 34 Log4perl Output rrdtool 'create' 'myrrdfile.rrd' '--start' '1211871600' '--step' '3600' 'DS:orders:GAUGE:7200:U:U' 'RRA:MAX:0.5:1:100' rrdtool 'update' 'myrrdfile.rrd' '1211875200:100' rrdtool 'update' 'myrrdfile.rrd' '1211878800:110' rrdtool 'update' 'myrrdfile.rrd' '1211882400:120' rrdtool 'update' 'myrrdfile.rrd' '1211886000:130' rrdtool 'update' 'myrrdfile.rrd' '1211889600:140' rrdtool 'update' 'myrrdfile.rrd' '1211893200:150' rrdtool 'graph' 'mygraph.png' '--vertical-label' 'Orders per Hour' '--end' '1211893200' '--start' '1211871600' 'DEF:draw1=myrrdfile.rrd:orders:MAX' 'AREA:draw1#0000FF:Web Orders

35. 35 Example: Stock Portfolio

36. 36 Example: Measure Power Usage Mastech MAS-345 Serial Interface (RS-232)

37. 37 Example: Measure Power Usage

38. 38 Example: Measure Power Usage

39. 39 Aberrant Behaviour

40. 40 The End Thanks!

41. 41 References RRDTool::OO on CPAN RRD::Simple on CPAN Portfolio Rendering article: http://perlmeister.com/talks/lm-200805.pdf RRDTool article: http://www.linux- magazine.com/issue/44/Perl_RDDtool.pdfhttp://www.linux- magazine.com/issue/44/Perl_RDDtool.pdf RRDTool article (in German): http://www.linux- magazin.de/Artikel/ausgabe/2004/06/perl/perl.html

42. 42 Bonus Slides

43. 43 Aberrant Behaviour Holt Winter Algorithm Data Smoothing/Prediction

44. 44 Aberrant Behaviour Holt Winter Algorithm RRAs: HWPREDICT SEASONAL DEVSEASONAL DEVPREDICT FAILURES RRA:HWPREDICT:rows:alpha:beta:sea sonal_period

45. 45 Aberrant Behaviour alpha (baseline): 0..1 1: recent observations carry greater weight 0: past observations carry greater weight beta (slope): 0..1 gamma (seasonal): 0..1 failures defaults to 7

46. 46 Aberrant Behaviour Example: alpha = 0.50 beta = 0.50 gamma = 0.01 seasonal_period = 2 threshold = 7 window_length = 9