1. Recent Developments in Directories: Performance Monitoring with “Look” Brendan Bellina, University of Notre Dame Spring 2003 Internet2 Member Meeting

2. Internet2 Member Meeting - April 9, 2003Middleware: Directories 2 Presentation Overview Why Monitor Performance What Performance Metrics To Monitor How To Gather Performance Metrics Tools for Performance Monitoring @ ND –Orca –Orcallator –Look

3. Internet2 Member Meeting - April 9, 2003Middleware: Directories 3 Why Monitor Performance Availability It’s up… isn’t it? –Directories often require 7x24 availability Responsiveness It’s fast enough… maybe. –Directories often require extremely fast response Scalability We can handle that… I think. –Structural changes, indices, increases in the # of entries or # of attributes can all affect performance.

4. Internet2 Member Meeting - April 9, 2003Middleware: Directories 4 What Performance Metrics To Monitor Response time Connection requests Bind requests Bind errors Search requests Search errors Average count & size of search results Directory Cache Hits Directory Cache Tries Bind response time Search response time Current connections Avg connection length Current binds Current searches # Bytes transmitted # Entries transmitted

5. Internet2 Member Meeting - April 9, 2003Middleware: Directories 5 Tools for Performance Monitoring @ ND Orca - http://www.orcaware.comhttp://www.orcaware.com Orcallator (collector for Solaris statistics) Look – LDAP operational Orca “k”ollector

6. Internet2 Member Meeting - April 9, 2003Middleware: Directories 6 Orca (http://www.orcaware.com) About Orca: Generates web friendly performance graphs based on collected statistical data. Free (will take donations) Open Source Perl Used by Yahoo!/Geocities Used at Notre Dame on 100+ servers Hourly, Daily, Weekly, Monthly, Quarterly, Yearly, All Did I say free?

7. Internet2 Member Meeting - April 9, 2003Middleware: Directories 7 ORCA @ Notre Dame CRON job runs every 15 minutes to regen graphs Orcallator (SymbEL script included with ORCA) –Written to monitor web servers and Solaris stats –Requires Sun Solaris SE Toolkit (free from http://www.setoolkit.com) http://www.setoolkit.com –Runs on each directory server to collect statistics – 5 minute averages, 30 second peaks –Data is consolidated on one server for ORCA processing –Modifications by ND to read iPlanet Dir Server 4.16 log

8. Internet2 Member Meeting - April 9, 2003Middleware: Directories 8 Look – LDAP operational Orca "k"ollector LDAP supplement to Orca and/or Orcallator Perl v5.6.0 script - platform independent Produces Orca compatible output files Processes iPlanet 4.16 & SunONE 5.1 SP1 log file Queries directory cn=monitor & cn=ldbm NMIr3 version (requires PerLDAP) code available from http://middleware.internet2.edu/dir/look/ Next version with contributions from Todd Piket of Michigan Tech will be based on Net::LDAP and monitor compare operations. To see this in production go to: https://secure.mtu.edu/docs/restricted/orca_stats/solaris_ldap/

9. Internet2 Member Meeting - April 9, 2003Middleware: Directories 9 Look – LDAP operational Orca "k"ollector Connection Rate Bind Rate Bind Error Rate Search Rate Search Error Rate Search Results Returned Rate Connections Operations Initiated Operations Completed Entries Sent Bytes Sent Entry Cache Hits Entry Cache Tries Cache Hit Ratio Database Cache Hits Database Cache Tries Database Cache Hit Ratio Peaks and Averages View live results at http://eds.nd.edu/perfmon/Look

10. Internet2 Member Meeting - April 9, 2003Middleware: Directories 10 ORCA Look Graph Example #1 LDAP search rate before and after email routing was transferred from email-specific directory to EDS:

11. Internet2 Member Meeting - April 9, 2003Middleware: Directories 11 ORCA Look Graph Example #2: Troubleshooting Daily abuse of open LDAP connections. What is different about the weekend?

12. Internet2 Member Meeting - April 9, 2003Middleware: Directories 12 ORCA Look Graph Example #3: Troubleshooting How long has this been going on? Week 49. What happened in mid-December? Hmmm…

13. Internet2 Member Meeting - April 9, 2003Middleware: Directories 13 ORCA Look Graph Example #4: Troubleshooting The culprit has been identified and shot.

14. Internet2 Member Meeting - April 9, 2003Middleware: Directories 14 ORCA Look Graph Example #5: Troubleshooting The sky is blue, the birds are chirping, and all is right with the world.

15. Internet2 Member Meeting - April 9, 2003Middleware: Directories 15 Links Notre Dame Enterprise Directory Service http://www.nd.edu/~eds Notre Dame EDS Performance Statistics - Orcallator http://www.nd.edu/~eds/perfmon or http://eds.nd.edu/perfmonhttp://www.nd.edu/~eds/perfmonhttp://eds.nd.edu/perfmon Notre Dame EDS Performance Statistics – Look http://www.nd.edu/~eds/perfmon/Look or http://eds.nd.edu/perfmon/Lookhttp://www.nd.edu/~eds/perfmon/Lookhttp://eds.nd.edu/perfmon/Look ORCA software http://www.orcaware.com Look software http://middleware.internet2.edu/dir/look/ Brendan Bellina (mailto:BBellina@nd.edu)mailto:BBellina@nd.edu http://www3.nd.edu/~eds/cgi-bin/nd_ldap_search.pl?ldapfilter=uid=bbellina

16. Internet2 Member Meeting - April 9, 2003Middleware: Directories 16 Copyright Statement Copyright © Brendan Bellina, 2003. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.