1. Tim Dyce Australia-ATLAS tjdyce@unimelb.edu.au Experiences from the other end of the network

2. About Australia-ATLAS The Australia-ATLAS site is located in Melbourne, Victoria Predominately an ATLAS site Also services some new local VOs and the biomed VO One of the furthest sites from Europe; posing many challenges Our distance makes us a canary for issues in the middleware

3. Our network path is long The longer the path the more vulnerable By and large our network links are very stable BDII Challenges – Network Fragility But.... Network links have many enemies Fibre cuts cause changes in routing, which can affect contact with BDIIs at other sites

4. BDII Challenges – Network Latency

5. The Old BDII Design Search Breathe Populate Active Search Breathe Populate Active 21722173 Search ldapsearch Breathe Populate Active 2172 Search Breathe Populate Active 2173 Search Breathe Populate 2173 Search 2172 Search Breathe Populate Active ldapsearch 2170 LDAP queries Time

6. Network Fragility & Latency Problems Problems The old BDII forked n ldapsearch processes during update – Forks fill up waiting for non-contactable sites – Forks take longer to complete on high latency networks The BDII may not sufficient time to contact all sites Results in an incomplete set of information in the BDII Solutions Increase the number of forks Lengthen the search phase of the cycle Search Breathe Populate Active Search ldapsearch

7. Network Fragility & Latency Problems Flow-on Problems At most European sites the search phase finished quickly, before the update time limit The update and breathe processes had more time to complete When the search time was longer the BDII displayed odd behaviour – Database seemed to be queried too early after start-up – Cached an empty result for everything – Underlying database contained data, but the BDII appeared empty Solution Lengthen the breathe phase of the update cycle

8. BDII Issues Design LDAP is a directory, not a database – Optimised for high performance read not write Constantly replacing the content is inefficient Distribution bundling BDII uses the BDB database OpenLDAP is particular about which versions of BDB it; – Is stable with, especially under high load – Performs well with Distributions bundle the BDB version which is current stable, not the version which will work best with OpenLDAP

9. OpenLDAP & BDB OpenLDAP versions OpenLDAP 2.2 (SL4) – Only works with BDB version 4.2 – Prefers some sub-versions over others OpenLDAP 2.4 (SL5) – Works with many versions of BDB – Is less particular about versions, but still works better with particular sub-versions The BDB database Is very high performing, but only if it is tuned well – To perform under high load, it's settings must be tuned to match the database size and structure it will be serving – It may be better if BDII were distributed with it's own version of BDB

10. The New BDII Design Search Update Active ldapsearch 2170 LDAP queries Time Search Update ldapsearch Search Update ldapsearch

11. The New BDII Testing the new BDII We have been testing the new BDII since May By design it; – Forks better for searching – Only updates what is necessary – Is much more stable Have only seen one crash, but under extremely high load – This may best be mitigated by haship rate limiting in the firewall

12. Visions for the future Where to from here? Now we have an information system which does not purge it's information with each cycle... Data retention Setup different retention levels for cached information; potentially only purging information when it has not been seen for several cycles, this keeps the BDII populated at sites with poor network and lots of network errors One LDAP to rule them all The information providers could update the site BDII directly, cutting out the middle-man and increasing freshness of the information

13. Visions for the future Syncrepl OpenLDAP 2.4 (under SL5) has a better synchronisation mechanism; syncrepl. This has potential to be very useful: – Under load balanced setups a single master site BDII can automatically replicate to slaves keeping their information actively consistent – May be useful to synchronise BDIIs within regions and between sites – Can be setup in a tiered design, and could be investigated as a next generation information distribution method

14. Summary Australia-ATLAS is a canary to issues within the middleware We are agnostic as to the information system solution LDAP is a good solution to the problem – But only if it is setup and tuned very carefully The new version of the BDII has been a massive improvement We are continuing to work with Laurence Field in testing the BDII

15. © Copyright The University of Melbourne 2009 Tim Dyce tjdyce@unimelb.edu.au