1. A Global Grid for Analysis of Arthropod Evolution Craig A. Stewart, Rainer Keller, Richard Repasky, Matthias Hess, David Hart, Matthias Müller, Ray Sheppard, Uwe Wössner, Martin Aumüller, Huian Li, Donald K. Berry, John Colbourne Indiana University – University Information Technology Services Höchstleistungsrechnencentrum Stuttgart (High Performance Computing Center Stuttgart) Indiana University – Center for Genomics and Bioinformatics

2. License Terms Please cite this presentation as: Stewart, C.A., R. Keller, R. Repasky, M. Hess, D. Hart, M. Müller, R. Sheppard, U. Wössner, M. Aumüller, H. Li, D.K. Berry and J. Colbourne. A Global Grid for Analysis of Arthropod Evolution. 2004. Presentation. Presented at: Grid2004 - 5th IEEE/ACM International Workshop on Grid Computing (Pittsburgh, PA, 8 Nov 2004). Available from: http://hdl.handle.net/2022/14784http://hdl.handle.net/2022/14784 Portions of this document that originated from sources outside IU are shown here and used by permission or under licenses indicated within this document. Items indicated with a © or denoted with a source url are under copyright and used here with permission. Such items may not be reused without permission from the holder of copyright except where license terms noted on a slide permit reuse. Except where otherwise noted, the contents of this presentation are copyright 2004 by the Trustees of Indiana University. This content is released under the Creative Commons Attribution 3.0 Unported license (http://creativecommons.org/licenses/by/3.0/). This license includes the following terms: You are free to share – to copy, distribute and transmit the work and to remix – to adapt the work under the following conditions: attribution – you must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work). For any reuse or distribution, you must make clear to others the license terms of this work.

3. Outline The biological problem The software used The global grid What we learned Acknowledgements

4. Biological problem Are Hexapods (animals with six legs) a single evolutionary group? Are ecdysozoans (animals that shed their skins) a single evolutionary group?

6. Phylogenetic inference Goal – reconstruct evolutionary history by comparison of DNA sequences NP-hard problem Heuristic approach used in maximum likelihood inference Data are available; analysis had never been attempted due to computational demands

7. Why this project on a grid? Important & time-sensitive biological question requiring massive computer resources A biologically-oriented code that scales well Grid middleware environment & collaboration tool well suited to the task at hand Opportunity to create a grid spanning every continent on earth (except Antarctica)

8. Software and data analysis Non-grid preparatory work –Download sequences from NCBI (67 Taxa, 12,162 bp, mitochondrial genes for 12 proteins) –Align sequences with Multi-Clustal –Determine rate parameters with TreePuzzle Grid preparatory work –Analyze performance of fastDNAml with Vampir –Meetings via Access Grid & CoVise The grid software –PACXMPI – Grid/MPI middleware –Covise – Collaboration and visualization –fastDNAml – Maximum Likelihood phylogenetics

9. fastDNAml ML analysis of phylogenetic trees based on DNA sequences Foreman/worker MPI program Fault tolerance for grid computing built into program since 1998 For 67 taxa: 2.12 ~10 109 trees Goal: 300 bootstraps, 10 jumbles per – 3000 executions (more than 3x typical!)

10. PACX-MPI (PArallel Computer eXtension) enables seamlessly execution of MPI-conforming parallel applications on a Grid. Application recompiled and linked w. PACX-MPI. Communication between MPI processes internally is done with the vendor MPI, while communication to other parts of the Metacomputer is done via the connecting network. Key advantages: –Optimized vendor MPI library is used. –Two daemons (MPI processes) take care of communication between systems – allows bundling of communication.

11. COVISE COllaborative VIsualization and Simulation Environment. Focus: collaborative & interactive use of supercomputers Interactive startup of calculation on Grid Real-Time visualization of the results

12. Application framework Work of Matthias Hess, HLRS

14. GleiderfüsslerGrid

15. The Metacomputers OneSGI Origin 200032CEBPA (Spain) Linux cluster 64AIST (Japan) Linux cluster 12ANU (Australia) TwoT3E 128HLRS (Germany) IBM SP 64IUB (US) Dec Alpha 4USP (Brazil) Sunfire 6800 16NUS (Singapore) ThreeHitachi SR8000 32Germany Cray T3E 128 MCC (UK) Cray T3E 32PSC (US) IBM SP (Blue Horizon) 32SDSC (US) FourDec Alpha (Lemieux) 64PSC (US) FiveLinux system 1ISET’com (Tunisia) 8 types of systems (several on Top500 list & TeraGrid); 6+ vendors; 641 processors; 9 countries; 6 continents

16. Results of one run

17. Conclusions Results –The grid actually worked (HPC Challenge award) –Real science was done (500 runs, 5,318,281 trees analyzed, 7800 CPU hours used) Lessons learned –Access Grid was essential –CVS is good –Importance of fault tolerance & interaction of fault tolerance with network speeds –Importance of the grid frameworks –Firewall issues & value of PACX-MPI Going forward –The key value of the grid approach was in reducing wall-clock time to amounts tolerable for the application scientists!

18. Acknowledgments This research was supported in part by the Indiana Genomics Initiative. The Indiana Genomics Initiative of Indiana University is supported in part by Lilly Endowment Inc. This work was supported in part by Shared University Research grants from IBM, Inc. to Indiana University. This material is based upon work supported by the National Science Foundation under Grant No. 0116050 and Grant No. CDA-9601632. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF). Assistance with this presentation: John Herrin, Malinda Lingwall, W. Les Teach, Jennifer Fairman Thanks to the SciNet team and SC2003 organizers!

19. Jennifer Steinbachs Center for Genomics and Bioinformatics, Indiana University Gary W. Stuart Center for Genomics and Bioinformatics, Indiana University Michael ReschHLRS, University of Stuttgart Eric Wernert UITS, Indiana University Markus Buchhorn Australia National University Hiroshi Takemiya National Institute of Advanced Industrial Science & Technology, Japan Rim Belhaj ISET'Com, Tunesia Wolfgang E. Nagel ZHR, Technical University of Dresden Sergui Sanielevici Pittsburgh Supercomputing Center Sergio takeo KofujiLCCA/CCE-USP David BannonVictorian Partnership for Advanced Computing, Australia Norihiro Nakajima Japan Atomic Energy Research Institute Rosa Badia CEPBA-IBM Research Institute Mark A. Miller San Diego Supercomputer Center Hyungwoo ParkKorea Institute of Science and Technology Information Rick Stevens Argonne National Laboratory Fang-Pang Lin National Center for High Performance Computing John Brooke Manchester Computing David Moffett Purdue University Tan Tin WeeNational University of Singapore Greg Newby Arctic Region Supercomputer Center J.C.T. Poole CACR, Cal-Tech Ramched Hamza Sup'com, Tunesia Mary Papakhian, John N. HuffmanUITS, Indiana University Leigh GrundhoefferUITS, Indiana University Ray SheppardUITS, Indiana University Peter Cherbas Center for Genomics and Bioinformatics, Indiana U. Stephen Pickles, Neil StringfellowCSAR, University of Manchester Arthurina BreckenridgeHLRS, University of Stuttgart

20. Our partners

21. Questions? Be sure to check out the current issue of Communications of the ACM Special Section on Bioinformatics – especially the article “ The Emerging role of BioGrids ”