1 Global Analysis of Arthropod Evolution – a successful grid project Craig A. Stewart, Rainer Keller, Matthias Hess, Uwe Woessner, Martin Aumüller, Matthias Müller, Richard Repasky, David Hart, Huian Li, Donald K. Berry University Information Technology Services, Indiana University High Performance Computing Center Stuttgart And many other contributors… © Copyright Trustees of Indiana University 2004

License Terms Please cite this presentation as: Stewart, C.A., R. Keller, M. Hess, U. Wössner, M. Aumüller, M. Müller, R. Repasky, D. Hart, H. Li and D.K. Berry. Global grid analysis of arthropod evolution – a successful grid project Presentation. Presented at: 7th HLRS Metacomputing and GRID Workshop (Stuttgart, Germany, 26 Apr 2004). Available from: Portions of this document that originated from sources outside IU are shown here and used by permission or under licenses indicated within this document. Items indicated with a © or denoted with a source url are under copyright and used here with permission. Such items may not be reused without permission from the holder of copyright except where license terms noted on a slide permit reuse. Except where otherwise noted, the contents of this presentation are copyright 2004 by the Trustees of Indiana University. This content is released under the Creative Commons Attribution 3.0 Unported license ( This license includes the following terms: You are free to share – to copy, distribute and transmit the work and to remix – to adapt the work under the following conditions: attribution – you must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work). For any reuse or distribution, you must make clear to others the license terms of this work.

3 Outline The SCxy conference and the HPC Challenge The biological problem The software used The global grid The results! Acknowledgements

4 The SCxy conference and the HPC Challenge Supercomputing Conference (sponsored by ACM and IEEE) High Performance Challenge –demonstrates new capabilities in advanced computing systems –(or sometimes silly supercomputer tricks)

5 Biological problem Are Hexapods a single evolutionary group? Are ecdysozoans a single evolutionary group?

6 A partial bestiary All organism illustrations copyright Jennifer Fairman, Used by agreement

7 Software and data analysis Non-grid preparatory work –Download sequences from NCBI (67 Taxa, 12,162 bp, mitochondrial genes for 12 proteins) –Align sequences with Multi-Clustal –Determine rate parameters with TreePuzzle Grid preparatory work –Analyze performance of fastDNAml with Vampir –Meetings via Access Grid & CoVise The grid software –PACXMPI – Grid/MPI middleware –Covise – Collaboration and visualization –fastDNAml – Maximum Likelihood phylogenetics

8 A project of HLRS (High Performance Computing Center Stuttgart) PACX-MPI (PArallel Computer eXtension) enables seamlessly execution of MPI-conforming parallel applications on a Grid. Application recompiled and linked w. PACX-MPI. Communication between MPI processes internally is done with the vendor MPI, while communication to other parts of the Metacomputer is done via the connecting network. Key advantages: –Optimized vendor MPI library is used. –Two daemons (MPI processes) take care of communication between systems – allows bundling of communication.

9 COVISE COllaborative VIsualization and Simulation Environment A project of HLRS (High Performance Computing Center Stuttgart) Focus on collaborative and interactive use of supercomputers Interactive startup of calculation on a Computational Grid Real-Time visualization of the results and the performance of computation.

10 fastDNAml ML analysis of phylogenetic trees based on DNA sequences Foreman/worker MPI program Heuristic search for best trees For 67 taxa: 2.12 ~ trees Goal: 300 bootstraps, 10 jumbles per – 3000 executions (more than 3x typical!)

11 Why this project on a grid? Important & time-sensitive biological question requiring massive computer resources A biologically-oriented code that scales well Grid middleware environment & collaboration tool well suited to the task at hand Opportunity to create a grid spanning every continent on earth (except Antarctica)

12 The metacomputers OneOrigin Spain Linux cluster 64Japan Linux cluster 12Australia IBM SP 32US TwoT3E128Germany IBM SP 64US Dec Alpha 4Brazil Sun fire Singapore ThreeHitachi SR Germany Cray T3E 128 UK Cray T3E 32US IBM SP (Blue Horiz) 32US FourDec Alpha (Lemieux) 64US FiveLinux system 1Tunisia Five functional units; 8 types of systems (several on Top500 list); 6+ vendors; 641 processors; 9 countries, 6 continents

13

14 The results ~200 trees were analyzed during the course of the week The biological results are still being analyzed Our HPC challenge project was awarded the prize for “ Most geographically distributed application ”

15 Things we learned Proper alignment of parallelism coarseness and network speeds was important There was real value to the use of the metacomputer concept within the overall grid You can distribute a lot of machine computations, but less of the human work. (=>simplicity is a virtue) There are today few large scale grids delivering computational services for biological computation in a persistent fashion. The temporary grid we created ranks as one of the larger grids ever created for biological computing

16 For further information fastDNAml: PACXMPI: COVISE: HLRS: UITS: uits.iu.edu Center for Genomics and Bioinformatics: SCxy: about.uits.iu.edu/divisions/rac/index.html about.uits.iu.edu/divisions/rac/pubsstaff.html ingen.iu.edu it.iu.edu

17 Acknowledgments This research was supported in part by the Indiana Genomics Initiative. The Indiana Genomics Initiative of Indiana University is supported in part by Lilly Endowment Inc. This work was supported in part by Shared University Research grants from IBM, Inc. to Indiana University. This material is based upon work supported by the National Science Foundation under Grant No and Grant No. CDA Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors) and do not necessarily reflect the views of the National Science Foundation (NSF). Assistance with this presentation: John Herrin, Malinda Lingwall, W. Les Teach Thanks to the SciNet team and SC2003 organizers! This project was an outcome of a kind invitation from Prof. Dr. Michael Resch and HLRS to Craig Stewart last year.

18 Our partners

19 Rainer Keller, Matthias Hess HLRS, University of Stuttgart Richard Repasky UITS, Indiana University John Colbourne Center for Genomics and Informatics, Indiana University Craig Stewart, David HartUITS, Indiana University Jennifer Steinbachs Center for Genomics and Bioinformatics, Indiana University Uwe Woessner HLRS, University of Stuttgart Donald BerryUITS, Indiana University Matthias MuellerHLRS, University of Stuttgart Huian LiUITS, Indiana University Gary W. Stuart Center for Genomics and Bioinformatics, Indiana University Michael ReschHLRS, University of Stuttgart Eric Wernert UITS, Indiana University Martin Aumüller, Ulrich LangHLRS, University of Stuttgart Markus Buchhorn Australia National University Hiroshi Takemiya National Institute of Advanced Industrial Science & Technology, Japan Rim Belhaj ISET'Com, Tunesia Wolfgang E. Nagel ZHR, Technical University of Dresden Sergui Sanielevici Pittsburgh Supercomputing Center Sergio takeo KofujiLCCA/CCE-USP David BannonVictorian Partnership for Advanced Computing, Australia Norihiro Nakajima Japan Atomic Energy Research Institute Rosa Badia CEPBA-IBM Research Institute Mark A. Miller San Diego Supercomputer Center Hyungwoo ParkKorea Institute of Science and Technology Information Rick Stevens Argonne National Laboratory Fang-Pang Lin National Center for High Performance Computing John Brooke Manchester Computing David Moffett Purdue University Tan Tin WeeNational University of Singapore Greg Newby Arctic Region Supercomputer Center J.C.T. Poole CACR, Cal-Tech Ramched Hamza Sup'com, Tunesia Mary Papakhian, John N. HuffmanUITS, Indiana University Leigh GrundhoefferUITS, Indiana University Ray SheppardUITS, Indiana University Peter Cherbas Center for Genomics and Bioinformatics, Indiana U. Stephen Pickles, Neil StringfellowCSAR, University of Manchester

20 Thank you! Questions?