1. Studying Protein Folding on the Grid: Experiences Using CHARMM on NPACI Resources under Legion University of Virginia Anand Natrajan Marty A. Humphrey Anthony D. Fox Andrew S. Grimshaw Scripps (TSRI) Michael Crowley Charles L. Brooks III SDSC Nancy Wilkins-Diehr http://legion.virginia.edu anand@virginia.edu

2. Outline CHARMM –Issues Legion The Run –Results –Lessons Portals Summary

3. CHARMM Routine exploration of folding landscapes helps in search for protein folding solution Understanding folding critical to structural genomics, biophysics, drug design, etc. Key to understanding cell malfunctions in Alzheimer’s, cystic fibrosis, etc. CHARMM and Amber benefit majority (>80%) of bio-molecular scientists Structural genomic & protein structure predictions

4. Folding Free Energy Landscape Molecular Dynamics Simulations 100-200 structures to sample (r,R gyr ) space R gyr 

5. Folding of Protein L Immunoglobulin-binding protein –62 residues (small), 585 atoms –6500 water molecules, total 20085 atoms –Each parameter point requires O(10 6 ) dynamics steps –Typical folding surfaces require 100-200 sampling runs CHARMM using most accurate physics available for classical molecular dynamics simulation Multiple 16-way parallel runs - maximum efficiency

6. Application Characteristics Parameter-space study –Parameters correspond to structures along & near folding path Path unknown - could be many or broad –Many places along path sampled for determining local low free energy states –Path is valley of lowest free energy states from high free energy state of unfolded protein to lowest free energy state (folded native protein)

7. Application Characteristics Many independent runs –200 sets of data to be simulated in two sequential runs Equilibration (4-8 hours) Production/sampling (8 to 16 hours) Each point has task name, e.g., pl_1_2_1_e

8. Legion Complete, Integrated Infrastructure for Secure Distributed Resource Sharing

9. Grid OS Requirements Wide-area High Performance Complexity Management Extensibility Security Site Autonomy Input / Output Heterogeneity Fault-tolerance Scalability Simplicity Single Namespace Resource Management Platform Independence Multi-language Legacy Support

10. Transparent System

11. npacinet

12. The Run

13. Computational Issues Provide improved response time Access large set of resources transparently –geographically distributed –heterogeneous –different organisations 6 organisations 6 queue types 10 queues 6 architectures ~1000 processors

14. IBM Blue Horizon SDSC 375MHz Power3 512/1184 IBM Blue Horizon SDSC 375MHz Power3 512/1184 Resources Available HP SuperDome CalTech 440 MHz PA-8700 128/128 HP SuperDome CalTech 440 MHz PA-8700 128/128 IBM SP3 UMich 375MHz Power3 24/24 IBM SP3 UMich 375MHz Power3 24/24 IBM Azure UTexas 160MHz Power2 32/64 IBM Azure UTexas 160MHz Power2 32/64 Sun HPC 10000 SDSC 400MHz SMP 32/64 Sun HPC 10000 SDSC 400MHz SMP 32/64 DEC Alpha UVa 533MHz EV56 32/128 DEC Alpha UVa 533MHz EV56 32/128

15. Scientists Using Legion Binaries for each type Script for dispatching jobs Script for keeping track of results Script for running binary at site –optional feature in Legion Abstract interface to resources –queues, accounting, firewalls, etc. Binary transfer (with caching) Input file transfer Job submission Status reporting Output file transfer

16. Mechanics of Runs Legion Register binaries Create task directories & specification Dispatch equilibration Dispatch equilibration & production

17. Distribution of CHARMM Work

18. LEGION Network slowdowns –Slowdown in the middle of the run –100% loss for packets of size ~8500 bytes Site failures –LoadLeveler restarts –NFS/AFS failures Legion –No run-time failures –Archival support lacking –Must address binary differences Problems Encountered UVaSDSCUMich 01101

19. Successes Science accomplished faster –1 month on 128 SGI Origins @Scripps –1.5 days on national grid with Legion Transparent access to resources –User didn’t need to log on to different machines –Minimal direct interaction with resources Problems identified Legion remained stable –Other Legion users unaware of large runs Large grid application run at powerful resources by one person from local resource Collaboration between natural and computer scientists

20. Portal Interface Easy Interface to Grid

21. Simple point-and-click interface to Grids –Familiar access to distributed file system –Enables & encourages sharing Application portal model for HPC –AmberGrid –RenderGrid –Accounting Legion GUIs Transparent Access to Remote Resources Intended Audience is Scientists

22. Logging in to npacinet

23. View of contexts (Distributed File System)

24. Control Panel

25. Running Amber

26. Run Status (Legion) Graphical View (Chime)

27. Summary CHARMM Run –Succeeded in starting big runs –Encountered problems –Learnt lessons for future AmberGrid –Showed proof-of-concept - grid portal –Need to resolve licence issues