Some Grid Experiences Laura Pearlman USC Information Sciences Institute ICTP Advanced Training Workshop on Scientific Instruments on the Grid *Most of these slides are from Lee Liming’s GlobusWorld 2006 presentation “A Globus ® Primer: What is the Grid and How Do I Use It?”

GlobusWORLD 2006Globus Primer2 To be Covered I. Grid computing problems II. Some notable U.S. Grids III. How grids are built and used in real life

GlobusWORLD 2006Globus Primer3 Grid Computing Problems l Scientific problems that are big enough that they require people in several organizations to collaborate and share computing resources, data, and instruments. u Interactive simulation (climate modeling) u Very large-scale simulation and analysis (galaxy formation, gravity waves, battlefield simulation) u Engineering (parameter studies, linked component models) u Experimental data analysis (high-energy physics) u Image and sensor analysis (astronomy, climate study, ecology) u Online instrumentation (microscopes, x-ray devices, etc.) u Remote visualization (climate studies, biology) u Engineering (large-scale structural testing, chemical engineering) u Biomedical applications

GlobusWORLD 2006Globus Primer4 Some Core Problems - Heterogeneity l Different authentication mechanisms across institutions l Different mechanisms for monitoring system and application status across institutions l Different ways to submit jobs l Different ways to store & access files and data l Different ways to keep track of data l Different preferences in programming languages and environments l Difficulty in tracking the causes of failures l Conflicting requirements among groups that need to interoperate

GlobusWORLD 2006Globus Primer5 Some Core Problems - Trust l Rigid use policies (authorization, QoS) vs rigid application assumptions. l Authorization needs to happen at many levels (communities, organizations, resource owners, etc.). l Complicated social structures exceed the abilities of simple authorization systems.

GlobusWORLD 2006Globus Primer6 Some Real-World Grids

GlobusWORLD 2006Globus Primer7 Earth System Grid Goal: Give climate scientists easier access to the distributed data and resources that they require to perform their research. Developed new technologies for (1) creating and operating "filtering servers" capable of performing sophisticated analyses, and (2) delivering results to users.

GlobusWORLD 2006Globus Primer8 Collaborative Engineering: NEES U.Nevada Reno

GlobusWORLD 2006Globus Primer9 UCSD UT UC/ANL NCSA PSC ORNL PU IU A National Science Foundation Investment in Cyberinfrastructure $100M 3-year construction ( ) $150M 5-year operation & enhancement ( ) NSF’s TeraGrid * l TeraGrid DEEP: Integrating NSF’s most powerful computers (60+ TF) u 2+ PB Online Data Storage u National data visualization facilities u World’s most powerful network (national footprint) l TeraGrid WIDE Science Gateways: Engaging Scientific Communities u 90+ Community Data Collections u Growing set of community partnerships spanning the science community. u Leveraging NSF ITR, NIH, DOE and other science community projects. u Engaging peer Grid projects such as Open Science Grid in the U.S. as peer Grids in Europe and Asia-Pacific. l Base TeraGrid Cyberinfrastructure: Persistent, Reliable, National u Coordinated distributed computing and information environment u Coherent User Outreach, Training, and Support u Common, open infrastructure services * Slide courtesy of Ray Bair, Argonne National Laboratory

GlobusWORLD 2006Globus Primer10 Open Science Grid l $30M over five years for effort to u sustain and evolve the distributed facility, u bring on board new communities & capabilities, u educate & train. l OSG hardware resources, applications and many other contributions come from OSG consortium members. l OSG technical work is performed together with collaborators & external projects l OSG has partners in Africa, Asia, Europe, North and South America. Text for this slide courtesy of Ruth Pordes

GlobusWORLD 2006Globus Primer11 OSG Partners l Autralian Partnerships for Advanced Computing (APAC) l Data Intensive Science University Network (DISUN) l Enabling Grids for E-SciencE (EGEE) l Grid Laboratory of Wisconsin (GLOW) l Grid Operations Center at Indiana University l Grid Research and Education Group at Iowa (GROW) l Nordic Data Grid Facility (NorduGrid) l Northwest Indiana Computational Grid (NWICG) l New York State Grid (NYSGrid) (in progress). l TeraGrid l Texas Internet Grid for Research and Education (TIGRE) l TWGrid (from Academica Sinica Grid Computing) Worldwide LHC Computing Grid Collaboration (WLCG) Slide courtesy of Ruth Pordes, OSG All Hands Meeting 2007

GlobusWORLD 2006Globus Primer Resources across production & integration infrastructures !ncrease in ~15 since Seattle 27 Virtual Organizations (+ 3 operations VOs) 25% non-physics. ~20,000 cores (from 30 to 4000 cores per cluster) ~6 PB accessible Tapes ~4 PB Shared Disk Sustaining through OSG submissions: Measuring ~180K CPUhours/day. ~Factor of 50% more (being measured) than in Seattle Using production & research networks OSG Snapshot Slide courtesy of Ruth Pordes, OSG All Hands Meeting 2007

GlobusWORLD 2006Globus Primer13 MEDICUS Picture courtesy of Stephan Erberich

II. How Grids are Built and Used

GlobusWORLD 2006Globus Primer15 Methodology l Building a Grid system or application is currently an exercise in software integration. u Define user requirements u Derive system requirements or features u Survey existing components u Identify useful components u Develop components to fit into the gaps u Integrate the system u Deploy and test the system u Maintain the system during its operation l This should be done iteratively, with many loops and eddies in the flow.

GlobusWORLD 2006Globus Primer16 What End Users Need Secure, reliable, on- demand access to data, software, people, and other resources (ideally all via a Web Browser!)

GlobusWORLD 2006Globus Primer17 How it Happens Web Browser Compute Server Data Catalog Data Viewer Tool Certificate authority Chat Tool Credential Repository Web Portal Compute Server Resources implement standard access & management interfaces Collective services aggregate &/or virtualize resources Users work with client applications Application services organize VOs & enable access to other services Database service Database service Database service Simulation Tool Camera Telepresence Monitor Registration Service

GlobusWORLD 2006Globus Primer18 How it Happens l Implementations are provided by a mix of u Application-specific code u “Off the shelf” tools and services u Tools and services from the Grid community (Globus + others using the same standards) l Glued together by… u Application development u System integration

GlobusWORLD 2006Globus Primer19 The Importance of Community l All Grid technology is evolving rapidly. u Web services standards u Grid interfaces u Grid implementations u Grid resource providers (ASP, SSP, etc.) l Community is important! u Best practices (OGF, OASIS, etc.) u Open source (Linux, Axis, Globus, etc.) l Application of community standards is vital. u Increases leverage u Mitigates (a bit) effects of rapid evolution u Paves the way for future integration/partnership