1. EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE www.eu-egee.org EGEE and gLite are registered trademarks Applications Using the EGEE Grid Infrastructure C. Loomis (CNRS/LAL), V. Floros (GRNET) EuroVO Conference Garching, Germany 7-11 April 2008

2. Apps. on EGEE – C. Loomis, V. Floros – EuroVO – 9 April 2008 Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 2 Contents Introduction Scientific Disciplines on the Grid Grid Functionality Highlighted Applications Evolution: Project & Middleware Conclusions

3. Apps. on EGEE – C. Loomis, V. Floros – EuroVO – 9 April 2008 Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 3 Introduction Grid: A hardware and software infrastructure… Federation –Sharing and coherent use of distributed resources. –Use of diverse resources (CPU, storage, DBs, infiniband, …). Collaboration –Platform to bring different people with different skills together. –Mechanism to analyze, publish, and combine previous results. Goal: Adoption and use by large number of users in wide spectrum of different scientific disciplines.

4. Apps. on EGEE – C. Loomis, V. Floros – EuroVO – 9 April 2008 Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 4 Scientific Disciplines Growth and diversification of applications. Reported apps. only  underestimate! 6/20062/20071/2008 Astron. & Astrophysics289 Computational Chemistry62721 Earth Science16 18 Fusion234 High-Energy Physics9117 Life Sciences233937 Others41421 Total62118117 Condensed Matter Physics Comp. Fluid Dynamics Computer Science/Tools Civil Protection Finance

5. Apps. on EGEE – C. Loomis, V. Floros – EuroVO – 9 April 2008 Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 5 Growing Usage ~22K CPUs in continuous use Usage doubled this last year

6. Apps. on EGEE – C. Loomis, V. Floros – EuroVO – 9 April 2008 Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 6 Usage by Scientific Discipline Wide (natural) differences in total CPU utilization. –Continued heavy use in HEP and LS. –Average 20x increase in other areas! Evidence of broad adoption of grid technology. 20062007TOTAL Astron. & Astrophysics5812721331 Comp. Chem.34773897736 Earth Science32255287 Fusion1713691386 High-Energy Physics70335166110236444 Life Sciences9073966618739 Others24981011212610 TOTAL82361196173278533

7. Apps. on EGEE – C. Loomis, V. Floros – EuroVO – 9 April 2008 Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 7 Summary of Use Large, growing overall utilization Long-term, habitual use of infrastructure. Broad adoption –Growing number of Virtual Organizations. –Uptake in broad spectrum of disciplines.

8. Apps. on EGEE – C. Loomis, V. Floros – EuroVO – 9 April 2008 Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 8 Functionality EGEE core middleware = gLite Job management –Gatekeeper, Workload Mgt. System –GridWay, GANGA/DIANE Data management: –SRM, FTS, LFC –SRB, … Metadata management –AMGA VO management –VOMS (user and group management, authorization)

9. Apps. on EGEE – C. Loomis, V. Floros – EuroVO – 9 April 2008 Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 9 CERN LHC 9 km © CERN Geneva

10. Apps. on EGEE – C. Loomis, V. Floros – EuroVO – 9 April 2008 Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 10 ATLAS Experiment 40 m 20 m 7000 tons ATLAS Image: ATL-PHO-GEN-2002-002

11. Apps. on EGEE – C. Loomis, V. Floros – EuroVO – 9 April 2008 Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 11 High-Energy Physics (LCG) Data Rate: –40 MHz interaction rate –100 Hz of filtered events –1-10 megabytes per filtered event –0.1-1 gigabytes/second Data Volume: –LHC runs 24/7. –Generates petabytes of data per year! –Plus 1-10 times that in simulated data. Data management is the real challenge for LHC. –Recording and retrieval. –Metadata management for locating interesting data. –Chaotic analysis and large productions. kilo-K10 3 mega-M10 6 giga-G10 9 tera-T10 12 peta-P10 15 exa-E10 18

12. Apps. on EGEE – C. Loomis, V. Floros – EuroVO – 9 April 2008 Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 12 Jürgen Knobloch Demanding Data Flow Real data: –CERN to others Analyzed data: –All to all Simulated data: –All to all

13. Apps. on EGEE – C. Loomis, V. Floros – EuroVO – 9 April 2008 Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 13 Docking: –Predict how small molecules bind to receptor with known 3D structure. Projects: –Proteins@Home –Rosetta@home –Docking@Home –AFRICA@home –malariacontrol.net –WISDOM Virtual Screening Process

14. Apps. on EGEE – C. Loomis, V. Floros – EuroVO – 9 April 2008 Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 14 WISDOM (http://wisdom.healthgrid.org/)http://wisdom.healthgrid.org/ –Developing new drugs for neglected and emerging diseases with a particular focus on malaria. –Reduced R&D costs for neglected diseases –Accelerated R&D for emerging diseases Three large calculations: –WISDOM-I (Q3’05), 1M compounds, 1 TB –Avian Flu (Q2’06), 300k compounds, 750 GB –WISDOM-II (Q4’06), 125M compounds, 2 TB WISDOM calculations used FlexX from BioSolveIT (3-6k free, floating licenses) in addition to Autodock. WISDOM

15. Apps. on EGEE – C. Loomis, V. Floros – EuroVO – 9 April 2008 Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 15 GEOSCOPE GEOSCOPE (http://geoscope.ipgp.jussieu.fr/)http://geoscope.ipgp.jussieu.fr/ –Localization of earthquakes –Determination of rupture modes of the faults –Results within a few hours of major earthquakes date = 6 Jan. 2008 time = 5:14:17 depth = 50.9 km magnitude = 6.1 latitude = 37.150° longitude = 22.934°

16. Apps. on EGEE – C. Loomis, V. Floros – EuroVO – 9 April 2008 Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 16 Noise Determination Process complete data set on EGEE –25 years of data –28 seismological stations and data center Impact on seismological data center design

17. Apps. on EGEE – C. Loomis, V. Floros – EuroVO – 9 April 2008 Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 17 Evolution Evolve from R&D project(s) to service infrastructure. –Continuous evolution since European DataGrid (2001). –EGEE-III starts 1 May 2008 for 2 years. –Planned switch to EGI after EGEE-III. Focus in EGEE-III –Support, Community Building, Advanced Functionality

18. Apps. on EGEE – C. Loomis, V. Floros – EuroVO – 9 April 2008 Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 18 Classic Support Direct User Support –Use GGUS system; grid ticketing system. –Provide team for handling tickets. –Documentation management and generation. VO Support –Registration. –Resource allocation. –Better tools for VO management (user mgt., communication, …). Application Porting Support –GILDA (INFN): porting/training using t-Infrastructure –GASuC (SZTAKI): porting to production grid infrastructure

19. Apps. on EGEE – C. Loomis, V. Floros – EuroVO – 9 April 2008 Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 19 Community Building Build strong, self-reliant user communities. Discipline-specific meetings –Techniques to aid each discipline (common data, tools, etc.) –Dissemination within that discipline Topical Meetings –Discuss common problems or needs –Highlight tools/techniques to address those needs

20. Apps. on EGEE – C. Loomis, V. Floros – EuroVO – 9 April 2008 Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 20 Community Building Conferences = Knowledge Transfer –Present results from using grid technology. –Discuss encountered problems and solutions. –Increase interactions between users. UF1 (CERN) EGEE’06 (Geneva) UF2-OGF20 (Manchester) EGEE’07 (Budapest) UF3 (Clermont- Ferrand)

21. Apps. on EGEE – C. Loomis, V. Floros – EuroVO – 9 April 2008 Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 21 Advanced Functionality gLite provides a reliable base for basic grid services. Core functionality must be augmented with higher-level software/services to provide complete stack for real applications. Continue with “cluster” development –Proven model to provide tight link with user requirements –Focus on “general”, high-level tools that can benefit others –AMGA, Dashboard, GANGA/DIANE, … RESPECT –Mechanism to highlight useful products that will work with gLite –Ensure “external” software provides support to user community

22. Apps. on EGEE – C. Loomis, V. Floros – EuroVO – 9 April 2008 Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 22 Working Groups EGEE supports a series of work groups to develop solutions for common needs or problems. Low-latency scheduling Parallel applications (MPI) Medical Data Management Job priorities Application portals Database Access

23. Apps. on EGEE – C. Loomis, V. Floros – EuroVO – 9 April 2008 Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 23 Commercial Software Gaussian –http://www.gaussian.com/http://www.gaussian.com/ –Predicts the energies, vibrational freq., … of molecular systems. –VO-based licensing model, actually in use in gaussian VO. MathWorks –http://www.mathworks.com/http://www.mathworks.com/ –Integrate MATLAB & Distributed Computing Engine with EGEE. –Both client and server are licensed in this model. Interactive Supercomputing –http://www.interactivesupercomputing.com/http://www.interactivesupercomputing.com/ –Similar to DCE; used from multiple clients (MATLAB, Python, R) –Server licensed, some clients licensed

24. Apps. on EGEE – C. Loomis, V. Floros – EuroVO – 9 April 2008 Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 24 Conclusions Scientists use the EGEE grid: –Routinely and heavily to speed and to enhance their analyses, –To federate (share) their resources, and –To collaborate effectively. Long-term goals: –Provide infrastructure and full range of support services. –Have grid technology routinely used in scientific community. –Build strong, self-reliant user communities.

25. Apps. on EGEE – C. Loomis, V. Floros – EuroVO – 9 April 2008 Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 25 Useful Links NA4 web site: –http://egeena4.lal.in2p3.fr/http://egeena4.lal.in2p3.fr/ –First point of contact for both new and existing users. gLite documentation: –http://glite.web.cern.ch/glite/documentation/userguide.asphttp://glite.web.cern.ch/glite/documentation/userguide.asp –http://glite.web.cern.ch/glite/documentation/default.asphttp://glite.web.cern.ch/glite/documentation/default.asp –Documentation for core middleware functionality. UIG “Use Cases” –http://egee-uig.web.cern.ch/egee- uig/production_pages/UIGindex.htmhttp://egee-uig.web.cern.ch/egee- uig/production_pages/UIGindex.htm –Simple HOWTOs for common tasks NA3 Training material –http://www.egee.nesc.ac.uk/trgmat/index.htmlhttp://www.egee.nesc.ac.uk/trgmat/index.html –Comprehensive catalog of training materials.