1. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 SA3 – Services for Heavy User Communities Jamie Shiers CERN SA3 – EGI-InSPIRE EC Review 20111

2. www.egi.eu EGI-InSPIRE RI-261323 SA3 Overview SA3 – EGI-InSPIRE EC Review 20112 9 Countries 11 Beneficiaries 732 PMs 20.4 FTEs WPTaskBeneficiaryTotal PMs WP6-GTSA3.1CERN18 WP6-GTSA3.2.1CNRS12 WP6-GTSA3.2.1CERN60 WP6-GTSA3.2.2ARNES3 WP6-GTSA3.2.2UI SAV18 WP6-GTSA3.2.2CERN60 WP6-GTSA3.2.3CNRS18 WP6-GTSA3.2.3INFN27 WP6-GTSA3.2.4CSIC18 WP6-GTSA3.2.4CSC18 WP6-GTSA3.2.4CYFRONET6 WP6-GTSA3.2.4EMBL18 WP6-GTSA3.2.5CSIC27 WP6-GTSA3.2.5TCD21 WP6-GTSA3.2.5INFN9 WP6-GTSA3.3INFN60 WP6-GTSA3.3CERN203 WP6-GTSA3.4CNRS29 WP6-GTSA3.4EMBL26 WP6-GTSA3.5INFN30 WP6-GTSA3.6KIT-G27 WP6-GTSA3.6CNRS24 CERN France Slovenia Slovakia Italy Spain Finland Poland EMBL Ireland Germany

3. www.egi.eu EGI-InSPIRE RI-261323 Task Breakdown SA3 – EGI-InSPIRE EC Review 20113 TaskActivities% Effort TSA3.1Activity Management2 TSA3.2Shared services and tools43 TSA3.3Services for High Energy Physics (HEP)36 TSA3.4Services for Life Sciences (LS)8 TSA3.5Services for Astronomy and Astrophysics (A&A) 4 TSA3.6Services for Earth Sciences (ES)7 Other HUCS: Computational Chemistry and Materials Sciences and Technologies (CCMST), Fusion (F)

4. www.egi.eu EGI-InSPIRE RI-261323 SA3 Objectives Transition to sustainable support: –Identify tools of benefit to multiple communities –Migrate these as part of the core infrastructure –Establish support models for those relevant to individual communities SA3 – EGI-InSPIRE EC Review 20114

5. www.egi.eu EGI-InSPIRE RI-261323 Communities & Activities 5 High Energy Physics TSA3.3 The LHC experiments use grid computing for data distribution, processing and analysis. Strong focus on common tools and solutions. Areas supported include: Data Management, Data Analysis and Monitoring. Main VOs: ALICE, ATLAS, CMS, LHCb but covers many other HEP experiments + related projects. Life Sciences Covers the European Extremely Large Telescope (E-ELT), the Square Kilometre Array (SKA) and Cerenkov Telescope Array (CTA) and others. Activities focus on visualisation tools and database/catalog access from the grid. Main VOs: Argo, Auger, Glast, Magic, Planck, CTA, plus others (total 23) across 7 NGIs. Large variety of ES disciplines. Provides also access from the grid to resources within the Ground European Network for Earth Science Interoperations - Digital Earth Community (GENESI-DEC); assists scientists working on climate change via the Climate-G testbed. Main VOs: esr, egeode, climate-g, env.see- grid-sci.eu, meteo.see-grid-sci.eu, seismo.see-grid-sci.eu- support by ~20 NGIs Astronomy & Astrophysics TSA3.5 Earth Sciences TSA3.6 Life Sciences TSA3.4 Focuses on medical, biomedical and bioinformatics sectors to connect worldwide laboratories, share resources and ease access to data in a secure and confidential way. Supports 4 VOs (biomed, lsgri, vlemed and pneumogrid) across 6 NGIs via the Life Science Grid Community. SA3 – EGI-InSPIRE EC Review 2011

6. www.egi.eu EGI-InSPIRE RI-261323 Communities & Activities 6 High Energy Physics TSA3.3 The LHC experiments use grid computing for data distribution, processing and analysis. Strong focus on common tools and solutions. Areas supported include: Data Management, Data Analysis and Monitoring. Main VOs: ALICE, ATLAS, CMS, LHCb but covers many other HEP experiments + related projects. Life Sciences Covers the European Extremely Large Telescope (E-ELT), the Square Kilometre Array (SKA) and Cerenkov Telescope Array (CTA) and others. Activities focus on visualisation tools and database/catalog access from the grid. Main VOs: Argo, Auger, Glast, Magic, Planck, CTA, plus others (total 23) across 7 NGIs. Large variety of ES disciplines. Provides also access from the grid to resources within the Ground European Network for Earth Science Interoperations - Digital Earth Community (GENESI-DEC); assists scientists working on climate change via the Climate-G testbed. Main VOs: esr, egeode, climate-g, env.see- grid-sci.eu, meteo.see-grid-sci.eu, seismo.see-grid-sci.eu- support by ~20 NGIs Astronomy & Astrophysics TSA3.5 Earth Sciences TSA3.6 Life Sciences TSA3.4 Focuses on medical, biomedical and bioinformatics sectors to connect worldwide laboratories, share resources and ease access to data in a secure and confidential way. Supports 4 VOs (biomed, lsgri, vlemed and pneumogrid) across 6 NGIs via the Life Science Grid Community. SA3 – EGI-InSPIRE EC Review 2011 These and other communities / projects supported by shared tools & services

7. www.egi.eu EGI-InSPIRE RI-261323 Communities & Activities 7 High Energy Physics TSA3.3 The LHC experiments use grid computing for data distribution, processing and analysis. Strong focus on common tools and solutions. Areas supported include: Data Management, Data Analysis and Monitoring. Main VOs: ALICE, ATLAS, CMS, LHCb but covers many other HEP experiments + related projects. Life Sciences Covers the European Extremely Large Telescope (E-ELT), the Square Kilometre Array (SKA) and Cerenkov Telescope Array (CTA) and others. Activities focus on visualisation tools and database/catalog access from the grid. Main VOs: Argo, Auger, Glast, Magic, Planck, CTA, plus others (total 23) across 7 NGIs. Large variety of ES disciplines. Provides also access from the grid to resources within the Ground European Network for Earth Science Interoperations - Digital Earth Community (GENESI-DEC); assists scientists working on climate change via the Climate-G testbed. Main VOs: esr, egeode, climate-g, env.see- grid-sci.eu, meteo.see-grid-sci.eu, seismo.see-grid-sci.eu- support by ~20 NGIs Astronomy & Astrophysics TSA3.5 Earth Sciences TSA3.6 Life Sciences TSA3.4 Focuses on medical, biomedical and bioinformatics sectors to connect worldwide laboratories, share resources and ease access to data in a secure and confidential way. Supports 4 VOs (biomed, lsgri, vlemed and pneumogrid) across 6 NGIs via the Life Science Grid Community. SA3 – EGI-InSPIRE EC Review 2011 These and other communities / projects supported by shared tools & services

8. www.egi.eu EGI-InSPIRE RI-261323 Shared Tools & Services Partner breakdown: see next slide

9. www.egi.eu EGI-InSPIRE RI-261323 9 Participant Number Participant Short name / Lead Beneficiary WP 6 (SA3): Services for Heavy User Communities Person Months per Task PM per Effort Type Total Person Months TSA3.1TSA3.2 +F+CCMST TSA3.3 HEP TSA3.4 LS TSA3.5 A&A TSA3.6 ES General 10KIT-G27 Fraunhofer27 12CSIC45 CSIC27 CIEMT18 13CSC18 14CNRS305383 19TCD21 INFN366030126 INFN60 UNIPG999 SPACI27 28CYFRONET666 31ARNES333 32UI SAV18 35CERN18120203341 37EMBL182644 TOTALS18315263793027732 SA3 – Partner Breakdown SA3 – EGI-InSPIRE EC Review 2011

10. www.egi.eu EGI-InSPIRE RI-261323 Achievements – Overview 1.Successfully supported major production computing at an unprecedented scale – both quantitatively and qualitatively 2.Successfully delivered common solutions in a variety of areas – with other activities in progress 3.Actively participated in EGI Technical & User Forum via presentations, tutorials, panels and demos 4.Broadened the use of grid technology and HUC services to related projects within the HUC domain (such as unfunded – by EGI-InSPIRE – LS / ES projects) 5.Completed first round of Milestones & Deliverables together with associated technical work 6.Identified – across all HUC communities – areas of common technology investigation for the future 7.Developed a S.W.O.T. analysis of each main discipline and made significant steps on the road to sustainability SA3 – EGI-InSPIRE EC Review 201110

11. www.egi.eu EGI-InSPIRE RI-261323 Service Quantity & Quality The first year of EGI-InSPIRE saw the use of grid computing at an unprecedented scale! –More than 100 CPU-millennia delivered; –More than 50 PB of data stored; –Data transfer rates of 200TB/day At the same time, its power in turning scientific data into publications at record speed was publically acknowledged (Economist, July 2010)July 2010 And service delivery was typically smooth with a small number of problems requiring in-depth investigation –Quality plots in backup slides SA3 contribution: tools, operations & WLCG Service Coordination SA3 – EGI-InSPIRE EC Review 201111

12. www.egi.eu EGI-InSPIRE RI-261323 Common Solutions – Examples SA3 – EGI-InSPIRE EC Review 201112 GangaExtensively used as a “gridification tool” by many projects / disciplines. This includes not only communities within EGI- InSPIRE but also others in many fields Numerous projects / disciplines Mini- Dashboard To be used with Ganga to monitor Ganga-based activity on the grid. Used by EnviroGRIDS and offered to NA3 EnviroGRIDS, NA3 Experiment Dashboards Common schema and code base for all job monitoring applications: job summary, historical view & task monitoring – implemented from July 2010 ATLAS + CMS GRelCExploited by the Earth Science (ES) community (Climate-G testbed) and by other projects/disciplines related to Environment and Bioinformatics LS, A&A, ES MPIHigh impact on multiple user communitiesCCMST, A&A, F FrameworksUse of LHCb’s DIRAC framework by LCD/ILC and Belle collaborations. Investigation of DIRAC by ES + others HEP (beyond LHC), ES Data Management Data popularity (dynamic data placement / caching), consistency of catalogs / storage LHC VOs Site Stress Testing HammerCloud service used at ATLAS, CMS and LHCb. Fully applicable to other VOs / communities HEP

13. www.egi.eu EGI-InSPIRE RI-261323 Common Solutions SA3 – EGI-InSPIRE EC Review 201113 Life Sciences Astronomy & Astrophysics Earth Sciences Life Sciences Connect various tools with multiple communities: the main disciplines and also Fusion, Computational Chemistry Fusion HEP Comp Chem

14. www.egi.eu EGI-InSPIRE RI-261323 Common Solutions SA3 – EGI-InSPIRE EC Review 201114 Life Sciences Astronomy & Astrophysics Earth Sciences Life Sciences Ganga: job definition and management in HEP (ATLAS and LHCb, Compass, Harp), Fusion, L&E Sciences Fusion HEP Comp Chem

15. www.egi.eu EGI-InSPIRE RI-261323 Common Solutions SA3 – EGI-InSPIRE EC Review 201115 Life Sciences Astronomy & Astrophysics Earth Sciences Life Sciences Hammercloud: a site stress testing system to validate site usability. Used by ATLAS, CMS and LHCb Fusion HEP Comp Chem

16. www.egi.eu EGI-InSPIRE RI-261323 Common Solutions SA3 – EGI-InSPIRE EC Review 201116 Life Sciences Astronomy & Astrophysics Earth Sciences Life Sciences Dashboards: single entry point to monitoring of all 4 LHC experiment activities on the grid. (Mini) Dashboard also used by E&L Sciences Fusion HEP Comp Chem

17. www.egi.eu EGI-InSPIRE RI-261323 Common Solutions SA3 – EGI-InSPIRE EC Review 201117 Life Sciences Astronomy & Astrophysics Earth Sciences Life Sciences DIRAC: workload and data management used by LHCb plus other HEP experiments with interest from Earth Sciences Fusion HEP Comp Chem

18. www.egi.eu EGI-InSPIRE RI-261323 Common Solutions SA3 – EGI-InSPIRE EC Review 201118 Life Sciences Astronomy & Astrophysics Earth Sciences Life Sciences MPI: used by A&A, Fusion, Computational Chemistry to handle parallel execution in grid environments Fusion HEP Comp Chem

19. www.egi.eu EGI-InSPIRE RI-261323 Common Solutions SA3 – EGI-InSPIRE EC Review 201119 Life Sciences Astronomy & Astrophysics Earth Sciences Life Sciences GRelC: set of advanced data grid services to manage Databases on the Grid. Used by L&E Sciences and A&A Fusion HEP Comp Chem

20. www.egi.eu EGI-InSPIRE RI-261323 Common Solutions – Examples SA3 – EGI-InSPIRE EC Review 201120 GangaExtensively used as a “gridification tool” by many projects / disciplines. This includes not only communities within EGI- InSPIRE but also others in many fields Numerous projects / disciplines Mini- Dashboard To be used with Ganga to monitor Ganga-based activity on the grid. Used by EnviroGRIDS and offered to NA3 EnviroGRIDS, NA3 Experiment Dashboards Common schema and code base for all job monitoring applications: job summary, historical view & task monitoring – implemented from July 2010 ATLAS + CMS GRelCExploited by the Earth Science (ES) community (Climate-G testbed) and by other projects/disciplines related to Environment and Bioinformatics LS, A&A, ES MPIHigh impact on multiple user communitiesCCMST, A&A, F FrameworksUse of LHCb’s DIRAC framework by LCD/ILC and Belle collaborations. Investigation of DIRAC by ES + others HEP (beyond LHC), ES Data Management Data popularity (dynamic data placement / caching), consistency of catalogs / storage LHC VOs Site Stress Testing HammerCloud service used at ATLAS, CMS and LHCb. Fully applicable to other VOs / communities HEP

21. www.egi.eu EGI-InSPIRE RI-261323 EGI TF & UF Actively participated in both EGI Technical Forum (September 2010, Amsterdam) and User Forum (April 2011, Vilnius) –TF: Two sessions covering overview of tasks and sub- tasks; one session dedicated to discussion of Common Requirements –UF: Numerous presentations, tutorials, demonstrations covering results in all areas of activity described above –All sessions well attended (50-100) people; good questions and feedback SA3 – EGI-InSPIRE EC Review 201121

22. www.egi.eu EGI-InSPIRE RI-261323 Draft Requirements Matrix SA3 – EGI-InSPIRE EC Review 201122 DisciplinePreservationVolumeAccessSecurity Earth Science MillenniaNo intrinsic limit – broken down into many different areas Cross- correlation Some data clearly sensitive Astronomy & Astrophysics MillenniaNo limit – current projects range from 100 TB – low PB range Cross- correlation between different observations Explicit policy on placing data in public domain after 1 year Life Science1+ centuries# people (life-forms) x data By Individual By condition Evolution Patient privacy; Copyrighted tools; Competing industries High Energy Physics A few decades More for educational purposes? 100PB – 1EB today; Previous generations, e.g. LEP, are in commodity market Range from sequential access to bulk data to many // accesses Often traded for performance & scalability

23. www.egi.eu EGI-InSPIRE RI-261323 Draft Requirements Matrix SA3 – EGI-InSPIRE EC Review 201123 DisciplinePreservationVolumeAccessSecurity Earth Science MillenniaNo intrinsic limit – broken down into many different areas Cross- correlation Some data clearly sensitive Astronomy & Astrophysics MillenniaNo limit – current projects range from 100 TB – low PB range Cross- correlation between different observations Explicit policy on placing data in public domain after 1 year Life Science1+ centuries# people (life-forms) x data By Individual By condition Evolution Patient privacy; Copyrighted tools; Competing industries High Energy Physics A few decades More for educational purposes? 100PB – 1EB today; Previous generations, e.g. LEP, are in commodity market Range from sequential access to bulk data to many // accesses Often traded for performance & scalability EGI-InSPIRE SA3 encourages and requires us to work together. By working with the contact points to the supported communities, we can reach out to other projects to build a more representative set of Common requirements and needs. This work started at the EGI TF 2010 (SA3 panel) and continued at the IEEE Massive Storage and Technologies symposium.

24. www.egi.eu EGI-InSPIRE RI-261323 Draft Requirements Matrix SA3 – EGI-InSPIRE EC Review 201124 DisciplinePreservationVolumeAccessSecurity Earth Science MillenniaNo intrinsic limit – broken down into many different areas Cross- correlation Some data clearly sensitive Astronomy & Astrophysics MillenniaNo limit – current projects range from 100 TB – low PB range Cross- correlation between different observations Explicit policy on placing data in public domain after 1 year Life Science1+ centuries# people (life-forms) x data By Individual By condition Evolution Patient privacy; Copyrighted tools; Competing industries High Energy Physics A few decades More for educational purposes? 100PB – 1EB today; Previous generations, e.g. LEP, are in commodity market Range from sequential access to bulk data to many // accesses Often traded for performance & scalability

25. www.egi.eu EGI-InSPIRE RI-261323 Broadening Use of the Grid Concrete examples: –Bi-directional sharing of solutions and techniques by other projects, such as PARTNER, ULICE (both LS) and EnviroGRIDS (ES) Other Earth Science projects in the pipeline Contacts also through IEEE conferences & symposia –e.g. Science: NSS + MIC, Technology: MSST –Solutions and expertise applied to 3 generations of HEP experiments – LEP to ILC SA3 – EGI-InSPIRE EC Review 201125

26. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 Domain-Specific Work A Summary of the Main Tasks: Details in Backup Slides SA3 – EGI-InSPIRE EC Review 201126

27. www.egi.eu EGI-InSPIRE RI-261323 Services for HEP ActivityResults Distributed Analysis Support for the tools used by the experiments; common error-logger for ATLAS/CMS/LHCb Data Management Dynamic caching / data popularity – move away from static data placement: common solutions deployed; others under development Persistency Framework Handles the event and detector conditions data from the experiments Monitoring / Dashboards All aspects of production and analysis: additional common solutions deployed Task Leader: Maria Girone SA3 – EGI-InSPIRE EC Review 201127

28. www.egi.eu EGI-InSPIRE RI-261323 Services for Life Sciences (SUB)TASKRESULTS TSA3.4Virtual Research Community building Life Sciences Grid Community (LSGC) Service development and provisioning TSA3.4.1Dashboard: design phase TSA3.4.2Data encryption service: prototype deployed TSA3.4.3Database interface: GRelC deployed TSA3.4.4Workflows: work on Taverna to start in year 2 TSA3.4.5CoreBio services: work to start in year 2 Task leader: Johan Montagnat, Deputy: Yannick Legré SA3 – EGI-InSPIRE EC Review 201128

29. www.egi.eu EGI-InSPIRE RI-261323 Services for A&A (SUB)TASKRESULTS Preparatory studies TSA3.5.2Visualisation tools: VisIVO integration (Vis. i/f to V Ob) TSA3.5.3Parallel (MPI/OpenMP) and GPU computing using CUDA TSA3.5.4Database services and integration with Virtual Observatory (V Ob) Achieved results TSA3.5.2MS608: Gridification of VisIVO and VisIVO Service TSA3.5.3Parallel programming: testing activities with cosmological simulations codes (FLY, Gadget, Flash) TSA3.5.4Database services analysis Task leader: Claudio Vuerli, Deputy: Giuliano Taffoni SA3 – EGI-InSPIRE EC Review 201129

30. www.egi.eu EGI-InSPIRE RI-261323 Services for ES (SUB)TASKRESULTS TSA3.6Support ES Activities in ES communities and projects, and carried out by researchers & students in Universities Main activity by proposal: access to GENESI-DEC Status: Webservice available and validated with application Extensions dependent on GENESI-DEC progress Further developments in Task: Integration with available GEOSS services to access Genesi and Climate Data from ESG Since January 2011 common developments with Earth System Grid (ESG) Access to ESG data from EGI infrastructure and vice versa Main problem: authentication / authorisation due to different federations Institute IPSL/CNRS, IPGP now unfunded partner in TSA3.6 Task leader: Horst Schwictenberg, Deputy: André Gemünd SA3 – EGI-InSPIRE EC Review 201130

31. www.egi.eu EGI-InSPIRE RI-261323 S.W.O.T. Analyses – D6.2 SA3 – EGI-InSPIRE EC Review 201131 DisciplineStrengthsWeaknessesOpportunitiesThreats HEP24 x 7 petascale computing Time to resolve some incidents Increased commonality Need to adapt to new technologies Life SciencesRamp-up of LSGC Issues with use of multi-grids Community growth No long-term funding Astronomy & Astrophysics Successful use of DCIs Coordination of overall community Wider use of DCIs Funding Earth SciencesExisting user community Diversity of demands and technologies Community growth Funding GRelCCommunity based approach, cross - discipline Little feedback about the SA3 DB questionnaire Community growth Lack of use of registry MPIWidely regarded as an important tool External factorsUse of large SMPs / GPGPUs Need for standardisation Notable overlap in these independent analyses

32. www.egi.eu EGI-InSPIRE RI-261323 S.W.O.T. Analyses – D6.2 SA3 – EGI-InSPIRE EC Review 201132 DisciplineStrengthsWeaknessesOpportunitiesThreats HEP24 x 7 petascale computing Time to resolve some incidents Increased commonality Need to adapt to new technologies Life SciencesRamp-up of LSGC Issues with use of multi-grids Community growth No long-term funding Astronomy & Astrophysics Successful use of DCIs Coordination of overall community Wider use of DCIs Funding Earth SciencesExisting user community Diversity of demands and technologies Community growth Funding GRelCCommunity based approach, cross - discipline Little feedback about the SA3 DB questionnaire Community growth Lack of use of registry MPIWidely regarded as an important tool External factorsUse of large SMPs / GPGPUs Need for standardisation

33. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 Use of Resources Breakdown by Task & Partner SA3 – EGI-InSPIRE EC Review 201133

34. www.egi.eu EGI-InSPIRE RI-261323 Effort Breakdown (1/2) SA3 – EGI-InSPIRE EC Review 201134 TaskPartnerWorked PM FundedCommitted PMAchieved PM % TSA3.135-CERN8.86.0147% TSA3.2.114A-CNRS00.70% TSA3.2.114C-HealthGrid0.33.38% TSA3.2.135-CERN22.420.0112% TSA3.2.231B-JSI0.51.047% TSA3.2.232-UI SAV2.66.043% TSA3.2.235-CERN17.120.086% TSA3.2.314A-CNRS7.53.3224% TSA3.2.314B-CEA02.70% TSA3.2.321E-SPACI4.09.044% TSA3.2.412C-CIEMAT6.96.0115% TSA3.2.413-CSC5.56.091% TSA3.2.428C-ICBP0.72.033% TSA3.2.437-EMBL06.00%

35. www.egi.eu EGI-InSPIRE RI-261323 Effort Breakdown (2/2) SA3 – EGI-InSPIRE EC Review 201135 TaskPartnerWorked PM FundedCommitted PMAchieved PM % TSA3.2.512A-CSIC5.69.062% TSA3.2.519-TCD7.0 100% TSA3.2.521D-UNIPG11.53.0385% TSA3.321A-INFN020.00% TSA3.335-CERN65.667.797% TSA3.414A-CNRS2.63.377% TSA3.414C-HealthGrid2.96.346% TSA3.437-EMBL08.70% TSA3.521C-INAF8.310.083% TSA3.610G-FRAUNHOFER2.19.023% TSA3.614A-CNRS4.58.057% Total:186.3244.076% Most discrepancies due either to reporting problems (being corrected) or start-up issues, e.g. hiring delays. Expect to “catch-up” quickly in PY2 in all cases

36. www.egi.eu EGI-InSPIRE RI-261323 Plans for Next Year Continue to identify / deliver common solutions Further work on sustainability; address key areas of weakness / concern No modifications to the DoW are foreseen Underspending of PM in PY1 to be made up in PYs 2 and 3 SA3 – EGI-InSPIRE EC Review 201136

37. www.egi.eu EGI-InSPIRE RI-261323 EGI TF 2011 Sessions planned to highlight main achievements of WP in meeting goals plus a mini-workshop on sustainability –High-light and show-case common solutions –Make concrete progress towards sustainability –Continue to invite related projects / disciplines to participate SA3 – EGI-InSPIRE EC Review 201137

38. www.egi.eu EGI-InSPIRE RI-261323 Review of Objectives SA3 – EGI-InSPIRE EC Review 201138 ObjectiveStatus Supporting the tools, services and capabilities required by different HUCs Achieved – work continues in PY2 / PY3 Identifying the tools, services and capabilities currently used by the HUCs that can benefit all user communities and to promote their adoption Several additional items identified and shared across other communities – work will expand and continue in PY2 / PY3 Migrating the tools, services and capabilities that could benefit all user communities into a sustainable support model as part of the core EGI infrastructure Not started – needs further work and discussion First steps: D6.2 (sustainability) & EGI TF w/s Establishing a sustainable support model for the tools, services and capabilities that will remain relevant to single HUCs Collaborative support is the basic model for sustainability that does not depend on individual partners nor specific project funding. A workshop on sustainability at the EGI TF is proposed – work continues and expands in PY2 and PY3. See also D6.2.

39. www.egi.eu EGI-InSPIRE RI-261323 Summary Successfully supported major production computing at an unprecedented scale – both quantitatively and qualitatively Successfully delivered common solutions in a variety of areas – with other activities in progress Actively participated in EGI Technical & User Forum via presentations, tutorials and demos Broadened the use of grid technology and HUC services to related projects within the HUC domain (such as unfunded – by EGI-InSPIRE – LS / ES projects) Completed first round of Milestones & Deliverables together with associated technical work Identified – across all HUC communities – areas of common technology investigation for the future Developed a S.W.O.T. analysis of each main discipline and made significant steps on the road to sustainability SA3 – EGI-InSPIRE EC Review 201139

40. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 BACKUP SLIDES SA3 – EGI-InSPIRE EC Review 201140

41. www.egi.eu EGI-InSPIRE RI-261323 Partner Breakdown 41 Taken from the Description of Work Participant numberParticipant NamePerson-months per participant 10KIT-627 12CSIC45 13CSC18 14CNRS83 19TCD21 INFN126 28CYFRONET6 31ARNES3 32UI SAV18 35CERN341 37EMBL44 Total732 SA3 – EGI-InSPIRE EC Review 2011

42. www.egi.eu EGI-InSPIRE RI-261323 Milestones & Deliverables SA3 – EGI-InSPIRE EC Review 201142 Milestone / Deliverable Due DateLead Partner (#)Title MS601PM1CSC (13)HUC Contact points and the support model MS602PM4INFN (21)HUC Software Roadmap MS603PM4CERN (35)Services for High Energy Physics D6.1PM4CERN (35)Capabilities offered by the HUCs to other communities MS604PM4CNRS (14)Services for the Life Science Community MS605PM8TCD (19)Training and dissemination event D6.2PM9CERN (35)Sustainability plans for the HUC activities MS606PM10INFN (21)HUC Software Roadmap D6.3PM11CERN (35)Annual Report on the Tools and Services of the HUCs MS607PM12CNRS (14)Hydra service deployment MS608PM12INFN (21)Integration of the VisIVO server with the production infrastructure

43. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 HEP SA3 – EGI-InSPIRE EC Review 201143

44. www.egi.eu EGI-InSPIRE RI-261323 WLCG Service Incidents SA3 – EGI-InSPIRE EC Review 201144

45. www.egi.eu EGI-InSPIRE RI-261323 SIRs – Time to Resolution SA3 – EGI-InSPIRE EC Review 201145

46. www.egi.eu EGI-InSPIRE RI-261323 GGUS Tickets – LHC VOs SA3 – EGI-InSPIRE EC Review 201146

47. www.egi.eu EGI-InSPIRE RI-261323 Data Management Crucial area for LHC and other HEP experiments –Data volumes: tens of PB/year, rates: up to 200TB/day between sites, several hundred active analysis users / experiment, 1M analysis jobs / day Experience from first data taking has shown that some assumptions on data placement are no longer optimal –based on decade+ old model “MONARC” which assumed network was scarce and expensive resource Working to adapt computing models to reflect such changes SA3 – EGI-InSPIRE EC Review 201147

48. www.egi.eu EGI-InSPIRE RI-261323 Data Placement & Dynamic Caching Based on MONARC, the initial phase of LHC data distribution was based on static pre-placement –Significant fraction of such data never read! Computing models now driving towards dynamic data placement –Replication is based on usage (“popularity”) – this results in better network and storage utilization Implemented first for ATLAS, now for CMS and LHCb SA3 – EGI-InSPIRE EC Review 201148

49. www.egi.eu EGI-InSPIRE RI-261323 Overview Data volumes managed by DDM on the WLCG Manage ATLAS data on the WLCG –Detector data –Simulated data –User data Provide functionalities for –Data placement & deletion –Bookkeeping & accounting –Data access & search Enforce ATLAS Computing Model STEP09 Data taking >50PetaBytes >180M files SA3 – EGI-InSPIRE EC Review 201149

50. www.egi.eu EGI-InSPIRE RI-261323 10GB/s 19 May 2010 Transfer Service throughput 6GB/s 20112010 JulyApril Oct 4GB/s 2GB/s 5GB/s Spring reprocessing Started data taking Autumn reprocessing and Heavy Ion runs MCDISK- DATADISK merging SA3 – EGI-InSPIRE EC Review 201150

51. www.egi.eu EGI-InSPIRE RI-261323 Storage usage accounting SA3 – EGI-InSPIRE EC Review 201151

52. www.egi.eu EGI-InSPIRE RI-261323 Monitoring of physics-groups space SA3 – EGI-InSPIRE EC Review 201152

53. www.egi.eu EGI-InSPIRE RI-261323 DDM Popularity Provides information about the usage of files and datasets Collects traces at file level by high-level tools used in ATLAS –DQ2 clients –PanDA –Ganga ~4M traces a day –Aggregation of traces into daily statistics Provide information through web site, CLI and API Angelos Molfetas (CERN PH-ADP) SA3 – EGI-InSPIRE EC Review 201153

54. www.egi.eu EGI-InSPIRE RI-261323 Replica reduction agent: Victor Automatic site cleaning: Optimize the utilization of storage resources Secondary replicas are guaranteed to be replicated on other sites Keep sites operationally full by deleting secondary, unpopular replicas Reduces manual operations and accidents Implementation follows plug-in approach for re-usage in other LHC experiments Victor DDM Accounting DDM Popularity DDM Deletion Service 1. Selection of full sites 2. Selection of unpopular replicas 3. Publication of decisions Space information Secondary replica popularity Full sites Replicas to delete Full sites Deleted replicas SA3 – EGI-InSPIRE EC Review 201154

55. www.egi.eu EGI-InSPIRE RI-261323 Catalogue Consistency With 50PB of data storage across a large number of sites worldwide, inconsistencies can easily arise! –Data that resides on Storage Elements but not in various catalogs (grid, experiment) referred to as “Dark Data” One site recently reported 70TB dark data! Using a messaging-based system, various catalogs and SEs can talk to each other and implement lazy synchronization SA3 – EGI-InSPIRE EC Review 201155

56. www.egi.eu EGI-InSPIRE RI-261323 Catalogue Consistency SA3 – EGI-InSPIRE EC Review 201156

57. www.egi.eu EGI-InSPIRE RI-261323 Data Analysis Support Covers the final stage of data processing leading on to publication –Large number of users/month (~1000) and analysis jobs/day (~1M) running across (~100) Tier2 and other sites – “chaotic” data access –All frameworks support heterogeneous back-ends Ganga (ATLAS, LHCb) used by 10 other communities and 500 – 600 users Common site stress testing system (Hammercloud) used by ATLAS, CMS and LHCb Areas of commonality and optimization –Move to “community support” model –Simplify data access and improve monitoring –Use common components and frameworks, such as for job submission and file transfer (built on gLite /EMI FTS) An area of potential future common work and simplification SA3 – EGI-InSPIRE EC Review 201157

58. www.egi.eu EGI-InSPIRE RI-261323 HammerCloud SA3 – EGI-InSPIRE EC Review 201158

59. www.egi.eu EGI-InSPIRE RI-261323 CRAB SA3 – EGI-InSPIRE EC Review 201159

60. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 LS SA3 – EGI-InSPIRE EC Review 201160

61. www.egi.eu EGI-InSPIRE RI-261323 Services for Life Sciences TSA3.4 - Virtual Research Community building –Life Sciences Grid Community (LSGC) Service development and provisioning –TSA3.2.1 - Dashboard: design phase –TSA3.2.3 - Data encryption service: prototype deployed –TSA3.2.3 - Database interface: GRelC deployed –TSA3.2.4 - Workflows: work on Taverna to start in year 2 –TSA3.4 - CoreBio services: work to start in year 2 SA3 – EGI-InSPIRE EC Review 201161

62. www.egi.eu EGI-InSPIRE RI-261323 Life Sciences Grid Community LSGC: http://wiki.healthgrid.org/LSVRC:Index http://wiki.healthgrid.org/LSVRC:Index –4 VOs, 6 NGIs, HG association, 2 EU projects –Communication channels: wiki, mailing lists, monthly phone conferences Technical team (shifters) –Infrastructure monitoring and troubleshooting (Nagios server with LS resource probes) User management tools – early design phase –User registration and management DB –To be integrated in LS Dashboard SA3 – EGI-InSPIRE EC Review 201162

63. www.egi.eu EGI-InSPIRE RI-261323 LS Dashboard Need to integrate –Nagios monitoring dedicated interface (possibly based on GOC dashboard) –VRC-level accounting information –User management tools (possibly based on VOMRS) SA3 – EGI-InSPIRE EC Review 201163

64. www.egi.eu EGI-InSPIRE RI-261323 Data encryption service Hydra server –Encryption keystore server based on Shamir’s secret sharing algorithm –Software packages available for gLite 3.1 –Installation procedure documented for gLite 3.0 Current status (MS607) –Working on gLite 3.1, not yet on gLite 3.2 –Client CLIs to be installed on all LS supporting sites Perspectives –Deployment of a three-heads hydra server on three sites SA3 – EGI-InSPIRE EC Review 201164

65. www.egi.eu EGI-InSPIRE RI-261323 Database interface GRelC (Grid Relational Catalog) service provision to support LS use cases Discussion about new LS use cases and data resources analysis Identification of a couple of biological databases to be ported in grid (relational DBs) Contribution to the SA3 questionnaire related to “grid-databases” SA3 – EGI-InSPIRE EC Review 201165

66. www.egi.eu EGI-InSPIRE RI-261323 LS issues, mitigation & perspectives VOMS and LFC servers are single point of failures –Replication procedures being set up Infrastructure monitoring is time consuming –Scheduled downtimes should be better reflected in BDII –Dedicated view of Nagios results in LS dashboard –Nagios probes improvements LSGC is a multi-VOs / multi-grids community –No tooling available to manage VRCs –Multi-grids hardly addressed in the context of EGI Few feedback on the SA3 questionnaire –More dissemination is needed in PY2 SA3 – EGI-InSPIRE EC Review 201166

67. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 A&A SA3 – EGI-InSPIRE EC Review 201167

68. www.egi.eu EGI-InSPIRE RI-261323 Services for A&A Preparatory studies: –Visualization tools: VisIVO integration (TSA3.5.2) –Parallel (MPI/OpenMP) and GPU computing using CUDA (TSA3.5.3) –Database services and integration with Virtual Observatory (TSA3.5.4) Achieved results: –MS608: Gridification of VisIVO and VisIVO Service (TSA3.5.2) –Parallel programming: testing activities with cosmological simulations codes (FLY, Gadget,Flash) (TSA3.5.3) –Database services analysis (TSA3.5.4) SA3 – EGI-InSPIRE EC Review 201168

69. www.egi.eu EGI-InSPIRE RI-261323 A&A Grid Community 23 VOs and 7 NGIs –The most part of active A&A VOs relate to the astroparticle physics community –Communication channels: Wiki Mailing lists –General list in place, others more specialized could follow Monthly phone conferences –A&A VRC meetings and workshops First of the EGI era will be in Paris –7 November 2011, ADASS Conference SA3 – Johan Montagnat – EGI-InSPIRE EC Review 2011 69SA3 – EGI-InSPIRE EC Review 201169

70. www.egi.eu EGI-InSPIRE RI-261323 VisIVO Porting of VisIVO server in Grid –Preparatory activity: enabling the usage of VisIVO directly within a code during the production phase A software layer has been developed using internal arrays – without the need of producing intermediate files A library of VisIVO was designed and implemented –The first issue of the gridified version of VisIVO server has been released in April 2011 (MS608) SA3 – Johan Montagnat – EGI-InSPIRE EC Review 2011 70SA3 – EGI-InSPIRE EC Review 201170

71. www.egi.eu EGI-InSPIRE RI-261323 A&A issues, mitigation & perspectives A&A is a complex community: its coordination is quite challenging. –No EU A&A projects currently funded  shortage of funds  coordination activity tricky  no easy deployment of tools and services We wish to continue the coordination and the efforts to strengthen the community –We rely on support of EGI.eu and NGIs –We exploit as much as possible collaborative tools and services to establish robust and stable communications SA3 – Johan Montagnat – EGI-InSPIRE EC Review 2011 71SA3 – EGI-InSPIRE EC Review 201171

72. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 ES SA3 – EGI-InSPIRE EC Review 201172

73. www.egi.eu EGI-InSPIRE RI-261323 Services for ES TSA 3.6 Support ES Activities in ES communities and projects, and carried out by researchers & students in Universities Main activity by proposal: access to GENESI-DEC –Status : Webservice available and validated with application –Extensions dependent on GENESI-DEC progress –Further developments in Task: Integration with available GEOSS services to access Genesi and CLimate Data from ESG Since Jan 2011 common developments with the climate Earth System Grid (ESG) –Access to ESG data from EGI-Infrastructure and vice versa –Main problem to solve: A & A due to different federations –Institute IPSL/CNRS, IPGP now unfunded partner in TSA3.6 SA3 – EGI-InSPIRE EC Review 201173

74. www.egi.eu EGI-InSPIRE RI-261323 ES HUC ES HUC VOs: 9 –“ESR VO” „catch all ES people“ provides resources and support for ES projects and anorganized ES users –EGEODE for Geocluster users or related activities –Climate-g : related to GrelC for Climate –3 SEE-Grid VOs for Environment, Meteorology and Seismology –Other VOs related with ES are in contact with ES HUC like Envirogrids and eo-grid.ikd.kiev.ua (Ukrainia), no specific contact with the cmip5 VO (Ireland) –EU project VERCE (seismology) will start with this VO and will work with SA3.6 ES HUC Activities: –Yearly session at European Geoscience Union, General Assembly 2011 about “ES and E-infrastructures SA3 – EGI-InSPIRE EC Review 201174

75. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 Other Tools & Services SA3 – EGI-InSPIRE EC Review 201175

76. www.egi.eu EGI-InSPIRE RI-261323 Experiment Dashboard SA3 – EGI-InSPIRE EC Review 201176

77. www.egi.eu EGI-InSPIRE RI-261323 Experiment Dashboard SA3 – EGI-InSPIRE EC Review 201177

78. www.egi.eu EGI-InSPIRE RI-261323 Experiment Dashboard SA3 – EGI-InSPIRE EC Review 201178

79. www.egi.eu EGI-InSPIRE RI-261323 SA3 – EGI-InSPIRE EC Review 201179 Mini-Dashboard

80. www.egi.eu EGI-InSPIRE RI-261323 Task Monitoring SA3 – EGI-InSPIRE EC Review 201180 user / user-support perspective. More than 350 unique users on a daily basis

81. www.egi.eu EGI-InSPIRE RI-261323 Historical Views SA3 – EGI-InSPIRE EC Review 201181 Site, Management perspective. Job metrics as a function of time.

82. www.egi.eu EGI-InSPIRE RI-261323 Job Summary SA3 – EGI-InSPIRE EC Review 201182 shifter/expert/site perspective. Real time job metrics.

83. www.egi.eu EGI-InSPIRE RI-261323 GRelC: Activity (I) NAx - > - EGI-InSPIRE EC Review 2011 83 User support through the implementation of a grid- database “registry” (EGI Database of Databases) – Easy search and discovery (cross-VO) of grid-DB resources distributed across the EGI grid. – Community-based approach to attract new users and address sustainability SA3 Questionnaire – A census about database resources, related needs and future plan – Distributed among the HUCs (end of Q3) – Few feedback; more dissemination during Y2 SA3 – EGI-InSPIRE EC Review 201183

84. www.egi.eu EGI-InSPIRE RI-261323 GRelC: Activity (II) NAx - > - EGI-InSPIRE EC Review 2011 84 The back-end modules of the “registry” finalized during Y1: – A MySQL catalog for the registry – Several Java classes to manage charts, grid-DB, VOs, GRelC services, community-oriented aspects Front-end modules related to the “registry” finalized in Y1 (now tested, on line during Q5) – Registry view completed – Grid-Database view completed – Integration of Web2.0 (Mash-up, Google Maps, permalinks) and community-oriented aspects (comments, scores, discussions groups, etc.) SA3 – EGI-InSPIRE EC Review 201184

85. www.egi.eu EGI-InSPIRE RI-261323 GRelC: Registry Snapshots NAx - > - EGI-InSPIRE EC Review 2011 85SA3 – EGI-InSPIRE EC Review 201185

86. www.egi.eu EGI-InSPIRE RI-261323 GRelC: User support NAx - > - EGI-InSPIRE EC Review 2011 86 HUC support in terms of: – grid-metadata management for the Earth Science and Environmental context (e.g. Climate-G, CMCC) – setup and hosting of a new GRelC service to implement and run LS use cases (e.g. biological) – grid-DBs census and requirements collection for the HUC through the SA3 Questionnaire (end of Q3) – Training and documentation resources Issues that arose: – Few feedback (from end of Q3) regarding the SA3 Questionnaire (further dissemination is needed during Y2, EGI-UF, EGI-TF, etc.) SA3 – EGI-InSPIRE EC Review 201186

87. www.egi.eu EGI-InSPIRE RI-261323 GRelC: Plans for next year The registry will be available online in PQ5 It will be the core part of the DashboardDB (available during Y2) The DashboardDB will provide specialized views, charts and statistics about the GRelC instances deployed across the EGI grid The SA3 Questionnaire will be refined and distributed among the HUCs –The feedback will be reported into the registry No changes to the current plan are foreseen SA3 – EGI-InSPIRE EC Review 2011 87

88. www.egi.eu EGI-InSPIRE RI-261323 Other Tools & Services Workflows & Metaschedulers TSA3.2.4 – Kepler: Kepler actors for gLite and Unicore fully operational. Tutorials provided, material available online. Use cases designed, created, and deployed. More use cases in year 2. TSA3.2.4 – GridWay: Available and used as standalone metascheduler. Integration GridWay - Kepler, to start in year 2. SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 88

89. www.egi.eu EGI-InSPIRE RI-261323 Workflows / Schedulers Kepler actors for gLite and Unicore fully operational First workflows with different use cases have been developed Still work required for GridWay Tutorials provided, online material Interest of communities reached SOMA2… (next) SA3 – EGI-InSPIRE EC Review 201189

90. www.egi.eu EGI-InSPIRE RI-261323 SOMA2 SOMA2 in EGI-InSPIRE –WP6: Services for the Heavy User Community (SA3) TSA3.2 Shared Services and Tools TSA3.2.4 Workflows and Schedulers –1 st Project Year Goals DCI integration Support for use of Grid middleware. Users’ X509 certificate handling. Grid enabled services’ setup Autodock 4 integration SOMA2 1.4 release Includes grid support + more –SOMA2 as a Service Currently provided for Finnish academic researchers In our roadmap we plan to offer the service to EGI as well (2 nd year) http://www.csc.fi/soma SA3 – EGI-InSPIRE EC Review 201190

91. www.egi.eu EGI-InSPIRE RI-261323 MPI/Parallel Computing (I) Cross disciplinary activity and support PY1 core objective achieved –Centralized documentation in the wiki: Admin Manual: https://wiki.egi.eu/wiki/MAN03 User Guide: https://wiki.egi.eu/wiki/MPI_User_Guide –Multiple application models and algorithms –Outreach, training and dissemination Training material centralized at user guide wiki: https://wiki.egi.eu/wiki/MPI_User_Guide#Application_ Execution Modest increase in #production sites –Improvements in monitoring => better service SA3 – EGI-InSPIRE EC Review 201191

92. www.egi.eu EGI-InSPIRE RI-261323 MPI / Parallel Computing (II) User defined processes per node allocation –implemented in EMI-1 WMS, CREAM and MPI- Start –However, not yet in production –Extensive exploitation/testing by CCMST/ INFN (See Laganà et al talk @ EGI UF 2011) –Hybrid OpenMP/MPI testing by TheoMPI (see Alfieri et al. talk @ EGI UF 2011) GP-GPU integration (with KVM) –Some open issues batch support, Lack of standardisation, Accounting etc SA3 – EGI-InSPIRE EC Review 201192

93. www.egi.eu EGI-InSPIRE RI-261323 CCMST (volunt. serv.) Virtual Research Community building –Computational Chemistry & Material Sciences and Technology (CCMST) members certification and user support Services development and provision –GriF - a user friendly tool for job distribution on the grid providing QoU and QoS information –GCres - a credit award system based on GriF –Workflows - evolution of Kepler, Pgrade and others –Packages and programs - Gaussian, GEMS, Chimere, DL_Poly, Gromacs, CMAST –Virtual laboratory - Insilico Lab SA3 – EGI-InSPIRE EC Review 2011 93