Www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE Review: Work Package 6 (SA3) Jamie Shiers CERN SA3 – Jamie Shiers.

www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE Review: Work Package 6 (SA3) Jamie Shiers CERN SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 1

www.egi.eu EGI-InSPIRE RI-261323 Activity Overview This slide will be provided by the PO It will summarise the activity in tables by: –The # partners, # people, # countries –The # PM and #FTE per country It will summarise the activity in graphics by: –The % effort of the activity within the project –The geographical spread across Europe SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 2

www.egi.eu EGI-InSPIRE RI-261323 Partner Breakdown 3 Taken from the Description of Work Participant numberParticipant NamePerson-months per participant 10KIT-627 12CSIC45 13CSC18 14CNRS83 19TCD21 INFN126 28CYFRONET6 31ARNES3 32UI SAV18 35CERN341 37EMBL44 Total732

www.egi.eu EGI-InSPIRE RI-261323 Task Breakdown SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 4 TaskActivities TSA3.1Activity Management TSA3.2Shared services and tools TSA3.3Services for High Energy Physics (HEP) TSA3.4Services for Life Sciences (LS) TSA3.5Services for Astronomy and Astrophysics (A&A) TSA3.6Services for Earth Sciences (ES) Other HUCS: Computational Chemistry and Materials Sciences and Technologies (CCMST), Fusion (F)

www.egi.eu EGI-InSPIRE RI-261323 Work Package Objectives Transition to sustainable support: –Identify tools of benefit to multiple communities –Migrate these as part of the core infrastructure –Establish support models for those relevant to individual communities SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 5

www.egi.eu EGI-InSPIRE RI-261323 HUC Activities – D6.1 SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 6 CommunityDescription, capabilities and services offered All HUCs (TSA3.2) This task provides support for grid tools and services that are used by more than one community, including Dashboards, applications such as Ganga, Services such as HYDRA and GRelC, Workflows and Schedulers (SOMA2, Kepler, Taverna) and MPI. High Energy Physics (TSA3.3) The High Energy Physics (HEP) HUC represents the 4 LHC experiments at CERN that fully rely on the use of grid computing for their offline data distribution, processing and analysis. Increasing focus is placed on common tools and solutions across these four large communities together with their re-use by other HEP experiments as well as numerous different disciplines and projects. Areas supported include: Data Management, Data Analysis, Persistency Framework and Monitoring Life Sciences (TSA3.4) The Life Science (LS) HUC originates from the use of grid technology in the medical, biomedical and bioinformatics sectors in order to connect worldwide laboratories, share resources and ease the access to data in a secure and confidential way through the health-grids. Astronomy and Astrophysics (TSA3.5) The A&A HUC is devoted to the evaluation of different solutions for the gridification of a rich variety of applications, as well as the accomplishment of a good level of interactivity among different technologies related to supercomputing, i.e. High Performance and High Throughput Computing, grid and cloud. Earth Sciences (TSA3.6) Earth Science (ES) applications cover various disciplines like seismology, atmospheric modelling, meteorological forecasting, flood forecasting and many others. Their presence in SA3 is currently centred in the implementation, deployment and maintenance of the EGDR service to provide access from the grid to resources within the Ground European Network for Earth Science Interoperations - Digital Repositories (GENESI-DR). The ES HUC includes also researchers and scientists working in the climate change domain. In particular most of them actively participate in the Climate-G use case. This use case exploits the GRelC service for distributed metadata management and the Climate-G portal as scientific gateway for this collaboration. Insert slides on shared tools and services

www.egi.eu EGI-InSPIRE RI-261323 Communities & Activities Maria Girone, IEEE MSST7 High Energy Physics The four LHC experiments use grid computing for data distribution, processing and analysis. Strong focus on common tools and solutions. Areas supported include: Data Management, Data Analysis and Monitoring. Main VOs: ALICE, ATLAS, CMS, LHCb Life Sciences Covers a variety of projects including the European Extremely Large Telescope (E-ELT), the Square Kilometre Array (SKA) and Cerenkov Telescope Array (CTA). Activities focus on visualisation tools and database/catalog access from the grid. Covers seismology, atmospheric modelling, meteorological forecasting, flood forecasting and climate change. Provides access from the grid to resources within the Ground European Network for Earth Science Interoperations - Digital Repositories (GENESI-DR). Also assists scientists working on climate change via the Climate-G testbed. Astronomy & Astrophysics Earth Sciences Life Sciences Focuses on medical, biomedical and bioinformatics sectors to connect worldwide laboratories, share resources and ease access to data in a secure and confidential way. Supports 4 VOs (biomed, lsgri, vlemed and pneumogrid) across 6 NGIs via the Life Science Grid Community.

www.egi.eu EGI-InSPIRE RI-261323 Communities & Activities Maria Girone, IEEE MSST8 High Energy Physics The four LHC experiments use grid computing for data distribution, processing and analysis. Strong focus on common tools and solutions. Areas supported include: Data Management, Data Analysis and Monitoring. Main VOs: ALICE, ATLAS, CMS, LHCb Life Sciences Covers a variety of projects including the European Extremely Large Telescope (E-ELT), the Square Kilometre Array (SKA) and Cerenkov Telescope Array (CTA). Activities focus on visualisation tools and database/catalog access from the grid. Covers seismology, atmospheric modelling, meteorological forecasting, flood forecasting and climate change. Provides access from the grid to resources within the Ground European Network for Earth Science Interoperations - Digital Repositories (GENESI-DR). Also assists scientists working on climate change via the Climate-G testbed. Astronomy & Astrophysics Earth Sciences Life Sciences Focuses on medical, biomedical and bioinformatics sectors to connect worldwide laboratories, share resources and ease access to data in a secure and confidential way. Supports 4 VOs (biomed, lsgri, vlemed and pneumogrid) across 6 NGIs via the Life Science Grid Community. These and other communities / projects supported by shared tools & services

www.egi.eu EGI-InSPIRE RI-261323 Services for HUCs 1.Successfully supported major production computing at an unprecedented scale – both quantitatively and qualitatively 2.Successfully delivered common solutions in a variety of areas – with other activities in progress 3.Actively participated in EGI Technical & User Forum via presentations, tutorials and demos 4.Broadened the use of grid technology and HUC services to related projects within the HUC domain (such as unfunded – by EGI-InSPIRE – LS / ES projects) 5.Completed first round of Milestones & Deliverables together with associated technical work 6.Identified – across all HUC communities – areas of common technology investigation for the future 7.Developed a S.W.O.T. analysis of each main discipline and made significant steps on the road to sustainability SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 9

www.egi.eu EGI-InSPIRE RI-261323 Service Quantity & Quality The first year of EGI-InSPIRE saw the use of grid computing at an unprecedented scale! –More than 100 CPU-millennia delivered; –More than 50 PB of data stored; –Data transfer rates of 200TB/day At the same time, its power in turning scientific data into publications at record speed was publically acknowledged (Economist, July 2010)July 2010 And service delivery was typically smooth with a small number of problems requiring in-depth investigation Quality plots in backup slides SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 10

www.egi.eu EGI-InSPIRE RI-261323 Common Solutions – Examples SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 11 GangaExtensively used as a “gridification tool” by many projects / disciplines. This includes not only communities within EGI- InSPIRE but also others in many fields Numerous non- HEP and non- HUC projects Mini- Dashboard To be used with Ganga to monitor Ganga-based activity on the grid. Used by EnviroGRIDS and offered to NA3 EnviroGRIDS, NA3 Experiment Dashboards Common schema and code base for all job monitoring applications: job summary, historical view & task monitoring – implemented from July 2010 ATLAS + CMS GRelCExploited by the Earth Science (ES) community (Climate-G testbed) and by other projects/disciplines related to Environment and Bioinformatics LS, A&A, ES MPIHigh impact on multiple user communitiesCCMST, A&A, F FrameworksUse of LHCb’s DIRAC framework by LCD/ILC and Belle collaborations. Investigation of DIRAC by ES + others HEP (beyond LHC), ES Data Management Data popularity (dynamic data placement / caching), consistency of catalogs / storage LHC VOs Site Stress Testing HammerCloud service used at ATLAS, CMS and LHCb. Fully applicable to other VOs / communities HEP

www.egi.eu EGI-InSPIRE RI-261323 Common Solutions Maria Girone, IEEE MSST12 Life Sciences Astronomy & Astrophysics Earth Sciences Life Sciences Connect various tools with multiple communities: the main disciplines and also Fusion, Computational Chemistry Fusion HEP Comp Chem

www.egi.eu EGI-InSPIRE RI-261323 Common Solutions Maria Girone, IEEE MSST13 Life Sciences Astronomy & Astrophysics Earth Sciences Life Sciences Ganga: job definition and management in HEP (ATLAS and LHCb, Compass, Harp), Fusion, L&E Sciences. Fusion HEP Comp Chem

www.egi.eu EGI-InSPIRE RI-261323 Common Solutions Maria Girone, IEEE MSST14 Life Sciences Astronomy & Astrophysics Earth Sciences Life Sciences Hammercloud: a site stress testing system to validate site usability. Used by ATLAS, CMS and LHCb. Fusion HEP Comp Chem

www.egi.eu EGI-InSPIRE RI-261323 Common Solutions Maria Girone, IEEE MSST15 Life Sciences Astronomy & Astrophysics Earth Sciences Life Sciences Dashboards: single entry point to monitoring of all 4 LHC experiment activities on the grid. (Mini) Dashboard also used by E&L Sciences. Fusion HEP Comp Chem

www.egi.eu EGI-InSPIRE RI-261323 Common Solutions Maria Girone, IEEE MSST16 Life Sciences Astronomy & Astrophysics Earth Sciences Life Sciences DIRAC: workload and data management used by LHCb plus other HEP experiments with interest from Earth Sciences. Fusion HEP Comp Chem

www.egi.eu EGI-InSPIRE RI-261323 Common Solutions Maria Girone, IEEE MSST17 Life Sciences Astronomy & Astrophysics Earth Sciences Life Sciences MPI: used by A&A, Fusion, Computational Chemistry to handle parallel execution in grid environments Fusion HEP Comp Chem

www.egi.eu EGI-InSPIRE RI-261323 Common Solutions Maria Girone, IEEE MSST18 Life Sciences Astronomy & Astrophysics Earth Sciences Life Sciences GRelC: set of advanced data grid services to manage Databases on the Grid. Used by L&E Sciences and A&A. Fusion HEP Comp Chem

www.egi.eu EGI-InSPIRE RI-261323 EGI TF & UF Actively participated in both EGI Technical Forum (September 2010, Amsterdam) and User Forum (April 2011 Vilnius) –TF: Two sessions covering overview of tasks and sub- tasks; One session dedicated to discussion of Common Requirements –UF: Numerous presentations, tutorials, demonstrations covering all areas of activity –All sessions well attended with good feedback EGI TF 2011: sessions planned to highlight main achievements of WP in meeting goals plus a mini- workshop on sustainability SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 19

www.egi.eu EGI-InSPIRE RI-261323 Broadening Use of the Grid Concrete examples: –Use of HUC solutions and expertise for “3 generations” of HEP experiments – LEP to ILC –Bi-directional sharing of solutions and techniques not only for those activities explicitly supported through EGI-InSPIRE but also other projects, such as PARTNER, ULICE and EnviroGRIDS –Other possible Earth Science projects in the pipeline SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 20

www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 Domain-Specific Work A Summary of the Main Sub-Tasks: Details in Backup Slides SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 21

www.egi.eu EGI-InSPIRE RI-261323 Services for HEP Distributed Analysis Support –Support for the tools used by the experiments; common error-logger for ATLAS/CMS/LHCb Data Management Support –Dynamic caching / data popularity: common solutions deployed Persistency Framework –Event and detector data: common to three LHC experiments Monitoring –All aspects of production and analysis: additional common solutions deployed SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 22

www.egi.eu EGI-InSPIRE RI-261323 Services for Life Sciences TSA3.4 - Virtual Research Community building –Life Sciences Grid Community (LSGC) Services development and provision –TSA3.2.1 - Dashboard: design phase –TSA3.2.3 - Data encryption service: prototype deployed –TSA3.2.3 - Database interface: GRelC deployed –TSA3.2.4 - Workflows: work on Taverna to start in year 2 –TSA3.4 - CoreBio services: work to start in year 2 NAx - > - EGI-InSPIRE EC Review 2011 23

www.egi.eu EGI-InSPIRE RI-261323 Services for A&A Preparatory studies: –Visualization tools: VisIVO integration (TSA3.5.2) –Parallel (MPI/OpenMP) and GPU computing using CUDA (TSA3.5.3) –Database services and integration with Virtual Observatory (TSA3.5.4) Achieved results: –ms608: Gridification of VisIVO and VisIVO Service (TSA3.5.2) –Parallel programming: testing activities with cosmological simulations codes (FLY, Gadget,Flash) (TSA3.5.3) –Database services analysis (TSA3.5.4) SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 24

www.egi.eu EGI-InSPIRE RI-261323 Services for ES TSA 3.6 Support ES Activities in ES communities and projects, and carried out by researchers & students in Universities Main activity by proposal: access to GENESI-DEC –Status : Webservice available and validated with application –Extensions dependent on GENESI-DEC progress –Further developments in Task: Integration with available GEOSS services to access Genesi and CLimate Data from ESG Since Jan 2011 common developments with the climate Earth System Grid (ESG) –Access to ESG data from EGI-Infrastructure and vice versa –Main problem to solve: A & A due to different federations –Institute IPSL/CNRS, IPGP now unfunded partner in TSA3.6 SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 25

www.egi.eu EGI-InSPIRE RI-261323 Shared Tools & Services SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 26

www.egi.eu EGI-InSPIRE RI-261323 S.W.O.T. Analyses – D6.2 SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 27 DisciplineStrengthsWeaknessesOpportunitiesThreats HEP24 x 7 petascale computing Time to resolve some incidents Increased commonality Need to adapt to new technologies Life SciencesRamp-up of LSGC Issues with use of multi-grids Community growth No long-term funding Astronomy & Astrophysics Successful use of DCIs Coordination of A&A activities Wider use of DCIs Funding Earth SciencesExisting user community Diversity of demands and technologies Community growth Funding GRelCCommunity based approach, cross - discipline Little feedback about the SA3 DB questionnaire Community growth Lack of use of registry MPIWidely regarded as an important tool External factorsUse of large SMPs / GPGPUs Need for standardisation Notable overlap in these independent analyses

www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 Use of Resources Breakdown by Task & Partner SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 28

www.egi.eu EGI-InSPIRE RI-261323 Effort Breakdown (1/2) SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 29 TaskPartnerWorked PM FundedCommitted PMAchieved PM % TSA3.135-CERN8.86.0147% TSA3.2.114A-CNRS00.70% TSA3.2.114C-HealthGrid0.33.38% TSA3.2.135-CERN22.420.0112% TSA3.2.231B-JSI0.51.047% TSA3.2.232-UI SAV2.66.043% TSA3.2.235-CERN17.120.086% TSA3.2.314A-CNRS7.53.3224% TSA3.2.314B-CEA02.70% TSA3.2.321E-SPACI4.09.044% TSA3.2.412C-CIEMAT6.96.0115% TSA3.2.413-CSC5.56.091% TSA3.2.428C-ICBP0.72.033% TSA3.2.437-EMBL06.00%

www.egi.eu EGI-InSPIRE RI-261323 Effort Breakdown (2/2) SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 33 TaskPartnerWorked PM FundedCommitted PMAchieved PM % TSA3.2.512A-CSIC5.69.062% TSA3.2.519-TCD7.0 100% TSA3.2.521D-UNIPG11.53.0385% TSA3.321A-INFN020.00% TSA3.335-CERN65.667.797% TSA3.414A-CNRS2.63.377% TSA3.414C-HealthGrid2.96.346% TSA3.437-EMBL08.70% TSA3.521C-INAF8.310.083% TSA3.610G-FRAUNHOFER2.19.023% TSA3.614A-CNRS4.58.057% Total:186.3244.076%

www.egi.eu EGI-InSPIRE RI-261323 Effort Breakdown (2/2) SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 34 TaskPartnerWorked PM FundedCommitted PMAchieved PM % TSA3.2.512A-CSIC5.69.062% TSA3.2.519-TCD7.0 100% TSA3.2.521D-UNIPG11.53.0385% TSA3.321A-INFN020.00% TSA3.335-CERN65.667.797% TSA3.414A-CNRS2.63.377% TSA3.414C-HealthGrid2.96.346% TSA3.437-EMBL08.70% TSA3.521C-INAF8.310.083% TSA3.610G-FRAUNHOFER2.19.023% TSA3.614A-CNRS4.58.057% Total:186.3244.076% If the Grid Observatory effort is removed, the underspend in PY1 is > 25%. Hopefully, this effort will be made up early in PY2.

www.egi.eu EGI-InSPIRE RI-261323 Plans for Next Year In PY2 we will continue to identify and deliver common solutions within and across existing communities, further the work on sustainability and address key areas of weakness / concern identified in PY1 This includes the evaluation – and eventual deployment – of new solutions / techniques No modifications to the DoW are foreseen Carry-over of PM to later in the project? SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 35

www.egi.eu EGI-InSPIRE RI-261323 Review of Objectives SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 36 ObjectiveStatus Supporting the tools, services and capabilities required by different HUCs Achieved – work continues in PY2 / PY3 Identifying the tools, services and capabilities currently used by the HUCs that can benefit all user communities and to promote their adoption Several additional items identified and shared across other communities – work will expand and continue in PY2 / PY3 Migrating the tools, services and capabilities that could benefit all user communities into a sustainable support model as part of the core EGI infrastructure Not started – needs further work and discussion Establishing a sustainable support model for the tools, services and capabilities that will remain relevant to single HUCs Collaborative support is the basic model for sustainability that does not depend on individual partners nor specific project funding. A workshop on sustainability at the EGI TF is proposed – work continues and expands in PY2 and PY3. See also D6.2.

www.egi.eu EGI-InSPIRE RI-261323 Summary Successfully supported major production computing at an unprecedented scale – both quantitatively and qualitatively Successfully delivered common solutions in a variety of areas – with other activities in progress Actively participated in EGI Technical & User Forum via presentations, tutorials and demos Broadened the use of grid technology and HUC services to related projects within the HUC domain (such as unfunded – by EGI-InSPIRE – LS / ES projects) Completed first round of Milestones & Deliverables together with associated technical work Identified – across all HUC communities – areas of common technology investigation for the future Developed a S.W.O.T. analysis of each main discipline and made significant steps on the road to sustainability SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 37

www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 BACKUP SLIDES SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 38

www.egi.eu EGI-InSPIRE RI-261323 Milestones & Deliverables SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 39 Milestone / Deliverable Due DateLead Partner (#)Title MS601PM1CSC (13)HUC Contact points and the support model MS602PM4INFN (21)HUC Software Roadmap MS603PM4CERN (35)Services for High Energy Physics D6.1PM4CERN (35)Capabilities offered by the HUCs to other communities MS604PM4CNRS (14)Services for the Life Science Community MS605PM8TCD (19)Training and dissemination event D6.2PM9CERN (35)Sustainability plans for the HUC activities MS606PM10INFN (21)HUC Software Roadmap D6.3PM11CERN (35)Annual Report on the Tools and Services of the HUCs MS607PM12CNRS (14)Hydra service deployment MS608PM12INFN (21)Integration of the VisIVO server with the production infrastructure

www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 HEP SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 40

www.egi.eu EGI-InSPIRE RI-261323 WLCG Service Incidents SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 41

www.egi.eu EGI-InSPIRE RI-261323 SIRs – Time to Resolution SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 42

www.egi.eu EGI-InSPIRE RI-261323 GGUS Tickets – LHC VOs SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 43

www.egi.eu EGI-InSPIRE RI-261323 Data Management Crucial area for LHC and other HEP experiments –Data volumes: tens of PB/year, rates: up to 200TB/day between sites, several hundred active analysis users / experiment, 1M analysis jobs / day Experience from first data taking has shown that some assumptions on data placement are no longer optimal –based on decade+ old model “MONARC” which assumed network was scarce and expensive resource Working to adapt computing models to reflect such changes Maria Girone, IEEE MSST44

www.egi.eu EGI-InSPIRE RI-261323 Data Placement & Dynamic Caching Based on MONARC, the initial phase of LHC data distribution was based on static pre-placement –Significant fraction of such data never read! Computing models now driving towards dynamic data placement –Replication is based on usage (“popularity”) – this results in better network and storage utilization Implemented first for ATLAS, now for CMS and LHCb Maria Girone, IEEE MSST45

www.egi.eu EGI-InSPIRE RI-261323 Catalogue Consistency With 50PB of data storage across a large number of sites worldwide, inconsistencies can easily arise! –Data that resides on Storage Elements but not in various catalogs (grid, experiment) referred to as “Dark Data” One site recently reported 70TB dark data! Using a messaging-based system, various catalogs and SEs can talk to each other and implement lazy synchronization Maria Girone, IEEE MSST46

www.egi.eu EGI-InSPIRE RI-261323 Data Analysis Support Covers the final stage of data processing leading on to publication –Large number of users/month (~1000) and analysis jobs/day (~1M) running across (~100) Tier2 and other sites – “chaotic” data access –All frameworks support heterogeneous back-ends Ganga (ATLAS, LHCb) used by 10 other communities and 500 – 600 users Common site stress testing system (Hammercloud) used by ATLAS, CMS and LHCb Areas of commonality and optimization –Move to “community support” model –Simplify data access and improve monitoring –Use common components and frameworks, such as for job submission and file transfer (built on gLite /EMI FTS) An area of potential future common work and simplification Maria Girone, IEEE MSST47

www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 LS SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 48

www.egi.eu EGI-InSPIRE RI-261323 Life Sciences Grid Community LSGC: http://wiki.healthgrid.org/LSVRC:Index http://wiki.healthgrid.org/LSVRC:Index –4 VOs, 6 NGIs, HG association, 2 EU projects –Communication channels: wiki, mailing lists, monthly phone conferences Technical team (shifters) –Infrastructure monitoring and troubleshooting (Nagios server with LS resource probes) User management tools – early design phase –User registration and management DB –To be integrated in LS Dashboard SA3 – Johan Montagnat – EGI-InSPIRE EC Review 2011 49

www.egi.eu EGI-InSPIRE RI-261323 LS Dashboard Need to integrate –Nagios monitoring dedicated interface (possibly based on GOC dashboard) –VRC-level accounting information –User management tools (possibly based on VOMRS) NAx - > - EGI-InSPIRE EC Review 2011 50

www.egi.eu EGI-InSPIRE RI-261323 Data encryption service Hydra server –Encryption keystore server based on Shamir’s secret sharing algorithm –Software packages available for gLite 3.1 –Installation procedure documented for gLite 3.0 Current status (MS607) –Working on gLite 3.1, not yet on gLite 3.2 –Client CLIs to be installed on all LS supporting sites Perspectives –Deployment of a three-heads hydra server on three sites SA3 – Johan Montagnat – EGI-InSPIRE EC Review 2011 51

www.egi.eu EGI-InSPIRE RI-261323 Database interface GRelC (Grid Relational Catalog) service provision to support LS use cases Discussion about new LS use cases and data resources analysis Identification of a couple of biological databases to be ported in grid (relational DBs) Contribution to the SA3 questionnaire related to “grid-databases” SA3 – Johan Montagnat – EGI-InSPIRE EC Review 2011 52

www.egi.eu EGI-InSPIRE RI-261323 LS issues, mitigation & perspectives VOMS and LFC servers are single point of failures –Replication procedures being set up Infrastructure monitoring is time consuming –Scheduled downtimes should be better reflected in BDII –Dedicated view of Nagios results in LS dashboard –Nagios probes improvements LSGC is a multi-VOs / multi-grids community –No tooling available to manage VRCs –Multi-grids hardly addressed in the context of EGI Few feedback on the SA3 questionnaire –More dissemination is needed in PY2 SA3 – Johan Montagnat – EGI-InSPIRE EC Review 2011 53

www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 A&A SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 54

www.egi.eu EGI-InSPIRE RI-261323 A&A Grid Community 23 VOs and 7 NGIs –The most part of active A&A VOs relate to the astroparticle physics community –Communication channels: wiki (under construction) mailing lists (a general mailing list currently in place, others more specialized could follow) phone conferences (organized on a monthly basis) –A&A VRC meetings and workshops The first one of the EGI era is under organization in Paris (7 November 2011, ADASS Conference) SA3 – Johan Montagnat – EGI-InSPIRE EC Review 2011 55

www.egi.eu EGI-InSPIRE RI-261323 VisIVO Porting of VisIVO server in Grid. –Preparatory activity: enabling the usage of VisIVO directly within a code during the production phase. A software layer has been developed using the internal arrays and without the need of producing intermediate files. A library of VisIVO was designed and implemented –The first issue of the gridified version of VisIVO server has been released in April 2011 (milestone MS608). SA3 – Johan Montagnat – EGI-InSPIRE EC Review 2011 56

www.egi.eu EGI-InSPIRE RI-261323 A&A issues, mitigation & perspectives A&A is a complex community: its coordination is quite challenging. No EU A&A projects currently funded  shortage of funds  coordination activity tricky  no easy deployment of tools and services Our willingness is to continue the coordination and the efforts to strengthen the community –We rely on EGI.eu and NGIs support –We exploit as much as possible collaborative tools and services to establish robust and stable communications SA3 – Johan Montagnat – EGI-InSPIRE EC Review 2011 57

www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 ES SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 58

www.egi.eu EGI-InSPIRE RI-261323 ES HUC ES HUC VOs: –“ESR VO” „catch all ES people“ provides resources and support for ES projects and anorganized ES users –EU project VERCE (seismology) will start with this VO and will work with SA3.6 VOs associated to ES HUC –See http://www.euearthsciencegrid.org/content/egee-virtual- organisations-earth-sciencehttp://www.euearthsciencegrid.org/content/egee-virtual- organisations-earth-science ES HUC Activities: –Yearly session at European Geoscience Union, General Assembly 2011 about “ES and E-infrastructures SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 59

www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 Other Tools & Services SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 60

www.egi.eu EGI-InSPIRE RI-261323 GRelC: Activity (I) NAx - > - EGI-InSPIRE EC Review 2011 61 User support through the implementation of a grid- database “registry” (EGI Database of Databases) – Easy search and discovery (cross-VO) of grid-DB resources distributed across the EGI grid. – Community-based approach to attract new users and address sustainability SA3 Questionnaire – A census about database resources, related needs and future plan – Distributed among the HUCs (end of Q3) – Few feedback; more dissemination during Y2

www.egi.eu EGI-InSPIRE RI-261323 GRelC: Activity (II) NAx - > - EGI-InSPIRE EC Review 2011 62 The back-end modules of the “registry” finalized during Y1: – A MySQL catalog for the registry – Several Java classes to manage charts, grid-DB, VOs, GRelC services, community-oriented aspects Front-end modules related to the “registry” finalized in Y1 (now tested, on line during Q5) – Registry view completed – Grid-Database view completed – Integration of Web2.0 (Mash-up, Google Maps, permalinks) and community-oriented aspects (comments, scores, discussions groups, etc.)

www.egi.eu EGI-InSPIRE RI-261323 GRelC: Registry Snapshots NAx - > - EGI-InSPIRE EC Review 2011 63

www.egi.eu EGI-InSPIRE RI-261323 GRelC: User support NAx - > - EGI-InSPIRE EC Review 2011 64 HUC support in terms of: – grid-metadata management for the Earth Science and Environmental context (e.g. Climate-G, CMCC) – setup and hosting of a new GRelC service to implement and run LS use cases (e.g. biological) – grid-DBs census and requirements collection for the HUC through the SA3 Questionnaire (end of Q3) – Training and documentation resources Issues that arose: – Few feedback (from end of Q3) regarding the SA3 Questionnaire (further dissemination is needed during Y2, EGI-UF, EGI-TF, etc.)

www.egi.eu EGI-InSPIRE RI-261323 GRelC: Plans for next year The registry will be available online in PQ5 It will be the core part of the DashboardDB (available during Y2) The DashboardDB will provide specialized views, charts and statistics about the GRelC instances deployed across the EGI grid The SA3 Questionnaire will be refined and distributed among the HUCs –The feedback will be reported into the registry No changes to the current plan are foreseen NAx - > - EGI-InSPIRE EC Review 2011 65

www.egi.eu EGI-InSPIRE RI-261323 Workflows / Schedulers –TSA3.2.4 – Kepler: Kepler actors for gLite and Unicore fully operational. Tutorials provided, material available online. Use cases designed, created, and deployed. More use cases in year 2. –TSA3.2.4 – GridWay: Available and used as standalone metascheduler. Integration GridWay - Kepler, to start in year 2. SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 66

www.egi.eu EGI-InSPIRE RI-261323 Workflows / Schedulers Kepler actors for gLite and Unicore fully operational First workflows with different use cases have been developed Still work required for GridWay Tutorials provided, online material Interest of communities reached SOMA2… (next)

www.egi.eu EGI-InSPIRE RI-261323 SOMA2 SOMA2 in EGI-InSPIRE –WP6: Services for the Heavy User Community (SA3) TSA3.2 Shared Services and Tools TSA3.2.4 Workflows and Schedulers –1 st Project Year Goals DCI integration Support for use of Grid middleware. Users’ X509 certificate handling. Grid enabled services’ setup Autodock 4 integration SOMA2 1.4 release Includes grid support + more –SOMA2 as a Service Currently provided for Finnish academic researchers In our roadmap we plan to offer the service to EGI as well (2 nd year) http://www.csc.fi/soma

www.egi.eu EGI-InSPIRE RI-261323 MPI/Parallel Computing (I) Cross disciplinary activity and support PY1 core objective achieved –Centralized documentation in the wiki: Admin Manual: https://wiki.egi.eu/wiki/MAN03 User Guide: https://wiki.egi.eu/wiki/MPI_User_Guide –Multiple application models and algorithms –Outreach, training and dissemination Training material centralized at user guide wiki: https://wiki.egi.eu/wiki/MPI_User_Guide#Application_ Execution Modest increase in #production sites –Improvements in monitoring => better service SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 69

www.egi.eu EGI-InSPIRE RI-261323 MPI / Parallel Computing (II) User defined processes per node allocation –implemented in EMI-1 WMS, CREAM and MPI- Start –However, not yet in production –Extensive exploitation/testing by CCMST/ INFN (See Laganà et al talk @ EGI UF 2011) –Hybrid OpenMP/MPI testing by TheoMPI (see Alfieri et al. talk @ EGI UF 2011) GP-GPU integration (with KVM) –Some open issues batch support, Lack of standardisation, Accounting etc SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 70

www.egi.eu EGI-InSPIRE RI-261323 CCMST (volunt. serv.) Virtual Research Community building –Computational Chemistry & Material Sciences and Technology (CCMST) members certification and user support Services development and provision –GriF - a user friendly tool for job distribution on the grid providing QoU and QoS information –GCres - a credit award system based on GriF –Workflows - evolution of Kepler, Pgrade and others –Packages and programs - Gaussian, GEMS, Chimere, DL_Poly, Gromacs, CMAST –Virtual laboratory - Insilico Lab NAx - > - EGI-InSPIRE EC Review 2011 71

Www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE Review: Work Package 6 (SA3) Jamie Shiers CERN SA3 – Jamie Shiers.

Similar presentations

Presentation on theme: "Www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE Review: Work Package 6 (SA3) Jamie Shiers CERN SA3 – Jamie Shiers."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE Review: Work Package 6 (SA3) Jamie Shiers CERN SA3 – Jamie Shiers.

Similar presentations

Presentation on theme: "Www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE Review: Work Package 6 (SA3) Jamie Shiers CERN SA3 – Jamie Shiers."— Presentation transcript:

Similar presentations

About project

Feedback