Download presentation
Presentation is loading. Please wait.
1
The EGI-InSPIRE Second Year Project Review
27 & 28 June 2012 EGI.eu, Amsterdam
2
SA3: Heavy User Communities
Jamie Shiers CERN
3
SA3 Overview 9 Countries 11 Beneficiaries 732 PMs 20.4 FTEs CERN
France Slovenia Slovakia Italy Spain Finland Poland EMBL Ireland Germany WP Task Beneficiary Total PMs WP6-G TSA3.1 CERN 18 TSA3.2.1 CNRS 12 60 TSA3.2.2 ARNES 3 UI SAV TSA3.2.3 INFN 27 TSA3.2.4 CSIC CSC CYFRONET 6 EMBL TSA3.2.5 TCD 21 9 TSA3.3 203 TSA3.4 29 26 TSA3.5 30 TSA3.6 KIT-G 24 9 Countries 11 Beneficiaries 732 PMs 20.4 FTEs
4
SA3 Objectives Transition to sustainable support:
Identify tools of benefit to multiple communities Migrate these as part of the core infrastructure Establish support models for those relevant to individual communities
5
Achievements Achievements and Future plans are based on D6.6 – the SA3 Annual Report Rather than make two passes over all of the communities, tools and services they are presented together: By community for TSA3.3 – TSA3.6 HEP, LS, A&A, ES By tool for TSA3.2 (shared tools and services)
6
Achievements in Context
To give an explicit example, we give the case of HEP and support for WLCG The 3 phases of EGEE (I/II/III) almost completely overlapped with final preparations for LHC data taking: WLCG Service Challenges 1-4, CCRC’08, STEP’09 EGI-InSPIRE SA3 covered virtually all the initial data taking run (3.5TeV/beam) of the LHC: first data taking and discoveries! The transition from EGEE to EGI was non-disruptive Continuous service improvement was seen Problems encountered during initial data taking were rapidly solved Significant progress in the identification and delivery of common solutions Active participation in the definition of the future evolution of WLCG
7
LHC(b) – Results! See also plenary at CHEP 2012: LHC experience so far
8
WLCG Service Incidents
STEP’09
9
Resolution of Incidents
Data taking
10
Achievements & Plans Comment: Summary tables need to be updated; first agree if this is a useful approach (used last year)
11
Communities & Activities
High Energy Physics TSA3.3 The LHC experiments use grid computing for data distribution, processing and analysis. Strong focus on common tools and solutions. Areas supported include: Data Management, Data Analysis and Monitoring. Main VOs: ALICE, ATLAS, CMS, LHCb but covers many other HEP experiments + related projects. Focuses on medical, biomedical and bioinformatics sectors to connect worldwide laboratories, share resources and ease access to data in a secure and confidential way. Supports 4 VOs (biomed, lsgri, vlemed and pneumogrid) across 6 NGIs via the Life Science Grid Community. Life Sciences TSA3.4 Life Sciences Covers the European Extremely Large Telescope (E-ELT), the Square Kilometre Array (SKA) and Cerenkov Telescope Array (CTA) and others. Activities focus on visualisation tools and database/catalog access from the grid. Main VOs: Argo, Auger, Glast, Magic, Planck, CTA, plus others (total 23) across 7 NGIs. Astronomy & Astrophysics TSA3.5 Large variety of ES disciplines. Provides also access from the grid to resources within the Ground European Network for Earth Science Interoperations - Digital Earth Community (GENESI-DEC); assists scientists working on climate change via the Climate-G testbed. Main VOs: esr, egeode, climate-g, env.see-grid-sci.eu, meteo.see-grid-sci.eu, seismo.see-grid-sci.eu- support by ~20 NGIs Earth Sciences TSA3.6
12
Communities & Activities
These and other communities supported by shared tools & services High Energy Physics TSA3.3 The LHC experiments use grid computing for data distribution, processing and analysis. Strong focus on common tools and solutions. Areas supported include: Data Management, Data Analysis and Monitoring. Main VOs: ALICE, ATLAS, CMS, LHCb but covers many other HEP experiments + related projects. Focuses on medical, biomedical and bioinformatics sectors to connect worldwide laboratories, share resources and ease access to data in a secure and confidential way. Supports 4 VOs (biomed, lsgri, vlemed and pneumogrid) across 6 NGIs via the Life Science Grid Community. Life Sciences TSA3.4 Life Sciences Covers the European Extremely Large Telescope (E-ELT), the Square Kilometre Array (SKA) and Cerenkov Telescope Array (CTA) and others. Activities focus on visualisation tools and database/catalog access from the grid. Main VOs: Argo, Auger, Glast, Magic, Planck, CTA, plus others (total 23) across 7 NGIs. Astronomy & Astrophysics TSA3.5 Large variety of ES disciplines. Provides also access from the grid to resources within the Ground European Network for Earth Science Interoperations - Digital Earth Community (GENESI-DEC); assists scientists working on climate change via the Climate-G testbed. Main VOs: esr, egeode, climate-g, env.see-grid-sci.eu, meteo.see-grid-sci.eu, seismo.see-grid-sci.eu- support by ~20 NGIs Earth Sciences TSA3.6
13
Communities & Activities
High Energy Physics TSA3.3 The LHC experiments use grid computing for data distribution, processing and analysis. Strong focus on common tools and solutions. Areas supported include: Data Management, Data Analysis and Monitoring. Main VOs: ALICE, ATLAS, CMS, LHCb but covers many other HEP experiments + related projects. Focuses on medical, biomedical and bioinformatics sectors to connect worldwide laboratories, share resources and ease access to data in a secure and confidential way. Supports 4 VOs (biomed, lsgri, vlemed and pneumogrid) across 6 NGIs via the Life Science Grid Community. Life Sciences TSA3.4 Life Sciences Covers the European Extremely Large Telescope (E-ELT), the Square Kilometre Array (SKA) and Cerenkov Telescope Array (CTA) and others. Activities focus on visualisation tools and database/catalog access from the grid. Main VOs: Argo, Auger, Glast, Magic, Planck, CTA, plus others (total 23) across 7 NGIs. Astronomy & Astrophysics TSA3.5 Large variety of ES disciplines. Provides also access from the grid to resources within the Ground European Network for Earth Science Interoperations - Digital Earth Community (GENESI-DEC); assists scientists working on climate change via the Climate-G testbed. Main VOs: esr, egeode, climate-g, env.see-grid-sci.eu, meteo.see-grid-sci.eu, seismo.see-grid-sci.eu- support by ~20 NGIs Earth Sciences TSA3.6
14
Task Leader: Maria Girone
Services for HEP Activity PY2 Results Distributed Analysis Support for the tools used by the experiments; common error-logger for ATLAS/CMS/LHCb Data Management Dynamic caching / data popularity – move away from static data placement: common solutions deployed; others under development Persistency Framework Handles the event and detector conditions data from the experiments Monitoring / Dashboards All aspects of production and analysis: additional common solutions deployed Task Leader: Maria Girone
15
Services for Life Sciences
(SUB)TASK PY2 RESULTS TSA3.4 Virtual Research Community building Life Sciences Grid Community (LSGC) Service development and provisioning TSA3.4.1 Dashboard: design phase TSA3.4.2 Data encryption service: prototype deployed TSA3.4.3 Database interface: GRelC deployed TSA3.4.4 Workflows: work on Taverna to start in year 2 TSA3.4.5 CoreBio services: work to start in year 2 Task leader: Johan Montagnat, Deputy: Yannick Legré
16
Task leader: Claudio Vuerli, Deputy: Giuliano Taffoni
Services for A&A (SUB)TASK PY2 RESULTS Preparatory studies TSA3.5.2 Visualisation tools: VisIVO integration (Vis. i/f to V Ob) TSA3.5.3 Parallel (MPI/OpenMP) and GPU computing using CUDA TSA3.5.4 Database services and integration with Virtual Observatory (V Ob) Achieved results MS608: Gridification of VisIVO and VisIVO Service Parallel programming: testing activities with cosmological simulations codes (FLY, Gadget, Flash) Database services analysis Task leader: Claudio Vuerli, Deputy: Giuliano Taffoni
17
Services for ES (SUB)TASK PY2 RESULTS TSA3.6
Support ES Activities in ES communities and projects, and carried out by researchers & students in Universities Main activity by proposal: access to GENESI-DEC Status: Webservice available and validated with application Extensions dependent on GENESI-DEC progress Further developments in Task: Integration with available GEOSS services to access Genesi and Climate Data from ESG Since January 2011 common developments with Earth System Grid (ESG) Access to ESG data from EGI infrastructure and vice versa Main problem: authentication / authorisation due to different federations Institute IPSL/CNRS, IPGP now unfunded partner in TSA3.6 Task leader: Horst Schwictenberg, Deputy: André Gemünd
18
(Detailed) Achievements & Plans
Summarized from D6.6 Comment: these slides will most likely be backup
19
Analysis Tools & Support
Services for HEP Analysis Tools & Support HammerCloud - a Grid site testing service. Modus operandi Site availability & functionality; frequent, short jobs; Site commissioning & benchmarking; on-demand stress testing. Development highlights Rationalisation of codebase; Extension of ‘job robot’ functionality for grid site validation; Migration of databases to CERN-IT hosted MySQL service. Future direction Standard benchmarks for storage element evaluation; Improvements to packaging and installation procedures; Central reporting for LHCb functionality tests; Pan-VO error and performance correlation; Interface for user-initiated testing of grid sites.
20
Analysis Tools & Support
Services for HEP Analysis Tools & Support CMS CRAB Client - The CMS Remote Analysis Builder Development highlights Maintenance of CRAB2. Development and commissioning of CRAB3. Future direction Wide scale testing of entire CRAB3 software chain. Reimplementation of monitoring service. Centralised CouchDB-based solution. Monitoring test results collected from distributed agents. Support for merging and publication of (user) output files. Support for HammerCloud-driven task submission. Further developments based on user-feedback.
21
Data Management Tools & Support
Services for HEP Data Management Tools & Support ATLAS Distributed Data Management - Replication, access and bookkeeping of ATLAS data. Development highlights DDM infrastructure visible in CERN-IT hosted Service Level Status overview pages. DDM site-service optimisation (e.g. data-placement utilities) Future direction Focus on code maintenance and support operations.
22
Data Management Tools & Support
Services for HEP Data Management Tools & Support CMS Data Management - The CMS Popularity Service; monitoring data-access patterns. Development highlights. Deployment of site cleaning agent. Based on original ATLAS codebase; both implementations share a common core. Modular, expandable architecture. Fully automated scanning of grid site storage elements. Obsolete and ‘unpopular’ datasets removed from sites approaching capacity. Database backend and web frontend improved. Future direction Usage-pattern analysis for XRootD accessed datasets. Dynamic data placement. Replication of ‘popular’ datasets.
23
Data Management Tools & Support
Services for HEP Data Management Tools & Support LHCb DIRAC – Framework for all grid computing activities for the LHCb VO (also used by others). Development highlights LHCb-specific functionality separated from DIRAC. Streamlined development cycle for LHCb community feature requests. Future direction Data consistency service consolidation. Support and improvement of recently deployed storage accounting tools. Handling anticipated changes to data-management middleware. Further development of data-popularity service. LHCb computing operations support.
24
WLCG Technical Evolution Groups
Services for HEP WLCG Technical Evolution Groups WLCG Technical Evolution Groups – (TEGs) established in the 2nd half of 2011. Recent work Reassessment of grid infrastructure implementation; based on LHC run experience and evolution of technology. Presentation of TEGs findings presented in February 2012. Future direction Document summarising TEG reports. Prepared by TEG chairs. Presented at WLCG Collaboration Workshop New York (May 2012). Recommendations from TEGs implemented within relatively short timescale (based on LHC shutdown schedule).
25
LSGC User Management Tools
Services for LS LSGC User Management Tools Life Sciences Grid Community User Management Tools User support Technical assistance for end users. Dissemination and knowledge transfer. Community resource monitoring of new resources All workload-management systems and compute element for the biomed VO Storage space monitoring tools Improvements to the Nagios monitoring probes Improving user experience Deployment of redundant VOMS server. Investigate viability of redundant LFC servers. Improvements to storage decommissioning procedures. Future direction Development of a HUC users database and management tools to assist VO administrators in their daily task
26
GReIC Data Access Interface
Services for LS GReIC Data Access Interface GReIC Data Access Interface - a general purpose Grid database service for Life Sciences. Development highlights Exposing the UNIPROT proteins data bank via the GRelC interface. Improvement of the Extraction-Transformation-Loading (ETL) tool. Definition of a new use case on biodiversity. Future direction New use cases will be defined jointly with the Life Sciences community. Existing use cases (e.g. biodiversity at the University of Salento) will be refined and finalized.
27
Hydra Encryption Service
Services for LS Hydra Encryption Service Hydra - an EMI-developed service for handling the encryption and decryption of sensitive files. Development highlights Hydra server: Deployed on gLite 3.1 and gLite 3.2 (beta-release) Improvements to the installation and configuration procedure. Hydra client: Work on Hydra client provision for sites providing resources to Life Sciences Discussion with EMI to integrate the client in the official m/w releases Service deployment: prototype service available for testing Future direction Distributed (3-servers) Hydra key-store deployment Client packages compatible with the latest middleware release
28
Coordination of the A&A Community
Services for A&A Coordination of the A&A Community Aims to provide requirements, use-cases and test-beds for EGI. Scope includes interactivity between heterogeneous e-Infrastructures (e.g. Grid, HCP, Cloud). Support for access and management of Grid-based astronomical databases. Recent work: Coordination of activities within the community concerning the usage of DCIs by both small-scale and large-scale projects has been intensified. Input encouraged during requirements gathering for EGI. A coordinating workshop was organized at the Astronomical Observatory of Paris on November 7th 2011 Major astronomical projects and research areas represented. Plans for the future: Based on conclusions at Paris workshop Community effort should focus on identifying major A&A projects and institutes suited to DCI adoption. The process aimed at the creation of new VRCs is currently in progress with regular checkpoints to verify its progress.
29
Services for A&A VisIVO
Visualization Interface for the Virtual Observatory (VisIVO) - a visualization and analysis software for astrophysical data. Recent work: Fully compliant implementation of the VisIVO MPI utility developed provide compatibility with the gLite grid infrastructure. A hybrid server was procured and deployed as a grid computing node to provide a combined GPU (Graphical Processing Unit) and CPU processing facility. Plans for the future: Ability of VisIVO to utilise GPUs on grid worker nodes. Optimisation of VisIVO visualization component when running on huge user data tables.
30
Services for A&A Grid & HPC
Working towards a stable environment for A & A activities. Recent work: Close coordination of activities between EGI and IGI (Italian NGI) resources. Workflows defined to facilitate smooth operation of complex algorithms running on very large data files. Plans for the future: Identify and define cosmological simulation routines, and validate their output. Deploy small-scale HPC resources at grid sites. Challenges include configuration of systems and integration into the grid middleware infrastructure. Validation of resources based on results from commonly used applications. HPC activity to continue in Italy, driven by demands of interested communities.
31
Services for A&A Access to Databases and interoperability with the VOBs Identification of use-cases and test-beds requiring simultaneous access to astronomical data and to computing resources. Recent work: Work in progress to integrate BaSTI (A Bag of Stellar Tracks and Isochrones) Astronomical Database and its feeding FARANEC code with grid technology. Web portal developed to facilitate grid execution of user-generated stellar evolution simulations. Plans for the future: GReIC is under evaluation for providing a mechanism to access databases from DCIs; a crucial requirement of the A&A community.
32
Services for ES GENESI-DR
A standardized data discovery interface for a federation of repositories. Recent work: A smarter data transfer component capable of discovering and downloading resources. The ability to set configurable parameters on a per-user basis from a dedicated configuration file. Improvements to end-user documentation, including typical use case examples and ‘getting started’ examples. A flexible web GUI was deployed. Plans for the future: Development of a command-line client to provide GENESI-DEC features to jobs running on the EGI grid infrastructure.
33
Introduction to Earth System Grid
Services for ES Introduction to Earth System Grid Distributed infrastructure developed to support CMIP5 The Coupled Model Intercomparison Project, Phase 5 Internationally co-ordinated set of climate model experiments Involving climate model centres from all over the world. Recent work: MPI implementation of ESG application code runs on EGI infrastructure Implementation of multi-threaded climate data transfer program to download data from the ESG nodes Investigation of a solution to streamline ESG data access from EGI infrastructure. Review and testing instance of NorduGrid Security Stack Problem communicated to other EGI representatives with situation description and possible solutions Collaboration between representatives of EGI, SA3.6 and Earth System Grid Federation
34
Introduction to Earth System Grid
Services for ES Introduction to Earth System Grid Recent work II: Improvements of the smart data transfer tool ‘synchro-data’ Contribution to the federated identity Management Workshop Maintenance of loosely coordination in ES VRC with EGU sessions and possible collaboration Plans for the future: ESG interoperability team is developing and testing a scenario based on an application from IPSL which uses CMIP5 data (climate model data stored on the ESG); A prototype modification of MyProxy is currently being developed Will issue ESG certificates based on EGI certificate authentication.
35
Other Communities & Shared Services & Tools
Monitoring & Dashboards Experiment Dashboard – Monitoring distributed computing infrastructure and activities. Development highlights Performance, scalability and functionality improvements driven by LHC computing activity growth. Hardware migration and Oracle upgrade for Dashboard database infrastructure. Common monitoring solutions for LHC communities. Migration to new Service Availability Monitoring architecture. Future direction Development guided by requests from LHC communities.
36
Other Communities & Shared Services & Tools
GReIC Grid database management service providing access and management functionalities related to relational and non-relational databases in a Grid environment. Recent work: The first production releases of the DashboardDB registry and global monitoring for GRelC servers were deployed on the GRelC website as web gadgets (connection with NA2 activity, EGI Gadgets; The porting of the GRelC software on the gLite 3.2 middleware release was completed. RPMS available; Two use cases (UNIPROT, GeneOntology) related to the LS community jointly defined with the bioinformatics group (University of Salento) and implemented; Another use case (Invasive Alien Species) to support a platform for sharing invasive species information across the Italian country in the Italian LifeWatch Virtual Laboratory was defined. Activity on-going. Plans for the future: The support for management, monitoring and control of the GRelC services provided through the DashboardDB will be further extended and improved; Strong dissemination about the DashboardDB registry to register the GRelC service instances deployed in EGI; Additional support to the HUC will be provided to address new user needs and requirements; Website, training events, tutorials, talks and papers will further disseminate the results of this activity.
37
GRelC Monitoring gadget
38
DashboardDB (social) gadget
38 38
39
Other Communities & Shared Services & Tools
SOMA2 A versatile modelling environment for computational drug discovery and molecular modelling. Recent work: Autodock 4 integration work continued and investigations started into how to setup a service for user communities; Program development in upgrading the core UI library components plus associated migration work. Basic Grid support in SOMA2 along with other minor fixes and improvements; New version release of SOMA2 – version 1.4 (Aluminium) was released – available on the web site; CSC maintained and operated the SOMA2 service. Plans for the future: Enable a Grid-enabled application service for interested user communities; Continue to operate and support the existing SOMA2 services; Advertise the Grid enabled SOMA2 service to different user communities; Longer term: expand the scientific applications integrated in to the SOMA2 service and integrate application services from different Grids hosted by different virtual organizations into SOMA2.
40
Other Communities & Shared Services & Tools
Workflow & Schedulers These tools are critical in integrating complex processes, generally involving multiple data sources and different computational resources, as needed within many disciplines. Recent work: New template use cases basing on user requirements, improving the performance of the Kepler actors, and performing tests of the different use cases; Use cases initially prepared for the Fusion community have been customised and reused by other communities; The initial work on this workflow has already started on DKES measuring the transport of particles using an equilibrium previously calculated by VMEC; The first astrophysics workflow has been customised and tested. It controls the production of realistic simulations of the anisotropies of the Cosmic Microwave Background (CMB) radiation; The first workflows from the CompChem area have been arranged and customised. Plans for the future: Dissemination activities have allowed us to show the impact of the work to other communities. The improvements introduced in the actors to improve reliability and scalability, can be of interest for many other communities and can potentially extend the usage of this tool.
41
Other Communities & Shared Services & Tools
MPI The MPI sub-task produces numerous workbenches of increasing complexity with specific high impact on the Computational Chemistry, Earth Sciences, Fusion, and Astronomy & Astrophysics communities. Recent work: TCD and UNIPG started independent work on exploiting GPGPU enabled devices; UNIPG worked on grid implementations of many parallel codes, including CHIMERE, RWAVEPR, ABC, DL_POLY, NAMD and VENU96; CSIC has made continuous improvements to the MPI documentation wiki. CSIC has also tested MPI-START integration with SGE and PBS like job managers with newer versions of MPICH2; TCD produced a proof-of-concept work using MPI-START to support generic parallel workloads. Plans for the future: TCD is investigating proof-of-concept execution of generic parallel codes using MPI-START; The current MPI-Virtual Team will come to an end in May The focused outcome of this groups work should be complete by then, including a better MPI testing infrastructure; TCD will propose a GPGPU virtual team. The aim of this group will be on how best to support GPGPU (and other generic resources) on the EGI grid infrastructure; CSIC continues to lead the MPI accounting sub-task ensuring correct accounting for MPI and parallel jobs. Aims to help produce better detection of PBS/Torque per-process limitations that affect parallel jobs at the EGI sites.
42
Plans for next year (1/3) Build on success in identifying and delivering common solutions Some significant successes seen – beyond expectations! Realisation of sustainability plans which have been demonstrated during project in some cases Collaborative support, less sensitive to changes at individual sites Minor modifications to the DoW proposed: Synchronisation of MS619 with EGI CF 2013 PC Merge of MS620 / D6.9 to reflect end of SA3 in PM 36 (See next slide) D6.8 (sustainability) & D6.9 (annual report) could also be merged
43
Plans for next year (2/3) The SA3 work package ends in April 2013
All of the objectives must therefore be achieved by then Any developments must be completed, fully documented, put into production & their long-term support handed over Even if, in some cases, there is on-going work the plans must take into account this reality
44
Plans for next year (3/3) All disciplines supported by SA3 rely on external funding, typically through national funding agencies (including international organisations such as CERN – funded by member states); The specific support provided by SA3 has ensured a smooth transition to EGI, brought new services to production and increased commonality, reducing long-term support costs; It will be missed but we will (need to) adapt within individual communities to ensure continued support; This adaption – together with completion of existing tasks – will define the work plan for PY3.
45
Review of Objectives 2011 Objective Status
Supporting the tools, services and capabilities required by different HUCs Achieved – work continues in PY2 / PY3 Identifying the tools, services and capabilities currently used by the HUCs that can benefit all user communities and to promote their adoption Several additional items identified and shared across other communities – work will expand and continue in PY2 / PY3 Migrating the tools, services and capabilities that could benefit all user communities into a sustainable support model as part of the core EGI infrastructure Not started – needs further work and discussion First steps: D6.2 (sustainability) & EGI TF w/s Establishing a sustainable support model for the tools, services and capabilities that will remain relevant to single HUCs Collaborative support is the basic model for sustainability that does not depend on individual partners nor specific project funding. A workshop on sustainability at the EGI TF is proposed – work continues and expands in PY2 and PY3. See also D6.2.
46
Review of Objectives 2012 Objective Status
Supporting the tools, services and capabilities required by different HUCs Achieved – work continues in PY3; support must be handed over prior to WP end! Identifying the tools, services and capabilities currently used by the HUCs that can benefit all user communities and to promote their adoption Several additional items identified and shared across communities – work expanded and will continue in PY3. By definition, the work has focused on production use, including enabling new (and old) VOs. Migrating the tools, services and capabilities that could benefit all user communities into a sustainable support model as part of the core EGI infrastructure The long-term support of these tools and even basic middleware looks as if it will depend on the communities that require them. Establishing a sustainable support model for the tools, services and capabilities that will remain relevant to single HUCs Collaborative support is the basic model for sustainability that does not depend on individual partners nor specific project funding. A workshop on sustainability at the EGI TF 2011 was held and is documented in D6.5.
47
Summary Successfully supported major production computing at an unprecedented scale – both quantitatively and qualitatively Successfully delivered common solutions in a variety of areas – with exciting new activities in progress Actively participated in EGI Technical & Community Forum via presentations, tutorials and demos Completed second round of Milestones & Deliverables together with associated technical work Demonstrated sustainability models (e.g. collaborative development) in a number of key areas Actively participated in WLCG Technology Evolution working groups and will be active in implementing the agreed recommendations during remainder of SA3.
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.