DEGREE IST 2006- 034619 Dissemination and Exploitation of GRids in Earth sciencE Earth Science Communit y RequirementsRequirements T echnologies FeedbackFeedback.

Slides:



Advertisements
Similar presentations
LEAD Portal: a TeraGrid Gateway and Application Service Architecture Marcus Christie and Suresh Marru Indiana University LEAD Project (
Advertisements

ESA Data Integration Application Open Grid Services for Earth Observation Luigi Fusco, Pedro Gonçalves.
European Commission Directorate-General Information Society Unit F2 – Grid Technologies INSERT PROJECT ACRONYM HERE BY EDITING THE MASTER SLIDE (VIEW /
EGEE-II INFSO-RI Enabling Grids for E-sciencE The gLite middleware distribution OSG Consortium Meeting Seattle,
Principles of Personalisation of Service Discovery Electronics and Computer Science, University of Southampton myGrid UK e-Science Project Juri Papay,
CNRIS CNRIS 2.0 Challenges for a new generation of Research Information Systems.
1 Software & Grid Middleware for Tier 2 Centers Rob Gardner Indiana University DOE/NSF Review of U.S. ATLAS and CMS Computing Projects Brookhaven National.
Cracow Grid Workshop, November 5-6, 2001 Towards the CrossGrid Architecture Marian Bubak, Marek Garbacz, Maciej Malawski, and Katarzyna Zając.
NextGRID & OGSA Data Architectures: Example Scenarios Stephen Davey, NeSC, UK ISSGC06 Summer School, Ischia, Italy 12 th July 2006.
DataGrid Kimmo Soikkeli Ilkka Sormunen. What is DataGrid? DataGrid is a project that aims to enable access to geographically distributed computing power.
Mike Smorul Saurabh Channan Digital Preservation and Archiving at the Institute for Advanced Computer Studies University of Maryland, College Park.
Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.
Sergey Belov, Tatiana Goloskokova, Vladimir Korenkov, Nikolay Kutovskiy, Danila Oleynik, Artem Petrosyan, Roman Semenov, Alexander Uzhinskiy LIT JINR The.
EU 2nd Year Review – Jan – WP9 WP9 Earth Observation Applications Demonstration Pedro Goncalves :
QCDgrid Technology James Perry, George Beckett, Lorna Smith EPCC, The University Of Edinburgh.
INFSO-RI Enabling Grids for E-sciencE FloodGrid application Ladislav Hluchy, Viet D. Tran Institute of Informatics, SAS Slovakia.
A Lightweight Platform for Integration of Resource Limited Devices into Pervasive Grids Stavros Isaiadis and Vladimir Getov University of Westminster
A Metadata Catalog Service for Data Intensive Applications Presented by Chin-Yi Tsai.
Grid Initiatives for e-Science virtual communities in Europe and Latin America DIRAC TEAM CPPM – CNRS DIRAC Grid Middleware.
1 School of Computer, National University of Defense Technology A Profile on the Grid Data Engine (GridDaEn) Xiao Nong
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks The EGEE Infrastructure and Remote Instruments.
National Earth Science Infrastructure Program AuScope Limited Headquarters School of Earth Sciences University of Melbourne Victoria 3010 Tel
QCDGrid Progress James Perry, Andrew Jackson, Stephen Booth, Lorna Smith EPCC, The University Of Edinburgh.
Jarek Nabrzyski, Ariel Oleksiak Comparison of Grid Middleware in European Grid Projects Jarek Nabrzyski, Ariel Oleksiak Poznań Supercomputing and Networking.
Grid Technologies  Slide text. What is Grid?  The World Wide Web provides seamless access to information that is stored in many millions of different.
The Grid System Design Liu Xiangrui Beijing Institute of Technology.
INFSO-RI Enabling Grids for E-sciencE ES applications in EGEEII – M. Petitdidier –11 February 2008 Earth Science session Wrap up.
INFSO-RI Enabling Grids for E-sciencE Supporting legacy code applications on EGEE VOs by GEMLCA and the P-GRADE portal P. Kacsuk*,
EarthCube Building Block for Integrating Discrete and Continuous Data (DisConBB) David Maidment, University of Texas at Austin (Lead PI) Alva Couch, Tufts.
Cracow Grid Workshop October 2009 Dipl.-Ing. (M.Sc.) Marcus Hilbrich Center for Information Services and High Performance.
NA-MIC National Alliance for Medical Image Computing UCSD: Engineering Core 2 Portal and Grid Infrastructure.
IODE Ocean Data Portal - ODP  The objective of the IODE Ocean Data Portal (ODP) is to facilitate and promote the exchange and dissemination of marine.
GEON2 and OpenEarth Framework (OEF) Bradley Wallet School of Geology and Geophysics, University of Oklahoma
Metadata Mòrag Burgon-Lyon University of Glasgow.
INFSO-RI Enabling Grids for E-sciencE A service oriented framework to create, manage and update metadata for earth system science.
INSERT PROJECT ACRONYM HERE BY EDITING THE MASTER SLIDE (VIEW / MASTER / SLIDE MASTER) Monique Petitdidier IPSL/CETP CEOS/WGISS.
EGEE is a project funded by the European Union under contract IST Presentation of NA4 Generic Applications Roberto Barbera NA4 Generic Applications.
INFSO-RI Enabling Grids for E-sciencE Intelligent Distributed Data Management in Earth System Science S. Kindermann, DKRZ, Germany.
Introduction to Grids By: Fetahi Z. Wuhib [CSD2004-Team19]
©2012 LIESMARS Wuhan University Building Integrated Cyberinfrastructure for GIScience through Geospatial Service Web Jianya Gong, Tong Zhang, Huayi Wu.
6/23/2005 R. GARDNER OSG Baseline Services 1 OSG Baseline Services In my talk I’d like to discuss two questions:  What capabilities are we aiming for.
NeuroLOG ANR-06-TLOG-024 Software technologies for integration of process and data in medical imaging A transitional.
International Symposium on Grid Computing (ISGC-07), Taipei - March 26-29, 2007 Of 16 1 A Novel Grid Resource Broker Cum Meta Scheduler - Asvija B System.
Development of e-Science Application Portal on GAP WeiLong Ueng Academia Sinica Grid Computing
An Overview of Scientific Workflows: Domains & Applications Laboratoire Lorrain de Recherche en Informatique et ses Applications Presented by Khaled Gaaloul.
EO Dataset Preservation Workflow Data Stewardship Interest Group WGISS-37 Meeting Cocoa Beach (Florida-US) - April 14-18, 2014.
EC Review – 01/03/2002 – WP9 – Earth Observation Applications – n° 1 WP9 Earth Observation Applications 1st Annual Review Report to the EU ESA, KNMI, IPSL,
2. WP9 – Earth Observation Applications ESA DataGrid Review Frascati, 10 June Welcome and introduction (15m) 2.WP9 – Earth Observation Applications.
ANALYSIS PHASE OF BUSINESS SYSTEM DEVELOPMENT METHODOLOGY.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI How to integrate portals with the EGI monitoring system Dusan Vudragovic.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Evaluating Metadata access strategies with.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI Monitoring of the LHC Computing Activities Key Results from the Services.
Data Services Task Team WGISS-22 meeting Annapolis, the US, September 12th 2006 Shinobu Kawahito, JAXA/RESTEC.
Tecnologías Grid aplicadas a Ciencias de la Tierra Manuel López-Puertas (IAA, Granada) Page 1 1ª Reuni ó n e-CA, Granada, Junio 2007.
K-WfGrid: Grid Workflows with Knowledge Ladislav Hluchy II SAS, Slovakia.
NTG Spatially Online: Accessing spatial data on the net SSI Seminar Series September 2007.
ACGT Architecture and Grid Infrastructure Juliusz Pukacki ‏ EGEE Conference Budapest, 4 October 2007.
SAM architecture EGEE 07 Service Availability Monitor for the LHC experiments Simone Campana, Alessandro Di Girolamo, Nicolò Magini, Patricia Mendez Lorenzo,
The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) gLite Grid Introduction Salma Saber Electronic.
INFSO-RI Enabling Grids for E-sciencE Earth Science Applications M. Petitdidier (CNRS/IPSL-Paris)
GGF OGSA-WG, Data Use Cases Peter Kunszt Middleware Activity, Data Management Cluster EGEE is a project funded by the European.
Network Requirements Javier Orellana
GSAF Grid Storage Access Framework
Dissemination and Exploitation of GRids in Earth sciencE
GSAF Grid Storage Access Framework
Leigh Grundhoefer Indiana University
Wide Area Workload Management Work Package DATAGRID project
Presentation transcript:

DEGREE IST Dissemination and Exploitation of GRids in Earth sciencE Earth Science Communit y RequirementsRequirements T echnologies FeedbackFeedback ApplicationsApplications DisseminationDissemination ES VOs EGEE & various GRIDS 1 Dissemination and exploitation of Grids in Earth Science Ladislav Hluchy,Monique Petitdidier II SAS, CNRS

DEGREE IST ─ 3 rd EGEE User Forum, Clermont-Ferrand, February Introduction ES community Atmosphere Ocean Biosphere Cryosphere Coupled interdisciplinary processes Sub-surface complex modelling Complex web of sensor Optimal Trajectory First guess Complex data analysis Noisy observations Global, regional, local applications –Alternative use of the data at different time and spatial resolution Large historical distributed archives –Long term data archives to be exploited Near real-time access to data –For processing, value adding and dissemination –For now-casting and alert Models to provide long term trends and forecast –Processing-intensive, data-intensive and complex applications –Data fusion, data assimilation, data mining, modelling … Integration of different data sources

DEGREE IST ─ 3 rd EGEE User Forum, Clermont-Ferrand, February Application examples Production (3 algorithms) and validation of 7 years of GOME Ozone profiles Rapid Earthquake analysis (mechanism and epicenter) CPUs Modeling seawater intrusion in costal aquifer (SWIMED) Geosciences: Geocluster for Academy and industry Flood of the Danube river-Cascade of models Specfem3D: Benchmark for MPI (2 to 2000CPUs) Specfem3D: seismic wave propagation model Benchmark for MPI (2 to 2000CPUs)

DEGREE IST ─ 3 rd EGEE User Forum, Clermont-Ferrand, February DEGREE project DEGREE aims to create a bridge between Earth Science and Grid communities In order to achieve this goal, the consortium members will: –Identify key Earth Science requirements –Disseminate Earth Science application requirements to Grid projects –Evaluate Grid middleware, tools and standards regarding Earth Science requirements –Provide feedbacks to Grid developers

DEGREE IST ─ 3 rd EGEE User Forum, Clermont-Ferrand, February Report on ES family of applications and their Grid requirements (D1.1) Document contains: –Definition of family of applications –Chapters on monitoring requirements –5 requirements categories: Security, Data and metadata management, Workflow, Application development, Miscellaneous –Contains both listed and described requirements (providing context) D1.1 is a living document: –contains previously stated requirements (DataGrid, EGEE) –contains analysis of 15 application scenarios –After delivery to EU, 5 more scenarios added –(re) analyze requirements –Will be used to monitor progress –Base for deliverable D1.2 Final progress report on ES families of applications and their Grid requirements –78 requirements defined Document and application descriptions available on DEGREE site ES requirements

DEGREE IST ─ 3 rd EGEE User Forum, Clermont-Ferrand, February ES key requirements Security –Fine grained on group and role level –enforce data policies Data and metadata management –Database access, both to databases inside and outside Grid –GIS integration Workflow –Complex workflow integration Portals –Web service interfaces to Grid middleware functionality Miscellaneous –Near Real Time: Not only ‘time to completion/delivery’, but also specific start time need –Use of licensed software Requirements might not be ‘unique’ to ES domain, but they are essential for ES

DEGREE IST ─ 3 rd EGEE User Forum, Clermont-Ferrand, February Test suite introduction Goals: Provide Grid developers clear case for our requirements Monitor on requirements fulfillment of Grid m/w (cover requirements) Consists of: Documentation: –Test suite description –Test procedures –Pass/fail criteria ES contact Real application* –Application software –Data needed –Data schemas –Workflow schemas * Application will be provided ‘as it is’ and as policies permit it. No programming done by DEGREE

DEGREE IST ─ 3 rd EGEE User Forum, Clermont-Ferrand, February Test suite Organization document (high level document) –Test suite organization –Test suite descriptions –Test coverage –Templates: Test suite template Test reporting template 8 Test suites, all containing: –Application description –Contact information (for application, data, etc.) –Test environment setup –Test cases –Report template All on the DEGREE website: Document content

DEGREE IST ─ 3 rd EGEE User Forum, Clermont-Ferrand, February ApplicationFamilySpecial Focus GOME level 2 product processing (Ozone profile) SimpleJob control, data management Centroid Seismic Moment Tensor (Seismology) SimpleJob control, data access GRIMI 2 test suite (Envisat/MIPAS level 2 processor) ComplexMPI need GeoCluster (GTC)ComplexJob control, Portal GOME validation (Ozone profile validation) Complex workflowData policy, data access, Portal Space Physics Interactive Data resources (SPIDR) (meteorology) Complex workflowData access Flood Forecasting Simulation Cascade (FFSC) Complex workflowComplex workflow, Portal PUMA (Atmospheric model)Complex workflowWorkflow, data access Test suites available

DEGREE IST ─ 3 rd EGEE User Forum, Clermont-Ferrand, February DB GIS Files DB GIS Work done Survey of the existing data technologies in ES Survey of data usage policies in ES Grid based data management for ES application Test cases Data Management

DEGREE IST ─ 3 rd EGEE User Forum, Clermont-Ferrand, February Data management - Results Survey of the existing data technologies and usage policies in ES –more than 20 ES application analyzed –focus on: Data provision and data flow Data access Data Policies Authorization, Authentication, Accounting –requirements gathered Test suites –data management test suits developed Grid based data management for ES applications –main grid middleware distributions analyzed according to identified requirements: gLite, ARC/NorduGrid, Globus Toolkit, Unicore 5/6, Naregi,GRIA –gaps identification –analysis of middleware services for specific problem areas (e.g. metadata services)

DEGREE IST ─ 3 rd EGEE User Forum, Clermont-Ferrand, February Requirements on data management Intensive use of file resources –large files / large number of files Data access –Databases –GIS systems Strong Data Access Policies Authorization, Authentication, Accounting

DEGREE IST ─ 3 rd EGEE User Forum, Clermont-Ferrand, February Gaps in data management metadata management database access from grid and replication (mirroring) toolkits providing integration of existing functionality as atomic operations QoS mechanisms for data management middleware and grid services for specialized data access protocols and interoperability at data level AMGA – missing support for spatial metadata attributes Some functionality required by ES can be achieved by combining functionality of multiple existing grid toolkits: Replicated data validity/integrity checks as part of replica management operations. File registration + metadata registration as an atomic operation Support for special protocols (e.g. SEED (seismic data)) Support for data format description Data format transformation Interoperability of different ES/environmental risk models Access protocols and transformation services are often ES specific; however well defined generic framework for specialized data access protocols and data transformation services would be advantageous. GRelC – filled gap for database acces from grid, important improvement for ES community Grid middleware for database replication and consistency missing

DEGREE IST ─ 3 rd EGEE User Forum, Clermont-Ferrand, February Job managements Applications Workflow managers Metaschedulers Tools Monitoring gLiteARC Globus Unicore Middleware LCG Storage Computing elements

DEGREE IST ─ 3 rd EGEE User Forum, Clermont-Ferrand, February Work done in Job managements Survey of existing technologies –Middlewares: LCG, gLite, Unicore –Tools: workflow managers, metaschedulers, monitoring tools Identifying most critical requirements of ES –Near-realtime execution –Reliability –Distributed job management Identifying of missing technologies Creating test suites

DEGREE IST ─ 3 rd EGEE User Forum, Clermont-Ferrand, February Gaps in job management Job execution: Near-realtime, reliability Workflow: automatic workflow composition, dynamic workflow Monitoring: expected job start/end time, notification, progress MPI support: Unified and transparent way to submit MPI jobs Licensing management and scheduling Co-scheduling of data gLite supports workflow management through specifications of job dependence in JDL document. Such JDL document must be submitted through a proxy service (WMproxy) which then submits the jobs described in the JDL according to the constraints which are specified for their dependency. What is missing: Real-time workflow modifications User interaction with the workflow management Fine-grained workflow monitoring Visualization of the workflow process EGEE provide support only for short jobs (Short deadline jobs) QoS and fault tolerance are still not guaranteed Partial support in gLite with job perusal Difficult to find correct setting for MPI jobs WMS should have integrated license manager to check license availability

DEGREE IST ─ 3 rd EGEE User Forum, Clermont-Ferrand, February ES portals

DEGREE IST ─ 3 rd EGEE User Forum, Clermont-Ferrand, February ES Portals for Grid, SOA and e-Collaboration Objective –Portal is in the critical path between User and Middleware –As such a key element to increase Grid uptake & exploitation –Where do we stand now and what needs to be done to ensure adequacy for the future on Middleware (& services) side on Portals side Approach to the task –Establish current state-or-art in ES in other e-Science communities –Analyze the present solutions what are ES Portals Requirements are they met ? what can be improved and how

DEGREE IST ─ 3 rd EGEE User Forum, Clermont-Ferrand, February ES Portals for Grid, SOA and e-Collaboration ES Grid Portals Survey –establish current state-of-art baseline –extent of uptake of Grid technologies in ES Portals –methods techniques and middleware used –types of solutions where Grid is used –ES Portals requirements –Deliverable D4.1 (available on the web) Generic Grid Portals Survey –looking outside ES Community –other e-science communities approach to Grid Portals –focus on generic Portals & middleware solutions –analysis of ES Grid Portals requirements vs. middleware gaps –recommendations and inputs for ES Grid Roadmap –Deliverable D4.2 (available on the web)

DEGREE IST ─ 3 rd EGEE User Forum, Clermont-Ferrand, February ES Portals for Grid, SOA and e-Collaboration Results –Wide range of ES Portals scenarios exploiting Grid, SOA, eCollaboration –ES Portals Classification Data Dissemination: "Discover, identify and access ES data" Collaborative: "Online collaboration in ES Virtual Communities" Grid-based: "ES data intensive processing" –ES Major Requirements Integration of heterogeneous distributed services (Grid & Geo- services) Support "Gridifcation" in Geo-services and Spatial Data standards Standard "off-the-shelf" tools for integrated Grid Security and User Management Big emphasis on Metadata and Data, its Discovery and Access...

DEGREE IST ─ 3 rd EGEE User Forum, Clermont-Ferrand, February Gaps in portal functionality Focus must be on ES functionality, grid working as back-end does not have to be visible. Higher level components targeted to ES –Computation submission without descending to grid-job level, maybe even hiding the grid completely Big emphasis on Metadata and Data, its Discovery and Access –Browsing and accessing datasets the ES way Support for Spatial Data: INSPIRE, SDI,... Spatial data searches and OGC services (e.g. WMS) Tools integrated with the Grid Interoperability and interchange –support for standard tools/protocols (ISO19115, OpenDAP, LAS, DODS, NETCDF, integration with OGSA-DAI) –ontology / semantic web (developing the rudiments) Publish, subscribe, notify Search, locate, access and process ES datasets of interest

DEGREE IST ─ 3 rd EGEE User Forum, Clermont-Ferrand, February Gaps in portal functionality Graphical interfaces for different kinds of ES data, activated by data type –Input specification, e.g. area selection on a map for subset selection –Output visualization and browsing: components for displaying time series of images, image layering –Such components exist, but they use different technologies and APIs Standard "off-the-shelf" tools for integrated Grid Security and User Management –Interfacing Grid security and ES security Portal login models : User management integrated with certificate management –Certificates generated on the portal by the portal, transparently –Loging-in into a portal should be enough to authenticate user There are existing activities and software to remedy this, e.g. PURSE (EarthScienceGrid), GAMA,...

DEGREE IST ─ 3 rd EGEE User Forum, Clermont-Ferrand, February Gaps in portal functionality Another approach – integrating grid into existing portals: Light-weight grid service interfaces to grid functionality for easy integration and/or mash-ups creation –Like Google Maps but for grids –Would allow easy integration of grid services into (existing) ES portals

DEGREE IST ─ 3 rd EGEE User Forum, Clermont-Ferrand, February Conclusion ES applications have important requirements The existing technologies have been reviewed and missing features have been identified: –The features are essential for ES applications Expected feedbacks: –Improvement of Grid technologies (both in existing and new Grid projects) to provide missing features –Better support for porting ES applications to Grid –Wider adoption of Grid technologies in ES community