Slides thanks to Steve Lynden Amy Krause EPCC Distributed Query Processing with OGSA-DQP Principles and Architectures for Structured Data Integration:

Slides:



Advertisements
Similar presentations
Tom Sugden EPCC OGSA-DAI Future Directions OGSA-DAI User's Forum GridWorld 2006, Washington DC 14 September 2006.
Advertisements

21 Sep 2005LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson.
OGSA-DQP - A Service-Based Distributed Query Processor for The Grid Arijit Mukherjee University of Newcastle Arijit Mukherjee University.
PHP I.
Dynasoar Dynamic Deployment of Web Services on a Grid or the Internet or Why its good to be Jobless Paul Watson School of Computing Science.
E-Science Data Information and Knowledge Transformation Eldas Building Service Grids with Enterprise Level Data Access Services Alan Gray
Service-Based Distributed Query Processing on the Grid M.Nedim Alpdemir Department of Computer Science University of Manchester.
Grid Database Projects Paul Watson, Newcastle Norman Paton, Manchester.
Eldas 1.0 Enterprise Level Data Access Services Design Issues, Implementation and Future Development Davy Virdee.
M.Nedim Alpdemir, Anastasios Gounaris¹, Arijit Mukherjee², Desmond Fitzgerald, Norman W. Paton¹, Paul Watson², Rizos Sakellariou¹, Alvaro A.A. Fernandes¹,
16-17 October 2003 Grids and Applied Language Theory: Declarative Grid Service Orchestration with OGSA-DQP (A A A Fernandes) 1 Declarative Grid Service.
Data Access & Integration in the ISPIDER Proteomics Grid N. Martin – A. Poulovassilis – L. Zamboulis
Data Access & Integration in the ISPIDER Proteomics Grid L. Zamboulis, H. Fan, K. Bellhajjame, J. Siepen, A. Jones, N. Martin, A. Poulovassilis, S. Hubbard,
Institute for Software Science – University of ViennaP.Brezany 1 Databases and the Grid Peter Brezany Institute für Scientific Computing University of.
Amy Krause Applications Consultant, EPCC Tom Sugden Applications Consultant, EPCC OGSA-DAI Client Toolkit Principles.
Amy Krause Applications Consultant, EPCC Tom Sugden Applications Consultant, EPCC OGSA-DAI Usage Scenarios Behind.
NextGRID & OGSA Data Architectures: Example Scenarios Stephen Davey, NeSC, UK ISSGC06 Summer School, Ischia, Italy 12 th July 2006.
1 An Introduction to OGSA-DAI Konstantinos Karasavvas 13 th September 2005.
Mike Jackson EPCC OGSA-DAI Today Release 2.2 Principles and Architectures for Structured Data Integration: OGSA-DAI.
Amy Krause Applications Consultant, EPCC Extending OGSA-DAI Principles and Architectures for Structured Data Integration: OGSA-DAI.
17 July 2006ISSGC06, Ischia, Italy1 Agenda Session 26 – 14:30-16:00 An Overview of OGSA-DAI OGSA-DAI today – and future features How to extend OGSA-DAI.
PARALLEL DBMS VS MAP REDUCE “MapReduce and parallel DBMSs: friends or foes?” Stonebraker, Daniel Abadi, David J Dewitt et al.
14-18 March 2004 EDBT'04 : Service-Based Distributed Query Processing for the Grid (M N Alpdemir) 1 Title, places, people, funding, projects Manchester.
Database Taskforce and the OGSA-DAI Project Norman Paton University of Manchester.
Data Management Kelly Clynes Caitlin Minteer. Agenda Globus Toolkit Basic Data Management Systems Overview of Data Management Data Movement Grid FTP Reliable.
GRAM: Software Provider Forum Stuart Martin Computational Institute, University of Chicago & Argonne National Lab TeraGrid 2007 Madison, WI.
DAIS Grid1 Database Access and Integration Services on the Grid * * Authors: N. Paton, M. Atkinson, V.
1 EPCC Sun Data and Compute Grids Project Using Sun Grid Engine and Globus to Schedule Jobs Across a Combination of Local.
Proteome data integration characteristics and challenges K. Belhajjame 1, R. Cote 4, S.M. Embury 1, H. Fan 2, C. Goble 1, H. Hermjakob, S.J. Hubbard 1,
INFSO-RI Module 01 ETICS Overview Alberto Di Meglio.
The ACGT Workflow Editing & Enactment Environment Giorgos Zacharioudakis Institute of Computer Science, Foundation for Research & Technology – Hellas (ICS-FORTH)
INFSO-RI Module 01 ETICS Overview Etics Online Tutorial Marian ŻUREK Baltic Grid II Summer School Vilnius, 2-3 July 2009.
OGSA-DAI Scenarios and Requirements OGSA-DAI for Developers GridWorld 2006, Washington DC 11 September 2006.
INFSO-RI Enabling Grids for E-sciencE Supporting legacy code applications on EGEE VOs by GEMLCA and the P-GRADE portal P. Kacsuk*,
FlexElink Winter presentation 26 February 2002 Flexible linking (and formatting) management software Hector Sanchez Universitat Jaume I Ing. Informatica.
1 1 EPCC 2 Curtin Business School & Edinburgh University Management School Michael J. Jackson 1 Ashley D. Lloyd 2 Terence M. Sloan 1 Enabling Access to.
Grid Execution Management for Legacy Code Applications Grid Enabling Legacy Code Applications Tamas Kiss Centre for Parallel.
Grids - the near future Mark Hayes NIEeS Summer School 2003.
Data access and integration with OGSA-DAI: OGSA-DQP Steven Lynden University of Manchester.
A Dynamic Service Deployment Infrastructure for Grid Computing or Why it’s good to be Jobless Paul Watson School of Computing Science.
INFSO-RI Enabling Grids for E-sciencE OGSA DAI Data Access and Integration Marek Ciglan Institute of Informatics, Slovac Academy.
Mike Jackson EPCC OGSA-DAI Architecture + Extensibility OGSA-DAI Tutorial GGF17, Tokyo.
INTRODUCTION TO DBS Database: a collection of data describing the activities of one or more related organizations DBMS: software designed to assist in.
Metadata Mòrag Burgon-Lyon University of Glasgow.
Grid Execution Management for Legacy Code Applications Grid Enabling Legacy Applications.
Amy Krause EPCC OGSA-DAI An Overview OGSA-DAI Technology Update GGF17, Tokyo (Japan)
OGSA-DQP:Service-Based Distributed Query Processing on the Grid M.Nedim Alpdemir Department of Computer Science University of Manchester.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE User Forum, Manchester, 10 May ‘07 Nicola Venuti
1 OGSA-DAI Status Report Neil P Chue Hong 20 th May 2005.
Chapter 1 Introduction Yonsei University 1 st Semester, 2014 Sanghyun Park.
Neil Chue Hong Project Manager, EPCC OGSA-DAI Requirements Gathering Exercise 2 nd DIALOGUE workshop eSI, 9-10.
Mapping the Data Warehouse to a Multiprocessor Architecture
Secure middleware patterns E.B.Fernandez. Middleware security Architectures have been studied and several patterns exist Security aspects have not been.
OGSA-DAI Open Grid Services Architecture – Data Access and Integration NeSC Review 18 March 2004.
NeOn Components for Ontology Sharing and Reuse Mathieu d’Aquin (and the NeOn Consortium) KMi, the Open Univeristy, UK
INFSO-RI Enabling Grids for E-sciencE Using of GANGA interface for Athena applications A. Zalite / PNPI.
Copyright 2007, Information Builders. Slide 1 iWay Web Services and WebFOCUS Consumption Michael Florkowski Information Builders.
Compilation of XSLT into Dataflow Graphs for Web Service Composition Peter Kelly Paul Coddington Andrew Wendelborn.
Reliable Web Service Execution and Deployment in Dynamic Environments * Markus Keidl, Stefan Seltzsam, and Alfons Kemper Universität Passau Passau,
OGSA-DQP Steven Lynden University of Manchester. Data access & integration with OGSA-DAI: GGF 17 2 Introduction OGSA-DQP is a service based distributed.
OGSA-DAI 简介及其它在 China-VO DAS 系统中的应用 杨阳 中国虚拟天文台研发团队 Chinese Virtual Observatory.
Grid Execution Management for Legacy Code Architecture Exposing legacy applications as Grid services: the GEMLCA approach Centre.
OGSA-DAI.
A Grid Data Integration Service (OGSA-DQP) Paul Watson, University of Newcastle-upon-Tyne based on the work of… Norman Paton, Tasos Gounaris,
Amy Krause EPCC OGSA-DAI An Overview OGSA-DAI on OMII 2.0 OMII The Open Middleware Infrastructure Institute NeSC,
System Software Laboratory Databases and the Grid by Paul Watson University of Newcastle Grid Computing: Making the Global Infrastructure a Reality June.
Open Source distributed document DB for an enterprise
Mapping the Data Warehouse to a Multiprocessor Architecture
Lecture 1: Multi-tier Architecture Overview
Presentation transcript:

Slides thanks to Steve Lynden Amy Krause EPCC Distributed Query Processing with OGSA-DQP Principles and Architectures for Structured Data Integration: OGSA-DAI as an example ISSGC06 (Ischia, Italy) 17 July 2006

ISSGC06, Ischia, Italy2 Introduction OGSA-DQP is a service based distributed query processor It evaluates queries over distributed data sources wrapped by OGSA- DAI It is built using OGSA-DAI extensibility points People involved: –University of Manchester –Steven Lynden, Alvaro Fernandes, Rizos Sakellariou, Norman Paton –University of Newcastle –Jim Smith, Arijit Mukherjee, Paul Watson –OGSA-DAI Prototype release 3.0 available from the OGSA-DAI website Release 3.1 will available soon

17 July 2006ISSGC06, Ischia, Italy3 OGSA-DQP mediator approach OGSA-DQP uses a middleware approach. It can be seen as a mediator over OGSA-DAI wrappers. Effectiveness: “leave to it to orchestrate your services”; Usability: “use it as an OGSA-DAI data service”. DBMS data OGSA-DQP QueryResults OGSA-DAI DBMS data

17 July 2006ISSGC06, Ischia, Italy4 OGSA-DQP parallelism OGSA-DQP queries can be evaluated across multiple nodes by DQP services deployed on those nodes Operators can be parallelised, e.g. a join can be executed across two nodes OGSA-DQP compiles, optimises and schedules queries for execution across available nodes An OGSA-DQP query is separated into a number of partitions, each of which encapsulates an individual service’s role in the query evaluation DBMS data OGSA-DAI DQP scan (A) DBMS data OGSA-DAI DQP scan (B) join (A 1,B 1 ) DQP join (A 2,B 2 ) DQP reduce node 1node 2 node 3 node 4 node 5

17 July 2006ISSGC06, Ischia, Italy5 DQP example Given two DBMSs and one analysis tool (i.e., a Web service): –goTerm : a GO Gene Ontology table within a MySQL DB, exposed by an OGSA- DAI data service –protein : a protein sequence table within a MySQL DB, exposed by an OGSA- DAI data service –Blast (sequence alignment scoring Web service); We want to obtain alignment scores for a sequence against proteins of a certain kind The user submits a single query referencing data stored at multiple sites. The author of the query need not be aware of how/where data is stored. Queries are written in Object Query Language (OQL): select p.proteinId, Blast(p.sequence) from protein p, goTerm t where t.termId = ‘GO: ’ and p.proteinId=t.proteinId

17 July 2006ISSGC06, Ischia, Italy6 OGSA-DQP architecture DQP evaluator services: –Are plain Web services –Implement the QueryEvaluation port type: –evaluate – the input is a query plan partition which is subsequently executed –receiveData – allows the evaluator to receive data from other evaluators OGSA-DAI extensions: –DQP resource – a resource which encapsulates a distributed query infrastructure: DQP evaluator services, OGSA-DAI data services etc. Implemented as a data resource accessor. –OQL query statement activity – enables the submission of a query in Object Query Language (OQL) –DQP factory activity – enables the creation and configuration of DQP resources.

17 July 2006ISSGC06, Ischia, Italy7 DQP query evaluation OGSA-DAI data service perform OQL query OGSA-DAI data service perform OQLQueryStatement DQP DSR Evaluator QE transport OGSA-DAI data service perform Analysis service... Evaluator QE Evaluator QE Result: WebRowSet XML Stream

17 July 2006ISSGC06, Ischia, Italy8 Conclusion OGSA-DQP is a service based distributed query processor that is: –Exposed as a service –Implemented as an orchestration of services It provides an example of how the OGSA-DAI extensibility points can be used… –The activity extensibility points are used –New data resource accessors are implemented –Dynamic resource deployment is used during configuration to create new resources Benefits: –OGSA-DAI manages activity concurrency – we didn’t need to write concurrent code –OGSA-DQP can take advantage of the host of delivery options provided by OGSA-DAI –OGSA-DQP is insulated from multiple platforms (WS-I, WSRF) by OGSA- DAI