1. Data access and integration with OGSA-DAI: OGSA-DQP Steven Lynden University of Manchester

2. Data access & integration with OGSA-DAI: GGF 17 2 Introduction OGSA-DQP is a service based distributed query processor It evaluates queries over distributed data sources wrapped by OGSA-DAI It is built using OGSA-DAI extensibility points People involved: University of Manchester Tasos Gounaris, Steven Lynden, Alvaro Fernandes, Rizos Sakellariou, Norman Paton University of Newcastle Jim Smith, Arijit Mukherjee, Paul Watson OGSA-DAI Prototype release 3.0 available from the OGSA-DAI website Install on OGSA-DAI WSRF/WS-I 2.1

3. Data access & integration with OGSA-DAI: GGF 17 3 OGSA-DQP high-level overview OGSA-DQP uses a middleware approach. It can be seen as a mediator over OGSA-DAI wrappers. Usability: use it as an OGSA- DAI data service. DQP is capable of planning, scheduling and executing in parallel the distributed queries Calls to analysis (Web) services can be declared within queries and invoked by DQP. DBMS data OGSA-DQP QueryResults OGSA-DAI DBMS data

4. Data access & integration with OGSA-DAI: GGF 17 4 Using OGSA-DQP All interactions are client-server based Firstly, configure OGSA-DQP by specifying the data sources and analysis services to be used (administration) DQP creates a global schema which can then be used to formulate queries The user may then submit queries Infrastructural requirements: - OGSA-DAI-wrapped relational databases - Analysis services (optional) - Evaluation infrastructure

5. Data access & integration with OGSA-DAI: GGF 17 5 OGSA-DQP architecture OGSA-DAI data service perform Evaluator QE Evaluator QE Evaluator QE The “OGSA-DQP service”, Grid Distributed Query Service (GDQS) AKA “Coordinator” AKA Grid Query Evaluation Service (GQES) DQP activities installed

6. Data access & integration with OGSA-DAI: GGF 17 6 OGSA-DQP architecture DQP evaluator services: Are plain Web services Implement the QueryEvaluation port type: evaluate – the input is a query plan partition which is subsequently executed receiveData – allows the evaluator to receive data from other evaluators OGSA-DAI extensions: DQP resource – a resource which encapsulates a distributed query infrastructure: DQP evaluator services, OGSA-DAI data services etc. Implemented as a data resource accessor. OQL query statement activity – enables the submission of a query in Object Query Language (OQL) DQP factory activity – enables the creation and configuration of DQP resources.

7. Data access & integration with OGSA-DAI: GGF 17 7 Example query Given two DBMSs and one analysis tool (i.e., a Web service): goTerm : a GO Gene Ontology table in a remote mySQL DB, exposed by an OGSA-DAI data service protein : a table in a protein sequence DB, exposed by an OGSA-DAI data service Blast (sequence alignment scoring Web service); We want to obtain alignment scores for a sequence against proteins of a certain kind The user submits a single query referencing data stored at multiple sites. The author of the query need not be aware of how/where data is stored. Queries are written in Object Query Language (OQL): select p.proteinId, Blast(p.sequence) from protein p, goTerm t where t.termId = ‘GO:0005942’ and p.proteinId=t.proteinId

8. Data access & integration with OGSA-DAI: GGF 17 8 Background: OQL Why? OGSA-DQP is based on a parallel distributed query processor for object databases (Polar*) The standard query language of object databases is OQL Polar* is still used by DQP to parse, optimise and schedule queries Instead of querying object databases, we are now querying relational databases OQL queries are compiled by Polar* into distributed query plans. During the execution of the query plan, DQP will query relational data sources using SQL.

9. Data access & integration with OGSA-DAI: GGF 17 9 Client interaction with OGSA-DQP Two main client/server interactions: 1. Configuration: the client sends a perform document requesting the service to create a DQP data service resource 2. Query submission: the client sends a perform document requesting the service to execute an Object Query Language (OQL) query, using a DQP data service resource created in (1) The data service resource created in (1) encapsulates the distributed query infrastructure used to execute queries. Differs from the typical OGSA-DAI data service resources e.g. relational data service resource

10. Data access & integration with OGSA-DAI: GGF 17 10 DQP configuration OGSA-DAI data service perform Evaluator URLs OGSA-DAI data service resources Web service URLs OGSA-DAI data service GetRP DQP factory activity OGSA-DAI data service GetRP creates DQP DSR Global schema of imported DBs & analysis services Set of evaluators that can be used Physical DB metadata (used to optimise queries) Result: resource ID of created DSR

11. Data access & integration with OGSA-DAI: GGF 17 11 DQP query evaluation OGSA-DAI data service perform OQL query OGSA-DAI data service perform OQLQueryStatement DQP DSR Evaluator QE transport OGSA-DAI data service perform Analysis service... Evaluator QE Evaluator QE Result: WebRowSet XML Stream

12. Data access & integration with OGSA-DAI: GGF 17 12 Interacting with an OGSA-DQP service Three options: A command line client Allows configuration and query submission via the execution of Apache Ant scripts Client toolkit classes Allow you to integrate OGSA-DQP into your own applications [The above utilities are part of the main OGSA-DQP download] GUI client

13. Data access & integration with OGSA-DAI: GGF 17 13 Command-line client Configuration example: $ ant factory -Ddqp.config.file=config.xml -Durl=http://rpc122.cs.man.ac.uk/axis/services/service1 -Dresource.id=dqp-factory Querying the global schema – example: $ ant getschemas -Durl=http://rpc122.cs.man.ac.uk/axis/services/service1 -Dresource.id=ogsadai-911acvd122

14. Data access & integration with OGSA-DAI: GGF 17 14 Command-line client Query submission example: $ ant query -Durl=http://rpc122.cs.man.ac.uk/axis/services/service1 -Dresource.id=ogsadai-911acvd122 -Dclient.query=“%print select i.id from i in go_goterms;” -Dclient.output.file=results.xml Results will be saved as a WebRowSet, the standard XML representation of relational results used by OGSA-DAI

15. Data access & integration with OGSA-DAI: GGF 17 15 Client toolkit classes Client toolkit classes are provided for the activities contributed by OGSA-DQP: GDQSFactory class used to construct DQPFactory activities OQLQuery class used to construct OQLQueryStatement activities The client toolkit allows the integration of DQP with other applications and seamless interaction with the OGSA-DAI client toolkit OGSA-DQP client toolkit is Java only…

16. Data access & integration with OGSA-DAI: GGF 17 16 Query execution using client toolkit 1 GenericServiceFetcher fetcher = GenericServiceFetcher.getInstance(); 2 DataService service = fetcher.getDataService(url,resourceID); 3 OQLQuery oqlQuery = new OQLQuery(query); 4 OutputStreamActivity outputStream = new OutputStreamActivity(); 5 outputStream.setInput( oqlQuery.getOutput() ); 6 ActivityRequest request = new ActivityRequest(); 7 request.add( oqlQuery ); 8 service.perform(request); 9 oqlQuery.getResultSet(); 10 java.sql.ResultSet rs = outputStream.getResultSet();

17. Data access & integration with OGSA-DAI: GGF 17 17 Demo: The GUI Client The GUI allows you to: Interact with OGSA-DQP services. The GUI is pre-configured with the URL of a OGSA-DQP service we have deployed at EPCC. View the configuration parameters of DQP data service resources View the global schema maintained by a DQP data service resource Submit OQL queries to DQP data service resources View the results of queries View graphical and XML representations of query plans

18. Data access & integration with OGSA-DAI: GGF 17 18 Services @ Newcastle University Evaluator service giga01.ncl.ac.uk OGSA-DAI data service GO Term DB Evaluator service giga02.ncl.ac.uk OGSA-DAI data service Protein interaction DB Evaluator service giga03.ncl.ac.uk OGSA-DAI data service Protein Term DB Evaluator service giga04.ncl.ac.uk OGSA-DAI data service Protein property DB

19. Data access & integration with OGSA-DAI: GGF 17 19 Services @ Newcastle University Evaluator service giga05.ncl.ac.uk OGSA-DAI data service Protein Sequence DB Evaluator service giga06.ncl.ac.uk Evaluator service giga07.ncl.ac.uk Evaluator service giga08.ncl.ac.uk Evaluator service giga09.ncl.ac.uk Entropy analyser service

20. Data access & integration with OGSA-DAI: GGF 17 20 Database tables namelengthsqlTypeName id32varchar type55varchar name255varchar GO Terms extent name: “goterms_goterms” Protein interactions Extent name: “interaction_protein_interactions” namelengthsqlTypeName ORF150varchar ORF250varchar baitProtein50varchar interactionType5varchar repeats11int experimenter100varchar

21. Data access & integration with OGSA-DAI: GGF 17 21 Database tables Protein terms extent name: “protein_term_protein_goterm” namelengthsqlTypeName ORF55varchar molecularWeight12float hydrophobicity12float Protein properties extent name: “protein_property_protein_propertys” namelengthsqlTypeName ORF50varchar sequence65535text Protein sequence extent name: “protein_sequence_protei n_sequences” namelengthsqlTypeName ORF55varchar GOTermIdenti fier 32varchar

22. Data access & integration with OGSA-DAI: GGF 17 22 DQP service @ EPCC OGSA-DAI data service DQP factory GIGA resource test.ogsadai.org.uk Encapsulates the distributed query environment deployed at Newcastle ogsadai-1092f60c1e1

23. Data access & integration with OGSA-DAI: GGF 17 23 Conclusion OGSA-DQP is a service based distributed query processor that is: Exposed as a service Implemented as an orchestration of services It provides an example of how the OGSA-DAI extensibility points can be used… The activity extensibility points are used New data resource accessors are implemented Dynamic resource deployment is used during configuration to create new resources Benefits: OGSA-DAI manages activity concurrency – we didn’t need to write concurrent code OGSA-DQP can take advantage of the host of delivery options provided by OGSA-DAI OGSA-DQP is insulated from multiple platforms (WS-I, WSRF) by OGSA-DAI