OGSA-DAI Usage Scenarios and Behaviour: Determining good practice Mario Antonioletti EPCC, University of Edinburgh AHM 2004, Data Services and Middleware, 2 nd September 2004

2http:// - AHM2004, 2nd September 2004 OGSA-DAI  OGSA-DAI is middleware –Middleware should be invisible –Provide additional functionality or hide complexity  Allows uniform access to data resources –data resources: relational and XML databases, files, …  Provides an extensible framework –You can extend functionality - fill any gaps  We think it works well –But need feedback  Recount how OGSA-DAI is being used  Some background first …

3http:// - AHM2004, 2nd September 2004 Basic Operational Model Data Resource Container DAISGR Client GDSF GDS

4http:// - AHM2004, 2nd September 2004 Why OGSA-DAI?  Why use OGSA-DAI over JDBC? –Language independence at the client end Do not need to use Java –Platform independence Do not have to worry about connection technology, drivers, etc –Can handle XML resources –Can embed additional functionality at the service end Transformations Third party delivery etc Avoiding unnecessary data movement –Provision of Metadata is powerful –Usefulness of the Registry for service discovery Dynamic service binding process

5http:// - AHM2004, 2nd September 2004 More Complex Behaviour Data Resource Container Client GDS GDT Data Resource Container GDS GDT Deliver data back to the client. Data Resource Deliver data to a third party. Deliver data another GDS. And there's a lot more that you can do …

6http:// - AHM2004, 2nd September 2004 Usage Patterns G A Q S+R Data Q - Query D - Delivery S - Status R - Result U - Update I - Data id Q+D A C G S R G C A Q S D R A G Q+U S RetrieveUpdate/InsertPipeline G2=C G1=P A I Q1 S2 S1 U/R Q2+D Q1+D G2=C A G1=P S2 S1 Q2 U/R Actors - OGSI process - Non-OGSI process A - Analyst C - Consumer G - GDS P - Producer Call Response Data Flow A P G U I Q S A P G U I S Q+D

7http:// - AHM2004, 2nd September 2004 Activities are the drivers  Express a task to be performed by a GDS  Three broad classes of activities: –Statement –Transformations –Delivery  Extensible: –Easy to add new functionality –Does not require modification to the service interface –Extension operate within the OGSA-DAI framework  Functionality: –Implemented at the service –Work where the data is (do not require to move data back)

8http:// - AHM2004, 2nd September 2004 OGSA-DAI Deck

9http:// - AHM2004, 2nd September 2004 Building Applications  Activities are grouped together –Perform document –Data can flow between activities  Optimisation –Avoids multiple message exchanges  Can deliver to other GDSs –Prerequisite for data integration  Base middleware for projects requiring data access –Some capability for data integration  That is the theory … now for the practice  OGSA-DAI being adopted by a number of projects …

10http:// - AHM2004, 2nd September 2004 Who is Using OGSA-DAI? OGSA-DAI ( AstroGrid ( BioSimGrid ( BioGrid ( Bridges ( eDiaMoND ( FirstDig ( GeneGrid ( GEON ( IU RGRBench ( myGrid ( N2Grid ( ODD-Genes ( OGSA-WebDB ( MCS ( INWA ( GridMiner (

11http:// - AHM2004, 2nd September 2004 Project classification

12http:// - AHM2004, 2nd September 2004 Projects using OGSA-DAI  These projects form a list of case studies –Need to capture requirements –How OGSA-DAI is being used –Where it succeeds and where it fails –Other issues that arise –An on-going process  Only time to outline salient points from a couple of projects –More detail in the paper … but this only gives a top level overview On-going process …  Solicit more –If you have more then please get in touch …

13http:// - AHM2004, 2nd September 2004  e-Digital MammOgraphy National Database  Built a prototype of a national database of mammographic images in support of the UK Breast screening programme  Employ Grid technologies to facilitate this process  Mike Brady gave a keynote that went over the details

14http:// - AHM2004, 2nd September 2004 DB2 Content Manager DB2 Content Manager DB2 Content Manager DB2 Content Manager DB2 Federation OGSA-DAI Database Files OGSA-DAI Core Services Core Services Core Services Core Services Data Load Training App Training Services UCL KCLUEDCHU Core API Training API Training Application Core & Training API OGSA-DAI Data Load Training App Core & Training API Data Load Training App Core & Training API Data Load Training App Core & Training API

15http:// - AHM2004, 2nd September 2004  eDiaMoND Findings: –OGSA-DAI provides a flexible framework –Dynamically configure the system through discovery –Activities can operate with different levels of granularity –Federation can introduced at various levels –Upgrading from R3  R4 broke some things Low level XML issues –Good documentation on how to extend the framework Extended Activities to access IBM DB2 Content Manager

16http:// - AHM2004, 2nd September 2004 INWA Objectives  Innovation Node Western Australia –Informing Business & Regional Policy: Grid-enabled fusion of global data and local knowledge  Project –Run from Nov Aug 2004 –Involved 10 partners (6 UK + 4 Australia)  Aim –Data mine commercially sensitive data –Security an absolute MUST –Employ Grid technologies –Need access to data and computational resources  Demonstrator using: –OGSA-DAI Incorporate data resources –Sun DCG's TOG (Transfer-queue Over Globus) Handle job submission to analyse micro array data

17http:// - AHM2004, 2nd September 2004 Curtin,Australia EPCC,UK INWA Grid Engine BankTelco Grid Engine BankTelco OGSA-DAI TOG Data Browser Telco data Bank data Australian property UK Property

18http:// - AHM2004, 2nd September 2004 INWA: Lessons Learned  Performing Data Integration: –TimeZone date problems  Security issues: –Bugs in JavaCoG in GT3 OGSA-DAI could not switch security for Grid data transfers TOG had no security option –All of these have been fixed  Middleware not mature enough for commercial deployment

19http:// - AHM2004, 2nd September 2004 ODD-Genes  OGSA-DAI Demo for Genetics  Collaboration between –EPCC –Scottish Centre for Genomic Technology and Informatics (GTI) –Human Genetics Unit (HGU)  ODD-Genes demonstrates: –Perform high-speed batch analysis of microarray data on the Grid –Browse the results of previous analyses stored in a database –View data from arbitrary databases as HTML –Discover related databases on the Grid –Perform coupled queries on newly- discovered databases to provide a richer analysis of gene data

20http:// - AHM2004, 2nd September 2004 ODD-Genes Actors HGU Mouse Genome Information OGSA-DAI GTI ODD-Genes Webapp OGSA-DAI Micro Array Data TOG GridEngine EPCC GridEngine Globus OGSA-DAI DAISGR 1. Client 2. EPCC is an example of a computational resource. 3. HGU is an example of a data repository.

21http:// - AHM2004, 2nd September 2004 ODD-Genes Findings  Data discovery perceived to be very important –Map data views: time -> spatial locations –Discovery of new resources  Transparency to data access had an XML database had a relational database –Deploy OGSA-DAI and not worry about databases  Issues –Registry maintenance policy –Semantics of the discovery process –Groups working the same area but different schemas, no generic metadata (schemas were the effective metadata)  Provides an additional tool for researchers

22http:// - AHM2004, 2nd September 2004 Other Projects  AstroGrid –Identified (and fixed) a number of bugs –Passed on requirements  FirstDig –Identified a number of bugs –Have contributed a data browser to OGSA-DAI  GeneGrid –Interfacing Perl through an OGSA-DAI service to access biological databases –Requirement for file support  EdSkyQuery-G –Collaboration between OGSA-DAI & Eldas –Based on SkyQuery project by John Hopkins University, Baltimore, USA

23http:// - AHM2004, 2nd September 2004 More on Projects  MSc at Edinburgh looking at data integration scenarios –Benchmarking OGSA-DAI –Investigating capabilities GDSActivity allowing perform documents to be executed at other GDSs Identifying further requirements for data integration - control flow –sequence –flow Question as to whether such capabilities should be included in OGSA-DAI or OGSA-DAI should interface with other workflow languages  MSc at Edinburgh looking at C bindings to the OGSA-DAI CTK –For language independance need to provide more of these… Perl, Python … Eiffel!!  GridMiner –Have a really cute logo –Have a member of that team currently at NeSC

24http:// - AHM2004, 2nd September 2004 Conclusions  Still early days –Standardisation process not stabilising quickly enough –Infrastructure still developing and prone to change  OGSA-DAI acting as an enabler –Showing people what can be done –However is it cracking a nut with a sledge hammer?  Usage patterns are similar –Call for people to work together to solve similar problems  Problems that are not OGSA-DAI specific –Metadata, Time zones, security, …  Data discovery perceived to be important –Is this in the scope of what OGSA-DAI should be doing?  Need to talk to users and gather war stories – –On going process …

25http:// - AHM2004, 2nd September 2004 Further Information  The OGSA-DAI Project Site: –  The DAIS-WG site: –  OGSA-DAI Users Mailing list –General discussion on grid DAI matters  Formal support for OGSA-DAI releases –  OGSA-DAI training courses