Grids, Grid Data Services and OGSA-DAI Mike Mineter NeSC-TOE

EU project: RIO31844-OMII-EUROPE 2 Acknowledgement Many slides from OGSA-DAI team. (Some slides from me.)

EU project: RIO31844-OMII-EUROPE 3 Contents What is a Grid? What is a Grid Data Service? Why the “OGSA-DAI” acronym?! Why does OGSA-DAI matter?! When should we use OGSA-DAI?

EU project: RIO31844-OMII-EUROPE 4 What is a Grid? - 1 This is

EU project: RIO31844-OMII-EUROPE 5 What is a Grid? ComputersDataPeople A Grid is all about the sharing of Resources

EU project: RIO31844-OMII-EUROPE 6 A Grid is.. … all about the sharing of Resources –Within and between virtual organisations (= collaborations) Resources accessed by abstractions –User wants a job to run, wants to access data,… »Rarely cares where this happens … a set of resources (and enabling services) that share mechanisms for –Authentication: communicate identity of user/provider X.509 certificate commonly used in “production grids” –Authorisation: what can this user be allowed to do Member of which VO, which group,… –Underpinned by agreement across VOs and resource providers … infrastructure that builds on the Internet to permit orchestration of services across administrative domains

EU project: RIO31844-OMII-EUROPE 7 Web services – software components that are… Accessible across a network Loosely coupled, defined by the messages they receive / send Service description that can be used to create client software Based on standards (for which tools do / could exist) Developed in anticipation of new uses Service Client

EU project: RIO31844-OMII-EUROPE 8 Using service B from service A Web services container service A: consumes B TCP/IP Messages O/S Web services container service B O/S A B messages

EU project: RIO31844-OMII-EUROPE 9 Using service B from service A Web services container service A: consumes B TCP/IP Messages O/S Web services container service B O/S A B messages These messages define service B

EU project: RIO31844-OMII-EUROPE 10 Using service B from service A Web services container service A: consumes B TCP/IP Messages O/S Web services container service B O/S A B messages http, https Commonly used for WS - original purpose: carry HTML HTTP request methods GET : specify a URL POST: key-value pairs Defaults to port 80 HTTPS: encrypts / decrypts

EU project: RIO31844-OMII-EUROPE 11 Messages in XML Using service B from service A Web services container service A: consumes B TCP/IP O/S Web services container service B O/S A B messages http, https XML eXtendable Markup Language Human readable (sort of) – so not tied to one architecture Extendable: can define new data types, belong to “namespaces”, expressed by schemas

EU project: RIO31844-OMII-EUROPE 12 Messages in XML WSDL Web services container service A: consumes B TCP/IP O/S Web services container service B O/S A B messages http, https Service B is described by WSDL, “Web Service Description Language”. Includes: Where: “ end point” URL where B receives messages What: Message definitions How: “Bindings” to use service

EU project: RIO31844-OMII-EUROPE 13 Messages in XML WSDL Web services container service A: consumes B TCP/IP O/S Web services container service B O/S A B messages http, https Service B is described by WSDL, “Web Service Description Language”. Includes: Where: “ end point” URL where B receives messages What: Message definitions How: “Bindings” to use service API’s for B API’s are derived from the WSDL by tools Developer adds code for specific client NO API LIBRARY SUPPLIED

EU project: RIO31844-OMII-EUROPE 14 SOAP Web services container service A: consumes B TCP/IP O/S Web services container service B O/S A B messages http, https API’s for B XML-based messaging protocol Header: extendable for security, accounting,.. Body: message content

EU project: RIO31844-OMII-EUROPE 15 SOAP Code languages Web services container service A: consumes B TCP/IP O/S Web services container service B O/S A B messages http, https API’s for B This determines whether code is in Java, C, Perl, Python…

Enabling Grids for E-sciencE INFSO-RI HTTP message (JAVA) Web Services A B.Op (a,b) invoke c return B Insert the web into the invoke and into the return Consumes B (client) Stack Sk eleton Provides B (server) Ret(c) Soap envelope A B- stub Stack B Bs.Op (a,b) invoke … … B.Op (a,b) invoke … c return c … Web Soap envelope HTTP message Op(a,b) WSDL for B Soap envelope Java program with 2 classes

Enabling Grids for E-sciencE INFSO-RI HTTP message (JAVA) Web Services A B.Op (a,b) invoke c return B Insert the web into the invoke and into the return Consumes B (client) Stack Sk eleton Provides B (server) Ret(c) Soap envelope A B- stub Stack B Bs.Op (a,b) invoke … … B.Op (a,b) invoke … c return c … Web Soap envelope HTTP message Op(a,b) WSDL for B Soap envelope Java program with 2 classes

18 Real Web Service Invocation Borja Sotomayor, Discover Describe Invoke

Enabling Grids for E-sciencE INFSO-RI WS-I core of Web Services WS-I (Interoperability) delivers practical guidance, best practices and resources for developing interoperable Web services solutions. Open standards: SOAP: protocol for message passing Web Service Description Language: to describe services UDDI: Universal Description, Discovery and Integration WS-Security: incorporates security

EU project: RIO31844-OMII-EUROPE 20 Web Services Grid Technology Grid Services Commerce- driven Standards Tools Research driven Data-intensive Compute intensive Collaboration – sharing of resources October 2001 View Open Grid Services Architecture

EU project: RIO31844-OMII-EUROPE View WSRF – Web Services Resource Framework WS-Notification Web Services –Basis for defining standards for different services –For services on grids: Need to manage state – interact with resources Need to be notified of change of state

© 2004 IBM Corporation © 2004 University of Chicago 22 context Interface Web Service message id message Using a Web service to access a WS-Resource id address resource Endpoint Reference The WS-Resource framework model Run-time environment

© 2004 IBM Corporation © 2004 University of Chicago 23  WS-Resource Properties Resource state and metadata “Projected” as an XML document Query and Set operations  WS-Resource LifeTime Explicit destruction or “Soft state” time-to-live Provides for cleanup of resource instances resource 5A34C1DE03 Power The WS-Resource framework model

Enabling Grids for E-sciencE INFSO-RI Acknowledgements: some slides are from

EU project: RIO31844-OMII-EUROPE 25 Among middleware that uses WSRF/ WS-N: –Globus Toolkit 4 –UNICORE

26 Custom Web Services WS-Addressing, WSRF, WS-Notification Custom WSRF Web Services GT4 WSRF Web Services WSDL, SOAP, WS-Security User Applications Registry Administration GT4 Container Globus Toolkit 4 Web Services Core Thanks to J. Schopf, ANL

EU project: RIO31844-OMII-EUROPE 27 Story so far…. Grids orchestrate resources They do this with many services that (increasingly often) are exposed as web-services “Grid services” have to manage state, notification: standards for this are WS-Resource Framework and WS-Notification - e.g. Globus Toolkit 4 Where do “grid data services” fit in?

EU project: RIO31844-OMII-EUROPE 28 What is a Grid? ComputersDataPeople A Grid is all about the sharing of Resources

EU project: RIO31844-OMII-EUROPE 29 Focus on Data Data OGSA-DAI enables the sharing of Data Resources

EU project: RIO31844-OMII-EUROPE 30 Types of data services Many user communities manage data in grid vaults (aka storage elements) –Experimental data …. –Replicated for resilience –And to be close to where computation will happen Many new user communities have more diverse data resources To facilitate new research need data to be accessible from Grid infrastructures Resources: –May pre-date Grids –Providers may have current ways to distribute data to users –May not be able to replicate data –Need AuthN and AuthZ

Web: Motivation Grid is about sharing resources OGSA-DAI is about sharing structured data resources

Web: Life before OGSA-DAI…. A few examples follow of alternative approaches to sharing data.

Web: Sharing data via web site download ZIP up data and put it on a web site Pros o Easy distribution for providers o Easy access for consumers Cons o Consumers have to download all the data o Consumers have to load data into local databases to use it o Static snapshot o Security

Web: Sharing data via direct access Providers tell consumers o Database URL – mycomputer.epcc.ed.ac.uk:3306 o Username – userID o Password – password Pros o Consumers have direct access Cons o Firewall issues o User and password management is hard o No consistent security model o Hard to use in grid/web service workflows

Web: Sharing data via direct access Cons (continued) o No server-side layer in which to standardize database heterogeneities o Myriad drivers o Different APIs across different data types Relational and JDBC XML and XMLDB Indexed files and Lucene

Web: Manipulate data using domain-specific operations, e.g. o Book findByISBN(ISBN) o List findByAuthor(Author) o List findByKeyword(Word) Pros o Fits with grid/web service approach o Abstraction hides back-end database details o Web services are programming language neutral o Operations likely to map well to authorization policies Domain-specific web services

Web: Cons o Slower than direct access Web service layer SOAP transport overhead – especially for large result sets o Domain-specific API prevents use of generic data exploration, mining and manipulation tools Domain-specific web services

Web: OGSA-DAI generic web services Manipulate data using OGSA-DAI’s generic web services

EU project: RIO31844-OMII-EUROPE 39 Importance of workflows Access service 3 activities in the workflow OGSA-DAI server is close to data

EU project: RIO31844-OMII-EUROPE 40 Usage Scenarios © Data Source Client OGSA-DAI Control message Data message Data Source Client OGSA-DAI FTP Server on Client Data Source 1 OGSA-DAI Data Source 2Data Source n Client

EU project: RIO31844-OMII-EUROPE 41 OGSA-DAI 3.0 Persistence and Configuration OMII ActivitiesData Resources GTAxisUNICOREgLiteWS-DAI?Embedded OGSA-DAI Core Resource management Activity management Workflow engine

EU project: RIO31844-OMII-EUROPE 42 Typical roles Researcher –Wants to use data from context of known application, easy portal, workflow.. Data publisher –Deploys OGSA-DAI server –Determines AuthN and AuthZ policies for their data –Establishes activities (= workflow components) Informatician / Application developer –Deploys client software –Uses Java to build workflow –Exposes client for…

EU project: RIO31844-OMII-EUROPE 43 OGSA-DAI 3.0 OGSA-DAI has evolved constantly since February 2002 OGSA-DAI 2.2 released April 2006 As the number of users grew so did the requirements –More effective data streaming –Standardisation of activity inputs and outputs –Targeting multiple data resources in a single workflow –Supporting application-specific presentation layers OGSA-DAI 2.2 was not suitable for addressing these OGSA-DAI 3.0 –A complete re-design and re-implementation of OGSA-DAI –A stable framework for the future –Released September 2007

EU project: RIO31844-OMII-EUROPE 44 Where might OGSA-DAI not be suitable? OGSA-DAI is not –A complete solution to every data-related problem –A replacement for or competitor to JDBC –Just about accessing relational databases It is not suitable if –You have a single data resource that isn’t going to change –You have no data transformation requirements –You want rapid access to data in a single data resource

EU project: RIO31844-OMII-EUROPE 45 What is OGSA-DAI? An extensible framework accessed via web services that executes data-centric workflows involving heterogeneous data resources for the purposes of data access, integration, transformation and delivery within a grid and is intended as a toolkit for building higher-level application-specific data services

EU project: RIO31844-OMII-EUROPE 46 Thank you!