San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida Data Grid Services and Pipelines Arun Jagatheesan Architect & Technical Lead, SDSC Matrix NPACI Summer Computing Institute August 18, 2003, San Diego

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 2 Credit / Acknowledgements Participants Allen Ding Lucas Gilbert Reena Mathew Erik Vandiekieft (IBM) Xi Cynthia Sheng Well Wishers Reagan Moore & SRB Team Kim Baldridge YOU !!! Sponsors NSF GriPhyN, NSF SCEC, NPACI REU, NIH BIRN

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 3 Lecture Outline Concepts Distributed Data Management Process Flow Pipelines Web Services; Grid Services Theory Data Grid Language (DGL) Practice (Hands-on) SDSC Matrix Web Demo Matrix Java API

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 4 Grid as Utility Computing

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 5 Logical Layers (bits,data,information,..) Storage Resource Transparency Storage Location Transparency E:\srbVault\image.jpg /users/srbVault/image.jpg Select … from srb.mdas.td where... Data Identifier Transparency image_0.jpg…image_100.jpg Data Replica Transparency image.sqlimage.cgiimage.wsdl Virtual Data Transparency Semantic data Organization (with behavior) patientRecordsCollectionmyActiveNeuroCollection Inter- organizational Information Storage Management

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 6 Is that all? We need more Hey, Who is this Guy?

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 7 Data  Discovery Digital entities Meta-data Services State New data updates relationships among data in collections Services invoked to analyze new relationships DGMS applications get notified of state updates

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 8 Distributed Data Management Data collecting Sensor systems, object ring buffers Data organization Collections, manage data context Data sharing Data grids, manage heterogeneity Data publication Digital libraries, support discovery Data preservation Persistent archives, manage technology evolution Data analysis Processing pipelines, choreograph data and knowledge extraction Data mediation Semantic data, mappings between data, information, knowledge Services, Data flow pipeline Management

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 9 Data process-flow pipelines Compute Archive Digital Library Research Input Coordinated execution amongst flows

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 10 Web Services Web Page (HTML) Searched and used by human being Any computer Useful for dissemination of information on any topic Web Service Searched and used by computer programs Any programming language, OS etc Useful for dissemination of services for any topic XML/ WSDL – Web Service Description SOAP (HTTP/SMTP) – Transport/Access UDDI - Discover WSDL SOAP (HTTP/SMTP) UDDI HTML – describe data layout HTTP – transport data Google – discover data

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 11 Lecture Outline Concepts Distributed Data Management Process Flow Pipelines Web Services; Grid Services Theory Data Grid Language (DGL) Practice (Hands-on) SDSC Matrix Web Demo Matrix Java API

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 12 Need for Standard DGL Database (DBMS) SQL 121.Event Hits.sql University of Gators 121.Event Thit.xml National Lab DGMS XML based, Invoke Operations Subset Xquery Process flow DGL DDL, DML, DQL

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 13 Data Grid Language XML based asynchronous protocol Describe data sets, collections, datagrid operations,... Access and Manage data grids, data-flow pipelines Query on data resource (based on W3C XQuery) Facilitates Grid Workflow Sharing of granular state information about execution of each datagrid operation amongst different processes or services

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 14 Data Grid Request (DReq) Datagrid Request Asynchronous requests for data/process-flow in datagrids Requests are either a Transaction or a Status Query Each Transaction consists of one or more Flows Each Flow consists of one ore more datagrid operations Datagrid operation = data transformation or data query A flow can be executed sequential or parallel

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 15 Data Grid Request

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 16 Data Grid Response Datagrid Response Either Transaction Acknowledgement or Status Response Status Response contains the results of a Transaction Response could be received at any granular level Status response is used for coordination of flows and inter-process notifications

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 17 Data Grid Response (DRes)

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 18 Lecture Outline Concepts Distributed Data Management Process Flow Pipelines Web Services; Grid Services Theory Data Grid Language (DGL) Practice (Hands-on) SDSC Matrix Web Demo Matrix Java API

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 19 “Lets play who wants to be a coder” Now its your turn to take the red pill from Matrix It gets interesting from here, lets us all do coding

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 20 SDSC Matrix Architecture Matrix Agent Abstraction In Memory StoreJDBC SRB AgentsOGSA Agent WSDL Agent Persistence (Store) Abstraction Termination Handler Matrix Data Grid Request Processor Transaction Handler Status Query Handler Data flow pipeline Meta data Manager JMS Messaging System JAXM Wrapper OGSARPC-Style for SOAP SOAP Service Wrapper Abstraction Flow Handler and Execution Manager Pipeline Query Processor XQuery Processor Event Publish Subscribe, Notification

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 21 Lesson – 1 : Data Grid Request Create Data Grid Request and its components

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 22 Learn it your self : Task - 1 Create Flow(0) in a Data Grid Request [DGREQ] Create a simple Data Grid Request using Web Demo Add Flow Make it Sequential Add Step Create Collection Collection Name : Click on Flow0 again, to add one more step in this Flow0 Create Container Container Name : Click on DGRequest link to see Flow0 with 2 steps

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 23 Learn it your self : Task - 2 Create Flow(1) in a Data Grid Request [DGREQ] Click on DGRequest link to see Flow0 with 2 steps Click on Add Flow Make it of type parallel Add Step Rename Collection Old Collection : to new name Click on Flow1 link, to add one more step in this Flow1 Create Collection Collection Name : Click on DGRequest link to see 2 Flows with 2 steps each

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 24 Learn it your self : Task - 3 Add Doc Meta for [DGREQ] Click on DOCMETA Fill your name (optional) Press >> to save details Doc Meta is just for reference. The Author is the process which created the request. The Author could have created the request for another user

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 25 Learn it your self : Task - 4 Add USERINFO for [DGREQ] Click on USERFINO Add user id : Organization: Challege Response: Home Directory </home/du22.npaci Storage Resource Press >> to save

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 26 Learn it your self : Task - 5 Add VOINFO for [DGREQ] Click on VOINO Add Server : Port: Click >> to save this in our demo VO Info is for Virtual Organization Information

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 27 Learn it your self : Task - 5 Send Data Grid Request First check if all components are ready We just learnt the components of a DReq. They all must be [Y] in demo, indicating they are ready Click Send If all the components are ok, the Data Grid Request is shown in XML Click Send DGReq

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 28 Lesson – 2 : Data Grid Acknowledgement, Status Get Data Grid Acknowledgement, Send Status Request, Receive Status Response

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 29 Data Grid Acknowledgement Data Grid Requests responded asynchronously Data Grid Acknowledgement Transaction ID to get status and result of DGReq All valid results are responded by this acknowledgement before they are processes Clients use this Acknowledgement Transaction ID The ID may be passed to third parties which can subscribe to these events (Grid Process Pipelines)

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 30 Data Grid Status Req and Response Transaction ID used to find status Later versions can use publish/subscribe Third party subscription also possible

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 31 Lesson – 3 : Query Data XQuery

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 32 XQuery W3C’s long waited answer – next SQL? As always, SDSC and our group lead the way A subset of Xquery on Data Grid has been implemented Built our own Xquery parser Demo CDL (in house project for NPACI Chemistry Digital Library)

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 33 Lesson – 4 : Java API Java API for Matrix

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 34 Demo Java Program Remember, its for programmatic exchange of state information for coordinated execution of data flow pipelines Java API. Sample Program Just download this zip file Unzip the file Type rundemo.bat Type rundemoquery.bat

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 35 Summary Coordinated execution of process-flow pipelines in Grid Environment necessary Data Grid Language in Data Grid like a SQL for databases SDSC Matrix Process flow pipelines Dynamic control of SRB and other services Discovery of process based on the data Check out our latest release Imagine what we can do for your project