Databases and the Grid OGSA-DAI Architecture & Status Malcolm Atkinson OGSA-DAI Chief Architect for all members of the OGSA-DAI team Director of National e-Science Centre 3 rd September 2002 UK e-Science All Hands Meeting Sheffield Hallam University

Overview Database Task Force & GGF DAIS-WG OGSA-DAI Project Scope, Scale, Participants, Plans Architecture Status Relationship with OGSA

Data Access & Integration Central to e-Science Collaboration Shared Databases Curated Knowledge Accumulated Observations Accumulated Simulations Computation Data mining Input to models Calibration of models Presentation Publication of results Visualisation

UK DBTF Malcolm Atkinson (NESC) Vijay Dialani (Southampton Uni.) Norman Paton (Manchester Uni.) Dave Pearson (Oracle UK) Tony Storey (IBM Hursley) Paul Watson (Newcastle Uni.) Membership GGF DAIS-WG OGSA-DAI Core Programme Project

GGF DAIS WG Chairs Norman Paton (Manchester Uni.) Leanne Guy (CERN) Dave Pearson (Oracle UK) Activity BoF GGF4 Toronto WG Meeting GGF5 Edinburgh Workshops & Mail lists Goals Agree Standards for Database Access & Integration Freely available reference implementations OGSA-DAI one source & focus for discussions

Particle Physics and Astronomy e-Science Projects GridPP links to EU DataGrid, CERN LHC Computing Project, US GriPhyN and PPDataGrid Projects, and iVDGL Global Grid Project AstroGrid links to EU AVO and US NVO projects From presentation by Tony Hey OGSA-DAI Early Adopter

EPSRC e-Science Projects (2) MyGrid: Personalised Extensible Environments for Data Intensive in silico Experiments in Biology Manchester, EBI, Southampton, Nottingham, Newcastle, Sheffield, GSK, Astra-Zeneca, IBM, Sun GEODISE: Grid Enabled Optimisation and Design Search for Engineering Southampton, Oxford, Manchester, BAE, Rolls Royce Discovery Net: High Throughput Sensing Applications Imperial College, Infosense, … From presentation by Tony Hey OGSA-DAI Early Adopter

Cambridge Oxford Glasgow Cardiff Southampton London Belfast Daresbury Lab RAL Hinxton OGSA-DAI Partners EPCC & NeSC Newcastle IBM USA IBM Hursley Oracle Manchester EPCC & NeSC IBM UK IBM USA Manchester e-SC Newcastle e-SC Oracle $5 million, 18 months, started 1 st February 2002

OGSA-DAI Scope Definition and development of generic Grid data services which provide access to and integration of data held in databases, and the management of data within a distributed environment. Database A stored, structured collection of data Accessed using an API that takes account of the structure of the data stored Includes Relational and object databases XML repositories Adequately described & managed collections of files

Databases in the Grid Computational Complexity Data Complexity

Scope of Database Services Discovery of Data by Content Query and Update Statements Metadata Management & Evolution Transactions (Flavours of) Distributed queries and updates Specialised types Encapsulated (safe) Function application Notification (driven by triggers, etc.)

OGSA-DAI Objectives Produce specifications for generic data services based on a common design framework consistent with Open Grid Service Architecture Design specifications as basis of standards recommendations via Database Access and Integration Services Working Group to the Global Grid Forum Deliver Grid data services software in future releases of the Globus Toolkit (GT3 December 2002) Refine identified requirements evaluate design options develop demonstrators transfer skills to the Grid community Develop reference implementations of generic data services Ensure that the Grid model and OGSA standards address fully the needs of data access and integration Ensure Grid data services meet the levels of service required performance, scalability, resilience, availability, and manageability evolution and distribution large user populations and large data volumes

OGSA-DAI Plan Two Phases Phase 1: Started Feb 02 ends 30 th September Detailed Plan – Requirements, Designs & Prototypes 6 Work Packages Project Management (Oracle, EPCC) Architecture (NeSC, DBTF) XML Data Management (NeSC & EPCC) Distributed Query Systems (Manchester & Newcastle) Metadata & Registries (NeSC & EPCC) Relational Databases (IBM UK) Phase 2: 12 months Structure and Objectives to be Refined in Major Review GGF5 DAIS WG meeting a major input

OGSA-DAI Time Line Feb 02May 02Jul 02Sep 02Dec 02Feb 03May 03Sep 03 Ship Alpha Release for GT3 Integration RDB + GT2 / OGSA Prototypes Available XML + OGSA Prototype Available Design Documents & Demos for DAIS GGF5 XML + OGSA Prototypes for Early Adopters WS + GSI UK support ( > 100 downloads) Phase 2 Starts Phase 1 Starts Presentation & GGF7 GGF6 WG Papers & Prototypes Productisation, RAMPS & Extension

Milestones & Deliverables 3 rd Jul 2002 GGF 5 Deliverables 1st Draft – OGSA-DAI Design Specification Working Grid data service prototype with workshop material Draft Phase 2 functional scope for each Work Package 30 th Sept 2002 End Phase 1 Phase 1 Review Report and recommendations including: revisions to Phase 2 streams of work, Work Package structure, content, and scope Completed, Tested, Work Package prototypes with evaluation report detailing functional scope and deficiencies, design options, measures for acceptance RDBMS/Globus-2 prototype implementation Phase 2 scope Agreed 2 nd Draft – OGSA-DAI design specification Dissemination programme for UK e-Science community Transition programme for UK Grid Support Team and Globus Development Team 31 st Dec 2002 Globus Toolkit Release 1 st Grid data services reference implementation for Globus Toolkit 3 1 st Grid data services specification for Globus Toolkit 3 Scope of functional content for 2 nd Globus Toolkit release and specification 1 st release training and support courses 31 st Mar 2003 Interim UK e-Science community release Interim Grid data services implementation for UK e-Science community Release training and support courses, with documentation 31 st Jul 2003 Globus Toolkit Release 2 nd Grid data services reference implementation for Globus Toolkit 3 2 nd Grid data services specification for Globus Toolkit 3 2 nd release training and support courses Publications and papers to support reference implementations through WG discussions and GGF standards processes Final Project Report

DAI Key Components GridDataServiceGDSAccess to data & DB operations GridDataServiceFactoryGDSFMakes GDS GridDataServiceRegistryGDSRDiscovery of GDS(F) & Data GridDataTransportVehicleGDTV Connects components + Moves Data GridDataTransportDepotGDTDGDTV with persistence

OGSA Relationship ClassGridServiceRegistryNotificationConsumerNotificationProducer GDSMandatory OptionalNormal GDSFMandatory OptionalNormal GDSRMandatory Normal GDTSMandatory GDTV GDTDMandatory OptionalNormal

DAI portType Usage ClassGridDataServiceGridDataTransportFactory GDSMandatoryNormal GDSFOptionalNormalMandatory GDSROptional GDTSOptionalMandatory GDTV GDTDOptionalMandatory

OGSA-DAI: Key Components Grid Database Services (GDS) GXDS, GRDS, GSFDS, … Perform DB actions Extra Data Service Elements DB-action-Management Functions Notifications from Triggers Grid Database Service Factories (GDSF) Create the above Extra Data Service Elements Database Service Registries (DSR) Specialised Registries to find DBs, Services & Factories Grid Data Transfer Services (GDTS) Described at Requirement Level Flexible & mapped to grid-FTP, MQ Series, …

OGSA-DAI Architecture 1 request for factory DSR GDSF client

OGSA-DAI Architecture 2 response with GDSFs GSHs 1 request for factory DSR GDSF client

OGSA-DAI Architecture 2 response with GDSFs GSHs 1 request for factory 3 script for 3 GDSs DSR GDSF client

4 creation of 3 GDSs OGSA-DAI Architecture 2 response with GDSFs GSHs 1 request for factory 3 script for 3 GDSs DSR GDSF GDS 1 GDS 2 GDS 3 client

4 creation of 3 GDSs OGSA-DAI Architecture 5 response with 3 GSHs 2 response with GDSFs GSHs 1 request for factory 3 script for 3 GDSs DSR GDSF GDS 1 GDS 2 GDS 3 client

4 creation of 3 GDSs OGSA-DAI Architecture 6 scripts requesting DB actions 5 response with 3 GSHs 2 response with GDSFs GSHs 1 request for factory 3 script for 3 GDSs DSR GDSF GDS 1 GDS 2 GDS 3 client

4 creation of 3 GDSs OGSA-DAI Architecture 6 scripts requesting DB actions 5 response with 3 GSHs 2 response with GDSFs GSHs 1 request for factory 3 script for 3 GDSs DSR GDSF GDS 1 GDS 2 GDS 3 client 7 transfer data batch to GDS 2 stream to GDS 3

4 creation of 3 GDSs OGSA-DAI Architecture 6 scripts requesting DB actions 5 response with 3 GSHs 2 response with GDSFs GSHs 1 request for factory 3 script for 3 GDSs DSR GDSF GDS 1 GDS 2 GDS 3 client 7 transfer data batch to GDS 2 stream to GDS 3 8 stream data to GDS 2

4 creation of 3 GDSs OGSA-DAI Architecture 6 scripts requesting DB actions 5 response with 3 GSHs 2 response with GDSFs GSHs 1 request for factory 3 script for 3 GDSs DSR GDSF GDS 1 GDS 2 GDS 3 client 9 transfer data batch to client 7 transfer data batch to GDS 2 stream to GDS 3 8 stream data to GDS 2

OGSA-DAI Architecture 4 creation of 3 GDSs 6 scripts requesting DB actions 5 response with 3 GSHs 2 response with GDSFs GSHs 1 request for factory 3 script for 3 GDSs DSR GDSF GDS 1 GDS 2 GDS 3 client 9 transfer data batch to client 7 transfer data batch to GDS 2 stream to GDS 3 8 stream data to GDS 2 10 stream data to specified destination

Status Teams & project coordination effective Relationship with Early Adopters with Grid Support Centre with Globus High rates of interaction XML GDS & GDSF prototypes available RDB demo available Distributed Query demo available Papers & designs presented at GGF5 Functional Scope & Architecture for Phase 2 Drafts & Intensive discussions

OGSA-DAI & OGSA <((-:} Description, e.g. portType Works Well Expect to make extensive use of Data Service Elements Special to DBs: Static & Dynamic Component Management Notification Grid-FTP Accounting Security: Authentication, Authorisation & Privacy Reliable invocation …

OGSA-DAI & OGSA <))-:} Lifetime Issues Conditions for termination Controlled clean-up opportunity Scope of State Evolution Notification Issues Registering & using same notification system For DBs, e.g. triggers do we have to construct a dummy Service Data Element? Type System Issues Standards needed for wide range of types Service Definition Issues How to create / obtain standard definitions for common services

OGSA-DAI Summary On Schedule & Going Well Contributions via GGF5, 6, 7, … Coordinating with GT3 Releases Ending Phase 1 (Design Exploration) Testing Architectural Design Using OGSA Working with Early Adopter Pilot Projects AstroGrid & MyGrid and others Many requests for access to the software Releasing prototypes Influence OGSA-DAI direction Via DAIS-WG & as Prototype users