A PPARC funded project AstroGrid and OGSA-DAI: Early experiences Kona Andrews Guy Rixon Institute of Astronomy University of Cambridge.

Slides:

Advertisements

Similar presentations

Instant JChem - current status and what's coming soon. Tim Dudgeon Solutions for Cheminformatics.

Advertisements

IVOA, Pune India September Data Access Layer Working Group Pune Workshop Summary Doug Tody National Radio Astronomy Observatory International.

2003 May 24Clive Page Simple ADQL Enhancements Clive Page, AstroGrid Project University of Leicester, UK.

3 September 2004NVO Coordination Meeting1 Grid-Technologies NVO and the Grid Reagan W. Moore George Kremenek Leesa Brieger Ewa Deelman Roy Williams John.

September 13, 2004NVO Summer School1 VO Protocols Overview Tom McGlynn NASA/GSFC T HE US N ATIONAL V IRTUAL O BSERVATORY.

September 13, 2004NVO Summer School1 VO Protocols Overview Tom McGlynn NASA/GSFC T HE US N ATIONAL V IRTUAL O BSERVATORY.

VO Standards – Catalog Access Tamás Budavári Johns Hopkins University.

Abstraction Layers Why do we need them? –Protection against change Where in the hourglass do we put them? –Computer Scientist perspective Expose low-level.

Enterprise Java and Data Services Designing for Broadly Available Grid Data Access Services.

OGSA-DAI Render Demo NeSC Review 18 March Description and Aims The OGSA-DAI Render demo is intended to demonstrate both the Process and Data aspects.

Eldas 1.0 Enterprise Level Data Access Services Design Issues, Implementation and Future Development Davy Virdee.

E-Science Data Information and Knowledge Transformation Edikt : e-Science Data, Information and Knowledge Transformation NeSC Review, 30 September 2003.

A PPARC funded project The Grid Data Warehouse Description of prototype work in progress by AstroGrid. Access-Grid lecture to Universities of Leeds and.

VO-DAS Chenzhou CUI Chao LIU, Haijun TIAN, Yang YANG, etc National Astronomical Observatories, CAS.

Simple 3 layer model From Database to User Interface - Modules © Juhani Välimäki 2004.

AstroGrid Consortium Meeting PM Report AstroGrid Consortium Meeting Overview Activities Finance Recruitment Collaboration Phase B.

What is MySQL? MySQL is a relational database management system (A relational database stores data in separate tables rather than putting all the data.

A PPARC funded project AstroGrid Grid Data Warehouse Iteration 5 workgroup meeting.

G O B E Y O N D C O N V E N T I O N WORF: Developing DB2 UDB based Web Services on a Websphere Application Server Kris Van Thillo, ABIS Training & Consulting.

Writing Perform Documents EPCC, University of Edinburgh Amy Krause ( Tom Sugden First International Summer.

Database Connectivity Rose-Hulman Institute of Technology Curt Clifton.

Performed by:Gidi Getter Svetlana Klinovsky Supervised by:Viktor Kulikov 08/03/2009.

AstroGrid Group 7: Teemu Toivola Tero Viitala. Problem several separate databases no common interface between databases difficulties of joining related.

BinX and Astronomy Bob Mann Institute for Astronomy and National e-Science Centre.

Overview of the ODP Data Provider Sergey Sukhonosov National Oceanographic Data Centre, Russia Expert training on the Ocean Data Portal technology, Buenos.

European Space Astronomy Centre (ESAC) Villafranca del Castillo, MADRID (SPAIN) Aurélien Stébé Homogeneous Access to Tabular Data Beijing, China - May.

Analysis of SQL injection prevention using a proxy server By: David Rowe Supervisor: Barry Irwin.

Astrogrid Resource Registry Querying the Registry 1.Mullard Space Science Laboratory, University College London, Holmbury St. Mary, Dorking, Surrey RH5.

A PPARC funded project AstroGrid: new technology for the virtual observatory SC2004 Pittsburgh, PA November 2004 Guy Rixon AstroGrid Technical Architect.

ADO.NET A2 Teacher Up skilling LECTURE 3. What’s to come today? ADO.NET What is ADO.NET? ADO.NET Objects SqlConnection SqlCommand SqlDataReader DataSet.

WSRF Supported Data Access Service (VO-DAS)‏ Chao Liu, Haijun Tian, Dan Gao, Yang Yang, Yong Lu China-VO National Astronomical Observatories, CAS, China.

EdSkyQuery-G Overview Brian Hills, December

Functions and Demo of Astrogrid 1.1 China-VO Haijun Tian.

AstroGrid AstroGrid increases scientific research possibilities by : enabling access to distributed astronomical data and information resources. providing.

OGSA-DAI User Guide The OGSA-DAI Team

Stephen Booth EPCC Stephen Booth GridSafe Overview.

ES Metadata Management Enabling Grids for E-sciencE ES metadata OGSA-DAI NA4 GA Meeting, D. Weissenbach, IPSL, France.

OGSA-DAI Architecture The OGSA-DAI Team

A PPARC funded project AstroGrid Intro & Demo John Taylor Institute for Astronomy, Edinburgh.

1 Database Management Systems: part of the solution or part of the problem? Clive Page 2004 April 28.

Objectives In this lesson, you will learn to: *Identify the need for ADO.NET *Identify the features of ADO.NET *Identify the components of the ADO.NET.

Introduction to the new mainframe © Copyright IBM Corp., All rights reserved. Chapter 12 Understanding database managers on z/OS.

Database Design and Management CPTG /23/2015Chapter 12 of 38 Functions of a Database Store data Store data School: student records, class schedules,

A PPARC funded project Astronomical services: situated software vs. commodity software Guy Rixon, AstroGrid/AVO/IVOA Building Service Based Grids - GGF11.

INFSO-RI Enabling Grids for E-sciencE OGSA DAI Data Access and Integration Marek Ciglan Institute of Informatics, Slovac Academy.

The OGSA-DAI Client Toolkit The OGSA-DAI Team

Solar and space physics datasets within a Virtual Observatory: the AstroGrid experience Silvia Dalla * and Nicholas A Walton  * School of Physics & Astronomy,

A PPARC funded project VOTech AstroGrid DSA Update Kona Andrews Institute for Astronomy University of Edinburgh.

Authentication and Authorization Architecture for AstroGrid and the VO Guy Rixon Tony Linde Elizabeth Auden Nic Walton TIVO, June 2002.

16 October 2003Registry Interface CallsIVOA Interoperability, Strasbourg IVOA Interoperability Elizabeth Auden & Registry Workgroup 16 – 17 October 2003.

A PPARC funded project PM Report AstroGrid Project Meeting Cambridge April 2004.

EBI is an Outstation of the European Molecular Biology Laboratory. easyDAS Automatic DAS server creation Bernat Gel Universitat Politècnicade Catalunya.

Basics of JDBC Session 14.

Chapter 9  Definition of terms  List advantages of client/server architecture  Explain three application components:

DGC Paris Spitfire A Relational DB Service for the Grid Leanne Guy Peter Z. Kunszt Gavin McCance William Bell European DataGrid Data Management.

AMGA-Bookkeeping Carmine Cioffi Department of Physics, Oxford University UK Metadata Workshop Oxford, 05 July 2006.

OGSA-DAI 简介及其它在 China-VO DAS 系统中的应用杨阳中国虚拟天文台研发团队 Chinese Virtual Observatory.

Securing Grid Services – OGF19 > Thijs Metsch > securing_grid_services_ogf19.ppt > Slide 1 Application Level Gateway Securing services using.

ODP V2 Data Provider overview. 22 Scope Data Provider provides access to data and metadata of the local data systems. Data Provider is a wrapper, installed.

Introduction: AstroGrid increases scientific research possibilities by enabling access to distributed astronomical data and information resources. AstroGrid.

Database: JDBC Overview

Database JDBC Overview CS Programming Languages for Web Applications

Optimising the OGSA-DAI Enactment Model

Client Access, Queries, Stored Procedures, JDBC

Azure's Performance, Scalability, SQL Servers Automate Real Time Data Transfer at Low Cost MINI-CASE STUDY “Azure offers high performance, scalable, and.

Introduction to NetDB2 IST210.

Cloud computing mechanisms

Chapter 10 ADO.

OGSA-DAI User Requirements and Scenarios

Presentation transcript:

A PPARC funded project AstroGrid and OGSA-DAI: Early experiences Kona Andrews Guy Rixon Institute of Astronomy University of Cambridge

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Data Set Access: The DataCenter DataCenter component Data Access Plug-in Data Web Services Tomcat / Axis

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences DataCenter JDBC plug-in DataCenter component JDBC DBMS Web Services Tomcat / Axis

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences DataCenter custom plug-in DataCenter component File querier FITS files Web Services Tomcat / Axis

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences DataCenter proxy plug-in DataCenter component Service proxy Vizier Web Services Tomcat / Axis

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences What we imagined OGSA- DAI Web Services DataCenter Component OGSA-DAI client plug-in Tomcat / Axis Grid Service Ports Tomcat / OGSA

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Problem OGSA- DAI Web Services DataCenter Component OGSA-DAI client plug-in Tomcat / Axis Grid Service Ports Tomcat / OGSA

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Problem OGSA- DAI Web Services DataCenter Component OGSA-DAI client plug-in Tomcat / Axis Grid Service Ports Tomcat / OGSA

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Nasty solution OGSA- DAI Web Services DataCenter Component OGSA-DAI client plug-in Tomcat / Axis Grid Ports Tomcat / OGSA OGSA-DAI client Separate JVM

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Current use of OGSA-DAI activities Data Center OGSA-DAI Postgres DBMS ADQL

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Current use of OGSA-DAI activities Data Center OGSA-DAI Postgres DBMS ADQL to SQL ADQL SQL

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Current use of OGSA-DAI activities Data Center OGSA-DAI Postgres DBMS ADQL to SQL sqlQueryStatement ADQL SQL

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Current use of OGSA-DAI activities Data Center OGSA-DAI Postgres DBMS ADQL to SQL sqlQueryStatement ADQL SQL Result rows SQL (to XML RowSet)

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Current use of OGSA-DAI activities Data Center OGSA-DAI Postgres DBMS ADQL to SQL sqlQueryStatement deliverFromURL (to XML RowSet) ADQL SQL Result rows SQL

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Current use of OGSA-DAI activities Data Center OGSA-DAI Postgres DBMS ADQL to SQL sqlQueryStatement xslTransform deliverFromURL (to XML RowSet) ADQL SQL Result rows SQL VOTable

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Current use of OGSA-DAI activities Data Center OGSA-DAI Postgres DBMS ADQL to SQL sqlQueryStatement xslTransform deliverToFile / deliverToGFTP deliverFromURL (to XML RowSet) ADQL SQL VOTable Result rows SQL

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Current use of OGSA-DAI activities Data Center OGSA-DAI Postgres DBMS ADQL to SQL sqlQueryStatement xslTransform deliverToFile / deliverToGFTP deliverFromURL (to XML RowSet) ADQL SQL VOTable Result rows SQL Deliver results

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Comments Astronomers often need LARGE result sets Thousands to millions of rows Memory usage is a big issue OGSA-DAI 3.1 much better than 3.0 xslTransform activity hogging memory

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences OGSA-DAI speed penalty Rough figures only Measured without xslTransform activity Will likely vary with available memory

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Planned extension: User tables OGSA- DAI DBMS Data Center R/O Datasets QueryQuery results

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Planned extension: User tables OGSA- DAI DBMS Data Center R/O Datasets User tables QueryQuery resultsData flow GDS to GDS transfer Bulk load OGSA- DAI

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences DBMS Planned extension: User tables OGSA- DAI DBMS Data Center Data Center OGSA- DAI Table Manager R/O Datasets User tables QueryQuery resultsData flow

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences DBMS Planned extension: User tables OGSA- DAI DBMS Data Center Data Center OGSA- DAI Table Manager Client User data R/O Datasets User tables QueryQuery resultsData flow

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Comments Will be relying on sqlBulkLoadRowSet (!) Table management facilities currently thin Not using X.509 auth at user granularity Still require W/S-based access to G/S

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Our needs Ability to handle very large result sets - Dont rely on loading whole result set into memory! Suite of DB/table management activities - List tables in DB - Describe table column types / space usage - Last access/modification time - Etc etc etc Ability to create and delete indices on tables Ultra-simple installation procedures

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences The End