Data Grid: GRASP Mike Smorul. Grid Retrieval and Search Platform Based on concepts developed in the Earth Science Data Interface (ESDI) developed at the.

Slides:

Advertisements

Similar presentations

Texas Digital Library Services Preservation Network.

Advertisements

National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center Data Grids for Collection Federation Reagan W. Moore University.

1 NASA CEOP Status & Demo CEOS WGISS-25 Sanya, China February 27, 2008 Yonsook Enloe.

Peter Berrisford RAL – Data Management Group SRB Services.

Data Grid: Storage Resource Broker Mike Smorul. SRB Overview Developed at San Diego Supercomputing Center. Provides the abstraction mechanisms needed.

NATIONAL PARTNERSHIP FOR ADVANCED COMPUTATIONAL INFRASTRUCTURE SAN DIEGO SUPERCOMPUTER CENTER Particle Physics Data Grid PPDG Data Handling System Reagan.

Chronopolis: Preserving Our Digital Heritage David Minor UC San Diego San Diego Supercomputer Center.

ADAPT An Approach to Digital Archiving and Preservation Technology Principal Investigator: Joseph JaJa Lead Programmers: Mike Smorul and Mike McGann Graduate.

PAWN: Producer-Archive Workflow Network University of Maryland Institute for Advanced Computer Studies Joseph Ja’Ja, Mike Smorul, Mike McGann.

May Archiving PAWN: A Policy-Driven Software Environment for Implementing Producer- Archive Interactions in Support of Long Term Digital.

Tools and Services for the Long Term Preservation and Access of Digital Archives Joseph JaJa, Mike Smorul, and Sangchul Song Institute for Advanced Computer.

Supporting Customized Archival Practices Using the Producer-Archive Workflow Network (PAWN) Mike Smorul, Mike McGann, Joseph JaJa.

Data Grids: Globus vs SRB. Maturity SRB  Older code base  Widely accepted across multiple communities  Core components are tightly integrated Globus.

Brief Overview of Major Enhancements to PAWN. Producer – Archive Workflow Network (PAWN) Distributed and secure ingestion of digital objects into the.

Jean-Yves Nief, CC-IN2P3 Wilko Kroeger, SCCS/SLAC Adil Hasan, CCLRC/RAL HEPiX, SLAC October 11th – 13th, 2005 BaBar data distribution using the Storage.

Robust Tools for Archiving and Preserving Digital Data Joseph JaJa, Mike Smorul, and Mike McGann Institute for Advanced Computer Studies Department of.

Metadata Server system software laboratory. Overview metadata service in Grid environment Grid environment Metadata server User query data search information.

PAWN: A Novel Ingestion Workflow Technology for Digital Preservation

Tools and Services for the Long Term Preservation and Access of Digital Archives Joseph JaJa, Mike Smorul, and Sangchul Song Institute for Advanced Computer.

PAWN: Producer-Archive Workflow Network University of Maryland Institute for Advanced Computer Studies Joseph JaJa, Mike Smorul, Mike McGann.

Mike Smorul Saurabh Channan Digital Preservation and Archiving at the Institute for Advanced Computer Studies University of Maryland, College Park.

UMIACS PAWN, LPE, and GRASP data grids Mike Smorul.

PAWN: A Novel Ingestion Workflow Technology for Digital Preservation Mike Smorul, Joseph JaJa, Yang Wang, and Fritz McCall.

Archival Prototypes and Lessons Learned Mike Smorul UMIACS.

3.1 © 2004 Pearson Education, Inc. Exam Managing and Maintaining a Microsoft® Windows® Server 2003 Environment Lesson 3: Introducing Active Directory.

Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.

UNIVERSITY of MARYLAND GLOBAL LAND COVER FACILITY High Performance Computing in Support of Geospatial Information Discovery and Mining Joseph JaJa Institute.

CISTI Source & SiteSearch OCLC User Meeting 2001 Danielle Langlois & Carol Serroul May 9, 2001.

High-Speed, High Volume Document Storage, Retrieval, and Manipulation with Documentum and Snowbound March 8, 2007.

GRACE Project IST EGAAP meeting – Den Haag, 25/11/2004 Giuseppe Sisto – Telecom Italia Lab.

Presenter: Dipesh Gautam.  Introduction  Why Data Grid?  High Level View  Design Considerations  Data Grid Services  Topology  Grids and Cloud.

SITools Enhanced Use of Laboratory Services and Data Romain Conseil

Jan Storage Resource Broker Managing Distributed Data in a Grid A discussion of a paper published by a group of researchers at the San Diego Supercomputer.

Global Land Cover Facility The Global Land Cover Facility (GLCF) is a member of the Earth Science Information Partnership (ESIP) Federation providing data,

The Global Land Cover Facility Presentation for WGISS-25 February 26, 2008 University of Maryland, Department of Geography.

INFSO-RI Module 01 ETICS Overview Alberto Di Meglio.

The Global Video Grid: DigitalWell Update & Plan For SRB Integration Myke Smith, Manager Streaming Media Technologies University of Washington / ResearchChannel.

Production Data Grids SRB - iRODS Storage Resource Broker Reagan W. Moore

INFSO-RI Module 01 ETICS Overview Etics Online Tutorial Marian ŻUREK Baltic Grid II Summer School Vilnius, 2-3 July 2009.

ISpheresImage iSpheresImage Feature Overview and Progress Summary.

, Key Components of a Successful Earth Science Subsetter Architecture ASDC Introduction The Atmospheric Science Data Center (ASDC) at NASA Langley Research.

1 ILDG Status in Japan  Lattice QCD Archive(LQA) a gateway to ILDG Japan Grid  HEPNet-J/sc an infrastructure for Japan Lattice QCD Grid A. Ukawa Center.

IPlant Collaborative Hands-on Cyberinfrastructure Workshop - Part 1 R. Walls University of Arizona Biodiversity Information Standards (TDWG) Sep. 28, 2015,

May 2003National Coastal Data Development Center Brief Introduction Two components Data Exchange Infrastructure (DEI) Spatial Data Model (SDM) Together,

User Working Group 2013 Data Access Mechanisms – Status 12 March 2013

The Global Land Cover Facility is sponsored by NASA and the University of Maryland.The GLCF is a founding member of the Federation of Earth Science Information.

Create Content Capture Content Review Content Edit Content Version Content Version Content Translate Content Translate Content Format Content Transform.

Mercury – A Service Oriented Web-based system for finding and retrieving Biogeochemical, Ecological and other land- based data National Aeronautics and.

Flexible intelligent solid video management software.

1 e-Science AHM st Aug – 3 rd Sept 2004 Nottingham Distributed Storage management using SRB on UK National Grid Service Manandhar A, Haines K,

1 Overall Architectural Design of the Earth System Grid.

Don’t Duck Metadata March 2005 Introducing Setting Up a Clearinghouse Node Topic: Introduction to Setting Up a Clearinghouse Node Objective: By.

Rights Management for Shared Collections Storage Resource Broker Reagan W. Moore

The Earth Information Exchange. Portal Structure Portal Functions/Capabilities Portal Content ESIP Portal and Geospatial One-Stop ESIP Portal and NOAA.

The Storage Resource Broker and.

EGI-InSPIRE RI EGI Webinar EGI-InSPIRE RI Porting your application to the EGI Federated Cloud 17 Feb

Building Preservation Environments with Data Grid Technology Reagan W. Moore Presenter: Praveen Namburi.

Active Directory Domain Services (AD DS). Identity and Access (IDA) – An IDA infrastructure should: Store information about users, groups, computers and.

Collection-Based Persistent Archives Arcot Rajasekar, Richard Marciano, Reagan Moore San Diego Supercomputer Center Presented by: Preetham A Gowda.

International Planetary Data Alliance Registry Project Update September 16, 2011.

AIRS Meeting GSFC, February 1, 2002 ECS Data Pool Gregory Leptoukh.

Sharing Maps and Layers to Portal for ArcGIS Melanie Summers, Tom Shippee, Ty Fitzpatrick.

A Solution for Maintaining File Integrity within an Online Data Archive Dan Scholes PDS Geosciences Node Washington University 1.

Policy-Based Data Management integrated Rule Oriented Data System

Web Services Primer Overview of Web Services

LCG Monte-Carlo Events Data Base: current status and plans

HingX Project Overview

Client/Server and Peer to Peer

Presentation transcript:

Data Grid: GRASP Mike Smorul

Grid Retrieval and Search Platform Based on concepts developed in the Earth Science Data Interface (ESDI) developed at the UMIACS GLCF. Provides a graphical interface into data grid holdings. Access to entire GLCF holdings through the Storage Resource Broker(SRB)

Earth Science Data Interface

ESDI Overview Designed to allow for intuitive browsing and searching of large geospatial data sets. Tightly integrated set of web, ftp, and file servers. (customized to GLCF) Distributes over 7Tb of data per month Over 27,000 Landsat scenes and 13Tb of data available for download

SRB Grid Testbed (original) Modified the SRB to hold spatial data Contributed Informix port of the SRB (v3.2) Linked three ESIP sites Tested replication between GMU and UMD. Remote registration at UNH

Lessons Learned The SRB can easily handle textual metadata. Spatial data could be stored into extended Informix attributes, but querying was available only through the DAI  Limited SRB MCAT to Informix based systems

GRASP Architecture

GRASP uses a data grid as an abstract storage repository. Metadata in the grid is mined from the grid itself or from external sources and published into a browsable form.  Data grids may allow for platform independent metadata, but may not be optimal for access

GRASP Screenshot

Grid Holdings Registered GLCF holdings  Over 338,000 registered files  4.4Tb total size  Granular permissions on registered holdings No need to move all data into grid, registered pre-existing holdings in place.

Current data grid Designed using newer SRB software that allows for Federated grid model. Created using standard SRB software configuration (postgreSQL) Large data sites can maintain independent MCATs  Administratively independent  Ability to customize data grid to site security/data requirements while maintaining compatibility across federation. Smaller peers can register as clients of UMD

Data grid overview

Growing data grid Additional MCATs can be federated Additional SRB aware clients can be added Remote data supplying sites can contribute data sets from their resources Sites needing quick local access to data can replicate to local SRB setup.

Questions?