1 Data Management for Internet Backplane Protocol by Tang Ming Assoc/Prof. Francis Lee School of Computer Engineering, Nanyang Technological University,

Slides:

Advertisements

Similar presentations

Recent Developments in Logistical Networking Micah Beck, Assoc. Prof. & Director Logistical Computing & Internetworking (LoCI) Lab Computer Science Department.

Advertisements

Giggle: A Framework for Constructing Scalable Replica Location Services Ann Chervenak, Ewa Deelman, Ian Foster, Leanne Guy, Wolfgang Hoschekk, Adriana.

The Replica Location Service In wide area computing systems, it is often desirable to create copies (replicas) of data objects. Replication can be used.

Building Portals to access Grid Middleware National Technical University of Athens Konstantinos Dolkas, On behalf of Andreas Menychtas.

ICS 434 Advanced Database Systems

EGEE-II INFSO-RI Enabling Grids for E-sciencE The gLite middleware distribution OSG Consortium Meeting Seattle,

Database Architectures and the Web

High Performance Computing Course Notes Grid Computing.

Aug Arie Shoshani Particle Physics Data Grid Request Management working group.

An End-to-End Approach to Globally Scalable Network Storage Presented in cs294-4 P2P Systems by Sailesh Krishnamurthy 15 October 2003.

USING THE GLOBUS TOOLKIT This summary by: Asad Samar / CALTECH/CMS Ben Segal / CERN-IT FULL INFO AT:

eGovernance Under guidance of Dr. P.V. Kamesam IBM Research Lab New Delhi Ashish Gupta 3 rd Year B.Tech, Computer Science and Engg. IIT Delhi.

ICPCA 2008 Research of architecture for digital campus LBS in Pervasive Computing Environment 1.

Magda – Manager for grid-based data Wensheng Deng Physics Applications Software group Brookhaven National Laboratory.

Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.

GRID COMPUTING: REPLICATION CONCEPTS Presented By: Payal Patel.

Database Architecture Introduction to Databases. The Nature of Data Un-structured Semi-structured Structured.

ATLAS DQ2 Deletion Service D.A. Oleynik, A.S. Petrosyan, V. Garonne, S. Campana (on behalf of the ATLAS Collaboration)

M i SMob i S Mob i Store - Mobile i nternet File Storage Platform Chetna Kaur.

A Metadata Catalog Service for Data Intensive Applications Presented by Chin-Yi Tsai.

1 School of Computer, National University of Defense Technology A Profile on the Grid Data Engine (GridDaEn) Xiao Nong

1 Configurable Security for Scavenged Storage Systems NetSysLab The University of British Columbia Abdullah Gharaibeh with: Samer Al-Kiswany, Matei Ripeanu.

ESP workshop, Sept 2003 the Earth System Grid data portal presented by Luca Cinquini (NCAR/SCD/VETS) Acknowledgments: ESG.

SmartGRID Ongoing research work in Univ. Fribourg and Univ. Applied Sciences of Western Switzerland (HES-SO) SwiNG Grid Day, Bern, Nov. 26th, 2009 Ye HUANG.

CYBERINFRASTRUCTURE FOR THE GEOSCIENCES Data Replication Service Sandeep Chandra GEON Systems Group San Diego Supercomputer Center.

Introduction to Logistical Networking Micah Beck, Assoc. Prof. & Director Logistical Computing & Internetworking (LoCI) Lab APAN Advanced.

Working together towards a next generation storage element Surya D. Pathak Advanced Computing Center for Research and Education.

1 Logistical Computing and Internetworking: Middleware for the Use of Storage in Communication Micah Beck Jack Dongarra Terry Moore James Plank University.

DIRAC Review (13 th December 2005)Stuart K. Paterson1 DIRAC Review Exposing DIRAC Functionality.

1 4/23/2007 Introduction to Grid computing Sunil Avutu Graduate Student Dept.of Computer Science.

Logistical Networking Micah Beck, Research Assoc. Professor Director, Logistical Computing & Internetworking (LoCI) Lab Computer.

Logistical Networking as an Advanced Engineering Testbed Micah Beck, Assoc. Prof. & Director Logistical Computing & Internetworking (LoCI) Lab

NOVA Networked Object-based EnVironment for Analysis P. Nevski, A. Vaniachine, T. Wenaus NOVA is a project to develop distributed object oriented physics.

Grid Execution Management for Legacy Code Applications Grid Enabling Legacy Code Applications Tamas Kiss Centre for Parallel.

Part Four: The LSC DataGrid Part Four: LSC DataGrid A: Data Replication B: What is the LSC DataGrid? C: The LSCDataFind tool.

Globus Replica Management Bill Allcock, ANL PPDG Meeting at SLAC 20 Sep 2000.

An Exposed Approach to Reliable Multicast in Heterogeneous Logistical Networks Micah Beck, Assoc. Prof. & Director Logistical Computing & Internetworking.

1 Mobile Management of Network Files Alex BassiMicah Beck Terry Moore Computer Science Department University of Tennessee.

The Replica Location Service The Globus Project™ And The DataGrid Project Copyright (c) 2002 University of Chicago and The University of Southern California.

NA-MIC National Alliance for Medical Image Computing UCSD: Engineering Core 2 Portal and Grid Infrastructure.

Wide Area Data Sharing with Logistical Networking Micah Beck, Assoc. Prof. & Director Logistical Computing & Internetworking (LoCI) Lab Computer Science.

Replica Management Services in the European DataGrid Project Work Package 2 European DataGrid.

1 Media Grid Initiative By A/Prof. Bu-Sung Lee, Francis Nanyang Technological University.

Grid-Powered Scientific & Engineering Applications Ho Quoc Thuan INSTITUTE OF HIGH PERFORMANCE COMPUTING.

T3 analysis Facility V. Bucard, F.Furano, A.Maier, R.Santana, R. Santinelli T3 Analysis Facility The LHCb Computing Model divides collaboration affiliated.

CGW 04, Stripped replication for the grid environment as a web service1 Stripped replication for the Grid environment as a web service Marek Ciglan, Ondrej.

Cole David Ronnie Julio. Introduction Globus is A community of users and developers who collaborate on the use and development of open source software,

© 2008 Open Grid Forum File Catalog Development in Japan e-Science Project GFS-WG, OGF24 Singapore Hideo Matsuda Osaka University.

NOVA A Networked Object-Based EnVironment for Analysis “Framework Components for Distributed Computing” Pavel Nevski, Sasha Vanyashin, Torre Wenaus US.

Replica Management Kelly Clynes. Agenda Grid Computing Globus Toolkit What is Replica Management Replica Management in Globus Replica Management Catalog.

Introduction to The Storage Resource.

Managing Data DIRAC Project. Outline  Data management components  Storage Elements  File Catalogs  DIRAC conventions for user data  Data operation.

An End-to-End Approach to Scalable Network Storage Micah Beck, Associate Professor Director, Logistical Computing & Internetworking (LoCI) Lab Terry Moore,

NorduGrid plans and questions for gLite Marko Niinimaki, NorduGrid 3 rd EGEE meeting Athens, April 2005.

Super Computing 2000 DOE SCIENCE ON THE GRID Storage Resource Management For the Earth Science Grid Scientific Data Management Research Group NERSC, LBNL.

An Architectural Approach to Managing Data in Transit Micah Beck Director & Associate Professor Logistical Computing and Internetworking Lab Computer Science.

1 Egrid portal Stefano Cozzini and Angelo Leto. 2 Egrid portal Based on P-GRADE Portal 2.3 –LCG-2 middleware support: broker, CEs, SEs, BDII –MyProxy.

August 28, 2003APAN, Logistical Networking WS DiDaS Distributed Data Storage Ludek Matyska Masaryk University, Institute of Comp. Sci. and CESNET, z.s.p.o.

1 A Scalable Distributed Data Management System for ATLAS David Cameron CERN CHEP 2006 Mumbai, India.

Logistical Networking: Buffering in the Network Prof. Martin Swany, Ph.D. Department of Computer and Information Sciences.

Software Architecture of Sensors. Hardware - Sensor Nodes Sensing: sensor --a transducer that converts a physical, chemical, or biological parameter into.

Scuola Grid INFN, Trieste, 1-12 Dic Managing Confidential Data in the gLite Middleware – The Secure Storage.

Data Grids, Digital Libraries and Persistent Archives: An Integrated Approach to Publishing, Sharing and Archiving Data. Written By: R. Moore, A. Rajasekar,

Oxana Smirnova, Jakob Nielsen (Lund University/CERN)

The Data Grid: Towards an architecture for Distributed Management

DLT Development Update

Grid Computing.

University of Technology

Distributed P2P File System

Scheduled Accomplishments

Presentation transcript:

1 Data Management for Internet Backplane Protocol by Tang Ming Assoc/Prof. Francis Lee School of Computer Engineering, Nanyang Technological University, Singapore

2 0. Outline Introduction The Logistical Networking & Globus Replica Catalog Data Management for IBP

3 1. Introduction Backgrounds of IBP Univ. of Tennessee Base layer of the Logistical Networking A technology to share network storages distributed across the globe Our work Integrate IBP and Globus Replica Catalog, develop the ExNode File Warehouse, and implement a data management system for IBP.

4 2. Logistical Networking IBP: a middleware to allow the sharing of distributed storage resources over the network L-Bone: a resource discovery service that maintains a list of public depots and metadata about those depots exNode: a data structure for aggregation storage allocations LoRS: provide API and command line tools

5 2.1 Examples: Upload / Download a File to IBP with LoRS Tools $ lors_upload blues.mp3 –f –c 1 –F 1 File “blues.mp3” is uploaded to IBP and the exNode file “blues.mp3.xnd” is generated. $ lors_download blues.mp3.xnd –f File “blues.mp3” is downloaded The data resources are not shareable to public Need a data management service for data sharing!

6 3. Globus Replica Catalog Provide the replica location information for Data Grid Logical File Name  Physical File Name LFN PFN File1gsiftp://ssim1.sce.ntu.edu.sg/dir/ File2ftp://ssim4.sce.ntu.edu.sg/dir2/ File3irm://vista.sas.ntu.edu.sg/tmp/

7 3.1 Motivation of using GRC Globus is the de facto standard of the Grid system The GRC has been used by many Data Grid projects It’s a way to integrate IBP and the existed Data Grids

8 4. Data Management for IBP System architecture Components The Logistical Networking Globus Replica Catalog (GRC) ExNode File Warehouse (EFW) IBP Data Management (IDM) Workflow examples (upload & download) Application

9 4.1 System Architecture

ExNode File Warehouse (EFW) EFW stores and maintains the exNode files for the data in IBP Also keeps the metadata about the data files, e.g., the expiration dates and time stamps EFW interacts with IDM and GRC client exNode info. (in memory) Storage Main thread Communication thread Expiration checking thread EFW Communication thread client Globus RC …

IBP Data Management (IDM) IDM interacts with the Logistical Networking, GRC, and EFW, and it provides APIs to applications. Functions: upload, download, refresh, delete, query a file or a file set

Workflow – Uploading a File IBP Data Management (IDM) IBP Data Management (IDM) IBP 1 Globus Replica Catalog (GRC) exNode file warehouse (EFW) User

Workflow – Downloading a File IBP Data Management (IDM) IBP Data Management (IDM) IBP 1 exNode file warehouse (EFW) User Globus Replica Catalog (GRC)

Support Grid Computing The system provides a global data cache to support Grid computing

15 5. Future Works Develop data discovery and replica selection service Support Globus new service: Replica Location Service (RLS) For further information:

16 Thanks!