SAN DIEGO SUPERCOMPUTER CENTER HDF5/SRB Integration August 28, 2006 Mike Wan SRB, SDSC Peter Cao

Slides:

Advertisements

Similar presentations

Data Grid: Storage Resource Broker Mike Smorul. SRB Overview Developed at San Diego Supercomputing Center. Provides the abstraction mechanisms needed.

Advertisements

NATIONAL PARTNERSHIP FOR ADVANCED COMPUTATIONAL INFRASTRUCTURE SAN DIEGO SUPERCOMPUTER CENTER Particle Physics Data Grid PPDG Data Handling System Reagan.

SAN DIEGO SUPERCOMPUTER CENTER HDF5/SRB Integration July 10, 2006 Mike Wan SRB, SDSC Peter Cao

October 15-17, 2008HDF and HDF-EOS Workshop XII, Denver, CO1 HDF5-iRODS Peter Cao The HDF Group Mike Wan San Diego Supercomputer Center HDF and HDF-EOS.

iRODS: Interoperability in Data Management

Chronopolis: Preserving Our Digital Heritage David Minor UC San Diego San Diego Supercomputer Center.

Presentation Outline  Project Aims  Introduction of Digital Video Library  Introduction of Our Work  Considerations and Approach  Design and Implementation.

Robust Tools for Archiving and Preserving Digital Data Joseph JaJa, Mike Smorul, and Mike McGann Institute for Advanced Computer Studies Department of.

Tools and Services for the Long Term Preservation and Access of Digital Archives Joseph JaJa, Mike Smorul, and Sangchul Song Institute for Advanced Computer.

Systems Integrator Benefits V7. Agenda Product Features  Faster Configuration  Online change deployment to clients  Faster server change deployment.

© Copyright 2000 M. Rodriguez-Martinez, All Rights Reserved Automatic Deployment of Application-Specific Metadata and Code in MOCHA Manuel Rodriguez-Martinez.

QA Automation Solution. Solution Architecture Test Management tool CI Tool Automation framework Testing Project BDD Tool Text of test to Testing Project.

6/1/2001 Supplementing Aleph Reports Using The Crystal Reports Web Component Server Presented by Bob Gerrity Head.

Linux Operations and Administration

Database System Concepts and Architecture Lecture # 3 22 June 2012 National University of Computer and Emerging Sciences.

XA R7.8 Upgrade Process and Technical Overview Ruth Anne Pharr Sr. IT Consultant, CISTECH Inc.

OSG Public Storage and iRODS

National Center for Supercomputing Applications The Computational Chemistry Grid: Production Cyberinfrastructure for Computational Chemistry PI: John Connolly.

A Framework for Relationship Discovery Among Files of Different Types Michal Ondrejcek, Jason Kastner and Peter Bajcsy National Center for Supercomputing.

San Diego Supercomputer Center Grid Physics Network (GriPhyN) University of Florida Dataflows in SRB using SDSC Matrix Arun Jagatheesan Architect & Team.

Appraisal and Data Mining of Large Size Complex Documents Rob Kooper, William McFadden and Peter Bajcsy National Center for Supercomputing Applications.

February 2-3, 2006SRB Workshop, San Diego P eter Cao, NCSA Mike Wan, SDSC Sponsored by NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration Object-level.

GUI For A Virtual Pipeline Simulation Testbed By, Revathi Manni Ranganathan Major Professor: Dr.Virgil Wallentine.

December 1, 2005HDF & HDF-EOS Workshop IX P eter Cao, NCSA December 1, 2005 Sponsored by NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration.

Integrated Collaborative Information Systems Ahmet E. Topcu Advisor: Prof Dr. Geoffrey Fox 1.

The Network Performance Advisor J. W. Ferguson NLANR/DAST & NCSA.

Collaborative Tools for Grid Support Laura F. M c Ginnis Pittsburgh Supercomputing Center Pittsburgh, PA, USA July 20, 2002 NeSC Workshop on Applications.

The Global Video Grid: DigitalWell Update & Plan For SRB Integration Myke Smith, Manager Streaming Media Technologies University of Washington / ResearchChannel.

Data Management BIRN supports data intensive activities including: – Imaging, Microscopy, Genomics, Time Series, Analytics and more… BIRN utilities scale:

CYBERINFRASTRUCTURE FOR THE GEOSCIENCES Data Replication Service Sandeep Chandra GEON Systems Group San Diego Supercomputer Center.

Event-Based Hybrid Consistency Framework (EBHCF) for Distributed Annotation Records Ahmet Fatih Mustacoglu Advisor: Prof. Geoffrey.

1 Ilkay ALTINTAS - July 24th, 2007 Ilkay ALTINTAS Director, Scientific Workflow Automation Technologies Laboratory San Diego Supercomputer Center, UCSD.

Integrating HDF5 with SRB The HDF5-SRB Architecture Peter Cao, HDF, NCSA February 24, 2005.

SAN DIEGO SUPERCOMPUTER CENTER HDF5/SRB Integration June 14, 2006 Mike Wan SRB, SDSC Peter Cao

March 17, 2006CIP Status Meeting March 17, 2006 Sponsored by NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration Project Report at CIP AG Meeting.

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure SRB + Web Services = Datagrid Management System (DGMS) Arcot.

Kurt Mueller San Diego Supercomputer Center NPACI HotPage Updates.

National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center Persistent Archive for the NSDL Reagan W. Moore Charlie Cowart.

_______________________________________________________________CMAQ Libraries and Utilities ___________________________________________________Community.

NEES Cyberinfrastructure Center at the San Diego Supercomputer Center, UCSD George E. Brown, Jr. Network for Earthquake Engineering Simulation Alternative.

Kepler includes contributors from GEON, SEEK, SDM Center and Ptolemy II, supported by NSF ITRs (SEEK), EAR (GEON), DOE DE-FC02-01ER25486.

GRIDS Center Middleware Overview Sandra Redman Information Technology and Systems Center and Information Technology Research Center National Space Science.

6/1/2001 Supplementing Aleph Reports Using The Crystal Reports Web Component Server Presented by Bob Gerrity Head.

A radiologist analyzes an X-ray image, and writes his observations on papers  Image Tagging improves the quality, consistency.  Usefulness of the data.

Cactus/TIKSL/KDI/Portal Synch Day. Agenda n Main Goals:  Overview of Cactus, TIKSL, KDI, and Portal efforts  present plans for each project  make sure.

Feedback Enhanced Environment for Learning (FEEL) May 3 rd, 2004 DLC Hands-on Project.

The Global Land Cover Facility is sponsored by NASA and the University of Maryland.The GLCF is a founding member of the Federation of Earth Science Information.

NEES Cyberinfrastructure Center at the San Diego Supercomputer Center, UCSD George E. Brown, Jr. Network for Earthquake Engineering Simulation NEES TeraGrid.

1 CYBERINFRASTRUCTURE FOR THE GEOSCIENCES IGEON 2007 at the University of Hyderabad, India, August Web Services – The Motivation.

19 May 2006 NCSA CIP Status Meeting 1 HDF5/SRB Integration Peter Cao & Mike Folk, NCSA Mike Wan & Reagan Moore, SDSC.

HDF and HDF-EOS Workshop VIII, October 26-28, /12 Peter Cao, National Center for Supercomputing Applications Ray Milnurn, Dave Buto, L-3 Communications.

1 Microsoft Windows 2000 Network Infrastructure Administration Chapter 4 Monitoring Network Activity.

Replica Management Kelly Clynes. Agenda Grid Computing Globus Toolkit What is Replica Management Replica Management in Globus Replica Management Catalog.

Introduction to The Storage Resource.

Mantid Stakeholder Review Nick Draper 01/11/2007.

K. Harrison CERN, 22nd September 2004 GANGA: ADA USER INTERFACE - Ganga release status - Job-Options Editor - Python support for AJDL - Job Builder - Python.

The Storage Resource Broker and.

Collection-Based Persistent Archives Arcot Rajasekar, Richard Marciano, Reagan Moore San Diego Supercomputer Center Presented by: Preetham A Gowda.

1 Particle Physics Data Grid (PPDG) project Les Cottrell – SLAC Presented at the NGI workshop, Berkeley, 7/21/99.

Data Infrastructure in the TeraGrid Chris Jordan Campus Champions Presentation May 6, 2009.

Review of PARK Reflectometry Group 10/31/2007. Outline Goal Hardware target Software infrastructure PARK organization Use cases Park Components. GUI /

Redmond Protocols Plugfest 2016 Jinghui Zhang Office Interoperability Test Tools (Test Suites and Open Source Projects) Software Engineer Microsoft Corporation.

Data Grids, Digital Libraries and Persistent Archives: An Integrated Approach to Publishing, Sharing and Archiving Data. Written By: R. Moore, A. Rajasekar,

Performance measurement of transferring files on the federated SRB

VisIt Project Overview

Problem: Ecological data needed to address critical questions are dispersed, heterogeneous, and complex Solution: An internet-based mechanism to discover,

OGCE OGCE The Open Grid Computing Environments Collaboratory

File service architecture

Gordon Erlebacher Florida State University

Presentation transcript:

SAN DIEGO SUPERCOMPUTER CENTER HDF5/SRB Integration August 28, 2006 Mike Wan SRB, SDSC Peter Cao HDF, NCSA Sponsored by CIP/NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration

SAN DIEGO SUPERCOMPUTER CENTER Current Status Improved transferring raw data (SDSC) Tested performance test #1 (270MB) from 18 to 15 (seconds) test #2 (550MB) from 231 to 35 (seconds) test #3 (820MB) from 734 to 113 (seconds) Project plan for SRB-FLASH

SAN DIEGO SUPERCOMPUTER CENTER Things to move forward Improve GUI component in HDFView to open remote files (NCSA) Support primitive writing capabilities for HDF5 files in SRB (NCSA and SDSC) Write documentation and extensive testing (NCSA and SDSC) Release the production system and publish documentation (NCSA and SDSC)

SAN DIEGO SUPERCOMPUTER CENTER SAP/SAC program: FLASH Slice (gas temperature) and isosurface (gas density) in particle/mesh simulation FLASH is a adaptive-mesh simulation code for astrophysical hydrodynamicsFLASH FLASH slice extractor retrieves a specific 2D slice from a 3D AMR (adaptive mesh refinement)

SAN DIEGO SUPERCOMPUTER CENTER SRB-FLASH: architecture

SAN DIEGO SUPERCOMPUTER CENTER SRB-FLASH: benefits No need for large storage on local machine. Terabytes of data files sit only on the sever machine Instant access to the data and metadata. Since only the selected data/metadata are transferred to the client instead of the whole multi-terabyte file, time on the network is tremendously reduced Data sharing among scientists. Since data sits on the server machine, it can be shared with scientists without multiple local copies of the same file. Also, everyone will have access to the updated data after a new simulation run

SAN DIEGO SUPERCOMPUTER CENTER SRB-FLASH: project plan This project will be divided into three phases: phase I, phase II and phase III Phase I and part of phase II are required Phase III and part of phase II are optional, depending on the duration and amount of the funding The time estimates are based on an half (50%) FTE

SAN DIEGO SUPERCOMPUTER CENTER SRB-FLASH: milestone Phase I: a command line client utility and server function to support selection of FLASH AMR mesh data. TasksStartFinish Collaboration between NCSA and SDSCDuration of the project Collecting requirements10/2/0610/13/06 Exploying the current tool10/16/0610/27/06 Implementing server function10/30/0612/22/06 Implementing client utility12/25/062/16/07 Testing with FLASH AMR mesh data2/19/073/16/07 Deploring to evaluation server3/19/074/13/07 Documentation and release4/16/075/11/07

SAN DIEGO SUPERCOMPUTER CENTER SRB-FLASH: milestone Phase II: enhanced utilities querying FLASH specific metadata and statistics, and performing data interpolation and simple mathematic manipulation. TasksStartFinish Collaboration between NCSA and SDSCDuration of the project Query and return simple statistics5/14/076/22/07 Extract and return particle information6/25/078/17/07 Analyze and interpolate line segment8/20/0710/12/07 Add support for png and ASCII text10/15/0711/23/07

SAN DIEGO SUPERCOMPUTER CENTER SRB-FLASH: milestone Phase III: development of general framework that can be adapted to other applications. We need to scope out the milestone for this phase based on the experience of phase I and II.