SAN DIEGO SUPERCOMPUTER CENTER HDF5/SRB Integration July 10, 2006 Mike Wan SRB, SDSC Peter Cao

Slides:

Advertisements

Similar presentations

National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center Data Grids for Collection Federation Reagan W. Moore University.

Advertisements

OGF-23 iRODS Metadata Grid File System Reagan Moore San Diego Supercomputer Center.

The Australian Virtual Observatory e-Science Meeting School of Physics, March 2003 David Barnes.

Inktomi Confidential and Proprietary The Inktomi Climate Lab: An Integrated Environment for Analyzing and Simulating Customer Network Traffic Stephane.

Jialin Liu, Bradly Crysler, Yin Lu, Yong Chen Oct. 15. Seminar Data-Intensive Scalable Computing Laboratory (DISCL) Locality-driven High-level.

University of Chicago Department of Energy The Parallel and Grid I/O Perspective MPI, MPI-IO, NetCDF, and HDF5 are in common use Multi TB datasets also.

Data Grid: Storage Resource Broker Mike Smorul. SRB Overview Developed at San Diego Supercomputing Center. Provides the abstraction mechanisms needed.

NATIONAL PARTNERSHIP FOR ADVANCED COMPUTATIONAL INFRASTRUCTURE SAN DIEGO SUPERCOMPUTER CENTER Early Experiences with Datastar: A 10TF Power4 + Federation.

October 15-17, 2008HDF and HDF-EOS Workshop XII, Denver, CO1 HDF5-iRODS Peter Cao The HDF Group Mike Wan San Diego Supercomputer Center HDF and HDF-EOS.

Visualization of ENZO - Tiger Simulation Amit Chourasia Sr. Visualization Scientist San Diego Supercomputer Center, UCSD AUS Telecon Jan 14, 2010 Collaborative.

SDM center Questions – Dave Nelson What kind of processing / queries / searches biologists do over microarray data? –Range query on a spot? –Range query.

Chronopolis: Preserving Our Digital Heritage David Minor UC San Diego San Diego Supercomputer Center.

Massive High-Performance Global File Systems for Grid Computing -By Phil Andrews, Patricia Kovatch, Christopher Jordan -Presented by Han S Kim.

© , Michael Aivazis DANSE Software Issues Michael Aivazis California Institute of Technology DANSE Software Workshop September 3-8, 2003.

HDF 1 NCSA HDF XML Activities Robert E. McGrath Mike Folk National Center for Supercomputing Applications.

Welcome to the Minnesota SharePoint User Group. Introductions / Overview Project Tracking / Management / Collaboration via SharePoint Multiple Audiences.

App-V, Configuration Manager, and You Douglas Henry Practice Lead, Services Software Logic.

STATUS UPDATE EM SUBCOMMITTEE Friedrich Roth, EM subcommittee chairman SEG 2012, Las Vegas Technical Standards Committee meeting.

Tutorial 1: Getting Started with Adobe Dreamweaver CS4.

Descriptive Data Analysis of File Transfer Data Sudarshan Srinivasan Victor Hazlewood Gregory D. Peterson.

Event Metadata Records as a Testbed for Scalable Data Mining David Malon, Peter van Gemmeren (Argonne National Laboratory) At a data rate of 200 hertz,

Amit Chourasia Visualization Scientist Visualization Services Presented at : Florida State University, Nov 20 th 2006 Scientific Visualization of Large.

Amit Chourasia Visualization Scientist San Diego Supercomputer Center Presented at : Cyberinfrastructure Internship Experiences for Graduate Students Spring.

SAN DIEGO SUPERCOMPUTER CENTER HDF5/SRB Integration August 28, 2006 Mike Wan SRB, SDSC Peter Cao

1 High level view of HDF5 Data structures and library HDF Summit Boeing Seattle September 19, 2006.

1 Overview of HDF5 HDF Summit Boeing Seattle The HDF Group (THG) September 19, 2006.

1 © 2008 Avaya Inc. All rights reserved. IPOffice Configuration Service Emil Ratnam.

February 2-3, 2006SRB Workshop, San Diego P eter Cao, NCSA Mike Wan, SDSC Sponsored by NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration Object-level.

December 1, 2005HDF & HDF-EOS Workshop IX P eter Cao, NCSA December 1, 2005 Sponsored by NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration.

VISUALIZING EARTHQUAKE SIMULATION DATA Amit Chourasia 1, Steve Cutchin 1, Alex DeCastro 1, Geoffrey Ely 2 1 San Diego Supercomputer Center 2 Scripps Institute.

CYBERINFRASTRUCTURE FOR THE GEOSCIENCES Data Replication Service Sandeep Chandra GEON Systems Group San Diego Supercomputer Center.

Technical Workshops | Esri International User Conference San Diego, California Creating Geoprocessing Services Kevin Hibma, Scott Murray July 25, 2012.

Integrating HDF5 with SRB The HDF5-SRB Architecture Peter Cao, HDF, NCSA February 24, 2005.

SAN DIEGO SUPERCOMPUTER CENTER HDF5/SRB Integration June 14, 2006 Mike Wan SRB, SDSC Peter Cao

March 17, 2006CIP Status Meeting March 17, 2006 Sponsored by NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration Project Report at CIP AG Meeting.

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure SRB + Web Services = Datagrid Management System (DGMS) Arcot.

Kurt Mueller San Diego Supercomputer Center NPACI HotPage Updates.

National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center Persistent Archive for the NSDL Reagan W. Moore Charlie Cowart.

A High performance I/O Module: the HDF5 WRF I/O module Muqun Yang, Robert E. McGrath, Mike Folk National Center for Supercomputing Applications University.

The HDF Group Support for NPP/NPOESS by The HDF Group Mike Folk, Elena Pourmal, Peter Cao The HDF Group November 5, 2009 November 3-5,

SAN DIEGO SUPERCOMPUTER CENTER Inca TeraGrid Status Kate Ericson November 2, 2006.

Scalable Systems Software for Terascale Computer Centers Coordinator: Al Geist Participating Organizations ORNL ANL LBNL.

GridFTP GUI: An Easy and Efficient Way to Transfer Data in Grid

NEES Cyberinfrastructure Center at the San Diego Supercomputer Center, UCSD George E. Brown, Jr. Network for Earthquake Engineering Simulation Enabling.

The Global Land Cover Facility is sponsored by NASA and the University of Maryland.The GLCF is a founding member of the Federation of Earth Science Information.

Michael Doherty RAL UK e-Science AHM 2-4 September 2003 SRB in Action.

Center for Computational Visualization University of Texas, Austin Visualization and Graphics Research Group University of California, Davis Molecular.

19 May 2006 NCSA CIP Status Meeting 1 HDF5/SRB Integration Peter Cao & Mike Folk, NCSA Mike Wan & Reagan Moore, SDSC.

HDF and HDF-EOS Workshop VIII, October 26-28, /12 Peter Cao, National Center for Supercomputing Applications Ray Milnurn, Dave Buto, L-3 Communications.

Introduction to The Storage Resource.

Parallel I/O Performance Study and Optimizations with HDF5, A Scientific Data Package MuQun Yang, Christian Chilan, Albert Cheng, Quincey Koziol, Mike.

Visualizing TERASHAKE Amit Chourasia Visualization Scientist Visualization Services San Diego Supercomputer center Geon Visualization Workshop March 1-2,

HDF-EOS Workshop IV September 19-21, 2000 Richard E. Ullman ESDIS Information Architect NASA/ GSFC, Code 423.

Visualization Efforts at San Diego Supercomputer center Amit Chourasia Visualization Scientist Visualization Services Presented to: SCEC-CME All Hands.

SAN DIEGO SUPERCOMPUTER CENTER Advanced User Support Project Overview Thomas E. Cheatham III University of Utah Jan 14th 2010 By Ross C. Walker.

National Archives and Records Administration1 Integrated Rules Ordered Data System (“IRODS”) Technology Research: Digital Preservation Technology in a.

The Storage Resource Broker and.

Visualizing large scale earthquake simulations Amit Chourasia Visualization Scientist San Diego Supercomputer Center Presented to: Advanced User Support,

SAN DIEGO SUPERCOMPUTER CENTER Replication Policies for Federated Digital Repositories Robert H. McDonald Chronopolis Project Manager

Programming Contest Management System Supervisor : Lecturer Phan Tr ư ờng Lâm Students : Hoàng Quang Mạnh Trần Đình Tuấn Nguyễn Thành Trung Phạm Thị Hồng.

Data Infrastructure in the TeraGrid Chris Jordan Campus Champions Presentation May 6, 2009.

Data Grids, Digital Libraries and Persistent Archives: An Integrated Approach to Publishing, Sharing and Archiving Data. Written By: R. Moore, A. Rajasekar,

SCEC Capability Simulations on TeraGrid

HDF Product Designer: Using Templates to Achieve Interoperability

HDF5 for Real-Time and/or Embedded Test Data

Large Data Visualization of Seismic Data (TeraShake)

Outline Problem DiskRouter Overview Details Real life DiskRouters

Lecture 15 Reading: Bacon 7.6, 7.7

Growing importance of metadata for synthetics: Calculating and Sharing Synthetic Seismic Data Dogan Seber University of California, San Diego San Diego.

Presentation transcript:

SAN DIEGO SUPERCOMPUTER CENTER HDF5/SRB Integration July 10, 2006 Mike Wan SRB, SDSC Peter Cao HDF, NCSA Sponsored by CIP/NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration

SAN DIEGO SUPERCOMPUTER CENTER Current Status Present the work at the TeraGrid '06 Publish HDF5 AIP documents White paper: HDF5 METS template: Finish h5ingest command line tool Create HDF5 METS template file Validate HDF5 METS document Setup a demo server to support SCEC files

SAN DIEGO SUPERCOMPUTER CENTER Current Status Work on test suite and bug fix Add code to separate HDF5 I/O time and SRB time Test large files and dataset (>2GB) Fix bug at srb client handler Work on performance improvement Implement a fairly large set of changes for the performance improvement by transfer raw data by byte- stream Need to test on large files

SAN DIEGO SUPERCOMPUTER CENTER Next Month More tests on performance for transferring raw data Add more features to HDFView for SRB support Integrate the software into the SRB configuration and distribution

SAN DIEGO SUPERCOMPUTER CENTER Potential SAC Projects SDSC ENZO project Enzo, 3D cosmological hydrodynamics code, simulating the process of massive star formation and destruction HDF5 is used as file format and parallel file I/O access FLASH Program The UC/DOE collaboration on creating three-dimensional, virtual reality projections of the cosmic explosions HDF5 is used for storing the data and high I/O access SCEC Terascale Earthquake Simulations Over 100 TB data/year Collections at SRB – 2.6 million files, 114 Terabytes

SAN DIEGO SUPERCOMPUTER CENTER TeraShake Surface Seismograms 4D Array (1.2 TB) Time (22,728) Horizontal (3,000) Vertical (1,500) Vector Component (3) Each file: 22,728 x 3,000 x 5 x 1 1,363,680,000 Bytes TeraShake scenario 900 files

SAN DIEGO SUPERCOMPUTER CENTER Example HDF5 File xhist00001hpss-scec xhist00002hpss-scec xhist00003hpss-scec xhist00004hpss-scec xhist00005hpss-scec HDF5 File 32-bit float 22,728 3,000 25

SAN DIEGO SUPERCOMPUTER CENTER File on SRB server

SAN DIEGO SUPERCOMPUTER CENTER Select a Subset

SAN DIEGO SUPERCOMPUTER CENTER HDFView