October 15-17, 2008HDF and HDF-EOS Workshop XII, Denver, CO1 HDF5-iRODS Peter Cao The HDF Group Mike Wan San Diego Supercomputer Center HDF and HDF-EOS Workshop XII October 16, 2008

October 15-17, 2008HDF and HDF-EOS Workshop XII, Denver, CO2 Imagine 1 GB 100 Frames x 1 GB = 100 GB HPSS DB HPC

October 15-17, 2008HDF and HDF-EOS Workshop XII, Denver, CO3 Outline HDF5-iRODS module Applications Demo (if time permits)

October 15-17, 2008HDF and HDF-EOS Workshop XII, Denver, CO4 What is iRODS? Stands for i Rule Oriented Data Systems. Developed by the Storage Resource Broker (SRB) team at the San Diego Supercomputer Center (SDSC). A data grid software system that enables a customizable architecture for sharing data distributed across heterogeneous resources.

October 15-17, 2008HDF and HDF-EOS Workshop XII, Denver, CO5 What is iRODS? Distributed StorageDatabase SystemRule System For more information and download, visit

October 15-17, 2008HDF and HDF-EOS Workshop XII, Denver, CO6 Motivation iRODSHDF5 Distributed data system Indexing and searching Access control, etc. Large and diverse data High-performance I/O Subsetting, etc. High-performance distributed data system

October 15-17, 2008HDF and HDF-EOS Workshop XII, Denver, CO7 Whole File Access client Get the file Transfer large file – slow! HDF5 server I need to see the eye of Hurricane Bob! HDF5

October 15-17, 2008HDF and HDF-EOS Workshop XII, Denver, CO8 HDF5 Object or Subset Level Access client HDF5 Get me the eye of hurricane Bob Small transfer – fast! server I need to see the eye of Hurricane Bob!

October 15-17, 2008HDF and HDF-EOS Workshop XII, Denver, CO9 HDF5-iRODS Module Distributed Storage Database System Rule System HDF5 iRODS Module Micro-services

October 15-17, 2008HDF and HDF-EOS Workshop XII, Denver, CO10 HDF5-iRODS Data Flow iRODS message (pack/unpack) clientserver HDF5 Object or Subset (File, Group, Dataset, Subset of Dataset, Attribute) HDF5 Library iRODS message (pack/unpack) HDF5 Object or Subset (File, Group, Dataset, Subset of Dataset, Attribute) HDF5

October 15-17, 2008HDF and HDF-EOS Workshop XII, Denver, CO11 New iRODS Micro-services Five iRODS micro-services  msiH5File_open  msiH5File_close  msiH5Dataset_read reads entire dataset or subset of dataset  msiH5Dataset_read_attribute  msiH5Group_read_attribute File Rule EnginemsiH5Dataset_readH5Dataset.read()

October 15-17, 2008HDF and HDF-EOS Workshop XII, Denver, CO12 HDF5-Enabled iRODS Server HDF5 library Other external libraries (SZIP, ZLIB) iRODS version 1.1 or later from Follow the README instruction at module/hdf5

October 15-17, 2008HDF and HDF-EOS Workshop XII, Denver, CO13 Client Application Requirements HDF5 object header files and client handlers iRODS client library and header files HDF5-iRODS JNI for Java applications only $HOME/.irods/.irodsEnv For more information and download, visit irodsHost 'kagiso.hdfgroup.uiuc.edu' irodsPort 1247 irodsUserName 'rods‘ …

October 15-17, 2008HDF and HDF-EOS Workshop XII, Denver, CO14 Example: HDFView Client ApplicationHDF5-Enabled iRODS Server

October 15-17, 2008HDF and HDF-EOS Workshop XII, Denver, CO15 Example: HDFView

October 15-17, 2008HDF and HDF-EOS Workshop XII, Denver, CO16 Example: islice For more information, visit flash.uchicago.edu Command-line tool to visualize data produced by FLASH simulation runs Data is huge (~ 100 GB) Interesting part is small adaptive mesh 16*16*16*47531 FLASH is an adaptive-mesh simulation code for astrophysical hydrodynamics problems

October 15-17, 2008HDF and HDF-EOS Workshop XII, Denver, CO17 Example: islice Breakout point Ash Flow Star A slice from a 3D simulation of The Detonation of a White Dwarf Star “./islice -t flash.pal -m rpv1 -p 2 rundir_055_8km_hdf5_plt_cnt_0424” 2048*2048*8 (32MB)

October 15-17, 2008 HDF and HDF-EOS Workshop XII, Denver, CO 18 This project is sponsored by CIP/NLADR, NSF PACI Project in Support of the Collaboration between the National Center for Supercomputing Applications (NCSA) and the San Diego Supercomputer Center (SDSC). The project is managed under the CyberInfrastructure Partnership (CIP), a joint effort led by NCSA and SDSC to help scientists and engineers take full advantage of the high-end CyberInfrastructure resources funded by the National Science Foundation (NSF). Thank You!

Questions/comments? October 15-17, 2008HDF and HDF-EOS Workshop XII, Denver, CO19