11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD1 HDF5 Software Process MuQun Yang, Quincey Koziol, Elena Pourmal The HDF Group

Purposes Demonstrate how we maintain HDF5 - Libraries and tools built on top of HDF5 HDF-EOS5, NetCDF4 and Pytables etc Hear your feedback 11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD2

Three pillars for robust software Correctness Performance Coding standard 11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD3

HDF5 software challenges - Portability Portability: IBM,SGI,windows, linux, Solaris, OSF1, cygwin, Cray,FreeBSD, Mac-OS Parallel IO: depends on MPI-IO, parallel File System and hardware - MPI-IO: IBM AIX, MPICH, SGI Altix - Parallel File System: GPFS, Lustre 11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD4

HDF5 software challenges - Features Programming languages - C, Fortran, C++ External libraries: szlib encoder and decoder, zlib Comprehensive internal library test suite - time-consuming tests: fractral heap 11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD5

HDF5 software challenges - Others 34 configuration features --enable-cxx, --enable-fortran etc. THE TESTING CHALLENGE machines x operating systems x compilers x languages x Szip (encoder + no encoder) x (serial + parallel) = a very large number Coordination among developers core library developers 5-6 developers for tools and others - subversion not enough 11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD6

Solutions HDF5 Daily Test on main-stream UNIX platforms -Rob Matzke started around Albert Cheng took over More platforms, testing with more features Different version of HDF5 1.6, 1.8 Other product: HDF4 Other platforms: Windows 11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD7

Daily automatic test procedure 11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD8 Start the automatic job 1. Configuring 2. Compiling library and tools 3. Running tests for library and tools 4. Installing the library 5. Testing examples Sending out the results to hdf5 library mailing lists 1. Platform watcher diagnoses the failure 2. Inform the corresponding developer if the failure is real The developer fixes the problem

An example for daily test Date: Tue, 6 Nov :00: [08:00:15 AM CST] From: HDF Tester To: Subject: kagiso HDF5_Daily_Tests_1106Tue_FAILED!!! 11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD9 Date: Tue, 6 Nov :23: [09:23:49 AM CST] From: Quincey Koziol To: Subject: Re: kagiso HDF5_Daily_Tests_1106Tue_FAILED!!!

11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD10

Other helpers Committest script - automatically test a few platforms before checking in source code Save developers’ time 11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD11

Performance High IO performance is always a goal for THG Detect bad performance in time Performance framework 11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD12

Performance framework Easy to Use for Various Benchmarks Multiple Platforms and Versions Long Term Regression Tests Help Debugging 11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD13

11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD14 Background Backend: Cron job / DB Storage Core: Performance C/C++ Library Frontend: PHP / jpgraph

11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD15 Solution Easy to Use A User’s Benchmark Performance Library Database Web Server cron www HDF5 1.6HDF5 1.8 PHP Graph/Text

11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD16 | | :51:14 | groups | creating empty groups | | hdfdap | | 4384 | for(i=0;i<1000 ;i++) { H5Gcreate(fileid,group_name,(size_t)0)); // Add groups } H5Perf_endTimer(&time); H5Perf_startTimer(&time); H5Perf_addInstance(db_host, date, time); Example Usage * * * /home/local/hyoklee/src/chicago/test-perf-hdfdap-3.sh TimestampInstance Name VersionPlatformTime

Demo 11/7/200717HDF and HDF-EOS Workshop XI, Landover, MD

Other Performance work Performance studies compression, chunking and parallel IO 11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD18

Coding standard Not much except seminars on HDF4/HDF5 coding standards – We definitely need to improve in this area 11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD19

Other work we have done to improve software process 11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD20

User involvements Public mailing lists Public RFCs Solicit comments for new HDF5 features etc. Ask special groups to give us feedback Subversion repo 11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD21

Trainings for developers Internally Book reading: Programming Pearls Attending Dr. Dobb’s software conference 11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD22

Near-term plan Enhance daily correctness tests  API compatibility tests: done  API Version tests: in the process  Java wrapper tests: done  Open source packages that use HDF o EOS2 with HDF4 o EOS5 with HDF5 o NetCDF4 with HDF5 Weekly “stable” code snapshots 11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD23

Long-term Plan Coding standard: code review Standards : - In the process of applying for ISO/ANSI standard for HDF5 500 random API tests to avoid ungraceful crash Collect existing HDF5 files such as EOS2, EOS5 files - Running all HDF4/HDF5 tools on these files periodically Daily correlation regression tests on external machines 11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD24

11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD25 Acknowledgement This work was supported basing upon the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA.