HDF5 and casacore Ger van Diepen ASTRON

casacore C++ library for astronomical data handling Array templated N-dim arrays (STL-conforming) Record dict-like container Table RDBMS-like storage mechanism; TaQL query language MeasurementSet visibility data storage and access (using Tables) Measures values in frame (direction, position, epoch, ...) Coordinates world coordinates for images Images N-dimensional image cubes with 0 or more masks (also supports HDF5, FITS, Miriad, expressions (LEL)) Used by LOFAR, ASKAP, ALMA, eVLA, MeqTrees, pyrap, pydal See Download casacore.googlecode.com Classes www.astron.nl/casacore/trunk/casacore/doc/html TaQL www.astron.nl/casacore/trunk/casacore/doc/notes/199.html LEL www.astron.nl/casacore/trunk/casacore/doc/notes/223.html MS definition www.astron.nl/casacore/trunk/casacore/doc/notes/229.html

Simple C++ classes on top of subset of HDF5 API HDF5 in casacore Simple C++ classes on top of subset of HDF5 API http://www.astron.nl/casacore/trunk/casacore/doc/html/group__HDF5__module.html Core HDF5Hid classes Resource handling (automatically close a hid) HDF5File HDF5DataType Create HDF5 data types for C++ data types HDF5DataSet HDF5 dataset of various types Uses chunking HDF5Record Map a Record (dict) to HDF5 groups

HDF5Lattice HDF5Image N-dim array in a HDF5DataSet http://www.astron.nl/casacore/trunk/casacore/doc/html/classcasa_1_1HDF5Lattice.html N-dim array in a HDF5DataSet Part of casacore Lattice/Image framework Handles iteration, subsetting, etc. Automatically sizes HDF5 chunk cache given an access pattern HDF5Image http://www.astron.nl/casacore/trunk/casacore/doc/html/classcasa_1_1HDF5Image.html HDF5Lattice with world coordinates Coordinates stored using HDF5Record Part of casacore Image framework PagedImage, FITSImage, MiriadImage, ImageExpression

Performance tLatticePerf Create a 3-dim float lattice using casacore Tables or HDF5 on MacBook 2 GB RAM Iterate by chunk, plane, and line line is done tile-wise to reduce required cache size Various array and chunk sizes 128^3 16^3 8 MB 256^3 16^3 64 MB 512^3 32^3 0.5 GB 1024,1024,128 64,64,16 0.5 GB 2048,2048,64 128,128,4 1 GB 4096,4096,32 128,128,4 2 GB 128^3 array with chunk sizes 2, 4, 8, 16, 32, 64, 128 (^3) Also compared file sizes

128^3 float; chunk 2^x HDF5 CCTS cache HDF5 CCTS cache File size (KB) getTile (sec) HDF5 CCTS cache more time for very large chunk

getplanexy,xz,yz, getlinex,y,z HDF5 CCTS cache CCTS mmap

8 MB Creating tLatticePerf_tmp.tab with shape [128, 128, 128] and tile shape [16, 16, 16] CCTS create 0.23 real 0.02 user 0.05 system Creating tLatticePerf_tmp.hdf with shape [128, 128, 128] and tile shape [16, 16, 16] HDF5 create 0.37 real 0.02 user 0.03 system CCTS getTiles 0.03 real 0.01 user 0.02 system HDF5 getTiles 0.02 real 0.02 user 0.01 system CCTS xy getPlane 0.03 real 0.01 user 0.02 system HDF5 xy getPlane 0.06 real 0.06 user 0.01 system CCTS xz getPlane 0.04 real 0.01 user 0.02 system HDF5 xz getPlane 0.08 real 0.07 user 0.01 system CCTS yz getPlane 0.06 real 0.04 user 0.02 system HDF5 yz getPlane 0.17 real 0.16 user 0.01 system CCTS x getLine 0.09 real 0.06 user 0.02 system HDF5 x getLine 0.85 real 0.83 user 0.01 system CCTS y getLine 0.08 real 0.06 user 0.02 system HDF5 y getLine 0.98 real 0.97 user 0.01 system CCTS z getLine 0.08 real 0.07 user 0.02 system HDF5 z getLine 1.16 real 1.03 user 0.01 system

64 MB Creating tLatticePerf_tmp.tab with shape [256, 256, 256] and tile shape [16, 16, 16] CCTS create 2.2 real 0.13 user 0.33 system Creating tLatticePerf_tmp.hdf with shape [256, 256, 256] and tile shape [16, 16, 16] HDF5 create 2.37 real 0.12 user 0.22 system CCTS getTiles 0.24 real 0.09 user 0.15 system HDF5 getTiles 0.15 real 0.1 user 0.05 system CCTS xy getPlane 0.31 real 0.11 user 0.19 system HDF5 xy getPlane 0.61 real 0.52 user 0.08 system CCTS xz getPlane 0.36 real 0.17 user 0.19 system HDF5 xz getPlane 0.76 real 0.68 user 0.08 system CCTS yz getPlane 0.56 real 0.37 user 0.19 system HDF5 yz getPlane 1.55 real 1.45 user 0.09 system CCTS x getLine 0.5 real 0.34 user 0.16 system HDF5 x getLine 6.61 real 6.53 user 0.06 system CCTS y getLine 0.48 real 0.32 user 0.16 system HDF5 y getLine 7.82 real 7.73 user 0.07 system CCTS z getLine 0.48 real 0.31 user 0.16 system HDF5 z getLine 8.31 real 8.19 user 0.08 system

0.5 GB Creating tLatticePerf_tmp.tab with shape [512, 512, 512] and tile shape [32, 32, 32] CCTS create 21.31 real 0.75 user 1.46 system Creating tLatticePerf_tmp.hdf with shape [512, 512, 512] and tile shape [32, 32, 32] HDF5 create 27.15 real 0.67 user 0.98 system CCTS getTiles 20.74 real 0.62 user 1.52 system HDF5 getTiles 19.89 real 0.36 user 0.82 system CCTS xy getPlane 19.17 real 0.74 user 1.66 system HDF5 xy getPlane 29.32 real 1.46 user 1.13 system CCTS xz getPlane 22.93 real 1.25 user 1.74 system HDF5 xz getPlane 26.51 real 1.94 user 1.12 system CCTS yz getPlane 62.63 real 6.73 user 2.27 system HDF5 yz getPlane 74.89 real 19.67 user 1.45 system CCTS x getLine 18.58 real 1.79 user 1.57 system HDF5 x getLine 43.91 real 27.01 user 0.92 system CCTS y getLine 46 real 2.41 user 2.18 system HDF5 y getLine 83.2 real 36.41 user 1.52 system CCTS z getLine 42.6 real 2.91 user 2.15 system HDF5 z getLine 84.69 real 41.87 user 1.41 system

0.5 GB Creating tLatticePerf_tmp.tab with shape [1024, 1024, 128] and tile shape [64, 64, 16] CCTS create 24.28 real 0.77 user 1.44 system Creating tLatticePerf_tmp.hdf with shape [1024, 1024, 128] and tile shape [64, 64, 16] HDF5 create 22.94 real 0.64 user 0.94 system CCTS getTiles 20.3 real 0.55 user 1.42 system HDF5 getTiles 19.9 real 0.29 user 0.78 system CCTS xy getPlane 19.06 real 0.83 user 1.57 system HDF5 xy getPlane 21.11 real 0.86 user 1 system CCTS xz getPlane 22.04 real 0.93 user 1.61 system HDF5 xz getPlane 28.69 real 1.51 user 1.02 system CCTS yz getPlane 48.88 real 7.08 user 2.08 system HDF5 yz getPlane 55.24 real 19.79 user 1.22 system CCTS x getLine 20.12 real 1.11 user 1.51 system HDF5 x getLine 30.67 real 13.22 user 0.88 system CCTS y getLine 41.32 real 1.72 user 1.97 system HDF5 y getLine 55.62 real 22.01 user 1.19 system CCTS z getLine 36.12 real 5.15 user 1.84 system HDF5 z getLine 100.49 real 67.79 user 1.25 system

1 GB Creating tLatticePerf_tmp.tab with shape [2048, 2048, 64] and tile shape [128, 128, 8] CCTS create 39.39 real 2.54 user 3.06 system Creating tLatticePerf_tmp.hdf with shape [2048, 2048, 64] and tile shape [128, 128, 8] HDF5 create 49.96 real 1.21 user 2.21 system CCTS getTiles 59.18 real 2.11 user 3.45 system HDF5 getTiles 73.55 real 0.68 user 2.18 system CCTS xy getPlane 45.98 real 2.17 user 3.34 system HDF5 xy getPlane 49.96 real 1.14 user 2.11 system CCTS xz getPlane 51.63 real 1.97 user 3.31 system HDF5 xz getPlane 43.11 real 2.58 user 1.9 system CCTS yz getPlane 73.26 real 15.21 user 3.64 system HDF5 yz getPlane 98.87 real 39.88 user 2.35 system CCTS x getLine 37.92 real 2.19 user 3.02 system HDF5 x getLine 51.17 real 13.35 user 1.73 system CCTS y getLine 61.6 real 3.19 user 3.53 system HDF5 y getLine 100.33 real 30.68 user 2.32 system CCTS z getLine 70.48 real 16.63 user 3.59 system HDF5 z getLine 300.85 real 241.81 user 2.63 system

2 GB Creating tLatticePerf_tmp.tab with shape [4096, 4096, 32] and tile shape [128, 128, 4] CCTS create 85.27 real 3.14 user 5.79 system Creating tLatticePerf_tmp.hdf with shape [4096, 4096, 32] and tile shape [128, 128, 4] HDF5 create 83.29 real 2.59 user 3.7 system CCTS getTiles 75.12 real 2.2 user 5.64 system HDF5 getTiles 84.17 real 1.21 user 3.31 system CCTS xy getPlane 76.88 real 3.41 user 7.21 system HDF5 xy getPlane 129.02 real 2.36 user 4.65 system CCTS xz getPlane 110.61 real 3.37 user 7.96 system HDF5 xz getPlane 154.15 real 9.12 user 5.03 system CCTS yz getPlane 179.2 real 29.51 user 8.04 system HDF5 yz getPlane 257.18 real 85.12 user 5.04 system CCTS x getLine 74.77 real 3.12 user 6.12 system HDF5 x getLine 105.67 real 26.36 user 3.49 system CCTS y getLine 170.27 real 5.56 user 7.92 system HDF5 y getLine 223.01 real 63.88 user 4.82 system CCTS z getLine 182.7 real 60.03 user 7.67 system HDF5 z getLine 1089.83 real 940.97 user 6.02 system

HDF5 line by line uses a lot of user time Results HDF5 line by line uses a lot of user time Verified by independent tHDF5 program submitted as bug Due to B-tree lookups? HDF5 uses less system time, but usually slower Files are about equal in size HDF5 slightly larger (more if tiles are smaller) due to B-tree? Note: writes faster because no fsync done for larger data sets kernel’s file cache not used