Download presentation
Presentation is loading. Please wait.
1
HDF5 and casacore Ger van Diepen ASTRON
2
casacore C++ library for astronomical data handling
Array templated N-dim arrays (STL-conforming) Record dict-like container Table RDBMS-like storage mechanism; TaQL query language MeasurementSet visibility data storage and access (using Tables) Measures values in frame (direction, position, epoch, ...) Coordinates world coordinates for images Images N-dimensional image cubes with 0 or more masks (also supports HDF5, FITS, Miriad, expressions (LEL)) Used by LOFAR, ASKAP, ALMA, eVLA, MeqTrees, pyrap, pydal See Download casacore.googlecode.com Classes TaQL LEL MS definition
3
Simple C++ classes on top of subset of HDF5 API
HDF5 in casacore Simple C++ classes on top of subset of HDF5 API Core HDF5Hid classes Resource handling (automatically close a hid) HDF5File HDF5DataType Create HDF5 data types for C++ data types HDF5DataSet HDF5 dataset of various types Uses chunking HDF5Record Map a Record (dict) to HDF5 groups
4
HDF5Lattice HDF5Image N-dim array in a HDF5DataSet
N-dim array in a HDF5DataSet Part of casacore Lattice/Image framework Handles iteration, subsetting, etc. Automatically sizes HDF5 chunk cache given an access pattern HDF5Image HDF5Lattice with world coordinates Coordinates stored using HDF5Record Part of casacore Image framework PagedImage, FITSImage, MiriadImage, ImageExpression
5
Performance tLatticePerf
Create a 3-dim float lattice using casacore Tables or HDF5 on MacBook 2 GB RAM Iterate by chunk, plane, and line line is done tile-wise to reduce required cache size Various array and chunk sizes 128^3 16^3 8 MB 256^3 16^3 64 MB 512^3 32^ GB 1024,1024,128 64,64, GB 2048,2048, ,128,4 1 GB 4096,4096, ,128,4 2 GB 128^3 array with chunk sizes 2, 4, 8, 16, 32, 64, 128 (^3) Also compared file sizes
6
128^3 float; chunk 2^x HDF5 CCTS cache HDF5 CCTS cache
File size (KB) getTile (sec) HDF CCTS cache more time for very large chunk
7
getplanexy,xz,yz, getlinex,y,z
HDF CCTS cache CCTS mmap
8
8 MB Creating tLatticePerf_tmp.tab with shape [128, 128, 128] and tile shape [16, 16, 16] CCTS create real user system Creating tLatticePerf_tmp.hdf with shape [128, 128, 128] and tile shape [16, 16, 16] HDF5 create real user system CCTS getTiles real user system HDF5 getTiles real user system CCTS xy getPlane real user system HDF5 xy getPlane real user system CCTS xz getPlane real user system HDF5 xz getPlane real user system CCTS yz getPlane real user system HDF5 yz getPlane real user system CCTS x getLine real user system HDF5 x getLine real user system CCTS y getLine real user system HDF5 y getLine real user system CCTS z getLine real user system HDF5 z getLine real user system
![8 MB Creating tLatticePerf_tmp.tab with shape [128, 128, 128] and tile shape [16, 16, 16]](http://slideplayer.com./slide/11727811/65/images/8/8+MB+Creating+tLatticePerf_tmp.tab+with+shape+%5B128%2C+128%2C+128%5D+and+tile+shape+%5B16%2C+16%2C+16%5D.jpg "CCTS create 0.23 real 0.02 user 0.05 system. Creating tLatticePerf_tmp.hdf with shape [128, 128, 128] and tile shape [16, 16, 16] HDF5 create 0.37 real 0.02 user 0.03 system. CCTS getTiles 0.03 real 0.01 user 0.02 system. HDF5 getTiles 0.02 real 0.02 user 0.01 system. CCTS xy getPlane 0.03 real 0.01 user 0.02 system. HDF5 xy getPlane 0.06 real 0.06 user 0.01 system. CCTS xz getPlane 0.04 real 0.01 user 0.02 system. HDF5 xz getPlane 0.08 real 0.07 user 0.01 system. CCTS yz getPlane 0.06 real 0.04 user 0.02 system. HDF5 yz getPlane 0.17 real 0.16 user 0.01 system. CCTS x getLine 0.09 real 0.06 user 0.02 system. HDF5 x getLine 0.85 real 0.83 user 0.01 system. CCTS y getLine 0.08 real 0.06 user 0.02 system. HDF5 y getLine 0.98 real 0.97 user 0.01 system. CCTS z getLine 0.08 real 0.07 user 0.02 system. HDF5 z getLine 1.16 real 1.03 user 0.01 system.")
9
64 MB Creating tLatticePerf_tmp.tab with shape [256, 256, 256] and tile shape [16, 16, 16] CCTS create real user system Creating tLatticePerf_tmp.hdf with shape [256, 256, 256] and tile shape [16, 16, 16] HDF5 create real user system CCTS getTiles real user system HDF5 getTiles real user system CCTS xy getPlane real user system HDF5 xy getPlane real user system CCTS xz getPlane real user system HDF5 xz getPlane real user system CCTS yz getPlane real user system HDF5 yz getPlane real user system CCTS x getLine real user system HDF5 x getLine real user system CCTS y getLine real user system HDF5 y getLine real user system CCTS z getLine real user system HDF5 z getLine real user system
![64 MB Creating tLatticePerf_tmp.tab with shape [256, 256, 256] and tile shape [16, 16, 16]](http://slideplayer.com./slide/11727811/65/images/9/64+MB+Creating+tLatticePerf_tmp.tab+with+shape+%5B256%2C+256%2C+256%5D+and+tile+shape+%5B16%2C+16%2C+16%5D.jpg "CCTS create 2.2 real 0.13 user 0.33 system. Creating tLatticePerf_tmp.hdf with shape [256, 256, 256] and tile shape [16, 16, 16] HDF5 create 2.37 real 0.12 user 0.22 system. CCTS getTiles 0.24 real 0.09 user 0.15 system. HDF5 getTiles 0.15 real 0.1 user 0.05 system. CCTS xy getPlane 0.31 real 0.11 user 0.19 system. HDF5 xy getPlane 0.61 real 0.52 user 0.08 system. CCTS xz getPlane 0.36 real 0.17 user 0.19 system. HDF5 xz getPlane 0.76 real 0.68 user 0.08 system. CCTS yz getPlane 0.56 real 0.37 user 0.19 system. HDF5 yz getPlane 1.55 real 1.45 user 0.09 system. CCTS x getLine 0.5 real 0.34 user 0.16 system. HDF5 x getLine 6.61 real 6.53 user 0.06 system. CCTS y getLine 0.48 real 0.32 user 0.16 system. HDF5 y getLine 7.82 real 7.73 user 0.07 system. CCTS z getLine 0.48 real 0.31 user 0.16 system. HDF5 z getLine 8.31 real 8.19 user 0.08 system.")
10
0.5 GB Creating tLatticePerf_tmp.tab with shape [512, 512, 512] and tile shape [32, 32, 32] CCTS create real user system Creating tLatticePerf_tmp.hdf with shape [512, 512, 512] and tile shape [32, 32, 32] HDF5 create real user system CCTS getTiles real user system HDF5 getTiles real user system CCTS xy getPlane real user system HDF5 xy getPlane real user system CCTS xz getPlane real user system HDF5 xz getPlane real user system CCTS yz getPlane real user system HDF5 yz getPlane real user system CCTS x getLine real user system HDF5 x getLine real user system CCTS y getLine real user system HDF5 y getLine real user system CCTS z getLine real user system HDF5 z getLine real user system
![0.5 GB Creating tLatticePerf_tmp.tab with shape [512, 512, 512] and tile shape [32, 32, 32]](http://slideplayer.com./slide/11727811/65/images/10/0.5+GB+Creating+tLatticePerf_tmp.tab+with+shape+%5B512%2C+512%2C+512%5D+and+tile+shape+%5B32%2C+32%2C+32%5D.jpg "CCTS create real 0.75 user 1.46 system. Creating tLatticePerf_tmp.hdf with shape [512, 512, 512] and tile shape [32, 32, 32] HDF5 create real 0.67 user 0.98 system. CCTS getTiles real 0.62 user 1.52 system. HDF5 getTiles real 0.36 user 0.82 system. CCTS xy getPlane real 0.74 user 1.66 system. HDF5 xy getPlane real 1.46 user 1.13 system. CCTS xz getPlane real 1.25 user 1.74 system. HDF5 xz getPlane real 1.94 user 1.12 system. CCTS yz getPlane real 6.73 user 2.27 system. HDF5 yz getPlane real user 1.45 system. CCTS x getLine real 1.79 user 1.57 system. HDF5 x getLine real user 0.92 system. CCTS y getLine 46 real 2.41 user 2.18 system. HDF5 y getLine 83.2 real user 1.52 system. CCTS z getLine 42.6 real 2.91 user 2.15 system. HDF5 z getLine real user 1.41 system.")
11
0.5 GB Creating tLatticePerf_tmp.tab with shape [1024, 1024, 128] and tile shape [64, 64, 16] CCTS create real user system Creating tLatticePerf_tmp.hdf with shape [1024, 1024, 128] and tile shape [64, 64, 16] HDF5 create real user system CCTS getTiles real user system HDF5 getTiles real user system CCTS xy getPlane real user system HDF5 xy getPlane real user system CCTS xz getPlane real user system HDF5 xz getPlane real user system CCTS yz getPlane real user system HDF5 yz getPlane real user system CCTS x getLine real user system HDF5 x getLine real user system CCTS y getLine real user system HDF5 y getLine real user system CCTS z getLine real user system HDF5 z getLine real user system
![0.5 GB Creating tLatticePerf_tmp.tab with shape [1024, 1024, 128] and tile shape [64, 64, 16]](http://slideplayer.com./slide/11727811/65/images/11/0.5+GB+Creating+tLatticePerf_tmp.tab+with+shape+%5B1024%2C+1024%2C+128%5D+and+tile+shape+%5B64%2C+64%2C+16%5D.jpg "CCTS create real 0.77 user 1.44 system. Creating tLatticePerf_tmp.hdf with shape [1024, 1024, 128] and tile shape [64, 64, 16] HDF5 create real 0.64 user 0.94 system. CCTS getTiles 20.3 real 0.55 user 1.42 system. HDF5 getTiles 19.9 real 0.29 user 0.78 system. CCTS xy getPlane real 0.83 user 1.57 system. HDF5 xy getPlane real 0.86 user 1 system. CCTS xz getPlane real 0.93 user 1.61 system. HDF5 xz getPlane real 1.51 user 1.02 system. CCTS yz getPlane real 7.08 user 2.08 system. HDF5 yz getPlane real user 1.22 system. CCTS x getLine real 1.11 user 1.51 system. HDF5 x getLine real user 0.88 system. CCTS y getLine real 1.72 user 1.97 system. HDF5 y getLine real user 1.19 system. CCTS z getLine real 5.15 user 1.84 system. HDF5 z getLine real user 1.25 system.")
12
1 GB Creating tLatticePerf_tmp.tab with shape [2048, 2048, 64] and tile shape [128, 128, 8] CCTS create real user system Creating tLatticePerf_tmp.hdf with shape [2048, 2048, 64] and tile shape [128, 128, 8] HDF5 create real user system CCTS getTiles real user system HDF5 getTiles real user system CCTS xy getPlane real user system HDF5 xy getPlane real user system CCTS xz getPlane real user system HDF5 xz getPlane real user system CCTS yz getPlane real user system HDF5 yz getPlane real user system CCTS x getLine real user system HDF5 x getLine real user system CCTS y getLine real user system HDF5 y getLine real user system CCTS z getLine real user system HDF5 z getLine real user system
![1 GB Creating tLatticePerf_tmp.tab with shape [2048, 2048, 64] and tile shape [128, 128, 8]](http://slideplayer.com./slide/11727811/65/images/12/1+GB+Creating+tLatticePerf_tmp.tab+with+shape+%5B2048%2C+2048%2C+64%5D+and+tile+shape+%5B128%2C+128%2C+8%5D.jpg "CCTS create real 2.54 user 3.06 system. Creating tLatticePerf_tmp.hdf with shape [2048, 2048, 64] and tile shape [128, 128, 8] HDF5 create real 1.21 user 2.21 system. CCTS getTiles real 2.11 user 3.45 system. HDF5 getTiles real 0.68 user 2.18 system. CCTS xy getPlane real 2.17 user 3.34 system. HDF5 xy getPlane real 1.14 user 2.11 system. CCTS xz getPlane real 1.97 user 3.31 system. HDF5 xz getPlane real 2.58 user 1.9 system. CCTS yz getPlane real user 3.64 system. HDF5 yz getPlane real user 2.35 system. CCTS x getLine real 2.19 user 3.02 system. HDF5 x getLine real user 1.73 system. CCTS y getLine 61.6 real 3.19 user 3.53 system. HDF5 y getLine real user 2.32 system. CCTS z getLine real user 3.59 system. HDF5 z getLine real user 2.63 system.")
13
2 GB Creating tLatticePerf_tmp.tab with shape [4096, 4096, 32] and tile shape [128, 128, 4] CCTS create real user system Creating tLatticePerf_tmp.hdf with shape [4096, 4096, 32] and tile shape [128, 128, 4] HDF5 create real user system CCTS getTiles real user system HDF5 getTiles real user system CCTS xy getPlane real user system HDF5 xy getPlane real user system CCTS xz getPlane real user system HDF5 xz getPlane real user system CCTS yz getPlane real user system HDF5 yz getPlane real user system CCTS x getLine real user system HDF5 x getLine real user system CCTS y getLine real user system HDF5 y getLine real user system CCTS z getLine real user system HDF5 z getLine real user system
![2 GB Creating tLatticePerf_tmp.tab with shape [4096, 4096, 32] and tile shape [128, 128, 4]](http://slideplayer.com./slide/11727811/65/images/13/2+GB+Creating+tLatticePerf_tmp.tab+with+shape+%5B4096%2C+4096%2C+32%5D+and+tile+shape+%5B128%2C+128%2C+4%5D.jpg "CCTS create real 3.14 user 5.79 system. Creating tLatticePerf_tmp.hdf with shape [4096, 4096, 32] and tile shape [128, 128, 4] HDF5 create real 2.59 user 3.7 system. CCTS getTiles real 2.2 user 5.64 system. HDF5 getTiles real 1.21 user 3.31 system. CCTS xy getPlane real 3.41 user 7.21 system. HDF5 xy getPlane real 2.36 user 4.65 system. CCTS xz getPlane real 3.37 user 7.96 system. HDF5 xz getPlane real 9.12 user 5.03 system. CCTS yz getPlane real user 8.04 system. HDF5 yz getPlane real user 5.04 system. CCTS x getLine real 3.12 user 6.12 system. HDF5 x getLine real user 3.49 system. CCTS y getLine real 5.56 user 7.92 system. HDF5 y getLine real user 4.82 system. CCTS z getLine real user 7.67 system. HDF5 z getLine real user 6.02 system.")
14
HDF5 line by line uses a lot of user time
Results HDF5 line by line uses a lot of user time Verified by independent tHDF5 program submitted as bug Due to B-tree lookups? HDF5 uses less system time, but usually slower Files are about equal in size HDF5 slightly larger (more if tiles are smaller) due to B-tree? Note: writes faster because no fsync done for larger data sets kernel’s file cache not used
Similar presentations
