NetCDF4 Performance Benchmark

Part I Will the performance in netCDF4 comparable with that in netCDF3? Will the performance in netCDF4 comparable with that in netCDF3?

Configurations Dataset Dataset 40 MB: 6 files 40 MB: 6 files 1 MB: 6 files 1 MB: 6 files Storage Layout Storage Layout Contiguous Contiguous Chunked (HDF5 default cache size: 1 MB) Chunked (HDF5 default cache size: 1 MB) Chunked (HDF5 cache size: 64 MB) Chunked (HDF5 cache size: 64 MB) System Cache System Cache

System Cache On On Use all caches and buffers provided by kernel Use all caches and buffers provided by kernel Drop Drop “drop_caches” to read data from disk “drop_caches” to read data from disk “fsync” to write data into disk “fsync” to write data into disk

10 cases Dataset Storage Layout System Cache 1 40 MB contiguouson 2 contiguousdrop 3 chunked (64 MB cache) on 4 40 MB chunked (64 MB cache) drop 5 40 MB chunked (1 MB cache) on 6 40 MB chunked (1 MB cache) drop 7 1 MB contiguouson 8 contiguousdrop 9 chunked (1 MB cache) on 10 1 MB chunked (1 MB cache) drop

Default Hyperslab One big hyperslab is selected One big hyperslab is selected

1. Contiguous layout with cache Dataset Storage Layout System Cache ≈ 40 MB contiguouson

2. Contiguous layout w/o cache Dataset Storage Layout System Cache ≈ 40 MB contiguousdrop

3. Chunked layout with cache Dataset Storage Layout System Cache ≈ 40 MB chunked (HDF5 cache size: 64 MB) on

4. Chunked layout w/o cache Dataset Storage Layout System Cache ≈ 40 MB chunked (HDF5 cache size: 64 MB) drop

5. Chunked layout with cache Dataset Storage Layout System Cache ≈ 40 MB chunked (HDF5 default cache size: 1 MB) on

H5Pset_alloc_time(EARLY) Dataset Storage Layout System Cache ≈ 40 MB chunked (HDF5 default cache size: 1 MB) on H5Pset_alloc_time(EARLY)

6. Chunked layout w/o cache Dataset Storage Layout System Cache ≈ 40 MB chunked (HDF5 default cache size: 1 MB) drop

7. Contiguous layout with cache Dataset Storage Layout System Cache ≈ 1 MB contiguouson

8. Contiguous layout w/o cache Dataset Storage Layout System Cache ≈ 1 MB contiguousdrop

9. Chunked layout with cache Dataset Storage Layout System Cache ≈ 1 MB chunked (HDF5 default cache size: 1 MB) on

10. Chunked layout w/o cache Dataset Storage Layout System Cache ≈ 1 MB chunked (HDF5 default cache size: 1 MB) drop

Part II Can I get better performance with netCDF4? If yes, under what circumstances can I get better performance? Can I get better performance with netCDF4? If yes, under what circumstances can I get better performance?

Non-contiguous Access Logical layout for 2-dimensional arrays Logical layout for 2-dimensional arrays

Non-contiguous Access Physical layout Physical layout Chunk size [16384][1] Chunk size [8192][1] Chunk size [4096][1]

11. Non-contiguous Access Dataset Storage Layout System Cache ≈ 16 MB contiguous; chunked (default chunk cache) drop

12. Chunked layout with cache Dataset Storage Layout System Cache ≈ 40 MB chunked (chunk cache varies) on

13. Compression Dataset Storage Layout System Cache Radar data chunked (default chunk cache) drop

13. Compression Compression ratio Compression ratio DatasetUncompressedCompressed Compression Ratio Tile172,132,8923,432,55921 Tile272,132,8925,129,48214 Tile372,132,8923,069,25423

Part III Can netCDF4 performance be bad? How can I avoid the bad performance? Can netCDF4 performance be bad? How can I avoid the bad performance?

14. Chunk size Too small chunk size is bad Too small chunk size is bad Little bit smaller than  (number of elements) / N  is bad Little bit smaller than  (number of elements) / N  is bad

14. Chunk size chunk0chunk1chunk2chunk3chunk0chunk1chunk2chunk3chunk4chunk5 chunk6chunk7chunk dataset chunk

14. Chunk size Dataset ≈ 64 MB Storage Layout chunked (default chunk cache) System Cache drop

14. Chunk size (more) Dataset ≈ 64 MB Storage Layout chunked (default chunk cache) System Cache drop  n   n  + 1  n  - 1

15. Many Hyperslab selections H5Pcreate() H5Dopen()

15. Many Hyperslab selections

Conclusion The performance in netCDF4 is comparable with that in netCDF3 The performance in netCDF4 is comparable with that in netCDF3 Improvement Improvement Non-contiguous access pattern Non-contiguous access pattern Adjusted cache size Adjusted cache size Compression Compression Pitfall Pitfall Small chunk size Small chunk size Many small hyperslab selections Many small hyperslab selections