1. NetCDF4 Performance Benchmark

2. Part I Will the performance in netCDF4 comparable with that in netCDF3? Will the performance in netCDF4 comparable with that in netCDF3?

3. Configurations Dataset Dataset 40 MB: 6 files 40 MB: 6 files 1 MB: 6 files 1 MB: 6 files Storage Layout Storage Layout Contiguous Contiguous Chunked (HDF5 default cache size: 1 MB) Chunked (HDF5 default cache size: 1 MB) Chunked (HDF5 cache size: 64 MB) Chunked (HDF5 cache size: 64 MB) System Cache System Cache

4. System Cache On On Use all caches and buffers provided by kernel Use all caches and buffers provided by kernel Drop Drop “drop_caches” to read data from disk “drop_caches” to read data from disk “fsync” to write data into disk “fsync” to write data into disk

5. 10 cases Dataset Storage Layout System Cache 1 40 MB contiguouson 2 contiguousdrop 3 chunked (64 MB cache) on 4 40 MB chunked (64 MB cache) drop 5 40 MB chunked (1 MB cache) on 6 40 MB chunked (1 MB cache) drop 7 1 MB contiguouson 8 contiguousdrop 9 chunked (1 MB cache) on 10 1 MB chunked (1 MB cache) drop

6. Default Hyperslab One big hyperslab is selected One big hyperslab is selected

7. 1. Contiguous layout with cache Dataset Storage Layout System Cache ≈ 40 MB contiguouson

8. 2. Contiguous layout w/o cache Dataset Storage Layout System Cache ≈ 40 MB contiguousdrop

9. 3. Chunked layout with cache Dataset Storage Layout System Cache ≈ 40 MB chunked (HDF5 cache size: 64 MB) on

10. 4. Chunked layout w/o cache Dataset Storage Layout System Cache ≈ 40 MB chunked (HDF5 cache size: 64 MB) drop

11. 5. Chunked layout with cache Dataset Storage Layout System Cache ≈ 40 MB chunked (HDF5 default cache size: 1 MB) on

12. H5Pset_alloc_time(EARLY) Dataset Storage Layout System Cache ≈ 40 MB chunked (HDF5 default cache size: 1 MB) on H5Pset_alloc_time(EARLY)

13. 6. Chunked layout w/o cache Dataset Storage Layout System Cache ≈ 40 MB chunked (HDF5 default cache size: 1 MB) drop

14. 7. Contiguous layout with cache Dataset Storage Layout System Cache ≈ 1 MB contiguouson

15. 8. Contiguous layout w/o cache Dataset Storage Layout System Cache ≈ 1 MB contiguousdrop

16. 9. Chunked layout with cache Dataset Storage Layout System Cache ≈ 1 MB chunked (HDF5 default cache size: 1 MB) on

17. 10. Chunked layout w/o cache Dataset Storage Layout System Cache ≈ 1 MB chunked (HDF5 default cache size: 1 MB) drop

18. Part II Can I get better performance with netCDF4? If yes, under what circumstances can I get better performance? Can I get better performance with netCDF4? If yes, under what circumstances can I get better performance?

19. Non-contiguous Access Logical layout for 2-dimensional arrays Logical layout for 2-dimensional arrays 256 16384 16 1 240

20. Non-contiguous Access Physical layout Physical layout Chunk size [16384][1] Chunk size [8192][1] Chunk size [4096][1]

21. 11. Non-contiguous Access Dataset Storage Layout System Cache ≈ 16 MB contiguous; chunked (default chunk cache) drop

22. 12. Chunked layout with cache Dataset Storage Layout System Cache ≈ 40 MB chunked (chunk cache varies) on

23. 13. Compression Dataset Storage Layout System Cache Radar data chunked (default chunk cache) drop

24. 13. Compression Compression ratio Compression ratio DatasetUncompressedCompressed Compression Ratio Tile172,132,8923,432,55921 Tile272,132,8925,129,48214 Tile372,132,8923,069,25423

25. Part III Can netCDF4 performance be bad? How can I avoid the bad performance? Can netCDF4 performance be bad? How can I avoid the bad performance?

26. 14. Chunk size Too small chunk size is bad Too small chunk size is bad Little bit smaller than  (number of elements) / N  is bad Little bit smaller than  (number of elements) / N  is bad

27. 14. Chunk size chunk0chunk1chunk2chunk3chunk0chunk1chunk2chunk3chunk4chunk5 chunk6chunk7chunk8 3162 791 3162 790 dataset chunk

28. 14. Chunk size Dataset ≈ 64 MB Storage Layout chunked (default chunk cache) System Cache drop

29. 14. Chunk size (more) Dataset ≈ 64 MB Storage Layout chunked (default chunk cache) System Cache drop  n   n  + 1  n  - 1

30. 15. Many Hyperslab selections H5Pcreate() H5Dopen()

31. 15. Many Hyperslab selections

32. Conclusion The performance in netCDF4 is comparable with that in netCDF3 The performance in netCDF4 is comparable with that in netCDF3 Improvement Improvement Non-contiguous access pattern Non-contiguous access pattern Adjusted cache size Adjusted cache size Compression Compression Pitfall Pitfall Small chunk size Small chunk size Many small hyperslab selections Many small hyperslab selections