1. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 1 Scaling Up Parallel I/O on the SP David Skinner, NERSC Division, Berkeley Lab

2. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 2 Motivation NERSC uses GPFS for $HOME and $SCRATCH Local disk filesystems on seaborg (/tmp) are tiny Growing data sizes and concurrencies often outpace I/O methodologies

3. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 3 Seaborg.nersc.gov

4. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 4 Case Study: Data Intensive Computing at NERSC Binary black hole collisions Finite differencing on a 1024x768x768x200 grid Run on 64 NH2 nodes with 32GB RAM (2 TB total) Need to save regular snapshots of full grid The first full 3D calculation of inward spiraling black holes done at NERSC by Ed Seidel, Gabrielle Allen, Denis Pollney, and Peter Diener Scientific American April 2002

5. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 5 Problems The binary black hole collision uses a modified version of the Cactus code to solve Einstein’s equations. It’s choices for I/O are serial and MPI-I/O CPU utilization suffers as time is lost to I/O Variation in write times can be severe

6. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 6 Finding solutions Data pattern is a common one Survey strategies to determine the rate and variation in rate

7. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 7

8. 8 Parallel I/O Strategies

9. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 9 Multiple File I/O if(private_dir) rank_dir(1,rank); fp=fopen(fname_r,"w"); fwrite(data,nbyte,1,fp); fclose(fp); if(private_dir) rank_dir(0,rank); MPI_Barrier(MPI_COMM_WORLD);

10. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 10 Single File I/O fd=open(fname,O_CREAT|O_RDWR, S_IRUSR); lseek(fd,(off_t)(rank*nbyte)-1,SEEK_SET); write(fd,data,1); close(fd);

11. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 11 MPI-I/O MPI_Info_set(mpiio_file_hints, MPIIO_FILE_HINT0); MPI_File_open(MPI_COMM_WORLD, fname, MPI_MODE_CREATE | MPI_MODE_RDWR, mpiio_file_hints, &fh); MPI_File_set_view(fh, (off_t)rank*(off_t)nbyte, MPI_DOUBLE, MPI_DOUBLE, "native", mpiio_file_hints); MPI_File_write_all(fh, data, ndata, MPI_DOUBLE, &status); MPI_File_close(&fh);

12. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 12 Results

13. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 13 Scaling of single file I/O

14. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 14 Scaling of multiple file and MPI I/O

15. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 15 Large block I/O MPI I/O on the SP includes the file hint IBM_largeblock_io IBM_largeblock_io=true used throughout, default values show large variation IBM_largeblock_io=true also turns off data shipping

16. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 16 Large block I/O = false MPI on the SP includes the file hint IBM_largeblock_io Except above IBM_largeblock_io=true used throughout IBM_largeblock_io=true also turns off data shipping

17. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 17 Bottlenecks to scaling Single file I/O has a tendency to serialize Scaling up with multiple files create filesystem problems Akin to data shipping consider the intermediate case

18. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 18 Parallel IO with SMP aggregation (32 tasks)

19. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 19 Parallel IO with SMP aggregation (512 tasks)

20. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 20 Summary 2048 1024 512 256 128 64 32 16 1 MB 10 MB 100 MB 1 GB 10 G 100 G Serial Multiple File mod n MPI IO MPI IO collective

21. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 21 Future Work Testing NERSC port of NetCDF to MPI-I/O Comparison with Linux/Intel GPFS NERSC/LBL Alvarez Cluster 84 2way SMP Pentium Nodes Myrinet 2000 Fiber Optic Interconnect Testing GUPFS technologies as they become available