2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Dealing with Data: Choosing a Good Storage Technology for Your Application Rick Wagner HPC Systems Manager July 1st, 2014

2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Application Focus Storage choices should be driven by application need, not just what’s available. But, applications need to adapt as they scale. Writing a few small files to an NFS server is fine… writing 1000’s simultaneously will wipe out the server. If you use binary files, don’t invent your own format. Consider HDF5.

2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Storage Technologies File Systems DevicesServices memory block Cloud MySQL CouchDB ext4 NFS Lustre PVFS FUSE

2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Storage Technologies File Systems DevicesServices memory block Cloud MySQL CouchDB ext4 NFS Lustre PVFS FUSE Each has its own performance characteristics Not all are available everywhere

2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO File Systems Classic access, POSIX, Windows Most relevant: Local Remote NFS, CIFS Parallel (Lustre, GPFS) Local file systems are good for small and temporary files Network file systems very convenient for sharing data between systems

2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Parallel File Systems

2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Parallel File Systems OSS72TBOSS72TB 32 OSS (Object Storage Servers) Provide 100GB/s Performance and >4PB Raw Capacity Arista G 10G 10G 10G Redundant Switches for Reliability and Performance 3 Distinct Network Architectures OSS72TBOSS72TBOSS72TBOSS72TBOSS72TBOSS72TB 64 Lustre LNET Routers 100 GB/s 64 Lustre LNET Routers 100 GB/s Mellanox 5020 Bridge 12 GB/s Mellanox 5020 Bridge 12 GB/s MDSMDS MDSMDS Myrinet 10G Switch 25 GB/s Myrinet 10G Switch 25 GB/s MDSMDS GORDON IB cluster GORDON TRITON Myrinet cluster TRITON TRESTLES IB cluster TRESTLES Metadata Servers

2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO A Cautionary Tale

2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Devices Raw block device (/dev/sdb) or RAM FS (/dev/shm) Useful in specific cases, like fast scratch Can be very good for small I/O

2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Services Things accessed programmatically Frequents the last thought for HPC applications: A MISTAKE Databases Cloud storage (Amazon S3) Document storage (MongoDB, CouchDB)

2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Know What You Need

2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Order of Magnitude Guide Storagefile/directoryfile sizesBWIOPs Local HDD1000sGB100 MB/s100 Local SSD1000sGBGB/s10000 RAM FS10000sGBGB/s10000 NFS100sGB100 MB/s100 Lustre/GPFS100sTB100 GB/s1000 CloudInfiniteTB10 GB/s0 DBN/A 10000

2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Choosing My application needs to: I should consider: Write a checkpoint dump from memory from a large parallel simulation. A parallel file system and a binary file format like HDF5.

2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Choosing My application needs to: I should consider: Run analysis on remote systems and return the results to a web portal for users. Cloud storage for results and input, and local scratch space for the job.

2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Choosing My application needs to: I should consider: Randomly access many small files, or read and write small blocks from large files. A database, RAM FS, or local scratch space.

2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Many Boxes Make a Sad Panda Database logos courtesy of RRZEiconsRRZEicons