1. Model-driven Data Layout Selection for Improving Read Performance Jialin Liu 1, Bin Dong 2, Surendra Byna 2, Kesheng Wu 2, Yong Chen 1 Texas Tech University 1 Lawrence Berkeley National Lab (LBNL) 2 HPDIC’14

2. Outline  Motivation  I/O Performance Model  Implementation  Evaluation  Related Work  Conclusion 2 HPDIC’14

3. Trend of Scientific Discovery: Data-intensive Scientific simulations tends to be data intensive  VPIC, plasma physics, 30TB in a single run  LHC, high energy physics, 1PB/yr, Hunt for the GOD Particle 3 Source: LBNL, CERN HPDIC’14

4. Read/Write Pattern Mismatching  The read pattern is usually different with the write pattern  Scientists choose optimal layouts for their application 4 Longitude Latitude Depth Longitude Latitude Depth Write PatternRead Pattern HPDIC’14

5. Various Data Layouts to Bridge the Gap  Read access patterns to a single file may vary over time  Scientists design various layouts: dimension transposing, hilbert curve, z-curve, etc. Z-curve Transposing 0 4 1 5 2 6 3 7 8 12 9 13 10 14 11 15 0 4 1 5 2 6 3 7 8 12 9 13 10 14 11 15 0 4 1 5 2 6 3 7 8 12 9 13 10 14 11 15 0 4 1 5 2 6 3 7 8 12 9 13 10 14 11 15 Hilbert-curve Original 5 HPDIC’14

6. Huge Parameter Space I/O (client)Layout (server) Independent/ Collective I/O Num(proc), num(aggregator) Access pattern: 1D, planes, sub- cube Buffer size, Request size Parallelism Chunk size Z-curve, Hilbert curve, Transposing Stripe count, Stripe size Block size, RAID level Data1 I/O Request1 Better Layouts? I/O Request2 6 Data2 Data3 Data4 Data5 Data6 HPDIC’14

7. Our Contributions 7 HPDIC’14  Multiple data layouts  A new disk-level I/O representation  A new parallel I/O performance model  Automatic layout selection strategy in SDS

8. Challenges 8  Transparently direct the I/O to better organization  Considering different I/O methods, e.g., independent and collective  Aware of various data organization  Low overhead HPDIC’14

9. Example I/O Architecture 9 Diagram of I/O Architecture on Hopper (NERSC) HPDIC’14  /scratch2, 1PB, Lustre, 13 lsi, 6 ost per oss, 2 oss per disk controller  156 OSTS, RAID6 (8+2)  Block size is 512B, read/write in 4K

10. Disk-level I/O Representation 10 Abstract Two Level Data Striping HPDIC’14  HDF5 as data format;  MPI-IO as middle layer;  Luster as file system. Lustre splits the data into "stripes" and stores evenly on OSTs  RAID 6 splits an OST stripe into smaller equal sized chunks of data and stores it on multiple disks

11. I/O Pattern Formalization 11 HPDIC’14 Given a logical I/O request in HDF5 format, e.g., I/O 1

12. BD&BG Estimation 12 HPDIC’14  Step 1: Map I/O to each strip  Step 2: Compute BD  Step 3: Compute BG

13. Model Design 13 Initial Observation HPDIC’14  The p-values of BG and BD are 0.000435 and 0.169462  BG is more significant than BD.

14. Model Design 14 Base linear I/O Model HPDIC’14 Parameter Space Reduce

15. Model Design 15 HPDIC’14 Refining for Independent I/O ost 0 ost 1 ost 2 p0 p1 pm Worst Case: I/O cost is proportional to Nproc Best Case P 6 P 5 P 4 P 3 P 2 P 1 P 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 Worst Case Block Order

16. Model Design 16 HPDIC’14 Process Aggregator ost 0 ost 1 ost 2 ost 3 Aggregated Requests Comm Phase< 20% Refining for Collective I/O

17. Implementation in SDS 17 HPDIC’14  The SDS Metadata Manager manages the metadata.  The Layout Recommender direct the I/O to the best layout.  The server periodically reads the frequently read patterns and contacts the Layout Recommender to identify the best data layouts, to partially re- organize.

18. Experimental Setup 18  Hopper, NERSC, Cray XE6 o Peak performance of 1.28 Petaflops/sec, 153,216 compute cores, 212 Terabytes of memory, and 2 Petabytes of disk  Dataset o 3-D, 128G each file, 100 OST, 1M  Library o HDF5  Test Layouts o 3 transposed datasets, various stripe size  Test I/O pattern o 1D, 2D plane, 3D sub-cube, I/O size, processes HPDIC’14

19. Results 19 HPDIC’14  Synthetic benchmark, 3D array with [1024, 1024, 65536] size in [x, y, z] dimensions.  Store data in [x, y, z], [x, z, y], and [z, x, y] dimensions that represent three different data layouts  Independent I/O, 100 training sets, 8 tests for each pattern with 512 processes  The average speedup for 1D patterns is 128.42X, that for 2D patterns is 68.46X, and that for 3D patterns is 9.26X.

20. Results 20 HPDIC’14  Average residual error is 40.7  Coefficient of determination R 2 is 0.9951

21. Results 21 HPDIC’14  Average residual error is 3.95  Coefficient of determination R 2 is 0.9969

22. Results 22 HPDIC’14

23. Related Work 23 HPDIC’14  Data Layout/Organization Optimization CHARISMA(Purakayastha), EDO(Tian), PDLA(Yin), Smart- IO(Tian), etc No optimal layout for all I/O requests, even for same access pattern  I/O Performance Model HPC-IO(Shan), Queue-Raid(Smirni), Cost-IO(Song), Auto- Tune(You), Machine-L(Wang, Yu), Disk-sim(Bucy) Desire new workload representation to be aware various layouts.

24. Conclusion and Future Work 24 Conclusion  Multiple layouts co-existing can benefit the I/O performance  The model automatically select the best layout  The model can be applied easily into the SDS Future Work  Partial Data Replication with layout cover set HPDIC’14

25. Model-driven Data Layout Selection 25 Thanks HPDIC’14 http://discl.cs.ttu.edu/doku.php