1. Out-of core Streamline Generation Using Flow-guided File Layout Chun-Ming Chen 788 Project 1

2. Background Visualize flow fields with streamlines Scientific data is huge – Traditional: Compute in clusters – Drawbacks: High Equipment Cost Inter-node communication http://vidi.cs.ucdavis.edu/publications http://nuweb2.neu.edu/ 2

3. Background Nowadays: multi-core CPU on single machine May not have enough memory capacity Out-of-core computation is needed – Out-of-core: data cannot be fully loaded into main memory 3

4. Goal Compute streamlines on a lower-cost multi-core machine with limited memory, given arbitrary seeds 4

5. Demand Paging Algorithm Preparation Stage: – Break flow fields into blocks Streamline Generation Stage: – Only load needed blocks during computation – Release least recently used (LRU) block when memory full 5 Load data from Disk Compute Release data (LRU) Store data in memory pool

6. Multi-core streamline computation 6 Threaded Computation Seeds for block 1 Seeds for block 2 Seeds for block 3 Seeds for block 4 Threaded Computation New seeds generated from block 1 Job Queue

7. Problem of Out-of-core Computation Earlier tests: 1Gb Data – Environment: 8-core Intel Machine Limit 25Mb memory usage – Time Generating streamlines: 8.436 s – Time Loading flow field : 32.524 s IO is the bottle neck 7

8. More tests Read all blocks in a 6Gb data Unit block size: float 16x16x16 (49152 bytes) Total 131,072 blocks – Random access: 249.146794 sec – Sequential read: 13.594309 sec – Reverse-Sequential read: 264.262752 sec Sequential read can be 20 times faster Reason: Disk Prefetching 8

9. File Layout Re-arrange data to increase more sequential reads Hilbert Curve Layout: 9

10. Result of Scheduling for Hilbert Curve Layout Scheduler: only read forward Test: 1Gb Data – Environment: 8-core Intel Machine Limit 25Mb memory usage Old test: – Time Generating streamlines: 8.436 s – Time Loading flow field : 32.524 s Hilbert layout: – Time Generating streamlines: 7.890 s – Time Loading flow field : 0.610 s 10

11. Layout By Flow Direction 11

12. Next And Conclusion Next: – Better layout? – Re-arrange data based on flow direction – NP-hard Problem Conclusion: – If we want to analyze large scientific data in a single machine, out-of-core computation is required now and also in the future – Good File layout is important for out-of-core computation 12