1. SC 2013 SDQuery DSI: Integrating Data Management Support with a Wide Area Data Transfer Protocol Yu Su*, Yi Wang*, Gagan Agrawal*, Rajkumar Kettimuthu # *The Ohio State University # The University of Chicago and Argonne National Laboratory

2. SC 2013 Motivation Science becomes increasingly data driven Strong requirements for efficient data analysis “Big Data” Challenge: –Fast data generation speed –Slow disk I/O and network speed –Some number from road-runner EC 3 simulation 4000 3 particles, 36 bytes per particle => 2.3 TB Network Bandwidth: GB level or even less Huge difference between simulation and network Gap will become bigger in future

3. SC 2013 Wide-Area Data Transfer Protocols Efficient data transfers over wide-area network Globus GridFTP: –Striped, Streaming, Parallel Data Transfer –Reliable and Restartable Data Transfer Limitation: volume? –The basic data transfer unit is file (GB or TB Level) –Strong requirements for transferring data subsets Climate Simulation, Tomography, XPCS An Example Goal: Integrate core data management functionality with wide-area data transfer protocols

4. SC 2013 Challenges How should the method be designed to allow easy use and integration with existing GridFTP installation? How can users view a remote file and conveniently specify the subsets of data that is of interest to them? How to support efficient data retrieval with different subsetting scenarios (index-based retrieval or data block loading + in-memory filter)? How can data retrieval be parallelized and benefits from multi-steaming?

5. SC 2013 Introduction GridFTP SDQuery DSI (Scientific Data Query Data Storage Interface) –Efficient Data Transfer over Flexible File Subset –Dynamic Loading / Unloading –HDF5 and NetCDF Data Formats –Standard SQL Embedded in Data Download Request –Multiple Query Types (Dims, Coordinates, Values) Bitmap Indexing Metadata View of Data File Features: –Performance Model based Hybrid Data Reading –Parallel Streaming Data Reading and Transferring

6. SC 2013 Background: Globus GridFTP Support Efficient Data Transfer in Grid Community –3500+ server, 1PB+ transfer/day DSI(Data Storage Interface): –Compatible with different file systems or platforms –An adapter between GridFTP and system SDQuery DSI: –Dynamic loading with small overhead –Seamless integration with GridFTP data transfer features (Fault Tolerance, Security, Automatic TCP optimization)

7. SC 2013 Background: Bitmap Indexing Widely used in scientific data management Suitable for float value by binning small ranges Run Length Compression(WAH, BBC) –Compress bitvector based on continuous 0s or 1s

8. SC 2013 System Architecture GridFTP Client GridFTP Server HDF5, NetCDF Dataset Indices and schema File Receiver Index Generation Schema Management Query Analysis Index Operations Data Reader File Sender Request Parser SDQuery DSI File DSI File ReceiverFile Reader data store request schema request data retrieve request Receive Data File Build Multi-level Bitmap Indexing Generate Metadata View Query Metadata View Parse SQL query Indexing and find all data pos Read Data based on data pos Send File

9. SC 2013 Metadata View Physical Storage Descriptor TEMP = /tmp/server/POP.nc VVEL = /tmp/server/POP.nc …… Logical Layout Descriptor Varname = “TEMP” Data Type: NC_FLOAT Dims (time, depth, t_lat, t_lon) Coordinate Values: t_lon… …… Value Distribute Descriptor Min/Max Value: (-21.1, 33.1) Logical Layout Descriptor Varname = “VVEL” Data Type: NC_FLOAT Dims (time, depth, u_lat, u_lon) Coordinate Values: u_lon… …… Value Distribute Descriptor Min/Max Value: (-246, 225)

10. SC 2013 An User Case Translate Analysis Requirement into Query: –Find the data elements under the depth of 50 meters of the ocean and the temperature is larger than 5 centigrade. Client-side Request Examples globus-url-copy "ftp://127.0.0.1:5000/tmp/server/POP.nc" file:///tmp/client/netcdfsubset/ globus-url-copy "ftp://127.0.0.1:5000/tmp/server/POP.nc( SELECT TEMP FROM POP.nc WHERE TEMP >=5 AND depth>50)" file:///tmp/client/netcdfsubset/ POP.nc TEMP(Query).nc Less Than 5% Data Transfer!

11. SC 2013 Performance Model-based Data Subset Retrieval Data Retrieval Process: –Query Analysis and Index Operations - Fast –Know how much data to fetch after index operations: –Data Reader – Slow Data Reading Choices: –Direct Access: Smaller Data Subset Directly read data by points or segments from disk –Memory Filter: Bigger Data Subset Load the data blocks into memory and filter –Which method is more efficient to choose is tricky Execution Environment, Data Format and Dataset

12. SC 2013 Performance Model Profiling and formulate data reading –Memory Filter: –Direct Access (Points): –Direct Access(Segments): Offline Training based on random query set –Parameters are trained and classified based on subset percent –Apply formulas for each real query –Select more efficient methods for data reading

13. SC 2013 Parallel Streaming Multi-Thread Data Retrieval and Transfer: –Data retrievals are performed in parallel –Data transfers are performed in parallel to better utilize the bandwidth –Data retrievals and data transfers are performed in a pipeline mode Bit-1 distribution based data partition: –Partition result bitset based on thread number –Great load balance for both data retrieval and transfer –Small partition cost One pass for both bits segmenting and partition Use multi-thread to speedup

14. SC 2013 Parallel Streams Example (2 streams) 01110100 11000110 00110110 00011100 01000000 00010000 Subset Size: 12 Subset Size: 5 Load Imbalance Subset Size: 8 Subset Size: 9 Chunk 0 Chunk n Chunk 1 Chunk 0 Chunk n Chunk 1 S0S0 S1S1 S2S2 …SnSn S0S0 S1S1 S2S2 …SnSn Sending Queue 1 Sending Queue 2 TCP stream T11: waiting… T10: reading… T21: waiting… T20: reading… T11: sending… T21: sending… …… Dim-based PartitionBit1-based Partition One pass: Generate Segs and Count

15. SC 2013 Experiment Results Goals: –Compare SDQuery DSI with GridFTP default File DSI –Show the effectiveness of perform-model based selection between direct access and memory filter –Speedup for using parallel streaming data transfer Datasets: –NetCDF: Parallel Ocean Programs (POP) –HDF5: Mediterranean Ocean Data Base (MODB) Environment: –RI Cluster: 100 nodes, 8 cores 2.53 GHz Intel(R) Xeon Processors, 12 GB memory

16. SC 2013 SDQuery DSI vs. File DSI Compare the total execution time between two DSIs in different network environments File DSI (GridFTP default DSI): –Read the entire data file and transfer over network Dataset: –140 GB POP data file –TEMP.nc(time(10), depth(42), lat(2400), lon(3600)) Three Network Environment: –LAN: 1 Gb/s bandwidth, 0.17 msec RTT –WAN: Avg. 200 Mb/s bandwidth, 24 msec RTT –WAN: Avg. 20Mb/s bandwidth, 60 msec RTT

17. SC 2013 SDQuery vs. File DSI (1Gb) SDQuery Query Processing Time: Query parsing and bitmap indexing time SDQuery Subset and Transfer Time: Data subset fetching and transfer time File Read and Transfer Time: Entire data file reading and transfer time Data file: 140 GB Input of SDQuery DSI: 2000 queries cover different data subset percentage When the data subset percentage is <50%, SDQuery DSI is better, the speedup is 1.26 to 9.41 Otherwise: FileDSI achieves better efficiency

18. SC 2013 SDQuery vs. File DSI (200 Mb) SDQuery Query Processing Time: Query parsing and bitmap indexing time SDQuery Subset and Transfer Time: Data subset fetching and transfer time File Read and Transfer Time: Entire data file reading and transfer time Same data and same input Network transfer time becomes the main bottleneck SDQuery DSI: Query Process Time: 9% - 40% of Total Execution Time Compared to File DSI, SDQuery DSI achieves better efficiency for all cases. The speedup is from 1.15 to 29.07

19. SC 2013 SDQuery vs. File DSI (20 Mb) SDQuery Query Processing Time: Query parsing and bitmap indexing time SDQuery Subset and Transfer Time: Data subset fetching and transfer time File Read and Transfer Time: Entire data file reading and transfer time In a common wide area network environment where bandwidth is really limited. Network transfer time becomes the dominant factor SDQuery DSI: Query Process Time: 1% - 9% of Total Execution Time SDQuery DSI achieves better efficiency for all cases. The speedup is from 1.21 to 81.32

20. SC 2013 Accuracy of Performance Model X axis: data subset percentage Y axis: only data subset reading time Direct Access, Memory Filter Data Access (points): frequent data seeking, inefficient Data Access (segments): average seg length: 300.36, speedup: 1.64 – 3.93 Memory Filter: Similar for all different cases Data Access (segments) and Memory Filter method achieve same performance when subset percentage is around 62% Hybrid Access: right choice in most case (except 60% - 70%)

21. SC 2013 Speedup Using Parallel Streaming X axis: data subset percentage Y axis: data retrieval and transfer time Non-overlapping: data is sent back only after all subset is loaded into memory Benefits: Parallel TCP Streams Parallel Data Retrieval Data Retrieval and transfer overlap Dataset: 10.5 GB MODB Network Speed: 200Mb/s 1 Steam allows the overlap between data retrieval and data transfer, the speedup is 1.19 – 1.52 compared with non overlapping Maximum speedup using 4 streams: 1.57 – 1.75 Bandwidth is fully utilized

22. SC 2013 Conclusion ‘‘Big Data’’ issue brings challenges for scientific data management SDQuery DSI: a GridFTP plug-in to support flexible data subsetting over HDF5 and NetCDF Seamless integration with GridFTP server Performance model based data retrieval method Parallel steaming data retrieval and transfer

23. SC 2013 Contact Us If You’re Interested! Yu Su Email: su1@cse.ohio-state.edu

24. SC 2013 Thanks 24

25. SC 2013 TEMP SALT UVEL VVEL Network I want to analyze TEMP within Indian Ocean! More Efficient! Entire Data File Data Subset Back POP.nc An Example of Ocean Simulation GridFTP Server