James A. Edwards, Uzi Vishkin University of Maryland

Introduction Lossless data compression Common tool  better use of memory (e.g., disk space) and network bandwidth. Burrows-Wheeler (BW) compression e.g., bzip2 Relatively higher compression ratio (pro) but slower (con) Snappy (Google) lower compression ratios but fast. Example For MPI on large machines speed is critical. Our motivation fast and high compression ratio Unexpected Prior work unknown to us made empirical follow-up … stronger Assumption throughout: fixed constant-size alphabet

State of the field Irregular algorithms: prevalent in CS curriculum and daily work (open-ended problems/programs). Yet, very limited support on today’s parallel hardware. Even more limited with strong scaling Low support for irregular parallel code in HW  SW developers limit themselves to regular algorithms  HW benchmarks  optimize HW for regular code … Namely, parallel data compression is of general interest as an undeniable application representing a big underrepresented “application crowd”

“Truly Parallel” BW compression Existing parallel approach: break input into blocks, compress blocks in parallel Practical drawback: good compression & speed only with large input Theory drawback: not really parallel Truly parallel: compress entire input using a parallel algorithm Works for both large and small inputs Can be combined with block-based approach Applications of small inputs: Faster (decompression) & greater compression  better use of main memory [ISCA05] & cache [ISCA12] Warehouse-scale computers. Bandwidth between various pairs of nodes can be extremely different; for MPI, MapReduce low bandwidth between pairs debilitating [HP 5th ed.] (i.e., Snappy was a solution)

Attempts at truly parallel BW compression A 2011 survey paper [Eirola] stipulates that parallelizing BW could hardly work on GPGPU, and decompression would fall behind further. Portions require “very random memory accessing” “…it seems unlikely that efficient Huffman-tree GPGPU algorithms will be possible.” The best GPGPU result: even more painful In 2012, Patel et al. concurrently attempted to develop parallel code for BW compression on GPUs but their best result was 2.8X slowdown. Patel reported separately 1.2X speedup for decompression (hence, not referenced in SPAA13 version.)

Stages of BW compression & decompression Block-Sorting Transform (BST) Move-to- Front (MTF) encoding Huffman encoding Inverse Block-Sorting Transform (IBST) Move-to- Front (MTF) decoding Huffman decoding Compression Decompression S S S BST S MTF S BW S BST S MTF

Inverse Block-Sorting Transform Serial algorithm: 1. Sort characters of S BST ; the sorted order T[i] forms a ring i → T[i] 2. Starting with $, traverse the ring to recover S Parallel algorithm: 1. Use parallel integer sorting to find T[i] 2. Use parallel list ranking to traverse the ring Both steps require O(log n) time and O(n) work On current parallel HW list ranking gets you – why we chose this step (END) banana$ S (read right to left) i S BST [i]annb$aa T[i]T[i] i rank[i] S BST [i] Linked ring i → T[i]

Conclusion and where to go from here? Despite being originally described as a serial algorithm, BW compression can be accomplished by a parallel algorithm. Material for a few good exercises on prefix sum & list ranking? For a more detailed description of our algorithm, see reference [4] in our brief announcement. This algorithm demonstrates the importance of parallel primitives such as prefix sums and list ranking. Requires support of fine-grained, irregular parallelism and sometimes also strong scaling  Issues on all current parallel hardware. Indeed: While recent work from UC Davis (2012) on parallel BW compression on GPUs that we missed taxed ~20% of our originality (same Step 2), It failed to achieve any speedup on compression. Instead a slowdown of 2.8x. For decompression: 1.2X speedup. On the UMD experimental Explicit Multi-Threading (XMT) architecture, we achieved speedups of 25x for compression and 13x for decompression [5]. On balance UC Davis paper huge gift: 70x vs. GPU for compression and 11X for decompression.

Where to go from here? Remaining options for the community Figure out how to do it on current HW Or, bash PRAM Or, the alternative we pursued Develop a parallel algorithm that will work well on buildable HW designed to support the best- established parallel algorithmic theory Final thought connecting to several other SPAA presentations This is an example where MPI on large systems works in tandem with PRAM-like support on small systems. Intra-node (of a large system) use PRAM compression & decompression algorithms for inter-node MPI messages Counter-argument to an often unstated position. That we need the same parallel programming model at very large and small scales

References [4] J. A. Edwards and U. Vishkin. Parallel algorithms for Burrows-Wheeler compression and decompression. TR, UMD, [5] J. A. Edwards and U. Vishkin. Empirical speedup study of truly parallel data compression. TR, UMD,

Block-Sorting Transform (BST) Goal: bring occurrences of characters together Serial algorithm: 1. Form a list of all rotations of the input string 2. Sort the list lexicographically 3. Take the last column of the list as output Equivalent to sorting the suffixes of the input string banana$ anana$b nana$ba ana$ban na$bana a$banan $banana a$banan ana$ban anana$b banana$ na$bana nana$ba banana$ Input to BST List of rotations annb$aa Output of BST Sort

Block-Sorting Transform (BST) Parallel algorithm: 1. Find the suffix tree of S (O(log 2 n) time, O(n) work)) 2. Find the suffix array SA of S by traversing the suffix tree (Euler tour technique: O(log n) time, O(n) work) 3. Permute characters according to SA (O(1) time, O(n) work) $ a $ na banana$ $ na$ na na$ $ i S[i]S[i]banana$ SA[i] S[SA[i]-1]annb$aa

Move-to-Front (MTF) encoding Goal: Assign low codes to repeated characters Serial algorithm: Maintain list of characters in order last seen Parallel algorithm: use prefix sums to compute the MTF list for each character (O(log n) time, O(n) work) Associative binary operator: X + Y = Y concat (X – Y) LiLi 130 j L0[j]L0[j] 0$ 1a 2b 3n i2103 S BST [i]annb 3 S MTF [i] j 0a 1$ 2b 3n L1[j]L1[j] j 0n 1a 2$ 3b L2[j]L2[j] j 0n 1a 2$ 3b L3[j]L3[j] a,$,b,n nba$ b,n$,an,ab,na,$a $,a,b,nb,n,aa,$ b,n,a,$a,$ assumed prefixS BST annb$aa

Move-to-Front (MTF) decoding Same algorithm as encoding, with the following changes Serial: The MTF lists are used in reverse Parallel: Instead of combining MTF lists, combine permutation functions S MTF Permutation function =

Huffman Encoding Goal: Assign shorter bit strings to more-frequent MTF codes The parallelization of this step is already well known Serial algorithm: 1. Count frequencies of characters 2. Build Huffman table based on frequencies 3. Encode characters using the table Parallel algorithm: 1. Use integer sorting to count frequencies (O(log n) time, O(n) work) 2. Build Huffman table using the (standard, heap-based) serial algorithm (O(1) time and work) 3. (a) Compute the prefix sums of the code lengths to determine where in the output to write the code for each character (O(log n) time, O(n) work) (b) Actually write the output (O(1) time, O(n) work)

Huffman Decoding Serial algorithm: Read through compressed data, decoding one character at a time Parallel algorithm: partition input and apply serial algorithm to each partition Problem: Decoding cannot start in the middle of the codeword for a character Solution: Identify a set of valid starting bits using prefix sums (O(log n) time, O(n) work)

Huffman Decoding How to identify valid starting positions: Divide the input string into partitions of length l (the length of the longest Huffman codeword) 1. Assign a processor to each bit in the input. Processor i decodes the compressed input starting at index i and stops when it crosses a partition boundary, recording the index where it stopped. (O(1) time, O(n) work) Now each partition has l pointers entering it, all of which originate from the immediately preceding partition. 2. Use prefix sums to merge consecutive pointers. (O(log n) time, O(n) work) Now each partition still has l pointers entering it, but they all originate from the first partition. 3. For each bit in the input, mark it as a valid starting position if and only if the pointer that points to that bit originates from the first bit (index 0) of the first partition (O(1) time, O(n) work)

Lossless data compression on GPGPU architectures (2011) Inverse BST: “Problems would possibly arise from poor GPU performance of the very random memory accessing caused by the scattering of characters throughout the string.” MTF decoding: “Speeding up decoding on GPGPU platforms might be more challenging since the character lookup is already constant time on serial implementations, and starting decoding from multiple places is difficult since the state of the stack is not known at the other places.” Huffman decoding: “Here again, decompression is harder. This is due to the fact that the decoder doesn’t know where one codeword ends and another begins before it has decoded the whole prior input.” “As for the codeword tables for the VLE, it seems unlikely that efficient Huffman-tree GPGPU algorithms will be possible.”