Manipulating Lossless Video in the Compressed Domain William Thies 1, Steven Hall 2, Saman Amarasinghe 2 1 Microsoft Research India 2 Massachusetts Institute of Technology ACM Multimedia October 20, 2009

Processing in the Compressed Domain Multimedia archives are growing rapidly –Monsters vs. Aliens production 100 TB –Facebook photos 400 TB –YouTube 600 TB How to analyze or modify the data? Uncompress Process Recompress Compressed Input Compressed Output Process Compressed Input Compressed Output Compressed-domain transformation Typical practice lossless prior to distribution

Prior Work: Focus on Lossy Formats DCT-based spatial compression (JPEG, MPEG stills) –Resizing [Dugad & Ahuja 2001] [Mukherjee & Mitra 2002] –Edge detection [Shen & Sethi 1996] –Image segmentation [Feng & Jiang 2003] –Shearing and rotating inner blocks [Shen & Sethi 1998] –Linear combinations of pixels [Smith & Rowe 1996] DCT-based temporal compression (MPEG video) –Captioning [Nang, Kwon, & Hong 2000] –Reversal [Vasudev 1998] –Distortion detection [Dorai, Ratha, & Bolle 2000] –Transcoding [Acharya & Smith 1998] Almost no work on lossless formats –Transpose and rotation of black/white images [Shoji 1995; Misra et al. 1999] –Pattern matching in compressed text [Farach & Thorup 1998; Navarro 2003] –Modifying pitch and playback of audio [Levine 1998]

Prior Work: Focus on Lossy Formats DCT-based spatial compression (JPEG, MPEG stills) –Resizing [Dugad & Ahuja 2001] [Mukherjee & Mitra 2002] –Edge detection [Shen & Sethi 1996] –Image segmentation [Feng & Jiang 2003] –Shearing and rotating inner blocks [Shen & Sethi 1998] –Linear combinations of pixels [Smith & Rowe 1996] DCT-based temporal compression (MPEG video) –Captioning [Nang, Kwon, & Hong 2000] –Reversal [Vasudev 1998] –Distortion detection [Dorai, Ratha, & Bolle 2000] –Transcoding [Acharya & Smith 1998] Almost no work on lossless formats –Transpose and rotation of black/white images [Shoji 1995; Misra et al. 1999] –Pattern matching in compressed text [Farach & Thorup 1998; Navarro 2003] –Modifying pitch and playback of audio [Levine 1998] Our Focus: Regular Processing of LZ77-Compressed Data Streams Our Focus: Regular Processing of LZ77-Compressed Data Streams

Example oooolalala OOOOLALALA Output: Input: to lowercase

AALALL Example OOOOALALLA OOOOLALALA Input: Compressed Input: oooolalala Output:

A Example OOOOLALLA OOOOLALALA 42 Input: oooolalala Output: Compressed Input:

Example OOOOLA OOOOLALALA 42 oooolalala Output: Input: Compressed Input: “Repeat Token” CountDistance

Example OOOOLA OOOOLALALA 4213 oooolalala Output: Input: Compressed Input: CountDistance “Repeat Token”

Example OLA OOOOLALALA 4213 oooolalala Output: Input: Compressed Input: CountDistance “Repeat Token”

Example OLA ola OOOOLALALA Compressed Output: Compressed Input: oooolalala Output:

Example OLA ola Compressed Output: Compressed Input: Compressed Domain Transformation OOOOLALALAoooolalala Output: Input:

Example

Our Contributions Handle the general case –Produce and consume more than one data item –Split and join data streams Implement in a compiler –Programmer thinks in terms of uncompressed data –Compiler translates to work on compressed data –Relies on StreamIt programming language Evaluate on video processing tasks –12 videos in Apple Animation format –Adjust colors or overlay two videos –Speedups proportional to compression ratio (median 15x)

In This Talk StreamIt Language Compressed Domain Transformation Experimental Evaluation

void->void pipeline FMRadio(freq1 low, float freq2, int N) { add AtoD(); add FMDemod(); add splitjoin { split duplicate; for (int i=0; i<N; i++) { add pipeline { add LowPassFilter(freq1 + i*(freq2-freq1)/N); add HighPassFilter(freq2 + i*(freq2-freq1)/N); } join roundrobin(); } add Adder(); add Speaker(); } Adder Speaker AtoD FMDemod LPF 1 Duplicate RoundRobin LPF 2 LPF 3 HPF 1 HPF 2 HPF 3 The StreamIt Language

Adder Speaker AtoD FMDemod LPF 1 Duplicate RoundRobin LPF 2 LPF 3 HPF 1 HPF 2 HPF 3 Applications –DES and Serpent [PLDI 05] –MPEG-2 [IPDPS 06] –SAR, DSP benchmarks, JPEG, … Programmability –StreamIt Language (CC 02) –Teleport Messaging (PPOPP 05) –Programming Environment in Eclipse (P-PHEC 05) Domain Specific Optimizations –Linear Analysis and Optimization (PLDI 03) –Optimizations for bit streaming (PLDI 05) –Linear State Space Analysis (CASES 05) Architecture Specific Optimizations –Compiling for Communication-Exposed Architectures (ASPLOS 02 & 06, dasCMP 07) –Phased Scheduling (LCTES 03) –Cache Aware Optimization (LCTES 05) –Load-Balanced Rendering (Graphics Hardware 05) Migrating Legacy Code to a Stream Representation –Using a Dynamic Analysis (MICRO 07) The StreamIt Language

Language Primitives FilterSplitterJoiner Filter pop 2 push 1 roundrobin(1,1) roundrobin(2,2) pop N push M roundrobin(N,M) Model of computation also known as cyclo-static dataflow

Example: Video Compositing roundrobin(1,1) Source 1 Source 2 Output MultiplyPixels 2 1

In This Talk StreamIt Language Compressed Domain Transformation Experimental Evaluation

Transforming Windows of Data OO–OOLA–L––LA– A LALAAOOOOL OO–OOLA–L––LA– A LALAAOOOOL Hyphenate Pairs Input: Output:

Transforming Windows of Data OO–OOLA–L––LA– A LALAAOOOOL OO–OOLA–L––LA– A LALAAOOOOL Hyphenate Pairs Input: Output:

Transforming Windows of Data OO–OOLA–L––LA– A LALAAOOOOL AOL 2431 L–A 36 Output: Compressed Input: Compressed Output:

Transforming Windows of Data OO–OOLA–L––LA– A LALAAOOOOL AOL 2431 L–A 36 Output: Compressed Input: Compressed Output:

Transforming Windows of Data OO–OOLA–L––LA– A LALAAOOOOL AOL 2431 OOL––A 36 AOOL Output: Coarsened, Expanded Compressed Input: Compressed Output:

General Case: Filters DN … … Filter I O D’O/ I N’’O/ I … … D’N’ …..… Filter I O I O … N’ % I items CoarsenTranslate D’ = LCM (D, I ) N’ = N – (D’ – D) N’’ = N’ – N % I

Compressed Input: Compressed Output: Splitting Streams LALAALALAL 1 1 ALALALALAL Input: Output:

Compressed Input: Splitting Streams LALAALALAL 2 2 AL 2 2 Input: Output:

Coarsened, Expanded Input: Compressed Output: Splitting Streams LAALALALAL

1 1 O X X O OOO Splitting and Joining: Transpose OOO OOO 4 4 OOO

O X 1 1 OOO OOO 4 4 O XOOO OO O

O X X O OOO OOO OOO OOO

XO O XO 31 O OO OOO

O O O 2 4 X O XO 12 O XO O 31

General Case: Joiners D1N1 … … W1 W2 D2N2 … … If D1 % W1 = 0 and D2 % W2 = 0 and D1/W1 = D2/W2 D1(W1+W2) N’ … … W1

In This Talk StreamIt Language Compressed Domain Transformation Experimental Evaluation

Implementation Implemented subset of transformations in StreamIt –User can change graph connectivity + filter functions Supported file format: Apple Animation (part of.MOV) –Standard format for interchange of lossless video –Compression: Run-length encoding within a line + difference encoding between frames Emit executable plugins for MEncoder and Blender –Allows integration with standard video editing workflow to-1 filter 1-to-1 joiner with 2-to-1 filter 1 1

Experimental Methodology Evaluated on 12 videos drawn from Internet video, computer animation, and stock digital television content Two classes of transformations: 1. Color adjustment: inverse, brightness, contrast 2. Composite transformations: alpha-under, multiply + = x= alpha under

Results: Execution Time Color Adjustment: - 2.5x to 471x (median 17x) Compositing: - 1.1x to 32x (median 6.6x) Compression Factor Following Re-compression Compression factor was low (≤1.1x) for one of source videos

Masked out areas not re-compressed Saturated colors not re-compressed Compression Factor Following Re-compression Results: File Bloat

Opportunity: Ignoring “Dead” Data Some pixels in composite frames do not depend on both input frames –Example: digital television mask (a low-performance case) If two data streams are multiplied, and one of them is repeatedly zero, then the repeat can be copied to the output (regardless of the values in the other stream) –We expect this would fix performance of our outlier cases –Requires pattern matching on stream graph x=

Extension to Other File Formats High-efficiency mappings –Flic Video –Microsoft RLE –Targa (with run-length encoding) Medium-efficiency mappings –Open EXR –Planar RGB  Re-arranges data by color or by byte Low-efficiency mappings –ZIP –GZIP –PNG  Performs Huffman coding prior to LZ77

Conclusions New method for direct processing of lossless-encoded data streams –Relies on LZ77 compression and stream programming model –Supports operations on windows of data –Supports splitting, joining, and reordering data Preliminary implementation in an automatic compiler –Write program on uncompressed data, run on compressed data Good speedups in the context of video processing –15x speedup (median) on color adjustment and compositing –Across 12 videos in Apple Animation format –May prove useful as more content authored in lossless formats Scope for extending technique, finding new applications

Extra Slides

General Case: Splitters DN … … Split U D’V U+V N’’V U+V … … D’N’ …..… Split … N’ % (U+V) items CoarsenTranslate D’ = LCM (D, U+V) N’ = N – (D’ – D) N’’ = N’ – N % (U+V) V U V U V