Final Project: Video Transcoding on Cloud Environments Queenie Wong CMPT 880

Introduction  Cloud computing technology has become mature and accessible to the public  Many complex and high computational operations can be distributed and processed on cloud environments  Complex video transcoding operations can be distributed to available nodes for paralleling processing – The encoding time of a MJEPG file was reduced from 7.5 hours to 2 minutes by scaling up to 6 nodes Queenie Wong2

Problem Statements  Excess Key-frame problem: Inappropriate splitting position on the original video file can create excess key- frames in the final output  How to perform video transcoding on Hadoop – Do not have native video transcoding library  Find out the right level of parallelism and split size for maps tasks – Too small split size: synchronization overheads – Too large split size: lack of dynamic load balancing Queenie Wong3

Proposed Solutions  Excess Key-frame problem: – Solution: MKVmerge program  Video transcoding on Hadoop – Solution: FFmpeg transcoding tool  The right level of parallelism and split size for maps tasks – Solutions:  Performance indicator to measure the efficiency of parallelism level  Video Transcoding Performance Test to find the optimal split size Queenie Wong4

Video Transcoding Performance Test  Studies suggested: – The right level of parallelism for maps seems to be around maps/node – The size of each map task is roughly 16MB to 64MB  Performance tests against different parallelism levels and split size to find out the ideal set up for video transcoding  Implementation: – Apache Hadoop, FFmpeg for video transcoding and MKVmerge for video splitting at key frames boundary Queenie Wong5

Parallel Nodes Test Results Queenie Wong6

Threshold for Adding Extra Nodes  Threshold: 5% of initial processing time (1 node) Reduction: (initial time – current time)/initial time Efficiency of parallelism level: (previous time – current time)/initial time ( ) / = 3.7 % Queenie Wong7

Split Size Test Results Queenie Wong8

Challenges  Tuning programs for files with different size  Get familiar with FFmpeg for video transcoding operations  Repositioning to key frame boundary  Measurement variance caused by I/O operations  Unstable software problems  Learn, setup, use and debug Hadoop and MapReduce program within a short timeframe Queenie Wong9

Conclusion  Video transcoding performance has been diminished when the system overhead exceed the benefit of parallel processing  Efficiency indicator is proposed to measure the efficiency of the level of parallelism  The optimal split sizes for transcoding is 64MB for files with common sizes  Enforce load balancing on every node in order to maximize the benefit of paralleling processing Queenie Wong10

References [1] R. Pereira, M. Azambuja, K. Breitman, and M. Endler, “An architecture for distributed high performance video processing in the cloud,” 2010 IEEE 3rd International Conference on Cloud Computing, pp. 482–489, [2] “mkvtoolnix - matroska tools for linux/unix and windows.” [3] “Ffmpeg.” [Online]. Available: [4] R. Schmidt and M. Rella, An approach for processing large and non-uniform media objects on mapreduce-based clusters. Springer Berlin Heidelberg, [5] “Yahoo! hadoop tutorial,” [Online]. Available: [6] J. Lin and C. Dyer, Data-Intensive Text Processing with MapReduce. Morgan & Claypool, [7] J. Dean and S. Ghemawat, “Mapreduce: simplified data processing on large clusters,” Comunications of the ACM, vol. 51, no. 1, pp. 107–113, Queenie Wong11

References [8] Z. Y, C. Wang, C.Thomborson, J. Wang, S. Lian, and A. Vasilakos, “Multimedia applications and security in mapreduce: opportunities and challenges,” Concurrency and Computation: Practice and Experience, vol. 17, no. 24, pp. 2083–2101, [9] A.Garcia, H.Kalva,and B.Furht,“Astudy of transcoding on cloud environments for video contentdelivery,” inProceedings of the 2010 ACM multimedia workshop on Mobile cloud media computing. ACM, 2010, pp. 13–18. [10] C. Yan, X. Yang, Z. Yu, M. Li, and X. Li, “Incmr: Incremental data processing based on mapreduce,” in 2012 IEEE 5th International Conference on Cloud Computing (CLOUD). IEEE, 2012, pp. 534–541. [11] N. S. Chahal and B. S. Khehra, “A comparative study for optimization of video file compression in cloud environment,” International Journal of Computer Applications, vol. 60, no. 13, pp. 27–30, [12] Yzyero. Hadoop + ffmpeg on mapreduce. [Online]. Available: [13] “Apache hadoop.” [Online]. Available: Queenie Wong12