1. Moving Pictures Implementing Video on Flickr Cal Henderson

2. Web 2.0 Expo NYC, 17th September 20082 Hello

3. Web 2.0 Expo NYC, 17th September 20083 Flickr Large scale kitten sharing website Started 2003, launched 2004 –5 years old this december Almost 3 billion photos

4. Web 2.0 Expo NYC, 17th September 20084 Enter: Video Video was added this year –Launched April 2008 Many hundreds of thousands of videos uploaded Many millions of playbacks

5. Web 2.0 Expo NYC, 17th September 20085

6. 6

7. 7

8. 8

9. 9

10. 10 Video? Thats just like photos! -Me, before Flickr Video

11. Web 2.0 Expo NYC, 17th September 200811 12 4 Steps 4 main tasks –Uploading –Transcoding –Storage –Serving & Playback

12. Web 2.0 Expo NYC, 17th September 200812 1. Upload

13. Web 2.0 Expo NYC, 17th September 200813 Simple upload Web forms –Just like any other file But files are large / huge

14. Web 2.0 Expo NYC, 17th September 200814 Issues Two components for uploading: –Sending from the client –Receiving on the server Both of these present problems

15. Web 2.0 Expo NYC, 17th September 200815 Sending from the client Multiple options –Simple form –Flash –Desktop app

16. Web 2.0 Expo NYC, 17th September 200816 Simple Forms Pros –Very easy to implement –Works on every browser out of the box Cons –Upload progress is harder –Slow –Select a single file at once

17. Web 2.0 Expo NYC, 17th September 200817 Flash Pros –Upload progress is easy –Fast –Multi select of files Cons –Harder to implement –Flash isnt quite ubiquitous

18. Web 2.0 Expo NYC, 17th September 200818 Desktop App Pros –Upload progress is easy –Very fast –Multi select of files –Drag and drop Cons –Hard to develop –Hard to deploy (relative to the web)

19. Web 2.0 Expo NYC, 17th September 200819 Making Progress Upload progress –Not impossible with plain forms –Just need to be able to query the upload progress via AJAX Multiple machines –The VIP issue –Enter Perlbal

20. Web 2.0 Expo NYC, 17th September 200820 Making Progress Browser Web 1 Web 2 1. Browser starts upload 2. Web server broadcasts progress via UDP 3. Browser queries progress via AJX call Load balancer

21. Web 2.0 Expo NYC, 17th September 200821 Receiving on the server File uploads are slow –Much slower than serving pages Apache processes are heavy –Waste of resources Use a poll based server (like jetty)

22. Web 2.0 Expo NYC, 17th September 200822 Receiving on the server Or, use a buffering layer –Perlbal is great for this Or a lightweight Apache –E.g. w/ mod_proxy BrowserBufferServer SlowFast

23. Web 2.0 Expo NYC, 17th September 200823 But wait Its not just the first step thats slow Moving files around between servers is slow –Do it out of band Asynchronous jobs are in order anyway –Do it!

24. Web 2.0 Expo NYC, 17th September 200824 2. Transcode

25. Web 2.0 Expo NYC, 17th September 200825 Transcode? Why transcode at all? Input comes from many sources –Point & shoots –DV Cams –Mobile devices –Video editing software All in different formats

26. Web 2.0 Expo NYC, 17th September 200826 So many formats But very few of these formats can be played back cross platform –Without special software or hardware Formats are designed to do one thing well –They dont always manage even that Transcoding puts all videos on an equal footing

27. Web 2.0 Expo NYC, 17th September 200827 Video file basics Most file types are really just containers –MOV, FLV, AVI The data inside can be in multiple formats –We call these codecs (encoder + decoder) Files contains multiple streams –Both audio and video

28. Web 2.0 Expo NYC, 17th September 200828 Interleave Audio and video are often interleaved –Hence AVI A video file looks like this: –Headers –Video chunk –Audio chunk –Video chunk –Audio chunk –Etc

29. Web 2.0 Expo NYC, 17th September 200829 Compress Raw video files are huge –A bitmap for every frame –Rarely used, even in post production At 30 fps, that gets crazy pretty quickly We dont need to store every frame –Static backgrounds dont change (much) between frames

30. Web 2.0 Expo NYC, 17th September 200830 Compresssss Intraframe –Treat each frame as a picture –Compress it (just like JPEG) –DCT (Discrete Cosine Transform) Interframe –Store the differences between frames –Treat the pixels as a 3D array to be compressed

31. Web 2.0 Expo NYC, 17th September 200831 The IPB Three frame types: I, P & B Intra coded pictures –A full raw frame Predicted pictures –Based on a single reference frame Bi-predictive pictures –Based on two or more reference frames

32. Web 2.0 Expo NYC, 17th September 200832 IPBIPBIPBIPB Reference frames may be I, P or B P & B frames may contain a mix of image data and motion vector displacements I frames require the most bits –Then P frames –Then B frames

33. Web 2.0 Expo NYC, 17th September 200833 Bad terminology We should really say picture –(Not frame) –Because of interlacing Really, we encode fields not frames –Picture is the general term And H.264 contains slices –Sub-regions of the field –Macroblocks & Artifacts

34. Web 2.0 Expo NYC, 17th September 200834 I-Frames Also called Key Frames Allow easy random seeking Twice a second for Digital TV & DVDs More widely spaced online

35. Web 2.0 Expo NYC, 17th September 200835 Seekable

36. Web 2.0 Expo NYC, 17th September 200836 Seekable

37. Web 2.0 Expo NYC, 17th September 200837 Oh, and audio too We can worry less about this –Older problem, well solved MP3 is pretty good –Who cares how it works? Syncing is the only issue –Presentation Time Stamps (PTS) and Decode Time Stamps (DTS) in MPEG-2

38. Web 2.0 Expo NYC, 17th September 200838 Flash! Woah-oh! The big question: Flash?

39. Web 2.0 Expo NYC, 17th September 200839 Non-flash sites QuickTime Windows Media This is gradually disappearing Flash player is ubiquitous Compression is good enough Interactive too But no 3D/VR as with QuickTime :(

40. Web 2.0 Expo NYC, 17th September 200840 The Flash Player Flash Player 6 –March 2002 Video: Sorenson Spark (H.263) Audio: MP3 –Or ADPCM / Uncompressed –Or Nellymoser Asao

41. Web 2.0 Expo NYC, 17th September 200841 Second Generation Flash Player 7 –August 2005 Video: On2 TrueMotion VP6 Audio: MP3

42. Web 2.0 Expo NYC, 17th September 200842 The hot shit Flash Player 9 (update 3) –December 2007 Video: H.264 (MPEG-4 Part 10) –w/ container formats from MPEG-4 Part 14 Audio: AAC (MPEG-4 Part 3) Plus 3GPP Timed Text (MPEG-4 Part 17)

43. Web 2.0 Expo NYC, 17th September 200843 TrueMotion VP6 Proprietary Reasonable compression Created by On2 –Patented –Probably illegal for GPL code YouTube uses it for lower quality and old streams

44. Web 2.0 Expo NYC, 17th September 200844 H.264 Not proprietary Good compression MPEG Standard –Open, but patented –Patent licenses from the MPEG LA –Unclear how this applies to (L)GPL code –But probably badly YouTube using it for higher quality streams iPhones and AppleTV

45. Web 2.0 Expo NYC, 17th September 200845 Software Open source transcode tools FFmepg –libavcodec for VP6 –Probably illegal – dubious –Also pretty shoddy –Can only decode H.264

46. Web 2.0 Expo NYC, 17th September 200846 More software MEncoder –libmpcodecs uses libavcodec VLC –libvlc uses libavcodec So basically the same –Different muxing, same codecs

47. Web 2.0 Expo NYC, 17th September 200847 Free H.264? Unfortunately, not really x264 is the only usable one –Its pretty good –MEncoder can use it –Still limited in options at this point –Again, dubiously legal

48. Web 2.0 Expo NYC, 17th September 200848 Non-free tools Flash encoder –Not automatable On2 FlixEngine –Creators of VP6 –Windows or Linux –Some support for H.264 Rhozet Carbon Coder –The new hot shit –Good H.264 support –Windows

49. Web 2.0 Expo NYC, 17th September 200849 Choices Video codec Resolution Bitrate (VBR, CBR) Keyframes Audio codec –Channels –Bit depth –Sampling rate

50. Web 2.0 Expo NYC, 17th September 200850 Doing it at scale Not really a problem Very easily parallelizable Amazon EC2 is awesome here –Exactly what it was design for –Grow/shrink as needed –But, per-CPU software licensing

51. Web 2.0 Expo NYC, 17th September 200851 3. Store

52. Web 2.0 Expo NYC, 17th September 200852 Easy! Really, just like photos –But with bigger files Same disk layout as any other serving But the serving part is slower

53. Web 2.0 Expo NYC, 17th September 200853 But.. Remember the files are huge Operations take time –More likely to fail halfway through –Checksums are your friend Do it all asynchronously

54. Web 2.0 Expo NYC, 17th September 200854 4. Serve & Playback

55. Web 2.0 Expo NYC, 17th September 200855 The choice Streaming vs Progressive

56. Web 2.0 Expo NYC, 17th September 200856 Streaming Pros –Easily seekable –Live feeds Cons –Special server software –Slower to start –Firewall troubles

57. Web 2.0 Expo NYC, 17th September 200857 Progressive download Pros –Just use a web server –Play offline –Firewall/proxy friendly Cons –Harder to seek ahead (but not impossible)

58. Web 2.0 Expo NYC, 17th September 200858 Streaming tech Non flash stuff –Well ignore that –Youre using flash, right? RTMP –Real-time Messaging Protocol –Proprietary (thanks Adobe!)

59. Web 2.0 Expo NYC, 17th September 200859 RTMP RTMP - Raw TCP socket stuffs RTMPT – RTMP tunneled over HHTP –For firewalls, etc Flash Media Server –previously Flash Communication Server Wowza Pro –$1000/server

60. Web 2.0 Expo NYC, 17th September 200860 Open source Its not all bad Red5 –Java implementation of RTMP server –Mostly feature complete –Beta quality, but usable in production –Facebook

61. Web 2.0 Expo NYC, 17th September 200861 Progressive Used by the majority of large sites Very simple! Seekable with server support

62. Web 2.0 Expo NYC, 17th September 200862 Seeking Serve the FLV starting at a different point Just add a simple FLV preamble before seeking into the file Simple to do in PHP, Perl, etc mod_flvx for Apache mod_secdownload for lighttpd

63. Web 2.0 Expo NYC, 17th September 200863 5. Other considerations

64. Web 2.0 Expo NYC, 17th September 200864 Review Videos are slow Expensive to review Review grids –Doesnt cover audio

65. Web 2.0 Expo NYC, 17th September 200865

66. Web 2.0 Expo NYC, 17th September 200866 Not enough? Social tools are useful here

67. Web 2.0 Expo NYC, 17th September 200867 Summary

68. Web 2.0 Expo NYC, 17th September 200868 Summing up Flash makes sense –For uploading too H.264 is probably your best bet today Transcoding software still costs money –Unless youre willing to take on the risk Progressive download is basically awesome

69. Web 2.0 Expo NYC, 17th September 200869 The end!

70. Web 2.0 Expo NYC, 17th September 200870 Awesome! These slides are available online: iamcal.com/talks/