Video Mosaics CS 294/283 Final Project Steve Martin U.C. Berkeley, Fall 2003

Overview Previous Work Method Overview Metrics for Comparing Video Preliminary Results Future Work Questions?

Previous Work on Mosaics Image Mosaics –1970s: Chuck Close/Salvadore Dali make image mosaics by hand –1990s: Papers at SIGGRAPH and commercial programs create image mosaics Video Mosaics –2002: Video Mosaic program created at Princeton, work published at NPAR 2002 Example... –2003: Wells Fargo commercial (done by hand) Example...

Method Overview Step 1: Get a LOT of source video –Quality of results highly dependant on quality and amount of component clips! –In our case, problem solved! Got stock video companies donate video to the war effort We have 200GB of high quality video to play with (more than 3000 clips)! Step 2: Preprocess video –Create a database of feature values for all video. –Each clip further subdivided into tiles Size determined by the user. With 20 by 20 pixel tiles, each movie has 864 tiles. –Metrics are run on each tile, values recorded –This is done with a separate program that crawls the video repository

Method Overview (2) Step 3: Create video mosaic –Load in original video –Parse the previously generated database –For each clip in the original video Run the same metrics as was used to generate the database –Search the database for the closest tile –Replace the original tile with one from another movie, continue Note that we cant just brute force compare video or well be computing forever. –How to come up with numbers that describe tiles?

Video Comparison Metric 1: Average Color Want replacement clips to have close to or same average color as original tile. To calculate: –Just add color values of pixels in movie tile and divide. Actually not terrible by itself... –As tiles get smaller, approach pixel size, eye blends them However, note that average color can be fooled –i.e. a clip that is half red and half green averages to olive brown: Need something else *+ =

In addition to average color, want some kind of measure of color frequency. –Helps solve previous problem, although not entirely To calculate: –For each pixel of each frame of each tile in movie: Convert from RGB to HSB Take hue value and histogram –Create histograms on a per-tile basis Number of histogram buckets is changeable –Usually use around 10. Video Comparison Metric 2: Color Histograms Vs.

Video Comparison Metric 3: Edge Histograms Also would like to have some kind of edge comparison between original and replacement tile. To calculate: –Foreach frame of each tile –Convert frame to black and white Dont care about colorwant edge data –Resize the image down to 10 x 10 We want major edges only Use a bicubic resize to blur the image slightly –For each pixel in the new image, Calc intensity gradient in the x and y directions Take the arctangent and histogram. –Do for each frame, create histogram. Example of this metric at work: θ

Finding the Best Match Currently, a linear search through a large unordered flat file. –Parse this into a simple row/column data structure, keep it in memory. –Search not as bad as it sounds; takes about a second with entries. –Coming soon: some tricks to make this much better. For each tile of the source movie, compare with each tile entry in the database. –Get the absolute difference between average colors –Get the distance between color histograms –Get the distance between edge histograms Add difference values to get a value for how good a match this tile is for the original. –Can weight different metric to emphasize different features Choose entry that minimizes this.

Video vs. Photo Mosaics Why bother using video? –Could mosaic each frame individually, and then string the frames together. –Example...Example... Would like to exhibit some kind of temporal consistency –I.e. each tile is recognizable as a movie instead of a compression artifact. –Neat effect; less distracting for the eye.

Preliminary Results

Whats Next? Need to implement some way of dividing tile clips into arbitrary length. –Approximation: subdivide tiles into groups of frames. Index the rest of the video so I can leverage the full library. Still working on good ways to preserve temporal locality –i.e. place video tiles in a manner that makes sense over time in addition to just per frame Include metrics that look at neighboring tiles to avoid drastically different edges –These just end up looking like compression artifacts –Princetons video has this problem Different sized tiles –I think a patchwork effect would look interesting, albeit harder to compute. Optimize, optimize, optimize! –Takes forever to make one of these!

References Klien, Grant, Finkelstein, and Cohen. Video Mosaics Proceedings of NPAR Microsoft Video For Windows developers site Apple Quicktime SDK developer resources James OBrien, Alexander Berg, Charless Fowlkes Questions? Comments?