Creating Music Videos using Automated Media Analysis Authored by Jonathan Foote, Matthew Cooper, and Andreas Girgensohn Presented by Sukhyung Shin, Ninad Dewal

One neat usage… Home videos are LONG –… AND generally have poor quality video & audio Video has fast motion Video has moments of extreme brightness –Too tedious to watch –Too precious to throw away Solution: –Automatic Music Video Creation

Key Guidelines to Keep in Mind Soundtrack quality  video quality –You think the video is better Synchronization helps both –Enhanced perception of quality Users choose clips –Fully automated not optimal –Need mix of both

What they did, in a Nutshell Automatic/Semi-automatic creation: –Source video –Arbitrary audio soundtrack Video clips aligned w/ audio changes –Audio: looked for tempo –Video: looked for unsuitability High level of synchronization

Audio Parameterization Self-similarity (SS) analysis –Independent of type of music –Past and future regions –Novel point between high SS regions –Standard spectral parameterization: Based on STFT (short term Fourier transform) Sampled at 22 kHz, quantized into 30 bins

Audio Self-Similarity Analysis Parameterized  2D representation Key = Dis-similarity measurement (cosine) –Can yield large scores for low magnitude vectors –Similarity Matrix S –Serves as visualization of audio file structure High similarity: bright

Not similar regions: darker Look for regions of: Low cross-similarity Then high self-similarity Compare with to obtain novelty N(i) for frame i:

Segmenting and Editing Video Video boundaries into takes and clips Discarding Unsuitable Video –Excessive camera motion or poor exposure –Unsuitability score First estimate camera speed and direction Compare this estimate vs. current camera motion Test exposure/brightness Discard clips with score > 0.5

Aligning Audio and Video So far, you have: –Peaks from audio –Clips from video boundaries Simple solution: –Rank audio peaks and match w/ video boundaries –Assuming: video longer than audio (what if not?) –Clip video clips even further if too big Assuming: High suitability score w/ audio region –Focus on audio segmenting; video usually poor –For fully automated: Algorithms used: sort, DP

User Control –Hitchcock System:

More Uses… Home Videos  Music Videos –Precious but tedious Music artists –MTV, VH1 Movie, TV Show, Anime Fans –Creating free MV as hobbies

Improvements Rhythmic synchronization –Distinctive tempo or beat Combining source and soundtrack audio –Has issues with edit boundaries

Conclusions Preliminary studies had positive outlook –Users could interact w/ Hitchcock Authors realized that… –…source video’s audio should be used Hitchcock interface combined w/ automated ordering worked well.