On the Use of Computable Features for Film Classification Zeeshan Rasheed,Yaser Sheikh Mubarak Shah IEEE TRANSCATION ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, JAN 2005

Outline Introduction Computable video features –Average Shot Length –Color Variance –Motion Content –Lighting Key Mean Shift Classification Results Conclusion

Introduction Films are a means of expression –Explicitly, with the delivery of lines by actors –Implicitly, with the background music, lighting, camera movements and so on Study domain is the movie preview, which often emphasizes the theme of a film

Introduction Maybe a need to extract the “genre” of scenes With scene-level classification, it would allow a more flexible system of scene ratings, ex filter and recommendation of movies

Related Work Work by Fischer et al. and Truong et al. distinguished between newscasts, cartoon, commercials, sports through decision tree with examples Kobla et.al used DCT coefficients, and motion vector information of MPEG video for indexing and retrieval

Computable video features Identify four major genres –Action, Comedy, Horror, and Drama –because most movies can be classified and low-level discriminant analysis is most likely to succeed Employ for features –Average Shot Length, Shot Motion Content, Lighting Key, Color Variance

Shot Detection and Average Shot Length First proposed by Vasconcelos Can direct audience’s attention with controling the tempo of the scene Ex. Dramas have larger average length, whereas action movies shorter shot length

Shot Detection and Average Shot Length Detection of shot boundaries using color histogram intersection in the HSV space –H: hue (color), 8 bins –S: saturation, 4 bins –V: value (brightness), 4 bins S(i) represent the intersection of histograms and of frames i and i-1

Shot Detection and Average Shot Length min bin1bin2bin1bin2 Frame iFrame i-1

Shot Detection and Average Shot Length min When S(i) is less than a fixed threshold shot boundaries !! bin1bin2bin1bin2 Frame iFrame i-1

Shot Detection and Average Shot Length 17 shots identified by a human observer Number of shots detected: 40; Correct: 15; False positive: 25; False negative: 2

Shot Detection and Average Shot Length To improve the accuracy, an iterative smoothing of the 1-D function is performed first Number of shots detected: 18; Correct: 16; False positive: 2; False negative: 1

Color Variance variance of color has a strong correlational structure with genres intuitively –For instance, comedies with a large variety of bright colors –whereas horror films with only darker hues Employ variance of CIE Luv –L: luminancy ( 發光度 ) –u,v: chrominancy ( 色差 )

Color Variance Generalized variance is obtain ps. All key frames presented in a preview are used to find this feature

Motion Content The visual disturbance of a scene can be represented as the motion content present Action films with higher value for such a measure, and dramatic or romantic movies with less visual disturbance

Motion Content Horizontal slice: I(x,t)

Motion Content Hx, Ht are the partial derivatives of I(x,t)

Lighting Key There are numerous ways to illuminate a scene, one of the common used is Three Point Lighting Keylight: The main source of light on the subject and it is the source of greatest illumination Backlight: Help emphasize the contour of the object, and it also separates it from a dark background Fill-light: Secondary illumination source which helps to soften some of the shadows thrown by the keylight and backlight

Lighting Key High-key lighting: –An abundance of bright light –More action, less dramatic –Ex. Comedy & action movies Low-key lighting: –Ex. Film noir or horror films

Lighting Key Many algorithms exist that compute the position of a light source in a given image Unfortunately, assumptions typically made in existing algorithms are violated, for example, single light source Compute the key of the lighting with brightness value of pixels

Lighting Key

Key frame i with m*n pixels, find the mean and standard deviation of the value component of the HSV space Lighting quantity –Horror movies with small value –Comedy movies with large value

Mean Shift Classification Mean shift procedure has been shown to have excellent properties for clustering and mode-detection with real data Xi: video features hi: their bandwidth parameters

Action + drama drama Comedy+ drama comedy Action+ comedy horror

Results Conduct 101 film previews obtained from the Apple website The total number of outliers in the final classification was 17 and 83% genre classification accurate

Conclusions Propose a method to perform genre classification of previews using low-level computable features Classification is performed using mean shift clustering in the 4-D feature space of average shot length, color variance, motion content, and the lighting key