Considerations for Video Scene Selection Margaret H. Pinson NTIA/ITS

Scene Complexity Should evenly span a wide range of coding difficulty Major impact to quality Why use an objective metric? Uneven distribution in available scenes More easy-to-code scenes available since they are easy-to-film

Complexity Algorithm #1 ITU-T P.910 section 6.3 “Two Criteria for Video Test Scene Selection”, ITU-T Study Group 12, 1994 Spatial Information SI = max time {std space [Sobel (F n )]} Temporal Information TI = max time {std space [F n -F n-1 ]}

Complexity Algorithm #2 “Perceptual Effects of Noise in Digitial Video Compression,” SMPTE 1998 o = log 10 {mean time [SI n * TI n ]} SI n = rms space [Sobel (F n )] TI n = rms space [F n -F n-1 ]

Scene Complexity Criteria One very difficult to code scene One very easy to code scene One high spatial detail scene Evenly span full range of complexity Some low Some medium Some high

Wide Variety of Scene Content Pro Relieves viewer boredom Prevents viewer overtraining Analysis results more general Con More scene content required Less HRCs for given sized full-matrix test Could use partial block design

Wider Quality Spread at Lower Bit Rates

Scene Content Selection Criteria Avoid repetition of scene type e.g., only one basketball scene Use each scene as few times as possible to allow desired HRC comparisons e.g., pair each scene with 4 HRCs Divide experiment into 2 or more full matrixes (block design)

Scene Cuts Masking effect Hides impairments Few frames before, ≈0.25 sec after Philosophy No scene cuts Television ≈3 to 5 sec between scene cuts Opening credits rapid scene cuts GOP structure interactions MPEG-2 GOP 15F for 30fps H.264 / MPEG-4 part 10 GOP 33F for 30fps

Scene Cuts Criteria One scene with rapid scene cuts ≈½ source no scene cuts ≈½ source with scene cuts Avoid scene cuts at very beginning or end of edited scene

Camera or Original Quality High quality camera recording Many people don’t have Medium quality camera recording Mild coding noise, mild analog noise Impacts HRC response Low quality camera recording VQEG does not use Commonly available to end users

Impact of Scene Content Despite viewer instructions “Please do not base your opinion on the content of the scene or the quality of the acting.” Ratings include opinion of content “Susie effect” popular scene

Impact of Scene Content “Aspen” HDTV 50Mbps MOS=4.7 excellent (17 viewers) good (8 viewers) fair (0 viewer) “ControlledBurn” 25 Mbps HDTV MOS=4.5 excellent (15 viewers) good (6 viewers) fair (3 viewer) coding artifacts “TouchdownDay” 100 Mbps HDTV MOS=4.0 excellent (4 viewers) good (17 viewers) fair (3 viewers) professional cameraman, no scene cuts TouchdownDay ControlledBurn Aspen

Original Scene Content & Quality Criteria No objectionable or controversial content No politics, violence, scantily clad people Avoid scenes with technical issues Strong judder, awkward scene cuts, poor focus, motion blur HDTV test – viewers more discriminating CIF test – viewers more tolerant

Include Animation Coder/decoder response very different H.264 higher quality MPEG-2 lower quality Animation overlay increasingly common Scrolling text and edge noise Preferably, use at least one scene with animation or animation overlay

Unusual Scene Characteristics for Coding Difficulty & Viewer Perception Fine detail Gravel, grass, hair, rug texture, pinstripes Sensitive to blurring, but viewers may not notice or care Sharp black/white edges Blurred background, foreground in focus Night scene or dim lighting At least one

Unusual Scene Characteristics for Coding Difficulty & Viewer Perception Ramped color with graduated change of color (e.g., sunset) Water, fire & smoke contribute unusual shapes and shifting patterns Picture jiggles or bounces significantly Hand-held camera Flashing lights or other very fast events Emergency lights, camera flash, falling snow, breaking glass

Unusual Scene Characteristics for Coding Difficulty & Viewer Perception High motion E.g., object moves 20+ pixels per frame at VGA resolution Action is in a small portion of the total picture Attention focus Colorful scene Important

Unusual Scene Characteristics for Coding Difficulty & Viewer Perception Small amounts of analog noise Multiple objects moving in a random, unpredictable manner

Include Close-up of People Popular Skin tones Knowledge of how people ought to appear & move Internal reference Internal Reference No Internal Reference

Close-up People Criteria Filming Privacy issues Releases Faces large enough to be easily seen Several scenes with close-up people Close-up Crowd Scene

Contrast and Brightness One scene with low contrast Soft edges One scene with high contrast Hard edges One scene with low brightness Dim One scene with high brightness Mostly white

Obtaining Content TV & movie content strictly controlled Purchase for own use Typically cannot redistribute Hire someone to make content for you Shoot it yourself! Royalty-free distribution More flexible usage No time expiration

2009, 4 th Quarter Consumer Digital Video Library Repository of high quality video User forums For quality measurement algorithms Subjectively rated databases For consumer video processing Coding, corrective/reconstructive processing, enhancement, re-formatting