1. Segmentation and Event Detection in Soccer Audio Lexing Xie, Prof. Dan Ellis EE6820, Spring 2001 April 24 th, 2001

2. 2 The problem Event detection in sports video In this project: the audio part Our approach Segmentation + Event Detection Incorporate domain knowledge

3. 3 Outline Related work Observations on soccer audio Segmentation Features Decision scheme Result Event detection Scope Feature metric Result Generalization Next step

4. 4 Related Work Audio segmentation Speech-silence discrimination [Rabiner78] Speech / music / mixture segmentation [Saunders96] [Scheirer97] [Williams99] Sports audio analysis Classify excited speech [Rui2000] Keyword/event template matching [Chang96] [Rui2000]

5. 5 Observations #1 Sound Types Foreground speech Noisy vocal sound with visible phoneme structure Background noise Ambient crowd, whistles, cheers, etc. Acoustics [Fahy2001] Sound intensity in open space: Sound attenuation in air Production conditions Frequency response of microphone Automatic Gain Control

6. 6 Observations #2 Large variety across games Commentator “verbosity” Audience “excitability”  not labeling and training In different languages  not ASR Not template-matching & training Assumptions on temporal characteristics Short-term dynamics  Long-term variety  -- Seg.Det. unit0.03sec0.5~1 context15>100

7. 7 Segmentation Algorithm Commentary vs. Crowd segmentation Decision Rules Energy > Global Avg. & adaptive threshold 1 st formant energy Fricative energy Feature extraction sound Morphological operations Post- processing Seg. boundary

8. 8 Segmentation Result Sound length Ground truth HitsMissesFalse Alarms 100 sec504642 crowd commentary

9. 9 Detection #1 Detecting audio events in crowd noise Examples: crowd cheering, whistle, … Subjective definition Spectral: centroid, roll-off Energies: E, Er1, Er2 feature contour and moments of the contours Pick up crowd, chop into units Distance metric Seg. boundaries Feature calculation Most distinctive segment

10. 10 Detection #2 Compute Mahalanobis distances [Duda 73] Feature element normalization and decorrelation Pick up distinctive segments Largest distance to all other segments (typically top 5~10%) Clustering: detecting outliers Merge adjacent segments

11. 11 Detection Results The game: River Plate vs. Los Andes Assumptions: The majority are Unimportant We do have Important parts! Cluster analysis helps Time (sec) 128 0 Start 49.1 Attacking.. 55.0 Foul! 95.2 Penalty kick 100.2 GOAL!

12. 12 Generalization Segmentation tasks Other Sports (baseball, tennis, etc.) Film sound track (Sense and Sensibility) Detection of sparse audio events Surveillance video Silence MusicSpeech

13. 13 Next step More experiments Improve decision scheme Improve GMM in segmentation Use cluster analysis in detection New features Wish list Classification of speech segments Other interesting noise patterns Investigate sound mixtures

14. 14 Summary Segmentation Use energy features Best result: precision 95%, recall 92% Event detection Use feature distance Interesting segments retrieved More work to follow

15. 15 Thanks!

16. 16