1. Multimodal Analysis Video Representation Video Highlights Extraction Video Browsing Video Retrieval Video Summarization

2. Multimodal Analysis Video Representation Video Browsing Video Retrieval Video Content Bottom-up Top-down Highlights based Summarization Table of Contents based Summarization Video Summarization content access data flow

3. Video with Audio Track Play / Break Audio-Visual Markers Highlight Candidates Highlight Groups Feature extraction & segmentation Key audio-visual object detection Audio-visual markers association Grouping Pla y Break Visual Marker Audio Marker A highlight

4. Scenes Groups Shots Key frames Visual Semantic Audio Camera motion Highlight Groups Highlight Candidates Audio-Visual Markers Play/Break Retrieval Highlights based Summarization ToC Index Highlights Browsing ToC based Summarization

5. Audio Video Audio Markers Detection Visual Markers Detection ApplauseCheers Baseball CatcherSoccer Goal Post Golfer Bending to Hit A-V Marker Negotiation Golf Swings Highlight Candidates Finer Resolution Highlights Golf Putts Which sport is it? Non-highlights Strikes, BallsBall-hitsNon-highlights Corner KicksPenalty KicksNon-highlights Excited Speech

6. Soccer video markers Baseball video marker Golf video markers

7. Time DomainFrequency Domain Audio Class Recognition (GMM) Applause Cheering Music Speech Excited Speech Feature Extraction Comparing Likelihoods Input Audio Classify the input audio into one of 5 Audio Markers using GMM Audio Class MFCC Coefficients. Training the GMM Classifiers using MFCC features and BIC. Feature Extraction Training Audio Clips

8. Time DomainFrequency Domain Audio Class Recognition (GMM) Excited Speech Other (Applause,Cheering, Music, Speech) Feature Extraction Comparing Likelihoods Input Audio Classify the input audio into one of 2 Audio Markers using GMM Audio Class MFCC Coefficients. Training the GMM Classifiers using MFCC features and CV Feature Extraction Training Audio Clips Task = Sports Highlights

10. Time DomainFrequency Domain Audio Class Recognition (GMM) Applause Cheering Music Speech Excited Speech Feature Extraction Comparing Likelihoods Input Audio Classify the input audio into one of 5 Audio Markers using GMM Audio Class MFCC Coefficients. Training the GMM Classifiers using MFCC features and BIC. Feature Extraction Training Audio Clips

11. Feature Extraction Audio Classifier Importance Level Calculation Task Input Audio MDCTs Class Label Importance Level

12. Generic Audio Classification Compare Likelihoods Applause Cheering Music Speech Excited Speech MDCTs Class Label Training Data for GMMs Feature Extraction Training Audio Clips

13. Feature Extraction Audio Classifier Importance Level Calculation Task Input Audio MDCTs Class Label Importance Level

14. Task Specific Audio Classification Compare Likelihoods MDCTs Class Label Feature Extraction Task = Sports Highlights Excited Speech Other (Applause,Cheering, Music, Speech) Training Audio Clips

15. Filter 1Filter 2 Catcher Model Interpretation Filter 1 Filter 2Filter 1Filter 2 Golfer view 1Golfer view 2 Golf Models Interpretation Filter 1Filter 2Filter 3 Goal post view 1 Filter 1Filter 2 Filter 3 Goal post view 2 Soccer Models Interpretation

16. Pla y Break Video clip A-V markers catcherbat swingcheer excited speech Index