1. Automated Summaries of Audio/Video Presentations Abigail Curden

2. Main Objectives To create automated summaries of audio/video presentations To integrate these summaries into a multimedia database and create a single visual display for retrieval/browsing

3. Possible Web Layout Electrical and Computer Engineering 20 th June 2003 Title: Presentation Analysis and Scene Change Detection QUESTIONS

4. Key Elements Of Project Slide location Slide change detection Slide identification Audio analysis Integration of all aspects of project- High level model

5. Presentation samples 1. Dark Background 2.Lightly Coloured background 3. Lightly Coloured background 4. Blank background left & right camera view 5. Lightly Coloured background left & right camera view 6. Blue background 7. Black background

6. Slide Location Purpose: To determine the co-ordinates of the slide in the video frame and so restrict further analysis to this region Strategy: The slide would most likely be rectangular shaped, coordinates given by corners Techniques Hough transform Corner detection

7. Slide Location: Hough Transform STEP 1: GRAYSCALE IMAGE STEP 3: HOUGH LINES STEP 2: HOUGH ARRAY STEP 4: INTER- SECTION POINTS

8. Slide Location: Harris Corner Detection,difference in y (row) direction,difference in x (column) direction,difference in x and y direction Corners:

9. Slide Location Combining Hough Transform and Corner Detector Hough Transform and Corner Detection applied to N frames chosen throughout the presentation Form a histogram of slide location points found by both methods Slide location points, assumed to correspond to count values greater than 75%N. hough intersection points>0.75N & corner points>0.75N

10. Slide Location Results

11. Slide Location Summary Affected by objects in room Suggestion: Have the system suggest to the user the probable slide coordinates and have the user verify or chose coordinates DESCRIPTION #CORNERS FOUND SPURIOUS POINTS? 1Dark slide background1yes 2Lightly Coloured slide background1yes 3Lightly Coloured slide background3yes 4 Blank slide background, left camera view4no Blank slide background, right camera view4no 5 Lightly Coloured slide background, left camera view3yes Lightly Coloured slide background, right camera view 3yes 6Light Blue slide background, camera at right angles4yes 7Black slide background, camera at right angles0yes

12. Slide Change Detection Purpose: To parse the video according to scene cuts. Performed only within region of slide coordinates Strategy: At slide change boundaries there will be large change in color/grayscale content Techniques Histogram Change Frame Difference

13. Histogram change The histogram of consecutive frames were found and the relative error found Slide Change Detection Techniques Frame difference The difference in intensity levels for consecutive frames is determined

14. Slide Change Detection Results: Histogram Change %error=11%error=25%error=41%

15. Slide Change Detection Results: Frame Difference Error=8Error=11 Error=20

16. Slide Change Results *histogram change error *frame difference error

17. Slide Change Detection: Combined Results and Threshold Frame difference error signal Median filter applied to reduce spikes Gaussian filter to further smooth signal Windowed std_dev calculated(diff ) Smoothed error signal minus original error signal Threshold taken as 5  std_dev

18. Slide Change: Occlusion Frame1Frame2Frame3 Motion mask previous two frames Current Frame difference Frame difference after masking

19. Slide Change Summary DESCRIPTION SLIDE CHANGES INCORRECT LOCATIONS 1Dark slide background 7/72(occlusion) 2Lightly Coloured slide background 3/30 3Lightly Coloured slide background 4/40 4 Blank slide background, left camera view 1/40 Blank slide background, right camera view 4/40 5 Lightly Coloured slide background, left camera view 4/50 Lightly Coloured slide background, right camera view 5/50 6Light Blue slide background, camera at right angles 0/67 7Black slide background, camera at right angles 4/40

20. Slide Identification Purpose: To obtain the sequence of the presentation Strategy: Slides can be identified by matching frame from video to ppt slide Techniques Warping of frame from video to ppt slide Displacement rectification Matching Penalty

21. Slide Identification-Warping To have the frame the same shape and size as the ppt slide Frame warped to the slide plane using projective transformation SLIDEFRAMEWARPED FRAME

22. Slide Identification Displacement Slide Identification Techniques Slide Identification Techniques dy Displacement in frame and ppt slides causing mismatch Only vertical displacement considered as displacement in the horizontal direction would be insignificant Sub-divide image into horizontal blocks Find displacement vector for each block (sum of absolute difference of means) Find position that minimizes error measure Error measure: If the ratio of error at zero displacement to the minimum error is greater than 5% Slide Identification Techniques

23. Slide Identification- Matching Correlation coefficients WARPED FRAME SLIDE14 SLIDE16 SLIDE15

24. Slide Identification - Penalty Penalty= Before Penalty After Penalty Slide Number

25. Slide Identification Results 1700190021002300 Frame Number Error

26. Slide Identification: Occlusion Four Different Presentations

27. Slide Identification Summary Correlation in gradient images more reliable In the cases where the slides were occluded, applying the penalty improved results IDPresentation %Corr Gradient %Corr Edge #Frames#SlidesOcclusionPenalty 1 Dark slide background10099.1613073Yes 2 Lightly coloured slide background 100 16393Yes 4 Blank Slide Background, left camera view 894025773Yes Blank slide background, right camera view 10099.9329893NoYes 6 Light Blue slide background, camera at right angles 100 11783No 7 Black slide Background, camera at right angles 100 35273No

28. Audio Analysis Purpose: Audio content could also provide useful information about presentation. Portions of long silence Pitch of the speaker Speaker change points Detecting Boundaries of silence implemented

29. Silence Detection Techniques Short term Zero crossing rate Short term energy Ideally, zero crossing rate and average energy zero for silence

30. Silence Detection SPEECH SIGNAL ENERGY SIGNAL ZERO CROSSING SIGNAL Time(s)

31. Silence Detection-Thresholds Zero crossing count and energy non-zero for silence Assume first 100ms of audio signal to be silence, find energy and zero crossing rate for this region Thresholds for energy and zero crossing taken as mean + 2  standard deviation Threshold for the minimum length of silence Silent regions found for the entire audio signal The number of times the signal stays below the threshold is counted Threshold for length taken as mean + 4  standard deviation of the count values

32. Silence Detection Results *start of silence *end of silence FEMALE SPEAKER PRESENTATION MALE SPEAKER PRESENTATION

33. Silence Detection Summary FEMALE SPEAKER PRESENTATION 22/23 corresponded to manually selected regions 1 false point Minimum duration =1.4s MALE SPEAKER PRESENTATION 25/28 corresponded to manually selected regions 6 false points Minimum duration 1.5s Different Speaker “ahhhhhhh”speech

34. High Level Model–Hidden Markov Model Purpose: To investigate Hidden Markov Model (HMM) and its application to: Slide change detection, slide identification HMM is a “ doubly embedded stochastic process that is not observable (it is hidden), but can only be observed through another set of stochastic processes that produce the sequence of observations” (Rabiner 1989 in Tutorial on HMM)

35. Hidden Markov Model Elements The number of states in model, N The number of distinct observations per state The initial state distribution matrix,  The state transition probability matrix, A The observation symbol probability matrix, B HMM model, =(A, B,  )

36. HMM for Presentation The HMM have been applied to video but used to classify camera motion HMM for this presentation, applied to classify information content The model, is represented by the presentation The states would then correspond to the slides in the presentation The observation sequence would be the histograms from slide change detection or the correlation matrices from slide identification

37. HMM Implementation Problem 1: Computing the probability of observation sequence, given the model Problem 2: Determining the optimal state sequence Problem 3: Training of the model REPRESENTATION OF HMM 11 44 22 nn 33 S2S2 S3S3 S1S1 SnSn S4S4 a 12 a 23 a 34 a 21 a 32 a 43 a nn a 44 a 33 a 22 a 11 O=[O 1,O 2,……..,O m ]

38. Further Work Further exploration of HMM Exploring non ppt presentations Implementation and testing-user feedback Future expansions Develop another HMM for Speaker change detection Keyword detection in speech Speech to text processing

39. Summary Objective: To created Automated Summaries Slide change detection Slide Identification Detecting boundaries of silence Limitations Slide location performance poor Processing time of some algorithms long Recommendations Clear or lightly coloured slide background Camera at right angles or near right angles Silent segment at start of presentation

40. Related Work Microsoft research Emphasis on minimizing length of presentation Slide change performed automatically by power point X Ju et al in IEEE transactions on circuits and systems for video technology Vol. 8, #5, Sep’98 Used motion estimation to detect slide change Sequencing was not achieved No audio analysis was performed Constraint in test samples

41. Acknowledgements Supervisors Dr. Anil Kokaram Dr. Cathy Radix Dr. Francis Asamoah The Multimedia Department, School of Education Colleagues Niall Rea, Rozenn Dahyot & Vijaya Ragoonanan CPR4AG fund from campus Research and Publication Fund committee, School for Graduate Studies & Research, U.W.I. St Augustine Financial Support from Department of Electrical and Computer Engineering Enterprise Ireland International Collaboration Grants 2001/02 and 2002/03