Characterizing activity in video shots based on salient points Nicolas Moënne-Loccoz Viper group Computer vision & multimedia laboratory University of Geneva

NML - CVML - UniGe2 Outline Context Video Activity extraction – Spatial salient points – Spatio-temporal salient points – Spatio-temporal salient regions Results Conclusion

NML - CVML - UniGe3 Context Describe visual content of video  Index, retrieve and browse video database Requirements – Generic approach (v.s. domain oriented) – Local approach (v.s. global description of the content) – Computationally efficient approach Video activity : salient region of the video 3D space

NML - CVML - UniGe4 Context Frames Activity #1 Activity #2  Description in space and time of video activity  Inference based on video object and event relationships  High level indexing

NML - CVML - UniGe5 Context Related approaches : Spatio-temporal segmentation – Segmentation problem – Computational efficiency Our approach : – Spatio-temporal salient points – Spatial grouping of salient points – Temporal matching of salient regions  Set of activities

NML - CVML - UniGe6 Overview Salient points & trajectories Global motion estimation Motion outliers Spatial grouping Video stream Salient extraction Temporal matching Salient extraction

NML - CVML - UniGe7 Salient points Points in the image space – Repetitive (robust) – High information content  Scale invariant interest points (Mikolajczyk, Schmid 2001) – One of the most robust – Salient points with characteristic scale

NML - CVML - UniGe8 Salient point extraction Linear Scale-Space : Harris function : Salient points (image space) : local maxima h(v,s) Laplacian over scale : Salient points (scale space) : local maxima l(v,s) & h(v,s)

NML - CVML - UniGe9 Salient point extraction Example : scale

NML - CVML - UniGe10 Salient point extraction scale

NML - CVML - UniGe11 Motion estimation Goal : – Find points having salient temporal behaviour  Estimate background motion model  Select points that do not follow this background motion model Estimation : – Compute salient point trajectories – Estimate corresponding affine motion model

NML - CVML - UniGe12 Trajectories Point descriptors : Local Grayvalue Invariants Point distance : Mahalanobis distance

NML - CVML - UniGe13 Trajectories Goodness of match : Candidate matching points – Matches with spatial distance below a threshold Relaxation process : – Disambiguating set of candidate matches – Greedy Winner-Takes-All algorithm

NML - CVML - UniGe14 Motion estimation Affine motion model : Estimate model from trajectories – Iterative least square error estimate (Tukey M-Estimator)  select points that belong to the global motion model  Assumption : +50% points belong to the background

NML - CVML - UniGe15 Motion estimation Points of the background and their motion estimated using the presented approach All points and their motion estimated by a dense motion estimator

NML - CVML - UniGe16 Spatio-temporal salient points Points whose trajectory does not fit the global motion model  Outliers (moving objects) Points without trajectory (no matching point)  New points (appearing or deformable objects)

NML - CVML - UniGe17 Spatio-temporal salient points Fixed camera Moving camera

NML - CVML - UniGe18 Salient regions Set of spatio-temporal salient points  Feature distribution of points (RGB colour features)  Spatial distribution of points Grouping process : Estimate salient region models

NML - CVML - UniGe19 Feature model Feature description – A salient point is characterized by the feature distribution of its neighbourhood – Assumption : maximum of four regions in the neighbourhood of the points – Compute the corresponding colour distributions : K-means clustering Gaussian model Gaussian models clustering – Greedy algorithm (AHC)  Set of Gaussian distributions representing the distribution of the neighbourhood of the salient points :

NML - CVML - UniGe20 Salient region model Feature models – Mixture of Gaussians  Corresponding weight of each Gaussian Spatial model : – Estimate spatial pdf from salient points & associated scale

NML - CVML - UniGe21 Salient region models Iterate a RanSaC algorithm Estimate salient region model – Robust estimation (Tukey M-estimator) – Cost function :

NML - CVML - UniGe22 Salient regions Fixed camera Moving camera

NML - CVML - UniGe23 Temporal matching Spatio-temporal salient regions of arbitrary length  Matching of salient regions Use salient points trajectories 1. Match regions with the highest number of matching points

NML - CVML - UniGe24 Results - Meetings

NML - CVML - UniGe25 Results – Misc

NML - CVML - UniGe26 Conclusion Contribution – Highly informational content descriptor – Generic content descriptor – Local in space and time content descriptor Limitation – Noisy & short activity Ongoing work – Temporal filtering of activity – Indexing of videos through the set of activity