Efficient Spatiotemporal Grouping Using the Nyström Method Charless Fowlkes, U.C. Berkeley Serge Belongie, U.C. San Diego Jitendra Malik, U.C. Berkeley

Grouping With Pairwise Affinities Sarkar and Boyer (1996), Shi and Malik (1997), Perona and Freeman (1998), Gdalyahu, Weinshall, and Werman (2000),..... Compute the similarities between pairs of points in the image Find groups of points which have high similarity with each other and low similarity with the rest of the image.

Normalized Cuts 2. Find the leading eigenvectors of the Normalized Laplacian 3. Segment the image using the leading eigenvectors 1. Compute matrix K which contains the pairwise similarities Computational Complexity: Need to find eigenvectors of an NxN matrix where N is the number of pixels. Other spectral partitioning techniques have same complexity.

Spatiotemporal grouping Adelson and Bergen (1985), Bolles, Baker and Marimont (1987), Shi and Malik (1998)

Computational Problem Hard to exploit pairwise clustering techniques since 256x384x30 frames entails ~10 13 pairwise similarities. How can we overcome this problem?

Coping with the computational burden 1 Zero out small entries in the affinity matrix Shi and Malik (97,98) 2 Exploit redundancy between rows of the affinity matrix (this talk)

Outline Exploiting Redundancy The Nyström approximation Application to segmenting video sequences

Exploiting Redundancy

Compute Affinity Matrix

Exploiting Redundancy Compute Leading Eigenvectors of Normalized Laplacian

Exploiting Redundancy Choose sample points

Exploiting Redundancy Compute strip of K

Exploiting Redundancy

Interpolate complete eigenvectors

Outline Exploiting Redundancy The Nyström approximation Application to segmenting video sequences

Approximating eigenfunctions E. J. Nyström (1929) Baker (1977) Williams and Seeger (2001) Interpolate eigenfunctions using The Nyström Extension: We would like to find numerical solutions to:

Matrix Completion Affinity Matrix: Approximate it with: Approximation Error:

Approximation Error

Extrapolating Eigenvectors Just matrix notation for the Nyström extension Diagonalize approximate K to get complete eigenvectors

Nyström-NCuts Algorithm 1 Choose sample points in image 2 Compute similarities for A and B blocks of K 3 Compute row sums to estimate degree 4 Normalize A and B blocks by degree 5 Compute approximate eigenvectors and orthogonalize 6 Cluster the embedded points using k-means

Outline Exploiting Redundancy The Nyström approximation Application to segmenting video sequences

Affinity Function for Video Pairwise affinity function between pixels in a video sequence makes of three cues - Similarity in color - Proximity in time and space - Common Fate (similarity in optical flow) We use squared-exponential kernel with diagonal weighting

Performance Segmenting a 5 frame video sequence at 120x150 resolution (~100,000 pixels) takes less than 1 minute in MATLAB on a PC

Conclusions Applied the Nystrom approximation to Normalized Cuts Exploited redundancy in image sequences in order to perform efficient spatiotemporal grouping

K-Way Normalized Cuts Find the leading eigenvectors of Normalized Laplacian Embed data and cluster j th pixel i th vector V

Sampling the image