A Probabilistic Framework for Video Representation Arnaldo Mayer, Hayit Greenspan Dept. of Biomedical Engineering Faculty of Engineering Tel-Aviv University, Israel Jacob Goldberger, CUTe Systems, Ltd.

Introduction In this work we describe a novel statistical video representation and modeling scheme. Video representation schemes are needed to enable segmenting a video stream into meaningful video- objects, useful for event detection, indexing and retrieval applications.

PACS: Picture Archiving & Communication Systems Storage Query/Retrieve InternetDatabaseManagement Query/Retrieve VisualInformation Tele-Medicine

Spatio-Temporal Segmentation of Multiple Sclerosis Lesions in MRI What are interesting events in medical data? Spatio-Temporal Tracking of Tracer in Digital Angiography

Analysis of a video as a single entity Vs analysis of video as a sequence of frames Inherent Spatio-temporal tracking Gaussian Mixture Modeling in color & space- time domain t x y Introduction

Learning a Probabilistic Model in Space-Time Feature Vectors [L,a,b,x,y,t] (6 - dimensional space) Expectation Maximization (EM) t y Gaussian Mixture Model x

Video Representation via Gaussian Mixture Modeling Each Component of the GMM Represents a Cluster in the Feature Space (=Blob) and a Spatio-temporal region in the video PdF For the GMM : With the Parameter set

Given a set of feature vectors and parameter values, the Likelihood expresses how well the model fits the data. The EM algorithm: iterative method to obtain the parameter values that maximize the Likelihood …

Expectation step: estimate the Gaussian clusters to which the points in feature space belong Maximization step: maximum likelihood parameter estimates using this data The EM Algorithm

Initialization & Model selection Initialization of the EM algorithm via K-means: –Unsupervised clustering method –Non-parametric Model selection via MDL (Minimum Description Length) –Choose k to maximize: –l k = #free parameters for a model with k mixture components

Static space-time blobDynamic space-time blob The GMM for a given video sequence can be visualized as a set of hyper-ellipsoids (2 sigma contour) within the 6 dimensional color-space-time domain. Video Model Visualization

Detection & Recognition of Events in Video C L a b x y t C xt C tt C tt - Duration of space-time blob Static/Dynamic blobs - thresholds on R xt (Hor. motion) & R yt (Ver. motion) Direction of motion - sign of R xt, R yt Correlation coefficient : C yt

Detection & Recognition of Events in Video C L a b x y t C xt C tt Blob motion (pixels per frame) via linear regression models in space & time : C yt Horizontal velocity of blob motion in image plane is extracted as the ratio of cov. parameters. Similar formalism allows for the modeling of any other motion in the image plane.

Probabilistic Image Segmentation A direct correspondence can be made between the mixture representation and the image plane. Each pixel of the original image is now affiliated with the most probable Gaussian cluster. Pixel labeling: Probability of pixel x to be labeled:

OriginalModel Segmentation

OriginalModel SegmentationDynamic Event Tracking

Limitations of the Global Model How can we represent non-convex spatio-temporal regions? All the data must be available simultaneously - Inappropriate for live video - Model fitting time increases directly with sequence length

Piecewise Gaussian Mixture Modeling Modeling the Video sequence as a succession of overlapping blocks of frames. Obtain a succession of GMMs instead of a single global model. Important issues: initialization; matching between adjacent segments for region tracking. (“gluing”)

Piecewise GMM : “Gluing” / Matching at Junctions Frame J 5 blobs via GMM 5 Frame J 5 blobs via GMM 6 Frame J 5 Ex: Blob matching

Original SequenceDynamic Event Tracking Model Sequence

Horizontal Velocity in function of Block of Frame Index Pix / frame BoF #

Vertical Velocity in function of Block of Frame Index Pix / frame BoF #

Original Sequence Segmentation Map Sequence BOF # Pix / frame Horizontal Velocity BOF # Pix / frame Vertical Velocity Sweater Trousers

Spatio-Temporal Segmentation of Multiple Sclerosis Lesions in MRI Sequences

Methodology Time K >= 4 1) CSF 2) White Matter 3) Gray Matter 4) Sclerotic Lesions Segmentation Maps Blobs in [L x y t] Feature Space Frame by frame Segmentation 3D (x,y,t) Connected Components GMM for Luminance

Original Sequence Dynamic Event Tracking Segmentation Maps Sequence

Area (in Pixels) Time point Time Evolution

Conclusions The modeling and the segmentation are combined to enable the extraction of video-regions that represent coherent regions across the video sequence, otherwise termed video-objects or sub-objects. Extracting video regions provides for a compact video content description, that may be useful for later indexing and retrieval applications. Medical applications: lesion modeling & tracking Acknowledgment Part of the work was supported by the Israeli Ministry of Science, Grant number