Using spatio-temporal probabilistic framework for object tracking By: Guy Koren-Blumstein Supervisor: Dr. Hayit Greenspan Emphasis on Face Detection & Tracking

Agenda ► Previous research overview (PGMM) ► Under-segmentation problem ► Face tracking using PGMM  Modeling skin color in [L,a,b] color space – over-segmentation problem ► Optical flow – overview ► Approaches for using optical flow ► Examples

Previous research ► Complementary research to an M.Sc. Thesis research conducted by A.Mayer under the supervision of Dr. H.Greenspan and Dr. J. Goldberger. ► Research Goal: Building a probabilistic framework for spatio-temporal video representation. ► Useful for:  Offline – automatic search in video databases  Online – characterization of events and alerting on those that are defined as ‘suspicious’

Previous research Parsing clip to BOF Build feature Space [L a b] Build GMM modelGMM model In [Lab] space Label BOF pixels Connect. Comp. On [L,a,b,x,y,t] Learn GMM model On [L,a,b,x,y,t] Source ClipBOF 1Labeled BOFBlob Extraction Under segmentation problem…

Building GMM model ► Each sample in the [L,a,b] feature space is a sample of random vector x  R d with the following PDF:

Building GMM model ► Given a set of feature vectors x1,..., xn the maximum likelihood estimation of  is : ► Obtaining  ML by using 2 stages iterative EM algorithm: Expectation step Maximization step

Labeling pixels ► The Labeling of a pixel related to the feature vector x is chosen as the maximum a posteriori probability, as follows:

Face Detection & Tracking ► Most of the known techniques can be divided into two categories :  Search for skin color and apply shape analysis to distinguish between facial and non-facial objects.  Search for facial features regardless of pixel color (eyes,nose,mouth,chin,symmetry etc.)

Apply framework to track faces ► The framework can extract and track after objects in an image sequence. ► Applying shape analysis to each skin- colored-blob can label the blob as ‘face’ or ‘non-face’. ► The face will be tracked by virtue of the tracking capabilities of the framework

Skin color in [L a b] ► Skin color is modeled in [a b] components only ► Supplies very good discriminability between ‘skin’ pixels and ‘not-skin’ pixels (high rate of True-Negative) ► Not optimal in terms of True-Positive (leads to mis- detection of skin color pixels)

Over-segmentation of faces ► Building blobs is done in [L a b] color space. ► More than one blob might have skin color [a b] components ► Solution : Unite all blobs whose [a b] are close enough to the skin color model (adaptive TH can be used)

Under Segmentation ► Faces moving in front of skin-color background are not extracted well. ► Applying shape analysis on the middle map yields mis-detection of faces.

Employing motion information ► Motion information helps to distinguish between foreground dynamic objects and static background ► 2 levels of motion information  Binary – indicates for each pixel whether it is in motion or not. Does not supply motion vector. Feature space: [L a b x y t m] where m={0,1}  Optical flow - supplies motion vector according to a given model. Feature space: [L a b x y t V x V y ]

Is binary information good enough?

Optical Flow ► Optical flow is an apparent motion of image brightness ► If I(x,y,t) is the brightness, two main assumptions can be made:  I(x,y,t) depends on coordinates x,y in greater part of the image  Brightness of every point of moving object does not change in time

Optical Flow ► If object is moving during time dt and its displacement is (dx,dy) then using Taylor series ► According to assumption 2: ► Dividing by dt gives the optical flow equation:

Optical Flow – Block Matching ► Does not use the equation directly. ► Divides the image to blocks ► For every block in I t it search for the best matching block in I t-1. ► Matching criteria: Cross Correlation, Square Difference, SAD etc.

Working with 8-D feature space ► Connected component analysis:  Does not require initialization of the order of the model  Hard decision prone ► GMM model via EM:  Initialized by K means. Requires initialization of K.  Impose elliptic shape on the objects  Soft Decision prone Parsing clip to BOF Build feature Space [L a b] Build GMM model In [Lab] space Label BOF pixels Connect. Comp. On [x,y,t,V x,V y ] Learn GMM model [x,y,t,V x,V y ] Frame By Frame Tracking

Frame by frame tracking ► Widely used in the literature ► Can handle variations in object’s velocity ► Tracking can be improved by employing Kalman filter to predict object’s location and velocity Predict params for next frame merge blobs Create new blobs split blobs Kill old blobs Label by updated parameters Update blob’s params Label by predicted parameters

Examples ► Opposite directions:  Optical Flow, Connected component (Extracted Faces), GMM Optical FlowConnected componentExtracted Faces GMM Optical FlowConnected componentExtracted Faces GMM ► Same direction, different velocity  Optical Flow, Connected component, GMM (Faces) Optical FlowConnected componentGMM Faces Optical FlowConnected componentGMM Faces ► Different directions – complex background  Optical Flow, Connected component, GMM: K=5,K=3,Faces Optical FlowConnected component K=5K=3Faces Optical FlowConnected component K=5K=3Faces ► Variable velocity  Optical Flow, Connected component, GMM, Frame By Frame Optical FlowConnected componentGMMFrame By Frame Optical FlowConnected componentGMMFrame By Frame

Real World Sequences ► Face tracking  Optical Flow Optical Flow Optical Flow  No motion info No motion info No motion info  Connected component Connected component Connected component  GMM GMM  Frame By Frame Frame By Frame By Frame ► Car Tracking  Optical Flow Optical Flow Optical Flow  No Motion info No Motion info No Motion info  GMM GMM ► Flower garden  Optical Flow Optical Flow  No motion info No motion info  Connected component Connected component  GMM GMM

Summary ► Applying probabilistic framework to track faces in video clips ► Working in [L,a,b] color space to detect faces ► Handling over segmentation ► Handling under segmentation by employing optical flow information in 3 different ways:  Connected Component Analysis  Learning GMM model  Frame By Frame tracking

Further Research ► Adaptive face color model ► Variable length BOF (using MDL) ► Using more complex motion model

Thank you for listening Questions ?