UNIVERSITY OF MURCIA (SPAIN) ARTIFICIAL PERCEPTION AND PATTERN RECOGNITION GROUP REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos Dept. de Informática y Sistemas University of Murcia - SPAIN

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, Introduction Main objective: develop a new technique to track human faces and facial features: –Working in real-time: fast processing –Under realistic conditions: robust to facial expressions, lighting conditions, 3D head pose and movements –With high location accuracy: facial features location (eyes, nose, mouth)

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, Introduction Index of the presentation: –Face integral projections –Integral projection models –Alignment of projections –The tracking process –Experimental results –Conclusions

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, Face integral projections Definition. Let i(x,y) be an image, and R(i) a region in it – Vertical integral projection P VR : {y min,..., y max }  R P VR (y) = i(x,y);  (x,y)  R(i) – Horizontal integral projection P HR : {x min,..., x max }  R P HR (x) = i(x,y);  (x,y)  R(i) FACE P VFACE (y) y EYES P HEYES (x) x

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, Face integral projections Dimensionality reduction: 3D world  2D images  1D integral projections Advantages: –Fast to compute and to process Disadvantages: –Loss of information. Is it relevant for the problem? What happens when applied to human faces?

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, Face integral projections Different individuals

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, Face integral projections Different facial expressions

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, Face integral projections Different segmented regions

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, Integral project. models Face integral projec. is an interesting and robust feature for tracking It has been applied using heuristic analysis: max-min search, fuzzy logic, thresholding projections Proposal: define and work with adaptable projection models How to model a variety of projection patterns?

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, Integral project. models A projection model is a pair: M : {m min,..., m max }  R (Mean) V : {m min,..., m max }  R(Variance)

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, Integral project. models Advantages of working with explicit projection models: –The model is learnt from examples. In tracking, it is adapted to tracked faces –We can define a signal to model distance:

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, Integral project. models Advantages of working with explicit projection models: –The model can be reprojected Reprojection (by outer product) using 1 vertical IP and 2 horizontal IP

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, Alignment of projections Corresponding facial features should be projected on the same locations Before alignment After alignment

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, Alignment of projections Alignment with respect to a model Problem formulation: –Let S: {s min,..., s max }  R be a signal –Let M,V: {m min,..., m max }  R be a model –Let S’ be a family of scale and translations alignments of S: –Find parameters ( a,b,c,d,e ) which minimize:

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, The tracking process Tracking is based on the alignment of integral projections Main steps: 1. Prediction and segmentation 2. Vertical alignment 3. Horizontal alignment 4. Orientation estimation

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, The tracking process Features to track Input to the tracker –Bounding ellipse –Facial features: eyes and mouth –Face model –State of tracking in frame t-1 –Frame t

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, The tracking process 1. Prediction and segmentation Null predictor: locations in frame t-1 are used to extract the face in frame t Predicted location Wrapped Segmented

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, The tracking process 2. Vertical alignment Using the vertical projection of the face, and the model, align the face vertically Segmented Model P VFACE (y) Align P VFACE to M VFACE y Align using the obtained parameters (a,b,c,d,e)

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, The tracking process 3. Horizontal alignment Using the horizontal projection of the eyes’ region, align the face horizontally Segmented after step 2 Model P HEYES (x) Align P HEYES to M HEYES y Align using the obtained parameters (a,b,c,d,e) x

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, The tracking process 4. Orientation estimation Using vertical projections of each eye, estimate the orientation of the face Segmented after step 3 Model P VEYE1, P VEYE2 Align P VEYEi to M VEYEi y

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, The tracking process Global structure of the tracker 1. Prediction and segmentation 2. Vertical alignment 3. Horizontal alignment 4. Orientation est.

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, Experimental results Experiments: –Location accuracy –Execution time per frame –Robustness to facial expressions, 3D pose, lighting conditions Different sources: TV, video- conference camera and DVD Compared with CamShift algorithm

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, Experimental results FILE NAME: tl5-02.aviSOURCE: TV FORMAT: 640x480 (25 fps)LENGTH: 280 frames MODEL SIZE (pixels): 97x123 AVG/MAX ERROR (mm): 3.41 / 12.9 TIME/FRAME (ms): 4.02

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, Experimental results FILE NAME: a3-05.aviSOURCE: TV FORMAT: 640x480 (25 fps)LENGTH: 541 frames MODEL SIZE (pixels): 101x136 AVG/MAX ERROR (mm): 1.95 / 9.76 TIME/FRAME (ms): 4.62

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, Experimental results FILE NAME: a3-2.aviSOURCE: TV FORMAT: 320x240 (25 fps)LENGTH: 440 frames MODEL SIZE (pixels): 75x95 AVG/MAX ERROR (mm): - TIME/FRAME (ms): 3.74

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, Experimental results FILE NAME: ggm2.aviSOURCE: QuickCam FORMAT: 320x240 (25 fps)LENGTH: 655 frames MODEL SIZE (pixels): 70x91 AVG/MAX ERROR (mm): 1.83 / 9.29 TIME/FRAME (ms): 3.69

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, Experimental results FILE NAME: sw2-1.aviSOURCE: DVD FORMAT: 320x240 (30 fps)LENGTH: 427 frames MODEL SIZE (pixels): 94x115 AVG/MAX ERROR (mm): - TIME/FRAME (ms): 3.57

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, Experimental results Location accuracy: –Errors in mm (in the face plane) using a ground-truth location of facial features –Average error below 4 mm, maximum error 14 mm –With CamShift: average error over 10 mm, maximum error 30 mm

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, Experimental results Execution time, per frame: –Off-the-self PC: AMD Athlon at 1.2 GHz –Average time below 5 ms, with 640x480 resolution, face size 100x120 pixels –With CamShift: average time about 10 ms, unable to work in one video sequence

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, Conclusions The tracking problem is decomposed into three main independent steps: –Vertical alignment –Horizontal alignment –Orientation estimation The process is fast, accurate and robust in the tested conditions It is exclusively based on integral projections

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, Conclusions The tracker is not affected by background distractors It can be applied either in color and grey-scale images Main limitation: maximum allowed movement (approx. 1 m/s, at 25 fps) Future work: improve the prediction step, e.g. with Kalman filters

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS Ginés García Mateos AVBPA’2003 GUILDFORD JUNE, Last This work has been supported by Spanish CICYT project DPI C03-01 Demo videos: Grupo PARP web page: Thank you very much