1. UNIVERSITY OF MURCIA (SPAIN) ARTIFICIAL PERCEPTION AND PATTERN RECOGNITION GROUP A PERCEPTUAL INTERFACE USING INTEGRAL PROJECTIONS Ginés García Mateos ginesgm@um.es Sergio Fructuoso Muñoz sergiofr@ono.com Dept. de Informática y Sistemas University of Murcia - SPAIN

2. A PERCEPTUAL INTERFACE USING INTEGRAL PROJECTIONS Ginés G.M. Sergio F.M. PRIA-7-2004 St. Petersburg OCT, 2004 2 System overview User of the per- ceptual interface Camera Face detector and tracker FACE LOCATION: X: 6 Y: 0 Z: 3 FACE ORIENTATION: Roll: 7.7 Pitch: 0 Yaw: 3 FACE MOVEMENT: ΔX: -2 ΔY: 1 ΔZ: 0 3D movement and pose estimation Control signals (virtual world movement) Virtual world rendering MOVE FORWARD/BACKWARD LEFT/RIGHT ROTATE LEFT/RIGHT LOOK UP/DOWN

3. A PERCEPTUAL INTERFACE USING INTEGRAL PROJECTIONS Ginés G.M. Sergio F.M. PRIA-7-2004 St. Petersburg OCT, 2004 3 Face integral projections Definition. Let i(x,y) be an image, and R(i) a region in it: Vertical Integral Projection Horizontal Integral Projection P VR : {y min,..., y max }  R P HR : {x min,..., x max }  R P VR (y) = i(x,y);  (x,y)  R(i) P HR (x) = i(x,y);  (x,y)  R(i) When applied to human faces, typical patterns of projection appear. This is used to design a face detector and tracker. Pv(y): Vertical I.P. of the face Ph1(y): Horizon- tal I.P. of eyes’ region Ph2(y): Horizon- tal I.P. of mouth’s region

4. A PERCEPTUAL INTERFACE USING INTEGRAL PROJECTIONS Ginés G.M. Sergio F.M. PRIA-7-2004 St. Petersburg OCT, 2004 4 Face tracking with I.P. Green line: Projection model Red line: Projection instance P VFACE (y) Align P VFACE y y EYES P HEYES (x) Align P HEYES x x EYE1, EYE2 P VEYE1,P VEYE2 Align P VEYEi y y STEP 1 STEP 2 STEP 3 FACE Face tracking is a 3-step process. 1. Vertical alignment: compute move- ment and scale in vertical direction 2. Horizontal alignment: compute move- ment and scale in horizontal direct. 3. Orientation alignment: compute orienta- tion of the face

5. A PERCEPTUAL INTERFACE USING INTEGRAL PROJECTIONS Ginés G.M. Sergio F.M. PRIA-7-2004 St. Petersburg OCT, 2004 5 Pose estimation (i) Depth estimation: inversely proportional to the size of the head in the image. Approximated with the eye-to-eye distance. 0.6740.9471.184 1.636 Relative distances Roll estimation: approximated with the perceived angle of the eyes. -33.4º-13.0º13.5º20.2º Roll angles

6. A PERCEPTUAL INTERFACE USING INTEGRAL PROJECTIONS Ginés G.M. Sergio F.M. PRIA-7-2004 St. Petersburg OCT, 2004 6 Pose estimation (ii) Pitch estimation: a heuristic measure is defined, using the vertical I.P. of the face Estimated pitch Yaw estimation: another heuristic, using the horizontal I.P. of the eyes. -7011 Estimated yaw 19-2 -9

7. A PERCEPTUAL INTERFACE USING INTEGRAL PROJECTIONS Ginés G.M. Sergio F.M. PRIA-7-2004 St. Petersburg OCT, 2004 7 Virtual environment The perceptual interface is used to control the movement in a virtual 3D world. We have used DirectX 9 and OpenCV 3. Virtual 3D world

8. A PERCEPTUAL INTERFACE USING INTEGRAL PROJECTIONS Ginés G.M. Sergio F.M. PRIA-7-2004 St. Petersburg OCT, 2004 8 Results and conclusion Conclusions Depth and roll estimation is very reliable. Pitch and yaw are less stable. Sample videos available at: http://dis.um.es/~ginesgm/fip/percint.html  . . ,10/2004