UNIVERSITY OF MURCIA (SPAIN) ARTIFICIAL PERCEPTION AND PATTERN RECOGNITION GROUP A PERCEPTUAL INTERFACE USING INTEGRAL PROJECTIONS Ginés García Mateos Sergio Fructuoso Muñoz Dept. de Informática y Sistemas University of Murcia - SPAIN

A PERCEPTUAL INTERFACE USING INTEGRAL PROJECTIONS Ginés G.M. Sergio F.M. PRIA St. Petersburg OCT, System overview User of the per- ceptual interface Camera Face detector and tracker FACE LOCATION: X: 6 Y: 0 Z: 3 FACE ORIENTATION: Roll: 7.7 Pitch: 0 Yaw: 3 FACE MOVEMENT: ΔX: -2 ΔY: 1 ΔZ: 0 3D movement and pose estimation Control signals (virtual world movement) Virtual world rendering MOVE FORWARD/BACKWARD LEFT/RIGHT ROTATE LEFT/RIGHT LOOK UP/DOWN

A PERCEPTUAL INTERFACE USING INTEGRAL PROJECTIONS Ginés G.M. Sergio F.M. PRIA St. Petersburg OCT, Face integral projections Definition. Let i(x,y) be an image, and R(i) a region in it: Vertical Integral Projection Horizontal Integral Projection P VR : {y min,..., y max }  R P HR : {x min,..., x max }  R P VR (y) = i(x,y);  (x,y)  R(i) P HR (x) = i(x,y);  (x,y)  R(i) When applied to human faces, typical patterns of projection appear. This is used to design a face detector and tracker. Pv(y): Vertical I.P. of the face Ph1(y): Horizon- tal I.P. of eyes’ region Ph2(y): Horizon- tal I.P. of mouth’s region

A PERCEPTUAL INTERFACE USING INTEGRAL PROJECTIONS Ginés G.M. Sergio F.M. PRIA St. Petersburg OCT, Face tracking with I.P. Green line: Projection model Red line: Projection instance P VFACE (y) Align P VFACE y y EYES P HEYES (x) Align P HEYES x x EYE1, EYE2 P VEYE1,P VEYE2 Align P VEYEi y y STEP 1 STEP 2 STEP 3 FACE Face tracking is a 3-step process. 1. Vertical alignment: compute move- ment and scale in vertical direction 2. Horizontal alignment: compute move- ment and scale in horizontal direct. 3. Orientation alignment: compute orienta- tion of the face

A PERCEPTUAL INTERFACE USING INTEGRAL PROJECTIONS Ginés G.M. Sergio F.M. PRIA St. Petersburg OCT, Pose estimation (i) Depth estimation: inversely proportional to the size of the head in the image. Approximated with the eye-to-eye distance Relative distances Roll estimation: approximated with the perceived angle of the eyes º-13.0º13.5º20.2º Roll angles

A PERCEPTUAL INTERFACE USING INTEGRAL PROJECTIONS Ginés G.M. Sergio F.M. PRIA St. Petersburg OCT, Pose estimation (ii) Pitch estimation: a heuristic measure is defined, using the vertical I.P. of the face Estimated pitch Yaw estimation: another heuristic, using the horizontal I.P. of the eyes Estimated yaw

A PERCEPTUAL INTERFACE USING INTEGRAL PROJECTIONS Ginés G.M. Sergio F.M. PRIA St. Petersburg OCT, Virtual environment The perceptual interface is used to control the movement in a virtual 3D world. We have used DirectX 9 and OpenCV 3. Virtual 3D world

A PERCEPTUAL INTERFACE USING INTEGRAL PROJECTIONS Ginés G.M. Sergio F.M. PRIA St. Petersburg OCT, Results and conclusion Conclusions Depth and roll estimation is very reliable. Pitch and yaw are less stable. Sample videos available at:  . . ,10/2004