The geometry of the system consisting of the hyperbolic mirror and the CCD camera is shown to the right. The points on the mirror surface can be expressed as: The intersection between a line through the origin of the coordinate system centered at one of the focal points of the mirror in the direction of the vector v can be described as: With a rotation between the camera and mirror coordinate systems a point on the mirror surface can then be calculated from the pixel coordinates in the captured image as: The matrix K is the camera calibration matrix: which relates points [x,y,z]T in space with image pixel coordinates u and q are normalized image coordinates. The Matlab Calibration Toolbox written by J.-Y. Bouguet was used for calibration. Imaging System for the Measurement of Head and Body Motion for the RIT-Wearable-Eye-Tracker Constantin Rothkopf, Advisor: Dr. Jeff Pelz Real part of spherical harmonics: vertically: 0  l  4 horizontally: -l  m  +l Research in visual perception and attention using eye movements has moved from signal detection paradigms and the assessment of the mechanics and metrics of eye movements to the study of complex behavior in natural tasks. In such tasks, subjects are able to move their head and body and interact in a purposeful way with a changing environment. Under such circumstances, the analysis of the eye movements is more difficult, because the eye tracker does not record the subject's head movements. Recovering the head movements can give additional information about the type of eye movement that was carried out, the overall gaze change in world coordinates, and insight into high-order perceptual strategies. The aim of this senior project is to develop a system that can make it possible to recover the head movements of a subject during natural tasks. The proposed solution utilizes an omnidirectional vision sensor consisting of a small CCD video camera and a hyperbolic mirror. The camera is mounted on an ASL eye tracker and records an image sequence at 60 Hz. Several algorithms for the estimation of rotational movement from omnidirectional image sequences have been developed. Because of the low resolution of a standard video image, a method based on the spherical harmonic decomposition developed by Makadia and Daniilidis (2003) has been implemented. The image sequence captured by the omnidirectional camera is remapped onto a sphere and represented in ,  space. The spherical harmonics: with the P l m being associated Legendre functions, are a set of orthonormal basis Functions on the sphere. The reprojected images are decomposed using the discrete spherical harmonics transformation. Under a rotation, parameterized with ZYZ Euler Angles as a sequence of two rotations: the coefficients of the decomposition can be expressed as: with the P l mn being generalized associated Legendre functions. The advantage of this parameterization is, that the unknown variables appear in the exponential terms and that the P l mn can be calculated as sums of binomial coefficients. The resulting equation was minimized using T.C.Kelley’s Matlab library implementation of Broyden’s method. A two state Hidden Markov Model was introduced by D. Salvucci (1999) for the classification of fixations and saccades from eye movement recordings in equation solving. This model was extended by incorporating the estimated head movement as a second observation variable in order to classify fixations, saccades, smooth pursuits, and VORs. The resulting probability distributions are modeled as bivariate Normal distributions. The initial guesses for the parameters of these distributions were obtained from experimental data. The Baum-Welsh algorithm is used to estimate the parameters and the transition probabilities from recorded data. The Hidden Markov Model is then used to decode the sequence of eye and head velocities in order to classify the types of eye movements. While the rotational motion estimates using synthetic images were accurate to less than one degree, a comparable accuracy could not be reached with the image sequence from the omnidirectional camera. A Fastrack system was used to obtain ground truth for the measurements. Further work should investigate methods for improving the rotational motion estimation. An increase of the spatial resolution of the camera is expected to have a significant impact. The classification algorithm was used on a sequence of 3 minutes length. One subject carried out sequences of smooth pursuits, VORs, fixations and saccades. The proposed algorithm classified the fixations, saccades, and VORs with 100% accuracy, and smooth pursuits with 65%. Further validation by trained experts should used to evaluate the algorithm. Introduction Specific Aim The omnidirectional image sensor Rotation estimation Eye movement classification Results from rotational motion estimation with synthetic images of size: 512x512: angle l  5 l  8 l  12  =5º 4.08º 5.62º 5.74º  =10º 9.26º 10.38º 10.57º  =25º 25.15º 24.19º 25.20º Schematic of the Hidden Markov Model Preliminary results Geometry of omnidirectional camera Remapping of image References: A.Makadia, K.Daniilidis: ‘Direct 3D-Rotation Estimation D.D.Salvucci: ’Mapping Eye Movements to Cognitive Processes’,T.Svoboda, T. Pajdla: ‘Epipolar Geometry for Central Catadioptric from Spherical Images via a generalized shift theorem’ PHD Thesis, Carnegie Mellon University, 1999Cameras’, IJCV 49(1), 2002