Predicting Visual Search Targets via Eye Tracking Data Dr. Ali Borji University of Central Florida Fawad Ahmed University of Central Florida Manisha Gupta manishagupta@utexas.edu University of Texas at Austin I. Purpose & Applications III. Gaze Information V. Euclidean Distances Participants’ gaze information was plotted against the collages Size of the fixation points corresponds to the duration Predict visual search targets in closed and open world settings Develop a learning mechanism over an unknown set of targets Applications: Implemented in cameras to scan the environment and pinpoint where a lost object (ex: set of keys) is located II. Dataset Extracted book covers Search target (top left) 6 participants searched for a target within a collage 20 variations of each collage 5 search targets (shown next to each collage) Yields 600 search tasks *Generated for all 600 search tasks *Repeated for the 20 variations of each collage per search target *Generated for 5 search tasks for each of the three types of collages Lower distances demonstrates higher similarities between the search target and the fixation point IV. Convolutional Neural Network Pre-trained Siamese model Extracts 2x512 vector features from linear layer Compares and outputs image similarities Pipeline: Architecture: Search Target ConvNet Decision Network Image Similarity Extracted Book Cover 1 Amazon book covers: Distinct structure/Distinct color Extracted Book Cover 2 ---- Extracted Book Cover N *Repeated for all six users The value of average Euclidean distances and lingering percentages imply that the set of Amazon collages was the most difficult search task. O’Reilly book covers: Low structure/Distinct color Decision Network Decision Network Branch Network 1 Image 1 Conv + ReLu Max pooling Max Pooling VI. Future Work VII. Acknowledgements Send feature vector and search targets to a binary SVM Add weights to fixation points to account for gaze duration Thank you to Dr. Mubarak Shah and Dr. Niels Lobo for their guidance during the program. The research project presented is funded by NSF REU Summer Program 2016. Un-shared (pseudo-siamese) Shared (siamese) Image 2 Conv + ReLu Max pooling Max Pooling Branch Network 2 Mugshots: Low structure/Low color