Vineeth Balasubramanian Shayok Chakraborty Sreekar Krishna Sethuraman Panchanathan C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING CUbiC Human Centered Machine Learning in a Social Interaction Assistant for Individuals with Visual Impairments ARIZONA STATE UNIVERSITY December 10, 2009

CUbiC Outline Introduction – Human Centered Machine LearningOverview of the Social Interaction AssistantExamples of Human Centered Machine LearningRelated Machine Learning Contributions

CUbiC Outline Introduction – Human Centered Machine LearningOverview of the Social Interaction AssistantExamples of Human Centered Machine LearningRelated Machine Learning Contributions

CUbiC Assistive Devices for the Visually Impaired Need to enrich the interaction of blind individuals with other individuals

CUbiC Social Interaction Hand Gestures Posture Eye Gaze Hair Clothing About 65% of the information during social interaction is communicated using non-verbal cues Krishna et al. A Systematic Requirements Analysis and Development of an Assistive Device to enhance the Social Interaction of people who are Blind or Visually Impaired. ECCV 2008

CUbiC Social Interaction for the blind The Need 1.1 million people in the US are legally blind, 37 million worldwide They face the fundamental challenges during social interaction Necessitates design of a Social Interaction Assistant for the visually impaired The Challenges Challenges such as person recognition/tracking, head pose estimation, gesture recognition, expression recognition to be addressed

CUbiC “The Human-Machine System” Human Machine  The human and the machine should be treated as a single entity  Both should jointly be used to deliver outputs

CUbiC Human-Centered Machine Learning Use the ability of the user to derive effective solutions

CUbiC Outline Introduction – Human Centered Machine LearningOverview of the Social Interaction AssistantExamples of Human Centered Machine LearningRelated Machine Learning Contributions

CUbiC The Social Interaction Assistant Haptic Belt Krishna et al. A wearable face recognition system for individuals with visual impairments. ACM SIG ASSETS, 2005 McDaniel et al. Using a Haptic Belt to convey Non Verbal Communication Cues during Social Interactions to Individuals who are Blind. IEEE HAVE, 2008

CUbiC Outline Introduction – Human Centered Machine LearningOverview of the Social Interaction AssistantExamples of Human Centered Machine LearningRelated Machine Learning Contributions

Integrated Face Localization/Recognition CUbiC Wearable camera on user Face detection (Viola – Jones: Adaboost) Face detected to my far right Face Recognition – an integral component in a Social Interaction Assistant

Integrated Face Localization/Recognition CUbiC Edwards et al. A Pragmatic Approach to the Design and Implementation of a Vibrotactile Belt and its Applications, IEEE HAVE, 2009 Direction conveyed through a vibrotactile cue in the haptic belt Human in the loop simplifies the problem The individual turns in the direction of vibration – camera captures frontal images

User Conformal Confidence Measures CUbiC  CP framework is used to quantify the level of certainty or “confidence” in decision making in machine learning applications.  Results are well-calibrated  Frequency of errors,, made by the system is exactly controlled by the confidence level, 1 -, defined by the user Shafer, Vovk. A Tutorial on Conformal Predictions, JMLR 2008

CUbiC Outline Introduction – Human Centered Machine LearningOverview of the Social Interaction AssistantExamples of Human Centered Machine LearningRelated Machine Learning Contributions

Online Active Learning for Person Recognition CUbiC Current Unlabeled Data Point Class A P-Value: p1 Class B P-Value: p2 Class C P-Value: p3 Largest eigenvalue used as a measure of discrepancy between the p-values V. Balasubramanian, S. Chakraborty and S. Panchanathan. Generalized Query by Transduction for Online Active Learning, IEEE ICCV 2009

Context Aware Batch Mode Active Learning CUbiC Data from a video stream  Select a batch of informative samples from a video stream to update the classifiers  Use of context based priors can further help select the salient instances  Select a batch of informative samples from a video stream to update the classifiers  Use of context based priors can further help select the salient instances Select the salient examples

Learning from Multiple Sources CUbiC Learning from Multiple Sources of information

CUbiC Questions ??.. Thank You !!..