C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING CUbiC ARIZONA STATE UNIVERSITY Sreekar Krishna, Vineeth Balasubramanian, Sethuraman (Panch) Panchanathan C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING CUbiC Enriching Social Situational Awareness in Remote Interactions - Insights and Inspirations from Disability Focused Research ARIZONA STATE UNIVERSITY

C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING 2 CUbiC Recognition & Learning Sensing & Processing Interaction & Delivery Assistive Tech. Technologies for Daily Living Rehabilitation Assessment & Training Medical Decision Support

C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING 3 CUbiC Success Stories - iCARE Reader Camera Phase prototypes developed Deployed in ASU and AzSDB Phase Personal size Customization capabilities Phase 3 In development Incorporating high resolution digital cameras on the glasses CENTER FOR COGNITIVE UBIQUITOUS COMPUTING

4 CUbiC Success Stories - NoteTaker The assistive technology for low vision and legally blind students for taking notes in the classroom Zoomed video of the lecturer’s presentation in real time Student notes with digital ink CENTER FOR COGNITIVE UBIQUITOUS COMPUTING Multi Touch Camera Control Winner of 2010 MS Imagine Cup Award

C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING Shopping Assistance 5

C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING Social Assistance - Origin of the Problem 6 Dr. Terri Hedgpeth – Director, Disability Resource Center, ASU Focus Group Study of Individuals who are Blind: “It would be nice to walk into a room and immediately get to know who are all in front of me before they start a conversation”. “It would be great to walk into a bar and identify a friend”.

C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING Enactor (Encoding) Recipient (Decoding) Interpersonal Social Interactions 7 N. Ambady and R. Rosenthal, “Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis.,” Psychological Bulletin. Vol. 111(2), vol. 111, Mar. 1992, pp Body Voice Speech Face 27% 19% Visual 18% 35% Audio Verbal Non-verbal 65% 35% Social Touch What role does touch play?

C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING Socio-Behavior Example – Hand Shake 8 Social Interactions Sensory Perceptual Cognitive Motor Handshake Step 1: Eye Contact Step 3: Move (Proxemics) Step 2: Intent to interact Step 4: Shake hands Step 5: Conversation distance

C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING Social Interactions 9 Social Touch Social Sight Social Hearing Social Stimulation Social Reciprocation Face Body Voice Social Cognition Social Stimulation Social Cognition Social Reciprocation Social Situational Awareness

C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING How many people? Where are they located? What are their facial expressions? Eye Gaze Eye Contact Body Mannerisms SSA in Various Settings 10 Social AssistanceDecision Making Remote Collaborations TeamSTEPPS Leadership Mutual Support Communication Attitude Situation Monitoring Patient Safety Expressing Opinion Managing Conflict Making Decision Speed of Decision Interaction with Colleagues Difficulty Establishing Rapport

C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING CUbiC Multidisciplinary Team 11 Technology, Psychology & Human Comm. Tech. Dissemination & Validation Assistive Technology Team Dr. Sethuraman Panchanathan Comp. Sci. ASU Human Centered Multimedia Dr. Sethuraman Panchanathan Comp. Sci. ASU Human Centered Multimedia Dr. John Black Jr. Comp. Sci. ASU Assistive Tech. Specialist Dr. Vineeth Balasubramanian Comp. Sci. ASU Decision Sys. & Risk Analysis Dr. Prasad Boradkar School of Design, ASU Assistive Tech. Design, Interdisciplinary Design Initiatives Dr. Jameson Wetmore Consortium of Science, Policy & Tech., ASU Asst. Tech. Ethics, Practices & Effective Dissemination Dr. Terri Hedgpeth Dir. Disability Resource Center, ASU Asst. Tech. Usability Expert, Early Adoption Specialist Dr. Michele Shiota Psychology, ASU Interpersonal Interactions, Facial Expression, Dyadic Communications Dr. Don Homa Psychology, ASU Visual Perception, Working Memory, Haptic Concepts Dr. Artemio Ramirez Human Comm., ASU Remote Interactions and Communications, Modeling Professional Meetings, Conflict Sreekar Krishna Elec. Engg. ASU Integration Engg. Behavioral Psychology and Communication Team

C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING Social Interaction Assistant 12 Stereotypy Face ReadingSocial Scene Analysis

C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING Stereotypy Any non-functional repetitive behavior Two main causes for stereotypy Lack of sensory feedback Lack of cognitive feedback Methods of control Stereotypy Curtail Behavior immediately Reward / Punishment Intervention Do not intervene directly Develop cognitive replacement Self Monitoring 13 Body Rocking is the most prevalent stereotypy for people who are blind and visually impaired

C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING Proposed solution X Y Z Rocking Non - Rocking 14 Rocking action can be recognized with an accuracy of 94% within 2 seconds Behavioral Psychology literature shows that one rock action is approximately 2.2 seconds long. Effectively, recognizing a rocking behavior well within one rock cycle.

C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING Dyadic Interaction – Face Reading 15 Camera Social Interaction Asst.

C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING Dynamic Delivery of Facial Mannerisms 16

C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING Social Gaze & Interaction Space 17 Intimate Personal Social Public 1.5’ 4’ 12’25’ 0’ Interpersonal Space

C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING Modeling Distance & Direction through Face Detection 18 Module 1: Color Analysis Module 2: Markov Random Field LPCD Module 3: Evidence Aggregation

C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING Structured Mode Searching Particle Filter (SMSPF) Initial Estimate Corrected Estimate Example Search Windows Motivation: Weak Temporal Redundancy Approach: Stochastic Search over a large search space (Color Histogram Comparison) Result: Approximate Estimate Motivation:ComplexObject Structure & Abrupt Motion Approach: Deterministic Search over a small probable search space (Histogram of Gradients with Chamfer Match) Result: Accurate Estimate 19

C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING Face/Person Detection/Tracking 20 Face Detection Person Detection Tracking Model Deliver

C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING Social Scene Delivery System 21

C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING Conveying Body Mannerisms 22 Body Posture Body Gestures Enactor Recipient Social Mirror Social Interaction Assistant

C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING Disability & Deficit Inspired Computing Disabled Population Activities of Daily Living (ADL) Observe Identify Barriers to ADL Design and Develop Assistive Tech. Refine Disseminate Extrapolate to the general population Did you know the typewriter was invented for the blind? Did you know the typewriter was invented for the blind? Blindness is only a concept

C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING Mayo Multi-disciplinary Simulation Center 24 Doctors, Nurses, Professionals, etc. On-body Affect Sensors Environment Affect Sensors Vision Audio Automated monitoring of group dynamics to determine communication breakdowns Automatic evaluation of the social affinity between team members Leadership evaluation and nomination through long term monitoring of teams and individuals

C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING Assisting Remote Interactions 25 Managing Conflict Making Decision Expressing Opinion Challenges in virtual teams compared to face-to-face teams Top five challenges faced during virtual team meeting Different leadership styles Insufficient time to build relations Colleague not participate Method of decision making Speed of decision making Personal challenges during virtual team meetings Difficulty seeing the whole picture Absence of collegiality Sense of isolation Reliance on and telephone Difficulty establishing rapport and trust Inability to read non-verbal cues The Challenges of Working in Virtual Teams: Virtual Teams Survey Report RW3 CultureWizard, 2010.

C ENTER FOR C OGNITIVE U BIQUITOUS C OMPUTING Socio-Behavioral Computing 26 Sensor & Actuator Technologies Human Computer Interaction Machine Learning and Pattern Recognition SBC Socio Behavioral Computing Affective Computing Social Robotics Human Communication Dynamics Human Centered Computing User Behavior Modeling Questions?