MAPVI: Meeting Accessibility for Persons with Visual Impairments Sebastian Günther, Reinhard Koutny, Naina Dhingra, Markus Funk, Christian Hirt, Klaus Miesenberger, Max Mühlhäuser, Andreas Kunz 02.07.2019 PETRA 2019

Table of Content Introduction Related Work Our Approach Summary & Outlook Q & A 02.07.2019 PETRA 2019

Distributed workspaces (artifact spaces) Introduction Distributed workspaces (artifact spaces) Logical context between workspaces  3D information space Explicit information (artifacts) is supplemented by non-verbal communication (NVC) Chosen application: Metaplan brainstorming method 02.07.2019 PETRA 2019

«Over there …. is the inaccessible information!» Introduction «Over there …. is the inaccessible information!» Setting the stage: Blind & sighted users in a collaborative brainstorming meeting Explicit information accessible using e.g. screen readers Audio channel “occupied” by ongoing discussion A lot of information is not made explicit, but transferred via NVC elements 02.07.2019 PETRA 2019

NVCs occur in the task space 136 NVCs are listed in [1] Introduction NVCs occur in the task space 136 NVCs are listed in [1] 55% of communication is non- verbal [2] [1] Brannigan, C.R.; Humphries, D.A.: «Human Non-Verbal Behavior, A Means of Communication»; Blurton-Jones (Ed.): Ethological Studies of Child Behavior; Cambridge University Press; 1972 [2] Mehrabian, A.: «Nonverbal Communication»; 1972 02.07.2019 PETRA 2019

Tracking pointing gestures above tabletop using Kinect [1] Related Work Tracking pointing gestures above tabletop using Kinect [1] Tracking pointing gestures using Leap Motion [2] Postures at a table [3] [1] Kunz, A.; Alavi, A.; Sinn, P.: «Integrating Pointing Gesture Detection for Enhancing Brainstorming Meetings using Kinect and PixelSense»; 8th International Conference on Digital Enterprise Technology; 205-212; 2014 [2] Schnelle-Walka, D.; Alavi, A.; Ostie, P.; Mühlhäuser, M.; Kunz, A.: «A Mimd Map for Brainstorming Sessions with Blind and Sighted Persons»; 14th International Conference on Computers Helping People with Special Needs; 214-219; 2014 [3] Kunz, A.; Schnelle-Walka, D.; Alavi, A.; Pölzer, S.; Mühlhäuser, M.; Miesenberger, K.: «Making Tabletop Interaction Accessible for Blind Users»; Interactive Tabletops and Surfaces; 327-332; 2014 02.07.2019 PETRA 2019

Our Approach NVCs to be captured Tracking methods Gaze direction Deictic gestures Postures (nodding, shrugging) Tracking methods SLAM (Simultaneous Localization and Mapping Structured light, time-of-flight (e.g. Kinect) Optical flow (e.g. RGB camera) Bluetooth, WiFi, etc. 02.07.2019 PETRA 2019

False alerts to blind & visually impaired should be avoided Our Approach No NVC should be missed Multiple sensors Various sensor types, including audio False alerts to blind & visually impaired should be avoided Logical context between measured signals Temporal context between measured signals Classification of behavior 02.07.2019 PETRA 2019

Eyetracker for gaze detection Our Approach Eyetracker for gaze detection Will be used as reference RGB camera for gaze detection  OpenGaze [1] SMI remote eye tracking device [1] Zhang, X.; Sugano, Y.; Andreas Bulling, A.: «Evaluation of A Appearance-Based Methods and Implications for Gaze-Based Applications»; Proceedings ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), 2019 02.07.2019 PETRA 2019

Eye tracker at ideal distance of 0.7 m, accuracy measured on screen Our Approach Eye tracker at ideal distance of 0.7 m, accuracy measured on screen Eye tracker at ideal distance of 0.7 m, accuracy measured on whiteboard Camera at ideal distance of 0.7 m, accuracy measured on whiteboard Camera on whiteboard, accuracy measured on whiteboard 02.07.2019 PETRA 2019

RGB for pose detection  OpenPose Our Approach RGB for pose detection  OpenPose [1] Cao, Z.; Hidalgo, G.; Simon, T.; Wei, S.-E.; Sheikh, Y.: «OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields»; IEEE Transactions on Pattern Analysis and Machine Intelligence; 2018 02.07.2019 PETRA 2019

Future Work Development of an accessible Metaplan tool Deep learning approaches for gesture classification Developing suitable output devices for spatial awareness of NVCs User studies on cognitive maps of BVIP and their relation to real spaces 02.07.2019 PETRA 2019

Q & A Thank you for your attention guenther@tk.tu-darmstadt.de reinhard.koutny@jku.at 02.07.2019 PETRA 2019