1. MAPVI: Meeting Accessibility for Persons with Visual Impairments
Sebastian Günther, Reinhard Koutny, Naina Dhingra, Markus Funk, Christian Hirt, Klaus Miesenberger, Max Mühlhäuser, Andreas Kunz
PETRA 2019

2. Table of Content Introduction Related Work Our Approach
Summary & Outlook
Q & A
PETRA 2019

3. Distributed workspaces (artifact spaces)
Introduction
Distributed workspaces (artifact spaces)
Logical context between workspaces  3D information space
Explicit information (artifacts) is supplemented by non-verbal communication (NVC)
Chosen application: Metaplan brainstorming method
PETRA 2019

4. «Over there …. is the inaccessible information!»
Introduction
«Over there …. is the inaccessible information!»
Setting the stage: Blind & sighted users in a collaborative brainstorming meeting
Explicit information accessible using e.g. screen readers
Audio channel “occupied” by ongoing discussion
A lot of information is not made explicit, but transferred via NVC elements
PETRA 2019

5. NVCs occur in the task space 136 NVCs are listed in [1]
Introduction
NVCs occur in the task space
136 NVCs are listed in [1]
55% of communication is non- verbal [2]
[1] Brannigan, C.R.; Humphries, D.A.: «Human Non-Verbal Behavior, A Means of Communication»; Blurton-Jones (Ed.): Ethological Studies of Child Behavior; Cambridge University Press; 1972
[2] Mehrabian, A.: «Nonverbal Communication»; 1972
PETRA 2019

6. Tracking pointing gestures above tabletop using Kinect [1]
Related Work
Tracking pointing gestures above tabletop using Kinect [1]
Tracking pointing gestures using Leap Motion [2]
Postures at a table [3]
[1] Kunz, A.; Alavi, A.; Sinn, P.: «Integrating Pointing Gesture Detection for Enhancing Brainstorming Meetings using Kinect and PixelSense»; 8th International Conference on Digital Enterprise Technology; ; 2014
[2] Schnelle-Walka, D.; Alavi, A.; Ostie, P.; Mühlhäuser, M.; Kunz, A.: «A Mimd Map for Brainstorming Sessions with Blind and Sighted Persons»; 14th International Conference on Computers Helping People with Special Needs; ; 2014
[3] Kunz, A.; Schnelle-Walka, D.; Alavi, A.; Pölzer, S.; Mühlhäuser, M.; Miesenberger, K.: «Making Tabletop Interaction Accessible for Blind Users»; Interactive Tabletops and Surfaces; ; 2014
PETRA 2019

7. Our Approach NVCs to be captured Tracking methods Gaze direction
Deictic gestures
Postures (nodding, shrugging)
Tracking methods
SLAM (Simultaneous Localization and Mapping
Structured light, time-of-flight (e.g. Kinect)
Optical flow (e.g. RGB camera)
Bluetooth, WiFi, etc.
PETRA 2019

8. False alerts to blind & visually impaired should be avoided
Our Approach
No NVC should be missed
Multiple sensors
Various sensor types, including audio
False alerts to blind & visually impaired should be avoided
Logical context between measured signals
Temporal context between measured signals
Classification of behavior
PETRA 2019

9. Eyetracker for gaze detection
Our Approach
Eyetracker for gaze detection
Will be used as reference
RGB camera for gaze detection  OpenGaze [1]
SMI remote eye tracking device
[1] Zhang, X.; Sugano, Y.; Andreas Bulling, A.: «Evaluation of A Appearance-Based Methods and Implications for Gaze-Based Applications»; Proceedings ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), 2019
PETRA 2019

10. Eye tracker at ideal distance of 0.7 m, accuracy measured on screen
Our Approach
Eye tracker at ideal distance of 0.7 m, accuracy measured on screen
Eye tracker at ideal distance of 0.7 m, accuracy measured on whiteboard
Camera at ideal distance of 0.7 m, accuracy measured on whiteboard
Camera on whiteboard, accuracy measured on whiteboard
PETRA 2019

11. RGB for pose detection  OpenPose
Our Approach
RGB for pose detection  OpenPose
[1] Cao, Z.; Hidalgo, G.; Simon, T.; Wei, S.-E.; Sheikh, Y.: «OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields»; IEEE Transactions on Pattern Analysis and Machine Intelligence; 2018
PETRA 2019

12. Future Work Development of an accessible Metaplan tool
Deep learning approaches for gesture classification
Developing suitable output devices for spatial awareness of NVCs
User studies on cognitive maps of BVIP and their relation to real spaces
PETRA 2019

13. Q & A Thank you for your attention guenther@tk.tu-darmstadt.de
PETRA 2019