1. On Wikipedia you can find the following definition of NUI: “In computing, a natural user interface, or NUI, or Natural Interface is the common parlance used by designers and developers of human-machine interfaces to refer to a user interface that is effectively invisible, and remains invisible as the user continuously learns increasingly complex interactions. The word natural is used because most computer interfaces use artificial control devices whose operation has to be learned.” Ultimately NUI stands for the acronym of "Natural User Interface“, that’s to say it’s a user interface designed with the goal of being used in a way as natural as possible. The main goal of NUI design consists in achieving: immediate experience progressive learning direct interaction, minimum cognitive load, low physical effort of interaction, an overall pleasant user experience.

2. Goal Design and implementation of a Natural User Interface based on Mid Air Gestures interaction. Instruments Kinect - Kinect V2 (coming soon … ) Kinect for Windows SDK Kinect Studio Device API Prime Sense OpenNI (not mandatory) Prime Sense NITE2 tracking libraries (not mandatory) Other available API & IDEs Programming languages C/C++, C# WPF, Windows Form Applications Industrial, Gaming.

3. Interaction metaphors elicitation (done). Gestures set definition under a user centric point of view (done). Gesture set usability assessment: Agreement (done) Guessability (done) Memorability (done) Ergonomics (improving) Gesture recognition (what about working together?) Accuracy (to do) Overall User Experience (to do)

4. User tracking Limited working area available in industrial environment -> no optimal distance from sensor. User occlusion.  Pseudo casual IR pattern projection  Depth image acquisition through IR sensor  Disparity matrix evaluation  Calibration on metric scale basing on each spot correlation with 64 near ones. User tracking is obtained through two phases:  Scene depht map acquisition  Skeletoning

5. Skeletoning consists in identifying:  number,  position,  skeletal junctions (joints) of users in the scene under sensor’s active range Depth imagePixels labelling3D joint proposal

6. Problems  Joints loos due to self occlusion during interaction.  False joint detection due to user’s working devices handling. If a digital multimeter does that, imagine what personal protective equipment could do!  Undesired false interactions -> robust gesture and attention detection.

7. Study of the state of the art of gesture recognition algorithms: Features detection. Classification systems (comparison between Neural Networks, Hidden Markov Models, SVM and so on). Implementation of a robust mid air gesture detection system to be used in a NUI aimed at navigation of an AR maintenance manual (4 gestures, remember 7 +/- 2 is the magic number!). System quality analysis: Accuracy FPR, ROC curve and so on. Workflow Detailed Kinect and Kinect V2 features analysis Device, API, libraries and IDE selection. State of the art of gesture recognition. Implementation. User test. Prototyping. Quality analysis.

8. Goal Extension and improvement of our testing framework aimed at mid air gestures NUIs’ usability assessment. Instruments Kinect - Kinect V2 (coming soon … ) Kinect for Windows SDK Kinect Studio OpenNI Other available API & IDEs Applications Industrial, Medical, Gaming.

9. Framework Usability metrics assessment: Agreement Guessability Ergonomics Goodness Easyness Accuracy

10. Further validation of previous CE metric User test Data analysis Shoulder-arm kinematics model improvement exploiting new Kinect V2 feature (i.e. finger tracking to take into account arm and wrist torsion).

11. Engine Implementation of an in-house made ergonomics assessment engine basing on state of the art of upper body kinematics model Metric validation through comparison with other used metrics (Borg CR10, Likert Scale, previous CE) Workflow Kinect V2 features analysis Shoulder-Arm kinematics model study and software implementation Reliability testing Framework integration NUI usability evaluation

12. Thanks for your attention! Vito Manghisi Contact information: Dipartimento di Meccanica, Matematica e Management Viale Japigia 182, Bari, IT Terza palazzina (stanza prof. Monno) Tel. +393334021125 Email vitomanghisi71@gmail.com