1. A novel depth-based head tracking and facial gesture recognition system by Dr. Farzin Deravi– EDA UoK Dr Konstantinos Sirlantzis– EDA UoK Shivanand Guness – EDA UoK Dr. Mohamed Sakel - EKHUFT Dr. Matthew Pepper - EKHUFT

2. Overview Clinical Background Objectives Kinect : RGB-D sensor Technical Approach Evaluation Technique Experimentation Results Conclusion Future Work

3. Clinical Background Individuals with conditions such as Motor Neuron Disease (MND), Cerebral Palsy (CP) and Multiple Sclerosis (MS): – Can lose the ability to speak – May only make small head/facial movements in cases where the person has MND or MS Capture and interpret the intentions and messages of the patients from their limited movement

4. Objectives Develop reliable automatic gesture recognition system – Tracking head movement; – Detect facial gestures such as eye blink, wink etc. Adaptive system to adapt to condition or user over time Develop a low cost assistive device

5. Kinect : RGB-D sensor

6. Kinect : RGB-D sensor(cont.) Projects a known pattern (Speckles) in Near-Infrared light. CMOS IR camera observes the scene. Calibration between the projector and camera has already been carried out and is known. Projection generated by a diffuser and diffractive element of IR light

7. Technical Approach

8. Technical Approach(cont.) Depth Map RGB Image Area of object nearest to sensor

9. Experimentation SetupFitts’ Test target screen

10. Evaluation Technique Fitts’ test for HCI is used to evaluate the tracking algorithms Initially developed by Paul Fitts in 1953 to model human movement Adapted to HCI by Scott MacKenzie in the 1992 ISO 9431-9:2000 (Ergonomic requirements for office work with visual display terminals (VDTs)—Part 9— Requirements for non-keyboard input devices) – is based on Fitts’ Test

11. Fitts’ Test Two Key Parameters

12. Fitts’ Test (cont) Effective index of difficulty ID e = log 2 (D/W e +1) – where (D) is the distance from the home to the target and W e is the effective width of the target. W e, is calculated Effective Width W e = 4.133 x SD – where SD is the standard deviation of the selection coordinates.

13. Fitts’ Test (cont)

14. Fitts’ Test Evaluation Width (W)Distance (D)Index of Difficulty(ID) 2550 1.5850 50200 2.3219 25400 4.0875 50400 3.1699

15. Result DevicesEffective Throughput (TP e ) DwellBlinkEyebrows Standard Mouse(ms) 0.84n/a CameraMouse (cm) 0.48n/a SmartNav (sn) 0.42n/a Vision head tracker (using webcam) 0.21 (ht-dwell) 0.15 (ht-blink) 0.08 (ht-brows) RGB-D head tracker (using Kinect) 0.30 (kht-dwell) 0.28 (kht-blink) 0.09 (kht-brows)

16. Result Throughput and Effective Throughput

17. Result – Index of Difficulty

18. Result – Effective Index of Difficulty

19. Result – Index of Difficulty

20. Result – Effective Index of Difficulty

21. Conclusion RGB-D head tracking system is shown to have an improved performance over the vision-based head tracking system – TP e of dwell clicking increased by a third (from 0.21 to 0.30 bits per second) – TP e of blink clicking double (from 0.15 to 0.28 bits per second) The eye blink detection algorithm performance is quite near to the performance of the dwell click switch

22. Future Work Increase the robustness of the HeadTracker Investigate the impact of the different facial gesture switches on performance Investigate additional facial gesture switches such as mouth (open-close), tongue etc. Conduct Fitts’ Test experimentation using about 10- 20 participants – with healthy volunteer

23. THANK YOU Project Website : http://www.eda.kent.ac.uk/research/theme_project.aspx?pid=164 Email : spg23@kent.ac.uk