1. Knowledge Systems Lab JN 1/15/2016 Facilitating User Interaction with Complex Systems via Hand Gesture Recognition MCIS Department Knowledge Systems Laboratory Jacksonville State University Joshua R. New, Erion Hasanbelliu, and Mario Aguilar

2. Knowledge Systems Lab JN 1/15/2016 Outline Motivation System Architecture Implementation Overview Proposed Approach Demonstration Future Directions

3. Knowledge Systems Lab JN 1/15/2016 Motivation Gesturing is a natural form of communication Interaction problems with the mouse –Have to locate cursor –Hard for some to control (Parkinsons or people on a train) –Limited forms of input from the mouse

4. Knowledge Systems Lab JN 1/15/2016 Motivation (2) Interaction Problems with the Virtual Reality Glove –Reliability –Always connected –Encumbrance

5. Knowledge Systems Lab JN 1/15/2016 System Architecture Standard Web Camera Rendering User Interface Display Hand Movement User Gesture Recognition System Image Capture Update Object Image Input

6. Knowledge Systems Lab JN 1/15/2016 Implementation Overview System: 1.6 Ghz AMD Athlon OpenCV and IPL libraries (from Intel) Input: 640x480 video image Hand calibration measure Output: Rough estimate of centroid Refined estimate of centroid Number of fingers being held up Manipulation of 3D skull in QT interface in response to gesturing

7. Knowledge Systems Lab JN 1/15/2016 Implementation Overview (2) Hand Calibration Measure: Max hand size in x and y orientations in # of pixels

8. Knowledge Systems Lab JN 1/15/2016 Implementation Overview (3) Saturation Channel Extraction (HSL space): Original Image Hue Lightness Saturation

9. Knowledge Systems Lab JN 1/15/2016 Proposed Approach

10. Knowledge Systems Lab JN 1/15/2016 Proposed Approach (2)

11. Knowledge Systems Lab JN 1/15/2016 Proposed Approach (3) The finger-finding function sweeps out a circle around the rCoM, counting the number of white and black pixels as it progresses A finger is defined to be any 10+ white pixels separated by 17+ black pixels (salt/pepper tolerance) Total fingers is number of fingers minus 1 for the hand itself

12. Knowledge Systems Lab JN 1/15/2016 Proposed Approach (4) System Runtime: Current time – 41 ms for one image from camera Processing Capability on 1.6 Ghz Athlon: 24 fps Process Steps Time (ms) Athlon XP 1900 (1.6 Ghz) 1) Extract Sat Channel9 2) Threshold3 3) Connected Contour Fill 14 4) Centroid2 5) Segment Hand From Arm 9 6) Refined Centroid4 7) Count Number of Fingers 0 Total Time41

13. Knowledge Systems Lab JN 1/15/2016 Demonstration System Configuration System GUI Layout

14. Knowledge Systems Lab JN 1/15/2016 Demonstration (2) Gesture to Interaction Mapping Number of Fingers: 2 – Roll Left 3 – Roll Right 4 – Zoom In 5 – Zoom Out

15. Knowledge Systems Lab JN 1/15/2016 Demonstration (3)

16. Knowledge Systems Lab JN 1/15/2016 Demonstration (4)

17. Knowledge Systems Lab JN 1/15/2016 Future Directions Optimization Calibration Phase Defining Hand Orientation Learning System Interface Extensions For additional information, please visit http://ksl.jsu.edu.