1. Hand Tracking for Virtual Object Manipulation
Thibaut Weise
Hi, I am Thibaut Weise.
I’d like to present you my final year project which is about hand tracking for virtual object manipulation.
Project Supervisor:
Second Marker:
Professor Guang-Zhong Yang
Dr Duncan Gillies

2. Motivation Virtual and Augmented Environments Robotic Control Surgery
Space, Deep Sea
Why would we want to track hands?
I’d like to give two examples where tracking hands would be beneficial:
Virtual and Augmented Environments:
Instead of using a mouse or wand, using the hands directly would be much more intuitive as it is the same movement as the user has to do for real world objects. It would make interaction for users much easier.
Robot Control:
Anywhere where hands are necessary for some work, but it is difficult to access, robotic arms can be used instead e.g. for surgery, mining, space, deep sea, etc. It would be beneficial to be able to control the robotic gripper directly by hand.

3. State of the Art
EM-Tracking
Acoustic Tracking
Data Glove
Optical

4. Optical Approaches Appearance-based Approach Model-based Approach
Reconstruct 3D Hand Model
Two methods:
Minimize Model against Features
Reconstruct from Features

5. Requirements Focus: Grab Gesture  Thumb and Index Finger Real-time
Responsive
Accurate

6. System Overview Calibration Marker-based Hand Tracking Marker-less
Virtual Environment

7. Reconstruction Triangulation
Requires Mapping between Pixels and 3D World Coordinates

8. Calibration
Optimize known World Coordinates against detected Image Coordinates
Planar Approach => Chessboard
OpenCV Implementation
Stereo Optimization

9. Calibration Accuracy
Corner Detector Accuracy
Why sub-pixel accurate?

10. X-Corner Detector Solution: Improved X-Corner Detector
Saddle Point
Product of Eigenvalues minimised: λ1λ2= fxxfyy-fxy2
Gradient zero: 0 = fx + s fxx + t fxy
0 = fy + s fxy + t fyy
Recalculate using weighted derivatives of neighbourhood

11. Corner Detector Comparison
Solution: Improved X-Corner Detector

12. Calibration Automation
Improved Chessboard Detector

13. Calibration Tool

14. Hand Tracking
Marker-based Marker-less

15. Marker Tracking 3 Markers Thumb, Index Finger and Hand
Uses Improved X-Corner Detector
Fast
Accurate
Simple
Uses improved X-corner detector as it is fast, accurate and simple

16. Hand Reconstruction Find all potential Markers in Images
Match Points using Epipolar Constraints
Uses improved X-corner detector as it is fast, accurate and simple

17. Hand Reconstruction Reconstruct all possible hand configurations
Different markers
Validate hand constraints
Uses improved X-corner detector as it is fast, accurate and simple

18. Hand Reconstruction Use best match based on temporal information
Minimise Euclidean distance to predicted position
Uses improved X-corner detector as it is fast, accurate and simple

19. Hand Reconstruction Initialisation based on Orientation Heuristic
Uses improved X-corner detector as it is fast, accurate and simple

20. Marker-less Tracking Hand Localisation 2 Fingertips
Reconstruct Hand Position
Hand Localisation
Skin Colour Detection
We have no markers, so we need to reconstruct the hand differently.
The hand position cannot found by direct feature point, as there is no feature!
Reconstruct hand position from finger axes.
First step is to localise the hand in the image.
Skin colour is in a distinct region of a certain colour space. We use skin colour detection using a simple heuristic Rule.

21. Hand Localisation
Background Subtraction -
Skin + Background +

22. Hand Localisation Find Hand Contour
We have no markers, so we need to reconstruct the hand differently.
The markers were at the finger tips, so detecting the fingertips is effective.
The hand position cannot found by direct feature point, as there is no feature!
Reconstruct hand position from finger axes.
First step is to localise the hand in the image.
Skin colour is in a distinct region of a certain colour space. We use skin colour detection using a simple heuristic
Rule.

23. Hand Reconstruction Find Peaks Calculate k-curvature:
[C(i)-C(i-k)]T[C(i+k)-C(i)]
Peaks where angle > 0o and < 90o
Once we have a contour as a list of pixel positions we find the fingertips as peaks in the contour.
For that we calculate the k-curvature at each contour pixel. The k-curvature is just the angle between
the vector from the contour pixel k steps before and the vector to the contour pixel k steps after.
If the angle is bigger than 90 and less than 180 degrees, that is the dot product is less than 0, then it is a potential peak.

24. Hand Reconstruction
Match peaks using Epipolar Constraints  find possible Finger Tips
Calculate Finger Axis

25. Hand Reconstruction Reconstruct hand position
Validate hand constraints
Use best hand based on temporal information

26. Evaluation
Marker-based Marker-less

27. Virtual Environment Virtual Tower of Hanoi Virtual Hand => Grabber
Fixed Dimensions
Grab and Push Objects
Perception
Colour Coding
Shadow

28. Implementation Application Video / Tracking Thread Thread
Settings
Video / Tracking Thread
Draw /
Settings
Hand Update
Virtual Environment

29. Demo

30. Conclusion Accurate semi-automatic Calibration Marker-based Tracking
Marker-less Tracking
Virtual Environment

31. Future Work Marker-based Tracking Marker-less Tracking
Handle Occlusion
Marker-less Tracking
Refine Skin Colour Detection
Use Gradient

32. Questions ?
Who wants to try?