Hand Tracking for Virtual Object Manipulation Thibaut Weise Hi, I am Thibaut Weise. I’d like to present you my final year project which is about hand tracking for virtual object manipulation. Project Supervisor: Second Marker: Professor Guang-Zhong Yang Dr Duncan Gillies

Motivation Virtual and Augmented Environments Robotic Control Surgery Space, Deep Sea Why would we want to track hands? I’d like to give two examples where tracking hands would be beneficial: Virtual and Augmented Environments: Instead of using a mouse or wand, using the hands directly would be much more intuitive as it is the same movement as the user has to do for real world objects. It would make interaction for users much easier. Robot Control: Anywhere where hands are necessary for some work, but it is difficult to access, robotic arms can be used instead e.g. for surgery, mining, space, deep sea, etc. It would be beneficial to be able to control the robotic gripper directly by hand.

State of the Art EM-Tracking Acoustic Tracking Data Glove Optical

Optical Approaches Appearance-based Approach Model-based Approach Reconstruct 3D Hand Model Two methods: Minimize Model against Features Reconstruct from Features

Requirements Focus: Grab Gesture  Thumb and Index Finger Real-time Responsive Accurate

System Overview Calibration Marker-based Hand Tracking Marker-less Virtual Environment

Reconstruction Triangulation Requires Mapping between Pixels and 3D World Coordinates

Calibration Optimize known World Coordinates against detected Image Coordinates Planar Approach => Chessboard OpenCV Implementation Stereo Optimization

Calibration Accuracy Corner Detector Accuracy Why sub-pixel accurate?

X-Corner Detector Solution: Improved X-Corner Detector Saddle Point Product of Eigenvalues minimised: λ1λ2= fxxfyy-fxy2 Gradient zero: 0 = fx + s fxx + t fxy 0 = fy + s fxy + t fyy Recalculate using weighted derivatives of neighbourhood

Corner Detector Comparison Solution: Improved X-Corner Detector

Calibration Automation Improved Chessboard Detector

Calibration Tool

Hand Tracking Marker-based Marker-less

Marker Tracking 3 Markers Thumb, Index Finger and Hand Uses Improved X-Corner Detector Fast Accurate Simple Uses improved X-corner detector as it is fast, accurate and simple

Hand Reconstruction Find all potential Markers in Images Match Points using Epipolar Constraints Uses improved X-corner detector as it is fast, accurate and simple

Hand Reconstruction Reconstruct all possible hand configurations Different markers Validate hand constraints Uses improved X-corner detector as it is fast, accurate and simple

Hand Reconstruction Use best match based on temporal information Minimise Euclidean distance to predicted position Uses improved X-corner detector as it is fast, accurate and simple

Hand Reconstruction Initialisation based on Orientation Heuristic Uses improved X-corner detector as it is fast, accurate and simple

Marker-less Tracking Hand Localisation 2 Fingertips Reconstruct Hand Position Hand Localisation Skin Colour Detection We have no markers, so we need to reconstruct the hand differently. The hand position cannot found by direct feature point, as there is no feature! Reconstruct hand position from finger axes. First step is to localise the hand in the image. Skin colour is in a distinct region of a certain colour space. We use skin colour detection using a simple heuristic Rule.

Hand Localisation Background Subtraction - Skin + Background +

Hand Localisation Find Hand Contour We have no markers, so we need to reconstruct the hand differently. The markers were at the finger tips, so detecting the fingertips is effective. The hand position cannot found by direct feature point, as there is no feature! Reconstruct hand position from finger axes. First step is to localise the hand in the image. Skin colour is in a distinct region of a certain colour space. We use skin colour detection using a simple heuristic Rule.

Hand Reconstruction Find Peaks Calculate k-curvature: [C(i)-C(i-k)]T[C(i+k)-C(i)] Peaks where angle > 0o and < 90o Once we have a contour as a list of pixel positions we find the fingertips as peaks in the contour. For that we calculate the k-curvature at each contour pixel. The k-curvature is just the angle between the vector from the contour pixel k steps before and the vector to the contour pixel k steps after. If the angle is bigger than 90 and less than 180 degrees, that is the dot product is less than 0, then it is a potential peak.

Hand Reconstruction Match peaks using Epipolar Constraints  find possible Finger Tips Calculate Finger Axis

Hand Reconstruction Reconstruct hand position Validate hand constraints Use best hand based on temporal information

Evaluation Marker-based Marker-less

Virtual Environment Virtual Tower of Hanoi Virtual Hand => Grabber Fixed Dimensions Grab and Push Objects Perception Colour Coding Shadow

Implementation Application Video / Tracking Thread Thread Settings Video / Tracking Thread Draw / Settings Hand Update Virtual Environment

Demo

Conclusion Accurate semi-automatic Calibration Marker-based Tracking Marker-less Tracking Virtual Environment

Future Work Marker-based Tracking Marker-less Tracking Handle Occlusion Marker-less Tracking Refine Skin Colour Detection Use Gradient

Questions ? Who wants to try?