1. Detection, Segmentation, and Pose Recognition of Hands in Images by Christopher Schwarz Thesis Chair: Dr. Niels da Vitoria Lobo

2. Outline Introduction Detection and Segmentation –Line Finding –Curve Finding –Detection –Grouping –Results Pose Recognition –Preprocessing –Matching –Results Discussions and Conclusions

3. Introduction Hands present an exciting challenge for Computer Vision researchers. –Foils traditional object detection due to nonrigidity and 21 DoF Uses: – Surveillance applications: Gang signs, obscene gestures, drawing of a weapon –Human-Computer Interaction Alternative input devices, motion capture, augmented reality.

4. Terminology Detection: Find presence of target Segmentation: Separate known target from background Pose Recognition: Determine what pose or posture a hand is in.

5. Related Work Huang [2000] Athitsos and Sclaroff [2003] Kölsch and Turk [2004] Baris Caglar [2005]

6. Detection and Segmentation Outline High-resolution images Monochromatic images Straight fingers Open fingers Part 1: Detection and Segmentation Input Image Generate Line Sketch Find Curves Find Candidate Fingers Group and Revisit

7. Line Sketch Image Use a Customized Line Finder –Modified Burns –Replace line combination with iterative method –Add a “cost of fit” measure per line Union results of running Line Finder over 5 varying inputs to obtain Line Sketch –4 varying scale –1 “Double Canny” input Large-gaussian Canny over output of small-gaussian canny to divide textured regions from untextured regions Part 1: Detection and Segmentation

8. Line Finder Iterative Joining of Lines –Find line segments –Find nearby, almost-parallel line pairs –If pair meets thresholds, combine them –Rejoins lines split from angle thresholds or gaps in the edge input. Part 1: Detection and Segmentation

9. Line Finder Cost of Fit Measure output with each line –Cost of fitting line model to underlying data These lines will have a higher Cost of Fit Part 1: Detection and Segmentation

10. Line Sketch Input image Unioned lines of length >= 15 Unioned Components: Part 1: Detection and Segmentation Blur 0Blur 1Blur 2Half-SizeDouble Canny

11. Line Sketch Examples Part 1: Detection and Segmentation

12. Line Sketch Examples Part 1: Detection and Segmentation

13. Curve Finder Second input to algorithm Discovers curves that may represent fingertips See Jan Prokaj’s thesis: Scale Space Based Grammar for Hand Detection Model: Part 1: Detection and Segmentation

14. Curve Finder Examples Part 1: Detection and Segmentation

15. FingerFinder Pseudocode For each pair of lines if pair meets criteria for all curves nearby curves if curve meets criteria add fingerCandidate Part 1: Detection and Segmentation

16. Finger Candidate Criteria “Finger Score” based on empirically found thresholds Criteria –Geometric –Other Part 1: Detection and Segmentation

17. Geometric Criteria 11 tests measuring how well a line pair and a curve approximates target configuration: Part 1: Detection and Segmentation

18. Non-Geometric Criteria Line Inaccuracy: Measure of line curvature found during line finding Canny Density: Amount of edge pixels detected in area. Variance in Canny Density: Sparse finger regions against cluttered background Part 1: Detection and Segmentation

19. Results First row: Input images Second row: Detected candidates Part 1: Detection and Segmentation

20. Grouping Candidate Fingers 1.Find finger groups possibly within the same hand using: –Locations, using Euclidian distance –Region intensities, comparing median values 2.Revisit weaker candidates to reinstate if supported by neighbors Part 1: Detection and Segmentation

21. Results First row: Input images Second row: "Strong" candidates before grouping Third row: Detected fingers, including those re-added during grouping Part 1: Detection and Segmentation

22. Grouping Result Breakdown Results show detections from all groups Often, individual groups divide false from true positives Part 1: Detection and Segmentation

23. Grouping Result Breakdown Part 1: Detection and Segmentation

24. Pose Recognition Goals Segmentation-based method using a database and an input contour Part 2: Pose Recognition Assumes: High-resolution Open fingers

25. Flowchart of Our Method Part 2: Pose Recognition

26. Preprocessing 1.Erode 2.Dilate 3.Compare with the original to find protrusions. Input contour silhouette Part 2: Pose Recognition Preprocessing is identical for the test and every database image.

27. Preprocessing 4. Ignore tiny protrusions as palm 5. Remove palm 6. Use K-Means clustering to find center of palm from wrist-palm segment 7. Count “finger” segments and find average direction Part 2: Pose Recognition

28. Preprocessing Examples Matching takes test and set of database images processed in this way Part 2: Pose Recognition

29. Matching Phase Overview Chamfer Distance Segment-Based Matching Part 2: Pose Recognition Matching via sum of two distance measures:

30. Chamfer Distance Numerical similarity between edge images For each point in X, find nearest point in Y The average is the chamfer distance Part 2: Pose Recognition

31. Chamfer Distance Direction c(X,Y) != c(Y,X) XY c(X,Y) < c(Y,X) “Undirected” Chamfer = c(X,Y) + c(Y,X) Part 2: Pose Recognition

32. Segment Based Matching: Overview Generate CODAA Vector for every pair of test segment and model segment. Vector contains five segment comparators Rank comparator vectors Rank database images with sum of comparator rankings Part 2: Pose Recognition

33. Segment Based Matching: CODAA Vectors C Chamfer distance between contours O Difference between orientations D Distance between centers A Difference in size A Difference in angle relative to palm center Part 2: Pose Recognition

34. Segment Based Matching 1.Score each CODAA vector via progressive thresholds of the five values. 2.Rank vectors according to scores 3.For each model image segment, find match in test image with highest score 4.For each segment in test image, find match in model image with highest score 5.Sum “forward” and “reverse” measures 6.Divide by number of fingers 7.Rank model images by score Part 2: Pose Recognition

35. Combination Combine results of Chamfer Distance and SBM by summing the Log (base 2) of a model’s rank in each measure. Rank models by this combined score Filter known-incorrect models: –Incorrect finger count –Incorrect average finger angle Part 2: Pose Recognition

36. Video Test Results Use video frames as a "database," to find ones matching an input pose Part 2: Pose Recognition

37. Still-Image Test Results Use a standard database Part 2: Pose Recognition

38. Publications Segment-Based Hand Pose Estimation. In IEEE CRV 2005. Hand Detection and Segmentation for Pose Recognition in Monochromatic Images. In progress. Line Sketch. To be written.

39. Future Work Develop and test bridge between segmentation and recognition algorithms Feasible to convert finger candidate regions into framework of SBM Results improved if palm center can be reliably located

40. Acknowledgements Thesis Committee –Dr. Niels da Vitoria Lobo –Dr. Charles Hughes –Dr. Mubarak Shah –Dr. Huaxin You Support –NSF REU Program