1. Converting sign language gestures from digital images to text
ASL2TXT
Converting sign language gestures from digital images to text
George Corser

2. Presentation Overview
Concept
Foundation: Barkoky & Charkari (2011)
Segmentation
Thinning
My Contribution: Corser (2012)
Segmentation (similar to Barkoky)
CED: Canny Edge Dilation (Minus Errors)
Assumption: User trains his own phone

3. Concept
Deaf and hearing people talking on the phone, each using their natural language
Sign-activated commands like voice-activated

4. Situation: Drive Thru Window
Think:
Stephen Hawking
Deaf person signs order
Phone speaks order
Confirmation on screen

5. Process Flow Requires several conversion processes
Many have been accomplished
Remaining: ASL2TXT

6. Goal: Find an Algorithm
Find an image processing algorithm that recognizes ASL alphabet
= A
Web site

7. Barkoky: Segmentation & Thinning
Barkoky counts
endpoints to
determine sign
(doesn’t work for ASL)

8. Barkoky Process Segmentation Thinning Capture RGB image Rescale
Extract using colors
Reduce noise
Crop at wrist
Result: hand segment
Input: hand segment
Apply thinning
Find endpoints, joints
Calculate lengths
Clean short lengths
Identify gesture by counting endpoints

9. 1. Capture RGB Image 2. Rescale
% Capture RGB image a = imread('DSC04926.JPG'); figure('Name','RGB image'),imshow(a); % Rescale image to 205x154 a10 = imresize(a, 0.1); figure('Name','Rescaled image'),imshow(a10);

10. 3. Extract Hand Using Colors
% Extract hand using color abw10 = zeros(205,154,1); for i=1:205, for j=1:154, if a10(i,j,2)<140 && a10(i,j,3)<100, abw10(i,j,1)=255; end; figure('Name','Extracted'),imshow(abw10);
Note: Color threshold code
differs from Barkoky

11. Colors: Training Set Histograms

12. Colors: Training Set (2)
Red
Green
Blue
Excel

13. Colors: Test Set Histograms

14. 4. Reduce Noise
% Reduce noise for i=2:204, for j=1:154, if abw10(i-1,j,1)==0 if abw10(i+1,j,1)==0, abw10(i,j,1)=0; end; end; if abw10(i-1,j,1)==255 if abw10(i+1,j,1)==255, end; end; abw10 = imfill(abw10,'holes');

15. 5. Identify Wrist Position
% Identify wrist position for i=204:-1:1, for j=1:154, if abw10(i,j,1)==255, break; end; end; if j ~= 154 && abw10(i+1,j,1)~=255, wristi=i+1; wristj=j+1; break;

16. Wrist Detection Algorithm searches bottom-to-top of image
Finds a leftmost white pixel above black pixel
Sets wrist position SE of found white pixel

17. Corser: Segmentation & CED
Segmentation (similar to Barkoky)
Color threshold technique slightly different
American Sign Language (ASL) alphabet, not Persian Sign Language (PSL) numbers
Image Comparison: Tried Several Methods
Full Threshold (Minus Errors)
Diced Segments (Minus Errors)
Endpoint Count Difference
CED: Canny Edge Dilation

18. ASL Training Set
Hit-or-miss: 23%
Barkoky: 8%

19. ASL Test Set
MATLAB

20. A

21. A

22. B

23. B

24. C

25. C

26. D

27. D

28. E

29. E

30. F

31. F

32. G

33. G

34. H

35. H

36. I

37. I

38. J

39. J

40. K

41. K

42. L

43. L

44. M

45. M

46. N

47. N

48. O

49. O

50. P

51. P

52. Q

53. Q

54. R

55. R

56. S

57. S

58. T

59. T

60. U

61. U

62. V

63. V

64. W

65. W

66. X

67. X

68. Y

69. Y

70. Z

71. Z

72. Z

73. Hybrid Algorithm Example
% MATLAB Code matchtotal = 0; if abs(x10range - x20range) < 20, matchtotal = matchtotal + 10; end; if abs(y10range - y20range) < 20, matchtotal = matchtotal + 11; matchtotal = matchtotal - abs(h10 - h20); % h10, h20 are vector magnitudes -----

74. Erosion Subtraction

75. Canny Edge

76. Canny Edge Dilation Code
% MATLAB Code se = strel('disk',5); a10 = edge(a10,'canny'); a20 = edge(a20,'canny'); a10 = imdilate(a10,se); a20 = imdilate(a20,se); % Then calculate matches minus errors

77. Experimental Results Technique Correct Full Threshold (Minus Errors)
19% (27%)
Diced Segments (Minus Errors)
23% (27%)
Barkoky Endpoint Count Diff. 8%
Hybrid - Height/Width/Endpoints
19%
Erosion Subtraction
15%
Canny Edge Dilation (Minus Errors)
12% (35%)

78. Disadvantages Dependent on lighting conditions
Fails with flesh-tone backgrounds
Requires calibration to a specific user
Limited applications: text messaging, activation (“sign” similar to voice activation)
ASL numbers (A=10, D=1, O=0, V=2, W=6)
Alphabet is tiny portion of full translation: complete translation maybe many years away

79. Future Work
Barkoky claims flesh tones can be detected, but I have yet to replicate (even Barkoky changed his color detection scheme)
Could write letter-by-letter algorithm
Could use range camera to compute distance of finger instead of shape of hand
Motion analysis or edge count
Many possibilities… we’ve only just begun!
Cue: music

80. The End