1. Feature-preserving Artifact Removal from Dermoscopy Images Howard Zhou 1, Mei Chen 2, Richard Gass 2, James M. Rehg 1, Laura Ferris 3, Jonhan Ho 3, Laura Drogowski 3 1 School of Interactive Computing, Georgia Tech 2 Intel Research Pittsburgh 3 Department of Dermatology, University of Pittsburgh

2. Skin cancer and melanoma Skin cancer : most common of all cancers

3. Skin cancer and melanoma Skin cancer : most common of all cancers [ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ] Basal Cell Carcinoma Hemangioma Compound nevusSeborrheic keratosis

4. Skin cancer and melanoma Skin cancer : most common of all cancers Melanoma : leading cause of mortality [ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ] Basal Cell Carcinoma Hemangioma Compound nevusSeborrheic keratosis Melanoma

5. Skin cancer and melanoma Skin cancer : most common of all cancers Melanoma : leading cause of mortality Early detection significantly reduces mortality [ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ] Basal Cell Carcinoma Hemangioma Compound nevusSeborrheic keratosis Melanoma

6. [ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ] Clinical View Dermoscopy view

7. Dermoscopy Skin surface microscopy Improve diagnostic accuracy by 30% for trained, experienced physicians Requires 5 or more years of experience Computer-aided diagnosis (CAD) to assist less experienced physicians Clinical viewDermoscopy view

8. Artifacts in dermoscopy images Hair, air-bubbles,… Interfering with computer-aided diagnosis [ Image courtesy of Grana et al. 2006]

9. Hair, air-bubbles,… Interfering with computer-aided diagnosis [ Image courtesy of Grana et al. 2006] Artifacts in dermoscopy images

10. Hair, air-bubbles,… Interfering with computer-aided diagnosis [ Image courtesy of Grana et al. 2006] Hair  lesion boundary

11. Artifacts in dermoscopy images Hair, air-bubbles,… Interfering with computer-aided diagnosis [ Image courtesy of Grana et al. 2006] Hair  lesion boundary

12. Artifacts in dermoscopy images Hair, air-bubbles,… Interfering with computer-aided diagnosis [ Image courtesy of Grana et al. 2006] Hair  lesion boundaryHair  pigmented network

13. Previous work Hair detection and tracing Fleming et al. 1998 Thresholding and averaging “DullRazor”, Tim K. Lee et al. 1997 Schmid et al. 2003 Thresholding and inpainting Paul Wighton et al. 2008 (right here in the conference)

14. Detection: thresholding Removal: morphological operations Schmid et al.

15. Thresholding  false detection Accidental removal of diagnostic features Schmid et al. 2003 Thresholding

16. Schmid et al. Morphological operation (neighbors’ average)  blurring Morphological operation Schmid et al. 2003

17. Feature-preserving artifact removal (FAR) Detection: Explicit curve modeling Removal: Exemplar- based inpainting Our method (FAR)Schmid et al. 2003

18. Our method (FAR) FAR Curve modeling  more accurate hair detection Thresholding Curve modeling Schmid et al. 2003

19. Our method (FAR) FAR Exemplar-based inpainting  preserving features Thresholding Curve modeling Morphological operation Exemplar-based inpainting Schmid et al. 2003

20. Our method (FAR) FAR Exemplar-based inpainting  preserving features Thresholding Curve modeling Morphological operation Exemplar-based inpainting Schmid et al. 2003

21. Our method (FAR) FAR Exemplar-based inpainting  preserving features Schmid et al. 2003

22. Our method (FAR) FAR Exemplar-based inpainting  preserving features Schmid et al. 2003

23. Our method (FAR) FAR Exemplar-based inpainting  preserving features Schmid et al. 2003

24. System overview Threholding Curve fitting & intersection analysis Exemplar patches Exemplar-based inpainting Dermoscopy image Hair removed Luminance difference  dark thin structure Line points Line segments Parameterized curves Mask Line points linking Line points Detection

25. Input dermoscopy image

26. Enhancing dark-thin structure Luminosity channel in CIE L*u*v* Difference b/a morphological closing [ Schmid-Saugeona et al. 2003, “Towards a computer-aided diagnosis system for pigmented skin lesions” ]

27. Detecting line points [ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ] Curve B(t)

28. Detecting line points [ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ] n(t) Curve B(t)Cross section n(t) f(x)

29. Detecting line points [ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ] n(t) Cross section n(t) f(x) Curve B(t)

30. Detecting line points [ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ] n(t) Cross section n(t) f(x) f’ = 0 |f’’| large Curve B(t)

31. Detecting line points [ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ] n(t) Cross section n(t) f(x) f’ = 0 |f’’| large Curve B(t) n(t) : direction ┴ curve B(t) eigenvector corresponding to the maximum absolute eigenvalue of the local Hessian

32. Detecting line points [ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ] n(t)

33. Detecting line points [ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ]

34. Linking line points Link the neighboring points to get line segments (sets of ordered line points)

35. Fitting polynomial curves A set of ordered points P i s P

36. Fitting polynomial curves A set of ordered points P i s Parametric curve P

37. Fitting polynomial curves A set of ordered points P i s Parametric curve B(t) P

38. Fitting polynomial curves B(t) P A set of ordered points P i s Parametric curve Minimize sum of squared distance

39. Fitting polynomial curves A set of ordered points P i s Parametric curve Minimize sum of squared distance Linear system (can be solved by Gaussian elimination) B(t) P

40. Handling hair intersection Configurations: Hair intersectionLine segments Intersection analysis Link Line segment ……

41. Before curve fitting and linking Line segments

42. After curve fitting and linking Parameterized curves

43. After curve fitting and linking Parameterized curves

44. After curve fitting and linking Hair mask

45. After curve fitting and linking Hair mask

46. Exemplar-based inpainting [ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ] [ Image courtesy of Criminisi et al. 2003 ] Fill in with patches from the image itself Patch ordering  structure propagation.

47. Exemplar-based inpainting Fill in with patches from the image itself Patch ordering  structure propagation. [ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

48. Exemplar-based inpainting Fill in with patches from the image itself Patch ordering  structure propagation. [ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

49. Exemplar-based inpainting Fill in with patches from the image itself Patch ordering  structure propagation. [ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

50. Exemplar-based inpainting Fill in with patches from the image itself Patch ordering  structure propagation. [ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

51. Exemplar-based inpainting Fill in with patches from the image itself Patch ordering  structure propagation. [ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

52. Exemplar-based inpainting Fill in with patches from the image itself Patch ordering  structure propagation. [ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

53. Exemplar-based inpainting Fill in with patches from the image itself Patch ordering  structure propagation. [ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

54. Before FAR

55. After FAR

56. More results Explicit curve modeling Exemplar-based inpainting Our method (FAR)Schmid et al. 2003

57. More results Explicit curve modeling Exemplar-based inpainting Our method (FAR)Schmid et al. 2003

58. Our method (FAR) FAR Exemplar-based inpainting  preserving features Schmid et al. 2003

59. When is FAR not suitable ? Oops, too much hair!

60. When is FAR not suitable ? Too much hair Makes explicit modeling difficult Schemid et al. 2003 (DullRazor)Our method (FAR)

61. Conclusion Automatic system that detects and removes curvilinear artifacts Feature-preserving artifact removal: Explicit curve modeling Exemplar-based inpainting

62. Future work Speed up exemplar-based inpainting

63. Future work Speed up exemplar-based inpainting Handle hair with arbitrary intensity

64. Future work Speed up exemplar-based inpainting Handle hair with arbitrary intensity Extend to removing air bubbles

65. Questions ?

66. Additional results Our method (FAR)Original Dermoscopy image