Feature-preserving Artifact Removal from Dermoscopy Images Howard Zhou 1, Mei Chen 2, Richard Gass 2, James M. Rehg 1, Laura Ferris 3, Jonhan Ho 3, Laura Drogowski 3 1 School of Interactive Computing, Georgia Tech 2 Intel Research Pittsburgh 3 Department of Dermatology, University of Pittsburgh

Skin cancer and melanoma Skin cancer : most common of all cancers

Skin cancer and melanoma Skin cancer : most common of all cancers [ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ] Basal Cell Carcinoma Hemangioma Compound nevusSeborrheic keratosis

Skin cancer and melanoma Skin cancer : most common of all cancers Melanoma : leading cause of mortality [ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ] Basal Cell Carcinoma Hemangioma Compound nevusSeborrheic keratosis Melanoma

Skin cancer and melanoma Skin cancer : most common of all cancers Melanoma : leading cause of mortality Early detection significantly reduces mortality [ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ] Basal Cell Carcinoma Hemangioma Compound nevusSeborrheic keratosis Melanoma

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ] Clinical View Dermoscopy view

Dermoscopy Skin surface microscopy Improve diagnostic accuracy by 30% for trained, experienced physicians Requires 5 or more years of experience Computer-aided diagnosis (CAD) to assist less experienced physicians Clinical viewDermoscopy view

Artifacts in dermoscopy images Hair, air-bubbles,… Interfering with computer-aided diagnosis [ Image courtesy of Grana et al. 2006]

Hair, air-bubbles,… Interfering with computer-aided diagnosis [ Image courtesy of Grana et al. 2006] Artifacts in dermoscopy images

Hair, air-bubbles,… Interfering with computer-aided diagnosis [ Image courtesy of Grana et al. 2006] Hair  lesion boundary

Artifacts in dermoscopy images Hair, air-bubbles,… Interfering with computer-aided diagnosis [ Image courtesy of Grana et al. 2006] Hair  lesion boundary

Artifacts in dermoscopy images Hair, air-bubbles,… Interfering with computer-aided diagnosis [ Image courtesy of Grana et al. 2006] Hair  lesion boundaryHair  pigmented network

Previous work Hair detection and tracing Fleming et al Thresholding and averaging “DullRazor”, Tim K. Lee et al Schmid et al Thresholding and inpainting Paul Wighton et al (right here in the conference)

Detection: thresholding Removal: morphological operations Schmid et al.

Thresholding  false detection Accidental removal of diagnostic features Schmid et al Thresholding

Schmid et al. Morphological operation (neighbors’ average)  blurring Morphological operation Schmid et al. 2003

Feature-preserving artifact removal (FAR) Detection: Explicit curve modeling Removal: Exemplar- based inpainting Our method (FAR)Schmid et al. 2003

Our method (FAR) FAR Curve modeling  more accurate hair detection Thresholding Curve modeling Schmid et al. 2003

Our method (FAR) FAR Exemplar-based inpainting  preserving features Thresholding Curve modeling Morphological operation Exemplar-based inpainting Schmid et al. 2003

Our method (FAR) FAR Exemplar-based inpainting  preserving features Thresholding Curve modeling Morphological operation Exemplar-based inpainting Schmid et al. 2003

Our method (FAR) FAR Exemplar-based inpainting  preserving features Schmid et al. 2003

Our method (FAR) FAR Exemplar-based inpainting  preserving features Schmid et al. 2003

Our method (FAR) FAR Exemplar-based inpainting  preserving features Schmid et al. 2003

System overview Threholding Curve fitting & intersection analysis Exemplar patches Exemplar-based inpainting Dermoscopy image Hair removed Luminance difference  dark thin structure Line points Line segments Parameterized curves Mask Line points linking Line points Detection

Input dermoscopy image

Enhancing dark-thin structure Luminosity channel in CIE L*u*v* Difference b/a morphological closing [ Schmid-Saugeona et al. 2003, “Towards a computer-aided diagnosis system for pigmented skin lesions” ]

Detecting line points [ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ] Curve B(t)

Detecting line points [ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ] n(t) Curve B(t)Cross section n(t) f(x)

Detecting line points [ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ] n(t) Cross section n(t) f(x) Curve B(t)

Detecting line points [ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ] n(t) Cross section n(t) f(x) f’ = 0 |f’’| large Curve B(t)

Detecting line points [ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ] n(t) Cross section n(t) f(x) f’ = 0 |f’’| large Curve B(t) n(t) : direction ┴ curve B(t) eigenvector corresponding to the maximum absolute eigenvalue of the local Hessian

Detecting line points [ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ] n(t)

Detecting line points [ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ]

Linking line points Link the neighboring points to get line segments (sets of ordered line points)

Fitting polynomial curves A set of ordered points P i s P

Fitting polynomial curves A set of ordered points P i s Parametric curve P

Fitting polynomial curves A set of ordered points P i s Parametric curve B(t) P

Fitting polynomial curves B(t) P A set of ordered points P i s Parametric curve Minimize sum of squared distance

Fitting polynomial curves A set of ordered points P i s Parametric curve Minimize sum of squared distance Linear system (can be solved by Gaussian elimination) B(t) P

Handling hair intersection Configurations: Hair intersectionLine segments Intersection analysis Link Line segment ……

Before curve fitting and linking Line segments

After curve fitting and linking Parameterized curves

After curve fitting and linking Parameterized curves

After curve fitting and linking Hair mask

After curve fitting and linking Hair mask

Exemplar-based inpainting [ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ] [ Image courtesy of Criminisi et al ] Fill in with patches from the image itself Patch ordering  structure propagation.

Exemplar-based inpainting Fill in with patches from the image itself Patch ordering  structure propagation. [ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

Exemplar-based inpainting Fill in with patches from the image itself Patch ordering  structure propagation. [ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

Exemplar-based inpainting Fill in with patches from the image itself Patch ordering  structure propagation. [ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

Exemplar-based inpainting Fill in with patches from the image itself Patch ordering  structure propagation. [ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

Exemplar-based inpainting Fill in with patches from the image itself Patch ordering  structure propagation. [ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

Exemplar-based inpainting Fill in with patches from the image itself Patch ordering  structure propagation. [ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

Exemplar-based inpainting Fill in with patches from the image itself Patch ordering  structure propagation. [ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

Before FAR

After FAR

More results Explicit curve modeling Exemplar-based inpainting Our method (FAR)Schmid et al. 2003

More results Explicit curve modeling Exemplar-based inpainting Our method (FAR)Schmid et al. 2003

Our method (FAR) FAR Exemplar-based inpainting  preserving features Schmid et al. 2003

When is FAR not suitable ? Oops, too much hair!

When is FAR not suitable ? Too much hair Makes explicit modeling difficult Schemid et al (DullRazor)Our method (FAR)

Conclusion Automatic system that detects and removes curvilinear artifacts Feature-preserving artifact removal: Explicit curve modeling Exemplar-based inpainting

Future work Speed up exemplar-based inpainting

Future work Speed up exemplar-based inpainting Handle hair with arbitrary intensity

Future work Speed up exemplar-based inpainting Handle hair with arbitrary intensity Extend to removing air bubbles

Questions ?

Additional results Our method (FAR)Original Dermoscopy image