1. Tomihisa (Tom) Welsh Michael Ashikhmin Klaus Mueller Tomihisa (Tom) Welsh Michael Ashikhmin Klaus Mueller Center for Visual Computing Stony Brook University Transferring Color to Greyscale Images

2. The Problem How to colorize greyscale images? How to colorize greyscale images?

3. The Problem How to colorize greyscale images? How to colorize greyscale images?

4. The Problem How to colorize greyscale images? How to colorize greyscale images? Is his tie blue or green? Is his tie blue or green?

5. The Problem How to colorize greyscale images? How to colorize greyscale images? Red!! Red!!

6. The Problem How to colorize greyscale images? How to colorize greyscale images? Issues: Issues: No “correct” solution Need to be creative How to minimize the manual labor involved? How to colorize greyscale images? How to colorize greyscale images? Issues: Issues: No “correct” solution Need to be creative How to minimize the manual labor involved?

7. Motivation Enhance Scientific Data Enhance Scientific Data Medical Imaging (MRI, CT, X-Ray) Enhance Scientific Data Enhance Scientific Data Medical Imaging (MRI, CT, X-Ray)

8. Motivation Enhance Scientific Data Enhance Scientific Data Medical Imaging (MRI, CT, X-Ray) Satellite Images (Landsat) Enhance Scientific Data Enhance Scientific Data Medical Imaging (MRI, CT, X-Ray) Satellite Images (Landsat)

9. Motivation Enhance Scientific Data Enhance Scientific Data Medical Imaging (MRI, CT, X-Ray) Satellite Images (Landsat) Scanning Electron Microscopy (SEM) Enhance Scientific Data Enhance Scientific Data Medical Imaging (MRI, CT, X-Ray) Satellite Images (Landsat) Scanning Electron Microscopy (SEM)

10. Motivation Enhance Scientific Data Enhance Scientific Data Medical Imaging (MRI, CT, X-Ray) Satellite Images (Landsat) Scanning Electron Microscopy (SEM) Colorize black/white photographs and movies Colorize black/white photographs and movies Enhance Scientific Data Enhance Scientific Data Medical Imaging (MRI, CT, X-Ray) Satellite Images (Landsat) Scanning Electron Microscopy (SEM) Colorize black/white photographs and movies Colorize black/white photographs and movies

11. Motivation Enhance Scientific Data Enhance Scientific Data Medical Imaging (MRI, CT, X-Ray) Satellite Images (Landsat) Scanning Electron Microscopy (SEM) Colorize black/white photographs and movies Colorize black/white photographs and movies Artistic Effects Artistic Effects Enhance Scientific Data Enhance Scientific Data Medical Imaging (MRI, CT, X-Ray) Satellite Images (Landsat) Scanning Electron Microscopy (SEM) Colorize black/white photographs and movies Colorize black/white photographs and movies Artistic Effects Artistic Effects

12. The Task The problem is fundamentally ill-posed The problem is fundamentally ill-posed It is an attempt to extrapolate from 1-D to 3-D It is an attempt to extrapolate from 1-D to 3-D Map scalar luminance (intensity) to vector RGB The problem is fundamentally ill-posed The problem is fundamentally ill-posed It is an attempt to extrapolate from 1-D to 3-D It is an attempt to extrapolate from 1-D to 3-D Map scalar luminance (intensity) to vector RGB

13. Previous Methods Coloring Book Method Coloring Book Method Photoshop: manually paint color with low opacity Coloring Book Method Coloring Book Method Photoshop: manually paint color with low opacity

14. Previous Methods Coloring Book Method Coloring Book Method Photoshop: manually paint color with low opacity Movie Industry: track polygons [Cinesite Press Article] Coloring Book Method Coloring Book Method Photoshop: manually paint color with low opacity Movie Industry: track polygons [Cinesite Press Article]

15. Previous Methods Coloring Book Method Coloring Book Method Photoshop: manually paint color with low opacity Movie Industry: track polygons [Cinesite Press Article] Pseudo-coloringPseudo-coloring Global Transformation/Color Map Coloring Book Method Coloring Book Method Photoshop: manually paint color with low opacity Movie Industry: track polygons [Cinesite Press Article] Pseudo-coloringPseudo-coloring Global Transformation/Color Map

16. Previous Methods Coloring Book Method Coloring Book Method Photoshop: manually paint color with low opacity Movie Industry: track polygons [Cinesite Press Article] Pseudo-coloringPseudo-coloring Global Transformation/Color Map Satellite Images Satellite Images Registration [R2V Software], Orthoimagery [Premoze] Coloring Book Method Coloring Book Method Photoshop: manually paint color with low opacity Movie Industry: track polygons [Cinesite Press Article] Pseudo-coloringPseudo-coloring Global Transformation/Color Map Satellite Images Satellite Images Registration [R2V Software], Orthoimagery [Premoze]

17. Previous Methods Coloring Book Method Coloring Book Method Photoshop: manually paint color with low opacity Movie Industry: track polygons [Cinesite Press Article] Pseudo-coloringPseudo-coloring Global Transformation/Color Map Satellite Images Satellite Images Registration [R2V Software], Orthoimagery [Premoze] Image Analogies Image Analogies Grey Source : Color Source :: Grey Target : Result Coloring Book Method Coloring Book Method Photoshop: manually paint color with low opacity Movie Industry: track polygons [Cinesite Press Article] Pseudo-coloringPseudo-coloring Global Transformation/Color Map Satellite Images Satellite Images Registration [R2V Software], Orthoimagery [Premoze] Image Analogies Image Analogies Grey Source : Color Source :: Grey Target : Result

18. Related Work “Color Transfer between Images” “Color Transfer between Images” [Reinhard et al. 2001] “Color Transfer between Images” “Color Transfer between Images” [Reinhard et al. 2001]

19. Related Work “Color Transfer between Images” “Color Transfer between Images” [Reinhard et al. 2001] “Color Transfer between Images” “Color Transfer between Images” [Reinhard et al. 2001] Target Source +

20. Related Work Target SourceFinal += “Color Transfer between Images” “Color Transfer between Images” [Reinhard et al. 2001] “Color Transfer between Images” “Color Transfer between Images” [Reinhard et al. 2001]

21. Reinhard et al. Decorrelated color space (lαβ) [Ruderman et al., 1998] Decorrelated color space (lαβ) [Ruderman et al., 1998] Scale and shift color distributions globally Scale and shift color distributions globally (Using mean and standard deviation) Decorrelated color space (lαβ) [Ruderman et al., 1998] Decorrelated color space (lαβ) [Ruderman et al., 1998] Scale and shift color distributions globally Scale and shift color distributions globally (Using mean and standard deviation) SourceTarget

22. Reinhard et al. Decorrelated color space (lαβ) [Ruderman et al., 1998] Decorrelated color space (lαβ) [Ruderman et al., 1998] Scale and shift color distributions globally Scale and shift color distributions globally (Using mean and standard deviation) Decorrelated color space (lαβ) [Ruderman et al., 1998] Decorrelated color space (lαβ) [Ruderman et al., 1998] Scale and shift color distributions globally Scale and shift color distributions globally (Using mean and standard deviation) Target (Scaled)Source

23. Reinhard et al. Decorrelated color space (lαβ) [Ruderman et al., 1998] Decorrelated color space (lαβ) [Ruderman et al., 1998] Scale and shift color distributions globally Scale and shift color distributions globally (Using mean and standard deviation) Decorrelated color space (lαβ) [Ruderman et al., 1998] Decorrelated color space (lαβ) [Ruderman et al., 1998] Scale and shift color distributions globally Scale and shift color distributions globally (Using mean and standard deviation) Target (Scaled & Shifted)Source

24. Our Approach Target

25. Our Approach Select color source image Select color source image + ??? Source Target ???

26. Our Approach Select color source image Select color source image Match each target pixel with a few source pixels Match each target pixel with a few source pixels Select color source image Select color source image Match each target pixel with a few source pixels Match each target pixel with a few source pixels + Source Target

27. Our Approach Select color source image Select color source image Match each target pixel with a few source pixels Match each target pixel with a few source pixels Choose best match using local pixel neighborhood statistics Select color source image Select color source image Match each target pixel with a few source pixels Match each target pixel with a few source pixels Choose best match using local pixel neighborhood statistics + Source Target

28. Our Approach Select color source image Select color source image Match each target pixel with a few source pixels Match each target pixel with a few source pixels Choose best match using local pixel neighborhood statistics Transfer color Transfer color Select color source image Select color source image Match each target pixel with a few source pixels Match each target pixel with a few source pixels Choose best match using local pixel neighborhood statistics Transfer color Transfer color + Source Target =

29. Our Approach Select color source image Select color source image Match each target pixel with a few source pixels Match each target pixel with a few source pixels Choose best match using local pixel neighborhood statistics Transfer color Transfer color Repeat for all pixels Repeat for all pixels Select color source image Select color source image Match each target pixel with a few source pixels Match each target pixel with a few source pixels Choose best match using local pixel neighborhood statistics Transfer color Transfer color Repeat for all pixels Repeat for all pixels =+ Source TargetFinal

30. Global Image Matching Procedure 1. Convert images to lαβ color space 2. Image Matching 3. Color Transfer 1. Convert images to lαβ color space 2. Image Matching 3. Color Transfer

31. 1. Convert to lαβ Space Luminance (l), alpha (α) and beta (β) channels Luminance (l), alpha (α) and beta (β) channels Minimizes correlation between axes Minimizes correlation between axes (i.e. cross-channel artifacts) Luminance (l), alpha (α) and beta (β) channels Luminance (l), alpha (α) and beta (β) channels Minimizes correlation between axes Minimizes correlation between axes (i.e. cross-channel artifacts) RGBlαβ Color Image

32. 2. Image Matching 1. Remap luminance histograms between src/target Target Source-Before Source-After

33. 2. Image Matching 1. Remap luminance histograms between src/target 2. Precompute neighborhood statistics for images 1. Remap luminance histograms between src/target 2. Precompute neighborhood statistics for images

34. 2. Image Matching 1. Remap luminance histograms between src/target 2. Precompute neighborhood statistics for images 3. Reduce samples using jittered sampling - Faster computation due to smaller search space 1. Remap luminance histograms between src/target 2. Precompute neighborhood statistics for images 3. Reduce samples using jittered sampling - Faster computation due to smaller search space

35. 2. Image Matching 1. Remap luminance histograms between src/target 2. Precompute neighborhood statistics for images 3. Reduce samples using jittered sampling - Faster computation due to smaller search space 1. Remap luminance histograms between src/target 2. Precompute neighborhood statistics for images 3. Reduce samples using jittered sampling - Faster computation due to smaller search space

36. 2. Image Matching 1. Remap luminance histograms between src/target 2. Precompute neighborhood statistics for images 3. Reduce samples using jittered sampling - Faster computation due to smaller search space 1. Remap luminance histograms between src/target 2. Precompute neighborhood statistics for images 3. Reduce samples using jittered sampling - Faster computation due to smaller search space

37. 2. Image Matching 1. Remap luminance histograms between src/target 2. Precompute neighborhood statistics for images 3. Reduce samples using jittered sampling - Faster computation due to smaller search space 4. Find best neighborhood match from samples - Weighted metric of luminance, mean, standard deviation 1. Remap luminance histograms between src/target 2. Precompute neighborhood statistics for images 3. Reduce samples using jittered sampling - Faster computation due to smaller search space 4. Find best neighborhood match from samples - Weighted metric of luminance, mean, standard deviation

38. 3. Color Transfer Transfer only alpha and beta channels (color) Transfer only alpha and beta channels (color) Target ImageColorized Results

39. 3. Color Transfer Transfer only alpha and beta channels (color) Transfer only alpha and beta channels (color) The original luminance value remains unchanged The original luminance value remains unchanged Transfer only alpha and beta channels (color) Transfer only alpha and beta channels (color) The original luminance value remains unchanged The original luminance value remains unchanged Target ImageColorized ResultsConvert to Greyscale (Photoshop)

40. Results: satellite + TargetSource

41. Results: satellite += Final TargetSource

42. Results: Textures + TargetSource

43. Results: Textures += TargetSourceFinal

44. Limitations (Global Approach) += TargetSourceFinal

45. Limitations (Global Approach) += TargetSourceFinal +=

46. Limitations (Global Approach) += TargetSourceFinal +=

47. User-Assisted Approach 1. User selects small number of swatches 2. Transfer color only to swatches (Global Matching Procedure) 3. Color entire target image (Only use swatch samples) 1. User selects small number of swatches 2. Transfer color only to swatches (Global Matching Procedure) 3. Color entire target image (Only use swatch samples)

48. 1. Selection of Swatches TargetSource

49. 2. Transfer Color to Swatches Transfer color using Global Matching Procedure Transfer color using Global Matching Procedure (described previously) Transfer color using Global Matching Procedure Transfer color using Global Matching Procedure (described previously)

50. 3. Colorize Entire Image Discard original source image Discard original source image

51. 3. Colorize Entire Image Colorize the full image Colorize the full image Match using L 2 Norm Colorize the full image Colorize the full image Match using L 2 Norm

52. Swatches: Notes Expect good results between swatches Expect good results between swatches

53. Swatches: Notes Expect good results between swatches Expect good results between swatches

54. Swatches: Notes Expect good results between swatches Expect good results between swatches Expect better matching within an image Expect better matching within an image (Allows more precise metric: L 2 Norm) Expect good results between swatches Expect good results between swatches Expect better matching within an image Expect better matching within an image (Allows more precise metric: L 2 Norm)

55. Swatches: Notes Expect good results between swatches Expect good results between swatches Expect better matching within an image Expect better matching within an image (L 2 Norm is more sensitive to image differences) Expect good results between swatches Expect good results between swatches Expect better matching within an image Expect better matching within an image (L 2 Norm is more sensitive to image differences) BetweenWithinBetweenWithin

56. Results: Swatches TargetSourceFinal (Previous Method) +=

57. Results: Swatches TargetSourceFinal += +=

58. Results: Swatches TargetSourceFinal += +=

59. Results: Swatches +

60. Final +=

61. Results: Textures +

62. Final +=

63. Results: Swatches TargetSource +

64. Results: Swatches TargetSourceFinal + =

65. Limitations

66. Comments Works well when target can be segmented well Works well when target can be segmented well

67. Comments Works well when target can be segmented well Works well when target can be segmented well Large shadows present a problem Large shadows present a problem (partial volume effect) To be more useful, combine with other tools To be more useful, combine with other tools Works well when target can be segmented well Works well when target can be segmented well Large shadows present a problem Large shadows present a problem (partial volume effect) To be more useful, combine with other tools To be more useful, combine with other tools

68. Running Time Pentium 3 (800 Mhz): 15 sec – 1 min Pentium 3 (800 Mhz): 15 sec – 1 min Typical Image size: 640x480 Typical Image size: 640x480 Implemented using MATLAB (Optimized) Implemented using MATLAB (Optimized) Factors: Factors: Image size Neighborhood Size Pentium 3 (800 Mhz): 15 sec – 1 min Pentium 3 (800 Mhz): 15 sec – 1 min Typical Image size: 640x480 Typical Image size: 640x480 Implemented using MATLAB (Optimized) Implemented using MATLAB (Optimized) Factors: Factors: Image size Neighborhood Size

69. Video 1. Colorize one frame using swatches 2. Use swatches to colorize the entire sequence If a single frame in a sequence is colorized well, then the entire sequence can be colorized well 1. Colorize one frame using swatches 2. Use swatches to colorize the entire sequence If a single frame in a sequence is colorized well, then the entire sequence can be colorized well

70. Video Procedure 1. Colorize one frame using swatches

71. Video Procedure 1. Colorize one frame using swatches 2. Use swatches to colorize the entire sequence 1. Colorize one frame using swatches 2. Use swatches to colorize the entire sequence …… …… =

72. Video: Waves

73. Video: Horses

74. Video: Visible Human

75. Conclusions Keep original luminance values Keep original luminance values Use local pixel neighborhood statistics to match Use local pixel neighborhood statistics to match Simple algorithms provide fast (and good) results Simple algorithms provide fast (and good) results Keep original luminance values Keep original luminance values Use local pixel neighborhood statistics to match Use local pixel neighborhood statistics to match Simple algorithms provide fast (and good) results Simple algorithms provide fast (and good) results

76. Future Work Robustness: more sophisticated matching Robustness: more sophisticated matching Multi-resolution, other pattern matching metrics Volumes Volumes Color Correction Color Correction Use local neighborhood statistics Color correct movies automatically Robustness: more sophisticated matching Robustness: more sophisticated matching Multi-resolution, other pattern matching metrics Volumes Volumes Color Correction Color Correction Use local neighborhood statistics Color correct movies automatically

77. Questions?