Colorization is a user-assisted color manipulation mechanism for changing grayscale images into colored ones. Surprisingly, only a fairly sparse set of color hints may be required. Here, we examine “minimalist” sets of hints, and find results are not very convincing, compared to using more user input. But less input is better! We argue that matching the color-output image contrast (gradient) to that of the input grayscale contrast is key to generating realistic-looking images in color. The problem is in part due to unconstrained generation of the color, with contrast not truly linked to that in the original, gray image. Instead, we suggest automatically generating a more realistic (and hence more pleasing) result, where the output colors may not match every input control color, but with output generated by requiring the gradient of the color image to match that of the input grayscale image. Realistic Colorization via the Structure Tensor Mark S. Drew 1 and Graham D. Finlayson 2 1 School of Computing Science, Simon Fraser University, Vancouver BC, Canada V5A 1S6 2 School of Computing Sciences, University of East Anglia, Norwich, U.K. NR4 7TJ Fig. 2. Matching color-gradient to grayscale gradient. For further information Please contact Fig. 1. Realistic colorization algorithm. Fig. 3. Meaning of maximum-contrast gray: Fig. 4. Gray; hints; standard colorization; realistic colorization. Input Grayscale Image + User-Supplied Hints Standard Colorization Introduction Method + Match Color Contrast to Grayscale Contrast Unrealistic Colorization More realistic f() ] [    [ f() ? What is (Color)?What is  Color Structure Tensor. [Di Zenzo, Sapiro, Socolinsky & Wolff] In the case of a 3-band color image   = {R, G, B}, we first consider the 2x3 array of color-channel gradient 2-vectors. Now form the  2x2 array Structure Tensor Z: Z is real symmetric, so its eigenvectors form an orthogonal matrix V, Therefore Z is the matrix of outer products of the 2D color-channel gradient components: The (normalized) eigenvectors of Z point in the direction of minimum- and maximum-contrast, for an underlying grayscale image with metric induced by the structure tensor. Let V={u, v} with 2-vectors u  v. Then the maximum-eigenvalue direction v is associated with maximum contrast in the grey image, with norm. First – Consider: Color to Gray Color to Gray; Gray to Color  Color to Gray-Gradient  Gray-Gradient to Gray: Re-integrating gray-gradient is accomplished by taking another derivative and solving Poisson’s equation: we end up with a grayscale g whose gradient best matches the desired grayscale gradient : Suppose the tentative colorized image has structure tensor, and the SVD decomposition of the 2x3 color –channel gradient is Closest Colorized Edges to Gray Edges: Then, knowing O, we can replace these color edges by ones that exactly give back the grayscale gradient: To re-integrate, we go to a color-opponent space, and adopt the input gray image as the luminance; and solve Poisson’s equation for the chrominance channels [with homogeneous Neumann boundary, and setting the free additive constant to match the mean user input hint pixels]. Results Gray to Color has the same Z. When do two images have the same structure tensor Z?  Since Z is formed as an outer product of the gradient, then if O is two rows of any 3x3 orthogonal matrix, then every Let’s start from the structure tensor for a grayscale image and derive the properties a matching color image must have so as to deliver that same tensor. From the grayscale image we know: desired max-contrast direction v and orthogonal direction u, and gray-contrast. Images with Equal Tensors