1. Color transfer between high-dynamic-range images
H. Hristova, R. Cozot,
O. Le Meur, K. Bouatouch
University of Rennes 1
Rennes, France

2. Outline Introduction Main objective Contributions
Extension to the HDR domain of a color transfer method
Results and evaluation
Generalization for state-of-the-art color transfer methods
Conclusion
Outline of the presentation:
Introduction to the main objective and the main contributions of the paper
Presenting an extension to the HDR domain of a novel local color transfer method
(Followed by) results and evaluation
Generalizing the developed extension to other state-of-the-art color transfer methods + results 2

3. Main goal
Carrying out a color transfer between two HDR images directly in the HDR domain
Our main goal is to carry out a color transfer between two HDR images directly in the HDR domain. -Why? -Because tone mapping operators compress the information contained in HDR images and therefore the plausibility of the tone-mapped images cannot be guaranteed.
Solution to the problem: to apply color transfer methods between two HDR images. Color transfer methods build a transformation mapping between two LDR images and aim at “borrowing” the look (mood) of a reference image. Can color transfer methods be directly applied to the HDR domain? -> next slide
Input
Reference
Solution: apply color transfer methods to stylize an HDR image with regards to a reference image 3

4. Why do LDR color transfer methods need to be extended to the HDR domain?
LDR color spaces
- well predict the color gamut for luminance levels between zero and the display white point
- uncertain applicability to HDR images
Color trend above the perfect diffuse white
The color transfer methods need to be extended to HDR images. -Why? -First: Traditional LDR color spaces, such as lab and CIE Lab, which are widely used in color transfer algorithms, predict well the color gamut for luminance levels below the display white point. However, when it comes to luminance levels way beyond the display white point, like those of HDR images, the applicability of these color spaces becomes uncertain. For example, when the lightness reaches high levels (above 255 in the figure), the imagery primaries (x, y) ranging from 0 to 1 cannot depict and predict the color gamut for these high levels of luminance. To predict the color trend above the perfect diffuse white, we need to perform some modifications to traditional color spaces. 4

5. Why do LDR color transfer methods need to be extended to the HDR domain?
Assumption: a unique multivariate Gaussian distribution
HDR domain: to fit the high range of lightness of HDR images we need to assume mixture of Gaussian distributions
Second: most color transfer methods build their transformations based on the assumption that a multivariate Gaussian distribution is sufficient to fit the luminance and chroma variations. However, this is hardly true in the LDR domain. The latter holds for the HDR domain. The high luminance range of an HDR image cannot be fitted only by one Gaussian distribution. Therefore, we need to assume a mixture of Gaussian distributions. 5

6. Why do LDR color transfer methods need to be extended to the HDR domain?
Lightness - approximated by luminance in the LDR domain
HDR domain - distinguish between the absolute luminance and the lightness (the L channel of CIE Lab)
Finally: Usually, the luminance is approximated by the lightness (the L channel from CIE Lab color space) in the LDR domain. Nevertheless, there is a distinguish between the luminance and the lightness in the HDR domain. On one hand, the luminance of HDR images is presented by the Y channel in XYZ color space and it is called absolute luminance. On the other hand, lightness is represented by the L channel of CIE Lab color space. Both of them are not equal in the context of HDR imagery. Therefore, color transfer methods which use these two notions in their algorithms need to be properly adapted to the HDR domain. 6

7. Contributions
Adaptation of [Hristova et al., 2015] color transfer method to HDR images
- HDR color spaces
- Modifications of the clustering step and of the image classification
Cluster-based local chromatic adaptation transform
Generalization for state-of-the-art color transfer methods
Our main contributions are:
the development of extension to HDR images of a new local color transfer method [Hristova et al., 2015], consisting of:
- modifications of traditional color spaces (their replacement with HDR-based ones)
- modifications of the clustering and classification algorithms of the local method: distinguishment between lightness and luminance
The development of novel cluster-based local chromatic adaptation transform for adapting the colors of the input HDr image to a reference white point. 7

8. Extension to HDR images
Input and reference images
Color space conversion
Image classification
Clustering and mapping
Color transfer
Chromatic adaptation transform
Final result
[Hristova et al., 2015]
Linear search for significant peaks in the image hue histogram
The number of significant peaks determines the cluster number
- Colors-based style images: hue histogram
- Light-based style images: luminance histogram
The local method by [Hristova et al., 2015] is a new method for color and style transfer consisting of several steps. These steps (either independently or as a combination) can be used in other color transfer methods. Therefore, we will first present the extension of [Hristova et al., 2015] and then, we will generalize that extension to other color transfer methods.
[Hristova et al., 2015] starts with a conversion to CIE Lab color space. Once both images are converted to that color space, they are classified according to their main features (either colors or light) as shown on the figure. Two types of images are considered: colors-based and light-based ones. The classification boils down to finding significant peaks in the hue histogram of both images. During the classification step, we determine also the number of clusters which will be passed to the clustering process. On one hand, the number of color clusters is determined by the hue histogram. On the other hand, the number of luminance clusters is defined by the number of significant peaks in the luminance histogram of the image.
After the classification, both images are clustered to Gaussian clusters and then, the input/reference clusters are mapped. Four policies are developed to map the clusters. They take into account both the luminance histogram and the luminance-hue distributions of images.
A color transfer is carried out between each pair of corresponding clusters. An optimal transformation is used (Monge-Kantorovich optimization problem). Finally, a chromatic adaptation transform is applied to adapt the colors of the output image to the reference illuminant. 8

9. Extension to HDR images
LDR images
CIE Lab
L channel of CIE Lab
Input and reference images
Color space conversion
Image classification
Clustering and mapping
Color transfer
Chromatic adaptation transform
Final result
[Hristova et al., 2015]
HDR images
hdr-CIELab
Log-
luminance
Dashed line: cubic function of L channel (CIE Lab)
Solid line: Michaelis-Menten function by which we replace the cubic function of L channel (CIE Lab)
hdr-CIELab color space [Fairchild et al., 2004]
The following extension of [Hristova et al., 2015] is proposed. The extension is applied to HDR images instead to LDR images.
One of the issues of the direct applying of the method to HDR images has been already discussed: the uncertainty in predicting the color gamut for high luminance values. Therefore, instead of using the LDR CIE Lab color space, we perform modifications to its channels to derive hdr-CIELab color space [Fairchild et al., 2004]. The cubic function of the L channel (the dashed line) is replaced by Michaelis-Menten function (the solid line) for better approximation of the lightness in HDR images. Moreover, this modification influences the predictability of the color trend for HDR images.
Furthermore, as the image classification depends strongly on the number of significant peaks in the image hue histogram, if we ensure a better predictability of the image hue, then we ensure the successful classification of images and the successful estimation of the number of color clusters (as the number of significant peaks corresponds to the number of color clusters). How about the number of light/luminance clusters? -> we adopt log-luminance histogram in the place of the L channel histogram -> explanation on the next slide.
[Fairchild et al., 2004] 9

10. Extension to HDR images
LDR images
CIE Lab
L channel of CIE Lab
L-based clustering
Input and reference images
Color space conversion
Image classification
Clustering and mapping
Color transfer
Chromatic adaptation transform
Final result
[Hristova et al., 2015]
HDR images
hdr-CIELab
Log-
luminance
Log-
luminance clustering
To find the optimal number of clusters for an image during the classification step, we need to find number of significant peaks in the luminance histogram of the image. Moreover, light-based images are clustered according to the luminance histogram. Finally, three of the mapping policies are based on luminance histogram information. In the LDR domain we can use the L channel of CIE Lab color space in the place of the luminance. However, in the HDR domain we use the absolute luminance and more precisely, the logarithmic transform of the absolute luminance (which we call log-luminance). The log-luminance is a good approximation of the brightness. Furthermore, the logarithmic transform preserves the regions of minima and maxima of a histogram and therefore we are able to recover the shadows, midtones and highlights from the log-luminance histogram. To this end, log-luminance histogram is used in the classification and clustering steps of the algorithm in the place of the lightness channel of hdr-CIELab color space.
Once we carry out the image clustering and we map the reference to the input clusters, we apply an optimal transformation between each pair of corresponding clusters. We call the obtained image an output. The last step adapts the colors of the output image to the reference illuminant.
Logarithmic
transform 10

11. Extension to HDR images
LDR images
CIE Lab
L channel of CIE Lab
L-based clustering
Local CAT
Input and reference images
Color space conversion
Image classification
Clustering and mapping
Color transfer
Chromatic adaptation transform
Final result
[Hristova et al., 2015]
HDR images
hdr-CIELab
Log-
luminance
Log-
luminance clustering
Cluster-
based local CAT
In the LDR domain, CAT algorithm adapts the input image to a global reference point. The novel cluster-based local CAT algorithm, however, first clusters both images into regions (highlights, midtones, shadows). We search for significant peaks in the log-luminance histogram and then, we define the region limits as the minima between two significant peaks. 11

12. Extension to HDR images
LDR images
CIE Lab
L channel of CIE Lab
L-based clustering
Local CAT
Input and reference images
Color space conversion
Image classification
Clustering and mapping
Color transfer
Chromatic adaptation transform
Final result
[Hristova et al., 2015]
HDR images
hdr-CIELab
Log-
luminance
Log-
luminance clustering
Cluster-
based local CAT
(h)
(sh)
(m)
(h)
Like in the LDR domain, we apply a low-pass Gaussian filter to the input image to compute locally (like a “white” image) the input illuminant. Then, we cluster the input image into regions. Each region corresponds to part of the “white” image which we take into account to carry out the transform region-wisely.
As far as the reference image is concerned, we estimate a representative white point for each reference region. Then, this white point becomes a global white point for the region. The reference white points for each region are shown in blue dashed lines.
(m)
Gaussian low-pass filter
(sh) 12

13. Extension to HDR images
LDR images
CIE Lab
L channel of CIE Lab
L-based clustering
Local CAT
Input and reference images
Color space conversion
Image classification
Clustering and mapping
Color transfer
Chromatic adaptation transform
Final result
[Hristova et al., 2015]
HDR images
hdr-CIELab
Log-
luminance
Log-
luminance clustering
Cluster-
based local CAT
Here is a video example of how the result changes over the iterations. Around the 10th iteration, the input image is overall adapted to the reference illuminant. However, there is a slight cast of the greenish input reference on the result. It is removed at the end of the iteration.
Input
Cluster-based local CAT
Reference 13

14. Extension to HDR images
LDR images
CIE Lab
L channel of CIE Lab
L-based clustering
Local CAT
Input and reference images
Color space conversion
Image classification
Clustering and mapping
Color transfer
Chromatic adaptation transform
Final result
[Hristova et al., 2015]
HDR images
hdr-CIELab
Log-
luminance
Log-
luminance clustering
Cluster-
based local CAT
Another example of illuminant adaptation (video).
Input
Cluster-based local CAT
Reference 14

15. Objective evaluation of the results
10 image pairs
Two tone-mapping operators: [Durand et al., 2002] and [Reinhard et al., 2002]
SSIM and Bhattacharya coefficient
We used 10 image pairs to obtain 10 HDR image results for the original [Hristova et al., 2015] method and its extension. We evaluated the results in the LDR domain after tone-mapping them. SSIM and Bhattacharya coefficient are two complementary metrics (used also by [Hristova et al., 2015]) measuring respectively the degree of artifacts and how successful the color transfer is.
The HDR extension obtains the highest scores for both metrics and for both tone-mapping operators. 15

16. Results [Hristova et al., 2015] HDR extension Input
Color transfer with CAT
Color transfer with CAT
The results support the objective evaluation.
The first column represents the input and reference images. The second column shows the results obtained with the method [Hristova et al., 2015] directly applied to HDR images (with and without local CAT). The third column shows the results obtained with the HDR extension of the method (with and without local CAT).
If we apply [Hristova et al., 2015] directly to HDR images, then the method it opt for artifacts. Moreover, the details are lost. On the other hand, the HDR extension manages to preserve the details and properly transfers the reference colors to the output image.
Reference
Color transfer without CAT
Color transfer without CAT 16

17. Generalization and results
Input
Reference
Now we show several results for state-of-the-art methods. The first row consists of the input and reference images. The first image on the second row is the result of the direct applying of the state-of-the-art method to HDR images and the second image on the second row presents the result of the same state-of-the-art method, modified in order to enhance the color transfer between two HDR images.
[Reinhard et al., 2002] is a global method which assumes that the light and colors of the image can be fitted by a unique Gaussian distribution. As discussed, one Gaussian distribution is not sufficient to fit the high luminance range of HDR images. Therefore, to enhance the effect of the color transfer, we recommend to carry out Reinhard’s transformation locally on pairs of clusters. To this end, we recommend [Tai et al., 2005] to be used in the place of [Reinhard et al., 2005] in the HDR domain.
[Tai et al., 2005] - clustering (local transformations)
[Reinhard et al., 2001] - global method 17

18. [Pitié et al., 2007] - hdr-CIELab
Generalization and results
Input
Reference
[Pitié et al., 2007] is a global method which applies optimal mapping between the input and reference images. In the LDR domain, the methods adopts CIE Lab color space. As we can see from the first result on the figure, CIE Lab is less effective from this type of transformation between HDR images than hdr-CIELab. The latter color space predicts better the color palette of the reference image and manages to transfer it better to the final result.
[Pitié et al., 2007] - CIE Lab
[Pitié et al., 2007] - hdr-CIELab 18

19. Generalization and results
Input
Reference
As a local method, [Bonneel et al., 2013] performs clustering to both input and reference images. The clustering is luminance-based. To extend the method to HDR images, we replace the luminance-based clustering with log-luminance-based clustering (as shown for [Hristova et al., 2015] earlier in the presentation). Moreover, instead of carrying out the transformation in CIE Lab color space, we adopt hdr-CIELab color space in the HDR domain. That way, the proposed extension lessens the visual artifacts in the final result (in comparison to the direct application of [Bonneel et al., 2013]).
[Bonneel et al., 2013] - log-luminance clustering
[Bonneel et al., 2013] - luminance clustering 19

20. Conclusion
Extension of a novel local color transfer method [Hristova et al., 2015]
Modifications to CIE Lab -> hdr-CIELab
Luminance/Lightness -> Log-luminance
Generalization to state-of-the-art methods
Future work
Need for a more precise color mapping/color transformation between two HDR images
HDR color spaces 20

21. Thank you for your attention!21