1. Palette-based Photo Recoloring
Huiwen Chang
Ohad Fried
Yiming Liu  Stephen DiVerdi
Adam Finkelstein
SIGGRAPH 2015

2. Outline
Introduction
Approach
Evaluation
Results
Conclusion

3. Introduction
Demo:

4. Introduction
Goal
original

5. Introduction
This paper introduces a tool that is easy for novices to learn while offering a broad expressive range.
Our approach specifies both the colors to be manipulated and the modifications to these colors via a color palette.
We show that our algorithm can easily be accelerated that allows it to run at interactive frame rates.

6. Introduction
We perform a study showing that with our tool untrained users can produce similar results to those of expert Photoshop users.
Finally, we show that our palette-based color transfer framework also supports other interfaces.
stroke-based interface
localized editing via a selection mask
fully-automatic palette improvement
editing a collection of images simultaneously

7. Approach Monotonic Luminance Transfer Color Transfer User interface
Automatic Palette Selection
Color Transfer Goals
Acceleration
Simple
Expressive
Intuitive
Responsive
(k + 1)-means
Interpolation
In Gamut
Pixel Continuity
Palette Continuity
One-to-One
Monotonicity in L
Dynamic Range

8. Approach(1) -User interface
These criteria are important for a color manipulation user interface:
Simple
Expressive
Intuitive
Responsive
While there are many existing tools that allow users to recolor images, to our knowledge none of them simultaneously achieve these goals.

9. Approach(1) -User interface
We describe a palette-based GUI
The user needs only to click on a palette color (C) and change it (to C’) via a HSL color picker.

10. Approach(2) - Automatic Palette Selection
Our automatic approach for creating a palette based on an image, using a variant of the k-means algorithm.
Our goal is to select a set of k colors { 𝐶 𝑖 } that distill the main color groups in the image, to be used as “controls” during editing.
We leave the choice of k up to the user and find that k ∈ [3, 7] works well for typical operations, and use k = 5 by default.

11. Approach(2) - Automatic Palette Selection
Organizing the data in K-D trees [Kanungo et al. 2002].
Exploiting the property that our data are colors restricted to R, G, B ∈ [0, 1], we assign them to bins in a b × b × b histogram (we use b = 16 in RGB).
For each bin we compute the mean color in Lab space, and these 𝒃 𝟑 colors 𝒄 𝒊 (or less because some bins may be empty) are the data we use for k-means.
Each data point 𝑐 𝑖 now represents the 𝑛 𝑖 pixels associated with that bin, we use a weighted mean (weighted by 𝑛 𝑖 ) when finding each of the k means in each iteration.
KANUNGO, T., MOUNT, D., NETANYAHU, N., PIATKO, C., SILVERMAN, R., AND WU, A An efficient k-means clustering algorithm: analysis and implementation. Pattern Analysis and Machine Intelligence, IEEE Transactions on 24, 7 (Jul), 881–892.

12. Approach(2) - Automatic Palette Selection
Instead of randomizing, we initialize the means as follows.
We initialize the first “mean” as the color 𝑐 𝑖 representing the bin with the largest weight 𝑛 𝑖 .
Next we attenuate all other weights 𝑛 𝑗 by a factor
Next we choose bin with the highest remaining weight 𝑛 𝑖 , repeating until k initial “means” have been chosen.
falloff

13. Approach(2) - Automatic Palette Selection
We discourage very dark palette entries as follows.
Rather than computing k-means, we actually compute (k + 1)-means, where one of them is initialized and perpetually locked to black.
The darkest colors in the image will be assigned to this mean, and will therefore not pull the other nearby means towards black.

14. Approach(2) - Automatic Palette Selection
For a comparison of our approach with the previously described methods.
(D) O’Donovan et al. [2011]
(L) Lin et al. [2013]
(G) Gaussian mixture models (GMMs) used by Shapira et al. [2009]
(K) K-means
(O) Our method
10s 2s
60ms
O’DONOVAN, P., AGARWALA, A., AND HERTZMANN, A Color Compatibility From Large Datasets. ACM Transactions on Graphics 30, 4.
LIN, S., AND HANRAHAN, P Modeling how people extract color themes from images. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’13.
SHAPIRA, L., SHAMIR, A., AND COHEN-OR, D Image appearance exploration by model-based navigation. In Computer Graphics Forum, vol. 28, 629–638.

15. Approach(3) - Color Transfer Goals
The association ({ 𝐶 𝑖 }, { 𝐶′ 𝑖 }) defines a transfer function 𝑓 that maps colors in the original image to colors in the edited image.
We assume that 𝑓 acts on colors independently of pixel location or context in the image.

16. Approach(3) - Color Transfer Goals
Interpolation.
In Gamut.
Pixel Continuity.
Palette Continuity.
One-to-One.
Monotonicity in L.
Dynamic Range
A gradient in the input maps to a gradient in the output (not a single value).

17. Approach(3) - Color Transfer Goals
To our knowledge no existing approaches can be easily adapted to meet the requirements listed above.
To make matters worse, the requirements themselves are inconsistent.

18. Approach(4)-Monotonic Luminance Transfer
In the GUI
The GUI constrains the relative ordering of luminance 𝐿′ 𝑖 of the edited palette colors 𝐶′ 𝑖 .
Whenever the user modifies 𝐶′ 𝑖 , the GUI also sets
To change the luminance of a palette entry we simply modify the L channel in Lab space.
This operation is evaluated in increasing order for all palette entries j > i; and the symmetric operation (involving min and 𝐿′ 𝑗+1 ) is applied in decreasing order for entries where j < i.
in the input palette
in the edited palette

19. Approach(4)-Monotonic Luminance Transfer
In the transfer function
we design a transfer function that has two orthogonal components – 𝑓 𝐿 and 𝑓 𝑎𝑏 .
The luminance transfer function simply takes a weighted combination of the two nearest palette entries.

20. Approach(5) - Color Transfer (ab)
The design of the function 𝑓 𝑎𝑏 is not guided by the monotonicity concern , but it does target the other requirements.
First we devise the function 𝑓 1 for the simple case where the original palette contains a single color C , and the user modifies it to be color C’.
For any color 𝑥 we would like to know 𝑥 ′ = 𝑓 1 𝑥 .
A naive strategy

21. Approach(5) - Color Transfer
We devise a scheme
Finally, we take 𝑓 1 𝑥 = 𝑥 ′ , the point on the ray from 𝑥 to 𝑥 𝑏 such that:
we determine if 𝒙 𝟎 = 𝒙 + 𝑪 ′ − 𝑪 is in gamut.
These intersections are found by binary search.

22. Approach(5) - Color Transfer
Define 𝑘 transfer functions 𝑓 𝑖 𝑥
For the weights we use radial basis functions (RBFs):
Found that the Gaussian kernel function works well in our application:
The 𝑘 2 unknown coefficients λ 𝑖𝑗 are found by solving a system of 𝑘 2 equations: 𝑘 of which require ω 𝑖 ( 𝐶 𝑖 ) = 1 and 𝑘 2 − 𝑘 of which require ω 𝑗≠𝑖 (𝐶 𝑖 ) = 0.

23. Approach(5) - Color Transfer
Unfortunately solving the system of equations set up by the RBFs can lead to negative weights, with two potential hazards.
It will add some component of the opposite behavior of some palette changes.
It can throw the result out of gamut.
We clamp any negative weights to zero and renormalize the non-zero weights.
a palette size of k = 3

24. Approach(6) - Acceleration
We further accelerate the computation by caching these weights ω 𝑖 only at a 𝑔×𝑔×𝑔 grid of locations uniformly sampled in the RGB cube.
During color editing when the 𝐶′ 𝑖 are known, we can compute the output color 𝑓(𝑥) at these 𝑔 3 locations using the precomputed weights.
Finally, to recolor each pixel in the image use trilinear interpolation on eight nearest grid values.

25. Approach(6) - Acceleration
Without acceleration, our runtime is
With acceleration, runtime is
Throughout this paper and in our demo we use g = 12, because it performs well across a range of computers and images, so
Results of varying grid size g
Update time
vs. grid size

26. Evaluation(1) -Other Recoloring Methods
Chen et al. [2014]
Shapira et al. [2009]
Wang et al. [2010]

27. Evaluation(2) -User Study
In order to judge expressiveness, the task we give our users is a matching task.
We collected 32 original-target pairs.
-Half (16) were selected from eight papers.
-The other half were created by two expert Photoshop users.
For the study we used Amazon’s Mechanical Turk framework.
Each worker was assigned randomly to one of three different conditions: our method, GMM, and hue-blend.

28. Evaluation(2) -User Study

29. Evaluation(2) -User Study
Aggregates all distance-from-target results.
better

30. Evaluation(2) -User Study
Shows completion times over all subjects, for each of the three methods.
roughly the same
slower
slowest

31. Results
Show examples of using our method with palettes of size 3 and 5.

32. Results
Our method supports masks, constraining the edits to specific image regions.

33. Results(1) -Video Recoloring
Our method is fast enough to be applied to a video, in real time. The interaction uses the same interface as for photos.

34. Results(2) -Duotone
Duotone reproduction is a traditional printing technique that typically involves two colors of ink applied via halftone patterns over white paper. Our method can easily produce a similar effect.

35. Results(3) -Automatic Color Manipulation
We can automatically pick a top rated palette and apply it to an image.

36. Results(4) -Editing an Image Collection
We show that our system can trivially operate on a collection of images simultaneously.
original
collection
result

37. Results(5) -Stroke-Based Interface
Our palette based editing method can easily be augmented with a stroke-based interface that relies on the same algorithms described.
We set the palette size:
Stroke-based editing.
the approach of Chen et al. [2012] our method
input
output
local
global
input
output

38. Conclusion
This paper introduces a simple, intuitive and interactive tool that allows non-experts to recolor an image by editing a color palette.
We show results for recoloring, image collection editing and video editing, and validate via a user study.

39. References
histogram adjustment methods in Photoshop
HSL