Author: Vera Kukić Supervisors: Shaun Bangay Adele Lobb George Wells Investigating the Development of Colour Interpolation Techniques for GIMP Author: Vera Kukić Supervisors: Shaun Bangay Adele Lobb George Wells

Overview Presentation Overview Introduction Colour Transfer Design Implementation Results Conclusion

Introduction Motivation for the study Hence our Objective: Recent colourisation and colour interpolation techniques Bring the power of colourisation to common graphics packages Hence our Objective: Explore and evaluate colour interpolation plug-ins in the GIMP framework Integrate colourization into GIMP Allows GIMP users the ability to use a colourisation tool The best way to describe a project is to state its WHAT WHY HOW WHY- The motivation There have been recent advances in automated image colourisation (2001, 2002 papers) – which warrant investigation Moreover, the integration of colourisation techniques with in a common graphics and image manipulation package is of great benefit to ordinary users. They are able to easily colourise images in a fairly automated fashion (as opposed to traditional colour paletting methods) Applications for colourising old black and white family photographs, or movies ect abound WHAT – the research objective Hence, it was decided to explore and evaluate these emerging colourisation techniques in a wide used open source graphics package – GIMP The benefits of this are thus – not only can we use the GIMP framework as an implementation and evaluation platform but also provide a useful colourisation plug-in for the wider GIMP community Integrated it into a prolific graphics package so that the images can be alter in other ways as well – a stand alone application is not as useful

Colour Transfer Colour Transfer Techniques Colouring Book Method Photoshop: manually paint colour with low opacity Movie Industry: track polygons Pseudocolouring Global Transformation/Colour Map Colour Transfer between Images Colour Transfer to Greyscale Images

Colour Transfer Colour Transfer between Images [Reinhard et al] + = Source Target Final

Colour Transfer Colour Transfer to Greyscale Images [Welsh et al] + = Select colour source image Match each target pixel with a few source pixels choose best match using local neighbourhood statistics Transfer Colour Repeat for all pixels Source Target Final

Colour Transfer Algorithm Select Source and Target Images Convert Images to the lαβ colour space Image Matching Colour Transfer

Colour Transfer Algorithm Convert Images to the lαβ colour space lαβ space was developed to reduce the correlation between the 3 coordinate axis of the colour space l –> chromatic luminance channel α –> yellow-blue channel β –> red –green channel Ruderman developed the colour space and the

lαβ colour space the α and β values approximate to zero for neutral colours white, grey and black. α as a measurement of red for α positive or green for α negative β as a measurement of yellow for β positive or blue for β negative.

Colour Transfer Algorithm Convert Images to the lαβ colour space Images are converted using matrix manipulation RGB -> XYZ tristimulus values XYZ -> LMS space LMS -> lαβ (using log conversion)

Colour Transfer Algorithm Image Matching Create luminance histograms of the two images The image: has a histogram: Example of luminance histogram This is the representation for twelve pixels only, histograms for complete images are denser cover the complete range of all the luminance values {0,..,255}.

Colour Transfer Algorithm Image Matching Remap luminance histograms between source and target image Scale and shift distributions globally Luminance remapping, proposed by Hertzman et al, describes the process of matching the luminance histograms of the two images. The luminance of the target image remains unchanged and the luminance of the source image is shifted and scaled to fit the histogram of the target image. This transformation brings the histograms of the colour image and the greyscale image into correspondence Source -> Scaled Target and Shifted

Colour Transfer Algorithm Image Matching Pre-compute neighbourhood statistics for images Use sampling techniques to get colour samples from the source image Find best match from samples Weighted metric of luminance, mean and standard deviation Different sampling techniques can be used but the best one is jittered sampling as it reduces the number of samples that need to be made. For each pixel in the greyscale image find a best match in the colour image – compare the luminance values of the greyscale image and the samples from the colour image Use weighted average to base decision 50% luminance value 50% std deviation

Colour Transfer Algorithm Transfer the α and β channels only (colour) After the best match is found only transfer the alpha and beta values while retaining the original luminance value Extracting luminance information + luminance adjustment + sampling and colour transfer are generic techniques different techniques can be implemented for each one. Target Image Colourized Result

Swatches User-assisted approach User selects a small number of swatches Transfer colour only to swatches Colour entire target image Allow more user interaction User can selectively transfer colours between corresponding swatches The user selectively chooses the regions to place swatches. The same colour transfer procedure is used as in the global matching procedure except only between the source and target swatches.

Swatches Select swatches = + Source Target Final Advantage is the selective colour transfer between the swatches which prevents the pixels with similar neighbourhood statistics but from the wrong part of the image from corrupting the target swatch colours. Swatches allow for interactive colouring and give the user the power to colour areas that do not correspond in composition The use of swatches reduces the number of comparisons needed to be made by the greyscale image to find the perfect match. The comparisons are thus only made between the relating swatches. The rest of the corresponding region is coloured in using texture synthesis Expect better results between swatches Expect better matching within an image Source Target Final

Design User Interface The plug-in is experimental designed in an extensible way – release the source code so that it can be perfected – extensible because of the way it has been designed GIMP 1.2 works under Linux and Windows (cross platform)

Design Extensible – can try deifferent matching and sampling techniques….

Implementation Environment Plug-in Architecture Development Issues

Implementation Environment GIMP – is the GNU Image Manipulation Package Open source – free  Suitable for image retouching, altering and image composition. Uses external modules known as plug-ins GIMP caters for C and Pearl Our choice was based on prior knowledge, available resources and the functionality that the language offers. Pearl is a scripting language and is not widely used. In terms of learning there is not much written in Pearl. C is a low level, powerful language with a large and diverse user base. GIMP is an open source image manipulation package. It has most of the functionality other image manipulation packages do

Implementation Plug-in definition: “A file containing data used to alter, enhance or extend the operation of a parent application program”. Free On-line Dictionary of Computing

Implementation Plug-in Architecture Implement a specified interface Plug-in must Register itself in the Procedure Database PDB Have a main() function Signal handling Communication PLUG_IN_INFO – global variable query() – call to gimp_install_procedure run() – sets the status code return value For GIMP to invoke a developed plug-in the plug-in needs to implement a specified interface GIMP interfaces with plug-ins by searching for files of the appropriate format and incorporating them into its menu structure. The search is only done across the plug-in sub-directories. The position where the plug-in is incorporated into the GIMP structure is defined within the plug-in Gimp plug-ins are queried on start up and this when they register themselves in the PDB Main() – is the first function that is called when the program is executed sets up signal handling and communication between gimp and the plug-in PLUG_IN_INFO – global variable used by the query procedure consists of pointers to four functions run() procedure makes sure that the plug-in is called correctly and that the return values are set correctly query() – call to gimp_install_procedure

Implementation Development Issues GIMP API poorly documented Wade through gimplib source Trial and Error implementation Negatively affected development timescales Experiences with C GUI development using GTK tedious Lack of Documentation is often synonymous with open source development  Frequently had to adopt a trial and error development style in order to discover the exact behaviour of functions in GIMP lib GIMP handling of pixels is byte level – wrapper data types that deal with pixels and pixel areas

Results = + Using grid sampling – divide image into grid and sample from middle of each grid

Results + =

Conclusion Application Enhance Scientific Data Medical Imaging Satellite Images Scanning Electron Microscopy Colourize black/white photographs and movies Artistic Effects

Conclusion Future Work Robustness: more sophisticated matching Colour correction Automated choice of source image Different metric for texture matches and similarity – allow for small surface colouring

Conclusion Implemented a new general, fast and user-friendly approach to colouring greyscale images

Questions? ?