Introduction to compositing

What is compositing?  The combination of two images to produce a single image  Many ways we can do this, especially in the digital domain  Perform operations on pixel values of each image according to a given equation  We treat an image as having three channels: red, green and blue.  These combine to produce the whole image

Simple example: add  Simply add the values for each channel (R, G and B) of the (N x M) images P and Q:  For i = 0 to N-1 For j = 0 to M-1 RGB_O ij = RGB_P ij + RGB_Q ij  Generally this won’t be very useful!

+ =

Other ways of combining images  Usually when we are combining images one can be considered a background image and one a foreground image  Often there may be more than two images, but they can usually be split into background and foreground pairs  What we need is a way to decide which pixels from each image we use at each location in the image

Simple example  We have a background image from which we want to use the left half combined with the right half of the foreground image + =

Simple example  For i = 0 to N-1 For j = 0 to M-1 If i<(N/2) RGB_O ij = RGB_BG ij Else RGB_O ij = RGB_FG ij

Compositing  We need a flexible way of deciding which pixels from which image we use at each position  To do this we can create an N x M array of values that define which pixels we use from which image.  As a simple example, use a value of 0 if we want to use the background pixel and 1 if we want to use the foreground pixel  This N x M array is itself an image, consider 0 as black and 1 as white  For the previous example:

Mattes  Such an image is a called a matte  The black and white pixels can occur anywhere and the image can be created using any image production technique (e.g. manually in Photoshop)  In general, we can use shades of grey to define different amounts of each image rather than a simple ‘on-off’ switch  Consider a more realistic example…

The process  We’ll now look at how this works in principle  We use the equation: O = (FG x M) + (BG x (1-M)) where BG is the background image, FG is the foreground image and M is the matte  This means that we multiply the foreground by the matte and the background by the inverse of the matte before adding the results

x x= = +=

Generating mattes  Mattes can be created in many ways, ranging from fully automatically to fully manually  If the images we’re using are generated by a computer (e.g. as part of a computer animation) then the alpha information may be generated at the same time as the images  If the images are from live action video or film, we need to create the matte either manually or automatically  For anything but simple images, this is complex

Creating mattes from images  If we have to create a matte for an existing image, there are ways that we can ease the process  If the pixels that we want replaced in the image have some common feature, we can use this to create our matte  This is what the ‘blue screen’ (or ‘green screen’) technique is based on

Colour keying  If all the pixels in our foreground image that we want replaced with our background image are the same colour it is easy to create a matte  With video/film, we can achieve this by filming the action against a constant colour backdrop  Usually a bright blue or green is used because these colours do not occur in natural scenes very often  When we post-process the images, we choose the backdrop colour to define our matte  This process is called colour keying

Example Blue screen originalGenerated matte Background imageComposited image

Colour keying  In practice, no matter how evenly we light the backdrop it will not be exactly one shade of colour  We therefore usually use colours within a certain threshold of the chosen one  We can also feather or soften the edges to help blend the two images and make the composite look more natural

Creating mattes  If we cannot shoot the original footage against a backdrop we need other ways to create a matte  One way is to import the images into an image manipulation package and create a matte manually

Manually creating mattes  This is obviously a labour intensive process, especially as in general we will be using sequences of moving images at between 24 and 30 frames per second!  There are ways of helping the process semi-automatically using image processing techniques

Image Processing  We can enhance the image to help define a matte by processing the image  One approach is to enhance the edges in the image by high pass filtering  Edges represent high frequencies in the image and so can be brought out by filtering the image

High pass filtering  We can high pass filter an image by looking at the difference between a pixel and its surrounding pixels  If they are very different, then there is a high frequency present and probably an edge  We move across the image pixel by pixel

Feature tracking  If the element of the image that we are trying to composite is similar in each frame (e.g. it is a an object that moves across the frame), then we can create a matte and track the object with it  This relies on automatically locating the object in each frame and moving the matte accordingly  To do this we can specify certain features to assist tracking

Example

Alpha channels  The matte information can be considered as another channel in addition to the RGB information  Hence, if we are using a 24 bit system for colour information (8 bits for each colour) we store an additional 8 bits of alpha  We then have an RGBA, 32 bit image

In Premier