9/28/04© University of Wisconsin, CS559 Fall 2004 Last Time Filtering –Box –Bartlett –Gaussian –Edge Detect (High-Pass) –Edge Enhance –How to apply filters Intro to resampling Project 1

9/28/04© University of Wisconsin, CS559 Fall 2004 Today Resampling Ideal Reconstruction Aliasing Compositing Homework 2 due

9/28/04© University of Wisconsin, CS559 Fall 2004 General Scenario You are trying to create a new image of some form, and you need data from a particular place in the existing image –Always: Figure out where the new sample comes from in the original image New Original

9/28/04© University of Wisconsin, CS559 Fall 2004 Resampling at a Point We want to reconstruct the original “function” at the required point We will use information from around the point to do this We do it using a filter –We will justify this shortly Which filter? –We’ll look at Bartlett (triangular) –Other filters also work You might view this as interpolation, but to understand what’s happening, we’ll view it as filtering ? Use these to reconstruct

9/28/04© University of Wisconsin, CS559 Fall 2004 Resampling at a Point Place a Bartlett filter at the required point Multiply the value of the neighbor by the filter at that point, and add them –Convolution with discrete samples The filter size is a parameter Say the filter is size 3, and you need the value at x=5.75 –You need the image samples, I(x), from x=5, x=6 and x=7 –You need the filter value, H(s), at s=-0.75, s=0.25 and s=1.25 –Compute: I(5)*H(-0.75)+I(6)*H(0.25)+I(7)*H(1.25)

9/28/04© University of Wisconsin, CS559 Fall 2004 Functional Form for Filters Consider the Bartlett in 1D: To apply it at a point x orig and find the contribution from point x where the image has value I(x) Extends naturally to 2D: w/2-w/20

9/28/04© University of Wisconsin, CS559 Fall 2004 Common Operations Image scaling by a factor k (e.g. 0.5 = half size): –To get x orig given x new, divide by k: Image rotation by an angle  : –This rotates around the bottom left (top left?) corner. It’s up to you to figure out how to rotate about the center –Be careful of radians vs. degrees: all C++ standard math functions take radians, but OpenGL functions take degrees

9/28/04© University of Wisconsin, CS559 Fall 2004 Ideal Image Size Reduction To do ideal image resampling, we would reconstruct the original function based on the samples A requirement for perfect enlargement or size reduction –Almost never possible in practice, and we’ll see why

9/28/04© University of Wisconsin, CS559 Fall 2004 An Reconstruction Example Say you have a sine function of a particular frequency And you sample it too sparsely You could draw a different sine curve through the samples

9/28/04© University of Wisconsin, CS559 Fall 2004 Some Intuition To reconstruct a function, you need to reconstruct every frequency component that’s in it –This is in the frequency domain, but that’s because it’s easy to talk about “components” of the function But we’ve just seen that to accurately reconstruct high frequencies, you need more samples The effect on the previous slide is called aliasing –The correct frequency is aliased by the longer wavelength curve

9/28/04© University of Wisconsin, CS559 Fall 2004 Nyquist Frequency Aliasing cannot happen if you sample at a frequency that is twice the original frequency – the Nyquist sampling limit –You cannot accurately reconstruct a signal that was sampled below its Nyquist frequency – you do not have the information –There is no point sampling at higher frequency – you do not gain extra information Signals that are not bandlimited cannot be accurately sampled and reconstructed –They would require an infinite sampling frequency –Can you reconstruct something with a sharp edge in it? How do you know where the edge should be?

9/28/04© University of Wisconsin, CS559 Fall 2004 Sampling in Spatial Domain Sampling in the spatial domain is like multiplying by a spike function You take some ideal function and get data for a regular grid of points 

9/28/04© University of Wisconsin, CS559 Fall 2004 Sampling in Frequency Domain Sampling in the frequency domain is like convolving with a spike function –Follows from the convolution theory: multiplication in spatial equals convolution in frequency –Spatial spike function in the frequency domain is also the spike function  Original Sampled

9/28/04© University of Wisconsin, CS559 Fall 2004 Reconstruction (Frequency Domain) To reconstruct, we must restore the original spectrum That can be done by multiplying by a square pulse  Sampled Original

9/28/04© University of Wisconsin, CS559 Fall 2004 Reconstruction (Spatial Domain) Multiplying by a square pulse in the frequency domain is the same as convolving with a sinc function in the spatial domain 

9/28/04© University of Wisconsin, CS559 Fall 2004 Aliasing Due to Under-sampling If the sampling rate is too low, high frequencies get reconstructed as lower frequencies High frequencies from one copy get added to low frequencies from another  

9/28/04© University of Wisconsin, CS559 Fall 2004 More Aliasing Poor reconstruction also results in aliasing Consider a signal reconstructed with a box filter in the spatial domain (square box pixels, which means using a sinc in the frequency domain):  

9/28/04© University of Wisconsin, CS559 Fall 2004 Aliasing in Practice We have two types of aliasing: –Aliasing due to insufficient sampling frequency –Aliasing due to poor reconstruction You have some control over reconstruction –If resizing, for instance, use an approximation to the sinc function to reconstruct (instead of Bartlett, as we used last time) –Gaussian is closer to sinc than Bartlett –But note that sinc function goes on forever (infinite support), which is inefficient to evaluate You have some control over sampling if creating images using a computer –Remove all sharp edges (high frequencies) from the scene before drawing it –That is, blur character and line edges before drawing

9/28/04© University of Wisconsin, CS559 Fall 2004 Compositing Compositing combines components from two or more images to make a new image –Special effects are easier to control when done in isolation –Even many all live-action sequences are more safely shot in different layers

9/28/04© University of Wisconsin, CS559 Fall 2004 Historically … The basis for film special effects (even before computers) –Create digital imagery and composite it into live action –It was necessary for films (like Star Wars) where models were used –It was done with film and masks, and was time consuming and expensive Important part of animation – even hand animation –Background change more slowly than foregrounds, so composite foreground elements onto constant background –It was a major advance in animation – the multiplane camera first used in Snow White (1937)

9/28/04© University of Wisconsin, CS559 Fall 2004 Perfect Storm

9/28/04© University of Wisconsin, CS559 Fall 2004 Animated Example over =

9/28/04© University of Wisconsin, CS559 Fall 2004 Mattes A matte is an image that shows which parts of another image are foreground objects Term dates from film editing and cartoon production How would I use a matte to insert an object into a background? How are mattes usually generated for television?

9/28/04© University of Wisconsin, CS559 Fall 2004 Working with Mattes To insert an object into a background –Call the image of the object the source –Put the background into the destination –For all the source pixels, if the matte is white, copy the pixel, otherwise leave it unchanged To generate mattes: –Use smart selection tools in Photoshop or similar They outline the object and convert the outline to a matte –Blue Screen: Photograph/film the object in front of a blue background, then consider all the blue pixels in the image to be the background

9/28/04© University of Wisconsin, CS559 Fall 2004 Compositing Compositing is the term for combining images, one over the other –Used to put special effects into live action Or live action into special effects

9/28/04© University of Wisconsin, CS559 Fall 2004 Alpha Basic idea: Encode opacity information in the image Add an extra channel, the alpha channel, to each image –For each pixel, store R, G, B and Alpha –alpha = 1 implies full opacity at a pixel –alpha = 0 implies completely clear pixels There are many interpretations of alpha –Is there anything in the image at that point (web graphics) –Transparency (real-time OpenGL) Images are now in RGBA format, and typically 32 bits per pixel (8 bits for alpha) All images in the project are in this format

9/28/04© University of Wisconsin, CS559 Fall 2004 Pre-Multiplied Alpha Instead of storing (R,G,B,  ), store (  R,  G,  B,  ) The compositing operations in the next several slides are easier with pre-multiplied alpha To display and do color conversions, must extract RGB by dividing out  –  =0 is always black –Some loss of precision as  gets small, but generally not a big problem

9/28/04© University of Wisconsin, CS559 Fall 2004 Compositing Assumptions We will combine two images, f and g, to get a third composite image –Not necessary that one be foreground and background –Background can remain unspecified Both images are the same size and use the same color representation Multiple images can be combined in stages, operating on two at a time

9/28/04© University of Wisconsin, CS559 Fall 2004 Basic Compositing Operation At each pixel, combine the pixel data from f and the pixel data from g with the equation: F and G describe how much of each input image survives, and c f and c g are pre-multiplied pixels, and all four channels are calculated To define a compositing operation, define F and G

9/28/04© University of Wisconsin, CS559 Fall 2004 Basic Compositing Operation F and G are simple functions of the alpha values F and G are chosen (independently) Different choices give different operations To code it, you can write one compositor and give it 6 numbers (3 for F, 3 for G) to say which function –Constant of 0 or 1 –  f is multiplied by -1, 0 or 1. Similar for  g

9/28/04© University of Wisconsin, CS559 Fall 2004 Sample Images Images Alphas

9/28/04© University of Wisconsin, CS559 Fall 2004 “Over” Operator If there’s some f, get f, otherwise get g over=

9/28/04© University of Wisconsin, CS559 Fall 2004 “Over” Operator Computes composite with the rule that f covers g

9/28/04© University of Wisconsin, CS559 Fall 2004 “Inside” Operator Get f to the extent that g is there, otherwise nothing inside=

9/28/04© University of Wisconsin, CS559 Fall 2004 “Inside” Operator Computes composite with the rule that only parts of f that are inside g contribute