Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Fall 2003

Outline  Project Introduction  Basic parts of the purposed system  Working principles of individual parts  Future Work  Q&A

Introduction  Post-TV production software changes the content of the original video clips.  Extensively used in video-making industries.  Why changing the content of a video?  Reducing video production cost  Performing dangerous actions  Producing effects those are impossible in reality

Difficulties to be overcome  Things in video can be treated individually called “video objects”.  Computers cannot perform object detection directly because…  Image is processed byte-by-byte  Without prior knowledge about the video objects to be detected  Result is definite, no fuzzy logic.  Though computers cannot perform object detection directly, it can be programmed to work indirectly.

Basic parts of the purposed system  Simple bitmap reader/writer  RGB/HSV converter  Edge detector with smoother  Edge equation finder  Equation processor  Translation detector  Texture mapper

RGB/HSV converter  Human eyes are more sensitive to the brightness rather than the true color components of an object.  More reasonable to convert the representation of colors into HSV (Hue, Saturation and Value (brightness)) model.  After processing, convert back to RGB and save to disk.

RGB/HSV converter RGB to HSV HSV to RGB

Basic parts of the purposed system  Simple bitmap reader/writer  RGB/HSV converter  Edge detector with smoother  Edge equation finder  Equation processor  Translation detector  Texture mapper

Edge detector  Usually, a sharp change in hue, saturation or brightness means that there exist a boundary line. HSV: (0,0,0) HSV: (0,255,255)

Edge detector Before edge highlightingAfter edge highlighting

Smoother for Edge detector  Sometimes noise will affect the edge detection result of low resolution images.  Include an image smoother to remove large noise points in the image.  In some cases performing a smoothing will greatly enhance the performance of edge detection due to the decrease in “fake” edge points.

Smoother for Edge detector

Basic parts of the purposed system  Simple bitmap reader/writer  RGB/HSV converter  Edge detector with smoother  Edge equation finder  Equation processor  Translation detector  Texture mapper

Edge equation finder  Derives mathematical facts out of the edge points.  Works with voting algorithm of Hough Transform.  Automatically adjusts tolerance value to minimize the effect of noise points.  This helps when the edge is not completely straight or blurred.

Edge equation finder (x1,y1) Angle in degree Frequency Desired linear equation in point-slope form:

Edge equation finder

Basic parts of the purposed system  Simple bitmap reader/writer  RGB/HSV converter  Edge detector with smoother  Edge equation finder  Equation processor  Translation detector  Texture mapper

Equation processor  After finding out the equation, constraints can be applied in order to remove redundant equations, get shadows or detect occultation.  Find out the corner points for translation detector and texture mapper.

Equation processor Before edge findingAfter edge and equation findingAfter extra equation removal

Equation processor  The following criteria are currently adopted in the equation processor:  Distance between the equations  Angle between the equations  Whether the equation intersects the object or not.  Since equation processor is a potential burden to the system, it may be replaced in the future by improving other parts.

Basic parts of the purposed system  Simple bitmap reader/writer  RGB/HSV converter  Edge detector with smoother  Edge equation finder  Equation processor  Translation detector  Texture mapper

Translation detector  A simple object motion tracker.  Collects the data of the first key frame to accelerate the process of the remaining video frames.  Can be beneficial if the video segment is long and the scene seldom changes.

Translation detector  Records the approximate location of the object in the key frame.  When processing the following video frames, just scan within a certain boundary near the recorded location.  Will improve its efficiency later.

Basic parts of the purposed system  Simple bitmap reader/writer  RGB/HSV converter  Edge detector  Edge equation finder  Equation processor  Texture mapper

Texture Mapper  A graphics design technique used to wrap a surface of a 3-D object with a texture map  The 3-D object acquires a surface texture similar to the texture map.  Colors, brightness values or altitudes

Texture Mapper Mapping (color) Texture mapImage

Texture Mapper  Definition of terms: Image coordinates (r, c) : location of pixel in the image Image coordinates (r, c) : location of pixel in the image Texture coordinates (u, v) : location in texture map which contains color information for image coordinates Texture coordinates (u, v) : location in texture map which contains color information for image coordinates Mapping function : determines how texture coordinates are mapped to image coordinates or vice versa. e.g. linear scan-line interpolation Mapping function : determines how texture coordinates are mapped to image coordinates or vice versa. e.g. linear scan-line interpolation

Texture Mapper Mapping function (u,v) (r,c) Texture coordinates Image coordinates

Texture Mapper  Definition of terms Forward mapping maps from the texture space to image space Forward mapping maps from the texture space to image space Inverse mapping maps from the image space to texture space Inverse mapping maps from the image space to texture space Scan-line conversion: an area-filling technique processing a surface line by line. It can be applied with forward/inverse mapping. Scan-line conversion: an area-filling technique processing a surface line by line. It can be applied with forward/inverse mapping.

Texture Mapper Forward mapping Inverse mapping

Texture Mapper Texture scanning Image scanning More important Scan-line conversion

 triangle/parallelogram scanning  line by line, from top to bottom  process each pixel on every line Scan-line conversion Scanline y k Scanline y k+1 scanning order for every line

Scan-line conversion  For a row scan, maintain a list of scanline / polygon intersections.  Intersection at scanline r+1 efficiently computed from scanline r. Scanline y k Scanline y k+1 Coordinates differ by Δx and Δy

Scan-line conversion  quadrilateral  triangles or parallelograms  scan each sub-polygon  special case: only 2 sub-polygons 1 2 3

Scan-line conversion with forward mapping Algorithm Algorithm texture scanning forward mapping functions r = R(u,v) c = C(u,v) copy pixel at source (u,v) copy pixel at source (u,v) to destination (r,c) to destination (r,c) for u = umin to umax for v = vmin to vmix for v = vmin to vmix

Scan-line conversion with inverse mapping texture scanning inverse mapping functions u = U(r,c) v = V(r,c) copy pixel at source (u,v) copy pixel at source (u,v) to destination (r,c) to destination (r,c) for (r,c) = polygon pixel Algorithm Algorithm

Comparison Possibility of aliasing NoYes Calculation of fractional area of pixel coverage a bit complicated as it involves image scanning easy if the mapping function is known Ease of implementation image  texture texture  image Principle Inverse mapping Forward mapping

Comparison Aliasing – filtering/resampling techniques can be applied for inverse mapping

Comparison Yes, but can be avoided with simple filtering/resampling Yes Possibility of aliasing NoYes Calculation of fractional area of pixel coverage a bit complicated as it involves image scanning easy if the mapping function is known Ease of implementation image  texture texture  image Principle Inverse mapping Forward mapping We used inverse mapping

Mapping functions  Simple linear transformations Translation, scaling, etc Translation, scaling, etc  parallelograms only  parallelograms only very fast very fast  Linear scan-line interpolation Based on proportion Based on proportion  any quadrilaterals  any quadrilaterals not suitable feasible

Linear scan-line interpolation  Idea of interpolation (x 1, y 1 ) (x 2, y 2 ) a 1-a (x, y) = a(x 2, y 2 ) + (1-a)(x 1, y 1 )

Linear scan-line interpolation (r 4, c 4 ) (r 1, c 1 ) (r 3, c 3 )(r 2, c 2 ) (r 5, c 5 ) (r, c) image (u 1, v 1 )(u 3, v 3 ) texture map (u 2, v 2 ) (u 4, v 4 ) (u 5, v 5 ) For a particular scanline, (u, v)

After mapping a bit strange !

Shadow mapping  Mapping of surface brightness  Retain the brightness of the original surface  Method: Convert the original surface  HSV Convert the original surface  HSV Get the V value Get the V value Replace the V value of the mapped surface Replace the V value of the mapped surface

After Shadow mapping More natural !

Future Work  Anti-aliasing  Mapping different shapes like cans  Speed optimization  Movie manipulation  Use of 3D markers

Q & A