1. Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Fall 2003

2. Outline  Project Introduction  Basic parts of the purposed system  Working principles of individual parts  Future Work  Q&A

3. Introduction  Post-TV production software changes the content of the original video clips.  Extensively used in video-making industries.  Why changing the content of a video?  Reducing video production cost  Performing dangerous actions  Producing effects those are impossible in reality

4. Difficulties to be overcome  Things in video can be treated individually called “video objects”.  Computers cannot perform object detection directly because…  Image is processed byte-by-byte  Without prior knowledge about the video objects to be detected  Result is definite, no fuzzy logic.  Though computers cannot perform object detection directly, it can be programmed to work indirectly.

5. Basic parts of the purposed system  Simple bitmap reader/writer  RGB/HSV converter  Edge detector with smoother  Edge equation finder  Equation processor  Translation detector  Texture mapper

6. RGB/HSV converter  Human eyes are more sensitive to the brightness rather than the true color components of an object.  More reasonable to convert the representation of colors into HSV (Hue, Saturation and Value (brightness)) model.  After processing, convert back to RGB and save to disk.

7. RGB/HSV converter RGB to HSV HSV to RGB

8. Basic parts of the purposed system  Simple bitmap reader/writer  RGB/HSV converter  Edge detector with smoother  Edge equation finder  Equation processor  Translation detector  Texture mapper

9. Edge detector  Usually, a sharp change in hue, saturation or brightness means that there exist a boundary line. HSV: (0,0,0) HSV: (0,255,255)

10. Edge detector Before edge highlightingAfter edge highlighting

11. Smoother for Edge detector  Sometimes noise will affect the edge detection result of low resolution images.  Include an image smoother to remove large noise points in the image.  In some cases performing a smoothing will greatly enhance the performance of edge detection due to the decrease in “fake” edge points.

12. Smoother for Edge detector

13. Basic parts of the purposed system  Simple bitmap reader/writer  RGB/HSV converter  Edge detector with smoother  Edge equation finder  Equation processor  Translation detector  Texture mapper

14. Edge equation finder  Derives mathematical facts out of the edge points.  Works with voting algorithm of Hough Transform.  Automatically adjusts tolerance value to minimize the effect of noise points.  This helps when the edge is not completely straight or blurred.

15. Edge equation finder (x1,y1) Angle in degree Frequency 0 1 45 3 90 1 135 1 Desired linear equation in point-slope form:

16. Edge equation finder

17. Basic parts of the purposed system  Simple bitmap reader/writer  RGB/HSV converter  Edge detector with smoother  Edge equation finder  Equation processor  Translation detector  Texture mapper

18. Equation processor  After finding out the equation, constraints can be applied in order to remove redundant equations, get shadows or detect occultation.  Find out the corner points for translation detector and texture mapper.

19. Equation processor Before edge findingAfter edge and equation findingAfter extra equation removal

20. Equation processor  The following criteria are currently adopted in the equation processor:  Distance between the equations  Angle between the equations  Whether the equation intersects the object or not.  Since equation processor is a potential burden to the system, it may be replaced in the future by improving other parts.

21. Basic parts of the purposed system  Simple bitmap reader/writer  RGB/HSV converter  Edge detector with smoother  Edge equation finder  Equation processor  Translation detector  Texture mapper

22. Translation detector  A simple object motion tracker.  Collects the data of the first key frame to accelerate the process of the remaining video frames.  Can be beneficial if the video segment is long and the scene seldom changes.

23. Translation detector  Records the approximate location of the object in the key frame.  When processing the following video frames, just scan within a certain boundary near the recorded location.  Will improve its efficiency later.

24. Basic parts of the purposed system  Simple bitmap reader/writer  RGB/HSV converter  Edge detector  Edge equation finder  Equation processor  Texture mapper

25. Texture Mapper  A graphics design technique used to wrap a surface of a 3-D object with a texture map  The 3-D object acquires a surface texture similar to the texture map.  Colors, brightness values or altitudes

26. Texture Mapper Mapping (color) Texture mapImage

27. Texture Mapper  Definition of terms: Image coordinates (r, c) : location of pixel in the image Image coordinates (r, c) : location of pixel in the image Texture coordinates (u, v) : location in texture map which contains color information for image coordinates Texture coordinates (u, v) : location in texture map which contains color information for image coordinates Mapping function : determines how texture coordinates are mapped to image coordinates or vice versa. e.g. linear scan-line interpolation Mapping function : determines how texture coordinates are mapped to image coordinates or vice versa. e.g. linear scan-line interpolation

28. Texture Mapper Mapping function (u,v) (r,c) Texture coordinates Image coordinates

29. Texture Mapper  Definition of terms Forward mapping maps from the texture space to image space Forward mapping maps from the texture space to image space Inverse mapping maps from the image space to texture space Inverse mapping maps from the image space to texture space Scan-line conversion: an area-filling technique processing a surface line by line. It can be applied with forward/inverse mapping. Scan-line conversion: an area-filling technique processing a surface line by line. It can be applied with forward/inverse mapping.

30. Texture Mapper Forward mapping Inverse mapping

31. Texture Mapper Texture scanning Image scanning More important Scan-line conversion

32.  triangle/parallelogram scanning  line by line, from top to bottom  process each pixel on every line Scan-line conversion Scanline y k Scanline y k+1 scanning order for every line

33. Scan-line conversion  For a row scan, maintain a list of scanline / polygon intersections.  Intersection at scanline r+1 efficiently computed from scanline r. Scanline y k Scanline y k+1 Coordinates differ by Δx and Δy

34. Scan-line conversion  quadrilateral  triangles or parallelograms  scan each sub-polygon  special case: only 2 sub-polygons 1 2 3

35. Scan-line conversion with forward mapping Algorithm Algorithm texture scanning forward mapping functions r = R(u,v) c = C(u,v) copy pixel at source (u,v) copy pixel at source (u,v) to destination (r,c) to destination (r,c) for u = umin to umax for v = vmin to vmix for v = vmin to vmix

36. Scan-line conversion with inverse mapping texture scanning inverse mapping functions u = U(r,c) v = V(r,c) copy pixel at source (u,v) copy pixel at source (u,v) to destination (r,c) to destination (r,c) for (r,c) = polygon pixel Algorithm Algorithm

37. Comparison Possibility of aliasing NoYes Calculation of fractional area of pixel coverage a bit complicated as it involves image scanning easy if the mapping function is known Ease of implementation image  texture texture  image Principle Inverse mapping Forward mapping

38. Comparison Aliasing – filtering/resampling techniques can be applied for inverse mapping

39. Comparison Yes, but can be avoided with simple filtering/resampling Yes Possibility of aliasing NoYes Calculation of fractional area of pixel coverage a bit complicated as it involves image scanning easy if the mapping function is known Ease of implementation image  texture texture  image Principle Inverse mapping Forward mapping We used inverse mapping

40. Mapping functions  Simple linear transformations Translation, scaling, etc Translation, scaling, etc  parallelograms only  parallelograms only very fast very fast  Linear scan-line interpolation Based on proportion Based on proportion  any quadrilaterals  any quadrilaterals not suitable feasible

41. Linear scan-line interpolation  Idea of interpolation (x 1, y 1 ) (x 2, y 2 ) a 1-a (x, y) = a(x 2, y 2 ) + (1-a)(x 1, y 1 )

42. Linear scan-line interpolation (r 4, c 4 ) (r 1, c 1 ) (r 3, c 3 )(r 2, c 2 ) (r 5, c 5 ) (r, c) image (u 1, v 1 )(u 3, v 3 ) texture map (u 2, v 2 ) (u 4, v 4 ) (u 5, v 5 ) For a particular scanline, (u, v)

43. After mapping a bit strange !

44. Shadow mapping  Mapping of surface brightness  Retain the brightness of the original surface  Method: Convert the original surface  HSV Convert the original surface  HSV Get the V value Get the V value Replace the V value of the mapped surface Replace the V value of the mapped surface

45. After Shadow mapping More natural !

46. Future Work  Anti-aliasing  Mapping different shapes like cans  Speed optimization  Movie manipulation  Use of 3D markers

47. Q & A