1. Conversion of Standard Broadcast Video Signals for HDTV Compatibility
Ph.D. Defense Presentation
Elham Shahinfard
Advisor: Prof. M. A Sid-Ahmed
16 July 2009

2. Contributions
Proposing a motion adaptive deinterlacing method. It includes:
An accurate hierarchical motion detection algorithm
A recursive threshold optimization method
A motion adaptive interpolation algorithm for estimating the missing lines.

3. Outline Introduction Background Information & Review of Previous Works
Proposed Motion Detection (MD) Method
Proposed Motion Adaptive Deinterlacing Method
Evaluation of Algorithm Performance and Comparison with Other Methods
Summary and Future Works

4. Problem Definition
Digital high definition TV is replacing analog TV all over the world
Canada and U.S. have adopted the Advanced Television System Committee fully digital system as their new TV standard
Analog TV standards use Interlaced format
Deinterlacing for converting interlaced to progressive has attracted attention
Vertical lines
Horizontal pixels per line
Aspect Ratio
Progressive/Interlaced
720
1280
16:9
Progressive
1080
1920
Interlace

5. Video Scanning Format
Interlacing

6. Deinterlacing Estimation De-Interlacing
Deinterlacing Objective: To find the best estimation for the missing lines with minimum distortion

7. Outline Introduction Background Information & Review of Previous Works
Proposed Motion Detection (MD) Method
Proposed Motion Adaptive Deinterlacing Method
Evaluation of Algorithm Performance and Comparison with Other Methods
Summary and Future Works

8. Deinterlacing Categories
Spatial deinterlacing (Intra-frame)
Temporal deinterlacing (Inter-frame)
Hybrid deinterlacing (Inter-frame)
Vertical-Temporal (VT) median deinterlacing
Motion compensated deinterlacing
Motion adaptive deinterlacing

9. Spatial Deinterlacing
Uses only spatial data from current field of video
: location of a pixel in the field
: spatial displacement vector
Examples
Line Repetition
Line Averaging

10. Spatial Deinterlacing Example (Line Averaging)
Original Progressive Frame
Interlaced Field
Advantages
Simplicity
Motion robustness
Disadvantages
Low quality; blurring effect
Deinterlaced Frame

11. Temporal Deinterlacing
Uses only temporal data from previous and/or subsequent fields
Examples
Field Insertion
Bilinear Field Averaging

12. Temporal Deinterlacing Example (Field Insertion)
Original Progressive Frame
Interlaced Field
Advantages
Simplicity
Perfect for static regions
Disadvantages
Severe distortion in dynamic regions
Deinterlaced Frame

13. VT Median Deinterlacing
A median filter is used to interpolate the missing lines
The median operation is done in both temporal and vertical directions
Example:
Three tap VT median deinterlacing:

14. VT Median Deinterlacing Example
Original Progressive Frame
Interlaced Field
Advantages
Easy implementation
Superior to temporal and spatial methods
Disadvantages
Low quality
Deinterlaced Frame

15. Motion Compensated Deinterlacing
Video sequence is virtually converted to a stationary sequence
A motion estimation method estimates the motion
The estimated motion is removed consequently
A deinterlacing method, which performs well in static regions, is applied to the stationary sequence
Motion data is added to the deinterlaced sequence in a later stage

16. Motion Compensated Deinterlacing
Advantages
High quality results
Perfect for videos with translational motion, such as panning camera
Disadvantages
Performance is highly dependant to motion estimation results
Vulnerable to common motion estimation obstacle, such as:
Object deformation
Appearance and disappearance of objects
Fast motions which goes beyond the search area
Sub-pixel motions

17. Motion Adaptive Deinterlacing
Benefits from both interframe and intraframe deinterlacing methods by:
Using a motion detector to divide a video sequence into static and dynamic regions
Using an interframe deinterlacing in static regions
Using an intraframe deinterlacing in dynamic regions
Combining the results to obtain best estimation
It has proven to be a proper choice for high quality deinterlacing
A high quality motion adaptive deinterlacing has been designed in this research

18. Outline Introduction Background Information & Review of Previous Works
Proposed Motion Detection (MD) Method
Proposed Motion Adaptive Deinterlacing Method
Evaluation of Algorithm Performance and Comparison with Other Methods
Summary and Future Works

19. Design Objectives & Contributions
Accurately detecting the presence of motion in video sequence
Measuring motion activity level of video sequence with high precision
Contributions
Proposing an accurate motion detection algorithm which:
Uses five consecutive fields of interlaced video
Has a hierarchical structure
Utilizes two LPF to improve algorithm accuracy

20. Input Sequence
Five consecutive fields of a video sequence are used for motion detection
The optimum number of correlated interlaced video fields is 5.
Relative position of missing lines in five consecutive fields

21. Motion Detector
Motion detection goal is detecting the possibility of motion
Motion direction is not important
Increase the accuracy of motion detector.
Assumptions:
Signal is large and noise is small
Low frequency energy in signal is greater than low frequency energy in noise and alias
2D square averaging filters are appropriate choices
Improves the consistency of the output
Assumption: moving objects are larger than pixels
an mxm median filter is an appropriate choice

22. Hierarchical Block Receives dif1, dif2, dif3
Partitions them into data blocks
Calculates the average intensity value for each block

23. Hierarchical Block
Compares the average intensity value of each block with its corresponding data blocks
Finds the maximum average intensity value of each three corresponding data blocks
Compares the maximum value with a predefined threshold value
If less than threshold value, the data block is considered static and its motion value is set to zero
If greater than threshold value, the data block is dynamic
A dynamic data block will be recursively partitioned to smaller data blocks
The recursive procedure may continue up to pixel level
The final output is motion value matrix

24. Hierarchical Block Average Intensity Value before Thresholding
Average Intensity Value after Thresholding

25. Threshold Value Determination
Threshold values have been found by experimental tests
A video sequence with tractable moving objects is a proper starting point
Average intensity values have been monitored for several test sequences to find initial values
Initial values have been applied to motion detection methods and recursively optimized for error minimization

26. Outline Introduction Background Information & Review of Previous Works
Proposed Motion Detection Method
Proposed Deinterlacing Method
Evaluation of Algorithm Performance and Comparison with Other Methods
Summary and Future Works

27. Proposed Motion Adaptive Deinterlacing
Improves motion detection consistency by reducing distortion
Reduces the possibility of missing motions.

28. Non-Linear Transformation
Converts Motion-Value (MV) to motion possibility value
and are predefined threshold values found by recursive error minimization.

29. Threshold Calculation
Pre Conditions:
are in the same range as pixel intensity values (0-255 in a general case)
Initial setup
Initial threshold values:
Initial step size: 10
Procedure:
Initial values have been applied to the proposed motion adaptive deinterlacing method
Deinterlacing error has been calculated and recorded for each initial value
Initial values have been changed based on calculated error.
Finer step size has been applied to the area with minimum error

30. Threshold Calculation
Same procedure has been applied to several test sequences
has proven to be optimum values for a general setup

31. Interpolation Algorithm
: motion possibility value
: intensity value of a pixel using Spatial deinterlacing method; Linear interpolation is chosen as spatial deinterlacing method
: intensity value of a pixel using Temporal deinterlacing method; Median filtering is chosen as temporal deinterlacing method

32. Deinterlacing Results
Stennis Original Progressive Video
Stennis Deinterlaced Video
Sflowg Original Progressive Video
Sflowg Deinterlaced Video

33. Implementation Results
(a) Grandmom; Original progressive frame
(b) Grandmom; Deinterlaced frame
(c) Mom; Original progressive frame
(d) Mom; Deinterlaced frame

34. (e) MomDaughter; Original progressive frame
(f) MomDaughter; Deinterlaced frame
(g) Stennis; Original progressive frame
(h) Stennis; Deinterlaced frame

35. (i) Heart; Original progressive frame
(j) Heart; Deinterlaced frame
(k) Sflowg; Original progressive frame
(l) Sflowg; Deinterlaced frame

36. (m) Movi; Original progressive frame
(n) Movi; Deinterlaced frame
(o) Disku; Original progressive frame
(p) Disku; Deinterlaced frame

37. Motion Detection Results
(a) Grandmom
(b) Mom
(c) MomDaughter
(d) Stennis

38. (e) Heart
(f) Sflowg
(g) Movi
(h) Disku

39. Outline Introduction Background Information & Review of Previous Works
Proposed Motion Detection (MD) Method
Proposed Motion Adaptive Deinterlacing Method
Evaluation of Algorithm Performance and Comparison with Other Methods
Summary and Future Works

40. Performance Evaluation Method

41. Evaluation Criterion
Objective Evaluation Criterion: Peak Signal to Noise Ratio
Subjective Evaluation Criterion (According to ITU-R BT ):
Mean Score & its
associated confidence
Interval
Grade
Impairment level 5 4 3 2 1
Imperceptible
Perceptible but not annoying
Slightly annoying
Annoying
Very Annoying

42. Objective Evaluation Results

46. Single Frame of Mom Seq. (a) Original progressive frame
(b) Deinterlaced by bilinear field averaging
(c) Deinterlaced by GA-HDTV method
(b) Deinterlaced by proposed MA method

47. Zoomed on a Moving Region
(a) Original progressive frame
(b) Deinterlaced by bilinear field averaging
(c) Deinterlaced by GA-HDTV method
(b) Deinterlaced by proposed MA method

48. Algorithm Robustness to Frame Rate

49. Evaluation of Proposed Motion Detection

51. Subjective Evaluation Results
25 observers have evaluated the algorithm
Non-professional random observers
Both male and female
Ages 15 to 65
Overall mean score is 4.74
Its 95% confidence interval

53. Outline Introduction: Deinterlacing Problem Statement
Review of Existing Methods
Proposed Motion Detection (MD) Method
Proposed Motion Adaptive Deinterlacing Method
Evaluation of Algorithm Performance and Comparison with Other Methods
Summary and Future Works

54. Summary A high accuracy motion detection algorithm was proposed
Has a hierarchical structure
Uses 5 consecutive video fields for motion detection
Is capable of detecting a wide range of motions from slow to fast motions
Provides superior PSNR compared to other mthods and improves deinterlacing overall performance by 18% on average
A motion adaptive deinterlacing method was proposed
Uses motion possibility values to combine line averaging with Vertical-Temporal median filtering and benefits from both
Has high performance and obtain high quality deinterlaced video

55. Recommendations for Future Works
Improvement in deinterlacing
Study of human eye frequency response to
Changes in video contents
Changes in motion speed while tracking an object
Improvement in motion adaptive deinterlacing
Including video content information
Automatic recognition of the type of the video for performance improvement
Combining video texture information with motion detection results
Combining motion detection results with motion estimation
Utilizing a motion compensated method instead of temporal method
Hardware Implementation (Architecture, Software/Hardware partitioning)