1. Quad-Tree Motion Modeling with Leaf Merging
Reji Mathew and David S. Taubman
CSVT 2010

2. Outline Introduction Quad-tree motion modeling
Quad-tree representation
Quad-tree motion modeling
Motion vector prediction strategies
Pruning algorithm
Merging principle
Motion signaling
R-D performance results
Hierarchical and polynomial motion modeling
Scalable motion modeling
Conclusion

3. Quad-tree Representation
Image modeling
Image to be recursively divided into smaller regions, each region represented by a suitable model.
Sub-optimal: dependency between neighboring leaf nodes with different parents is not exploited

4. Quad-tree Representation
Image modeling
Rate-distortion optimization, allowing a Lagrangian cost function(D+λR) to be minimized using tree pruning with leaf merging step.
[1] R. Shukla, P. Dragotti, M. Do, and M. Vetterli, “Rate-distortion optimized tree structure compression algorithms for piecewise polynomial images,” IEEE Trans. Image Process., vol. 14, no. 3, pp. 343–359, Mar

5. Quad-tree Motion Modeling
forward-only, backward only or bi-directional motion with two reference frames.
Motion vector prediction strategies
Hierarchical motion coding
H.264 spatial motion vector prediction strategy
Motion models

6. Quad-tree Motion Modeling
Pruning Algorithm
Produce a quad-tree structure that minimizes the Lagrangian cost objective Df + λRf
Given a parent node p, the four children ci , 1 ≤ i ≤ 4, are pruned away if
When pruning occurs, and Otherwise, and
=Rp in hierarchical coding
=0 at all times in spatial coding
R-D optimally pruned quad-tree:
Tree pruning yields a globally minimal value for Df + λRf for hierarchical coding; while it is somewhat greedy for spatially predictive coding.

7. Quad-tree Motion Modeling
Merging principle
possibility of jointly coding and optimizing neighboring nodes that belong to different parents.
Merge target contains nieghboring node located at a higher level or at the same level.
Merging is allowed to take place only if it reduces the overall Lagrangian cost.
The same parent

8. Quad-tree Motion Modeling
Motion signaling
Anchor node:
Hierarchical: the only member node of the region that is not signaled as being merged
Spatial: the first node in the region that is encountered during decoding.(the top-left block)

9. Quad-tree Motion Modeling
R-D performance results
25%
35%
35%
45%
once merging is included the performance of hierarchical motion representation can be brought close to that achieved by spatial prediction with merging.

10. Hierarchical and Polynomial Motion Modeling
Further improve the performance of hierarchical motion representation by polynomial motion models.
Formation of larger regions during merging process
Smoother motion representations
Motion models
The parameters of the motion model are obtained by a weighted least squares fitting procedure.
Pruning phase
Merging phase
: mv belonging to node b at level k
: motion corresponding to translation, linear and affine flows

11. Hierarchical and Polynomial Motion Modeling
Motion compensation
Generate a set of MVs for each descendants at level K (4*4 block)
R-D performance with motion models
depend on the motion model and the central location of block b’

12. Scalability Motion Modeling
Scalability objective
Modified Lagrangian cost function
When terminating decoding at an intermediate resolution level, motion compensation is performed using leaf nodes that may already be available; in those cases where leaf nodes are not available, information contained in branch nodes is utilized.
: The total distortion of all nodes for which motion compensation is performed
Level k :
: The costs for each level k of the quad-tree
: The weights assigned to each level,
and
Leaf node b
Branch node b
Contribution to
terminate

13. Scalability Motion Modeling
Scalability performance
α0 = α1= α2=0.1, α3=0.7

14. Scalability Motion Modeling
Residual coding
JPEG2000: full resolution motion compensated residual frames
Total rate for coding motion and residual frames

15. Scalability Motion Modeling
Wavelet-based video encoding results
integrate the quad-tree motion model with the wavelet-based scalable interactive video (SIV) codec[9]
[9] A. Secker and D. S. Taubman, “Lifting-based invertible motion adaptive transform framework for highly scalable video compression,” IEEE Trans. Image Process., vol. 12, no. 12, pp. 1530–1542, Dec

16. Conclusion
The merging step can be incorporated into quad-tree motion representations for a range of motion modeling contexts.
R-D performance that can be gained by introducing merging for the two cases of hierarchical and spatially predictive motion coding (such as that employed by H.264).
Report on the benefits of polynomial modeling and hierarchical coding, once merging has been incorporated into the conventional quad-tree approach.