1. Xinqiao LiuRate constrained conditional replenishment1 Rate-Constrained Conditional Replenishment with Adaptive Change Detection Xinqiao Liu December 8, 2000 EE368B Project

2. Xinqiao LiuRate constrained conditional replenishment2 Motivation Conditional replenishment ---- method of reducing temporal redundancy between successive frames –Efficient in video conferencing with stationary cameras and slow motion. –Study shows that less than 3% of the pixels need to be replenished in most head-and-shoulders scenes in desktop video Computational complexity is significant simpler than other video compression methods –Software-only CODEC is possible –Appealing for on-sensor compression where pixel array and simple image processing are integrated on the same chip, i.e, camera system-on-chip

3. Xinqiao LiuRate constrained conditional replenishment3 Previous Work Most of the research concentrate mainly on the image quality (Haskell, et al’72, Haskell’79) Recently, a perception-based change detection method was proposed (Chiu&Berger ’96, Chiu&Berger’99) –Reduces the perceptual redundancy in addition to the spatial and temporal redundancy –Change detection threshold is set based on Web’s law However, the correlation between transmission bit-rate and the choice of change detection schemes still need to be explored.

4. Xinqiao LiuRate constrained conditional replenishment4 Outline Introduction & Problem formulation Context-based Arithmetic Encoder Change detection --- direct methods –Subsampling –Threshold adjusting Adaptive change detection –Noise characteristic –Adaptive algorithm Conclusion

5. Xinqiao LiuRate constrained conditional replenishment5 Conditional Replenishment Diagram Goal: Given a rate-constrained transmission channel, find the optimal change detection algorithm that minimizes the distortion

6. Xinqiao LiuRate constrained conditional replenishment6 Model and Assumptions Assumptions: 1.Transmitted separately under certain bit-rate constrain R1, R2 2.Lossless coding for both mask and signal 3.Only intra-frame compression is considered

7. Xinqiao LiuRate constrained conditional replenishment7 Rate-Constrained Change Detection Three ways to control the bit rate in the change detector: 1.Subsampling the mask and signal after detection 2.Adjusting the detection threshold 3.Using adaptive threshold for each pixel based on the noise characteristics -----eliminate those pixels that have changed due to noise rather than the input Use unconstrained Lagrangian cost function to find the optimum detection parameters for each method

8. Xinqiao LiuRate constrained conditional replenishment8 Problem Formulation (I) The mean-square distortion is defined as: The above assumption allows us to study the rate-distortion function of conditional replenishment by only implementing the compression scheme of the mask. Assume R 1 = kR 2 since they are proportional to the number of changed pixels. The total bit-rate R is Given previous frame A 1, current frame A 2, binary change mask C, the reconstructed frame at decoder end is:

9. Xinqiao LiuRate constrained conditional replenishment9 Problem Formulation (II) The constrained problem of: Can be converted to the unstrained problem by introducing the Lagrangian cost function given Lagrange multiplier : where s is the adjustable change detection parameter. The optimal value of s is given by : The desired optimal slop value  is not known a priori but can be obtained using a fast bisection search algorithm

10. Xinqiao LiuRate constrained conditional replenishment10 Outline Introduction & problem formulation Context-based Arithmetic Encoder Change detection --- direct methods Adaptive change detection Conclusion

11. Xinqiao LiuRate constrained conditional replenishment11 Test Video Sequence Captured by a stationary high-speed digital camera with a person moving cross the screen:

12. Xinqiao LiuRate constrained conditional replenishment12 Context-based Arithmetic Encoder (CAE) Binary bitmap-based shape coding scheme used in the MPEG-4 standard Three types of 16x16 macroblocks: –"black" block: none of the pixel changed (all 0) –"white" block: all pixels changed and to be replenished (all 1) –“boundary” block: encoded with a template of 10 pixels to define the causal context for predicting the binary value of the current pixel (S0). For black and white blocks, only the block type need to be transmitted For boundary blocks, use conditional entropy:

13. Xinqiao LiuRate constrained conditional replenishment13 Outline Introduction & problem formulation Context-based Arithmetic Encoder Change detection --- direct methods –Subsampling –Threshold adjusting Adaptive change detection Conclusion

14. Xinqiao LiuRate constrained conditional replenishment14 Change Masks With Subsampling Subsample the macroblock by a factor of 2, 4 or 8 Subblocks are encoded using the CAE Upsample at the decoder end using pixel replication filter combined with a 3x3 median filter

15. Xinqiao LiuRate constrained conditional replenishment15 Rate-distortion of Subsampling

16. Xinqiao LiuRate constrained conditional replenishment16 Change Masks With Threshold-adjusting Control the bit-rate by globally adjusting the change detector threshold. As the threshold increased, few pixels will be detected

17. Xinqiao LiuRate constrained conditional replenishment17 Rate-distortion of Threshold-adjusting

18. Xinqiao LiuRate constrained conditional replenishment18 Outline Introduction & problem formulation Context-based Arithmetic Encoder Change detection --- direct methods Adaptive change detection –Noise characteristics –Adaptive algorithm Conclusion

19. Xinqiao LiuRate constrained conditional replenishment19 Noise Characteristics A fundamental problem in designing an optimum change detector is how to separate pixels whose change is due to noise from pixels whose change is due to real input signal change For cameras using either CCD or CMOS image sensors, the final image is formed by the photo-charge Q i,j (or voltage) integrated on each photo-detector during the exposure time. Two independent additive noise corrupt the output signal: –Shot noise U i,j which is zero mean signal dependent gaussian distribution with: –Readout circuit and reset noise V i,j (including quantization noise) with zero mean and variance  V .

20. Xinqiao LiuRate constrained conditional replenishment20 Adaptive Change Detection Thus the total noise variance of pixel (i,j) is: –The noise is signal dependent –The stronger the luminance level, the noisier the pixel will be The threshold T i,j is thus set as: –where m is the sensitivity factor that is set globally – is local average value over a small area with size 8x8. Note that by changing m, we effectively adjusting the detection sensitivity while the threshold is still locally adapted

21. Xinqiao LiuRate constrained conditional replenishment21 Adaptive Threshold

22. Xinqiao LiuRate constrained conditional replenishment22 Change Masks With Adaptive Threshold

23. Xinqiao LiuRate constrained conditional replenishment23 Rate-distortion of Adaptive Threshold

24. Xinqiao LiuRate constrained conditional replenishment24 Performance Comparison Subsampling is the most efficient in reducing bit-rate Adaptive thresholding achieves the best PSNR

25. Xinqiao LiuRate constrained conditional replenishment25 Conclusion Studied three change detection algorithms: 1.Subsampling 2.Threshold-adjusting 3.Adaptive threshold based on the noise characteristics The adaptive change detection algorithm efficiently separates pixels whose change is due to noise from pixels whose value change is due to real input signal change Simulation proves that the adaptive change detection algorithm achieves the best PSNR among all the three algorithms