1. המעבדה למערכות ספרתיות מהירות High speed digital systems laboratory הטכניון - מכון טכנולוגי לישראל הפקולטה להנדסת חשמל Technion - Israel institute of technology department of Electrical Engineering 2007 winter Final presentation (part B) Performed by: Cohen Ido, Volokitina Irina Instructor: Rivkin Ina, Technion Almog Asaf, Intel Denoising video in real time

2. Agenda Motivation and background. Optional concepts and solutions (Why bi-lateral filter?). Validation’s expectations Results’ validation (and more). Implementation in hardware (gidel and synplify). Conclusions Demonstration

3. Image noise – definition and more The term noise usually refers to the high frequency random perturbations. corresponds to visible grain or particles present in the image. Generally caused by the electronic noise in the input device sensor and circuitry (e.g. scanner, digital camera).

4. Type of noise

5. The solution is DENOISING Removing noise from data is often the first step in data analysis. Denoising techniques should not only reduce the noise, but do so without blurring or changing the location of the edges.

6. Optional concepts and solutions Diffused image. Bilateral filter.

7. Bilateral Simple implementation. Noniterative. Local. YUV. (CIE) Diffusion Complicating implementation. Iterative. Not local. RGB. Bilateral and diffusion filtering comparison Conclusion: local and noniterative characterizations make us to choose in bilateral algorithm

8. Bilateral filter Bilateral filtering smoothes images while preserving edges, by means of a nonlinear combination of nearby image values. The bilateral filter can enforce the perceptual metric underlying the CIE-Lab color space, and smooth colors and preserve edges in a way that is tuned to human perception.

9. VALIDATION Implement the algorithm with high level design tool MATLAB, and compare the denoising image and the original one.

10. Validation the solution’s correctness The indication for “success” or “fail” is MSE. The destination is to reduce the MSE by 50%.

11. Validation Process Noise (normal distribution) denoising unit (bilateral) + Input File (without noise) Output File MSE measurement RGBRGB RGBRGB RGBRGB RGBRGB RGBRGB RGBRGB RGBRGB configuration noisemaker MSE redaction percentage

12. Implementation and validation process MSE tester. Normal distribution noisemaker. Denoising unit. Stages in the implementation rgb2yuv function and opposite. Adaptive LPF. Fir select. Adaptive HPF.

13. Implementations stages Denoising unit (bilateral) RGB YUVYUV RGB RGBRGBRGBRGB YUV

14. Implementations stages Denoising unit (bilateral) RGB YUVYUV RGB RGBRGBRGBRGB YUV Adaptive LPF YUV configuration

15. Implementations stages Denoising unit (bilateral) RGB YUVYUV RGB RGBRGBRGBRGB YUV Adaptive LPF YUV configuration Fir select MUXMUX

16. Implementations stages Denoising unit (bilateral) RGB YCbCrYCbCr RGB RGBRGB RGBRGB YCbCr Adaptive LPF YCbCr configuration Fir select MUXMUX Adaptive HPF

17. expectations rgb2yuv and opposite – changing up to 10% in MSE. LPF – reducing MSE by at least 35% Adding fir select – reducing MSE by 5-10% HPF – reducing 10% MSE. Destination: reduction the MSE by 50%

18. Results rgb2yuv and opposite – changing up to 10% in MSE => less than 1%. LPF – reducing MSE by at least 35% => 44% Adding fir select – reducing MSE by 5-10% => 20% × HPF – reducing 10% MSE => add about 10% to MSE. Bottom line: reduction the MSE by 50% (and more) achieved

19. Original

20. noisy

21. filtered

22. final

23. Goals of project Implement denoise bilateral algorithm Denoise bilateral algorithm Noise No Noise = Video in Video out configuration

24. System Block Diagram Bilateral filterYCbCr RGB Denoise algorithm RGB YCbCr configuration Video in Video out Synplify implemented GIDEL implemented

25. Bilateral filter block Diagram HPF (3X3) LPF (3X3) MEMORYMEMORY Image analysis & Fir select Controller YCbCr configuration X X + Synplify implemented GIDEL implemented

26. Memory block Diagram MEMORYMEMORY YCbCr L[n] L[n-2] L[n-1] L[n] SYNPLIFY implemented YCbCr

27. HPF/LPF block HPF/LPF SYNPLIFY implemented L[n-2] L[n-1] L[n] FIR L[n-2] L[n-1] L[n] + L[n-1]

28. Image analysis & Fir select block diagram L[n-2] L[n-1] L[n] MIN last 3 clock MAX last 3 clock - >=< SYNPLIFY implemented

29. Controller block diagram SYNPLIFY implemented HPF coefficients LPF coefficients Threshold & weight parameter User definition

30. System block diagram Denoising unit (bilateral) RGB YCbCrYCbCr RGB RGBRGB RGBRGB YCbCr Adaptive LPF YCbCr configuration Fir select MUXMUX HPF Noise No Noise GiDEL PROCWizard

31. Implementation : System level top view (a) – input domain; (b) – delay line block; (c) – RGB to YCbCr transform blocks – one for each line; (d) – delay sample blocks; (e) – min max block; (f), (g) – synchronizing pipes; (h1) – LPF block; (h2) –HPF block; (k) – filter select domain; (l) – synchronizing domain; (m) – YCbCr to RGB block; (n) – output domain (p) – simplify tool button

32. Flow stages Stage 1 - break input stream to lines Input stream divided to 3 lines data stream while line to line delay is 800 pixels (screen resolution) by 3 registers when each one delayed from others according to line its responsible for.

33. Conversion algorithm taken from : http://en.wikipedia.org/wiki/YCbCr RGB data is 24 bit pattern while each 8bit of it is a value for each one of 3 colors. e.i YCbCr from "digital 8-bit RGB“ Y = 16 + 1/256*(65.738*R + 129.057*G + 25.064*B) Cb = 128 + 1/256*(-37.945*R - 74.494*G + 112.439*B) Cr = 128 + 1/256*(112.439*R - 94.154*G - 18.285*B) The inverse transform equivalent to the above is: 8-bit RGB from YCbCr R = ( 298.082*Y + 408.583*Cr) / 256 - 222.921 G = ( 298.082*Y - 100.291*Cb - 208.120 * Cr) / 256 + 135.576 B = ( 298.082*Y + 516.412*Cb) / 256 - 276.836 Flow stages (cont.) Stage 2 - RGB to YCbCr transform one for each line

34. (a) – input domain; (b),(e) – synchronizing domain; (c) – expand domain to expand 24 bit incoming pattern to three 8 bit patterns (see General description); (d) - Conversion domain (see general); (f) – output domain;

35. Flow stages (cont.) Stage 3 - get 3x3 matrix surrounding for each pixel X 3 =

36. Flow stages (cont.) Stage 4 – min/max evaluation (a) – In_y1 to In_y9 - data input domain; (b) – Cb and Cr components of input; (c) – Control signals inputs ( de_enable, vsync. hsync); (d) – Min, max evaluation domain ; (e) – In_y1 to In_y2 data transfer domain; (f) – Control signals synchronizing domain ; (g) – min and max values output; (k) – In_y1 to In_y2 data output; (h) – Cb and Cr components of output; (j) – Control signals inputs; Blue outlined vertical areas are synchronizing pipelines.

37. Flow stages (cont.) Stage 4(cont.)– min/max evaluation Max-min = Delta = Filter select indicator

38. Flow stages (cont.) Stage 5 - LPF

39. Flow stages (cont.) Stage 6 - HPF HPF bypass mechanism

40. Flow stages (cont.) Stage 7 - Filter select mechanism (a) – threshold input; (b) – comparator element; (c) – MUX; (d) –input from HPF_3x3 block ; (e) - input from LPF_3x3 block; (f) – filtered Y component of pixel All other element are for synchronizing.

41. Flow stages (last ) Stage 7 - YCbCr 2 RGB (a) – input domain; (b) - Conversion domain (see General description); (c) – synchronizing domain; (d) – concat domain to concat the three 8 bit patterns bits to 24 bit outgoing pattern to (see General description); (e) – output domain;

42. Summary and conclusions Summary We have implemented denoising bilateral algorithm using GIDEL's hardware and High Level Design tools as SynplifyDSP and SynplifyPro. The algorithm performs real-time adaptive filtering per pixel of video stream coming from the DVI input. Project goals are fully achieved. Conclusions  Not easy to implement on such data rates but - Can be done  Better results using HPF bypass -> Implemented HPF algorithm is not a best solution  Possible improvement :  other HPF algorithm or another filter selection algorithm (few threshold etc.)

43. Questions Demonstration