Non-photorealistic Video Effects in the Compressed Domain Dept. of Computer Science National Chengchi University Student : Fu-Liang Hsu Advisor : Wen-Hung Liao 2005/7/18

2 Outline Motivation and introduction Issues in non-photorealistic rendering (NPR) Objectives of this research NPR in the spatial domain NPR in the compressed domain Conclusion and future work

3 Motivation Sony EyeToy Apple toySight ImageTech ( 台灣夢工場科技 )

4 NPR: Introduction Non-Photorealistic Rendering  Photorealistic Rendering

5 NPR Example: Images Oil paint effect using Ulead PhotoImpact® Image size=450x315. Took 4 seconds on machines with P4 2.4G CPU. Source ImageOil paint Image

6 NPR Example: 3D Models Applying NPR to 3D model Can be done in real-time.

7 NPR Effects of Interest Static NPR algorithm NPR animation Real-time NPR SIGGRAPH 1997 SIGGRAPH 2004

8 Drawbacks of Frame-by-Frame NPR Generation Processing time is demanding Coherence problem Flickering Oil paint Image

9 Reducing Processing Time Post-processing Modifying existing NPR algorithms Developing hierarchical NPR algorithms Applying NPR to regions of interest (ROI) in the video

10 Dealing with the Coherence Problem Coherence  Stroke-based NPR  Optical flow Flickering  Paint-over

11 Objectives of this Research Develop near real-time NPR algorithms (frame rate >= 10 fps) to facilitate interactive applications. Attempt to employ existing NPR algorithms and generate similar effects. Try to devise methods that are applicable to most NPR algorithms.

12 Possible Enhancements In the spatial domain In the compressed domain

13 NPR in the Spatial Domain Most NPR algorithms’ complexities are dependent on image size. Apply NPR to regions-of-interest (ROIs) can effectively reduce the processing time.

14 Combine with Source Video SkinFace NPR in the Spatial Domain: Framework Full Frame Edge Detected Area Motion Background Random Image Filter Source Video NPR Video

15 NPR Algorithm Oil paint  Time Complexity K= 7 For color = 1:3 For y= 1: height For x=1:width find most frequent value M in (x,y)'s n*n neighborhood oil_image(x,y)= M;

16 Development Environment CPU:Pentium-4 2.4G Hz with 1GB of RAM Visual C Intel OpenCV Library Asiamajor V-Gear MaxCam1300  USB 2.0  Frame rate:15 frame/sec

17  Frame rate : 0.75 ~ 1.0 fps Full Frame NPR Demo

18 Edge Image NPR Demo  Frame rate : 1.5 ~2.0 fps Edge Image Canny edge detection + Dilation Frame rate : 15 fps

19 Moving Region  Foreground vs. background region  Motion filter: Frame rate :15 fps D t (x)=Difference(t,t+1); If(| D t (x) | > Threshold) M t (x)= 1 else M t (x)=0 B(t+1)= B t + [ a 1 *( 1-M t ) + a 2 *M t ]*D t // a x : 降低改變大的區域對背景的影響, a1+a2=1, a 1 >a 2

20 Moving Region NPR Demo  Frame rate : 1.5~ 8.0 fps

21 Background Region NPR Demo  Frame rate : 1.0~ 2.0 fps

22 Random Region NPR Demo  Frame rate : 1.8~4.0 fps Random region selection: 15 fps

23 Possible Enhancements In the spatial domain: image filter  Edge  Moving region  Background  Random Detected area  Face  Skin

24 NPR in the Spatial Domain Face detection based on Viola and Jones’ algorithm proposed in ”Rapid object detection using a boosted cascade of simple features”

25 Face NPR Demo  Frame rate: 3.8~7.5 fps Region of interest: face Frame rate :15 fps

26 Color-Based Skin Detection  Hue value 0.3~1.5  Frame rate : 15 fps

27 Skin NPR demo  Frame rate : 3.2~4.5 fps

28 Summary of Spatial Domain Processing Frame rate NPR 效果影響速度關鍵 Full frame1~1.2 frame/sec 最佳，速度最慢整張影像，負擔過重 邊緣 2.5~3 frame/sec 邊緣區域較少，視覺效果較 差，可搭配其他層次使用 偵測邊緣演算法的門檻 值，影響邊緣數量的多 寡 移動區塊 2.5~4.0 frame/sec 使用移動區塊可以和使 用者互動，有額外的效果 移動區塊大小 背景區塊 1.8~4.0 frame/sec 使用背景區塊可以和使用者 互動 背景區塊大小 隨機選取 3.9~6.0 frame/sec 搭配其他的方式會有比較好 的效果 隨機選取區塊大小 臉部區塊 3.8~7.5 frame/sec 針對人體的部分套用 NPR 演 算法，互動效果佳 臉部偵測函式及臉部區 塊大小 膚色區塊 3.2~4.5 frame/sec 針對人體的部分套用 NPR 演 算法，互動效果佳 膚色區塊大小

29 Possible Enhancements In the spatial domain  Edge  Moving region  Background  Random  Face  Skin In the compressed domain

30 MPEG-I Compression Format Forward prediction of P-frame Forward prediction of B-frame Backward prediction of B-frame MPEG Display Order

31 Applying NPR in the Compressed Domain

32 NPR in the Compressed Domain Encode Decode all frames to spatial domain Decode I-frame to Spatial domain Change AC in the Compressed domain Compressed Video Image Compressed NPR Video Decode I-frame and large difference P, B-frame to Spatial domain

33 Development Environment CPU Pentium-4 2.4G Hz  Memory 1GB  Visual C  Intel OpenCV library  Dali library for video compression  MPEG-1 standard video 320x frames, 13sec GOP:IBBPBBPBBPBBPBB Source Video Encoding source image captured from Webcam to MPEG-I using hardware. Source Video

34 Macro Block 320 x 240 Image ……... AC Macro block 8x8 DC

35 NPR in the Compressed Domain: Changing the DC Coefficient Making changes to DC value in the compressed domain is equivalent to adding/subtracting a constant to every pixel in the spatial domain.

36 Changing the AC Coefficients: Frequency Domain Filtering Model  Butterworth Lowpass Filter 

37 Butterworth Lowpass Filter Demo I-frame only  D 0 =1,n=2  sec / 419 frames SourceI-frame BLPF

38 Butterworth Lowpass Filter Demo All I,P,B frames  D 0 =1,n=2  sec / 419 frames SourceIPB-frame BLPF

39 NPR in the Compressed Domain Gaussian Lowpass Filter 

40 Gaussian Lowpass Filter Demo I-frame only  D 0 =2  sec / 419 frames SourceI-frame GLPF

41 Gaussian Lowpass Filter Demo All I,P,B frames  D 0 =2  sec / 419 frames SourceIPB-frame GLPF

42 NPR in the Compressed Domain: Highpass Filtering Butterworth Highpass Filter 

43 Butterworth Highpass Filter Demo I-frame only  D 0 =4,n=4  sec / 419 frames SourceI-frame BHPF

44 Butterworth Highpass Filter Demo All I,P,B frames  D 0 =1,n=2  sec / 419frames SourceIPB-frame BHPF

45 Gaussian Highpass Filter I-frame only  D 0 =4  sec / 419 frames SourceI-frame GHPF

46 Gaussian Highpass Filter Demo All I,P,B frames  D 0 =4  sec / 419 frames SourceIPB-frame GHPF

47 Summary of DCT Domain Filtering Butterworth Lowpass Filter Gaussian Lowpass Filter Butterworth Highpass Filter Gaussian Highpass Filter I-frame 套用 1.071sec / 419 Frames 1.219sec / 419 Frames 1.109sec / 419 Frames 1.125sec / 419 Frames IPB-frame 套用 1.375sec / 419 Frames 1.968sec / 419 Frames 1.328sec / 419 Frames 1.813sec / 419 Frames 視覺效果 smear mosaic

48 Possible Enhancements In the compressed domain  Changing DC,AC coefficients  Apply NPR to I-frame  Applying NPR to I frames and to P,B- frames with discontinuities

49 Frame-by-Frame Oil Paint  sec / 419 frames  Frame rate : fps  Flickering Sourceframe by frame NPR

50  sec / 419 frames  frame rate : fps  Lost frame Applying NPR to I-frame Only frame by frameI-frame NPR

51 Applying NPR to I Frames and P,B-Frames with Discontinuities

52 Computing Image Differences Can be done in DCT or spatial domain. Spatial domain approach: For all pixels if | ( D t +1 (i,j) - D t (i,j) )| > constant diff++ if( diff > percentage * pixels ) ApplyNPR( D t +1 )

53 Order of Computation

54 I-and discontinuous P,B-Frames NPR  Difference>5  Percentage=60% I-and Discontinuous P,B-Frames NPR I-frame NPR I-and discontinuous P,B- frames NPR

55 I-and Discontinuous P,B- frames NPR: Performance I-and discontinuous P,B-frames NPR  Difference+NPR sec/419 frames Frame rate : 7.65 fps  NPR sec/419 frames Frame rate : fps

56 Summary of I,P,B-frame NPR IPB-frame NPRI-frame NPR I-frame & Difference NPR 花費時間 sec28.407sec54.469sec Frame rate1.048 frame/sec frame/sec7.65 frame/sec 相對於逐張 frame 套用 NPR 增進效率百分 比 0%93% 87% ( Difference threshold = 60% ) 效果 每張皆有 NPR 效 果，但會有閃爍 情形，無法即時 場景差異過大時 P,B-frame 無 NPR 效果，達到即時 有適當的差異門檻值 可兼顧效果並接近即 時

57 Summary of Compressed Domain NPR 改變 DCT 係數僅套用於 I-framePB-frame 補強 NPR 套用速度 速度最快，可以達 到即時的效果 可達到即時效果近似即時 額外負擔無解壓縮 I-frame 解壓縮 I,PB-frame, 計算 Difference 特點速度快但效果有限 可以達到即時，可 套用空間上的 NPR 演算法 效果最好，但是否 達到即時視差異值 而定

58 Optimize Pixel-Based NPR Calculate variance in the compressed domain for pixel-based NPR Sum of AC 2 is equal to calculate variance in the spatial domain If the macro block is quite uniform, do not apply the NPR effect. AC Macro block 8x8 DC

59 Applying NPR to Selective Macro Blocks Variance =sum of all AC 2 in the macro block if( Variance > threshold ) ApplyNPR(macro block) else macro block = macro block of source image

60 Threshold =100

61 Threshold =200

62 Summary of Applying NPR to Selective Macro Blocks 門檻值 花費時間 1.09sec0.875sec0.719sec0.65sec 減少區塊比例 0%21.5%34.5%45%47% 相對於整張影 像套用 NPR 加 速時間比例 0%20%33%40%

63 Conclusions Applying NPR to regions-of-interest can indeed reduce processing time, making interactive applications feasible. Compressed domain processing has proven to be effective.

64 Future Work Encoding source image captured from Webcam to MPEG-I using hardware. Incorporating information in motion vectors to avoid the need to perform optical flow analysis.

65 Future Work (cont’d) MPEG-2  Resolution is higher  Difficult to achieve real-time performance.  Hardware acceleration is required. MPEG-4  Video Object (VO)

66 Q & A

67 Calculate Variance in the Compressed Domain for pixel-based NPR Sum of AC 2 DFT Domain Parseval's Theorem

68 NPR in the Compressed Domain Model Change DC value

69 NPR in the Compressed Domain Change DC value

70 NPR in the Compressed Domain Change DC value  α> 1 increase, α < 1 decrease 