1. Background Blurring shadow@cmlab.csie.ntu.edu.tw 1

2. 2

3. Outline Introduction Documentation ◦ Object Segmentation ◦ Depth Map Generation ◦ Image Defocus 3

4. Method 4 Lazy Snapping Perspective Box Pop-up Card Camera Settings Defocus Blur Privacy Preserving Photo Browser Object Segmentation Depth Map Generation Applications Image Defocus Lazy Snapping Perspective Box Pop-up Card Camera Settings

5. Method Segment ◦ Lazy snapping ◦ Mean shift 5 Resize to 20% Do hard graph cut by lazy snapping User Stroke Mean shift Lazy Snapping. ACM Trans. On Graphics 2004. Mean Shift: A Robust Approach Toward Feature Space Analysis. PAMI 2002. A Bayesian Approach to Digital Matting. CVPR 2001.

6. Method Depth map ◦ Perspective Box  Vanish point  Rear wall Depth Map Perspective Box Normal vector Pop-up card 6 Depth Map Normal vector Pop-up card Tour Into the Picture: Using a spidery mesh user interface to make animation from a single image. SIGGRAPH 1997

7. Method Depth map ◦ Pop-up card Depth Map Perspective Box Normal vector Pop-up card 7 Depth Map Normal vector Pop-up card

8. Method Depth map ◦ Normal vector Depth Map Perspective Box Normal vector Pop-up card 8 Depth Map Normal vector

9. Method Defocus blur ◦ bokeh 9 x

10. Method 10 Defocus blur

11. Method ◦ Near-by object case 11 Focus on the flower

12. ◦ Near-by object case Method Blur circle diameter Objects in front of the focus Objects behind the focus Objects behind the focus Defocus Blur Texture synthesis inpainting Blur shape as alpha map Defocus Blur Interpolation by alpha map Shallow focus image 12

13. Experimental Result 1 Defocus blur method proposed in the system comparing with other blur filter results 13 Deep focus Shallow focus

14. Experimental Result 1 Defocus blur method proposed in the system comparing with other blur filter results Gaussian blur Defocus blur Defocus blur + Bokeh 14

15. Experimental Result 1 15 Deep focus Shallow focus

16. Defocus blur Experimental Result 1 16 Gaussian blur Defocus blur + Bokeh Original photograph Defocus blur method proposed in the system comparing with other blur filter results

17. Experimental Result 2 ◦ Near-by object case 17 Focus on the flower

18. ◦ Near-by object case Without inpainting Interpolation result Experimental Result 2 18

19. Applications - Privacy Preserving 19

20. Applications - Image Transition at Photo Browser 20

21. 21 Applications - Image Transition at Photo Browser

22. Movie with defocus blur effect 22 Applications - Image Transition at Photo Browser

23. Documentation 23

24. ./ 系統主要分成三部份，前兩塊產生出 depth 和 segment 的 資訊之後，可以在第三部份根據不同參數設定產生出不同的 效果 ◦ system  1_segmentation  Tool1: lazy-snapping with simple UI  Tool2: 光影魔術手  2_depthmap  depth of background scene (3d-box)  depth of each segment  3_refocusblur  camera parameter  refocusing (inpainting, defocus blur) ◦ sample // sample input, parameter and result 24

25. Segmentation ◦ Lazy snapping  Tool1: preprocessing (resize for mean shift ) + lazy snapping  Lzsegment.m  Input : parameter file name 先做 preprocessing ，這裡可以用 meanshift 或是直接 downsample 成 64 色階，並且在 meanshift 的地方 resize 為五分之一加快速度， 讀取 stroke_1.bmp, stroke_2.bmp… 等前後景筆劃圖，前景是紅色， 後景是藍色。然後執行 lazy snapping 處理 ◎每個 segment 都存一個灰階的 bmp 檔，檔名分別是 seg_1.bmp, seg_2.bmp… 沒有先後順序。將每個 segment 扣除剩下的 background 的部份，即是 seg_segN.bmp ◎需要 libgil2 的 library  Tool2: 光影魔術手  每個 segment 都存一個 png 檔，檔名分別是 obj_1.png, obj_2.png… 將這些 png 檔的 alpha 值轉成 bmp, 並且扣除只剩 下 background 的部份, 即是 seg_segN.bmp 25

26. Depth map ◦ depth.m (function depth(parafilename))  Input:  parameter file name  dep_1.dat, dep_2.dat, …dep_(segN-1).dat ★ 每個 dep_n.dat 的內容 : n x n y p1 x p1 y p2 x p2 y … ★ n 表示 normal vector, 預設是 (0,0), vector 方向代表物體在平面上 的法向量方向，長度表示深度變化量 ★ p1,p2… 是 user 劃下的筆劃座標，即此 segment 的深度參考點，預設 是 segment 最下方的點。如果沒有 p1,p2… 系統會自動取 segment 最 下方一列的點作為參考點。  Output:  dep_1.bmp, dep_2.bmp … dep_segN.bmp 26

27. Refocus Blur ◦ refocus.m ( function [finalresult]=defocus(parafilename) )  Get parameter from file, and load the segment/depth bmp files  Get the focus position from focusposition.dat  Compute diffusion blur circle diameter for each pixel  Devide image into two parts, the part in front of the focus, and run defocus_blur.m  Use alpha.m to get alpha map, and find the pixels to run inpaint.m  Use the result by inpaint.m to call defocus_blur.m  Combine the front part and back part by alpha map  return the result image 27

28. others ◦ Inpainting function needs the estexemplarhelper.c run mex estexemplarhelper.c before the function call at first time The inpainting code (inpaint.m) is from http://www.cc.gatech.edu/~sooraj/inpainting/ http://www.cc.gatech.edu/~sooraj/inpainting/ ◦ Meanshift code is downloaded from ◦ http://www.caip.rutgers.edu/riul/ http://www.caip.rutgers.edu/riul/ ◦ 28

29. 29 中間檔案 / 參數 Generate fromfor focuspoint.dat UI 畫定 refocus.m 的聚焦點資訊 dep_1.dat, dep_2.dat … UI 畫定 depth.m 每個 segment 的 深度參考點和 normal 值， 如果沒有給予參考點則 視為最下方一排 stroke_1.bmp, stroke_2.bmp… UI 畫定 Lazy snapping 的前後景 分割依據 seg_1.bmp, seg_2.bmp… lzsegment.m 產生，或 是用 tool2 取得 obj.png 之後，用 seg_png2bmp.m 產生 depth.m 和 refocus.m 都 需要用到的 segment map dep_1.bmp, dep_2.bmp… depth.m 產生 refocus.m 需要用到的深 度資訊 v1,v2,a1,a2,b1,b2.. Vanish box 和消失點座 標, 可用 UI 介面畫定 depth.m 產生背景深度資 訊所需要 其他 camera 參數使用者自行調整

30. Input parameter file 1.dirnamediratory of the files, include '/' on the tail, ex: 'sample1/' 2.imagenamefile name of input image, ex: test.jpg 3.segNnumber of segment image, include the background. ex: 2 (foreground*1+background) 4.vx position of vanish point, must be in the rear wall, 5.vybut can be outside of the image 6.ax 7.ayleft_top position of the rear wall 8.bx 9.byright_bottom position of the rear wall 10.dc_flens focal length,(mm) ex: 50 mm, 100 mm...etc. 11.dpmaxthe depth of pixels with color level 255 in depth map, (mm) 12.dnearthe depth of pixels with color level 0 in depth map, (mm) 13.dc_Nthe aperture number 14.thr=250threshold of bokeh color level 15.light_e=0.5little constant number for light_weight function 16.lens shape default setting 0 is flat disk 30

31. Camera parameter dc_f 相機的焦距，單位是 mm ，數值越大景深越淺， 建議在 50~100 之間 Dpmaxrear wall 距離成像平面的距離，建議為 5~50 公尺 (5000~50000) Dnear 場景裡距離成像平面最近的物體距離， 建議為 1 公尺，至少要在兩倍焦距以上 dc_N 光圈值，數值越大，光圈孔隙越小，景深越深， 為達到淺景深效果建議值在 1.4~2.8 之間，越小效果越強 thr=250 將 pixel 產生散景效果的臨界值，建議不要低於 240 light_e=0.5weight function 的明度微調參數，為一個大於 0 的值， 數值越大明度影響越低，建議為 0.5~1 之間 lens shape 光圈孔隙的形狀，預設為 0 對應到 refocus/lens/ 資料夾底下的 0.bmp, 1.bmp 31

32. Sample ◦./  parameter01-.dat ◦./sample01  original image  seg_1.bmp seg_2.bmp …  dep_1.dat dep_2.dat …  可由 UI 產生  focusposition.dat  可由 UI 產生，給定要聚焦的點，可以為單一或是多個點，將會 取其對應到的深度平均值，格式為 x1 y1 x2 y2 …  dep_1.bmp dep_2.bmp … 32

33. 執行： ◦ 用 system 的前兩步驟產生出前頁所述的參數和 bmp files ，並放在所屬資料夾下 ◦ result = refocus(parameter file name); 33

34. END THANK YOU 34