1. Northeastern University, Fall 2005 CSG242: Computational Photography Ramesh Raskar Mitsubishi Electric Research Labs Northeastern University Oct 19th, 2005 Course WebPage : http://www.merl.com/people/raskar/photo/course/

2. Plan for Today “The Eye As a Camera” Michael Sandberg“The Eye As a Camera” Michael Sandberg Computational IlluminationComputational Illumination Second Programming AssignmentSecond Programming Assignment Mid TermMid Term –Oct 26 th Project Proposals DueProject Proposals Due –November 2 nd Paper readingPaper reading 2 per student, 15 mins each, Reading list on the web2 per student, 15 mins each, Reading list on the web Starts Nov 2 ndStarts Nov 2 nd

3. Credits Assignments:Assignments: Five project-oriented assignmentsFive project-oriented assignments Requires programming in MatlabRequires programming in Matlab 8 points each (Last assignment format is flexible)8 points each (Last assignment format is flexible) Mid-term ExamMid-term Exam 20 points20 points Paper reading (two papers per student, 15 min presentation, 5pts each)Paper reading (two papers per student, 15 min presentation, 5pts each) 10 points (Was Term Paper, 15 points)10 points (Was Term Paper, 15 points) Final ProjectFinal Project Individual or in a group of 2Individual or in a group of 2 20 points20 points Discretionary creditDiscretionary credit 10 points (Was 5)10 points (Was 5)

4. Tentative Schedule Oct 26: Midterm examOct 26: Midterm exam Nov 2 nd Project Proposals DueNov 2 nd Project Proposals Due Nov 9 th Class ? Likely on 10 thNov 9 th Class ? Likely on 10 th Nov 16 th Class ?Nov 16 th Class ? Nov 23 rd -> Likely on Nov 22 ndNov 23 rd -> Likely on Nov 22 nd Nov 30 thNov 30 th Dec 7 thDec 7 th Dec 15 th (Exam week) ProjectsDec 15 th (Exam week) Projects

5. Mid-Term Oct 26 th at 6pm,Oct 26 th at 6pm, Duration: 90 minutesDuration: 90 minutes Questions: Think, Explore, SolveQuestions: Think, Explore, Solve –No need to remember all the formulas in detail –More concepts than math problems –Drawing diagrams to explain concepts 20 points20 points TopicsTopics All material covered till Oct 19 thAll material covered till Oct 19 th Slides, assignments and in-class discussionsSlides, assignments and in-class discussions Basics, Dynamic Range, Focus, IlluminationBasics, Dynamic Range, Focus, Illumination

6. Focus

9. Computational Illumination

10. Synthetic Lighting Paul Haeberli, Jan 1992

11. Computational Photography Novel Illumination Novel Cameras Scene : 8D Ray Modulator Display Generalized Sensor Generalized Optics Processing Recreate 4D Lightfield Light Sources

12. Photography Artifacts: Flash Hotspot AmbientFlash Flash Hotspot

13. UnderexposedReflections AmbientFlash Reflections due to Flash

14. Flash Brightness Falloff with Distance Flash Distant people underexposed

15. Combining Flash/No-flash Images for High Dynamic Range (HDR) Imaging

16. Need Both Ambient and Flash!! FlashAmbient HDR Scene: Underexposed Well-lit in Flash Well-lit in Ambient

17. Exposure Time 1/1001/201/5141/250 Conventional Exposure HDR: Varying Exposure Time

18. Flash Brightness 42 7 1 0 Flash HDR: Varying Flash Brightness Scene distance dependence

19. Exposure Time Flash Brightness Flash-Exposure Sampling Flash-Exposure HDR: Varying both

20. Varying Exposure timeVarying Flash brightnessVarying both Capturing HDR Image

21. Do We Need All Images ? Regular 2D Sampling 24 Pictures Adaptive Sampling 5 pictures Next Best Picture ? Exposure Time Flash Brightness Exposure Time Flash Brightness Based on all previous picturesBased on all previous pictures Maximize well-lit pixels over the imageMaximize well-lit pixels over the image Exclude pixels already captured as well-exposedExclude pixels already captured as well-exposed

22. HDR Image using N images HDR Image using N+1 images UnderexposedStill UnderexposedWell-exposed Exposure Time Flash Brightness ? ? ? N+1 th picture ?

23. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Ramesh Raskar, Karhan Tan, Rogerio Feris, Jingyi Yu, Matthew Turk Mitsubishi Electric Research Labs (MERL), Cambridge, MA U of California at Santa Barbara U of North Carolina at Chapel Hill Non-photorealistic Camera: Depth Edge Detection and Stylized Rendering using Multi-Flash Imaging

24. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

25. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Car Manuals

26. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera What are the problems with ‘real’ photo in conveying information ? Why do we hire artists to draw what can be photographed ?

27. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Shadows Clutter Many Colors Highlight Shape Edges Mark moving parts Basic colors

28. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Shadows Clutter Many Colors Highlight Edges Mark moving parts Basic colors A New Problem

29. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Why Non-photorealistic (NPR) Images ? Easy to Understand Easy to Display Require not-so-rich (3D) data Can we directly capture using a camera ? –Quick comprehensible images for the masses –Tools for the artists

30. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Depth Edge Camera

31. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

32. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

33. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

34. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

35. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

36. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

37. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

38. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

39. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

40. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

41. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

42. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

43. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Depth Discontinuities Internal and external Shape boundaries, Occluding contour, Silhouettes

44. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Depth Edges

45. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Sigma = 9Sigma = 5 Sigma = 1 Canny Intensity Edge Detection Our method captures shape edges

46. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Our MethodCanny

47. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Our Method Photo

48. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Canny Intensity Edge Detection Our Method Photo Result

49. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

50. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Canny Intensity Edge Detection Our Method

51. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

52. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

53. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Imaging Geometry Shadow lies along epipolar ray

54. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Shadow lies along epipolar ray, Epipole and Shadow are on opposite sides of the edge Imaging Geometry m

55. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Shadow lies along epipolar ray, Shadow and epipole are on opposite sides of the edge Imaging Geometry m

56. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Depth Edge Camera Light epipolar rays are horizontal or vertical

57. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Normalized Left / Max Right / Max Left Flash Right Flash InputU{depth edges}

58. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Normalized Left / Max Right / Max Left Flash Right Flash InputU{depth edges}

59. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Normalized Left / Max Right / Max Left Flash Right Flash InputU{depth edges} Negative transition along epipolar ray is depth edge Plot

60. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Normalized Left / Max Right / Max Left Flash Right Flash Input Negative transition along epipolar ray is depth edge PlotU{depth edges}

61. Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera % Max composite maximg = max( left, right, top, bottom); % Normalize by computing ratio images r1 = left./ maximg;r2 = top./ maximg; r3 = right./ maximg;r4 = bottom./ maximg; % Compute confidence map v = fspecial( 'sobel' ); h = v'; d1 = imfilter( r1, v ); d3 = imfilter( r3, v ); % vertical sobel d2 = imfilter( r2, h ); d4 = imfilter( r4, h ); % horizontal sobel %Keep only negative transitions silhouette1 = d1.* (d1>0); silhouette2 = abs( d2.* (d2<0) ); silhouette3 = abs( d3.* (d3<0) ); silhouette4 = d4.* (d4>0); %Pick max confidence in each confidence = max(silhouette1, silhouette2, silhouette3, silhouette4); imwrite( confidence, 'confidence.bmp'); No magic parameters !