Northeastern University, Fall 2005 CSG242: Computational Photography Ramesh Raskar Mitsubishi Electric Research Labs Northeastern University Oct 19th, 2005 Course WebPage :

Plan for Today “The Eye As a Camera” Michael Sandberg“The Eye As a Camera” Michael Sandberg Computational IlluminationComputational Illumination Second Programming AssignmentSecond Programming Assignment Mid TermMid Term –Oct 26 th Project Proposals DueProject Proposals Due –November 2 nd Paper readingPaper reading 2 per student, 15 mins each, Reading list on the web2 per student, 15 mins each, Reading list on the web Starts Nov 2 ndStarts Nov 2 nd

Credits Assignments:Assignments: Five project-oriented assignmentsFive project-oriented assignments Requires programming in MatlabRequires programming in Matlab 8 points each (Last assignment format is flexible)8 points each (Last assignment format is flexible) Mid-term ExamMid-term Exam 20 points20 points Paper reading (two papers per student, 15 min presentation, 5pts each)Paper reading (two papers per student, 15 min presentation, 5pts each) 10 points (Was Term Paper, 15 points)10 points (Was Term Paper, 15 points) Final ProjectFinal Project Individual or in a group of 2Individual or in a group of 2 20 points20 points Discretionary creditDiscretionary credit 10 points (Was 5)10 points (Was 5)

Tentative Schedule Oct 26: Midterm examOct 26: Midterm exam Nov 2 nd Project Proposals DueNov 2 nd Project Proposals Due Nov 9 th Class ? Likely on 10 thNov 9 th Class ? Likely on 10 th Nov 16 th Class ?Nov 16 th Class ? Nov 23 rd -> Likely on Nov 22 ndNov 23 rd -> Likely on Nov 22 nd Nov 30 thNov 30 th Dec 7 thDec 7 th Dec 15 th (Exam week) ProjectsDec 15 th (Exam week) Projects

Mid-Term Oct 26 th at 6pm,Oct 26 th at 6pm, Duration: 90 minutesDuration: 90 minutes Questions: Think, Explore, SolveQuestions: Think, Explore, Solve –No need to remember all the formulas in detail –More concepts than math problems –Drawing diagrams to explain concepts 20 points20 points TopicsTopics All material covered till Oct 19 thAll material covered till Oct 19 th Slides, assignments and in-class discussionsSlides, assignments and in-class discussions Basics, Dynamic Range, Focus, IlluminationBasics, Dynamic Range, Focus, Illumination

Focus

Computational Illumination

Synthetic Lighting Paul Haeberli, Jan 1992

Computational Photography Novel Illumination Novel Cameras Scene : 8D Ray Modulator Display Generalized Sensor Generalized Optics Processing Recreate 4D Lightfield Light Sources

Photography Artifacts: Flash Hotspot AmbientFlash Flash Hotspot

UnderexposedReflections AmbientFlash Reflections due to Flash

Flash Brightness Falloff with Distance Flash Distant people underexposed

Combining Flash/No-flash Images for High Dynamic Range (HDR) Imaging

Need Both Ambient and Flash!! FlashAmbient HDR Scene: Underexposed Well-lit in Flash Well-lit in Ambient

Exposure Time 1/1001/201/5141/250 Conventional Exposure HDR: Varying Exposure Time

Flash Brightness Flash HDR: Varying Flash Brightness Scene distance dependence

Exposure Time Flash Brightness Flash-Exposure Sampling Flash-Exposure HDR: Varying both

Varying Exposure timeVarying Flash brightnessVarying both Capturing HDR Image

Do We Need All Images ? Regular 2D Sampling 24 Pictures Adaptive Sampling 5 pictures Next Best Picture ? Exposure Time Flash Brightness Exposure Time Flash Brightness Based on all previous picturesBased on all previous pictures Maximize well-lit pixels over the imageMaximize well-lit pixels over the image Exclude pixels already captured as well-exposedExclude pixels already captured as well-exposed

HDR Image using N images HDR Image using N+1 images UnderexposedStill UnderexposedWell-exposed Exposure Time Flash Brightness ? ? ? N+1 th picture ?

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Ramesh Raskar, Karhan Tan, Rogerio Feris, Jingyi Yu, Matthew Turk Mitsubishi Electric Research Labs (MERL), Cambridge, MA U of California at Santa Barbara U of North Carolina at Chapel Hill Non-photorealistic Camera: Depth Edge Detection and Stylized Rendering using Multi-Flash Imaging

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Car Manuals

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera What are the problems with ‘real’ photo in conveying information ? Why do we hire artists to draw what can be photographed ?

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Shadows Clutter Many Colors Highlight Shape Edges Mark moving parts Basic colors

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Shadows Clutter Many Colors Highlight Edges Mark moving parts Basic colors A New Problem

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Why Non-photorealistic (NPR) Images ? Easy to Understand Easy to Display Require not-so-rich (3D) data Can we directly capture using a camera ? –Quick comprehensible images for the masses –Tools for the artists

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Depth Edge Camera

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Depth Discontinuities Internal and external Shape boundaries, Occluding contour, Silhouettes

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Depth Edges

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Sigma = 9Sigma = 5 Sigma = 1 Canny Intensity Edge Detection Our method captures shape edges

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Our MethodCanny

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Our Method Photo

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Canny Intensity Edge Detection Our Method Photo Result

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Canny Intensity Edge Detection Our Method

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Imaging Geometry Shadow lies along epipolar ray

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Shadow lies along epipolar ray, Epipole and Shadow are on opposite sides of the edge Imaging Geometry m

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Shadow lies along epipolar ray, Shadow and epipole are on opposite sides of the edge Imaging Geometry m

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Depth Edge Camera Light epipolar rays are horizontal or vertical

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Normalized Left / Max Right / Max Left Flash Right Flash InputU{depth edges}

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Normalized Left / Max Right / Max Left Flash Right Flash InputU{depth edges}

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Normalized Left / Max Right / Max Left Flash Right Flash InputU{depth edges} Negative transition along epipolar ray is depth edge Plot

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera Normalized Left / Max Right / Max Left Flash Right Flash Input Negative transition along epipolar ray is depth edge PlotU{depth edges}

Mitsubishi Electric Research Labs Raskar, Tan, Feris, Yu, Turk MultiFlash NPR Camera % Max composite maximg = max( left, right, top, bottom); % Normalize by computing ratio images r1 = left./ maximg;r2 = top./ maximg; r3 = right./ maximg;r4 = bottom./ maximg; % Compute confidence map v = fspecial( 'sobel' ); h = v'; d1 = imfilter( r1, v ); d3 = imfilter( r3, v ); % vertical sobel d2 = imfilter( r2, h ); d4 = imfilter( r4, h ); % horizontal sobel %Keep only negative transitions silhouette1 = d1.* (d1>0); silhouette2 = abs( d2.* (d2<0) ); silhouette3 = abs( d3.* (d3<0) ); silhouette4 = d4.* (d4>0); %Pick max confidence in each confidence = max(silhouette1, silhouette2, silhouette3, silhouette4); imwrite( confidence, 'confidence.bmp'); No magic parameters !