1. Northeastern University, Fall 2005 CSG242: Computational Photography Ramesh Raskar Mitsubishi Electric Research Labs Northeastern University Course WebPage : http://www.merl.com/people/raskar/photo/course/

2. CSG242: Computational Photography Course WebPage http://www.merl.com/people/raskar/photo/course Google phrase ‘Northeastern Computational Photography’

3. Welcome Understanding Film-like PhotographyUnderstanding Film-like Photography –Parameters, Nonlinearities, Ray-based concepts Image Processing and Reconstruction ToolsImage Processing and Reconstruction Tools –Multi-image Fusion, Gradient domain, Graph Cuts Improving Camera PerformanceImproving Camera Performance –Better dynamic range, focus, frame rate, resolution Future DirectionsFuture Directions –HDR cameras, Gradient sensing, Smart optics/lighting

4. Welcome Topics: Digital Imaging Beyond Film-like PhotographyDigital Imaging Beyond Film-like Photography Computational Aspects of Lenses, Image Sensors and ProcessingComputational Aspects of Lenses, Image Sensors and Processing Algorithmic Solutions for Camera Sensor, Lens and Lighting LimitationsAlgorithmic Solutions for Camera Sensor, Lens and Lighting Limitations Adaptive Fusion of Multiple Images for Impossible PhotosAdaptive Fusion of Multiple Images for Impossible Photos Image Reconstruction from Coded Images SamplesImage Reconstruction from Coded Images Samples Future Directions in Smart Lighting, Optics and SensorsFuture Directions in Smart Lighting, Optics and Sensors

5. Goals –Understand computational aspects of cameras Discuss issues in traditional camerasDiscuss issues in traditional cameras Explore alternative imaging methodsExplore alternative imaging methods Learn vision and optics techniquesLearn vision and optics techniques –Discuss image processing and reconstruction tools –Review of 30+ recent papers

6. Goals –What we will not cover Composition, Aesthetics, ColorComposition, Aesthetics, Color Film Cameras, film issuesFilm Cameras, film issues Lighting equipmentLighting equipment Color issuesColor issues Traditional image processing/editing (Photoshop)Traditional image processing/editing (Photoshop) –Histogram, Artistic filters Digital camera user manual, StorageDigital camera user manual, Storage Novel view rendering (IBR)Novel view rendering (IBR)

7. Instructor: Ramesh Raskar Ramesh Raskar is a Senior Research Scientist at MERL. His research interests include projector-based graphics, computational photography and non-photorealistic rendering. He has published several articles on imaging and photography including multi-flash photography for depth edge detection, image fusion, gradient-domain imaging and projector-camera systems. His papers have appeared in SIGGRAPH, EuroGraphics, IEEE Visualization, CVPR and many other graphics and vision conferences. He was a course organizer at Siggraph 2002, 2003 and 2004. He is a panel organizer at the Symposium on Computational Photography and Video in Cambridge, MA in May 2005. He is a member of the ACM and IEEE. http://www.merl.com/people/raskar/raskar.html

8. Opportunities –Unlocking Photography How to expand camera capabilitiesHow to expand camera capabilities Digital photography that goes beyond film-like photographyDigital photography that goes beyond film-like photography –Opportunities Computing corrects for lens, sensor and lighting limitationsComputing corrects for lens, sensor and lighting limitations Computing merges results from multiple imagesComputing merges results from multiple images Computing reconstructs from coded image samplesComputing reconstructs from coded image samples Cameras benefit from computerized light sourcesCameras benefit from computerized light sources –Think beyond post-capture image processing Computation well before image processing and editingComputation well before image processing and editing –Learn how to build your own camera-toys

9. Traditional Photography Lens Detector Pixels Image

10. Computational Photography: Optics, Sensors and Computations Generalized Sensor Generalized Optics Computations Picture 4D Ray Bender Upto 4D Ray Sampler Ray Reconstruction

11. Computational Photography Novel Cameras Generalized Sensor Generalized Optics Processing

12. Computational Photography Novel Illumination Novel Cameras Generalized Sensor Generalized Optics Processing Light Sources

13. Computational Photography Novel Illumination Novel Cameras Scene : 8D Ray Modulator Generalized Sensor Generalized Optics Processing Light Sources

14. Computational Photography Novel Illumination Novel Cameras Scene : 8D Ray Modulator Display Generalized Sensor Generalized Optics Processing Recreate 4D Lightfield Light Sources

15. Computational Photography Novel Illumination Novel Cameras Scene : 8D Ray Modulator Display Generalized Sensor Generalized Optics Processing 4D Ray Bender Upto 4D Ray Sampler Ray Reconstruction Generalized Optics Recreate 4D Lightfield Light Sources Modulators 4D Incident Lighting 4D Light Field

16. A Teaser: Dual Photography Scene Photocell Projector

17. A Teaser: Dual Photography Scene Photocell Projector

18. A Teaser: Dual Photography Scene Photocell Projector

19. A Teaser: Dual Photography Scene Photocell Projector Camera

20. camera The 4D transport matrix: Contribution of each projector pixel to each camera pixel scene projector

21. camera The 4D transport matrix: Contribution of each projector pixel to each camera pixel scene projector Sen et al, Siggraph 2005

22. camera The 4D transport matrix: Which projector pixel contribute to each camera pixel scene projector Sen et al, Siggraph 2005 ?

23. Dual photography from diffuse reflections the camera’s view Sen et al, Siggraph 2005

24. Rollout Photographs © Justin Kerr: Slide idea: Steve Seitz http://research.famsi.org/kerrmaya.html Are BOTH of a ‘photograph’?

25. New Ways of Seeing the World “Multiple-Center-of-Projection Images” Rademacher, P, Bishop, G., SIGGRAPH '98

26. What rays are most expressive? Andrew Davidhazy, RIT : http://www.rit.edu/~andpph/ http://www.rit.edu/~andpph/

27. Thick photography: interaction What other ways better reveal shape to human viewers? (Without direct shape measurement? ) Time for space wiggle. Time for space wiggle. Gasparini, 1998. Can you understand this shape better? this shape better?

28. Topics: Digital Imaging Beyond Film-like PhotographyDigital Imaging Beyond Film-like Photography Computational Aspects of Lenses, Image Sensors and ProcessingComputational Aspects of Lenses, Image Sensors and Processing Algorithmic Solutions for Camera Sensor, Lens and Lighting LimitationsAlgorithmic Solutions for Camera Sensor, Lens and Lighting Limitations Adaptive Fusion of Multiple Images for Impossible PhotosAdaptive Fusion of Multiple Images for Impossible Photos Image Reconstruction from Coded Images SamplesImage Reconstruction from Coded Images Samples Future Directions in Smart Lighting, Optics and SensorsFuture Directions in Smart Lighting, Optics and Sensors

29. Camera Obscura, Gemma Frisius, 1558 1558 A Brief History of Images

30. Lens Based Camera Obscura, 1568 1558 1568 A Brief History of Images

31. Still Life, Louis Jaques Mande Daguerre, 1837 1558 1837 1568 A Brief History of Images

32. Silicon Image Detector, 1970 1558 1837 1568 1970 A Brief History of Images

33. 1558 1837 1568 1970 1994 A Brief History of Images Digital Cameras

34. Dream of A New Photography Old New People and Time ~Cheap Precious Each photo Precious Free Lighting Critical Automated* External Sensors No Yes ‘Stills / Video’ Disjoint Merged Exposure Settings Pre-select Post-Process Exposure Time Pre-select Post-Process Resolution/noise Pre-select Post-Process ‘HDR’ range Pre-select Post-Process

35.  Marc Levoy The digital photography marketplace 90+% of American households own a camera, 20% own a digital camera, 3% own only digital 57 million film cameras sold last year (down 10%) + 100+ million disposable film cameras (up 8%) + 53 million digital cameras (up 15%) + 53 million camera phones (skyrocketing) Surveillance camera, robotics, Webcams..

36.  Marc Levoy Digital cameras are boring roughly the same features and controls as film cameras –zoom and focus –aperture and exposure –shutter release and advance –one shutter press = one snapshot but things are changing…

37.  Marc Levoy Digital camera technology #1: real-time in-camera processing fast auto-focus systems –sharp photographs of moving objects optical image stabilization –long handheld exposures automatic object recognition –adaptive metering (e.g. of faces) → helps film cameras as well

38.  Marc Levoy Digital camera technology #2: unusual acquisition protocols continuous auto-focusing –sequences of sharp photographs of moving objects large DRAM buffer –16MB on Canon EOS-D30 = 10 JPEG images –permits burst-mode photography free and plentiful permanent storage –2 GB microdrive = 1000 JPEG images –permits extended shooting → replacing film cameras for newsgathering

39.  Marc Levoy Digital camera technology #3: digital post-processing Photoshop –replacing traditional darkroom techniques –also replacing exposure compensation, color filtering, and other specialized shooting techniques second generation tools –warping images, stitching panoramas –will eventually replace the view and panoramic camera around the bend –high-X imaging (resolution, dynamic range, focus, etc.) –techniques based on multiple images

40. Survey How many of you are photographers ?How many of you are photographers ? How many of you are photo-artists ?How many of you are photo-artists ? How many of you are digiphoto-artists ?How many of you are digiphoto-artists ? How many do active programming ?How many do active programming ? Field of work: Academics? Industry ? Research ? Art ?Field of work: Academics? Industry ? Research ? Art ? Brief IntroductionsBrief Introductions

41. Credit (Tentative) Assignments: Five project-oriented assignments Requires programming in Matlab 8 points each Mid-term Exam 20 points Term Paper Individual or a group of 2, 8 to 10 pages, 15 minute presentation, demo encouraged 15 points Final Project Individual or in groups of up to 3 20 points Discretionary credit 5 points Each student is expected to prepare notes for one lecture These notes should be prepared and emailed to the instructor no later than the following Sunday (midnight EST). Revisions and corrections will be exchanged by email and after changes the notes will be posted to the website before class the following week. Discretionary credit will be given for this. No late submissions

42. Structure of class: 1st half 6:00PM – 7:20PM break 7:20PM – 7:40PM break 7:20PM – 7:40PM 2nd half 7:40PM – 9:00PM 2nd half 7:40PM – 9:00PM Please be on time. Please keep your cell-phones switched off during class hours. Please do not email/IM during class. Every class has a break of 20 minutes for your convenience, please use this time to take/make calls, check emails, etc. Please be on time. Please keep your cell-phones switched off during class hours. Please do not email/IM during class. Every class has a break of 20 minutes for your convenience, please use this time to take/make calls, check emails, etc. Course mailing list: Please make sure that your emailid is on the course mailing list – csg242@ccs.neu.edu. Send email to raskar@merl.com Instructor: Ramesh Raskar, 617-621-7533, raskar@merl.com Email is the best way to get in touch with me. Office hours: -- Location: --