Course 1: Room 6DE Computational Photography A.2: Concepts Tumblin.

Slides:



Advertisements
Similar presentations
Computer Graphics Inf4/MSc Computer Graphics Lecture Notes #16 Image-Based Modelling, Rendering and Lighting.
Advertisements

Paul Debevec Computational Photography: Advanced Topics Computational Photography: Advanced Topics.
Lecture 35: Photometric stereo CS4670 / 5670 : Computer Vision Noah Snavely.
3D Graphics Rendering and Terrain Modeling
Measuring BRDFs. Why bother modeling BRDFs? Why not directly measure BRDFs? True knowledge of surface properties Accurate models for graphics.
Light Fields PROPERTIES AND APPLICATIONS. Outline  What are light fields  Acquisition of light fields  from a 3D scene  from a real world scene 
Vision Sensing. Multi-View Stereo for Community Photo Collections Michael Goesele, et al, ICCV 2007 Venus de Milo.
Digital Photography By Mike Volker. What is Digital Photography? A form of Photography that uses digital technology to make images of subjects Replaced.
 CBo CBo.
Graphics and Games IS 101Y/CMSC 101 Computational Thinking and Design Carolyn Seaman University of Maryland Baltimore County.
Lecture 23: Photometric Stereo CS4670/5760: Computer Vision Kavita Bala Scott Wehrwein.
Advanced Computer Graphics CSE 190 [Spring 2015], Lecture 14 Ravi Ramamoorthi
Advanced Computer Graphics (Fall 2010) CS 283, Lecture 16: Image-Based Rendering and Light Fields Ravi Ramamoorthi
Advanced Computer Graphics (Spring 2005) COMS 4162, Lecture 21: Image-Based Rendering Ravi Ramamoorthi
Representations of Visual Appearance COMS 6160 [Spring 2007], Lecture 4 Image-Based Modeling and Rendering
Image-Based Modeling and Rendering CS 6998 Lecture 6.
CS 395/495-25: Spring 2003 IBMR: Week 10A Dimensionality of IBMR Methods Jack Tumblin
CSCE 641 Computer Graphics: Image-based Rendering (cont.) Jinxiang Chai.
Computational Photography Light Field Rendering Jinxiang Chai.
CSCE 641: Computer Graphics Image-based Rendering Jinxiang Chai.
Stereo Guest Lecture by Li Zhang
Project 1 artifact winners Project 2 questions Project 2 extra signup slots –Can take a second slot if you’d like Announcements.
Measure, measure, measure: BRDF, BTF, Light Fields Lecture #6
DIGITAL PHOTOGRAPHY: BACK TO BASICS Source:
Course 3: Computational Photography Ramesh Raskar Mitsubishi Electric Research Labs Jack Tumblin Northwestern University Course WebPage :
Northeastern University, Fall 2005 CSG242: Computational Photography Ramesh Raskar Mitsubishi Electric Research Labs Northeastern University Course WebPage.
1/22/04© University of Wisconsin, CS559 Spring 2004 Last Time Course introduction Image basics.
Capturing and controlling digital images. Great images are not made by digital cameras. They are made by photographers who understand what to look for.
Light and shading Source: A. Efros.
PRINCIPLES OF DESIGN PHOTOGRAPHY. BALANCE Visual center is above geometric center. Visual weight is determined by many variables Size Darkness – A strong.
Northeastern University, Fall 2005 CSG242: Computational Photography Ramesh Raskar Mitsubishi Electric Research Labs Northeastern University September.
Real-Time High Quality Rendering CSE 291 [Winter 2015], Lecture 6 Image-Based Rendering and Light Fields
Integration Of CG & Live-Action For Cinematic Visual Effects by Amarnath Director, Octopus Media School.
OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography:
Structure from images. Calibration Review: Pinhole Camera.
Technology and Historical Overview. Introduction to 3d Computer Graphics  3D computer graphics is the science, study, and method of projecting a mathematical.
1 CS6825: Image Formation How are images created. How are images created.
Digital Photography A tool for Graphic Design Graphic Design: Digital Photography.
Advanced Computer Graphics (Spring 2013) CS 283, Lecture 15: Image-Based Rendering and Light Fields Ravi Ramamoorthi
Image-Based Rendering. 3D Scene = Shape + Shading Source: Leonard mcMillan, UNC-CH.
 The adjustable opening in a camera lens used to control the amount of light reaching the film. The size of this hole is called the f-stop.
Image-Based Rendering from a Single Image Kim Sang Hoon Samuel Boivin – Andre Gagalowicz INRIA.
Dynamically Reparameterized Light Fields Aaron Isaksen, Leonard McMillan (MIT), Steven Gortler (Harvard) Siggraph 2000 Presented by Orion Sky Lawlor cs497yzy.
Tone Mapping Software Photomatix Pro Application to Photography Konferenz und Workshop '05 Reality-Based Visualization.
Computational photography CS4670: Computer Vision Noah Snavely.
A Case Study using the Hugh Morton Photograph Collection A Photographic Journey brought to you by the Digital Production Center.
Lecture Exposure/histograms. Exposure - Four Factors A camera is just a box with a hole in it. The correct exposure is determined by four factors: 1.
© 1999 Rochester Institute of Technology Introduction to Digital Imaging.
. Wild Dreams for Cameras Jack Tumblin Northwestern University From May 24 Panel Discussion on cameras.
CSC 461: Lecture 3 1 CSC461 Lecture 3: Models and Architectures  Objectives –Learn the basic design of a graphics system –Introduce pipeline architecture.
Capturing light Source: A. Efros.
Computational Photography
Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-based Graphics with Global Illumination and High Dynamic Range Photography.
CSE 185 Introduction to Computer Vision Stereo. Taken at the same time or sequential in time stereo vision structure from motion optical flow Multiple.
Interreflections : The Inverse Problem Lecture #12 Thanks to Shree Nayar, Seitz et al, Levoy et al, David Kriegman.
DIGITAL CAMERAS Prof Oakes. Overview Camera history Digital Cameras/Digital Images Image Capture Image Display Frame Rate Progressive and Interlaced scans.
Student Camera Concepts Examples. Concepts The underlying principles that apply regardless of the camera you are using. The underlying principles that.
Announcements Office hours today 2:30-3:30 Graded midterms will be returned at the end of the class.
112/5/ :54 Graphics II Image Based Rendering Session 11.
Don Norman’s Fall 2003 Seminar 8-Dimensional Digital Photography for Historical Preservation Jack Tumblin
Analyzing Impossible Images Steve Seitz University of Washington Computational Photography Symposium May 23, 2004.
Introduction to Photography To take beautiful photographs you do not need an expensive camera and a bag full of equipment. What is important is the photographer’s.
BASICS of the Camera A brief look at the origins and key features of the modern camera.
Hi, I’m Michele Del Core! I’m 18 years old and photography is one of my biggest passions. Practicing and doing researches about it, I discovered that.
Presented by 翁丞世  View Interpolation  Layered Depth Images  Light Fields and Lumigraphs  Environment Mattes  Video-Based.
Image-Based Rendering
3D Graphics Rendering PPT By Ricardo Veguilla.
Image Based Modeling and Rendering (PI: Malik)
A.2 Concepts in Computational Photography (Tumblin, 15 minutes)
Introduction to Photography
Presentation transcript:

Course 1: Room 6DE Computational Photography A.2: Concepts Tumblin

Jack Tumblin Northwestern University A2: Core Concepts (30 minutes) Computational Photography

Focus, Click, Print: ‘Film-Like Photography’ Angle( ,  ) Position(x,y) 2D Image: ‘Instantaneous’ Intensity Map Light + 3D Scene: Illumination, shape, movement, surface BRDF,… ‘Center of Projection’ (P 3 or P 2 Origin) Ray Bundles

Film-Like Photography Thought Experiment: Side-by-side digital camera & film camera. COMPARE:COMPARE: –Digital Camera result. –Digitized (Scanned) Film result. ? Can we See more, Do more, Feel more? ? Has photography really changed yet ? ? Has photography really changed yet ?

scene scene display SceneLightIntensities DisplayLightIntensities ‘Pixel values’ (scene intensity? display intensity? (scene intensity? display intensity? perceived intensity? ‘blackness/whiteness’ ?) perceived intensity? ‘blackness/whiteness’ ?) display Digitally Perfected Photography?

‘Film-Like’ Photography Film Camera design assumptions: –‘Instantaneous’ light measurement… –Of focal plane image behind a lens. –Reproduce those amounts of light. Implied: “What we see is  focal-plane intensities.” well, no…we see much more! (seeing is deeply cognitive) “What we see is  focal-plane intensities.” well, no…we see much more! (seeing is deeply cognitive)

Our Definitions ‘Film-like’ Photography:‘Film-like’ Photography: –Static ‘instantaneous’ record of the 2D image formed by a lens Display image  sensor image ‘Computational’ Photography:‘Computational’ Photography: –displayed image  sensor image –A more expressive, controllable displayed result, transformed, merged, decoded sensor data

What is Photography? Safe answer: A wholly new, expressive medium (ca. 1830s) A wholly new, expressive medium (ca. 1830s) Manipulated display of what we think, feel, want, …Manipulated display of what we think, feel, want, … –Capture a memory, a visual experience in tangible form –‘painting with light’; express the subject’s visual essence –“Exactitude is not the truth.” –Henri Matisse

What is Photography? A ‘bucket’ word: a neat container for messy notions (e.g. aviation, music, comprehension)A ‘bucket’ word: a neat container for messy notions (e.g. aviation, music, comprehension) A record of what we see, or would like to see, in tangible form.A record of what we see, or would like to see, in tangible form. Does ‘film’ photography always capture it?Does ‘film’ photography always capture it? No! What do we see? What is missing?What do we see? What is missing? Harold ‘Doc’ Edgerton 1936

Display RGB(x,y,t n ) Image I(x,y,λ,t) Light & Optics 3D Scene light sources, BRDFs, shapes, positions, movements, …Eyepoint position, movement, projection, … PHYSICALPERCEIVED What is Photography? Exposure Control, tone map Scene light sources, BRDFs, shapes, positions, movements, …Eyepoint position, movement, projection, … Vision Photo: A Tangible Record Editable, storable as Film or Pixels

3D Scene? light sources, BRDFs, shapes, positions, movements, …Eyepoint? position, movement, projection, …Meaning… VisualStimulus 3D Scene light sources, BRDFs, shapes, positions, movements, …Eyepoint position, movement, projection, … PHYSICAL PERCEIVED or UNDERSTOOD Ultimate Photographic Goals Vision Sensor(s) Computing Light & Optics Photo: A Tangible Record

Millions of delicate, fascinating treasures: –< 1% of Smithsonian collection ever exhibited –sparse $, displays; conservation limits access A Driving Problem: Museum Artifacts

Current Archives: Not rich enough Fixed, static viewpointFixed, static viewpoint Fixed, static lightingFixed, static lighting Custom light: impracticalCustom light: impractical Conflates shapes, materials, shadows, texture, highlights, …Conflates shapes, materials, shadows, texture, highlights, … Can you understand this shape?

Current Archives: Not rich enough Fixed, static viewpointFixed, static viewpoint Fixed, static lightingFixed, static lighting Custom light: impracticalCustom light: impractical Conflates shapes, materials, shadows, texture, highlights, …Conflates shapes, materials, shadows, texture, highlights, … Can you understand this shape? What ‘digital archive’ What ‘digital archive’ can best match in-hand, can best match in-hand, direct examination ? direct examination ? What is missing? What is missing?

Missing: Reliable Visual Boundaries 5 ray sets  explicit geometric occlusion boundaries Ramesh Raskar, MERL, 2004

Rollout Photographs © Justin Kerr: Slide idea: Steve Seitz Rollout Photographs © Justin Kerr: Slide idea: Steve Seitz Missing: Occlusion Removal BOTH capture visual appearance; BOTH should be easy to make! BOTH capture visual appearance; BOTH should be easy to make!

Missing: Viewpoint Freedom “Multiple-Center-of-Projection Images” Rademacher, P, Bishop, G., SIGGRAPH '98

Missing: Interaction… Adjust everything: lighting, pose, viewpoint, focus, FOV,… Winnemoller EG 2005: after Malzbender, SIGG2001

Missing: Expressive Time Manipulations What other ways better reveal appearance to human viewers? (Without direct shape measurement? ) Time for space wiggle. Time for space wiggle. Gasparini, Can you understand this shape better? this shape better?

Photographic Signal: Pixels Rays Core ideas are ancient, simple, seem obvious:Core ideas are ancient, simple, seem obvious: –Lighting: ray sources –Optics: ray bending/folding devices –Sensor: measure light –Processing: assess it –Display: reproduce it Ancient Greeks: ‘eye rays’ wipe the world to feel its contents…Ancient Greeks: ‘eye rays’ wipe the world to feel its contents…

The Photographic Signal Path Claim: Computing can improve every step Light Sources Sensors Data Types, Processing Display Rays OpticsOptics Scene Rays Eyes

Review: How many Rays in a 3-D Scene? A 4-D set of infinitesimal members. Imagine: –Convex Enclosure of a 3D scene –Inward-facing ray camera at every surface point –Pick the rays you need for ANY camera outside. 2D surface of cameras, 2D ray set for each camera, 2D surface of cameras, 2D ray set for each camera,  4D set of rays.  4D set of rays. (Levoy et al. SIGG’96) (Gortler et al. ‘96) +

4-D Light Field / Lumigraph Measure all the outgoing light rays.

4-D Illumination Field Same Idea: Measure all the incoming light rays

4D x 4D = 8-D Reflectance Field Ratio: R ij = (outgoing ray i ) / (incoming ray j )

[Debevec et al. 2002] [Debevec et al. 2000] [Masselus et al. 2002] [Masselus et al. 2003] [Malzbender et al. 2002] [Matusik et al. 2002] Is a 4-D Light Source Required?

Is A 4D Camera Required? e.g. MIT Dynamic Light Field Camera 2002 Multiple dynamic Virtual Viewpoints Efficient Bandwidth usage: ‘send only what you see’ Yang, et al tightly packed commodity CMOS webcams 30 Hz, Scaleable, Real-time: or is it just “more film-like cameras, but now with computers!” ? or is it just “more film-like cameras, but now with computers!” ? Is this the whole answer?

Or do Ray Changes Convey Appearance? 5 ray sets  explicit geometric occlusion boundaries Ramesh Raskar, MERL, 2004

Or do Ray Changes Convey Appearance? These rays + all these rays give me…These rays + all these rays give me… MANY more useful details I can examine…MANY more useful details I can examine…

Mild Viewing & Lighting Changes; Are these Enough? Convicing visual appearance: Is Accurate Depth really necessary? a few good 2-D images may be enough… “Image jets, Level Sets, and Silhouettes“ Lance Williams, talk at Stanford, 1998.

‘The Ideal Photographic Signal’ I CLAIM IT IS: All Rays? Some Rays? Changes in Some Rays All Rays? Some Rays? Changes in Some Rays Photographic ray space is vast and redundant >8 dimensions: 4D view, 4D light, time,, ? Gather only ‘visually significant’ ray changes ? ? What rays should we measure ? ? How should we combine them ? ? How should we display them ?

Future Photography Novel Illuminators Novel Cameras Scene : 8D Ray Modulator Generalized Sensors Generalized Processing 4D Ray Sampler Ray Reconstructor General Optics: 4D Ray Benders Recreated 4D Light field Lights Modulators 4D Incident Lighting Viewed 4D Light Field General Optics: 4D Ray Benders Generalized Display Generalized Display Novel Displays

Beyond ‘Film-Like’ Photography Call it ‘Computational Photography’: To make ‘meaningful ray changes’ tangible, To make ‘meaningful ray changes’ tangible, Optics can do more…Optics can do more… Sensors can do more…Sensors can do more… Light Sources can do more…Light Sources can do more… Processing can do more…Processing can do more… by applying low-cost storage, computation, and control. by applying low-cost storage, computation, and control.