Check web page often T,R 12:30-1:50pm PSCB (Phy Sci Class Blg) 140 Course intro handout Computational Photography and Vision (CS116) Deva Ramanan Slides adapted from Alyosha Efros, Rick Szeliski and Steve Seitz

Agenda Intros Computational photography/vision overview Course overview Image processing – let’s dive in Readings (due next class) Online Book (link from course website): Richard Szeliski, Computer Vision: Algorithms and Applications –Intro: Ch 1.0 & 2.1 Ungraded HW (due next class) MATLAB tutorial (link from course website)

About me Deva Ramanan Relatively new faculty (2cnd class ever!) My research focus is on computer vision Part of new Computational Vision Lab

What is computational photography? Convergence of image processing, computer vision, computer graphics and photography Digital photography: Simply replaces traditional sensors and recording by digital technology Involves only simple image processing Computational photography More elaborate image manipulation, more computation New types of media (panorama, 3D, etc.) Camera design that take computation into account

GRAPHICS What is computer graphics? (3D->2D) 3D geometry physics Simulation projection

What is computer vision? (2D->3D) 3D geometry physics Estimation

What is computer vision? Terminator 2

Every picture tells a story Goal of computer vision is to write computer programs that can interpret images

Can computers match (or beat) human vision? Yes and no (but mostly no!) humans are much better at “hard” things computers can be better at “easy” things

Human perception has its shortcomings… Sinha and Poggio, Nature, 1996

Copyright A.Kitaoka 2003A.Kitaoka

Current state of the art The next slides show some examples of what current vision systems can do

Earth viewers (3D modeling) Image from Microsoft’s Virtual EarthVirtual Earth (see also: Google Earth)Google Earth

Photo Tourism

Optical character recognition (OCR) Digit recognition, AT&T labs Technology to convert scanned docs to text If you have a scanner, it probably came with OCR software License plate readers

Face detection Many new digital cameras now detect faces Canon, Sony, Fuji, …

Smile detection? Sony Cyber-shot® T70 Digital Still Camera

Object recognition (in supermarkets) LaneHawk by EvolutionRobotics “A smart camera is flush-mounted in the checkout lane, continuously watching for items. When an item is detected and recognized, the cashier verifies the quantity of items that were found under the basket, and continues to close the transaction. The item can remain under the basket, and with LaneHawk,you are assured to get paid for it… “

Face recognition Who is she?

Vision-based biometrics “How the Afghan Girl was Identified by Her Iris Patterns” Read the storystory

Login without a password… Fingerprint scanners on many new laptops, other devices Face recognition systems now beginning to appear more widely

Object recognition (in mobile phones) This is becoming real: Microsoft Research Point & Find, NokiaPoint & FindNokia

The Matrix movies, ESC Entertainment, XYZRGB, NRC Special effects: shape capture

Pirates of the Carribean, Industrial Light and Magic Special effects: motion capture

Special effects: image-based rendering

Sports Sportvision first down line Nice explanation on

Smart cars Mobileye Vision systems currently in high-end BMW, GM, Volvo models By 2010: 70% of car manufacturers. Slide content courtesy of Amnon Shashua

Vision-based interaction (and games) Nintendo Wii has camera-based IR tracking built in. See Lee’s work at CMU on clever tricks on using it to create a multi-touch display!Lee’s work at CMU multi-touch display DigimaskDigimask: put your face on a 3D avatar. “Game turns moviegoers into Human Joysticks”, “Game turns moviegoers into Human Joysticks”, CNET Camera tracking a crowd, based on this work.this work

Robotics NASA’s Mars Spirit Rover

Medical imaging Image guided surgery Grimson et al., MIT 3D imaging MRI, CT

Current state of the art You just saw examples of current systems. Many of these are less than 5 years old This is a very active research area, and rapidly changing Many new apps in the next 5 years To learn more about vision applications and companies David Lowe maintains an excellent overview of vision companiesDavid Lowe –

This course Prerequisites Calculus, linear algebra + probability helpful Interest in playing with images Emphasis on programming projects! Best way to learn is to build something from scratch MATLAB has a low learning curve 5 projects (15% of grad) + final exam (25%) Project due every 2 weeks For larger projects, “part 1” due first week

Project 1: Demosaicing 1)Get feet wet with MATLAB 2)Turn raw output of digital camera into a color image

Project 2: hole-filling and blending The fun stuff! Tools: bayesian modelling, differential equations

Project 3: Image re-targeting Click on video Tools: combinatorial optimization, dynamic programming

Project 4: Automatic mosaicing Tools: linear algebra, signal processing

Project 5: Face detection & recognition Tools: probabilistic modeling

Cameras Don’t need for class, but really cool Digital SLRs are ideal Point–and-shoots still nice and not too expensive (<$200) e.g. Canon A550

References There is no required text. Various course notes and papers will be made available. We will often use an online draft of an upcoming book: Richard Szeliski, Computer Vision: Algorithms and Applications There is a number of other fine texts that you can use for general reference: Computer Vision: The Modern Approach, Forsyth and Ponce Vision Science: Photons to Phenomenology, Stephen Palmer Multiple View Geometry in Computer Vision, Hartley & Zisserman The Computer Image, Watt and Policarpo Linear Algebra and its Applications, Gilbert Strang

A little bit of teaching philosophy… 1) Prefer discussion vs lectures – ask questions! 2) Readings before class help ‘set the stage’ 3) We learn best by doing it ourselves – progamming projects important! 4) Don’t like powerpoint (makes students sleep) -But visual aides are nice -Slides will be available online *after* class -I encourage you to take notes during class

Favor to ask… Need to boost enrollment - spread the gospel about this cool class!