Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro Computer Vision and Applications

Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro Computer Vision Computer vision is the study of extracting content from digital image data (my definition) the analysis of digital images by a computer (Shapiro’s definition) the science and technology of machines that see (Wikipedia definition) One textbook says: “The goal of computer vision is to make useful decisions about real physical objects and scenes based on sensed images.”

Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro Applications Image databases

Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro More applications Robot vision Mars exploration rover Stanley – winner of 2005 DARPA Grand Challenge

Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro More applications Face detection and recognition

Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro More Applications Surveillance Modeling for graphics and animation

Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro More Applications liver kidney Medical imaging Document analysis

Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro More Applications Photo tourism

Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro More Applications Games

Industrial inspection Biometrics (faces, fingerprints) Motion analysis (including gestures and actions) Road / traffic analysis Real-time tracking Augmented reality Human-computer interaction Visual navigation Image / video indexing and retrieval Motion capture and entertainment … More Applications

Related fields Machine vision –Sometimes refers to industrial applications Image processing –Transforming one image into another Pattern recognition –Concerned with classification or description of observations –Data could be anything (not necessarily images) Photogrammetry –Science of obtaining accurate measurements and maps from photographs (images) Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro

Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro Why Study Computer Vision? Images and movies are everywhere Fast-growing collection of useful applications Interesting scientific mysteries –how does object recognition work? Better understanding of human vision

Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro Complexity Computer vision is far from a solved problem Successful systems exist –Usually for controlled situations –Often dependent on parameter settings There are many visual tasks that people perform better than computers

Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro Imaging Geometry

Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro Pinhole Cameras Abstract camera model - box with a small hole in it Pinhole cameras work in practice

Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro Distant Objects Are Smaller

Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro Parallel Lines Meet Common to draw film plane in front of the focal point. Moving the film plane merely scales the image.

Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro Cartesian coordinates: –We have, by similar triangles, that: –(X, Y, Z) ~ (f X/Z, f Y/Z, f) –f is called the focal length. The equation of projection [X, Y, Z] [fX/Z, fY/Z]

Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro The reason for lenses We won’t worry much about lenses in this class.

Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro Lens distortion “Barrel distortion” of rectangular grid is common for inexpensive lenses Precision lenses can be expensive Zoom lenses often show severe distortion Fish-eye lenses also have severe distortion

Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro Image capture Images are not continuous Typically captured with a CCD camera (charge-coupled-device) The amount of light striking each location on a grid is integrated over some time period Rows are read out one at a time For color images, successive pixels usually correspond to different colors High quality color cameras use a beam splitter and 3 separate CCD chips

Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro Resolution Resolution often (but not always) refers to the number of pixels in the image. Lower resolution has fewer pixels. Interestingly, faces of people you know can usually be recognized at 64 x 64 (or less) pixels. Squint and look at the lowest resolution image.

Programming in OpenCV In OpenCV, images are represented as matrices (as in linear algebra). Mat image = imread("photo.jpg");// Most generic declaration The image could have a number of underlying data types for each pixel: uchar – unsigned byte (greyscale image) Vec3b – vector of 3 bytes (color image) Point2f – point in two dimensions, float many others… Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro

Creating images Images can be created using a number of methods: using namespace cv;// all my code assumes this Mat image;// creates 0x0 image Mat image = …// uses copy constructor Mat image(rows, cols, type);// type is CV_8U, for example Mat image(rows, cols, type, scalarValue); –Example: Mat allBlue(360, 480, CV_8UC3, Scalar(255, 0, 0)); Mat_ colorImage = imread(“color.jpg”); // Can be convenient, but now limited to Vec3b images (matrices) // Also, must declare as a similar parameter type when passed Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro

Copying images Be careful to remember that most image copy and pass by value methods do NOT perform a deep copy. image2 = image1; // shallow copy void someMethod(Mat imageParam);// shallow copy If you want a deep copy, then use clone (or copyTo): image2 = image1.clone(); // deep copy image1.copyTo(image2);// deep copy Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro

Memory management Memory management is handled by the Mat class. –This is different from the IplImage class in OpenCV 1 –This works correctly even if multiple images share the same data –A reference count is kept for each image –The data is deallocated only when the reference count goes to zero –However, this can allow privacy leaks unless you are careful Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro

Backwards compatibility OpenCV 2 is backwards compatible with OpenCV 1. IplImage *iplIm = cvLoadImage("photo.jpg"); // Do work with image here cvReleaseImage(&iplIm); // necessary to prevent memory leak Can convert to Mat simply: Mat converted(iplIm);// do not release image until finished Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro

Image manipulation OpenCV provides many methods to manipulate entire images: Filtering: blur, smooth, median, gradient, laplacian Transformations: resize, affine, perspective, color space, threshold, flood fill Segmentation: grabCut, watershed Feature detection: edges, corners, lines, circles, template matching, SIFT Computer Vision Set: Applications Slides by D.A. Forsyth, C.F. Olson, J. Ponce, L.G. Shapiro