Jack Pinches INFO410 & INFO350 S INFORMATION SCIENCE Computer Vision I

INFO410 S COMPUTER VISION I SLIDE 2 INFORMATION SCIENCE Topics  Definition  Related Fields  Application Examples  Application Overview  Typical Tasks  Methods  Feature Detection  Feature Description  Feature Matching  Exam Questions  Resources

Defining Computer Vision

INFO410 S COMPUTER VISION I SLIDE 4 INFORMATION SCIENCE Formal Definition  Making useful decisions about real physical objects and scenes based on images (Shapiro & Stockman, 2001).  Extracting descriptions of the world from pictures of sequences of pictures (Forsyth & Ponce, 2003).  Analysing images and producing descriptions that can be used to interact with the environment (Horn, 1986).  Designing representations and algorithms for relating images to models of the world (Ballard & Brown, 1982).

INFO410 S COMPUTER VISION I SLIDE 5 INFORMATION SCIENCE Informal Definition  Consists of acquiring, processing, analysing images/video of the real world  To produce information or descriptions  Often used as part of decision making

INFO410 S COMPUTER VISION I SLIDE 6 INFORMATION SCIENCE Definition  Extract “meaning” from pixels What we seeWhat a computer sees

INFO410 S COMPUTER VISION I SLIDE 7 INFORMATION SCIENCE Why is it important?  Images and videos are everywhere, and growing in numbers.  Trying to mimic human vision and understanding with computers.  Humans have good visual understanding – large amount of the brain devoted to visual processing.  Difficult to mimic.  Human level visual perception may be strong AI or “AI complete”.

INFO410 S COMPUTER VISION I SLIDE 8 INFORMATION SCIENCE Related Fields

Application Examples

INFO410 S COMPUTER VISION I SLIDE 10 INFORMATION SCIENCE Applications AgricultureAugmented RealityAutonomous VehiclesBiometricsCharacter RecognitionForensicsIndustrial Quality InspectionFace RecognitionMedical Image AnalysisPollution MonitoringProcess ControlRemote SensingRoboticsSecurity and SurveillanceTransport

INFO410 S COMPUTER VISION I SLIDE 11 INFORMATION SCIENCE Application Examples  Face Detection:  “Feature” included on many digital cameras

INFO410 S COMPUTER VISION I SLIDE 12 INFORMATION SCIENCE Application Examples  Automatic number plate recognition:  Used by law enforcement  Automatic tolling stations  Road surveillance

INFO410 S COMPUTER VISION I SLIDE 13 INFORMATION SCIENCE Application Examples  Facial Recognition:  Airport Security  Check the person matches the passport photo  Used in NZ

Application Overview

INFO410 S COMPUTER VISION I SLIDE 15 INFORMATION SCIENCE Typical Tasks  Detection:  Image scanned for a specific condition. E.g. detection of humans as part of a security system, or vehicles at automatic toll system.  Recognition:  Pre-learned objects can be recognized. E.g. recognition of a specific album cover.  Description:  Extracting data about specific object of interest. E.g. colour information, 2D position or 3D position.

INFO410 S COMPUTER VISION I SLIDE 16 INFORMATION SCIENCE Application Methods  Image Acquisition:  Image is produced by one or several image sensors. The image may have some associated meta data.  Pre-processing:  Process image to ensure it meets requirements.  Examples:  Re-sample the image to check it’s what’s expected to be outputted.  Noise reduction so sensor noise doesn’t add false information.  Contrast alteration so important features can be detected.

INFO410 S COMPUTER VISION I SLIDE 17 INFORMATION SCIENCE Application Methods  Feature extraction:  Image features are extracted from the image data.  Examples:  Lines, edges, intersections  Complex examples could include texture, shape, or motion  Detection:  Decide which image regions are required for further processing.

INFO410 S COMPUTER VISION I SLIDE 18 INFORMATION SCIENCE Application Methods  High level processing:  Usually small set of the original data. Image region which assume contains a specific object.  Examples:  Estimation of object pose or size  Object recognition  Final decision:  Make final decision for the application.  Examples:  Match/no match in recognition application  Flag if human review needed (medical, military applications)

Feature Detection

INFO410 S COMPUTER VISION I SLIDE 20 INFORMATION SCIENCE Feature Detection  Methods that try to extract image information.  Decisions made if there is an image feature or not.  Features will be a subset of the image.  Different approaches available.

INFO410 S COMPUTER VISION I SLIDE 21 INFORMATION SCIENCE Feature Detection – Image Features  Edges:  Points where there is a boundary between two image regions.  Can be almost any shape.

INFO410 S COMPUTER VISION I SLIDE 22 INFORMATION SCIENCE Feature Detection – Image Features  Corners (interest points):  Originally called corner as early algorithms performed edge detection. Then looked for edges which have rapid direction change.  Can look for high level of curvature.

INFO410 S COMPUTER VISION I SLIDE 23 INFORMATION SCIENCE Feature Detection – Image Features  Blobs:  Aimed at detecting regions of an image.  The region may differ in properties (such as brightness or colour) compared to surrounding images.

Feature Description

INFO410 S COMPUTER VISION I SLIDE 25 INFORMATION SCIENCE Feature Description  Feature Description Methods:  SIFT: Scale Invariant Feature Transform  SURF: Speeded up Robust Features  GLOH: Gradient Location and Orientation Histogram  HOG: Histogram of Orientated Gradients

INFO410 S COMPUTER VISION I SLIDE 26 INFORMATION SCIENCE Feature Description - SIFT  Detect and describe features in images.  Developed by David Lowe in  Algorithm described in:  Lowe, David G. (1999). "Object recognition from local scale-invariant features". Proceedings of the International Conference on Computer Vision 2. pp. 1150–1157. doi: /ICCV  C++ Implementation: 

INFO410 S COMPUTER VISION I SLIDE 27 INFORMATION SCIENCE  Input an image.  Output is a list of points on the image each associated to a vector of low level descriptors.  These points are called the keypoints.  Provide a local image description.  The points and their descriptions are invariant by rescaling, orientation, and some changes of illumination.  Used to find visual correspondences between images for different applications, such as image alignment or object recognition. Feature Description - SIFT

INFO410 S COMPUTER VISION I SLIDE 28 INFORMATION SCIENCE Feature Description – SIFT Image Description 813 Keypoints (slide 4)

INFO410 S COMPUTER VISION I SLIDE 29 INFORMATION SCIENCE Feature Description – SIFT Application  Align two images to produce a new image.

Feature Matching

INFO410 S COMPUTER VISION I SLIDE 31 INFORMATION SCIENCE Feature Matching – Why do it?  Fusing image from different sensors or times  Detect changes between images  Produce a new large image by overlapping/stitching smaller images E.g. Panorama  Object recognition by comparing an image to a reference image or model

INFO410 S COMPUTER VISION I SLIDE 32 INFORMATION SCIENCE Feature Matching

INFO410 S COMPUTER VISION I SLIDE 33 INFORMATION SCIENCE Feature Matching  Different Techniques:  SIFT  RANSAC (Random Sample Consensus)  Cross Correlation  Nearest Neighbour Techniques  Exhaustive Comparison

INFO410 S COMPUTER VISION I SLIDE 34 INFORMATION SCIENCE Feature Matching – What makes a good feature?  Uniqueness:  No ambiguous matches between images  Look for “interest points”: image regions that unique are (or at least unusual)  Defining unusual?

INFO410 S COMPUTER VISION I SLIDE 35 INFORMATION SCIENCE Feature Matching – With SIFT  Can detect interest points, create feature descriptors.  Compare feature descriptors between images.

INFO410 S COMPUTER VISION I SLIDE 36 INFORMATION SCIENCE Feature Matching – With Cross Correlation  Determine the important features:  Unique colour/structure of the neighbourhood  Geometry or topology of neighbourhood Vectors show the movement of the neighbourhood.

INFO410 S COMPUTER VISION I SLIDE 37 INFORMATION SCIENCE Feature Matching 1)Determine points 2)Match Neighbourhoods

Potential Exam Questions

INFO410 S COMPUTER VISION I SLIDE 39 INFORMATION SCIENCE Exam Questions  Define computer vision, and include the typical tasks that could be expected in a computer vision application.  Explain why it is important to use good feature descriptors for feature matching, use applications examples to support your reasoning.  Explain how you would design and implement a license plate detection and recognition application.

INFO410 S COMPUTER VISION I SLIDE 40 INFORMATION SCIENCE Resources  Computer Vision:    SIFT:    Features, Matching, and Detection:     