Internet-scale Imagery for Graphics and Vision James Hays cs195g Computational Photography Brown University, Spring 2010.

Slides:



Advertisements
Similar presentations
Distinctive Image Features from Scale-Invariant Keypoints
Advertisements

Feature Detection. Description Localization More Points Robust to occlusion Works with less texture More Repeatable Robust detection Precise localization.
Distinctive Image Features from Scale-Invariant Keypoints David Lowe.
Presented by Xinyu Chang
The SIFT (Scale Invariant Feature Transform) Detector and Descriptor
TP14 - Local features: detection and description Computer Vision, FCUP, 2014 Miguel Coimbra Slides by Prof. Kristen Grauman.
Object Recognition using Invariant Local Features Applications l Mobile robots, driver assistance l Cell phone location or object recognition l Panoramas,
CSE 473/573 Computer Vision and Image Processing (CVIP)
Distinctive Image Features from Scale- Invariant Keypoints Mohammad-Amin Ahantab Technische Universität München, Germany.
Instructor: Mircea Nicolescu Lecture 13 CS 485 / 685 Computer Vision.
Object Recognition with Invariant Features n Definition: Identify objects or scenes and determine their pose and model parameters n Applications l Industrial.
Fast High-Dimensional Feature Matching for Object Recognition David Lowe Computer Science Department University of British Columbia.
Robust and large-scale alignment Image from
A Study of Approaches for Object Recognition
Object Recognition with Invariant Features n Definition: Identify objects or scenes and determine their pose and model parameters n Applications l Industrial.
Distinctive Image Feature from Scale-Invariant KeyPoints
Feature extraction: Corners and blobs
Distinctive image features from scale-invariant keypoints. David G. Lowe, Int. Journal of Computer Vision, 60, 2 (2004), pp Presented by: Shalomi.
Scale Invariant Feature Transform (SIFT)
Automatic Matching of Multi-View Images
SIFT - The Scale Invariant Feature Transform Distinctive image features from scale-invariant keypoints. David G. Lowe, International Journal of Computer.
Blob detection.
Automatic Image Alignment (feature-based) : Computational Photography Alexei Efros, CMU, Fall 2006 with a lot of slides stolen from Steve Seitz and.
The SIFT (Scale Invariant Feature Transform) Detector and Descriptor
776 Computer Vision Jan-Michael Frahm, Enrique Dunn Spring 2013.
Opportunities of Scale, Part 2 Computer Vision James Hays, Brown Many slides from James Hays, Alyosha Efros, and Derek Hoiem Graphic from Antonio Torralba.
Distinctive Image Features from Scale-Invariant Keypoints David G. Lowe – IJCV 2004 Brien Flewelling CPSC 643 Presentation 1.
Scale-Invariant Feature Transform (SIFT) Jinxiang Chai.
Creating and Exploring a Large Photorealistic Virtual Space INRIA / CSAIL / Adobe First IEEE Workshop on Internet Vision, associated with CVPR 2008.
Distinctive Image Features from Scale-Invariant Keypoints By David G. Lowe, University of British Columbia Presented by: Tim Havinga, Joël van Neerbos.
Computer vision.
Bag of Visual Words for Image Representation & Visual Search Jianping Fan Dept of Computer Science UNC-Charlotte.
Object Tracking/Recognition using Invariant Local Features Applications l Mobile robots, driver assistance l Cell phone location or object recognition.
Overview Harris interest points Comparing interest points (SSD, ZNCC, SIFT) Scale & affine invariant interest points Evaluation and comparison of different.
Reporter: Fei-Fei Chen. Wide-baseline matching Object recognition Texture recognition Scene classification Robot wandering Motion tracking.
Computational Photography Tamara Berg Features. Representing Images Keep all the pixels! Pros? Cons?
CVPR 2003 Tutorial Recognition and Matching Based on Local Invariant Features David Lowe Computer Science Department University of British Columbia.
CSCE 643 Computer Vision: Extractions of Image Features Jinxiang Chai.
Lecture 7: Features Part 2 CS4670/5670: Computer Vision Noah Snavely.
Distinctive Image Features from Scale-Invariant Keypoints Ronnie Bajwa Sameer Pawar * * Adapted from slides found online by Michael Kowalski, Lehigh University.
Distinctive Image Features from Scale-Invariant Keypoints David Lowe Presented by Tony X. Han March 11, 2008.
Feature extraction: Corners and blobs. Why extract features? Motivation: panorama stitching We have two images – how do we combine them?
A Tutorial on using SIFT Presented by Jimmy Huff (Slightly modified by Josiah Yoder for Winter )
Project 3 questions? Interest Points and Instance Recognition Computer Vision CS 143, Brown James Hays 10/21/11 Many slides from Kristen Grauman and.
Local features: detection and description
Scale Invariant Feature Transform (SIFT)
Presented by David Lee 3/20/2006
Tentative Future Courses Fall `11 : Computer Vision – emphasis on recognition Spring `11 : Graduate seminar Fall `12 : Computational Photography.
Recognizing specific objects Matching with SIFT Original suggestion Lowe, 1999,2004.
Distinctive Image Features from Scale-Invariant Keypoints Presenter :JIA-HONG,DONG Advisor : Yen- Ting, Chen 1 David G. Lowe International Journal of Computer.
Blob detection.
SIFT.
776 Computer Vision Jan-Michael Frahm Spring 2012.
SIFT Scale-Invariant Feature Transform David Lowe
Presented by David Lee 3/20/2006
Lecture 07 13/12/2011 Shai Avidan הבהרה: החומר המחייב הוא החומר הנלמד בכיתה ולא זה המופיע / לא מופיע במצגת.
Distinctive Image Features from Scale-Invariant Keypoints
Scale Invariant Feature Transform (SIFT)
Project 1: hybrid images
TP12 - Local features: detection and description
Local features: detection and description May 11th, 2017
Cheng-Ming Huang, Wen-Hung Liao Department of Computer Science
CAP 5415 Computer Vision Fall 2012 Dr. Mubarak Shah Lecture-5
The SIFT (Scale Invariant Feature Transform) Detector and Descriptor
Interest Points & Descriptors 3 - SIFT
SIFT keypoint detection
SIFT.
ECE734 Project-Scale Invariant Feature Transform Algorithm
SIFT SIFT is an carefully designed procedure with empirically determined parameters for the invariant and distinctive features.
Presented by Xu Miao April 20, 2005
Presentation transcript:

Internet-scale Imagery for Graphics and Vision James Hays cs195g Computational Photography Brown University, Spring 2010

Recap from Monday What imagery is available on the Internet What different ways can we use that imagery – aggregate statistics – sort by keyword – visual search category / scene recognition instance / landmark recognition

How many images are there? Torralba, Fergus, Freeman. PAMI 2008

Lots Of Images A. Torralba, R. Fergus, W.T.Freeman. PAMI 2008

Lots Of Images A. Torralba, R. Fergus, W.T.Freeman. PAMI 2008

Lots Of Images

Automatic Colorization Result Grayscale input High resolution Colorization of input using average A. Torralba, R. Fergus, W.T.Freeman. 2008

Automatic Orientation Many images have ambiguous orientation Look at top 25% by confidence: Examples of high and low confidence images:

Automatic Orientation Examples A. Torralba, R. Fergus, W.T.Freeman. 2008

Tiny Images Discussion Why SSD? Can we build a better image descriptor?

Gist Scene Descriptor Better than SSD, why?

Gist Scene Descriptor Hays and Efros, SIGGRAPH 2007

Gist Scene Descriptor Gist scene descriptor (Oliva and Torralba 2001) Hays and Efros, SIGGRAPH 2007

Gist Scene Descriptor Gist scene descriptor (Oliva and Torralba 2001) Hays and Efros, SIGGRAPH 2007

Gist Scene Descriptor Gist scene descriptor (Oliva and Torralba 2001) Hays and Efros, SIGGRAPH 2007

Gist Scene Descriptor + Gist scene descriptor (Oliva and Torralba 2001) Hays and Efros, SIGGRAPH 2007

Scene matching with camera transformations

Image representation Color layout GIST [Oliva and Torralba’01] Original image

3. Find a match to fill the missing pixels Scene matching with camera view transformations: Translation 1. Move camera 2. View from the virtual camera 4. Locally align images 5. Find a seam 6. Blend in the gradient domain

4. Stitched rotation Scene matching with camera view transformations: Camera rotation 1. Rotate camera 2. View from the virtual camera 3. Find a match to fill-in the missing pixels 5. Display on a cylinder

Scene matching with camera view transformations: Forward motion 1. Move camera 2. View from the virtual camera 3. Find a match to replace pixels

Navigate the virtual space using intuitive motion controls Tour from a single image

Video

Distinctive Image Features from Scale-Invariant Keypoints David Lowe Slides from Derek Hoiem and Gang Wang

object instance recognition (matching)

Challenges Scale change Rotation Occlusion Illumination ……

Strategy Matching by stable, robust and distinctive local features. SIFT: Scale Invariant Feature Transform; transform image data into scale-invariant coordinates relative to local features

SIFT Scale-space extrema detection Keypoint localization Orientation assignment Keypoint descriptor

Scale-space extrema detection Find the points, whose surrounding patches (with some scale) are distinctive An approximation to the scale-normalized Laplacian of Gaussian

Maxima and minima in a 3*3*3 neighborhood

Keypoint localization There are still a lot of points, some of them are not good enough. The locations of keypoints may be not accurate. Eliminating edge points.

(1) (2) (3)

Eliminating edge points Such a point has large principal curvature across the edge but a small one in the perpendicular direction The principal curvatures can be calculated from a Hessian function The eigenvalues of H are proportional to the principal curvatures, so two eigenvalues shouldn’t diff too much

Orientation assignment Assign an orientation to each keypoint, the keypoint descriptor can be represented relative to this orientation and therefore achieve invariance to image rotation Compute magnitude and orientation on the Gaussian smoothed images

Orientation assignment A histogram is formed by quantizing the orientations into 36 bins; Peaks in the histogram correspond to the orientations of the patch; For the same scale and location, there could be multiple keypoints with different orientations;

Feature descriptor

Based on 16*16 patches 4*4 subregions 8 bins in each subregion 4*4*8=128 dimensions in total

Application: object recognition The SIFT features of training images are extracted and stored For a query image 1.Extract SIFT feature 2.Efficient nearest neighbor indexing 3.3 keypoints, Geometry verification

Extensions PCA-SIFT 1.Working on 41*41 patches 2.2*39*39 dimensions 3.Using PCA to project it to 20 dimensions

Surf Approximate SIFT Works almost equally well Very fast

Conclusions The most successful feature (probably the most successful paper in computer vision) A lot of heuristics, the parameters are optimized based on a small and specific dataset. Different tasks should have different parameter settings. Learning local image descriptors (Winder et al 2007): tuning parameters given their dataset. We need a universal objective function.