Shape-Representation and Shape Similarity Dr. Rolf Lakaemper Part 1: Shapes

May I introduce myself… Rolf Lakaemper PhD (Doctorate Degree) 2000 Hamburg University, Germany Currently Assist. Professor at Department of Computer and Information Sciences, Temple University, Philadelphia, USA Main Research Area: Computer Vision

Research Goal Teaching robots to recognize the world they see …using SHAPE

Motivation WHY SHAPE ?

Motivation These objects are recognized by…

Motivation These objects are recognized by… TextureColorContextShape XX XX X X X XX

Why Shape ? Several applications in computer vision use shape processing: Object recognition Image retrieval Processing of pictorial information Video compression (eg. MPEG-7) … This presentation focuses on object recognition and image retrieval.

Motivation Typical Application: Multimedia: Image Database Query by Shape / Texture / … (Color / Keyword)

ISS Database Example : ISS-Database

The Interface (JAVA – Applet)

The Sketchpad: Query by Shape

The First Guess: Different Shape - Classes

Selected shape defines query by shape – class

Result

ISS Database ISS: Query by Shape / Texture Sketch of Shape Query: by Shape only Result: Satisfying ?

ISS Database SHAPE recognition seems to be possible and leads to satisfying results !

ISS Database We’ll talk about the ISS Database a bit later, so stay alert !

Overview Part 1: General thoughts about shape recognition Feature based approaches Part 2: Part based, direct approaches The ISS database Applications

Data Retrieval The most obvious sensor to gain the data for shape recognition is a camera. But shape is not only perceived by visual means: tactical sensors can also provide shape information that are processed in a similar way. robots’ range sensor provide shape information, too. Hence shape is a general, widely applicable object descriptor!

Shape Typical problems: How to describe shape ? What is the matching transformation? No one-to-one correspondence Occlusion Noise

Shape Partial match: only part of query appears in part of database shape

What is Shape ? … let’s start with some properties easy to agree on: Shape describes a spatial region Shape is a (the ?) specific part of spatial cognition Typically addresses 2D space

What is Shape ? Shape or Not ? Continuous transformation from shape to two shapes: Is there a point when it stops being a single shape?

What is Shape ? But there’s no doubt that a single, connected region is a shape. Right ?

What is Shape ? A single, connected region. But a shape ? A question of scale !

What is Shape ? There’s no easy, single definition of shape In difference to geometry, arbitrary shape is not covered by an axiomatic system Different applications in object recognition focus on different shape related features Special shapes can be handled Typically, applications in object recognition employ a similarity measure to determine a plausibility that two shapes correspond to each other

Similarity So the new question is: What is Shape Similarity ? or How to Define a Similarity Measure

Similarity Again: it’s not so simple (sorry). There’s nothing like THE similarity measure

Similarity Measure Requirements to a similarity measure Should not incorporate context knowledge (no AI), thus computes generic shape similarity

Similarity Measure Requirements to a similarity measure Must be able to deal with noise Must be invariant with respect to basic transformations Next: Strategy Scaling (or resolution) Rotation Rigid / non-rigid deformation

Similarity Measure Requirements to a similarity measure Must be able to deal with noise Must be invariant with respect to basic transformations Must be in accord with human perception

Similarity Measure Desired Properties of a Similarity Function C (Basri et al. 1998) C should be a metric C should be continuous C should be invariant (to…)

Properties Metric Properties S set of patterns Metric: d: S  S  R satisfying 1. Self-identity :  x  S, d(x,x)=0 2. Positivity :  x  y  S, d(x,y)>0 3. Symmetry :  x, y  S, d(x,y)= d(y,x) 4. Triangle inequality :  x, y, z  S, d(x,z)  d(x,y)+d(y,z) Semi-metric: 1, 2, 3 Pseudo-metric: 1, 3, 4 S with fixed metric d is called metric space

Properties 1.Self-identity :  x  S, d(x,x)=0 2.Positivity :  x  y  S, d(x,y)>0 …surely makes sense

Properties

In general: a similarity measure in accordance with human perception is NOT a metric. This leads to deep problems in further processing, e.g. clustering, since most of these algorithms need metric spaces !

Similarity Measures: Overview Similarity Measure depends on Shape Representation Boundary Area (discrete: = point set) Structural (e.g. Skeleton) Comparison Model feature vector direct

Similarity Measures directfeature based Boundary Spring model, Cum. Angular Function, Chaincode, Arc Decomposition (ASR- Algorithm) Central Dist. Fourier Distance histogram … Area (point set) Hausdorff … Moments Zernike Moments … Structure Skeleton … ---

Feature Based Coding This category defines all approaches that determine a feature-vector for a given shape. Two operations need to be defined: a mapping of shape into the feature space and a similarity of feature vectors. RepresentationFeature ExtractionVector Comparison

Feature Based Coding Again: TWO operations need to be defined… We hence have TWO TIMES an information reduction of the basic representation, which by itself is already a mapping of the ‘reality’. RepresentationFeature ExtractionVector Comparison

Example Vector of Elementary Descriptors Shape A,B given as Area (continous) or Point Sets (discrete)

Vector Comparison

Similarity (scalar value)

Vector Comparison All Feature Vector approaches have similar properties: Provide a compact representation this is especially interesting for database indexing ! Works for any shape Requires complete shapes (global comparison) Sensible to noise (except Zernike moments which are computationally demanding) Map dissimilar shapes to similar feature vectors (!) They can be used as a prefilter for database applications ! Make the choice of a similarity function difficult

Direct Comparison End of Feature Based Coding ! Next: Direct Comparison

Part II: Behind The Scenes of the ISS - Database: Modern Techniques of Shape Recognition and Database Retrieval

Overview Topics: The Shape Recognition Algorithm Implemented in ISS Possible Applications in Different Areas of Computer Vision

Results first… Image Database providing query by Keyword Texture Shape Shape is given by user-sketch, a mouse- drawn outline

ISS - GUI

The Sketchpad: Query by Shape

The First Guess: Different Shape - Classes

Selected shape defines query by shape – class

Result

Key Steps Retrieval by Vantage Objects Retrieval by Direct Shape Comparison

Wide range of applications recognition of complex and arbitrary patterns... invariance to basic transformations... results which are in accord with human perception... parameter-free operation Requirements Robust automatic recognition of arbitrary shaped objects which is in accord with human visual perception Industrial requirements robustness... low processing time... applicable to three main tasks of recognition

Wide range of applications recognition of complex and arbitrary patterns... invariance to basic transformations... results which are in accord with human perception... parameter-free operation Requirements Robust automatic recognition of arbitrary shaped objects which is in accord with human visual perception Industrial requirements robustness... low processing time Next: Strategy Scaling (or resolution) Rotation Rigid / non-rigid deformation... applicable to three main tasks of recognition

Wide range of applications recognition of complex and arbitrary patterns... results which are in accord with human perception... applicable to three main tasks of recognition... parameter-free operation Requirements Robust automatic recognition of arbitrary shaped objects which is in accord with human visual perception... robustness Industrial requirements robustness... low processing time... invariance to basic transformations... low processing time Simple Recognition (yes / no) Common Rating (best of...) Analytical Rating (best of, but...)

The 2 nd Step First: Shape Comparison Developed by Dr. Latecki / Dr. Lakaemper in cooperation with Siemens AG, Munich, for industrial applications in robotics... multimedia (MPEG – 7) ISS implements the ASR (Advanced Shape Recognition) Algorithm

MPEG 7 MPEG-7: ASR outperformes classical approaches ! Similarity test (70 basic shapes, 20 different deformations): Wavelet Contour Heinrich Hertz Institute Berlin67.67 % Multilayer EigenvectorHyundai70.33 % Curvature Scale SpaceMitsubishi ITE-VIL75.44 % ASRHamburg Univ./Siemens AG76.45 % DAG Ordered TreesMitsubishi/Princeton University60.00 % Zernicke MomentsHanyang University70.22 % (Capitulation :-)IBM--.-- %

The shape similarity algorithm behind the ISS- database is a direct, part based similarity measure.

Motivation WHY PARTS ?

Motivation

Global similarity measures fail at: Occlusion Global Deformation Partial Match (actually everything that occurs under ‘real’ conditions)

Requirements for a Part Based Shape Representation Principal approach: Hoffman/Richards (’85): ‘Part decomposition should precede part description’ => No primitives, but general principles

Parts No primitives, but general principals “When two arbitrarily shaped surfaces are made to interpenetrate they always meet in a contour of concave discontinuity of their tangent planes” (transversality principle)

Parts How should parts be defined ? Some approaches: Decomposition of interior Skeletons Maximally convex parts Best combination of primitives Boundary Based High Curvature Points Constant Curvature Segments

Visual Parts Motivated by psychological experiments (Hoffmann/Richards): split bounding-curve into convex / concave arcs

ASR: Strategy Source:2D - Image Arc – Matching Contour – Segmentation Contour Extraction Object - Segmentation Evolution

Curve Evolution Target: reduce data by elimination of irrelevant features, preserve relevant features... noise reduction... shape simplification:

Curve Evolution: Tangent Space Transformation from image-space to tangent-space bild s.22

Tangent Space: Properties In tangent space the height of a step shows the turn-angle... monotonic increasing intervals represent convex arcs... height-shifting corresponds to rotation... the resulting curve can be interpreted as 1 – dimensional signal => idea: filter signal in tangent space (demo: 'fishapplet')

Curve Evolution: Step Compensation New nonlinear filter: merging of 2 steps with area – difference F given by:  pq p + q F F  F  q p

Curve Evolution: Step Compensation Interpretation in image – space:... Polygon – linearization... removal of visual irrelevant vertices p q removed vertex

Curve Evolution: Step Compensation Interpretation in image – space:... Polygon – linearization... removal of visual irrelevant vertices next: Iterative SC

Curve Evolution: Iterative Step Compensation Keep it simple: repeated step compensation ! Remark: there are of course some traps...

The evolution reduces the shape-complexity... is robust to noise... is invariant to translation, scaling and rotation... preserves the position of important vertices... extracts line segments... is in accord with visual perception... offers noise-reduction and shape abstraction... is parameter free Curve Evolution: Properties... is translatable to higher dimensions

Curve Evolution: Properties Robustness (demo: noiseApplet)(demo: noiseApplet)

Curve Evolution: Properties Preservation of position, no blurring !

Strong relation to digital lines and segments Curve Evolution: Properties

Noise reduction as well as shape abstraction Curve Evolution: Properties

Parameter free Curve Evolution: Properties

Extendable to higher dimensions Curve Evolution: Properties

Extendable to higher dimensions Curve Evolution: Properties

Extendable to higher dimensions Curve Evolution: Properties

Extendable to higher dimensions Curve Evolution: Properties

Shape Comparison: Measure Tangent space offers an intuitive measure:

Shape Comparison: Measure Drawback: not adaptive to unequally distributed noise Solution: partition bounding curve

Shape Comparison: Contour Segmentation Solution: partition bounding curve

Shape Comparison: Contour Segmentation Motivated by psychological experiments (Hoffmann/Richards): split bounding-curve into convex / concave arcs

Shape Comparison: Correspondence Optimal arc-correspondence: find one to many (many to one) correspondence, that minimizes the arc-measure !

Graph of Correspondence a0 a1 a2 a3 b0 b1 b2 b3 a0 b0 a1 a2 a3 b1 b2 b3 Graph:... edge represents correspondence... node represents matched arcs arc correspondence

Shape Comparison: Correspondence Example: a0 a1 a2 a3 b0 b1 b2 b3 a0 b0 a1 a2 a3 b1 b2 b3

Shape Comparison: Correspondence Result: Optimal correspondence is given by cheapest way

Correspondence: Results

(Movie Deer.avi)

Correspondence: Results Correspondence and arc-measure allow the identification of visual parts as well as... the identification of the entire object... a robust recognition of defective parts... a shape matching which is in accord with human perception

ASR: Applications in Computer Vision Robotics: Shape Screening (Movie: Robot2.avi) Straightforward Training Phase Recognition of Rough Differences Recognition of Differences in Detail Recognition of Parts

ASR: Applications in Computer Vision Application 2: View Invariant Human Activity Recognition (Dr. Cen Rao and Mubarak Shah, School of Electrical Engineering and Computer Science, University of Central Florida)

Application: Human Activity Recognition Human Action Defined by Trajectory Action Recognition by Comparison of Trajectories (Movie: Trajectories) Rao / Shah: Extraction of ‘Dynamic Instants’ by Analysis of Spatiotemporal Curvature Comparison of ‘Dynamic Instants’ (Sets of unconnected points !) ASR: Simplification of Trajectories by Curve Evolution Comparison of Trajectories

Application: Human Activity Recognition Trajectory Simplification

Activity Recognition: Typical Set of Trajectories

Trajectories in Tangent Space

Trajectory Comparison by ASR: Results

Recognition of 3D Objects by Projection Background: MPEG 7 uses fixed view angles Improvement: Automatic Detection of Key Views

Automatic Detection of Key Views (Pairwise) Comparison of Adjacent Views Detects Appearance of Hidden Parts

Automatic Detection of Key Views Expected Result (work in progress):

Conclusion: Research in Shape Similarity has a lot of challenges, some solutions, and for sure is fun ! That’s it, Thanks !