Applications of Shape Similarity

Robotics: Shape Screening ASR: Applications in Computer Vision Robotics: Shape Screening (Movie: Robot2.avi) Straightforward Training Phase Recognition of Rough Differences Recognition of Differences in Detail Recognition of Parts

View Invariant Human Activity Recognition ASR: Applications in Computer Vision Application 2: View Invariant Human Activity Recognition (Dr. Cen Rao and Mubarak Shah, School of Electrical Engineering and Computer Science, University of Central Florida)

Human Action Defined by Trajectory Application: Human Activity Recognition Human Action Defined by Trajectory Action Recognition by Comparison of Trajectories (Movie: Trajectories) Rao / Shah: Extraction of ‘Dynamic Instants’ by Analysis of Spatiotemporal Curvature Comparison of ‘Dynamic Instants’ (Sets of unconnected points !) ASR: Simplification of Trajectories by Curve Evolution Comparison of Trajectories

Application: Human Activity Recognition Simplification Trajectory

Activity Recognition: Typical Set of Trajectories

Trajectories in Tangent Space

Trajectory Comparison by ASR: Results

Recognition of 3D Objects by Projection Background: MPEG 7 uses fixed view angles Improvement: Automatic Detection of Key Views

Automatic Detection of Key Views (Pairwise) Comparison of Adjacent Views Detects Appearance of Hidden Parts

Automatic Detection of Key Views Result (work in progress):

The Database Implementation Application: ASR The Database Implementation

The Main Application: Back to ISS Task: Create Image Database Problem: Response Time Comparison of 2 Shapes: 23ms on Pentium1Ghz ISS contains 15,000 images: Response Time about 6 min. Clustering not possible: ASR failed on measuring dissimilarities !

Vantage Objects Solution: Full search on entire database using a simpler comparison Vantage Objects (Vleugels / Veltkamp, 2000) provide a simple comparison of n- dimensional vectors (n typically < 100)

Vantage Objects The Idea: Compare the query-shape q to a predefined subset S of the shapes in the database D The result is an n-dimensional Vantage Vector V, n = |S| s1 v1 s2 v2 q s3 v3 … sn vn

Vantage Objects - Each shape can be represented by a single Vantage Vector - The computation of the Vantage Vector calls the ASR – comparison only n times - ISS uses 54 Vantage Objects, reducing the comparison time (needed to create the Vantage Vector) to < 1.5s - How to compare the query object to the database ?

Vantage Objects - Create the Vantage Vector vi for every shape di in the database D - Create the Vantage Vector vq for the query-shape q - compute the (euclidean) distance between vq and vi - best response is minimum distance Note: computing the Vantage Vectors for the database objects is an offline process !

How to define the set S of Vantage Objects ?

k=1..i-1 e(di , sk) maximal. (e = eucl. dist.) Vantage Objects Algorithm 1 (Vleugels / Veltkamp 2000): Predefine the number n of Vantage Objects S0 = { } Iteratively add shapes di  D\Si-1 to Si-1 such that Si = Si-1  di and k=1..i-1 e(di , sk) maximal. (e = eucl. dist.) Stop if i = n.

Vantage Objects Result: Did not work for ISS.

Algorithm 2 (Latecki / Henning / Lakaemper): Vantage Objects Algorithm 2 (Latecki / Henning / Lakaemper): Def.: A(s1,s2): ASR distance of shapes s1,s2 q: query shape ‘Vantage Query’ : determining the result r by minimizing e(vq , vi ) vi = Vantage Vector to si ‘ASR Query’: determining the result r by minimizing A(q,di ) Vantage Query has certain loss of retrieval quality compared to ASR query. Define a loss function l to model the extent of retrieval performance

Vantage Objects Given a Database D and a set V of Vantage Vectors, the loss of retrieval performance for a single query by shape q is given by: lV,D (q) = A(q,r), Where r denotes the resulting shape of the vantage query to D using q. Property: lV,D (q) is minimal if r is the result of the ASR-Query.

L(S) = 1/n  lS,D\{si} (si) Vantage Objects Now define retrieval error function L(S) of set S={s1 ,…, sn }  D of Vantage Vectors of Database D: L(S) = 1/n  lS,D\{si} (si) Task: Find subset S  D such that L(S) is minimal.

Vantage Objects Algorithm: V0={ } iteratively determine sj in D\Sj-1 such that Sj =Sj-1  sj and L(Vj) minimal. Stop if improvement is low

Number of Vantage Objects Result: Worked fine for ISS, though handpicked objects still performed better. Handpicked Algorithm 2 L(S) Number of Vantage Objects

Vantage Objects …some of the Vantage Objects used in ISS:

Vantage Objects and ISS The Vantage Objects are used in the ASR in the first (handdrawn) query. The query is compared to 54 Objects, then a vector comparison is computed with the whole database. The first result, also called ‘first guess’, is the result of the vantage vector search. Searching for a ‘grabbed’ a shape on the user interface leads to direct comparison with the ASR, these results are precomputed, since the query is a known shape !

Vantage Objects and ISS A: the handdrawn sketch B: the result of the Vantage search C: the result of the exact match