November 2002 Scene recognition by inexact graph matching1/39 ALIO/EURO Workshop Scene recognition by inexact graph matching Celso C. Ribeiro * Claudia Boeres Isabelle Bloch IV ALIO/EURO Workshop on Applied Combinatorial Optimization Pucón, Chile, November 4-6, 2002

November 2002 Scene recognition by inexact graph matching2/39 ALIO/EURO Workshop Summary Motivation and applications Optimal graph matching formulation Model: objective function and constraints Randomized construction algorithm Local search GRASP heuristic Numerical results Current and future work

November 2002 Scene recognition by inexact graph matching3/39 ALIO/EURO Workshop Motivation and applications Recognition and understanding of complex scenes require a detailed description of … –… the objects involved –… the spatial relationships between the objects The diversity of the forms of the same object in different instantiations of a scene, and also the similarities of different objects in the same scene, make the relationships between objects of prime importance to help the recognition of objects with similar appearance.

November 2002 Scene recognition by inexact graph matching4/39 ALIO/EURO Workshop Motivation and applications Graph-based representations often used for scene representation: –Nodes represent the objects in the scene. –Edges represent the relationships between the objects. Relevant information is extracted from the scene and represented by relational attributed graphs. Model-based recognition: both the model and the scene are represented by graphs.  Graph matching problem

November 2002 Scene recognition by inexact graph matching5/39 ALIO/EURO Workshop Motivation and applications Matching: graph or subgraph isomorphism –The assumption of a bijection between the elements in two instantiations of the same scene is too strong for some problems. –Usually, the model has a schematic aspect. –Construction of the image graph often relies on segmentation techniques that mail fail in accurately representing the image in meaningful entities, therefore no isomorphism can be expected. –Recognition can be better expressed as an inexact (non-bijective) graph matching problem.

November 2002 Scene recognition by inexact graph matching6/39 ALIO/EURO Workshop Motivation and applications Application: –Recognition of brain structures from 3D magnetic resonance images previously processed by a segmentation model (ENST and hospital Saint Vincent de Paul, Paris) –Model is an anatomical atlas: nodes represent the anatomical structures, while edges represent the spatial relationships between them. –Inaccuracies: oversegmented image, unexpected objects found (e.g. pathologies), expected objects not found, deformations, imprecise attributes

November 2002 Scene recognition by inexact graph matching7/39 ALIO/EURO Workshop Motivation and applications Axial slices of a brain (each grey level corresponds to a unique connected structure): middle dark structures (lateral ventricles) are much bigger in (b) than in (a), the white hyper-signal structure (tumor) in (b) does not appear in (a) or (c). (a) Normal brain (c) Brain atlas (b) Pathological brain

November 2002 Scene recognition by inexact graph matching8/39 ALIO/EURO Workshop Motivation and applications Other applications: –Aerial or satellite image interpretation using a map –Face recognition –Character recognition

November 2002 Scene recognition by inexact graph matching9/39 ALIO/EURO Workshop Optimal graph matching formulation Attributed graphs: one single vertex for each image region Typical attributes: adjacencies, distances, relative positions, grey levels Vertex similarities and edge similarities are computed from the values of the attributed graphs, relating each pair of vertices and each pair of edges (one from the model, the other from the image).

November 2002 Scene recognition by inexact graph matching10/39 ALIO/EURO Workshop Optimal graph matching formulation Model: G 1 =(N 1,E 1 ) Oversegmented image: G 2 =(N 2,E 2 ), with |N 2 |>|N 1 | Assign one single node of N 1 to each node of N 2 : Set of nodes associated with i  N 1 :

November 2002 Scene recognition by inexact graph matching11/39 ALIO/EURO Workshop Optimal graph matching formulation Edges (a,b)  E 1 and (p,q)  E 2 are associated if node a  N 1 is associated with p  N 2 and node b  N 1 is associated with q  N 2.

November 2002 Scene recognition by inexact graph matching12/39 ALIO/EURO Workshop Optimal graph matching formulation Search for the optimal graph matching which best represents the scene recognition is based on similarity values.

November 2002 Scene recognition by inexact graph matching13/39 ALIO/EURO Workshop Optimal graph matching formulation Similarity matrices: s v (i,j ): vertex-similarity between vertices i  N 1 and j  N 2. s e (u,v ): edge-similarity between edges u = (a,b )  E 1 and v = (c,d )  E 2. A good matching is a solution in which the vertex associations and the edge associations correspond to high similarity values.

November 2002 Scene recognition by inexact graph matching14/39 ALIO/EURO Workshop Model: objective function node-similarities contribution edge-similarities contribution

November 2002 Scene recognition by inexact graph matching15/39 ALIO/EURO Workshop Model: objective function High similarity associations are privileged: s v (i,j ) high  x ij = 1 maximizes node-similarities contribution: Low similarity associations are penalized: s v (i,j ) low  x ij = 0 maximizes

November 2002 Scene recognition by inexact graph matching16/39 ALIO/EURO Workshop Model: constraints Each vertex of N 2 has to be associated with exactly one vertex of N 1 : –Image segmentation is performed by an appropriate algorithm which preserves the boundaries. –In consequence, it avoids situations in which one region of the segmented image is located in the frontier of two adjacent regions of the model and no undersegmentation occurs. Constraint (1): For every j  N 2, there exists exactly one node i  N 1 such that x ij = 1, i.e.

November 2002 Scene recognition by inexact graph matching17/39 ALIO/EURO Workshop Model: constraints Vertices of N 2 associated with the same vertex of N 1 should be all connected among themselves: –An oversegmentation method can split an object into several pieces, but it does not change their relative positions (connectivity constraint). Constraint (2): For every i  N 1, the graph induced by A (i ) in G 2 = (N 2,E 2 ) is connected.

November 2002 Scene recognition by inexact graph matching18/39 ALIO/EURO Workshop Model: constraints Pairs of vertices with null similarity cannot be associated: Constraint (3): s v (i,j ) = 0  x ij = 0. All objects in the model should appear in the image: Constraint (4): For every i  N 1, there exists at least one node j  N 2 such that x ij = 1, i.e.

November 2002 Scene recognition by inexact graph matching19/39 ALIO/EURO Workshop Randomized construction algorithm MaxTrials: maximum number of attempts to build a feasible solution. Repeat for at most MaxTrials attempts: 1.Set A (i ) =  for every i  N 1. 2.Randomly select a node j  N 2 and set N 2  N 2 – {j }. 3.Randomly select a node i  N 1, until s v (i,j ) > 0 and the graph induced in G 2 by A (i )  {j } is connected. 4.Set A (i )  A (i )  {j }. 5.If N 2 ≠  go back to step 2 above. 6.If A (i ) ≠   i  N 1 and A -1 (j ) ≠   j  N 2, then stop.

November 2002 Scene recognition by inexact graph matching20/39 ALIO/EURO Workshop Randomized construction algorithm The algorithm stops and returns the first feasible solution found within at most MaxTrials attempts. The algorithm enforces feasibility, good solutions are not necessarily obtained. It may also be extended to return the best among the first MaxSolutions feasible solutions built. Each attempt has time complexity.

November 2002 Scene recognition by inexact graph matching21/39 ALIO/EURO Workshop Randomized construction algorithm

November 2002 Scene recognition by inexact graph matching22/39 ALIO/EURO Workshop Test problems 12 test problems –Some randomly generated, others from real problems associated with medical images –Two difficult instances with isolated nodes –Largest instance has |N 1 |=50 and |E 1 |=88, |N 2 |=250 and |E 2 |=1681. Construction algorithm is very fast and very often finds a feasible solution in the first attempt. Instances with isolated nodes are harder, but even in these cases a feasible solution is built.

November 2002 Scene recognition by inexact graph matching23/39 ALIO/EURO Workshop Test problems: I-6 (human brain) Model: 12 vertices 42 edges Image: 95 vertices 1434 edges

November 2002 Scene recognition by inexact graph matching24/39 ALIO/EURO Workshop Model: 14 vertices 27 edges Image: 28 vertices 63 edges Test problems: I-7 (muscle)

November 2002 Scene recognition by inexact graph matching25/39 ALIO/EURO Workshop Test problems: I-7 (muscle) modeloversegmented image

November 2002 Scene recognition by inexact graph matching26/39 ALIO/EURO Workshop Test problems: I-7 (feasible solutions) First feasible solution: 4 regions correctly recognized Best of ten first feasible solutions: 5 nodes correctly recognized Best of 100 first feasible solutions: 12 regions correctly recognized

November 2002 Scene recognition by inexact graph matching27/39 ALIO/EURO Workshop Local search Solutions built by the construction algorithm are not necessarily optimal. Local search algorithm successively replaces the current solution by a better one in a neighborhood of the first, terminating at a local optimum. First improving strategy: the current solution is replaced by the first neighbor whose cost function value improves that of the current solution.

November 2002 Scene recognition by inexact graph matching28/39 ALIO/EURO Workshop Local search: neighborhoods Neighborhood A: all solutions that can be obtained from the current one by modifying A -1 (j ) for some node j  N 2 (the new node associated with j has to enforce all feasibility constraints). Neighborhood B: all solutions that can be obtained from the current one by exchanging the nodes A -1 (p) and A - 1 (q ) of N 1, currently associated with nodes p and q of N 2.

November 2002 Scene recognition by inexact graph matching29/39 ALIO/EURO Workshop Local search: neighborhoods Neighborhood A: associate a new node (a) with node 2 Neighborhood B: exchange the nodes b and c associated with nodes 4 and 7

November 2002 Scene recognition by inexact graph matching30/39 ALIO/EURO Workshop Local search VND procedure using neighborhoods A and B and a first improving strategy: 1.Start with a solution built by the construction algorithm. 2.Move to the first improving solution in neighborhood A of the current solution until a local optimum is found. 3.If the current solution is also locally optimal with respect to neighborhood B, then stop. 4.Otherwise, move to the first improving solution in neighborhood B and go back to step 2.

November 2002 Scene recognition by inexact graph matching31/39 ALIO/EURO Workshop Local search Each local search iteration (neighborhood search and move) has time complexity. Local search improved the solution value for all test problems.

November 2002 Scene recognition by inexact graph matching32/39 ALIO/EURO Workshop Local search: I-7 Local search applied to the best of the 100 first feasible solutions built: 13 regions correctly recognized (out of 28)

November 2002 Scene recognition by inexact graph matching33/39 ALIO/EURO Workshop GRASP heuristic GRASP: m ultistart metaheuristic Each iteration has two phases: construction and local search MaxIterations: maximum number of iterations Repeat for MaxIterations iterations: –Build a feasible solution using the randomized construction algorithm. –Use the local search procedure to improve the solution built. –Update the best solution found.

November 2002 Scene recognition by inexact graph matching34/39 ALIO/EURO Workshop GRASP heuristic Good solutions obtained with a few hundred GRASP iterations. Improved solutions for all test problems Small computation times Problem I-7, GRASP iterations: 216 seconds on a Pentium II 450 MHz

November 2002 Scene recognition by inexact graph matching35/39 ALIO/EURO Workshop GRASP heuristic Solution values for α = 0.9: construction algorithm with MaxTrials = 500, first improving local search strategy, and 200 GRASP iterations Proble m |N1||N1||E1||E1||N2||N2||E2||E2|Constructi on Local search GRASP I I I I

November 2002 Scene recognition by inexact graph matching36/39 ALIO/EURO Workshop GRASP heuristic: I GRASP iterations, with local search starting from the first feasible solution built in the construction phase: 26 regions correctly recognized (out of 28)

November 2002 Scene recognition by inexact graph matching37/39 ALIO/EURO Workshop GRASP heuristic: 1000 iterations Problem I-6 Problem I-7

November 2002 Scene recognition by inexact graph matching38/39 ALIO/EURO Workshop Current and future current work Improved construction algorithm incorporating a greedy (randomized) insertion criterion Improvement of the GRASP heuristic using path-relinking Parallelization on a 64-node Linux cluster using the MPI library for communication Comparison with other approaches in the literature, in particular with genetic algorithms Application to the recognition of facial features

November 2002 Scene recognition by inexact graph matching39/39 ALIO/EURO Workshop Slides and publications Slides of this talk can be downloaded from: rio/~celso/talks Papers about the randomized construction heuristic with local search for scene recognition (shortly) available at: rio.br/~celso/publicacoes (also available: other papers about GRASP, parallelization, and path-relinking)