1 The Odyssey of: Shape Matching with Ordered Boundary Point Shape Contexts Using a Least Cost Diagonal Method By Dr. Carl E Abrams Jan 20th, 2007

2 Odyssey: a long wandering and eventful journey Or If we knew what we were doing, it wouldn't be called research, would it? – A Einstein

3 Agenda Topic Selection The Eventful Journey –Elation –Mild Desperation –Elation –Beyond Desperation –Relief (Success) Discussion / Lessons Learned

4 Topic Selection or… First Idea: Requirements Engineering and Business Process Modeling I despised the topic area

5 Topic Selection or…Roots Identification of Automotive Vehicles Using Semantic Feature Extraction (Dec 2004) Elation!

6 Dissertation Timeline Carl E. Abrams 9/031/041/051/06 Complete Draft Proposal Advisor Selection First Paper at Pace Day Draft Idea Paper Committee Formation Complete Dissertation Proposal Draft Chap 1&2 Draft Chap 3&4 Final Draft Chap 1-3 Draft Chap 4&5 Final Manuscript and Paper Defense Now!

7 The Eventful Journey

8 Semantic Geometric Features: A Preliminary Investigation of Automobile Identification Carl E. Abrams Sung-Hyuk Cha, Michael Gargano, and Charles Tappert

9 Agenda Overview of the Problem The Experiments Results Going Forward

10 Overview Object recognition remains a hard problem The human mind uses shapes to recognize objects Can semantic features defined by their shapes be more effective in the recognition and identification of objects?

11 The Experiments 10 test images of cars Directly form the manufactures websites Images were restricted to side views of the cars taken from 90 degrees All 2005 models Feature vectors calculated/measured from the images

12 The Vehicles

13 Experiments used Euclidean Distance as the Measure the xi and ti are measurements from two different vehicles

14 Experiments used Euclidean Distance as the Measure (x1,y1) (x2,y2) c a b c = (a 2 +b 2 ) 1/2 c = ((x1-x2) 2 +(y1-y2) 2 ) 1/2

15 Manufacturers Specifications First Experiment

16 Boundary Description using Rays Second Experiment

17 Semantic Features Third Experiment

18 Challenge: Determine the qualitative ability of the feature vectors to separate the vehicles Within each experiment compute the distance of each vehicle from all the others Evenly divide the measures into 5 bins Observe the distribution of the measures

19 The Results

20 Distance Matrix – Semantic Features / , Honda Civic Honda Accord Mazda 3Mazda 6Porsche Carerra Toyota Camry Toyota Celica Toyota Corolla Toyota Echo VW Passat Honda Civic Honda Accord Mazda 3 Mazda 6 Porsche Carerra Toyota Camry Toyota Celica Toyota Corolla Toyota Echo VW Passat

21 End of First Dead End Mild Desperation

22 Dissertation Timeline Carl E. Abrams 9/031/041/051/06 Complete Draft Proposal Advisor Selection First Paper at Pace Day Draft Idea Paper Committee Formation Complete Dissertation Proposal Draft Chap 1&2 Draft Chap 3&4 Final Draft Chap 1-3 Draft Chap 4&5 Final Manuscript and Paper Defense Now! Mild Desperation sets in

23 Shape Contexts Shape Contexts are a novel shape descriptor introduced be Belongie[1] Describes a shape by quantifying each point on the boundary of a shape by its relationship to all the on the boundary points on the shape Compares shapes by comparing shape contexts [1] S. Belongie, "Image segmentation and shape matching for object recognition," vol. PhD, 2000, pp. 60.

24 Shape Contexts-Constructing

25 Shape Contexts-Constructing

26 Shape Contexts-Comparing Unknown Known CHI^2 Test where K= # of bins, g is unknown histogram and h is the known histogram

27 Shape Contexts-Properties: Translation

28 Shape Contexts-Properties: Scale

29 Shape Contexts-Properties: Rotation

30 Progress Report The Role of Semantic Features in Automobile Identification as of December 10 th, 2005 Carl E. Abrams

31 Agenda Review of Topic and Approach (Elevator Pitch) Summary Results to Date Next Steps

32 Review of Topic and Approach Develop an image segmentation and feature extraction / classification scheme for automobiles that employs the shapes of “semantic” parts and their geometric relationships. –Semantic features are the shapes of : windows, doors, front and rear quarter panels.

33 Review of Topic and Approach Approach: Develop a test database of vehicles by collecting side images Develop/beg borrow or steal/software to interactively extract the shape and geometric information from the images Work through all the classification test database In parallel continue building the master database –Develop, test and compare segmentation / extraction method for semantic shapes

34 Preliminary Results-Test DB As of Oct 15: All Ford models back to 1990 ~ 50 images As of Nov 12: Acura, Audi, Chrylser, Dodge, Ford, Honda, Mercury, Nissan, Pontiac, Saab, Saturn, Toyota, Volvo, VW models back to 1990 ~125 images

35 Preliminary Results-Image Segmentation Software Modified CTMRedit: a matlab GUI for viewing, segmenting, and interpolating CT and MRI Images –Written by Mark Hasegawa-Johnson and Jul Cha Simplified GUI and added capability to store shapes specific to vehicle identification

36 Preliminary Results-Image Segmentation Software Examples

37 Preliminary Results Create a feature vector that allows the comparison of one shape to another: Vector_Known( ws1,ws2,ws3,ws4,ds1,ds2,body shape ) Vector_Unknown( ws1,ws2,ws3,ws4,ds1,ds2,body shape ) Feature vector depends on shape descriptor in this case “Shape Contexts”

38 Preliminary Results – Shape Contexts How to effectively describe a shape? r o

39 Preliminary Results – Shape Contexts How to effectively describe a shape? r o r (5 bins) O (12 Bins)

40 Preliminary Results – Shape Contexts How to effectively describe a shape? Develop the Shape Context histograms for every point on the shape

41 Preliminary Results – Distance Between Shape Contexts How to effectively describe a shape? Unknown Known CHI^2 Test where K= # of bins, g is unknown histogram and h is the known histogram Each shape has 128 points, creates a 128x128 cost matrix

42 Preliminary Results – Distance Between Shape Contexts What is the best fit (minimum cost) to align all the points? The Assignment Problem – Hungarian Method for bi-partite matching problem We will be working with the following problem: assign n = 9 candidates to n=9 jobs to minimize the total salary cost paid by the department. The individual salaries of each candidate at each job position depend on their qualification and are given by the cost matrix (in $ per hour): Sa m JillJohnLizAnnLoisPeteAlexHerb Administrator Secretary to the Chair Undergraduate Secretary Graduate Secretary Financial Clerk Secretary Web designer Receptionist Typist If we start with the position of Administrator and assign it to Alex (he gets the minimal salary for this position), then we assign the position of Secretary to Chair to Lois (he gets the minimal salary for this position), and so on, up to the position of Typist, then the assignment is given by the assignment matrix:

43 Preliminary Results Experimental Setup – Run 5 test cars against known database of 50 cars Test cars re-segmented form known database Plot out matches based on Euclidean Distance Repeat by adding more cars of a different manufacturer to known DB

44 Preliminary Results Unknown: 2003FordMustang2DGT

45 Preliminary Results Unknown: 2003FordMustang2DGT-Volvos Added to Known DB

46 Preliminary Results Unknown: 2004FordFocus4DZX5.dh1

47 Preliminary Results Unknown: 2004FordFocus4DZX5.dh1-Volvos Added to Known DB

48 Preliminary Results Unknown: 2004FordTaurus4DSES.dh1

49 Preliminary Results Unknown: 2004FordTaurus4DSES.dh1-Volvos Added to Known DB

50 Preliminary Results Unknown: 2005FordMustangCoupe2DV6Deluxe.dh1

51 Preliminary Results Unknown: 2005FordMustangCoupe2DV6Deluxe.dh1-Volvos Added to Known DB

52 Preliminary Results Unknown: 2005FordTaurusSE4D.ws2

53 Preliminary Results Unknown: 2005FordTaurusSE4D.ws2-Vovlos Added to Known DB

54 Known vs Known (as of Nov 12 th )

55 Preliminary Results Unknown:2005FordFocusZX32D vs Only 2D Vehicles

56 Next Steps Preliminary Conclusion: Creating a feature vector composed of shape descriptors using shape contexts demonstrates a high discriminatory power in vehicle identification. ELATION!

57 Dissertation Timeline Carl E. Abrams 9/031/041/051/06 Complete Draft Proposal Advisor Selection First Paper at Pace Day Draft Idea Paper Committee Formation Complete Dissertation Proposal Draft Chap 1&2 Draft Chap 3&4 Final Draft Chap 1-3 Draft Chap 4&5 Final Manuscript and Paper Defense Now! Elation

58 Desperation! This is not new it is just an application of what is already known!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Now what do I do?????

59 Dissertation Timeline Carl E. Abrams 9/031/041/051/06 Complete Draft Proposal Advisor Selection First Paper at Pace Day Draft Idea Paper Committee Formation Complete Dissertation Proposal Draft Chap 1&2 Draft Chap 3&4 Final Draft Chap 1-3 Draft Chap 4&5 Final Manuscript and Paper Defense Now! Desperation --- but never give in

60 Next Steps Preliminary Conclusion: Creating a feature vector composed of shape descriptors using shape contexts demonstrates a high discriminatory power in vehicle identification. Research Exploration: How well can shape contexts perform in the role of image segmentation??? 1.Develop semantic extraction using Shape Contexts 2.Test segmentation / extraction method for semantic shapes 3.Compare segmentation / extraction method for semantic shapes to other methods

61 Shape Extraction using Shape Contexts 1.From known database develop invariant models for window shape 1,2 and 3, door shapes and body shape using clustering analysis 2.For these known shapes and their contexts, develop shape histograms for neighborhood of high information context points on shape using entropy measures 3.Scan across an edge detected image using a window, computing the local neighborhood in the window 4.Compare the local neighborhood to the known shape contexts local neighborhood and find best fits 5.Focus on best fits and match known shape to points in the best fit area

62 For known shapes and their contexts, develop shape histograms for neighborhood of high information context points on shape using entropy measures Use entropy calculation to determine the local set of points to use as reference on known shape context. ( Note: Shape Context is intentionally biased toward close in points ) Use a subset of points on the know shape, slide around the shape calculating entropy select the highest point subset (“The Entropy Strategy for Shape Recognition”, D Geman, Proc IEEE-IMS workshop on Information Theory and Statistics, Alexandria, VA, October, 1994

63 Shape Entropy Example (“The Entropy Strategy for Shape Recognition”, D Geman, Proc IEEE-IMS workshop on Information Theory and Statistics, Alexandria, VA, October, discrete samples from the curve 12 flat line segments – 0 deg turning angle 4 corners - 90 deg turning angle Prob 0 deg =3/4 and Prob 90 deg = ¼ Slide window

64 Segmentation using Local Shape Contexts by looking for similar local shape histograms Select the closet match and then fit the invariant model to the available points in the image by scaling for best fit

65 Dissertation Timeline Carl E. Abrams 9/031/041/051/06 Complete Draft Proposal Advisor Selection First Paper at Pace Day Draft Idea Paper Committee Formation Complete Dissertation Proposal Draft Chap 1&2 Draft Chap 3&4 Final Draft Chap 1-3 Draft Chap 4&5 Final Manuscript and Paper Defense Now! Despair -- What cap and gown?

66 Trying to regain the will to live And then: ……. Beyond Desperation

67 Introduction and Overview The research focuses on and extends the work done on a new shape descriptor called “Shape Contexts” Constraining shapes to be continuous outlines (no holes) it is proved using graph theory and then confirmed through experiments that the original matching method which is modeled on the “Assignment Problem” is subject to degenerate shape description A simpler matching algorithm called the “Least Cost Diagonal” which utilizes the physical relationship of points on a boundary is introduced and compared to the original method The efficacy of the Least Cost Diagonal method is confirmed through experiments to the original method in a Nearest Neighbor comparison using a previously classified pottery database

68 Shape Contexts-Assumptions Comparing two shapes was modeled after “The Assignment Problem” Assumption: The boundary points are in no particular order[2] [2] T. B. Sebastian, P. N. Klein, and B. B. Kimia, "Recognition of shapes by editing their shock graphs," Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 26, pp , 2004.

69 Shape Contexts-The Assignment Problem What is the best fit (minimum cost) to align all the points? The Assignment Problem – Hungarian Method for bi-partite matching problem We will be working with the following problem: assign n = 9 candidates to n=9 jobs to minimize the total salary cost paid by the department. The individual salaries of each candidate at each job position depend on their qualification and are given by the cost matrix (in $ per hour): SamJillJohnLizAnnLoisPeteAlexHerb Administrator Secretary to the Chair Undergraduate Secretary Graduate Secretary Financial Clerk Secretary Web designer Receptionist Typist If we start with the position of Administrator and assign it to Alex (he gets the minimal salary for this position), then we assign the position of Secretary to Chair to Lois (he gets the minimal salary for this position), and so on, up to the position of Typist, then the assignment is given by the assignment matrix:

70 Shape Contexts-Observations Using the assignment problem model means that the neighborhood-ness of the boundary points is not used. The potential of degenerate shape description exists ( i.e. very different shapes that use the same boundary points but in which the points occurs in different orders will be computed to be the same)

71 Shape Contexts-Degenerate Behavior

72 Shape Contexts-Degenerate Behavior Formalization Graph Theoretic Proof with Examples Step 1: Using the definition of a Hamiltonian Cycle prove using Jordan's Theorem that each different Hamiltonian Cycle of a set of vertices is a distinct shape corresponding to that Hamiltonian Cycle Step 2: Prove that the Hamiltonian Cycles of a particular graph which have been shown to represent distinct shapes are computed to be identical by the shape context method developed by Belongie (solving point assignment matching using the Assignment Problem as the model) Jordan Curve Theorem If J is a simple closed curve in R 2, then the Jordan curve theorem, also called the Jordan-Brouwer theorem (Spanier 1966) states that R2 J -has two components (an "inside" and "outside"), with J the boundary of each. The Jordan curve theorem is a standard result in algebraic topology with a rich history. A complete proof can be found in Hatcher (2002, p. 169), or in classic texts such as Spanier (1966). Recently, a proof checker was used by a Japanese-Polish team to create a "computer-checked" proof of the theorem (Grabowski 2005).

73 Shape Contexts-Proof of Degenerate Behavior Step 1 [4] O. Veblen, "Theory on plane curves in non-metrical analysis situs," Transactions of the American Mathematical Society vol. 6, pp , [4]

74 Shape Contexts-Proof of Degenerate Behavior Step 1 Example

75 Shape Contexts-Proof of Degenerate Behavior Step 2

76 Shape Contexts-Proof of Degenerate Behavior Step 2 Example

77 The Least Cost Diagonal Constrains the domain of shapes to outlines (no holes) Constrains matches to made up of boundary ordered points Avoids degenerate shape matching Runs much more quickly than the original method Gives an indication of degree of rotation of one shape to another

78 The Least Cost Diagonal-How it works 180 degrees 270 degrees

79 Shape Contexts-Degenerate Behavior

80 Experimental Results Basic Shape Context Experiments Classification Performance Timing and Boundary Point dependence

81 Experimental Results Basic Shape Context Experiments –Translation, Scale and Rotation Invariance –Shape Matching with the Least Cost Diagonal –Demonstration of Degenerate Shape Description/Matching

82 Experimental Results Classification Performance –Use a NN analysis on a known database which had already been grouped into classes[5]. Compare the Assignment Model Matching to the Least Cost Diagonal at varying number of boundary points –Experiments performed: One pottery class against each other class One pottery class against groups of two classes One pottery class against groups of three classes [5] RG. Bishop, S. Cha, and C.C. Tappert, "Identification of Pottery Shapes and Schools Using Image Retrieval Techniques," Proc. MCSCE, CISST, Las Vegas, NV, June What happened to the cars?????

83 Experimental Results Timing and Boundary Point dependence –Timing: From all the classification runs plus additional runs at higher number of boundary points verify the order of computational complexity –Point dependence: Run the classification on particular classes that did not perform well at additional boundary points(150 and 250) to demonstrate that a relationship between the number of points and performance exists

84 Classification Experimental Results Experimental Procedure: Compute % correctly classified for various numbers of boundary points (3,4,5,10,20,100)[6,7] Class 1 ShapesClass 2 Shapes SA1SA2SA3SA4SA5SA6SA7SB1SB2SB3SB4SB5 Class 1 Shapes SA1 SA2 SA3 SA4 SA5 SA6 SA7 Hungarian or LCD Costs [6] S. Belongie, J. Malik, and J. Puzicha, "Matching Shapes," presented at International Conference on Computer Vision (ICCV'01) Volume [7] K. Mikolajczyk and C. Schmid, "A performance evaluation of local descriptors," Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 27, pp , 2005.

85 Classification Experimental Results Example: Class 1 vs Class 2 and Class 3

86 Classification Experimental Results: Timing Averaged the timing for each set of boundary points-60 runs per boundary point. (Added six additional runs to extend the timings up to 250 points) Fitted a polynomial to the data to derive the computational order of complexity

87 Classification Experimental Results: Timing Curve Fitting

88 Classification Experimental Results: Boundary Points Class 1 Class 5

89 Classification Experimental Results: Boundary Points

90 Classification Experimental Results: Boundary Points Class 3vsClass 11,12, points points3points4points5points10points20points 100points150points250points Number of Points % Correct Diagonal Hungarian Class 3vs Class 11,12, points3points4points5points10points20points 100points Number of Points % Correct Diagonal Hungarian

91 Summary The research has confirmed the properties of the Shape Context and proved that when the domain of the shape matching is limited to outlines, erroneous shape matching based on degenerate shape descriptors can occur A new matching method has been introduced( Least Cost Diagonal) and its efficacy verified against real images and compared to the original method developed by Belongie Further work was identified: Is there a relationship between the number boundary points to the accuracy of shape matching? Are there better distance measures for shape context histograms? Are there better quantizing schemes for shape contexts?

92 Dissertation Timeline Carl E. Abrams 9/031/041/051/06 Complete Draft Proposal Advisor Selection First Paper at Pace Day Draft Idea Paper Committee Formation Complete Dissertation Proposal Draft Chap 1&2 Draft Chap 3&4 Final Draft Chap 1-3 Draft Chap 4&5 Final Manuscript and Paper Defense Now! Defense was on July 14 th, 2006

93 Post-Defense Work Apply the new method to my existing car database –Compare to “Hungarian Method” for accuracy and speed Make the dissertation longer

94 Discussion/Lessons Learned Really have an interest in your topic!! Get comfortable with uncertainty See your faculty advisors early and often Work on your project ever day!!! Don’t throw anything away Make it at least 100 pages long without the references “The way to get good ideas is to get lots of ideas, and throw the bad ones away” –Dr. Linus Pauling (The trick is recognizing the bad ones)! Dr. Carl Abrams