1. Competence Centre on Information Extraction and Image Understanding for Earth Observation 29th March 2007 Category - based Semantic Search Engine 1 Mihai Costache Télécom Paris

2. Competence Centre on Information Extraction and Image Understanding for Earth Observation 29th March 2007 Category - based Semantic Search Engine 2 Outline  General problem description  Relevance Feedback in Satellite CBIR  SVM based search engine  Results  Discussions

3. Competence Centre on Information Extraction and Image Understanding for Earth Observation 29th March 2007 Category - based Semantic Search Engine 3 Problem statement  Context  Huge volume of Earth Observation (EO) images  High volume of information due to the increasing resolution (0.6 – 2.5 metesr/pixel)  Need to perform  Automatic indexation and annotation of satellite image archives (Content Based Image Retrieval, Relevance Feedback)  Detection and recognition of structures and objects within the satellite images (classification)  Reasearch areas  Information theory and statistics  Machine learning and Bayesian Inference – Incremental learning  Discriminative learning

4. Competence Centre on Information Extraction and Image Understanding for Earth Observation 29th March 2007 Category - based Semantic Search Engine 4 Satellite Content Based Image Retrieval System  CBIR attempts to automate the indexing process of images in the database  Descriptions based on inherent properties of images (texture, color, shape etc.)  Problems:  Gap between low-level descriptors and semantic image content  Different image semantic interpretations : Same image but two persons Same person but different moments of time  Thus small retrieval precision  Solution: incorporating human perception subjectivity Relevance Feedback

5. Competence Centre on Information Extraction and Image Understanding for Earth Observation 29th March 2007 Category - based Semantic Search Engine 5 RF in SCBIR Systems Relevance Feedback  Finding relevant information in satellite CBIR by human interaction  The retrieval process is improved by human decision  The user scores the retrieved documents as relevant / irrelevant  Annotations used in the learning process  Structure of a satellite indexation system with RF Primitive feature extraction Classification Relevance Feedback (RF) Quadrature Mirror Filters (QMF) SVM RF-SVM Objective Subjective

6. Competence Centre on Information Extraction and Image Understanding for Earth Observation 29th March 2007 Category - based Semantic Search Engine 6 Structure of a RF system  Human-machine communication in RF Subjective Objective

7. Competence Centre on Information Extraction and Image Understanding for Earth Observation 29th March 2007 Category - based Semantic Search Engine 7 RF Algorithms  Query point movement - Try to find an optimal vector model for retrieval purposes  Rocchio’s Algorithm J. J. Rocchio, Relevance feedback information retrieval, 1971. Viper, MARS D. McG. Squire et al., Content-based query of image databases: inspirations from text retrieval, 2000. Y. Ruiu et al., Content-based Image Retrieval with Relevance Feedback in MARS, 1997.  Re-weighting algorithms - Try to find an optimal similarity measure  MindReader Y. Ishikawa et al., Mindreader: Query Databases Through Multiple Examples,1998.  RF – Support Vector Machine (SVM) S. Thong and Chang E., Support vector Machine Active Learning for Image Retrieval, 2001. M. Costache, H. Maitre and M. Datcu,Categorization based Relevance Feedback Search Engine for Earth Observation Images Repositories, 2006.

8. Competence Centre on Information Extraction and Image Understanding for Earth Observation 29th March 2007 Category - based Semantic Search Engine 8 Important points in RF  Feature selection  Texture, shape, color etc.  Learning scheme  Bayesian, Kernel learning methods, Neural Networks, Decision Trees  Scalability  In terms of storage space and query processing times  Type of search:  Category search  Target search

9. Competence Centre on Information Extraction and Image Understanding for Earth Observation 29th March 2007 Category - based Semantic Search Engine 9 Proposed search engine  Search engine concept based on RF algorithm  Field of application: Satellite CBIR  Search tool: Support Vector Machines (SVM)  Bayesian inference used to infer data models  Hierarchy of information Data driven User driven Learing/Unlearning

10. Competence Centre on Information Extraction and Image Understanding for Earth Observation 29th March 2007 Category - based Semantic Search Engine 10 Support Vector Machine  Linear separation case  Labeled data training set  Find a separation surface  Decision function f = sign(g(x))  d + = distance from g to closest {+1}  d - = distance from g to closest {-1}  Margin area = d + +d - =  Find a separating hyperplane with largest margin xixi xjxj g > 0 g < 0 g = 0 margin area SV + SV - Most Relevant (MR) Most Ambiguous (MA)

11. Competence Centre on Information Extraction and Image Understanding for Earth Observation 29th March 2007 Category - based Semantic Search Engine 11 SVM based Search Engine  Algorithm  Query selection  Ranking  User’s relevance: {+1,-1}  Retrieve the Most Ambiguous (MA) images (g~0)  Go to 3  Return the Most Relevant images (MR) (max g)  Model data (GMM) inference – generation of categories  No of Gaussian components K obtained by Minimum Description Length (MDL)  Parameters estimation by Expectation and Maximization (EM) Memory

12. Competence Centre on Information Extraction and Image Understanding for Earth Observation 29th March 2007 Category - based Semantic Search Engine 12 System memory  Hierarchy of information  Classes Λ  Categories C  SV sets  Adding memory  Queries sessions for images within the database  Semantical annotation of the SVM generated classes  Categories generation  Save the set of Support Vectors (SV)  Algorithm  Query example  Compute the likelihod based on the saved data models  Choose the set of SV for which the likelihood is maximized  Start RF with the chosen SV set

13. Competence Centre on Information Extraction and Image Understanding for Earth Observation 29th March 2007 Category - based Semantic Search Engine 13  D Query image  Θ Search engine  Λ  C  U airports,……, village,……….. Hierarchy construction

14. Competence Centre on Information Extraction and Image Understanding for Earth Observation 29th March 2007 Category - based Semantic Search Engine 14 RF quality evaluation  Measure the capability of retrieving relevant data  Relevance has to be experienced  Relevance is personal & situational  Precision – Recall graphs  B – retrieved imagesA – relevant images  Recall: fraction of the relevant images which have been retrieved R = |B∩A|/ |B |  Precision: fraction of the retrieved images which are relevant P = |B∩A|/ |A|  Mean Precison  Combines precision, relevance ranking and overall recall

15. Competence Centre on Information Extraction and Image Understanding for Earth Observation 29th March 2007 Category - based Semantic Search Engine 15 Satellite database  SPOT5 ©CNES scenes 3000x3000 pixels  Resolution 5m/pixel  Cropped 64x64 pixels small images  46 scenes used to create an 11 classes database  100 examples per class  Primitive features: QMF coefficients (11)  Gaussian kernel:  8 images per RF step  15 RF steps  Evaluation of the top 60 MP images  Clouds  Sea  Desert  City  Forest  Fields  Airports  Village  Savanna  Boats  Traffic circle

16. Competence Centre on Information Extraction and Image Understanding for Earth Observation 29th March 2007 Category - based Semantic Search Engine 16 SVM-RF simulation results (1) – primitive feature evaluation  Database composed of 600 SPOT5 images divided in six classes  Used features: Gabor, Haralick, QMF and GMRF  Gaussian Kernel  System evaluation: Precision-Recall graphs

17. Competence Centre on Information Extraction and Image Understanding for Earth Observation 29th March 2007 Category - based Semantic Search Engine 17 SVM-RF simulation results (2) – automatic indexation  Database composed of 1100 SPOT5 images divided in eleven classes  The system has memory  QMF features  System evaluation: Precision-Recall graphs Search engineSpeed of learningMean precision SVM based15 RF steps0.9477 Category based1 RF step0.9368

18. Competence Centre on Information Extraction and Image Understanding for Earth Observation 29th March 2007 Category - based Semantic Search Engine 18 SVM-RF simulation results (3) - classification  Classification (structure detection / recognition)  SPOT5 image over Paris  Cemeteries areas

19. Competence Centre on Information Extraction and Image Understanding for Earth Observation 29th March 2007 Category - based Semantic Search Engine 19 Conclusion  Use of hierarchy of information suitable for EO image interpretation  Semantical annotations of the generated classes  Category Bayesian based learning enhances the search capabilities by speeding the learning process  SVM supports classification of EO scenes.

20. Competence Centre on Information Extraction and Image Understanding for Earth Observation 29th March 2007 Category - based Semantic Search Engine 20 Thank You!