1. A Genetic Algorithm-Based Approach to Content-Based Image Retrieval Bo-Yen Wang( 王博彥 ) 894410003

2. Outline  An Introduction to Content-Based Image Retrieval  Genetic Algorithm: Primary  Example of Applying Genetic Algorithm to CBIR  Conclusion

3. What is Content-Based Image Retrieval  Inspired from CSI (Crime Scene Investigation)

4. Conventional Retrieval Method  Text-based search: Search images with keywords  Manual tagging images  Limitations  Maintain huge image databases  Lack of languages independent  Inconsistent between keyword and human perception

5. Content Based Approach  Versatile content within images  Various content feature  Color  Texture  Local Shape

6. Content Based Image Retrieval  Typical System Feature Extraction Attribute Comparison using distance function f(x) Query Processor outputs the list of similar images Similar Images I4, I8, … Ik F(Iq) Color histogram Texture Description Shape details Iq - Query Image I1, I2,.. –Image Files

7. Dataflow of Content-Based Retrieval Active Picture Set Data Reference source: Smeulders et al.“Content-Based Image Retrieval at the End of the Early Years” Annotate And Interact Compute Local Property Compute Object Feature Store In I-File Store In F-File Compute Similarity Annotate And Interact Compute Local Property Selection Query Picture Store In Session Display And Feedback Compute Object Feature Query

8. Sample CBIR architecture Feature Extraction Indexing Mechanism Database Handler Database Insert/Delete Images Search For Images User System Administrator

9. Challenges of CBIR  Sensory Gap  The sensory gap is the gap between the object in the world and the information in a (computational) description derived from a recording of that scene.  Semantic Gap  The semantic gap is the lack of coincidence between the information that one can extract from the visual data and the interpretation that the same data have for a user in a given situation. Reference source: Smeulders et al.“Content-Based Image Retrieval at the End of the Early Years”

10. What visual properties do these images of tomatoes all have in common?

11. What is this? tomato?, setting sun?, clown’s nose?….

12. Semantic Gap “democracy” SEMANTIC GAP

13. Semantic Gap or Sensory Gap

14. Bridging Semantic Gap  A critical point in the advancement of content-based retrieval is the semantic gap, where the meaning of an image is rarely self-evident.  The aim of content-based retrieval systems must be to provide maximum support in bridging the semantic gap between the simplicity of available visual features and the richness of the user semantics. Reference source: Smeulders et al.“Content-Based Image Retrieval at the End of the Early Years”

15. Bridging Semantic Gap  Imagery Features and Similarity Measure  Select effective imagery features  Subjective experiments  Image categorization

16. Another approach: Relevance Feedback  Limitations of the computer-centric approach, recent research focus in CBIR has moved to an interactive mechanism that involves a human as part of the retrieval process  Human and computer system interacted to refine high- level queries to representations based on low-level features Reference source: Yong Rui et al.“Relevance Feedback: A Power Tool for Interactive Content-Based Image Retrieval” ”

17. Dataflow of Relevance Feedback Images Query Image Result Database Index and Storage Refined Result Refined Result ……… Feature Extraction Query Feedback

18. Example of Relevance Feedback 1 st Round User Feedback Display 2 nd Round Display User Feedback Similarity Measuring Feedback to system

19. Genetic Algorithm: Primary  Charles Robert Darwin  Diversity drives changes  Survival from fitness  Population based Optimization

20. Evaluation (fitness) Generation cycle of Genetic Algorithm Genetic operators Population (chromosomes) Mating pool parents offsprings Reproduction (selection) Mates Natural Selection (Evaluated by Fitness Values) individual (chromosomes) Crossover Mutation

21. A Simple Genetic Algorithm Simple Genetic Algorithm() { Initialize population; evaluate population; while(termination criterion not reached) { select solutions for next population(reproduction); perform crossover and mutation; evaluate population; }

22. Major parts of GA  Representation  Genetic operators  Selection method  Objective function

23. Representation of GA  Binary encoding  Real-coded encoding  real-valued vector  Order-based encoding  Permutation problem

24. Selection Methods  Proportionate selection operators  roulette wheel  Tournament selection operators  Ranking selection operators

25. Example of roulette wheel  01101 169 14.4%  11000 576 49.2% 01000 64 5.5%  10011 361 30.9%    Maximize f(s)

26. Crossover Methods  Real valued  Binary valued  Uniform crossover

27. Example of Binary valued Crossover  Single-point(one-point) crossover crossover parent1 parent2 child1 child2

28. Example of Binary valued Crossover  Multi-point crossover parent1 parent2 child1 child2

29. Example of Uniform Crossover individual 1 0 1 1 1 0 0 1 1 0 1 0 individual 2 1 0 1 0 1 1 0 0 1 0 1 parent sample 1 0 1 1 0 0 0 1 1 0 1 0 sample 2 1 0 0 1 1 1 0 0 1 0 1 Random offspring 1 1 1 1 0 1 1 1 1 1 1 1 offspring 2 0 0 1 1 0 0 0 0 0 0 0 children

30. Mutation  Real valued mutation  Binary valued mutation before mutation 01110011010 after mutation01100011010

31. Why GA is robust than traditional search methods  Working with a coding of parameter set  Searching from a population of points, not a single point  using fitness information, not derivatives  probabilistic transition (guided random search)

32. Example of Applying Genetic Algorithm to CBIR  GA-based approach to relevance feedback using LSP  Genetic algorithm (GA) based method used for finding an optimal assignment of similarity criteria to image regions, and incorporated in the relevance feedback mechanism Reference source: Zoran Stejic et al. “Genetic algorithm-based relevance feedback for image retrieval using local similarity patterns”

33. Definition of Local Similarity Patterns (LSP)  The assignment of image features to image regions, which are obtained by uniform partitioning of the image area. Reference source: Zoran Stejic et al. “Genetic algorithm-based relevance feedback for image retrieval using local similarity patterns”

34. Definition of Local Similarity Patterns (LSP) Reference source: Zoran Stejic et al. “Genetic algorithm-based relevance feedback for image retrieval using local similarity patterns”

35. LSP-based image queries  LSP is proposed for computing similarity between the query image and the database images.  Well-defined query consists of the query image together with the intended interpretation (corresponding similarity criteria )  An LSP-based query of the query image that the user selects, and the LSP system automatically infers based on the user’s relevance feedback. Reference source: Zoran Stejic et al. “Genetic algorithm-based relevance feedback for image retrieval using local similarity patterns”

36. Adapting GA for Image Retrieval  Given a user’s query image, a GA is used to automatically generate a LSP, and complete a LSP based query  GA is to realize the feature assignment function to assign image features to the image regions obtained by the uniform partitioning of the image area

37. Operation of GA in LSP  Chromosomes coding  uniformly partitioned into N*N regions  Each gene of a chromosome corresponds to one image region, and stores the type of image feature assigned to that region

38. Chromosomes coding

39. Operation of GA in LSP  Fitness measure  fitness of a chromosome is expressed as the retrieval precision of the corresponding LSP-based query  GA parameters  Selection: roulette-wheel  Crossover: uniform with probability 0.6  Mutation: standard with probability 0.1  Population size : 50  Generations: 250

40. Methods for comparison(1/3)  Conventional method without relevance feedback  Similarity computation with common feature  Color  Shape  Texture  Color-Texture  Color-Shape  Shape-Texture

41. Methods for comparison(2/3)  Relevance feedback with Standard deviation-based method  Variance is computed over all the relevant images, and the weight of a feature is set to be the inverse of the corresponding variance  The smaller the variance, the bigger the weight

42. Methods for comparison(3/3)  Relevance feedback with Genetic Algorithms-based method  Find an optimal assignment of the weights to the image features  Maximize the average rank of the relevant images (Maximizing the retrieval precision)

43. Experiments Results

47. Conclusion  Genetic algorithm-based approach introduced a slow process but acceptable solution in content-based image retrieval  Population-based characteristic of content-based image retrieval drives feasibility of using evolutionary computation.  Limited stochastic process causes complex problem reduced.