1. Content-Based Image Retrieval: Reading One’s Mind and Making People Share Oral defense by Sia Ka Cheung Supervisor: Prof. Irwin King 31 July 2003

2. 2 Flow of Presentation Content-Based Image Retrieval Reading One’s Mind  Relevance Feedback Based on Parameter Estimation of Target Distribution Making People Share  P2P Information Retrieval  DIStributed COntent-based Visual Information Retrieval

3. 31 July 20033 Content-Based Image Retrieval How to represent and retrieve images?  By annotation (manual) Text retrieval Semantic level (good for picture with people, architectures)  By the content (automatic) Color, texture, shape Vague description of picture (good for pictures of scenery and with pattern and texture)

4. 31 July 20034 Feature Extraction R B G

5. 31 July 20035 Indexing and Retrieval Images are represented as high dimensional data points (feature vector) Similar images are “close” in the feature vector space  Euclidean distance is used

6. 31 July 20036 Typical Flow of CBIR Images Database Index and Storage Feature Extraction Query Result Query Image Lookup

7. 31 July 20037 Reading One’s Mind Relevance Feedback

8. 31 July 20038 Why Relevance Feedback? The gap between semantic meaning and low-level feature  the retrieved results are not good enough Images Database Index and Storage Feature Extraction Result Query Image Lookup Better Result Feedback Better Result Feedback

9. 1 st iteration User Feedback Display 2 nd iteration Display User Feedback Estimation & Display selection Feedback to system

10. 31 July 200310 Problem Statement Assumption: images of the same semantic meaning/category form a cluster in feature vector space Given a set of positive examples, learn user’s preference and find better result in the next iteration

11. 31 July 200311 Former Approaches Multimedia Analysis and Retrieval System (MARS)  IEEE Trans CSVT 1998  Weight updating, modification of distance function Pic-Hunter  IEEE Trans IP 2000  Probability based, updated by Bayes’ rule  Maximum Entropy Display

12. 31 July 200312 Comparisons AspectModelDescription Modeling of user’s target MARSWeighted Euclidean distance Pic-HunterProbability associated with each image Our approachUser’s target data point follow Gaussian distribution Learning method MARSWeight updating, modification of distance function Pic-HunterBayes’ rule Our approachParameter estimation Display selection MARSK-NN neighborhood search Pic-HunterMaximum entropy principle Our approachSimulated maximum entropy principle

13. 31 July 200313 Estimation of Target Distribution Assume the user’s target follows a Gaussian distribution Construct a distribution that best fits the relevant data points into some “specific” region Data points selected as relevant

14. 31 July 200314 Estimation of Target Distribution Assume the user’s target follows a Gaussian distribution Construct a distribution that best fits the relevant data points into some “specific” region Data points selected as relevant

15. 31 July 200315 Estimation of Target Distribution Assume the user’s target follows a Gaussian distribution Construct a distribution that best fits the relevant data points into some “specific” region Data points selected as relevant

16. 31 July 200316 Expectation Function Best fit the relevant data points to medium likelihood region The estimated distribution represents user’s target

17. 31 July 200317 Updating Parameters After each feedback loop, parameters are updated  New estimated mean = mean of relevant data points  New estimated variance  found by differentiation  Iterative approach

18. 31 July 200318 Display Selection Why maximum entropy principle?  K-NN is not a good way to learn user’s preference  The novelty of result set is increased, thus allowing user to browse more from the DB How to use maximum entropy?  PicHunter – Select a subset of images which entropy is maximized.  Our approach – data points inside boundary region (medium likelihood) are selected

19. 31 July 200319 Simulating Maximum Entropy Display Data points around the region of 1.18 δ away from μ are selected Why 1.18?  2P(μ+1.18 δ)=P(μ) Query target cluster center Selected by knn search Selected by Max. Entropy P(μ+1.18 δ) P(μ)

20. 31 July 200320 Experiments Synthetic data forming mixture of Gaussians are generated Feedbacks are generated based on ground truth (class membership of synthetic data) Investigation  Does the estimated parameters converge?  Does it performs better? DimensionNo. of classNo. of data points in each class Range of μRange of δ 450 [-1,1][0.2,0.6] 67050[-1.5,1.5][0.2,0.6] 88550[-1.5,1.5][0.15,0.45]

21. 31 July 200321 Convergence of Estimated Parameters More feedbacks are given, estimated parameters converge to original parameters used to generate mixtures

22. 31 July 200322 Precision-Recall Red – PE Blue – MARS More experiments in later section

23. 31 July 200323 Precision-Recall

24. 31 July 200324 Problems What if user’s target distribution forms several cluster?  Indicated in Qcluster (SIGMOD’03)  Parameters estimation failed because single cluster is the assumption Qcluster solve it by using multi-points query Merge different clusters into one cluster !!

25. 31 July 200325 The Use of Inter-Query Feedback Relevance feedback information given by users in each query process often infer a similar semantic meaning (images under the same category) Feature vector space can be re-organized  Relevant images are moved towards to the estimated target  Similar images no longer span on different clusters Parameters estimation method can be improved

26. 31 July 200326 1 st Stage of SOM Training Large number of data points   SOM is used to reduce data size   Each neuron represent a group of similar images   original feature space is not changed directly

27. 31 July 200327 Procedure of Inter-query Feedback Updating User marked a set of images as relevant or non-relevant in a particular retrieval process The corresponding relevant neurons are moved towards estimated target Where  M ’ R – set of relevant neurons  c – estimated target  α R – learning rate The corresponding non-relevant neurons are moved away from estimated target

28. 31 July 200328 SOM-based Approach Neuron Class 1 Neuron Class 2 Neuron Class 3

29. 31 July 200329 SOM-based Approach After each query process Relevant Neuron Non- Relevant Neuron

30. 31 July 200330 SOM-based Approach Estimated Target

31. 31 July 200331 SOM-based Approach Relevant neurons are moved towards estimated target

32. 31 July 200332 SOM-based Approach

33. 31 July 200333 SOM-based Approach Feature vector space re-organized

34. 31 July 200334 SOM-based Approach After several iterations (users’ queries)

35. 31 July 200335 SOM-based Approach

36. 31 July 200336 SOM-based Approach Similar images cluster together instead of spanning across different clusters in the new, re-organized feature vector space

37. 31 July 200337 Experiments Real data from Corel image collection  4000 images from 40 different categories Feature extraction methods  RGB color moment (9-d)  Grey scale cooccurence matrix (20-d) 80 queries are generated evenly among 40 classes Evaluations  MARS  PE without SOM-based inter-query feedback training  PE with SOM-based inter-query feedback training

38. 31 July 200338 Precision vs Recall

39. 31 July 200339 Conclusion We propose a parameters estimation approach for capturing user’s target as a distribution A display set selection scheme similar to maximum entropy display is used to capture more user’s feedback information A SOM-based inter-query feedback is proposed  Overcome the single cluster assumption of most intra- query feedback approach

40. 31 July 200340 Making People Share DIStributed COntent-based Visual Information Retrieval

41. 31 July 200341 P2P Information Retrieval Images Peer databases Feature Extraction Query Result Query Image Lookup … How to locate relevant images In an efficient manner?

42. 31 July 200342 Contributions Migrate centralized architecture of CBIR to distribution architecture Improve existing query scheme in P2P applications A novel algorithm for efficient information retrieval over P2P  Peer Clustering  Firework Query Model (FQM)

43. 31 July 200343 Existing P2P Architecture Centralized  Napster, SETI (Berkeley), ezPeer (Taiwan)  Easy implementation  Bottleneck, single point failure  Legal problems update query answer transfer

44. 31 July 200344 Existing P2P Architecture Decentralized Unstructured  Gnutella (AOL, Nullsoft), Freenet (Europe)  Self-evolving, robust  Query flooding Peer TCP connection

45. 31 July 200345 Existing P2P Architecture Decentralized Structured  Chord (SIGCOMM’01), CAN(SIGCOMM’01), Tapestry (Berkeley)  Efficient retrieval and robust  Penalty in join and leave Distributed Hash Table (DHT) Peer in the network Files shared by peers CAN model TCP connection

46. 31 July 200346 DISCOVIR Approach Decentralized Quasi-structured  DISCOVIR (CUHK)  Self-organized, clustered, efficient retrieval attractive connections random connections

47. 31 July 200347 Design Goal and Algorithms used in DISCOVIR Peers sharing “similar” images are interconnected Reduce flooding of query message Construction of self-organizing network 1. Signatures calculation 2. Neighborhood discovery 3. Attractive connections establishment Content-based query routing 1. Route selection 2. Shared file lookup

48. 31 July 200348 Construction of Self-Organizing Network Signatures calculation Signatures discovery of neighborhoods Comparison of signatures Attractive connection establishment

49. 31 July 200349 Signatures Calculation Feature vector space

50. 31 July 200350 Signatures Calculation Peer A Peer B Centroid of peer

51. 31 July 200351 Signatures Calculation Peer A A1A1 A2A2 A3A3 Peer B B1B1 B2B2 B3B3 Centriod of sub-cluster Centroid of peer

52. 31 July 200352 Signatures Calculation Peer A A1A1 A2A2 A3A3 Peer B B1B1 B2B2 B3B3 Centriod of sub-cluster Centroid of peer

53. 9 1 7 8 5 3 10 11 2 4 6 New attractive connection Random connection Existing clustered P2P network (1.5,1.2) (1.3,2.4) (1.6,1.8) (5.2,8.5) (5.6,8.8) (4.9,9.7) (4.7,9.3) (2.9,6.5) (2.7,6.0) (2.6,5.9) (2.0,5.8) Random connection Attractive connection (x,y) Signature value

54. 31 July 200354 Content-based Query Routing Incoming query Signature value left? Similarity < threshold? Forward to random link if not forwarded before Forward to attractive link End N Y Y N

55. 31 July 200355 Content-based Query Routing Similarity < threshold Similarity > threshold Random connection Attractive connection

56. 31 July 200356 Comparison of Content-based Query Routing and Address-based Query Routing AspectSchemeDescription ApplicationABRInternet Protocol (IP), Domain Name System (DNS) CBRWide Area Information System (WAIS), DISCOVIR ProblemABRWe know where to go, but not the path CBRWe don’t know where to go, nor the path EmphasisABRCorrectness, speed CBRRelevance of result retrieved GoalABRAvoid unnecessary traffic CBRAvoid unnecessary traffic

57. 31 July 200357 Experiments Dataset  RBG color moment, 9-d 10000 images from Corel database, 100 classes  Synthetic data, 9-d, 10000 points, 100 classes Operation  Distribute data-points into peers (1 class per peer)  Simulate network setup and query (averaged 50 queries) Investigation  Scalability (against number of peers)  Property (against TTL of query message)  Data resolution (different number of signatures per peer)

58. 31 July 200358 Network Model Small world characteristic, power-law distribution  Few peers are connected with many peers  Many peers are connected with few peers

59. 31 July 200359 Performance Metrics Relevance of retrieved result  Recall Number of query traffic generated  Query scope Effectiveness of query routing scheme  Query efficiency Number of retrieved relevant result Total number of relevant result Number peers visited by query message Total number of peers

60. 31 July 200360 Recall vs Peers

61. 31 July 200361 Recall vs TTL

62. 31 July 200362 Query Scope vs Peers

63. 31 July 200363 Query Scope vs TTL

64. 31 July 200364 Query Efficiency vs Peers

65. 31 July 200365 Query Efficiency vs TTL

66. 31 July 200366 Difference Between Synthetic Data and Real Data Inter-cluster distanceMean of variances realsyntheticrealsynthetic max1.44671.8207max0.01530.0128 min0.02720.3556min0.00060.0042 avg0.32981.1159avg0.01120.0086

67. 31 July 200367 Effects of Data Resolution Assign 2-4 classes of image to each peer  High data resolution (use 3 signatures)  Low data resolution (use 1 signatures)

68. 31 July 200368 Conclusion CBIR is migrated from centralized server approach to peer-to-peer architecture Efficient retrieval is achieved by  Constructing a self-organizing network  Content-based query routing Scalability, property and effects on data resolution are investigated Query efficiency are at least doubled under the proposed architecture

69. Questions and Answers

70. 31 July 200370 Precision, Recall, Novelty Precision  range [0,1] Recall  range [0,1] Set of retrieved result Set of relevant result Set of retrieved known to user before hand

71. 31 July 200371 Update Equation, Learning Rate Update equation of non-relevant neurons Update equation of neighboring neurons

72. Original SOM MModified SOM M’ Modified model vector space data-size reduced to |M’| Original feature vector space data-size is |I| 1-1 mapping function – f -1 Table lookup Retrieval process with SOM to capture feedback information

73. 31 July 200373 Multiple Clusters Version

74. 31 July 200374 DISCOVIR – System Architecture Built on LimeWire, Java-based Plug-in architecture for feature extraction module Query by example, sketch, thumbnail previewing Connection Manager Packet Router Plug-in Manager HTTP Agent Feature Extractor Image Indexer DISCOVIR Network Shared Collection DISCOVIR Core DISCOVIR User Interface WWW Image Manager

75. 31 July 200375 DISCOVIR – Screen Capture

76. DISCOVIR-Protocol Modification Minimum SpeedFeature Name0Feature Vector0 Image Query 0x80 0 12 … Number of HitsPortIP AddressSpeedResult Set Image Query Hit 0x81 0 12 Servant Identifier 3 671011…nn+16 File IndexFile SizeFile NameThumbnail information, similarity0 034 0 78… Minimum SpeedDISCOVIRSIGNATURE 0 0 DISCOVIR Signature Query 0x80 0 12 20 Number of HitsPortIP AddressSpeedResult Set DISCOVIR Signature Query Hit 0x81 0 12 Servant Identifier 3 671011…nn+16 Dummy Feature Extraction nameSignature value0 034 0 78…

77. 31 July 200377 Query Message Utilization

78. 31 July 200378 Average Reply Path Length