Content-Based Image Retrieval: Reading One’s Mind and Making People Share Oral defense by Sia Ka Cheung Supervisor: Prof. Irwin King 31 July 2003

2 Flow of Presentation Content-Based Image Retrieval Reading One’s Mind  Relevance Feedback Based on Parameter Estimation of Target Distribution Making People Share  P2P Information Retrieval  DIStributed COntent-based Visual Information Retrieval

31 July Content-Based Image Retrieval How to represent and retrieve images?  By annotation (manual) Text retrieval Semantic level (good for picture with people, architectures)  By the content (automatic) Color, texture, shape Vague description of picture (good for pictures of scenery and with pattern and texture)

31 July Feature Extraction R B G

31 July Indexing and Retrieval Images are represented as high dimensional data points (feature vector) Similar images are “close” in the feature vector space  Euclidean distance is used

31 July Typical Flow of CBIR Images Database Index and Storage Feature Extraction Query Result Query Image Lookup

31 July Reading One’s Mind Relevance Feedback

31 July Why Relevance Feedback? The gap between semantic meaning and low-level feature  the retrieved results are not good enough Images Database Index and Storage Feature Extraction Result Query Image Lookup Better Result Feedback Better Result Feedback

1 st iteration User Feedback Display 2 nd iteration Display User Feedback Estimation & Display selection Feedback to system

31 July Problem Statement Assumption: images of the same semantic meaning/category form a cluster in feature vector space Given a set of positive examples, learn user’s preference and find better result in the next iteration

31 July Former Approaches Multimedia Analysis and Retrieval System (MARS)  IEEE Trans CSVT 1998  Weight updating, modification of distance function Pic-Hunter  IEEE Trans IP 2000  Probability based, updated by Bayes’ rule  Maximum Entropy Display

31 July Comparisons AspectModelDescription Modeling of user’s target MARSWeighted Euclidean distance Pic-HunterProbability associated with each image Our approachUser’s target data point follow Gaussian distribution Learning method MARSWeight updating, modification of distance function Pic-HunterBayes’ rule Our approachParameter estimation Display selection MARSK-NN neighborhood search Pic-HunterMaximum entropy principle Our approachSimulated maximum entropy principle

31 July Estimation of Target Distribution Assume the user’s target follows a Gaussian distribution Construct a distribution that best fits the relevant data points into some “specific” region Data points selected as relevant

31 July Estimation of Target Distribution Assume the user’s target follows a Gaussian distribution Construct a distribution that best fits the relevant data points into some “specific” region Data points selected as relevant

31 July Estimation of Target Distribution Assume the user’s target follows a Gaussian distribution Construct a distribution that best fits the relevant data points into some “specific” region Data points selected as relevant

31 July Expectation Function Best fit the relevant data points to medium likelihood region The estimated distribution represents user’s target

31 July Updating Parameters After each feedback loop, parameters are updated  New estimated mean = mean of relevant data points  New estimated variance  found by differentiation  Iterative approach

31 July Display Selection Why maximum entropy principle?  K-NN is not a good way to learn user’s preference  The novelty of result set is increased, thus allowing user to browse more from the DB How to use maximum entropy?  PicHunter – Select a subset of images which entropy is maximized.  Our approach – data points inside boundary region (medium likelihood) are selected

31 July Simulating Maximum Entropy Display Data points around the region of 1.18 δ away from μ are selected Why 1.18?  2P(μ+1.18 δ)=P(μ) Query target cluster center Selected by knn search Selected by Max. Entropy P(μ+1.18 δ) P(μ)

31 July Experiments Synthetic data forming mixture of Gaussians are generated Feedbacks are generated based on ground truth (class membership of synthetic data) Investigation  Does the estimated parameters converge?  Does it performs better? DimensionNo. of classNo. of data points in each class Range of μRange of δ 450 [-1,1][0.2,0.6] 67050[-1.5,1.5][0.2,0.6] 88550[-1.5,1.5][0.15,0.45]

31 July Convergence of Estimated Parameters More feedbacks are given, estimated parameters converge to original parameters used to generate mixtures

31 July Precision-Recall Red – PE Blue – MARS More experiments in later section

31 July Precision-Recall

31 July Problems What if user’s target distribution forms several cluster?  Indicated in Qcluster (SIGMOD’03)  Parameters estimation failed because single cluster is the assumption Qcluster solve it by using multi-points query Merge different clusters into one cluster !!

31 July The Use of Inter-Query Feedback Relevance feedback information given by users in each query process often infer a similar semantic meaning (images under the same category) Feature vector space can be re-organized  Relevant images are moved towards to the estimated target  Similar images no longer span on different clusters Parameters estimation method can be improved

31 July st Stage of SOM Training Large number of data points   SOM is used to reduce data size   Each neuron represent a group of similar images   original feature space is not changed directly

31 July Procedure of Inter-query Feedback Updating User marked a set of images as relevant or non-relevant in a particular retrieval process The corresponding relevant neurons are moved towards estimated target Where  M ’ R – set of relevant neurons  c – estimated target  α R – learning rate The corresponding non-relevant neurons are moved away from estimated target

31 July SOM-based Approach Neuron Class 1 Neuron Class 2 Neuron Class 3

31 July SOM-based Approach After each query process Relevant Neuron Non- Relevant Neuron

31 July SOM-based Approach Estimated Target

31 July SOM-based Approach Relevant neurons are moved towards estimated target

31 July SOM-based Approach

31 July SOM-based Approach Feature vector space re-organized

31 July SOM-based Approach After several iterations (users’ queries)

31 July SOM-based Approach

31 July SOM-based Approach Similar images cluster together instead of spanning across different clusters in the new, re-organized feature vector space

31 July Experiments Real data from Corel image collection  4000 images from 40 different categories Feature extraction methods  RGB color moment (9-d)  Grey scale cooccurence matrix (20-d) 80 queries are generated evenly among 40 classes Evaluations  MARS  PE without SOM-based inter-query feedback training  PE with SOM-based inter-query feedback training

31 July Precision vs Recall

31 July Conclusion We propose a parameters estimation approach for capturing user’s target as a distribution A display set selection scheme similar to maximum entropy display is used to capture more user’s feedback information A SOM-based inter-query feedback is proposed  Overcome the single cluster assumption of most intra- query feedback approach

31 July Making People Share DIStributed COntent-based Visual Information Retrieval

31 July P2P Information Retrieval Images Peer databases Feature Extraction Query Result Query Image Lookup … How to locate relevant images In an efficient manner?

31 July Contributions Migrate centralized architecture of CBIR to distribution architecture Improve existing query scheme in P2P applications A novel algorithm for efficient information retrieval over P2P  Peer Clustering  Firework Query Model (FQM)

31 July Existing P2P Architecture Centralized  Napster, SETI (Berkeley), ezPeer (Taiwan)  Easy implementation  Bottleneck, single point failure  Legal problems update query answer transfer

31 July Existing P2P Architecture Decentralized Unstructured  Gnutella (AOL, Nullsoft), Freenet (Europe)  Self-evolving, robust  Query flooding Peer TCP connection

31 July Existing P2P Architecture Decentralized Structured  Chord (SIGCOMM’01), CAN(SIGCOMM’01), Tapestry (Berkeley)  Efficient retrieval and robust  Penalty in join and leave Distributed Hash Table (DHT) Peer in the network Files shared by peers CAN model TCP connection

31 July DISCOVIR Approach Decentralized Quasi-structured  DISCOVIR (CUHK)  Self-organized, clustered, efficient retrieval attractive connections random connections

31 July Design Goal and Algorithms used in DISCOVIR Peers sharing “similar” images are interconnected Reduce flooding of query message Construction of self-organizing network 1. Signatures calculation 2. Neighborhood discovery 3. Attractive connections establishment Content-based query routing 1. Route selection 2. Shared file lookup

31 July Construction of Self-Organizing Network Signatures calculation Signatures discovery of neighborhoods Comparison of signatures Attractive connection establishment

31 July Signatures Calculation Feature vector space

31 July Signatures Calculation Peer A Peer B Centroid of peer

31 July Signatures Calculation Peer A A1A1 A2A2 A3A3 Peer B B1B1 B2B2 B3B3 Centriod of sub-cluster Centroid of peer

31 July Signatures Calculation Peer A A1A1 A2A2 A3A3 Peer B B1B1 B2B2 B3B3 Centriod of sub-cluster Centroid of peer

New attractive connection Random connection Existing clustered P2P network (1.5,1.2) (1.3,2.4) (1.6,1.8) (5.2,8.5) (5.6,8.8) (4.9,9.7) (4.7,9.3) (2.9,6.5) (2.7,6.0) (2.6,5.9) (2.0,5.8) Random connection Attractive connection (x,y) Signature value

31 July Content-based Query Routing Incoming query Signature value left? Similarity < threshold? Forward to random link if not forwarded before Forward to attractive link End N Y Y N

31 July Content-based Query Routing Similarity < threshold Similarity > threshold Random connection Attractive connection

31 July Comparison of Content-based Query Routing and Address-based Query Routing AspectSchemeDescription ApplicationABRInternet Protocol (IP), Domain Name System (DNS) CBRWide Area Information System (WAIS), DISCOVIR ProblemABRWe know where to go, but not the path CBRWe don’t know where to go, nor the path EmphasisABRCorrectness, speed CBRRelevance of result retrieved GoalABRAvoid unnecessary traffic CBRAvoid unnecessary traffic

31 July Experiments Dataset  RBG color moment, 9-d images from Corel database, 100 classes  Synthetic data, 9-d, points, 100 classes Operation  Distribute data-points into peers (1 class per peer)  Simulate network setup and query (averaged 50 queries) Investigation  Scalability (against number of peers)  Property (against TTL of query message)  Data resolution (different number of signatures per peer)

31 July Network Model Small world characteristic, power-law distribution  Few peers are connected with many peers  Many peers are connected with few peers

31 July Performance Metrics Relevance of retrieved result  Recall Number of query traffic generated  Query scope Effectiveness of query routing scheme  Query efficiency Number of retrieved relevant result Total number of relevant result Number peers visited by query message Total number of peers

31 July Recall vs Peers

31 July Recall vs TTL

31 July Query Scope vs Peers

31 July Query Scope vs TTL

31 July Query Efficiency vs Peers

31 July Query Efficiency vs TTL

31 July Difference Between Synthetic Data and Real Data Inter-cluster distanceMean of variances realsyntheticrealsynthetic max max min min avg avg

31 July Effects of Data Resolution Assign 2-4 classes of image to each peer  High data resolution (use 3 signatures)  Low data resolution (use 1 signatures)

31 July Conclusion CBIR is migrated from centralized server approach to peer-to-peer architecture Efficient retrieval is achieved by  Constructing a self-organizing network  Content-based query routing Scalability, property and effects on data resolution are investigated Query efficiency are at least doubled under the proposed architecture

Questions and Answers

31 July Precision, Recall, Novelty Precision  range [0,1] Recall  range [0,1] Set of retrieved result Set of relevant result Set of retrieved known to user before hand

31 July Update Equation, Learning Rate Update equation of non-relevant neurons Update equation of neighboring neurons

Original SOM MModified SOM M’ Modified model vector space data-size reduced to |M’| Original feature vector space data-size is |I| 1-1 mapping function – f -1 Table lookup Retrieval process with SOM to capture feedback information

31 July Multiple Clusters Version

31 July DISCOVIR – System Architecture Built on LimeWire, Java-based Plug-in architecture for feature extraction module Query by example, sketch, thumbnail previewing Connection Manager Packet Router Plug-in Manager HTTP Agent Feature Extractor Image Indexer DISCOVIR Network Shared Collection DISCOVIR Core DISCOVIR User Interface WWW Image Manager

31 July DISCOVIR – Screen Capture

DISCOVIR-Protocol Modification Minimum SpeedFeature Name0Feature Vector0 Image Query 0x … Number of HitsPortIP AddressSpeedResult Set Image Query Hit 0x Servant Identifier …nn+16 File IndexFile SizeFile NameThumbnail information, similarity … Minimum SpeedDISCOVIRSIGNATURE 0 0 DISCOVIR Signature Query 0x Number of HitsPortIP AddressSpeedResult Set DISCOVIR Signature Query Hit 0x Servant Identifier …nn+16 Dummy Feature Extraction nameSignature value …

31 July Query Message Utilization

31 July Average Reply Path Length