ACM MM 2000, LA, USA 1 Giving Meanings to WWW Images Heng Tao Shen Beng Chin Ooi Kian Lee Tan

ACM MM Outline Image Representation Model Semantic Measure Model Relevance Feedback Experiments

ACM MM Background Image: indispensable component in WWW – 1 image = 1000 words WWW: rich resource of images – Some 100 billions? Tradition: poor performance – Keywords Content_based: no enough semantic – Like object, event, and relationship Not effective for images from WWW

ACM MM Cont Semantics of embedded images in HTML – Image Title, ALT, Page Title, Image Caption -> ChainNet model Similarity between query and image – List space model Relevance feedback: – Improve precision further

ACM MM Weight ChainNet model Lexical chain(LC) – A sentence that carries certain semantics by its words 6 types of LC – TLC: Title Lexical Chain – PLC: Page Lexical Chain – ALC: Alt Lexical Chain – SLC: Sentence Lexical Chain – RSLC: Reconstructed Sentence Lexical Chain – CLC: Caption Lexical Chain

ACM MM Title Caption ALT SLC: 1->2->3->4->5 RSLC: 1->2->8->9 CLC: 1->2->…-> Page Title

ACM MM Semantic measure model Computing similarity between two LCs – List space model Where e i and e j are matched terms in list 1 and list 2 respectively.

ACM MM Semantic measure model – Match scale: closeness in view of match order Here v1 and v2 represent the children of first and second original lists respectively. Inspired from the angle between two vectors Where v2j is the matched word in v2 for v1i in v1

ACM MM Semantic measure model LC Match Level(LC1, LC2): the number of distinct matched words by two LCs – Match level threshold: The minimum match level for LC to keep its original semantic – LC Semantic similarity: similarity(list1, list2) in its LC Match Level

ACM MM Semantic measure model Image Match Level(image, query) = MAX ( TLC.weight * LCMatchLevel( TLC, QLC), ALC.weight * LCMatchLevel( ALC, QLC), PLC.weight * LCMatchLevel( PLC, QLC), SLC.weight * LCMatchLevel( SLC, QLC), RSLC.weight *LCMatchLevel( RSLC, QLC), CLC.weight * LCMatchLevel( CLC, QLC) )

ACM MM Relevance Feedback Semantic Accumulation – Choose one best image as feedback – Accumulate the previous feedback images’ semantics to construct a new QLC – Results are more close to the specific image selected – More noise

ACM MM Semantic accumulation Weight F/Q ChainNet QLC New query Last feedback image Image ALT Image Title Image Caption Page Title

ACM MM Semantic Integration and Differentiation – Choose several Good and Bad images as feedback – Integrate Good semantics to construct new query – Differentiate irrelevant images by Bad images – Results are more diverse and less noise

ACM MM Semantic integration and differentiation Similar weight F/Q ChainNet QLC New query Good feedback images Image iImage 3Image 2Image 1 LC1LC2LC3LCi Most related LC

ACM MM Experiments Set up – Web crawler to collect images – 5232 images from over 2000 URLs – 12 general queries

ACM MM Tuning the LC Weights

ACM MM Tune the match level MatchLevel Threshold= coef * query.length()+ constant

ACM MM Impact of match scale – explore the importance of match order

ACM MM Feedback Mechanisms

ACM MM Feedback Mechanisms One-step feedback of Accu and I&D for Q1.

ACM MM Conclusion – Inner semantic structure of surrounding text is explored well for good precision achievement – ChainNet model and list space model work well – RF techniques help to return more accurate results

ACM MM Future work – Explore LC meanings by AI technique – Extract semantics from visual content, then integrate with our system to construct a more advanced semantic retrieval system – Object-oriented detection

ACM MM DEMO ON THURSDAY SEE YOU THEN…