1. ACM MM 2000, LA, USA 1 Giving Meanings to WWW Images Heng Tao Shen Beng Chin Ooi Kian Lee Tan

2. ACM MM 2000 2 Outline Image Representation Model Semantic Measure Model Relevance Feedback Experiments

3. ACM MM 2000 3 Background Image: indispensable component in WWW – 1 image = 1000 words WWW: rich resource of images – Some 100 billions? Tradition: poor performance – Keywords Content_based: no enough semantic – Like object, event, and relationship Not effective for images from WWW

4. ACM MM 2000 4 Cont Semantics of embedded images in HTML – Image Title, ALT, Page Title, Image Caption -> ChainNet model Similarity between query and image – List space model Relevance feedback: – Improve precision further

5. ACM MM 2000 5 Weight ChainNet model Lexical chain(LC) – A sentence that carries certain semantics by its words 6 types of LC – TLC: Title Lexical Chain – PLC: Page Lexical Chain – ALC: Alt Lexical Chain – SLC: Sentence Lexical Chain – RSLC: Reconstructed Sentence Lexical Chain – CLC: Caption Lexical Chain

6. ACM MM 2000 6 Title Caption 7 8 9 14 4 5 ALT SLC: 1->2->3->4->5 RSLC: 1->2->8->9 CLC: 1->2->…->14 3 2 1 4 Page Title

7. ACM MM 2000 7 Semantic measure model Computing similarity between two LCs – List space model Where e i and e j are matched terms in list 1 and list 2 respectively.

8. ACM MM 2000 8 Semantic measure model – Match scale: closeness in view of match order Here v1 and v2 represent the children of first and second original lists respectively. Inspired from the angle between two vectors Where v2j is the matched word in v2 for v1i in v1

9. ACM MM 2000 9 Semantic measure model LC Match Level(LC1, LC2): the number of distinct matched words by two LCs – Match level threshold: The minimum match level for LC to keep its original semantic – LC Semantic similarity: similarity(list1, list2) in its LC Match Level

10. ACM MM 2000 10 Semantic measure model Image Match Level(image, query) = MAX ( TLC.weight * LCMatchLevel( TLC, QLC), ALC.weight * LCMatchLevel( ALC, QLC), PLC.weight * LCMatchLevel( PLC, QLC), SLC.weight * LCMatchLevel( SLC, QLC), RSLC.weight *LCMatchLevel( RSLC, QLC), CLC.weight * LCMatchLevel( CLC, QLC) )

11. ACM MM 2000 11 Relevance Feedback Semantic Accumulation – Choose one best image as feedback – Accumulate the previous feedback images’ semantics to construct a new QLC – Results are more close to the specific image selected – More noise

12. ACM MM 2000 12 Semantic accumulation Weight F/Q ChainNet QLC New query Last feedback image Image ALT Image Title Image Caption Page Title

13. ACM MM 2000 13 Semantic Integration and Differentiation – Choose several Good and Bad images as feedback – Integrate Good semantics to construct new query – Differentiate irrelevant images by Bad images – Results are more diverse and less noise

14. ACM MM 2000 14 Semantic integration and differentiation Similar weight F/Q ChainNet QLC New query Good feedback images Image iImage 3Image 2Image 1 LC1LC2LC3LCi Most related LC

15. ACM MM 2000 15 Experiments Set up – Web crawler to collect images – 5232 images from over 2000 URLs – 12 general queries

16. ACM MM 2000 16 Tuning the LC Weights

17. ACM MM 2000 17 Tune the match level MatchLevel Threshold= coef * query.length()+ constant

18. ACM MM 2000 18 Impact of match scale – explore the importance of match order

19. ACM MM 2000 19 Feedback Mechanisms

20. ACM MM 2000 20 Feedback Mechanisms One-step feedback of Accu and I&D for Q1.

21. ACM MM 2000 21 Conclusion – Inner semantic structure of surrounding text is explored well for good precision achievement – ChainNet model and list space model work well – RF techniques help to return more accurate results

22. ACM MM 2000 22 Future work – Explore LC meanings by AI technique – Extract semantics from visual content, then integrate with our system to construct a more advanced semantic retrieval system – Object-oriented detection

23. ACM MM 2000 23 DEMO ON THURSDAY SEE YOU THEN… http://efoto.geofoto.com