1 LIST – DTSI – Service Réalité virtuelle, Cognitique et Interfaces sensorielles A Conceptual Approach to Web Image Retrieval Adrian Popescu Gregory Grefenstette CEA LIST, France OntoImage 2008 Marrakech, Morocco Contacts :

2 LIST – DTSI – Service Réalité virtuelle, Cognitique et Interfaces sensorielles Some facts 60% of Internet image queries include only one concept People are stupid or lazy? What are they looking for? Common nouns Places Other (famous) people Products Sexual content

3 May 2008 DTSI LIST – DTSI – Service Réalité virtuelle, Cognitique et Interfaces sensorielles Our Image Browsing Strategy  Introduce conceptual hierarchy into image search applications  Expand common words by more specific terms  Present results reflecting hierarchy  Dual browsing: image or text query  Exploit conceptual hierarchy and image spidering to structure image base

4 May 2008 DTSI LIST – DTSI – Service Réalité virtuelle, Cognitique et Interfaces sensorielles Architecture of OLIVE Olive picture retrieval framework employing a large scale conceptual hierarchy to provide a dual access to Web images

5 May 2008 DTSI LIST – DTSI – Service Réalité virtuelle, Cognitique et Interfaces sensorielles Adapting WordNet  Concept ranking Based on Web counts  Relevance (C1) = webCount(C1, parent) Web counts and WordNet structure  Relevance (C1, C2) = webCount(C1, parent) * dist(C1,C2)/senses(C1) Example for dog  (Web counts) pooch, pug, Newfoundland, basset  (Web+WordNet) collie, basset, german shepherd, doberman  Query reformulation Query concept + representative leaf nodes

6 May 2008 DTSI LIST – DTSI – Service Réalité virtuelle, Cognitique et Interfaces sensorielles Knowledge for dog Organism, canine, domestic animal, living thing, physical entity Parents Wolf, fox, hyena, wild dog, domestic cat, bitch, jackal Related types Poodle, corgi, spitz, cur, hunting dog, working dog, toy dog, Dalmatian, griffon Narrower concepts pooch, pug, Newfoundland, basset, beagle, cairn, Airedale, Doberman, German shepherd Leaf Subtypes yesPolysemic Cat, monkey, elephant, tiger, lion, golden retriever, kittens, Chihuahua, snake Expanded search Rabbit, bird, Scooby Doo Related names Puppy, Alaskan husky Narrower search

7 May 2008 DTSI LIST – DTSI – Service Réalité virtuelle, Cognitique et Interfaces sensorielles First response page for dog - Google Image

8 May 2008 DTSI LIST – DTSI – Service Réalité virtuelle, Cognitique et Interfaces sensorielles First response page for dog - Olive

9 May 2008 DTSI LIST – DTSI – Service Réalité virtuelle, Cognitique et Interfaces sensorielles First response page for Doberman - Olive

10 May 2008 DTSI LIST – DTSI – Service Réalité virtuelle, Cognitique et Interfaces sensorielles Visual similarity search for image of Doberman

11 May 2008 DTSI LIST – DTSI – Service Réalité virtuelle, Cognitique et Interfaces sensorielles Evaluation – Precision on first two pages

12 May 2008 DTSI LIST – DTSI – Service Réalité virtuelle, Cognitique et Interfaces sensorielles Precision over ten pages

13 May 2008 DTSI LIST – DTSI – Service Réalité virtuelle, Cognitique et Interfaces sensorielles Qualitative evaluation Experiment setting  10 users compared Olive and Google Image  Questionnaire about: results structuring; enriched interaction; automatic reformulation; Positive reactions from the users to the new features introduced in Olive The users also provided useful comments under the form of free text.

14 May 2008 DTSI LIST – DTSI – Service Réalité virtuelle, Cognitique et Interfaces sensorielles Conclusions  A semantic retrieval architecture can cover a significant part of Web picture queries.  Dual access to picture content, textual and visual Simple but efficient solution to introduce CBIR in large scale architectures  With semantic structures, existing image indexes can be better used than in current applications OLIVE outperforms Google Image, qualitatively and quantitatively  Future work: The extension of the hierarchy (e.g. using Wikipedia)‏

15 May 2008 DTSI LIST – DTSI – Service Réalité virtuelle, Cognitique et Interfaces sensorielles A glimpse of the future

16 May 2008 DTSI LIST – DTSI – Service Réalité virtuelle, Cognitique et Interfaces sensorielles Advertising  Website  I am looking for work Starting November 2008