1. Using HTML Textual and Structural Data for Web Image Search Cheng Thao, Ethan Munson, Jim Dabrowski, Nikolas D. Bohne University of Wisconsin-Milwaukee

2. Which image is George Bush or has George Bush?

3. Which images are similar to this image?

5. <IMG SRC="http://www.gopbi.com/community/groups/bigband/images/bill%20cosby.jpg" alt="" BORDER=0 VSPACE=3 HSPACE=3> Bill Cosby <IMG SRC="http://www.gopbi.com/community/groups/bigband/images/betty%20white.jpg" alt="" BORDER=0 VSPACE=3 HSPACE=3> Betty White <IMG SRC="http://www.gopbi.com/community/groups/bigband/images/tom%20brokaw.jpg" alt="" BORDER=0 VSPACE=3 HSPACE=3> Tom Brokaw <IMG SRC="http://www.gopbi.com/community/groups/bigband/images/george%20bush.jpg" alt="" BORDER=0 VSPACE=3 HSPACE=3> Pres. George Bush <IMG SRC="http://www.gopbi.com/community/groups/bigband/images/ed%20mcmahon.jpg" alt="" BORDER=0 VSPACE=3 HSPACE=3> Ed McMahon <IMG SRC="http://www.gopbi.com/community/groups/bigband/images/bob%20barker.jpg" alt="" BORDER=0 VSPACE=3 HSPACE=3> Bob Barker Does the HTML source tell which images is George Bush?

6. Introduction -image search is difficult - performance is slow - image identification is a complex, inaccurate task -most research on image search has emphasized analysis of image content -few Web image search engines - commercial: Alta Vista, Google - research: WebSeek -little research in textual image search

7. HTML overview -HTML document composed of: -head -title -meta -body -paragraph, -table, -text, -link, -image, …

8. Sample HTML HTML overiew first paragraph Simple Table 1 2 3 4 5 6 7 8 9 Here is a photo of George Bush.

10. Previous work - Previous work - Yelena Tsymbalenko -studied HTML constructs and determine what can be used in image search. -found the following to be effective - title of the page - image filename - image alt attribute

11. Research Goals -What HTML features make good clues to the content of images? - Structural features (document, table) - File names or URLs - Formatting of material (bold, heading) - How can clues be combined into a single relevance rating?

12. Image Search Study Process -Downloading pages with matching text - Use existing search engine to identify matches - These pages provide a corpus of images - We download pages so that our corpus remains static -Download acts as a snap shot -Clue extraction -Analyze each page in corpus for all possible clues to image content -Human relevance ratings -human rates if an image is relevant to the query -Statistical analysis to find clue-based relevance functions

13. Downloading Software query URLs Process: Downloading Web Pages queries Search Engine Web Pages images Web pages and images are saved to local disk.

14. <query> 1 1 2 2 George George Bush Bush </query><query> 2 2 3 3 Bill Bill Clinton Clinton </query> Design: Queries in XML Multiple queries are stored in an XML file Engine: 1=Altavista, 2=Excite, 3=Hotbot, 4=Google Method: 1 =or, 2 = and, 3 = expression Search for George Bush using Alta Vista and must have all the words Search for Bill Clinton using Hotbot and search for exact expression

15. Process: Clue Extraction Extraction Software cluesqueries Clues Extraction Software clues

16. Data to be analyzed For each page –Query used to find page –Source URL –For each image Source URL Attributes Position in document –For each clue Whether clue feature occurs in document at all If feature occurs with text matching the query –Position in document for each occurrence

17. Relevant Rating Software Query & image Relevant/ not Process: Relevance Rating queries Human Presents images from each query to the user from the database, and record the human relevance rating back to the database.

18. Clues : global Global Clues - clues that apply every image on the page - filename of page - path of page - host of page - title element of the web page - keywords found in meta element - description found in meta element Why do we break the URL into three clues? Different parts of the URL contributes different relevance factor to the overall relevance of the image in that page.

19. Clues : global Apple <META NAME="keywords" CONTENT="Apple Computer, Power Macintosh, PowerBook, AppleWorks, WebObjects, iMovie, QuickTime, Desktop Movies, Software, Operating Systems, Mac OS, iMac, iBook"> <META NAME="Description" CONTENT="Visit www.apple.com for the latest news, the hottest products, and technical support resources from Apple Computer, Inc.">

20. Clues : image file Image file properties - external properties - filename - path - host An image can be from another host, and have different paths.

21. Clues : common attributes Elements have common attributes -title - describe what the element is -id- used in identifying the element -name- same as id, older HTML Clues that use these attributes: link, image, object, table, cell, row

22. Clues: Image Container Link to an image - text enclosed within the link element Embed image element -alt attribute (usually describes what the image is) Object element -text that enclosed within the object element

23. Clues: table Table ( ): - summary attribute - describes the table content - caption - describes table content - row heading - row - column heading - column - cell - neighboring cells (above, below, right, left)

24. Clues: table

31. Clues: headings Heading elements(h1, h2,..h6) - headings above image - headings below image header above image header below image Heading can indicate a topic and images below the heading maybe relate to the heading. Some use headings as caption above images, and sometimes below images. Some headers are used where fonts should be used or bold should be used.

32. In this photo, the heading comes after the image. Often if it is used as a topic, it usually comes before the image. But some images have heading as caption below the images.

33. Clues : text Emphasized text elements - bold - italic - underline - strong - emphasis - big Body text - text that surrounds the image - distance

34. Current Project Status - Prototype download and clue extraction software nearly complete - now testing implementation - data (without human relevance ratings) in early November - Recruiting students to build on-line relevance rating system - hope to get students outside lab to help with ratings via Web interface

35. Challenges for image search systems - computing word distance from image - Stylesheet used for presentation - table pattern - pattern of HTML elements usage - CGI returned images - structural boundaries -patterns in Web page design -HTML generators

36. Cheng Thao, chengt@uwm.edu