1. WWW Challenges : Supporting Users in Search and Navigation Natasa Milic-Frayling Microsoft Research, Cambridge UK SOFSEM 2004 January 28, 2004

2. Introduction Research: Research: Web usage and interfaces Optimization of service architectures Text Classification – support for document classification, routing, filtering Presentation Focus WWW challenges in designing effective services and applications. WWW challenges in designing effective services and applications. Intersection Browser Interface – Internet, Intranet, services, local drives. Browser Interface – Internet, Intranet, services, local drives. Devices and applications: TabletPC, PDA, eBook Devices and applications: TabletPC, PDA, eBook Services: MSN Portal and Search - on-line searching, reading, and browsing Services: MSN Portal and Search - on-line searching, reading, and browsing

3. Introduction Intersection Browser Interface – Internet, Intranet, services, local drives. Browser Interface – Internet, Intranet, services, local drives. Devices and applications: TabletPC, PDA, eBook Devices and applications: TabletPC, PDA, eBook Services: MSN Portal and Search - on-line searching, reading, and browsing Services: MSN Portal and Search - on-line searching, reading, and browsing Research: Research: Web usage and interfaces Optimization of service architectures Text Classification – support for document classification, routing, filtering Presentation Focus WWW challenges in designing effective services and applications. WWW challenges in designing effective services and applications.

4. Introduction Research: Research: Web usage and interfaces Optimization of service architectures Text Classification – support for document classification, routing, filtering Presentation Focus WWW challenges in designing effective services and applications. WWW challenges in designing effective services and applications. Intersection Browser Interface – Internet, Intranet, services, local drives. Browser Interface – Internet, Intranet, services, local drives. Devices and applications: TabletPC, PDA, eBook Devices and applications: TabletPC, PDA, eBook Services: MSN Portal and Search - on-line searching, reading, and browsing Services: MSN Portal and Search - on-line searching, reading, and browsing

5. Characteristics of the Web Highly distributed: distributed data and processes Highly dynamic Evolving content, with still inadequate content publishing practice. IMPLICATIONS 

6. On-line Experience Web access is a combination of search and navigation Search to find URL of relevant pages Search to find URL of relevant pages Navigation to explore result space Navigation to explore result space Reading on devices of various display sizes. Reading on devices of various display sizes. Only limited “context” in both activities preserved and exposed  Ineffective search  Lost in hyperspace  Lost within a document, on small screen devices.

7. ‘Diagnoses’ Three aspects of the Web Separation of search and document delivery Separation of search and document delivery Separation of document authoring and generation of metadata about the documents required by services and applications Separation of document authoring and generation of metadata about the documents required by services and applications Lack of generic publishing format to support flexible display of content across devices. Lack of generic publishing format to support flexible display of content across devices.

8. Part I Separation of search and document delivery  Ineffective Search MIDAS - SiteExplorer

9. Query URLs URLs User’s Information Need User’s Information Need Web Server Search Engine Web Server HTTP Request HTTP Request Search processes Web page delivery

10. MS READ Service MS READ Service Highlighting - How is it done ? Query URLs URLs Query syntactic Analysis Semantic Expansion Highlighting Regime Thumbnail Creation Query syntactic Analysis Semantic Expansion Highlighting Regime Thumbnail Creation User’s Information Need User’s Information Need Topic Description Web Server Search Engine Web Server HTTP Request HTTP Request

11. MS READ Service MS READ Service Link Evaluation - How is it done ? NLPNLP IndexingIndexing Search Over Local IndexSearch Over Local Index Web Server TopicStorage: Topic 1 Topic 2 Topic 3 Topic 4 HTTP Requests for Text Only Mark Links for Relevance Download Text Only

12. MS Read Users have difficulty locating relevant parts of a Web page while reviewing search results (MSN Search Diary and Field Interviews) Users have difficulty evaluating search results and refining their search (Anne Cohen-Kiel’s ethnographic study in Spain, UK and Canada; MSN Search Diary Study and Site Interviews). Solution: Preserve user’s topic of interest and provide highlighting of topic terms on the pages that the user is viewing. Allow the users to enhance the topic by adding new query terms or resources (lists of concepts, entities, etc.) and perform search over the page content Allow the user to search the content of the pages that are linked to the current page. When the page is the search result page, this is equivalent to refining the search over the previous top N search results. When the page is the search result page, this is equivalent to refining the search over the previous top N search results.

13. MSRead – Supporting search

14. MIDAS and SiteExplorer Separation of document authoring and generation of metadata about the documents required by services and applications  User lost in the hyperspace Part II

15. Problem Crawling - Services, such as search engines, collect the data and create metadata but do not deliver the content Out of sync with the data on the Web servers  ‘broken links’ Out of sync with the data on the Web servers  ‘broken links’ Services can perform only basic analysis of the context No information about structure of information resources No information about structure of information resources No sophisticated linguistic process. No sophisticated linguistic process.

16. Solution: MIDAS Framework Distributed metadata generation Generate & store meta-information alongside contents At authoring or publishing time At authoring or publishing time Synchronised with publishing Synchronised with publishing Deliver metadata upon request In case of centralized services Services do not crawl for data but only for metadata Services do not crawl for data but only for metadata Obtain data through ‘push’ by authors/web servers. Obtain data through ‘push’ by authors/web servers. Site structure Page structure METADATA: Linguistic analysis Statistical analysis Visual representation Site structure Page structure METADATA: Linguistic analysis Statistical analysis Visual representation

17. AUTHORCLIENTSERVER Web Server Web Content

18. AUTHORCLIENT Metadata Server SERVER Web Server Automatically Generated Metadata Web Content FrontPage Site Template and Structure in XML Format SiteExplorer Author generated metadata Web metadata (XML)

19. MIDAS is NOT… …an element of the Semantic Web Not adding “knowledge” explicitly into the Web Simple metadata Easily authored/easily computable at authoring/publishing time Easily authored/easily computable at authoring/publishing time Presently available but dismissed Presently available but dismissed

20. Problems addressed Users have difficulty choosing the right website from the result set Users want overviews of sites in a list of search results (Anne Cohen-Kiel’s ethnographic study in Spain, UK and Canada) Users want overviews of sites in a list of search results (Anne Cohen-Kiel’s ethnographic study in Spain, UK and Canada) Users have difficulty evaluating search results and refining their search (MSN Search Diary Study and Site Interviews) Users have difficulty evaluating search results and refining their search (MSN Search Diary Study and Site Interviews) Users have difficulty locating relevant information within a destination site once they get to the site (MSN Search Diary Study and Site Interviews) Site Explorer’s Solutions: Providing users with an overview of the site content as interactive sitemap Providing users with an overview of the site content as interactive sitemap Supporting exploration of the site through local search Supporting exploration of the site through local search

21. “Anyone who has been to a shopping mall knows the value of the ‘you are here’ dot on the map … Site maps must become more aware of users’ website navigation…” Jakob Nielsen, Site Map Usability January 6, 2002 External studies External studies

22. SiteExplorer Bar Search Box Site Overview Site Structure Page details

23. SiteExplorer Bar Search Box Site Overview Site Structure Page details

24. SmartView and SearchMobil Viewing Web on PDAs and Mobile Phones Lack of generic publishing format to support flexible display of content across devices Lack of generic publishing format to support flexible display of content across devices  Ineffective reading on mobile devices Part III

25. Lost in Hyperspace - Small Complex pages on small screens Overview Overview – none provided at the moment – none provided at the moment Extensive horizontal/vertical scrolling Extensive horizontal/vertical scrolling

26. Lost in Hyperspace - Small Location of search hits on result page Difficulty even on desktop screens Difficulty even on desktop screens Reason: disassociation of search service and document delivery Reason: disassociation of search service and document delivery

27. SmartView

28. SmartView Prototype

29. SmartView Plus

30. SearchMobil SearchMobil Web Service Collection of search results – “booklet” of Web pages Collection of search results – “booklet” of Web pages Creation of the “local” full text index Creation of the “local” full text index Search within a designated set of pages Annotated booklets (hit highlighting) Annotated booklets (hit highlighting)

31. Web Search On-line search: Google On-line search: Google Automatic download of pages Automatic download of pages Processing of pages – structure discovery and content indexing Processing of pages – structure discovery and content indexing Creation of a booklet of overviews Creation of a booklet of overviews Indicators of search hits Indicators of search hits Indicator of the best region – scroll down the ‘red’ section Indicator of the best region – scroll down the ‘red’ section Select the region and access the detailed view Select the region and access the detailed view SearchMobil Features

32. Web Search – Detail View SearchMobil Features On-line search: Google On-line search: Google Automatic download of pages Automatic download of pages Processing of pages – structure discovery and content indexing Processing of pages – structure discovery and content indexing Creation of a booklet of overviews Creation of a booklet of overviews Indicators of search hits Indicators of search hits Indicator of the best region – scroll down the ‘red’ section Indicator of the best region – scroll down the ‘red’ section Select the region and access the detailed view Select the region and access the detailed view

33. Local Search SearchMobil Features – Cont. Local search – focussed on the set of pages in the booklet Local search – focussed on the set of pages in the booklet Indicators of relevance at the page and the booklet level Indicators of relevance at the page and the booklet level

34. SearchMobil Prototype

35. Summary Simple proposition: Save metadata about structure and content generated by authoring applications Save metadata about structure and content generated by authoring applications Benefits on the client side: Rich context for search and navigation Rich context for search and navigation Interactive download of document elements and metadata for small devices Interactive download of document elements and metadata for small devices Benefit for services: Metadata collected and in s Metadata collected and in s Opportunity for new services based on rich metadata Opportunity for new services based on rich metadata Opportunity for push based services – reduce the need for crawling. Opportunity for push based services – reduce the need for crawling.

36. Thank you!