1. Feb. 1, 2007Search and Sensemaking1 CoLiDeS and SNIF-ACT: Complementary Models for Searching and Sensemaking on the Web Muneo Kitajima, AIST Peter G. Polson & Marilyn H. Blackmon University of Colorado

2. Feb. 1, 2007Search and Sensemaking2 Decoding Acronyms In Title CoLiDeS Comprehension-based Linked-model of Deliberate Search (Kitajima, Blackmon, and Polson, 2000,2005) –Derived from Kintsch’s (1998) construction-integration cognitive architecture –Basis for usability engineering method called Cognitive Walkthrough for the Web (CWW) (Blackmon et al., 2002). SNIF-ACT Scent-based Navigation and Information Foraging in the ACT Cognitive Architecture (Fu and Pirolli, In press; …..)

3. Feb. 1, 2007Search and Sensemaking3 Goals To Integrate: –CoLiDeS Searching and sensemaking at the level of a Web page With –SNIF-ACT Searching and sensemaking at higher levels To Show: –Role of background knowledge –Multiple patches on a single web page –Importance of comprehension processes in searching and sensemaking

4. Feb. 1, 2007Search and Sensemaking4 Outline 1.Information Foraging Theory (Card and Pirolli, 1999; Pirolli, 2005) 2.Brief Summary of SNIF-ACT 3.Scent AND Background Knowledge 4.CoLiDeS 5.Multiple Patches on A Web Page 6.Interactions Between Search and Sense Making 7.Conclusions 8.CoLiDeS +SNIF-ACT

5. Feb. 1, 2007Search and Sensemaking5 1. Information Foraging Theory

6. Feb. 1, 2007Search and Sensemaking6 Insights from Information Foraging Theory World Wide Web is a collection of patches of relevant information (Websites) Forager faces two basic decisions –Continue foraging in current patch –Leave and find a new patch Decisions based on comparison of expected gain of staying in current patch versus cost of finding new patch Scent-based navigation within a site

7. Feb. 1, 2007Search and Sensemaking7 Scent-Based Navigation Consensus –Users perform website navigation tasks by exploration by following a trail of information scent In Pete’s presentation this morning –Showing that very reliable scent cues are required for successful navigation (Hogg and Huberman,1987) –Derivation of theory of scent from first principles

8. Feb. 1, 2007Search and Sensemaking8 Search, Sensemaking, and Comprehension Information Foraging Theory: Two subproblems –Search –Sensemaking CoLiDeS: Both search and sensemaking require comprehension –Search requires comprehension of structure and content of a web page –Sensemaking entails comprehension of retrieved information

9. Feb. 1, 2007Search and Sensemaking9 2. Brief Summary of SNIF-ACT

10. Feb. 1, 2007Search and Sensemaking10 SNIF-ACT ACT-R Model of searching a Website for specific information –Based on Information Foraging Theory (Card and Pirolli, 1999; and Pirolli, 2005) A user’s goal and hyperlink texts are represented as collections of chunks –Spreading activation from link chunks to goal chunks determines scent –Link strengths can be computed from on-line corpora

11. Feb. 1, 2007Search and Sensemaking11 SNIF-ACT 1 and 2 Knowledge –Actions are represented as productions Probability of an action is determined by its utility in relation to utilities of other actions under consideration Utility of clicking on a link is determined by its scent Control –Evaluate links on webpage one link at a time moving down through links in sequential order –Use satisficing problem solving strategy

12. Feb. 1, 2007Search and Sensemaking12 SNIF-ACT 2.0 Page Level Loop During each cycle Decide –Click on best link found so far* –Process another link –Click on back button Utilities (U) of alternative actions –U (Click on best link)  Scent of that link –U (Process another link)  Starts high and decreases with number of links evaluated –U (Click on back button)  Average scents of links processed on previous pages and this page

13. Feb. 1, 2007Search and Sensemaking13 Strengths of SNIF-ACT 2.0 Treatment of Actions with No Scent –Derived from rational analysis of foraging in information patches (Pirolli, 2005) Press back button Leave website Theory of Information Scent –Derived from first principles (Pirolli, 2005) –Linked to spreading activation memory mechanisms in ACT-R

14. Feb. 1, 2007Search and Sensemaking14 3. Scent AND Background Knowledge

15. Feb. 1, 2007Search and Sensemaking15 Scent and Background Knowledge –CoLiDeS assumes that scent is product of successful comprehension of link –Familiarity => Having necessary background knowledge to comprehend words and categories that make up a link –Familiarity of words and categories in link is as or more important than scent

16. Feb. 1, 2007Search and Sensemaking16 Our Version of Scent Used Latent Semantic Analysis (LSA) to estimate similarities between goals and –Descriptions of patches (Headings) –Links Values of Scent –1.0 < s < 1.0, analogous to correlation s 0.3strong scent

17. Feb. 1, 2007Search and Sensemaking17 Larson and Czerwinski (1998) Task –Search for articles in experimental website simulating Encarta online encyclopedia –One or two words described target articles Unfamiliar targets: Pink Floyd, Tlingit, Trilobite, … –Unfamiliar links Anthropology, Paleontology, Theology & Practices, … –Search for articles involving Unfamiliar targets Unfamiliar correct links Far more difficult than predicted by model that only describes scent following

18. Feb. 1, 2007Search and Sensemaking18 Blackmon et al (2002) Experiment Partial replication of Larson & Czerwinski (1998) Fix unfamiliar target problems –Provided participants with four or five sentence definitions of target articles 16 and 32 link conditions –One click –Select correct topic heading from list of 16 or 32 links –Link texts from Larson and Czerwinski

19. Feb. 1, 2007Search and Sensemaking19 Tlingit culture, Tlingit tribes

20. Feb. 1, 2007Search and Sensemaking20 Observed first-click behavior: Unfamiliar problem hid high-scent Anthropology link UNFAMILIAR LINK

21. Feb. 1, 2007Search and Sensemaking21 12 unfamiliar tasks were significantly more difficult (p<.002)

22. Feb. 1, 2007Search and Sensemaking22 …but unfamiliar tasks had same mean scent for correct link

23. Feb. 1, 2007Search and Sensemaking23 Experiments with Ethnic Minority Parents and Children Parents and under-18 children came together & did experiment at adjacent computers Most participants were adult parents with high school or middle school students, but a few children were elementary school students Each task asked them to search for things familiar to children, for example, ferns, pets, earth-moving equipment, or Mexican art 9 category headings always in left hand navigation column, level-2 pages reveal links for one category (e.g., history) in content area

24. Feb. 1, 2007Search and Sensemaking24 Goals had pictures to help participants quickly grasp what they were looking for, and texts written at 3rd-grade level

25. Feb. 1, 2007Search and Sensemaking25 College-level vs. ethnic minority adults and children (3rd-grade reading level) Encarta links 13% unfamiliar scattered sparsely across the page < 5% low frequency words Powerful, consistent evidence that even college- level population flounders on unfamiliar links but helped by grouping of links into coherent patches Encarta links 56% unfamiliar, and some patches are virtually all unfamiliar links 52% low frequency words Our preliminary observations: ethnic minority population frequently resorted to trial-and-error exhaustive searching of entire patches

26. Feb. 1, 2007Search and Sensemaking26 9 top-level category links were surprisingly difficult though predicted Top-level categories familiar (e.g., Social Science, History) 0% low frequency words at top-level College-level population good at scent following on 9 top-level category headings 44% (4 of 9) of top-level categories are unfamiliar 41% low frequency words at top level Ethnic minority population poor at scent following on 9 top-level category headings

27. Feb. 1, 2007Search and Sensemaking27 4. CoLiDeS

28. Feb. 1, 2007Search and Sensemaking28 CoLiDeS is a Comprehension-Based Model in HCI Simulates a user searching for information relevant to her goal on a Web site Based on the Construction-Integration (C-I) architecture (Kintsch, 1998) –C-I is a detailed story about utilization of background knowledge to comprehend or action plan –C-I can fill in gaps in knowledge using hill climbing or pure forward search as a problem solving strategy –Hill climbing controlled by information scent

29. Feb. 1, 2007Search and Sensemaking29 CoLiDeS Starts with a Complex Goal I am interested in reading recent articles that deal with prediction of sea level rise in the near future caused by global warming. I am going to browse the Science section of the New York Times Website to articles. Can include: –Description of search target –Navigation

30. Feb. 1, 2007Search and Sensemaking30 CoLiDeS Parses Web Page into Patches to Select a Link For each page for a given goal: Attention phase: Select a patch –Parse page into collection of patches –Describe each patch –Select a patch whose description is comprehended AND is most similar to the goal Action selection phase: Select a link –Describe each link –Select a link whose description is comprehended AND is most similar to the goal

31. Feb. 1, 2007Search and Sensemaking31 Dec. 14 NY Times

32. Feb. 1, 2007Search and Sensemaking32 A Page Parsed Into Patches Logo Site Nav Bar Ad Search Window Ad Site Nav Links Articles Ad Topics Ad Info Ad

33. Feb. 1, 2007Search and Sensemaking33 Attended to Patch

34. Feb. 1, 2007Search and Sensemaking34 Determining Patches on A Page –A mixture of bottom-up and top-down processes –Bottom-up processes Driven by perceptual features that define visually related regions –Top-down processes Controlled by the user’s knowledge of Web page layout conventions and typical pages for a given Web site or type of Web site

35. Feb. 1, 2007Search and Sensemaking35 Examples of Phenomena to be Explained by Attention Phase Interactions between user’s background knowledge, goals, and attention to different patches on a Web page –Banner “Blindness” –Eye movement patterns during interactions with a Web site

36. Feb. 1, 2007Search and Sensemaking36 CoLiDeS’s Key Assumptions Core process underlying Web navigation is comprehension of texts and images Comprehension processes –Build mental representations of goals, patches on a page, hyperlinks, images, and other targets for action on a page –Compare goal with representations of patches and hyperlinks, images, and other targets for action on a page. –Select a patch to attend to or object on page to act on based on comprehension AND similarity of descriptions

37. Feb. 1, 2007Search and Sensemaking37 5. Multiple Patches on a Web Page

38. Feb. 1, 2007Search and Sensemaking38 Blackmon, et al. (2003, 2005) Experiments Search for target article on Encarta-like website First two levels of hierarchy presented on one or two web page(s) Prototype is 93 links nested under 9 category headings –Each heading and subordinate links defined a patch –Participant task: Select correct link in correct patch –Clicking on link lead to web page with 8 article titles on page –Click on matching title Time limit: 130 seconds Example task: Find encyclopedia article on Dome of the Rock

39. Feb. 1, 2007Search and Sensemaking39 Example of One Level Web Page Summary of Target Article

40. Feb. 1, 2007Search and Sensemaking40 Religion & Philosophy is the Most Attractive Patch

41. Feb. 1, 2007Search and Sensemaking41 Webpage for Find Dome of the Rock Task One competing patch on page –Art, Language & Literature (correct patch) –Religion & Philosophy (competing patch) –Geography –History –Social Science –Performing Arts –3 other patches 4 competing links in most attractive patch: –Theology & Practices –Religions & Religious Groups –Scripture –Religious Figures –3 more links in patch

42. Feb. 1, 2007Search and Sensemaking42 First-click Distribution Correct Patch

43. Feb. 1, 2007Search and Sensemaking43 Performance on Dome of the Rock Task N=38 Mean total clicks on links = 8.3 Mean time = 115 seconds 58% time expired 42% finally clicked the correct link, “Architecture” nested under Art, Language, & Literature

44. Feb. 1, 2007Search and Sensemaking44 6. Interactions Between Search and Sensemaking

45. Feb. 1, 2007Search and Sensemaking45 Blackmon, et al. (2002, 2003, 2005) Experiments Examining combined impact of –Design errors that mislead a scent following heuristic Competing links in incorrect patches Weak scent correct links Unfamiliar links interfering with sensemaking Cognitive Walkthrough for the Web can: –Correctly identify the above problems –Guide successful correction of these problems

46. Feb. 1, 2007Search and Sensemaking46 Combined Analysis of Blackmon, et al. (2002, 2003, 2005) Task => College students search for a target article in Encarta like website Mean clicks and time for 324 tasks –N’s ranged from 22 to 50, mean N about 40 Extensive exploration using multiple regression techniques Independent Variables –Total competing links in incorrect patches –Weak scent correct link (yes, no) –Unfamiliar correct link (yes, no) Examples of other variables….

47. Feb. 1, 2007Search and Sensemaking47 Examples of Other Independent Variables Serial positions of links Serial positions of patches Correct patch scent values Correct link scent values Number of competing patches Lots of other variables…

48. Feb. 1, 2007Search and Sensemaking48 Over View of Results Mean number of clicks range from 1.1 to over 10 Percent solvers ranged from 100% to 25% Correlation between clicks and percent solvers =.93 Three Variables Account for 51% of Variance –Total competing links in incorrect patches –Weak scent correct link (yes, no) –Unfamiliar correct link (yes, no)

49. Feb. 1, 2007Search and Sensemaking49 Competing Links in Incorrect Patches Increase Task Difficulty (p<.0001) n=236 tasks n=88 tasks n=33 tasks n=112 tasks

50. Feb. 1, 2007Search and Sensemaking50 Weak-Scent Correct Links Increase Task Difficulty (p<.0001) n=271 tasks n=53 tasks n=15 tasks n=112 tasks

51. Feb. 1, 2007Search and Sensemaking51 Unfamiliar Correct Links Increase Task Difficulty (p<.0001) n=268 tasks n=25 tasks n=112 tasks n=56 tasks

52. Feb. 1, 2007Search and Sensemaking52 Conclusions Competing Links in Incorrect Patches Unfamiliarity Means End Analysis

53. Feb. 1, 2007Search and Sensemaking53 Competing Links in Incorrect Patches Patch contains 0 high or moderate scent links –Little or no impact in performance –Leave patch quickly –Low cost of going to new patch on same page Patch contains 2 or more high or moderate scent links –Large impact on performance Garden path effects determined by TOTAL number of competing links in incorrect patches

54. Feb. 1, 2007Search and Sensemaking54 Number of Moderate or High Scent Incorrect Links vs. Observed Clicks

55. Feb. 1, 2007Search and Sensemaking55 Unfamiliarity Two Levels –Incomplete knowledge of meaning of a superordinate concept e.g., Anthropology, college students Only 5% of links in Blackmon et al. (2002,…) –No knowledge of a word Specialized technical terms for college students Many superordinate terms for individuals with 3rd to 6th grade reading skills

56. Feb. 1, 2007Search and Sensemaking56 Impact of Unfamiliarity 5% of link terms (1st year college reading level) –Problem solving skills necessary to make inferences –Infer meaning from other terms in patch –Minor problem 50% of link terms (3rd grade reading level) –Locus of unfamiliarity is superordinate concepts –Too many unknown words to be able to make inferences –May lack necessary problem solving skills

57. Feb. 1, 2007Search and Sensemaking57 Scent Following Is Means-End Analysis Limited by ability to comprehend links Information Foraging by scent following is a simple form of Means-Ends Analysis (MEA) –Exhibits all MEA’s failure modes Classical problem solving literature is directly relevant –Operator subgoaling Navigation goals Patch enrichment activities

58. Feb. 1, 2007Search and Sensemaking58 CoLiDeS + SNIF-ACT CoLiDeS –A webpage defines multiple patches –Both search and sensemaking require comprehension Search requires comprehension of structure and content of a web page Sensemaking entails comprehension of retrieved information

59. Feb. 1, 2007Search and Sensemaking59 CoLiDeS + SNIF-ACT (Cont.) SNIF-ACT –ACT-R cognitive architecture Learning mechanisms Perceptual-motor mechanisms –Satisficing decision cycle Click on best link found so far* Process another link Click on back button

60. Feb. 1, 2007Search and Sensemaking60 Contact Information Marilyn Blackmon –Marilyn.Blackmon@Colorado.eduMarilyn.Blackmon@Colorado.edu Muneo Kitajima –kitajima@ni.aist.go.jpkitajima@ni.aist.go.jp Peter Polson –Peter.Polson@Colorado.eduPeter.Polson@Colorado.edu