The Web Changes Everything How Dynamic Content Affects the Way People Find Online Jaime Teevan Microsoft Research (CLUES) with S. Dumais, D. Liebling, E. Adar, J. Elsas, and R. Hughes

Information Artifacts Change

Digital Dynamics Easy to Capture

The Web Changes Everything JanuaryFebruaryMarch April May JuneJuly August September Content Changes People Revisit JanuaryFebruaryMarch April May JuneJuly August September Today’s Browse and Search Experiences But, ignores …

The Web Changes Everything JanuaryFebruaryMarch April May JuneJuly August September Content Changes Large scale Web crawl over time – Revisited pages Unique users, visits/user, time between visits 55,000 pages crawled hourly for 18+ months – Judged pages (relevance to a query) 6 million pages crawled every two days for 6 months

Measuring Web Page Change Summary metrics – Number of changes – Time between changes – Amount of change Change curves – Fixed starting point – Measure similarity over different time intervals Knot point

Measuring Within-Page Change DOM structure changes Term use changes – Divergence from norm cookbooks salads cheese ingredient bbq – Staying power in page Time Sep. Oct. Nov. Dec.

JanuaryFebruaryMarch April May JuneJuly August September Revisitation on the Web Content Changes People Revisit JanuaryFebruaryMarch April May JuneJuly August September Revisitation patterns – Log analysis – Toolbar logs for revisitation – Query logs for re-finding – User survey for intent What’s the last Web page you visited?

Measuring Revisitation Summary metrics – Unique visitors – Visits/user – Time between visits Revisitation curves – Histogram of revisit intervals – Normalized Time Interval

Four Revisitation Patterns Fast – Hub-and-spoke – Navigation within site Hybrid – High quality fast pages Medium – Popular homepages – Mail and Web applications Slow – Entry pages, bank pages – Accessed via search engine

Search and Revisitation How to measure? Repeat query (33%) – hci research at stanford Repeat click (39%) – – Query  stanford hci Big opportunity (43%) – Navigational (24%)

Repeat Clicks for Repeat Queries Minutes Days Weeks

How Revisitation and Change Relate JanuaryFebruaryMarch April May JuneJuly August September Content Changes People Revisit JanuaryFebruaryMarch April May JuneJuly August September Why did you revisit the last Web page you did?

Possible Relationships Interested in change – Monitor Effect change – Transact Change unimportant – Find Change can interfere – Re-find

Understanding the Relationship Compare summary metrics Revisits: Unique visitors, visits/user, interval Change: Number, interval, similarity 2 visits/user 3 visits/user 4 visits/user 5 or 6 visits/user 7+ visits/user Number of changesTime between changesSimilarity 2 visits/user visits/user visits/user or 6 visits/user visits/user

Comparing Change and Revisit Curves Three pages – New York Times – Woot.com – Costco Similar change patterns Different revisitation – NYT: Fast – Woot: Medium – Costco: Slow (news, forums) (retail) Time

Within-Page Relationship Page elements change at different rates Pages revisited at different rates Resonance can serve as a filter for interesting content

Building Support for Web Dynamics JanuaryFebruaryMarch April May JuneJuly August September Content Changes JanuaryFebruaryMarch April May JuneJuly August September People Revisit

DiffIE Changes to page since your last visit DiffIE toolbar

Interesting Features of DiffIE Always on In-situ New to you Non-intrusive

EXAMPLES OF DiffIE IN ACTION

Expected New Content

Monitor

Unexpected Important Content

Serendipitous Encounters

Unexpected Unimportant Content

Understand Page Dynamics

Attend to Activity

Edit

Unexpected Unimportant Content Attend to Activity Edit Understand Page Dynamics Serendipitous Encounter Unexpected Important Content Expected New Content Monitor Expected Unexpected

Monitor

Find Expected New Content

Methods for Studying DiffIE Feedback buttons Survey – Prior to installation – After a month of use Logging – URLs visited – Amount of change when revisited Experience interview In situ Representative Experience Longitudinal

People Revisit More Perception of revisitation remains constant – How often do you revisit? – How often are revisits to view new content? Actual revisitation increases – First week: 39.4% of visits are revisits – Last week: 45.0% of visits are revisits Why are people revisiting more with DiffIE? 14%

Revisited Pages Change More Perception of change increases – What proportion of pages change regularly? – How often do you notice unexpected change? Amount of change seen increases – First week: 21.5% revisits changed by 6.2% – Last week: 32.4% revisits changed by 9.5% DiffIE is driving visits to changed pages 51+% 17% 8%

Change by Page Type Perceptions of change reinforced Pages that change a lot  change more Pages that change a little  change less News pages Message boards, forums, news groups Search engine results Blogs you read Pages with product information Wikipedia pages Company homepages Personal home pages of people you know Reference pages (dictionaries, yellow pages, maps) Change little Change a lot

The Web Changes Everything JanuaryFebruaryMarch April May JuneJuly August September Content Changes JanuaryFebruaryMarch April May JuneJuly August September Web content changes: Page-level, term-level People revisit and re-find Web content Relating revisitation and change allows us to – Identify pages for which change is important – Identify interesting components within a page Explicit support for Web dynamics can impact how people use and understand the Web People Revisit

Change Adar, Teevan, Dumais & Elsas. The Web changes everything: Understanding the dynamics of Web Content. WSDM ’09 (Best Student Paper). Elsas & Dumais. Leveraging temporal dynamics of document content in relevance ranking. WSDM ’10 (to appear). Revisitation Adar, Teevan & Dumais. Large scale analysis of Web revisitation patterns. CHI ’08 (Best Paper). Teevan, Adar, Jones & Potts. Information re-retrieval: Repeat queries in Yahoo’s logs. SIGIR ’07. Tyler & Teevan. Large scale query log analysis of re-finding. WSDM ’10 (to appear). Relationship Adar, Teevan & Dumais. Resonance on the Web: Web dynamics and revisitation patterns. CHI ’09. DiffIE Teevan, Dumais, Liebling & Hughes. Changing how people view changes on the Web. UIST ’09. Teevan, Dumais & Liebling. A longitudinal study of how highlighting Web content change affects people’s Web interactions. CHI ’10 (under submission). Thank you. Jaime Teevan

EXTRA SLIDES

Web Dynamics JanuaryFebruaryMarch April May JuneJuly August September Content Changes Number of studies of change [2, 7, 10, 20] Frequency and degree of change characterized Visited pages are more likely to change [2]

Web Dynamics JanuaryFebruaryMarch April May JuneJuly August September Content Changes People Revisit JanuaryFebruaryMarch April May JuneJuly August September People revisit on the Web a lot – Over half of page visits are revisits [2, 22] – Over a third of searches are for re-finding [23] Revisitation relates to change – 66% of revisits are to changed pages [2] – 20% of the content changes [2] – Revisiting often motivated by change [2, 15] – Change interferes with revisiting [21, 23]

Systems That Expose Web Change Historical access to pages – Internet Archives (archive.org) Subscription to change – RSS, Web slices – Monitoring support [15] In-situ awareness of change – symbols – Dynamo [3], Difference Engine [9], WebCQ [17]

HOW DiffIE WORKS

DiffIE DiffIE Architecture Web Cache Toolbar ComponentComparison Component IE Client Machine

Toolbar Compare to older versions Status message Feedback buttons See previous version Hide highlighting

Cache Web page representation – Hash of text in leaf DOM nodes – Some information about DOM structure Cache multiple versions of pages visited Small footprint (15-20KB) – Exact duplicates stored as pointer files – Cap count (only 6% of pages visited >5 times) Privacy preserving

Thread title# Posts# Views This is the title of a thread3 Something else is discussed here!1582 Read This Thread Today217 0 Comparison Component Highlighted: Additions, changes Not highlighted: Moves, deletions Look for text not present in previous version Semantic grouping of tabular information 2 Same

Alternate Comparison Change Deletion Addition Movement A A B B D D C C E E F F A A B B C C D D E E E E D D Node has fewer children Node has more children, child new Node has new child, child present Node has same children, child changes

Alternate Comparison Change Deletion Addition Movement Highlighted: Additions, changes Not highlighted: Moves, deletions Node has fewer children Node has more children, child new Node has new child, child present Node has same children, child changes

UNDERSTANDING DiffIE

DiffIE Received Positively Feedback buttons – 51% of unsolicited feedback positive (v %) Survey – Prior to use: 2.83 / 5 – After a month: 3.20 / 5 Experience interview (conditioned on change) – 61% positive – 18% neutral – 21% negative 13%

Feedback Positive – “It helps make a new block of content stand out on a long page containing mostly old content.” – “Please don't take away my DiffIE!” Negative – Bugs: “Seemed to stop working on my system.” – Performance: “Had the feeling that my browser took too much time to load sites.” – Robust prototype important

Reported Experience with DiffIE

Higher value Lower value Unexpected Unimportant Content Attend to Activity Edit Understand Page Dynamics Serendipitous Encounter Unexpected Important Content Monitor Expected New Content

Performance Highlighting shown on page load event Appears 10s to 100s of milliseconds after load Does not interfere with browsing experience Often appears after interaction begins Notification of delay important