Measuring Learning During Search: Differences in Interactions, Eye-Gaze, and Semantic Similarity to Expert Knowledge Florian Groß 1 28. Mai 2019 1

Outline Introduction Methodology Results Discussion 2 Florian Groß 2

Introduction Information Seeking – “a process, in which humans purposefully engage in order to change their state of knowledge” Marchioni Tie between information search and learning in prior works Consider learning as integral part of information search process How to measure learning? Learning as changes in verbal knowledge from before and after a search session Marchioni -> information search driven by higher lvl human needs.  Consider Information seeking changes state of searchers knowledge… TIE, consider… -> in this work, measure learning not new in IR, learning… -> Our interest is in the learning that takes place at the remembering and factual knowledge level 3 Florian Groß 3

Background Learning outcomes from search are a good evaluation measure of IR systems Typically requires collecting explicit responses from users Which measurement techniques of learning? Which techniques work outside laboratory? Eye tracking to measure change in searchers learning 2 punkt: assessing learning …, eye tracking: prior work used eye gazing to assess differences in lvls of users domain knowledge 4 Florian Groß 4

Goal Construct learning measures that … require minimal input from users do not require users to answer topic-specific comprehension tests do not expose user to topic of search in pre-task assessment attempt to assess a users true knowledge level with minimal scope for guessing 5 Florian Groß 5

Approach 30 participants perform search on the web Two types of learning measure Topic-independent measure Measure based on semantic similarity with expert vocabulary Expectation: searchers that invest more effort and consume more result pages learn more 6 Florian Groß 6

Experimental Design Participants asked to search for health-related information on internet Pre-screened participants Native-level English familiarity Non-expert topic familiarity Uncorrected 20/20 vision (min. problems with eye tracking) All participants reported Using internet for over an hour per day Daily usage of google majority of the participants had been using Google for longer than seven years, and considered themselves proficient in searching for information online 7 Florian Groß 7

Task Each participant performed two search tasks in counterbalanced order On health related topic Simulated work-task approach  Triggering realistic information-need find useful information for helping a family member and a friend. The tasks were designed to be complex, and contained multiple facets 8 Florian Groß 8

Interface Customized version of Google Browser with additional sidebar No advertisements Search engine result page showed seven results per page  Eye fixation tracked on each individual result Browser with additional sidebar Current search task Bookmarking section – save URLs of relevant pages Notes section – note relevant text from web page All other (content) pages shown in their true form Recorded interaction with computer(eye gaze, mouse clicks, key strokes) 9 Florian Groß 9

Procedure Florian Groß 10 Green/yellow patches in (a) are eye-tracking fixation heatmaps, The circle with number in (b) is an eye-fixation with duration, Each experimental session started with the assessment of participant’s working memory capacity and health literacy, Next, the participants performed a training task to familiarize themselves with the custom user interfaces (bookmarking and notetaking) and the study procedure, list as many words or phrases as you can on the topic of the search task. 10 Florian Groß 10

Procedure Assessment of working memory capacity and health literacy Training task to familiarize with interface Pre-task knowledge assessment Searching for information using google Visiting search result webpages Bookmarking webpages that contain relevant information Taking notes (not available in post-task) Assessing knowledge change through post-task questionnaire Measure workload Literacy = bildung, because the task was on a medical topic, NASA-TLX (Task Load Index), for working memory capacity : using memory span task; eHealth Literacy scale for health literacy; 11 Florian Groß 11

Measures All measures calculated for each user task pair Knowledge change Free-recall as many words/phrases on topic Relative difference in number of items before and after task Vocabulary of expert words Semantic similarity pre/post-task recall with expert vocabulary Difference and ratio between post task- and pre task similarity with expert vocabulary pre_exp_sim = sim(pre_task, expert), post_exp_sim = sim(post_task, expert), 12 Florian Groß 12

Measures Eye-tracking Search Interaction Reflects process of reading Calculated on serps, content pages and relevant content pages Eye fixation = reading Eye regression = moving back to fixate on previous word Calculate total duration, count of fixation, number of eye regression Search Interaction Number of visited serps and content pages Dwell time on page type, number of queries/ query reformulations  Search effort = search interaction + acquiring text from web pages Relevant content pages because expected participants to learn most from such pages, search interaction(visiting serps/content pages, entering queries, clicking links), acquiring(nr + duration of reading fixations, length of reading sequence) ; Search-effort is operationalized as a two part, multiple-component construct, composed of the above two groups of measures, SI and ET 13 Florian Groß 13

Results Split KC measure in HI and LO group based on median-split of score Higher knowledge change score Less frequent/ uncommon words in queries Less amount of reading on webpages Reported higher mental workload No difference between groups Search interactions Working memory capacity Online health literacy Expectation: searchers that invest more effort and consume more result pages learn more  higer kc with less reading. 14 Florian Groß 14

Limitations and future work Using just two tasks of similar nature Performing data analysis at the task level Uniform group of participants (Recruitment?) Short-time frame of experimental session Use wider range of tasks More diverse participant samples Additional individual difference tests Assessment of verbal skills Multiple session study to measure learning over a longer period of time Assessment = beurteilung, 16 females; mean age 24.5 years, they did not mention how they recruited their participants, maybe had bigger study in mind but wanted to check results on smaller study first 15 Florian Groß 15

Discussion 16 Florian Groß 16

Backup (crowdsourced and evaluated with doctor) Angular similarity 17 Florian Groß 17