1. Measuring Learning During Search: Differences in Interactions, Eye-Gaze, and Semantic Similarity to Expert Knowledge
Florian Groß 1
28. Mai 2019 1

2. Outline
Introduction
Methodology
Results
Discussion 2
Florian Groß 2

3. Introduction
Information Seeking – “a process, in which humans purposefully engage in order to change their state of knowledge” Marchioni
Tie between information search and learning in prior works
Consider learning as integral part of information search process
How to measure learning?
Learning as changes in verbal knowledge from before and after a search session
Marchioni -> information search driven by higher lvl human needs.  Consider Information seeking changes state of searchers knowledge… TIE, consider… -> in this work, measure learning not new in IR, learning… -> Our interest is in the learning that takes place at the remembering and
factual knowledge level 3
Florian Groß 3

4. Background
Learning outcomes from search are a good evaluation measure of IR systems
Typically requires collecting explicit responses from users
Which measurement techniques of learning?
Which techniques work outside laboratory?
Eye tracking to measure change in searchers learning
2 punkt: assessing learning …, eye tracking: prior work used eye gazing to assess differences in lvls of users domain knowledge 4
Florian Groß 4

5. Goal Construct learning measures that …
require minimal input from users
do not require users to answer topic-specific comprehension tests
do not expose user to topic of search in pre-task assessment
attempt to assess a users true knowledge level with minimal scope for guessing 5
Florian Groß 5

6. Approach 30 participants perform search on the web
Two types of learning measure
Topic-independent measure
Measure based on semantic similarity with expert vocabulary
Expectation: searchers that invest more effort and consume more result pages learn more 6
Florian Groß 6

7. Experimental Design
Participants asked to search for health-related information on internet
Pre-screened participants
Native-level English familiarity
Non-expert topic familiarity
Uncorrected 20/20 vision (min. problems with eye tracking)
All participants reported
Using internet for over an hour per day
Daily usage of google
majority of the participants had been using Google for longer than seven years, and considered themselves proficient in searching for information online 7
Florian Groß 7

8. Task
Each participant performed two search tasks in counterbalanced order
On health related topic
Simulated work-task approach
 Triggering realistic information-need
find useful information for helping a family member and a friend. The tasks were designed to be complex, and contained multiple facets 8
Florian Groß 8

9. Interface Customized version of Google Browser with additional sidebar
No advertisements
Search engine result page showed seven results per page
 Eye fixation tracked on each individual result
Browser with additional sidebar
Current search task
Bookmarking section – save URLs of relevant pages
Notes section – note relevant text from web page
All other (content) pages shown in their true form
Recorded interaction with computer(eye gaze, mouse clicks, key strokes) 9
Florian Groß 9

10. Procedure Florian Groß 10
Green/yellow patches in (a) are eye-tracking fixation heatmaps, The circle with number in (b) is an eye-fixation with duration, Each experimental session started with the assessment of participant’s working memory capacity and health literacy, Next, the participants performed a training task to familiarize
themselves with the custom user interfaces (bookmarking and notetaking) and the study procedure, list as many words or phrases as you can on the topic of the
search task. 10
Florian Groß 10

11. Procedure Assessment of working memory capacity and health literacy
Training task to familiarize with interface
Pre-task knowledge assessment
Searching for information using google
Visiting search result webpages
Bookmarking webpages that contain relevant information
Taking notes (not available in post-task)
Assessing knowledge change through post-task questionnaire
Measure workload
Literacy = bildung, because the task was on a medical topic, NASA-TLX (Task Load Index), for working memory capacity : using memory span task; eHealth Literacy scale for health literacy; 11
Florian Groß 11

12. Measures All measures calculated for each user task pair
Knowledge change
Free-recall as many words/phrases on topic
Relative difference in number of items before and after task
Vocabulary of expert words
Semantic similarity pre/post-task recall with expert vocabulary
Difference and ratio between post task- and pre task similarity with expert vocabulary
pre_exp_sim = sim(pre_task, expert), post_exp_sim = sim(post_task, expert), 12
Florian Groß 12

13. Measures Eye-tracking Search Interaction Reflects process of reading
Calculated on serps, content pages and relevant content pages
Eye fixation = reading
Eye regression = moving back to fixate on previous word
Calculate total duration, count of fixation, number of eye regression
Search Interaction
Number of visited serps and content pages
Dwell time on page type, number of queries/ query reformulations
 Search effort = search interaction + acquiring text from web pages
Relevant content pages because expected participants to learn most from such pages, search interaction(visiting serps/content pages, entering queries, clicking links), acquiring(nr + duration of reading fixations, length of reading sequence) ; Search-effort is operationalized as a two part, multiple-component construct, composed of the above two groups of measures, SI and ET 13
Florian Groß 13

14. Results
Split KC measure in HI and LO group based on median-split of score
Higher knowledge change score
Less frequent/ uncommon words in queries
Less amount of reading on webpages
Reported higher mental workload
No difference between groups
Search interactions
Working memory capacity
Online health literacy
Expectation: searchers that invest more effort and consume more result pages learn more  higer kc with less reading. 14
Florian Groß 14

15. Limitations and future work
Using just two tasks of similar nature
Performing data analysis at the task level
Uniform group of participants (Recruitment?)
Short-time frame of experimental session
Use wider range of tasks
More diverse participant samples
Additional individual difference tests
Assessment of verbal skills
Multiple session study to measure learning over a longer period of time
Assessment = beurteilung, 16 females; mean age 24.5 years, they did not mention how they recruited their participants, maybe had bigger study in mind but wanted to check results on smaller study first 15
Florian Groß 15

16. Discussion 16
Florian Groß 16

17. Backup (crowdsourced and evaluated with doctor) Angular similarity17
Florian Groß 17