evaluation of a new web design for the dictionary portal owid Institute for the German Language, Mannheim evaluation of a new web design for the dictionary portal owid Sascha Wolfer, Carolin Müller-Spitzer

Eye-tracking and research into dictionary usage Eye-tracking methodology still not widely used in research into dictionary usage. Simonsen (2011), Tono (2011), Lew, Grzelak, & Leszkowicz (2013) Widely used: Log files (number of visits to specific entries). Eye-tracking could provide another type of data. … not necessarily better data. Log files: Overall look-up preferences for specific words. Eye-tracking: Process of researching in a dictionary and perception of specific entries and their structure.

Aims of study Evaluate a new way of data collection for research into online dictionary usage. Evaluate new web design for the OWID dictionary portal ("Online German Lexical Information System"). This study is also reported in Müller-Spitzer, C., Michaelis, F., & Koplenig, A. (2014). Evaluation of a new web design for the dictionary portal OWID. In: Müller-Spitzer (Ed.): Using Online Dictionaries. S. 207-228. – de Gruyter, 2014. (Lexicographica: Series Maior 145).

This talk Introduction to OWID. Method and procedure of the study. Results. Discussion and critical evaluation.

Online dictionary portal OWID

General access structure OWID Integrating various academic electronic dictionary resources of contemporary German into one internet portal. general monolingual dictionary neologisms discourse dictionaries proverbs fixed multiword expressions … OWID General access structure B A C …

dictionary overview entry section navigation bar (keyword list) Kurze Vorstellung des neuen Designs. dictionary overview entry section

Erklären, was es mit den Farb-Kästen auf sich hat.

Eye-tracking study

Research questions Is it easy to recognize OWID as a dictionary portal, i.e. that it integrates different dictionaries? Does the colour scheme work for identifying the individual dictionaries? Is specific lexicographic information easy to locate with the new layout? Wiederaufgreifen der Forschungsfragen. Dieses Mal konkreter formuliert und auf die einzelnen Designelemente bezogen.

Method / APPARATUS 38 participants, aged 20-30, 10 € compensation Sensomotoric Instruments (SMI) RED eye-tracker remote system 60 Hz In cooperation with the Mannheim EyeLab at the University of Mannheim (promotional image by SMI)

Procedure Stimuli: Screenshots of the new OWID layout. Live version would be much more naturalistic (browsing, scrolling). However: Eye-tracking studies with dynamic content are much more expensive! Instruction 1 Instruction 2 Screenshot + Eye-tracking Screenshot + Eye-tracking Questions Question

Results 1 Is OWID being identified as a dictionary portal? Stichwortliste zieht bei zweiter Instruktion deutlich mehr Aufmerksamkeit auf sich. „Gain an initial overview.“ „Which dictionaries are included?“

Results 1 Which of the following dictionaries is included in OWID? Neologism dictionary Woxikon dictionary of synonyms PONS German orthography elexiko Guilt discourse 1945-1955 Dictionary of fixed multiword expressions OBELEX bibliography Only 32% chose all and only the correct answers. Note, that the instruction did not mention that a question will be asked … … and the distractors are quite popular dictionaries.

Discussion 1 Dictionary list attracted more attention when participants were asked to find out which dictionaries are included. Problems: No contrastive condition with old design. No contrastive condition with other positions of the dictionary list.

Results 2 Does the colour scheme work for associating entries with their respective dictionary? „Which headword is described?“ „Which dictionary is the entry from?“

Results 2 Does the colour scheme work for associating entries with their respective dictionary? Mehr Aufmerksamkeit auf Wörterbuchliste; Verbindung zu Stichwortliste (links) „Which headword is described?“ „Which dictionary is the entry from?“

Discussion 2 After the first instruction („which headword is described?“), participants mainly concentrated on the center of the screen. After the second instruction („which dictionary is it from?“), fixations were shifted to the dictionary list on the right and the headword list on the left. Problems: No contrastive condition with old design. No contrastive condition with alternative layouts.

Results 3 Is specific lexicographic information easily accessible (here: style information)? Diese Folie soll verdeutlichen, dass Heatmaps nicht immer geeignet sind, um einen bestimmten Effekt zu zeigen. Das ist ein eher methodologischer Punkt. Die nächste Folie (scanpaths) ist da schon aussagekräftiger. Erst dort kann man den Lerneffekt wirklich sehen, weil im Sequenz-Plot schneller zur relevanten Information gefunden wird. First Stimulus Second Stimulus

Results 3 Is specific lexicographic information easily accessible (here: style information)? First fixation after 5 seconds (average) First fixation after 2.5 seconds First Stimulus Second Stimulus

Discussion 3 Clear learning effect between first and second stimulus. Aggregation methods for gaze data (e.g., heatmaps) are not suitable for all effects! Problems: Learning effect is no real surprise. Without a contrastive condition (e.g., old design), we cannot state that the new design is really better for finding relevant information.

GENERAL DISCUSSION / Critical evaluation Our study showed that the given tasks were solvable with the new OWID layout. Design decisions were not outright confusing for users. Interesting gaze behavior while exploring a dictionary entry can be observed. However, main problem remains: no contrastive conditions were introduced! Old vs. new design. Different layouts crossed with different tasks.  Methodologically sound experimental design is highly important – not only in eye-tracking experiments.

Thank You! Thank you for your attention!