1. Analyzing the Integration of Figures and Text Using Eye Movements Jason Rubinstein With Eileen Kowler, Nick Ross, Elio Santos, Min Zhao University of Delaware and Rutgers

2. Introduction Figures + Text Same “page” vs. different “page” Semantic integration of information over long timescales Building up a knowledgebase Between saccades “rapid movements of the eyes between positions of rest” Between “page turns” Meaning People tend to limit memory load when possible The L–D differences in the ventilatory and metabolic variables occurred even between 15-min epochs of identical activity levels, whether these were below or above the daily mean (Fig. 5). VE/VO2 in the D remained higher than in the L, by 13% when activity was low and 9% at the higher activity level. The results of this analysis did not depend on the animal ’s activity immediately preceding the epoch analyzed, as there was no correlation between the L –D difference in VE for iso-activity epochs and the level of activity prior to the epochs analyzed.

3. Ballard, Hayhoe, & Pelz, 1995

4. Epelboim & Suppes, 2001

5. Question What strategies do people use to extract meaning from figures and text (in combination)?

6. Why? Real world Textbooks Educational material Standardized tests SATs Newspapers Websites Instruction manuals Captions

9. The percentage statements during which the participant looked at the person was used as an indicator of gaze seeking. There was not an effect of gaze cueing and no interaction between gaze cueing and full or partial instructions. For full instructions, the person was fixated less often when he provided gaze cues, than when he did not provide gaze cues. For partial instructions, gaze cues were often sought.

12. Which of the following is true? A. The percentage of fixations on the person was high for both full and partial instructions independently of whether the gaze cues were provided by the person or not. B. Fixations on the person were more frequent following full instructions with or without gaze cues. C. Gaze cues increased the percentage of fixations on the person for both full and partial instructions. D. For partial instructions, fixations on the person were rare whether gaze cues were provided or not.

13. Question What strategies do people use to extract meaning from figures and text (in combination)? Patterns in Rule-governed transitions Revisits “Effector” (motor action) Eye (simultaneous) or arm (sequential)

15. VIDEO

16. Results Revisit figure after reading key words in text Words that refer to the legend, specific aspects of figure, etc

17. Results Revisit figure after reading key words in text Words that refer to the legend, specific aspects of figure, etc On average… Switch more in simultaneous condition (!)

18. Number of visits to figureNumber of visits to text

19. Results Revisit figure after reading key words in text Words that refer to the legend, specific aspects of figure, etc On average… Switch more in simultaneous condition (!) Though “easier” (cognitively) in alternating Less “work”

20. Results Revisit figure after reading key words in text Words that refer to the legend, specific aspects of figure, etc On average… Switch more in simultaneous condition (!) Though “easier” (cognitively) in alternating Less “work” Periphery -- planning

21. Results Revisit figure after reading key words in text Words that refer to the legend, specific aspects of figure, etc On average… Switch more in simultaneous condition (!) Though “easier” cognitively in alternating Less “work” Periphery -- planning Spend more time in text in both conditions Longer visits + more time overall

22. Proportion of time spent in figureProportion of time spent in text

23. Results Revisit figure after reading key words in text Words that refer to the legend, specific aspects of figure, etc On average… Switch more in simultaneous condition (!) Though “easier” cognitively in alternating Less “work” Periphery -- planning Spend more time in text in both conditions Longer visits + more time overall However, proportion of overall time between simultaneous and alternating about the same

24. Average length of visits to figure (s)Average length of visits to text (s)

25. Results Revisit figure after reading key words in text Words that refer to the legend, specific aspects of figure, etc On average… Switch more in simultaneous condition (!) Though “easier” cognitively in alternating Less “work” Periphery -- planning Spend more time in text in both conditions Longer visits + more time overall However, proportion of overall time between simultaneous and alternating about the same Reading… Images vs. words?

26. What next? MORE Participants Stimuli Shorter trials/harder questions Accuracy of various strategies Specifics of transitions Word in text → which part of figure? Vice versa Model revisits Simultaneous trials Saccade triggers stimuli Periphery

27. ?

28. Thank you! Eileen Kowler Nick Ross, Elio Santos, Min Zhao Matthew Stone Joe, Henry, Avery, Rachel, Lara The Academy Everyone else