Translating between multiple representations: discussion EARLI Symposium Padua, August, 2003 Richard Cox HCT Group, University of Sussex

Overview Definitions, framing issues, the translation ‘problem space’ How do learners come to understand relationships between representations? How we might discriminate between occasions when a learner is engaged in ER comprehension activity and when s/he is successfully reasoning with an ER ? Supporting users at various levels of expertise

Framing issues… 1. Translation “..convey..from one place to another..”, “..turn from one language into another retaining the meaning” (OED) 2. Consider dimensions in which translation possible between (multiple) ERs - extending Palmer (1978) to the 2 ER case...

Represented world (Palmer, 78) abcd

Representing world (ER) taller than --> longer than a bcd

Representing worlds taller than--> longer than taller than --> shorter than a bcd a b cd ER 1 ER 2

a bcd a b cd Both ERs represent same aspects of represented world - YES Same aspect of representation doing the mapping - YES

a bcd a b c d Both ERs represent same aspects of represented world - YES Same aspect of representation doing the mapping - NO taller than -> longer thantaller than --> points to

a bcd a bcd Both ERs represent same aspects of represented world - NO Same aspect of representation doing the mapping - YES taller than -> longer than WIDER than -> longer than

ab cd a b c d Both ERs represent same aspects of represented world - NO Same aspect of representation doing the mapping - NO WIDER than - -> LARGER than taller than --> points to

same aspect of world? same ER mapping? YESNO YES NO

a 3rd dimension - multimodality... GRAPHICAL LINGUISTIC

and a 4th, 5th, 6th... static vs dynamic ERs (interactive, animated...) not to mention other factors... –degree of abstraction of ER(s) –type of graphical representation (picture, non-picture) –number of ERs co-present in display –3D, VR... combinatorial explosion of translation routes!

How do learners come to understand relationships between representations? Depends on relationships and translation ‘route’ If same aspect of world represented - can explore ER-to-ER links (dynalinks), perhaps without reference to the world and maybe solely via other (constraining) ERs If different aspects of world represented - then ER-to-ER translation is usually via represented world

How do learners come to understand relationships between representations? Learning may not be so much `more’ or `less’ but different in mode –exploration of dynalinks, constraining ERs and correlated displays ---> implicit learning (eg. Berry & Broadbent, 1984) –need to consider learning outcomes... how important is need for explicit (verbalizable) knowledge? if implicit knowledge acquired - assess differently? –interaction of methodology with learning outcome - `think aloud’ likely to keep knowledge and reasoning explicit (with extra cognitive load perhaps)

`Staring at’ versus using - telling the difference.. how can we discriminate between when a learner is engaged in a) ER comprehension activity and b) reasoning with the ER ? 1. correlate ER behaviour with performance (as many researchers do!) here rich process data very useful - innovative methodologies pay off... –DEMIST: logging, dynalinking, analyser –SDE: RFV, think aloud and logging –SimQuest: think-aloud and cognitive load probes

`Staring at’ versus using - telling the difference.. 2. Assess learner’s background knowledge of ERs beforehand examine mental organisation of ER knowledge - category organisation differs in poor versus better reasoners (Cox & Grawemeyer, 2003) and in people with different backgrounds (Lohse et al 1994)

Supporting users at various levels of expertise The `intermediate learner’ effect (du Boulay et al., Seufert) –some knowledge (of ERs, of domain) necessary to benefit from support –support should segue into integration thru’ degrees of compartmentalisation and complexity of domain k. and ERs need to know more about what `knowing an ER’ actually means - characterise partial knowledge and misconceptions (eg viewing graphs as pictures der Meij & de Tong) - look at different levels of cognitive processing system - ie. perceptual, semantic memory organisation and output levels

Supporting users at various levels of expertise by making aspects of world that are modelled explicit (very nice colour coding in der Meij & de Jong) assists learner to assess redundancy level of MERs in display, directs attention in dynalinking and exploration sometimes a tension between learner-centred design and traditional ‘ease of use’ HCI... ask who is system for, should there be different versions for different users?

References p.1of 2 Berry, D.C. & Broadbent, D.E. (1984) On the relationship between task performance and associated verbalizable knowledge. The Quarterly Journal of Experimental Psychology, 36a, Cox, R. & Grawemeyer, B. (2003) The mental organisation of external representations. Proceeding of the European Cognitive Science conference (EuroCogSci03), Osnabruck, Sep.

References p. 2 of 2 Lohse, G.L., Biolsi, K., Walker, N. & Rueter, H. (1994) A classification of visual representations. Communications of the ACM, 37(12), Palmer, S.E. (1978) Fundamental aspects of cognitive representation. In E. Rosch & B.B.Lloyd (eds) Cognition and categorization. Hillsdale, NJ: Lawrence Erlbaum Associates. Other works cited were papers presented at this symposium