From icons to mixed realities: how external representations help us learn Yvonne Rogers SLIS/Informatics

Overview How can we design graphical representations to facilitate learning and understanding? How do they work? –Public symbols –Icons –Diagrams –Animations –Multimedia –Mixed realities

Claims about their added value Symbols can be universally understood without ever having been seen before Icons are easy to infer meaning and remember Diagrams can make relationships explicit Animations can convey dynamic changes Multimedia can ‘enrich’ learning Mixed realities provoke reflection

My research agenda What are the main cognitive benefits of different forms of external representations? Can we develop a generalisable theoretical account of how external representations work? Can this provide design principles and guidance? Can we show whether the theory and principles are valid and useful?

Sometimes a (good) picture can be worth a 1000 words

Sometimes not… 1/5 people thought it signified a kite flying area 3/5 people though it signified foot-and-mouth disease

And sometimes they are just confusing…

Public symbols Can be easily understood Can be amusing Can be ambiguous Context is important May need to learn a ‘picture’ grammar

Icons - why use instead of text? Many objects and operations can be succinctly expressed in iconic form –e.g., folders, files, trash can, to save, to delete Many functions are poorly expressed in an iconic form –e.g., format, preferences, to quit

Icons Can be easily understood and meaning remembered Can be easy to scan and locate in a set Can be confusing if look alike - difficult to discriminate Direct mappings best, followed by symbolic and then abstract

Diagrams Why use diagrams instead of text?

Diagrams Help us understand difficult and complex subjects (e.g. an ecosystem) Make relationships and interdependencies between elements explicit Make it easier to ‘read off’ information and see connections

Animated diagrams What is the added value? Same benefits as static ones but also… Can show invisible and dynamic processes

A static and animated diagram of how the heart works

Is a moving picture worth 1000 static ones? + Animations can be effective at displaying dynamic changes + They make explicit what is implicit in diagrams - Difficult to keep track and ‘digest’ - Learners simply watch and assume they have understood

Is Multimedia any better? Multimedia typically consists of graphics, text, speech and links Provides more representations and more information

An early example of multimedia: Mayo’s Health CD-ROM

text...

search...

...graphics

Multimedia Early educational CD-ROMs were designed as electronic encyclopedias Knowledge represented in multiple media based on a resource/book model Full potential of multimedia not being realized How could it be designed to facilitate the learning process?

Designing it to teach the ‘difficult stuff’ An example of ‘difficult stuff’: reasoning about a complex system using an abstraction

Example: What would happen if the mice were wiped out? To answer need to: know factual knowledge (who eats what) understand canonical links (arrows) reason about knock on-effects (using representations in temporal and spatial order)

Pondworld multimedia environment Goal was to enable children to reason more effectively about ecosystems using food web diagrams Get the children to relate explicitly their concrete knowledge with the abstract formalism To achieve this by enabling them to manipulate and construct dynamic food web diagrams and to get feedback on their actions

Multimedia can be much more… Different elements can be conveyed simultaneously Can be interacted with in novel ways Studies of children using PondWorld showed significant improvement in ability to reason using diagram

External cognition approach (Scaife and Rogers, 1996) explain how we interact with external representations Identify the cognitive benefits and costs of using different representations provide principles to inform their design

Computational offloading the extent to which different external representations ( 5, five, |||||, V) reduce the amount of cognitive effort required to solve equivalent problems 45 x Forty-five times twelve IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII XXXXV 12 IIIIIIIIIIII XII the amount of cognitive work the learner is expected to do when using the external representation

“Dynalinking” linking of multiple representations so that making changes in one makes changes in the other (e.g., Pondworld) Dynalinking allows people to 'read' and connect information explicitly, helping them to understand the relationship between the two

Mixed realities What is Mixed Reality? - physical world is combined with virtual world Coupling the familiar with the unexpected Offers new opportunities for learning and understanding, e.g., reflection

Mixing colors Very familiar activity using wet paints and brushes What happens when you provide new ways of mixing colors? –e.g. combining digital paints to cause a physical object to change color What do young children do? How do they reflect upon the ‘old-new’ experiences?

The Chromarium project (i)Wet paints (ii)Combine digital colors (iii)Combine physical cubes with colored sides to mix digital paints on a wall (iv)Combine digital colors to cause a physical windmill’s sails to rotate and change color

(ii) Move digital colored discs on top of each other, resulting in new digital colors

(iii) Place two physical cubes next to each other on a tabletop resulting in new digital color appearing on wall

(iv) Move digital windmill sails causing physical windmill sails to move and change color

Findings from Chromarium study Children were most inventive and creative when mixing colors using the physical cubes Encouraged more collaboration and explaining of the ‘what’ and ‘why’

Summary: From icons to mixed realities Many cognitive benefits of using external representations in learning –understanding –reasoning –problem-solving –reflecting But can’t just assume a picture is worth a 1000 words or an animation a 1000 pictures or a mixed reality is better than a physical reality… Understand the properties of each and apply principles of design

Thanks All my colleagues and collaborators, especially those at the Interact Lab, Sussex University and those working on the Equator IRC in the UK