Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How to integrate images and words The Narrative thread

Consider that hieroglyphs gave way to more abstract symbols Why turn back the clock? -17K years

Imagens Pavio’s dual coding theory Visual System Auditory System Verbal Visual Image Information From speech Text Information Associate structure Logogens Verbal responses Non-verbal responses

Capacity of visual working memory (Vogal, Woodman, Luck, 2001) Task – change detection Can see 3.3 objects Each object can have several attributes Orientation Shape size, texture 1 second

Capacity of verbal working memory Used to be though of a 7 +/- 2 It is now thought of as more a duration of ~ 2sec and a store of proto-verbal codes. Words have an enormously rich semantic structure Sometimes we have a choice Spatial prepositions: above, across, alongside, amidst, apart, around, back, behind, below, beneath, beside, between, beyond, by (out of approx 80: From Pinker)

Reasoning with words and graphics Get a set of numbers, such that 3 of them add to 15.

Can we have a visual language?

The nature of language Words are arbitrary Chomsky, grammar with innate deep structures. Common to computer languages Austin - Illocutionary force (speech acts) Critical period for language development But, being verbal is not essential to language development Sign languages for the deaf are the most perfect examples of visual language

Sign languages ASL Danish SLChinese SL Arose spontaneously Are not related to verbal languages Have grammar Become more abstract over time NSL Goldin-Meadow

What can we do with language? Description The ability to communicate procedures and sequences of operations – including logic – if, but, causes, do a then b then c A vehicle for thought when internalized Communicate goals, intentions (illucutionary acts). Make things happen Nit picking (mutual grooming)

“If there is no fresh trout at Whole Foods then go to the fishmarket On 19 th and get salmon.”

There are special brain areas for language Production Wernicke Broca Understanding Production Visual Cortex Auditory Cortex

Can we have a visual language? Yes – but …

If we don’t learn it early some some capabilities are best handled verbally

The visual system gives us Rapid recognition and pattern finding

AbstractionPattern Jane is Jim’s boss Jim is Joe’s boss Anne works for Jane Mark works for Jim Anne is Mary’s boss Anne is Mike’s boss Org chart

Visual and verbal pseudo-code While letters in stack Take a letter Put a stamp on it Put it in the ‘out tray’ Visual programming languages have a history of failure Data flow diagrams are defunct

Examples of visual languages Sanscrit Petri-nets Khoros

Sanscrit Sum integers from 1 to 3 for i = 1 to 3 do sum = sum + i;

Petri Net language Petri nets are stochastic – timed attributed (tokens on nodes, transitions)

Khoros

Attaching images and words Pointing, Diexis and the diectic gesture Can be a glance or a nod An elementary speech act Shown to disambiguate verbal communications Why the mouse is so powerful The basis of shared environments

Attaching words to images - use gestalt laws

Integrated pictures and words more Effective: Chandler and Sweller 1991

Narrative: Visualizations can be used to help tell a story Can something visual carry a narrative. In psychological terms the important thing is a leading of the thread of ideas (=attention). In data analysis the analyst has the narrative thread. In a presentation the audience is following the presenter’s thread. (actually a hybrid is best).

Narrative Can be accomplished with either images or words A leading of attention

Degrees of narrative Strong leading  Free exploration

XPlane

Recap Words for procedure, logic and abstract meanings/ images for pattern and structure Both words and images can sustain narratives (leading attention)

Emotional and motivational effects (Images can be more efficient) and automatic.

Gillnet Set Surface Buoy Lobster Trap Trawl Surface Buoy Landry 2002

Theory: Graphics and Words Words for procedural logic: conditionals, qualifiers, if-then else, while. + causality. Not the formal logic of math but a kind of concrete logic. (A vowel must have an odd number on the back).

Other kinds of gestures Beat gestures for emphasis Verb gestures showing how to do something McNeil Hand and mind

Issues in shared environments Speech + Pointer + Visuals – most important components Subtle ways of directing attention also important in meeting dynamics.

Peter