1. The Human Processing and Memory
Human Computer Interaction, 2nd Ed.
Dix, Finlay, Abowd, and Beale
Chapter 1

2. Model Human Processor + Attention Recall, “purely and engineering abstraction”
Sensory store
Rapid decay “buffer” to hold sensory input for later processing
Perceptual processor
Recognizes symbols, phonemes
Aided by LTM
Cognitive processor
Uses recognized symbols
Makes comparisons and decisions
Problem solving
Interacts with LTM and WM
Motor processor
Input from cog. proc. for action
Instructs muscles
Feedback
Results of muscles by senses
Attention
Allocation of resources

3. Overview
Will look at elements of human information processing from a slightly different orientation than “engineering abstraction”
A bit more fine grained analysis, following from psychological studies
But, it is these psychological studies from which the “engineering abstraction” is derived
3 stage model of human memory
Iconic buffer, STM, LTM
Models of LTM
Reasoning
Problem solving

4. Model Human Processor + Attention Recall, “purely and engineering abstraction”
Sensory store
Rapid decay “buffer” to hold sensory input for later processing
Perceptual processor
Recognizes symbols, phonemes
Aided by LTM
Cognitive processor
Uses recognized symbols
Makes comparisons and decisions
Problem solving
Interacts with LTM and WM
Motor processor
Input from cog. proc. for action
Instructs muscles
Feedback
Results of muscles by senses
Attention
Allocation of resources

5. 3-Stage Model of Human Memory
Sensory (here, iconic) memory – “very” short term memory
lasts 1-2 seconds, infinite capacity
Short-term memory (Working memory)
lasts ~ 18 seconds, holds 1.75 (7+/-2 items)
Long-term memory
infinite capacity; short of damage is permanent
Recall vs. Recognition (Remember vs. Know)
Retrieval cues
Will demonstrate later in class …

6. “Executive” - Attention
Central “executive” controls tasking
Pays, or allocates, attention
Bandwidth of attention is limited
Tasks that require the same resources interfere with one another
Attention is both a low-level and high-level property of vision

7. Sensory Memory: “Very” Short Term Memory
Sensory buffers for stimuli received through senses
iconic memory: visual stimuli
echoic memory: aural stimuli
haptic memory: tactile stimuli
Examples
“sparkler” trail
stereo sound
Continuously overwritten – demo follows

8. A Test – of Visual Iconic Memory
Will present figure briefly (~1/2 second)
Try to remember as many elements as you can
Write them down

12. The Phenomenon
After presentation, did you continue to “see” the items?
Some purely physiological based “seeing”:
Afterimage
Bleaching of pigments
“bright, or colored, stuff”
But also, there is a more “memory-based” image (process further downstream in memory system)
Iconic memory
“dark, or veridical, stuff”
Reading from the iconic buffer

13. Reading from the Iconic Buffer, 1
Typically can list 3 – 7 items named
Short lived visual, or iconic, buffer
holds the image for a second or two
Read images and place in STM
3-stage model
Can get about 5-7 items until run out of short term (working) memory capacity
Limitation of 5-7 comes from:
Decay of iconic memory
Rate can read from visual buffer
Capacity of working memory
In each fixation between saccadic eye movements, image of world captured
Set of miscellaneous symbols

14. Reading from the Iconic Buffer, 2
Again, Limitation of 7 comes from:
Decay of iconic memory
Rate can read from visual buffer
Capacity of working memory
From each image,
brain must identify objects,
match them with objects previously perceived, and
take information into working memory for symbolic analysis
Search light model of attention (for vision)
Visual information is acquired by pointing fovea at regions of visual field that are interesting
Then using a scanning process in which objects are read from an image buffer from more extensive processing
Set of miscellaneous symbols

15. Attention Spotlight metaphor Visual dominance:
Spotlight moves serially from one input channel to another
Can focus attention (and perceptual processor) on only one input channel at a time
Location in visual field, voice in auditory field, …, anything
Visual dominance:
Easier to attend to visual channels than auditory channels
All stimuli within spotlighted channel are processed in parallel
Whether you want to or not
Can cause “interference” - demo

16. Say the Colors of the Words
Easy enough – didn’t take too long

17. Say the Colors of the Words
Took longer … Stroop effect
For design:
Choose secondary characteristics of display to reinforce message

18. Again, Human Memory Stages
Sensory (here, iconic) memory
lasts 1-2 seconds, infinite capacity
Short-term memory (Working memory)
lasts ~ 18 seconds, holds 1.75 (7+/-2 items)
Long-term memory
infinite capacity; short of damage is permanent
Recall vs. Recognition (Remember vs. Know)
Retrieval cues

19. Short-term memory (STM)
“Scratch-pad” (or buffer) for temporary recall
rapid access ~ 70ms
rapid decay ~ 200ms
limited capacity - 7± 2 chunks
Chunking, recoding, etc.
affects amount of information retained, entering LTM

20. Example - Chunking
HEC ATR ANU PTH ETR EET

21. Long-term Memory (LTM)
Repository for all our knowledge
slow access ~ 1/10 second
slow decay, if any
huge or unlimited capacity
Episodic and semantic memory
Episodic (episodes): Serial memory of events
Semantic (“meanings”): Structured memory of facts, concepts, skills
Also, procedural and declarative memory
“Processes” vs. “facts”

22. LTM – Models of Semantic Memory
Semantic memory structure
Contains LTM knowledge of world
Provides access to information
Generic knowledge -- specific details lost
Represents relationships between bits of information
Important for rule-based behavior
Supports inference
Many models, theories, accounts, schemata proposed
Semantic network model (example next slide):
E.g., Inheritance – child nodes inherit properties of parent nodes
Relationships between bits of information explicit
Supports inference through inheritance
Other Models (examples follow):
Scripts, frames, production rules

23. Early Model of Semantic Memory Collins and Quillian
Collins & Quillian’s Teachable Language Comprehender
Semantic memory is organized as a network of interrelated concepts
Each concept is represented as a node
Concepts are linked together by pathways
Economy of representation
Activation of one concept spreads to interconnected nodes
Remind you of anything from computer science?

24. Early Model of Semantic Memory Collins and Quillian
Collins & Quillian’s Teachable Language Comprehender

25. Early Model of Semantic Memory Collins and Quillian
Spreading Activation
Working memory is activated LTM
When a concept becomes active, activation spreads to all other interconnected nodes
Activation spreads to all related nodes
How do you evaluate sentences like “Is a robin is an animal”?

26. Early Model of Semantic Memory Collins and Quillian
Spreading Activation
Activation spreads from each of the concept nodes (Robin & Animal)
When two spreading activations meet, an intersection is formed
Robins ==> BIRD <== Animals
If no intersection, relatively fast no
If intersection, decision stage operates to determine if sentence is valid
Is a robin an animal?

27. Tests of Spreading Activation
Sentence verification task
Time to respond yes or no
Takes time for activation to spread
Greater distances ==> longer RT
Verification time for items 0, 1, and 2 links

28. But, it’s not that simple ...
E.g. typicality effects - how many links separate:
A canary is a bird?
A robin is a bird?
A chicken is a bird?
An ostrich is a bird?
But, RT varied - less typical birds took longer than more typical birds

29. Semantic Relatednes and Semantic Priming
Semantic relatedness
Spreading activation between related concepts
Activation of one concept partially activates semantically related concepts
Semantic priming
Stimulus 1 ==> Stimulus 2
(Prime) ==> (Probe)
Test spreading activation by manipulating semantic relationship between prime & probe
Concepts linked by spreading activation
Prime: Probe:
Doctor Nurse
Bread Butter
Doctor Butter
Bread Nurse
Sometimes prime facilitates processing

30. Semantic Relatednes Recall, semantic relatedness
Spreading activation between related concepts
Activation of one concept partially activates semantically related concepts
So, can focus on relatedness, without explicitly indicating links

31. Spreading Activation - Fin
Semantic priming commonplace
Can exploit in design
Indeed, in design can exploit all information about how human operates
Spreading activation is thought to be automatic
Governed by data-driven aspects of processing
How do expectancies affect semantic access?
Automatic vs Conscious Strategies (attentional)
Fast vs Slow
Effortless vs Effortful
Benefits vs Costs & Benefits

32. Fyi – Another semantic network

33. Models of LTM – Frames, or Schemata
COLLIE
Fixed
breed of: DOG
type: sheepdog
Default
size: 65 cm
Variable
colour
Information organized in “memorial data structures”
Schemata
Stored framework or body of knowledge
Conceptual framework for interpreting information
Biased information processing to relate new material to what we already know
Alters way we perceive things
Individual differences in perception and memory
Frames
Slots in structure instantiated with values for instance of data
Type–subtype relationships
DOG
Fixed
legs: 4
Default
diet: carniverous
sound: bark
Variable
size: colour

34. Script for a visit to the vet
Models of LTM - Scripts
Model of stereotypical information required to interpret situation
Script has elements that can be instantiated with values for context
Script for a visit to the vet
Entry conditions: dog ill
vet open
owner has money
Result: dog better
owner poorer
vet richer
Props: examination table
medicine
instruments
Roles: vet examines
diagnoses
treats
owner brings dog in
pays
takes dog out
Scenes: arriving at reception
waiting in room
examination
paying
Tracks: dog needs medicine
dog needs operation

35. Models of LTM - Production Rules
Representation of procedural knowledge.
Condition/action rules
if condition is matched
then use rule to determine action.
IF dog is wagging tail,
THEN pat dog
IF dog is growling,
THEN run away

36. LTM - Storage of information
LTM much studied in psychology:
Rehearsal
information moves from STM to LTM
Total time hypothesis
amount retained proportional to rehearsal time
Distribution of practice effect
optimized by spreading learning over time
Structure, meaning and familiarity
information easier to remember

37. LTM - Forgetting Decay Interference
information is lost gradually but very slowly
Interference
new information replaces old: retroactive interference
old may interfere with new: proactive inhibition
So, ... may not forget at all, memory is selective …!
Also, affected by emotion – can subconsciously `choose' to forget

38. LTM - Retrieval Should be familiar from heuristics Recall
information reproduced from memory can be assisted by cues, e.g. categories, imagery
Recognition
information gives knowledge that it has been seen before
less complex than recall - information is cue

39. Thinking – Cognitive Processing
Humans reason, process information, like, well, humans
Recall, any theory is an abstraction and, thus, captures some elements of phenomenon, and misses others
Question is …
Is the account (theory, model) useful in the context and for the purpose for which it is used?
Basic forms of reasoning, or, forming inferences, are useful in understanding broad outlines of human cognition
Deduction
Induction
Abduction
Problem solving
Gestalt
Problem Space
Analogy
Skill acquisition

40. Reasoning Deduction: Induction: Abduction:
derive logically necessary conclusion from given premises
e.g., If it is Friday, then she will go to work - It is Friday, therefore she will go to work
Logical conclusion not necessarily true:
e.g., If it is raining, then the ground is dry - It is raining, therefore the ground is dry
Induction:
Generalize from cases seen to cases unseen
e.g., All elephants we have seen have trunks - therefore all elephants have trunks.
Unreliable (but useful):
Can only prove false not true
Abduction:
Reasoning from event to cause
e.g., Sam drives fast when drunk.
If I see Sam driving fast, assume drunk.
Unreliable:
can lead to false explanations

41. FYI - Induction vs. Deduction
Induction: Make observations first, then draw conclusions
Organized data survey (structured analysis, visualization) of raw data provide basis for interpretation process
Interpretation process will produce knowledge that is being sought
Experience of individual scientist (observer) is crucial
Important: selection of relevant data, collection method, and analysis method
Data mining is an important knowledge discovery strategy
ubiquitious data collection, filtering, classification, and focusing is crucial
Deduction: Formulate hypothesis first, then test hypothesis
Via experiment and accept/reject
Data collection more targeted than in induction
Only limited data mining opportunities
Mueller, 2003

42. Problem Solving
Process of finding solution to unfamiliar task using knowledge
Complex, time consuming process
Selections not immediately obvious
May require many steps
May involve insight
May use analogy
Solutions often counterintuitive
Several theories, or accounts
Gestalt
Problem solving both productive and reproductive
Productive draws on insight and restructuring of problem
Attractive but not enough evidence to explain “insight” etc.
Move away from behaviourism and led towards information processing theories
Others: Insight, Functional fixedness, Analogy

43. Problem Solving Cycle
One schema – consider “task performance”

44. Insight
Early - Kohler (a Gestalt psychologist) in Canary islands in WWI
Studied problem solving in chimpanzees
Sultan and the Banana:
Learned how to get banana with longer pole
Then given shorter poles that wouldn’t reach
Flash of “insight”, Sultan put the poles together
Sudden perception of useful or proper relations
Solutions will sometimes “spring to mind”
Pieces fall into place
First attempts to solve don’t work
Production hindered by unwarranted assumptions.
Insight occurs when the assumption is removed
Strayer, Utah:

45. Problem solving (cont.)
Problem space theory
Problem space comprises problem states
Problem solving involves generating states using legal operators
Heuristics may be employed to select operators e.g. means-ends analysis
Operates within human information processing system e.g. STM limits etc.
Largely applied to problem solving in well-defined areas e.g. puzzles rather than knowledge intensive areas

46. Problem solving (cont.)
Analogy
Analogical mapping:
novel problems in new domain?
use knowledge of similar problem from similar domain
Analogical mapping difficult if domains are semantically different
Skill acquisition – e.g., “expert” performance
Skilled activity characterized by chunking
lot of information is chunked to optimize STM
Conceptual rather than superficial grouping of problems
Information is structured more effectively

47. Individual Differences
Long term – Sex, physical and intellectual abilities
Short term – Effect of stress or fatigue
Changing – Age
Dix says ask:
Will design decision exclude section of user population?
(or, more generally) How does design differentially affect sections of the population?
i.e., Universal Usability

48. The “Model Human Processor + Attention” is Similar to Ware (2004) Model
We’ll look at one more model of cognitive (and visual) processing
All are in fact much the same, but focus on different goals

49. The “Model Human Processor + Attention” is Similar to Ware (2004) Model
We’ll look at one more model of cognitive (and visual) processing
All are in fact much the same, but focus on different goals
Card et al. model was developed in context of predicting user performance
E.g., set parameters and perform simulation
Ware’s model includes much the same elements, but focuses on those which are most relevant for processing of visual information in context of task performance

50. The “Model Human Processor + Attention” is Similar to Ware (2004) Model
Sensory store
Rapid decay “buffer” to hold sensory input for later processing
Perceptual processor
Recognizes symbols, phonemes
Aided by LTM
Cognitive processor
Uses recognized symbols
Makes comparisons and decisions
Problem solving
Interacts with LTM and WM
Motor processor
Input from cog. proc. for action
Instructs muscles
Feedback
Results of muscles by senses
Attention
Allocation of resources

51. A Model of Perceptual Processing Quick Overview
What we do is design information displays!
An information processing (the dominant paradigm) model
“Information” is transformed and processed
Physical light does excite neurons, but at this “level of analysis” consider information
Gives account to examine aspects important to visualization
Here, clearly, many neural subsystems and mapping of neural to ip is pragmatic
In spirit of visualization as evolving discipline, yet to develop its theories, laws, …
Stage 1: Parallel processing to extract low-level properties of the visual science
Stage 2: Pattern perception
Stage 3: Sequential goal-directed processing

52. Stage 1: Parallel Processing to Extract Low-level Properties of Visual Scene
(Very first) neurons fire
Visual information 1st processed by
large array of neurons in eye
primary visual cortex at back of brain
Individual neurons selectively tuned to certain kinds of information
e.g., orientations of edges or color of light
Evoked potential experiments
In each subarea large arrays of neurons work in parallel
extracting particular features of environment (stimulus)

53. Stage 1: Parallel Processing to Extract Low-level Properties of Visual Scene
At early stages, parallel processing proceeds involuntarily
Largely independent of what choose to attend to (though not where look)
Is rapid,
If want people to understand information fast, should present in way so is easily detected by these large, fast computational systems in brain
Stage 1 processing is:
Rapid and parallel
Entails extraction of features, orientation, color, texture, and movement patterns
“transitory”, only briefly held in iconic store
Bottom up, data-driven

54. Stage 2: Pattern Perception
Rapid processes
Divide visual field into regions and simple patterns, e.g.,
Continuous contours
Regions of same color
Regions of same texture …
“Active”, but not conscious processes
Specialized for object recognition
Visual attention and memory
E.g., for recognition must match features with memory
Task performing will influence what perceived
Bottom up nature of Stage 1, influenced by top down nature of Stage 3

55. Stage 2: Pattern Perception
Specialized for interacting with environment
E.g., tasks involving eye-hand coordination
“Two-visual system hypothesis”
One system for locomotion and eye-hand coordination
The “action system”
One system for symbolic object manipulation
The “what system”
Characteristics:
Slow serial processing
Involvement of both working (vs. iconic) and long-term memory
Both bottom up and top down
More emphasis on arbitrary aspects of symbols than Stage 1
Top-down processing
Different pathways for object recognition and visually guided motion

56. Stage 3: Sequential Goal-Directed Processing
At highest level of perception are the objects held in visual memory by demands of active attention
To use an external visualization, we construct a sequence of visual queries that are answered through visual search strategies
Only a few objects can be held at a time
They are constructed from available patterns providing answers to the visual queries

57. Stage 3: Sequential Goal-Directed Processing
They are constructed from available patterns providing answers to the visual queries
E.g., if use a road map to look for a route, the visual query will trigger a search for connected red contours (representing major highways) between two visual symbols (representing cities)
Are other subsystems, as well
Visual object identification process interfaces with the verbal linguistic subsystems of the brain so that words can be connected to images
The perception-for-action subsystem interfaces with the motor systems that contril muscle movements

58. End .