1. Erin Harris & Jerry Weinberg
Facilitator Induce Errors: The Effects of Social Facilitation and Electronic Monitoring on Usability Testing
Erin Harris & Jerry Weinberg
Department of Computer Science
Sue Thomas & Danica Geaslin
Department of Psychology

2. St. Louis Area Usability Labs

3. Electronic Performance Monitoring
Increases levels of stress
Decreases both the levels of productivity and quality of work done.

4. EPM Examples
A study involving 762 telecommunications workers found that those who were monitored reported higher levels of stress and anxiety.
Other studies have found that participants working on tasks requiring much thought and skill performed significantly worse under EPM while participants working on simple repetitive tasks performed better

5. Social Facilitation
Social psychology concept that the behavior of an animal, including humans, is affected by the presence of another.
An individual’s dominant response is enhanced while the non-dominate response is impaired.

6. Two Social Facilitation Theories
Mere Presence Theory
This theory states that the presence of other people alone increases one’s arousal
Evaluation-Apprehension Theory
This theory goes further to state that it is not the presence of another individual alone, but also the anticipation that the individual is assessing your performance that increases one’s anxiety.

7. Social Facilitation Examples
Cockroaches completing a maze
Simple mazes were completed much faster in the presence of other cockroaches.
Complex mazes were much more difficult to complete in the presence of other cockroaches.
Pool players’ shot-making ability
Good players increased their shot-making ability
71% to 80%
Less talented players dropped their percentage
36% to 25%

8. Research Hypothesis
The presence of a facilitator will increase the effects of social facilitation
These effects will be magnified by viewing the observation lab and meeting the design team due to the increased salience of the evaluation
These effects will also be magnified by being constantly reminded of the observation

9. SIUE’s Usability Lab

10. SIUE’s Usability Lab

11. Procedures
Each subject was informed that they were going to be videotaped and observed over a closed circuit camera
Each subject was given a series of 16 Microsoft Word tasks; 8 simple and 8 complex
Each task had a time limit
The number of errors per task was recorded

12. Example Tasks Simple Tasks Complex Tasks
Center and underline the title of the document.
Insert a page break at the end of the document
Complex Tasks
Add a center tab stop at the 3-inch mark from the left side of the screen.
When you add a header to Microsoft Word, the header is added to all pages of the document. Setup Word so that the header of the first page of the document is different from the rest of the pages.

13. The Experimental Set-up
Facilitator Group
Served as control group of the study
No Facilitator Group

14. The Experimental Set-up
Facilitator + Reminding Group
Frequent communication with the software design team
Facilitator + Design Team Group
Touring the lab and meeting the design team prior to the test

15. Results – Mean Errors
7.2
9.4
15.1
48.7
62.3
45.0
56.8
2 & 4 evaluation apprehension
Simple weren’t automatic ( 2 4)
Constant reminding – suppressed dominant response

16. Results – Mean Errors
35.82
35.98
27.98
26.10
2 & 4 evaluation apprehension
Simple weren’t automatic ( 2 4)
Constant reminding – suppressed dominant response

17. Results – ANOVA for Errors
Main effect for type of task
Significantly fewer errors were made on simple tasks (M = 9.74) than on the complex tasks (M = 53.2)
F(1, 96) = , p =
Main effect for group
The No Facilitator and Facilitator + Constant Reminding groups committed significantly fewer errors than the Facilitator and Facilitator + Design Team groups.
F(3, 96) = 3.50, p = .018.
No Facilitator
Facilitator +
Constant Reminding
Facilitator
Design Team
Mean Errors
27.98
26.10
35.82
35.98
The interaction between group and the type of task did not reach statistical significance
F(3, 96) = 2.10, p = .106

18. Results Social facilitation is present in usability testing
Simple tasks were not really simple
Simple tasks needed to be so well learned that they were automatic.
Constant reminding
Our constant reminding caused the users to explain their actions.
Providing these explanations caused the users to think harder about what they were doing, improving performance
Wright & Converse (1992) had similar findings when instructing users to provide explanations for their actions

19. Putting it into Practice
Minimize interjection into user’s awareness reduces the errors
Will this provide a clearer picture of usability?
Is it preferable to induce more errors?
Subjects’ Remarks
“More aware of the observers when I didn’t know what I was doing.”
“Am I going to flunk?”

20. Putting it into Practice
Evaluation apprehension is subconscious
Regardless of how much you try to put the user at ease, the feeling of evaluation is still present subconsciously
How should we interpret results when “White Coat Response” effects performance?
Subjects’ Remarks
“Out of sight, out of mind”
Didn’t mind having the facilitator there, because she had met me, but the other observers made her nervous

21. Putting it into Practice
The social-psychological context of the usability test can have a significant impact on a user’s performance.
Other psychological factors that may impact usability results
Different Lab Set-ups
Gender differences between the facilitator and the user
Different introductions into the process
Our study only measured user performance
Measure the user’s state anxiety