1. This slide deck is part of a presentation given by Saul Greenberg at a UIST 2007 panel on Evaluating user interface systems research (see paper by Dan Olson in ACM DL)
Its content was based on a paper written by Saul Greenberg and Bill Buxton, available at:

2. I need to
do an evaluation

3. What’s the problem?

4. It won’t get accepted if I don’t. Duh!

5. Dogma
Course Introduction

6. Good HCI
Course Introduction

7. Time Learning Brian Gaines Breakthrough Replication Empiricism Theory
major creative advances
Replication
ideas mimicked and altered; increasing experience
Empiricism
lessons formulated as empirical design rules
Theory
underlying causal systems
Automation
accepted
Maturity
used routinely
Brian Gaines

8. Evaluation sweet spot Interface validation and refinement*
Breakthrough
Replication
Empiricism
Theory
Automation
Maturity
Interface validation and refinement*
*70% quantitative, 30% qualitative

9. Concepts, Sketches Breakthrough Replication Empiricism Theory
Automation
Maturity
Concepts, Sketches

10. Memex Bush
Concept

11. Unimplemented and untested design. Microfilm is impractical
Unimplemented and untested design. Microfilm is impractical. The work is premature. Resubmit after you build and evaluate this design.

12. Sketchpad Sutherland
Working idea

13. Complex controls, overly simplistic drawings Recommend you evaluate a trained draftsman using Sketchpad vs. paper and pencil with real drafting tasks

14. Augment Engelbart
Deployed idea

15. While ‘tested’ by your team, this is not an objective validation.

16. What to do?

17. Reflection
Bill Buxton, Sketching the User Experience

18. Cultural Appropriation
Breakthrough
Replication
Empiricism
Theory
Automation
Maturity
Cultural Appropriation

19. Hypertext

24. What to do?

25. Case / field studies Cultural probes Reflection Extreme uses…

26. Evaluation
Course Introduction

27. More Appropriate Evaluation
Course Introduction

28. We decide what is good research

29. Cultural appropriation
Time
Learning
Breakthrough
Breakthrough
Replication
Replication
Empiricism
Empiricism
Theory
Theory
Automation
Automation
Maturity
Maturity
But is this lack of tools really a bottleneck?
Well, in the 1980s, Computer Scientist Brian Gaines introduced a model of how science technology develops over time, and how people learn from this development
It unfolds from <READ>
Breakthrough -> Invention
Replication -> Research
Empiricism -> Product Innovation
Theory -> Product Lines
Automation -> Low cost products
Maturity -> Throw away products
Sketches
Interface validation
Cultural appropriation
Brian Gaines

30. grouplab.cpsc.ucalgary.ca/~papers/