Research Design and Evaluation Colin Ware

Goals for empirical research Uncover fundamental truths and test theories (late stage science). Discover the nature of the world. Find out about how people interact with information artifacts. Find out how people respond to a particular display. (survey of recreational boaters). Does theory apply in practice. E.g. good continuity applied to flow vis. Compare two (or more) display or interaction methods. Is this method adequate?

Continued Compare two entire systems. (GIS 1, vs GIS 2) Measure task performance (e.g how fast can we find a certain class of patterns). Determine user preferences. _________________________________

Non-empirical contributions Invent a better interface for some particular domain (Social network analysis, interactive flow visualization) Proof of concept prototype Constructivist analysis Invent a better evaluation method Invent a new “theory” (information foraging theory)

Methods for evaluation Measure errors (how accurate are we) Target detection (rapid response) Binary data Method of adjustment Measure time to perform a task How much work can we do per unit time. Rating scales (Likert scales) Turn judgments into numbers. Task analysis + Structural analysis. A cognitive theory is constructed with observation as evidence (constructivism). (e.g. narrative). Goes hand-in-hand with proor of concept applications

Semi-stuctured interviews. Expose the user to a problem + solution in a systematic way. (+questions) Wizard of Oz Cognitive walkthroughs. Have the user perform a task and talk aloud as they do it.

Spiral design method Must document the design process

Spiral design One way of proceeding is though a spiral design method. Construct a rapid prototyping environment (tool kit) – demo or die Construct a prototype Evaluate (and document findings) Refine prototype Evaluate (and document findings)

Methodological Issues If you want to measure time/error data Best to have a simple representative task Justify it. If you want to get opinions (ratings, interviews). Domain experts often ideal. But they may have established ideas – they like the status quo.

Information interface design Determine data and problem Determine tasks Create mappings from data to representation such that important patterns are “transparent”. Create interfaces for rapid task performance Create support for hypothesis

Things to consider Aim for important targets. Keep it simple Maximize payoffs for (your) time expended Be sure to have control conditions. Defacto standard Alternative display methods. Choose dependent variable ranges. Remove all un-necessary variables (e.g. background). Have sufficient observations.

Statistics Means. Standard deviations. Standard error of the mean =  /sqrt(N); 2se ~ 95% confidence intervals. ANOVA e.g. color and motion.

The End

Velocity glyphs Curved Compact Show direction clearly Easily readable: more looks “greater” Capable of 100 levels

Visualization design pattern Patterns of common low-level tasks Relating patterns- e.g. bioinformatics. Pattern searching Finding paths Finding groups Finding target symbols