1. A crowd-sourcing framework for automated visualization evaluation
Radu Jianu (with Mershack Okoe)
Florida International University

2. A proof of concept: GraphUnit
GraphUnit is a web-service that supports a semi-automatic evaluation of graph visualizations; it leverages crowdsourcing, a library of graph-tasks linked to benchmark graph data sets, automatic user-study deployment, and result collection and analysis. If a web-visualization is available, GraphUnit lets you configure and deploy an online user study of it in about 30 minutes.

3. Running a study with GraphUnit

4. Running a study with GraphUnit

5. Running a study with GraphUnit

9. Running a study with GraphUnit
Return to see intermediate and final results (charts, R-statistical analyses, raw results); GraphUnit will create a unique URL where you can access them

10. Fine print

11. GraphUnit’s ingredients
Interface methods
Task taxonomy
Specific benchmark data set
Study design choices
Prototypical tasks
“Are two nodes connected?”
Input: two nodes
Answer: yes/no
Study protocol
(e.g., alternate conditions)
Task instances
“Are nodes N1 and N2 connected?”
Input: N1, N2
Answer: Yes
Data analysis

12. The next step: VisUnit Set and group module Multidimensional module
Graph Module
Prototypical tasks
Benchmark data and task instances
Interface methods
Vector field module
Easy to create such modules (so define the tasks, define interface methods, etc). Create the infrastructure for this.
Glyph module

13. The next step: VisUnit Problems:
Real data-sets are not ‘pure’; they are often combinations of graph data, multidimensional data, spatio-temporal data
Solution: prototypical tasks are still valuable; stream-line the process of registering one’s own data set and task instances
How do we design VisUnit to be sufficiently flexible to replicate existing user study designs?
E.g.: Answers might not come in widget form -> use the interface methods to accept any answer from a user (e.g., a selection of a data object, or a click on the screen)

14. Why now?
Controlled user study designs and data analyses become more standardized
Lam et al. (2012)

15. Motivation
Controlled user study designs and data analyses are standardized
Lam et al. (2012)
Evaluated tasks are becoming increasingly standardized into task taxonomies
Graphs (Lee 2006), Multidimensional (Valiati 2006), Group+Graph (Saket 2014)

16. Motivation
Controlled user study designs and data analyses are standardized
Lam et al. (2012)
Evaluated tasks are becoming increasingly standardized into task taxonomies
Graphs (Lee 2006), Multidimensional (Valiati 2006), Group+Graph (Saket 2014)
Online crowdsourcing has been validated as a mechanism to run user studies; crowdsourcing implements ‘human macros’
Heer and Bostock (2010), Kittur et al. (2008), Bernstein et al. (2010)

17. Motivation
Controlled user study designs and data analyses are standardized
Lam et al. (2012)
Evaluated tasks are becoming increasingly standardized into task taxonomies
Graphs (Lee 2006), Multidimensional (Valiati 2006), Group+Graph (Saket 2014)
Online crowdsourcing has been validated as a mechanism to run user studies; crowdsourcing implements ‘human macros’
Heer and Bostock (2010), Kittur et al. (2008), Bernstein et al. (2010)
Visualizations are migrating to the web
D3, WebGL

18. Benefits Evaluating visualizations is important (and lacking?):
Lam et al. (2012) showed that 42% of 850 major vis papers between 2002 and 2012 reported an evaluation.
Conducting user studies is challenging, time consuming, and expensive
Standardized benchmark evaluation can lead to comparable results and in turn to user study results that can aggregate over time
Can we move evaluations from after the design process to within the design process?

19. Questions?