A crowd-sourcing framework for automated visualization evaluation Radu Jianu (with Mershack Okoe) Florida International University http://vizlab.cs.fiu.edu/graphunit/

A proof of concept: GraphUnit GraphUnit is a web-service that supports a semi-automatic evaluation of graph visualizations; it leverages crowdsourcing, a library of graph-tasks linked to benchmark graph data sets, automatic user-study deployment, and result collection and analysis. If a web-visualization is available, GraphUnit lets you configure and deploy an online user study of it in about 30 minutes. http://vizlab.cs.fiu.edu/graphunit/

Running a study with GraphUnit http://vizlab.cs.fiu.edu/graphunit/

Running a study with GraphUnit

Running a study with GraphUnit http://vizlab.cs.fiu.edu/graphunit/

Running a study with GraphUnit Return to see intermediate and final results (charts, R-statistical analyses, raw results); GraphUnit will create a unique URL where you can access them http://vizlab.cs.fiu.edu/graphunit/

Fine print

GraphUnit’s ingredients Interface methods Task taxonomy Specific benchmark data set Study design choices Prototypical tasks “Are two nodes connected?” Input: two nodes Answer: yes/no Study protocol (e.g., alternate conditions) Task instances “Are nodes N1 and N2 connected?” Input: N1, N2 Answer: Yes Data analysis

The next step: VisUnit Set and group module Multidimensional module Graph Module Prototypical tasks Benchmark data and task instances Interface methods Vector field module Easy to create such modules (so define the tasks, define interface methods, etc). Create the infrastructure for this. Glyph module

The next step: VisUnit Problems: Real data-sets are not ‘pure’; they are often combinations of graph data, multidimensional data, spatio-temporal data Solution: prototypical tasks are still valuable; stream-line the process of registering one’s own data set and task instances How do we design VisUnit to be sufficiently flexible to replicate existing user study designs? E.g.: Answers might not come in widget form -> use the interface methods to accept any answer from a user (e.g., a selection of a data object, or a click on the screen) http://vizlab.cs.fiu.edu/graphunit/

Why now? Controlled user study designs and data analyses become more standardized Lam et al. (2012) http://vizlab.cs.fiu.edu/graphunit/

Motivation Controlled user study designs and data analyses are standardized Lam et al. (2012) Evaluated tasks are becoming increasingly standardized into task taxonomies Graphs (Lee 2006), Multidimensional (Valiati 2006), Group+Graph (Saket 2014) http://vizlab.cs.fiu.edu/graphunit/

Motivation Controlled user study designs and data analyses are standardized Lam et al. (2012) Evaluated tasks are becoming increasingly standardized into task taxonomies Graphs (Lee 2006), Multidimensional (Valiati 2006), Group+Graph (Saket 2014) Online crowdsourcing has been validated as a mechanism to run user studies; crowdsourcing implements ‘human macros’ Heer and Bostock (2010), Kittur et al. (2008), Bernstein et al. (2010) http://vizlab.cs.fiu.edu/graphunit/

Motivation Controlled user study designs and data analyses are standardized Lam et al. (2012) Evaluated tasks are becoming increasingly standardized into task taxonomies Graphs (Lee 2006), Multidimensional (Valiati 2006), Group+Graph (Saket 2014) Online crowdsourcing has been validated as a mechanism to run user studies; crowdsourcing implements ‘human macros’ Heer and Bostock (2010), Kittur et al. (2008), Bernstein et al. (2010) Visualizations are migrating to the web D3, WebGL http://vizlab.cs.fiu.edu/graphunit/

Benefits Evaluating visualizations is important (and lacking?): Lam et al. (2012) showed that 42% of 850 major vis papers between 2002 and 2012 reported an evaluation. Conducting user studies is challenging, time consuming, and expensive Standardized benchmark evaluation can lead to comparable results and in turn to user study results that can aggregate over time Can we move evaluations from after the design process to within the design process? http://vizlab.cs.fiu.edu/graphunit/

Questions? http://vizlab.cs.fiu.edu/graphunit/