1. FlowVUS Zhanping Liu Shangshu Cai J. Edward Swan II
A 2D Flow Visualization User Study
Using
Explicit Flow Synthesis and Implicit Task Design
VisWeek 2011
FlowVUS
IEEE TVCG
University of Pennsylvania
Kentucky State University
University of California
at Santa Barbara
Mississippi State University
Lockheed Martin Corp.
Army Research Lab
Zhanping Liu
Shangshu Cai
J. Edward Swan II
Robert J. Moorhead II
Joel P. Martin
T. J. Jankun-Kelly
I would like to present our work on a 2D flow vis user study called FlowVUS.

2. FlowVUS Outline Brief Introduction Experimental Components
Synthetic Flow Datasets
Flow Visualization Techniques
Flow Analysis Tasks
 Explicit Flow Synthesis
 Diverse Evaluation Aspects
 Implicit Task Design
First I will give a brief introduction to flow vis user study.
Then I will describe in detail three experimental components of our user study.
Next I will talk about our test strategy.
Finally I provide the test results and conclude this talk with some future plans.
Test Strategy
Test Results
Concluding Remarks
VIS 2011
FlowVUS

3. FlowVUS Brief Introduction Flow Representation
Geometry-based / glyph-based
Texture-based / image-based
arrow plots, streamlines, pathlines, streak lines, time lines
stream ribbons, stream tubes, stream surfaces, streak surfaces, ……
graphical primitives rendered for a sparse or discrete representation
good survey by McLoughlin et al (EuroGraphics 09)
topology-based methods use graphical primitives for a representation
Roughly speaking from the visual representation perspective,
flow visualization techniques may be categorized into geometry-based and texture-based.
spot noise, LIC, UFLIC, LEA, IBFV, IBFVS, ISA, UFAC, ……
texture convolution / advection for a dense continuous representation
good survey by Laramee et al (Computer Graphics Forum 04)
VIS 2011
FlowVUS

4. FlowVUS Brief Introduction Flow Visualization User Study
NIH-NSF report on Visualization Research Challenges (Johnson etc 06)
different techniques may be advantageous in different aspects
only a few have been evaluated to determine their effectiveness
the best methods might not have been integrated into vis. systems
domain scientists may not yet have access to cutting-edge techniques
insufficient user feedback for visualization researchers and developers
 more user studies are needed to
examine flow representations
improve existing techniques
design innovative techniques
Different techniques may be advantages in different aspects.
The NIH-NSF report calls for more user studies to evaluate the effectiveness of flow vis techniques, to bridge the gaps between research, development, and deployment.
bridge the long-lasting gaps
between research, development, and deployment
VIS 2011
FlowVUS

5. FlowVUS Brief Introduction Flow Visualization User Study Previous work
2D flow visualization user study (Laidlaw et al, TVCG 05)
3D flow visualization user study (Forsberg et al, TVCG 09) ……
insufficient research on effective user study methodologies
There have seen only a few papers in this area.
One issue is that we have not seen sufficient research on effective user study methodologies.
VIS 2011
FlowVUS

6. distorted by various bias issues
Brief Introduction
Flow Visualization User Study
Previous work
given a user study framework or platform
for evaluating flow visualization techniques
distorted by various bias issues
the data collected and the analysis
results are distorted too, failing
to provide objective conclusions
flow visualization techniques
Without necessary anti-bias methodologies, the framework or platform would be very much distorted.
As a consequence the data we collect and the conclusions we draw would not be convincing.
VIS 2011
FlowVUS

7. FlowVUS Brief Introduction Flow Visualization User Study Previous work
2D flow visualization user study (Laidlaw et al, TVCG 05)
3D flow visualization user study (Forsberg et al, TVCG 09) ……
insufficient research on effective user study methodologies
There is more to a flow visualization user study than
the scenarios being considered
the techniques being evaluated
the flow features being examined
the specific yet usually ad-hoc conclusions being drawn
e.g., surface flows, volume flows, time-varying flows, ……
e.g., UFLIC, LEA, IBFV, IBFVS, ISA, UFAC, ……
e.g., separation, attachment, vortex core, periodic orbit, ……
In this sense, we cannot over-emphasize the importance of effective anti-bias methodologies compared to
the specific scenarios, flow vis techniques, and flow features considered in a user study as well as the usually ad-hoc conclusions.
VIS 2011
FlowVUS

8. FlowVUS Brief Introduction Flow Visualization User Study
 Conducting objective 2D flow visualization user studies
even with traditional and well-known techniques
remains an open problem
requires valid methodologies — an anti-bias platform
refines our understanding of some 2D flow vis. techniques
offers quantitative support for qualitative evidence or anecdotal
advice in terms of the effectiveness of flow vis. techniques
So conducting objective 2D flow vis user studies remains an open problem.
it helps formulate an anti-bias framework that is necessary for user studies with more complex configurations.
helps formulate a general framework
that is necessary for
carrying out convincing flow visualization user studies
with more complex configurations
VIS 2011
FlowVUS

9. FlowVUS Brief Introduction
Our 2D Flow Visualization User Study — FlowVUS
motivated by the necessity for and significance of
effective flow visualization user study methodologies
builds on Laidlaw et al’s work
features new strategies and important improvements
explicit flow synthesis
implicit task design
flow data bias
task design bias
Our user study was motivated by the necessity for and significance of effective anti-bias methodologies.
It builds on Laidlaw et al’s work, but demonstrates new strategies and important improvements.
Two major features are explicit flow synthesis and implicit task design.
VIS 2011
FlowVUS

10. equipped with anti-bias methodologies
Brief Introduction
Our 2D Flow Visualization User Study — FlowVUS
given a user study framework or platform
for evaluating flow visualization techniques
equipped with anti-bias methodologies
the data collected and the analysis
results are convincing, leading to
a better understanding of techniques
flow visualization techniques
Because the framework of our user study is equipped with effective anti-bias methodologies,
the data we collect and the conclusions we draw are convincing.
VIS 2011
FlowVUS

11. FlowVUS Brief Introduction
Our 2D Flow Visualization User Study — FlowVUS
Major contributions
explicit flow synthesis
 combats data-related bias by automatically generating many flows
with similar topological complexities but with different structures
implicit task design
 reduces task-related bias by designing sample-free pattern-based flow
analysis tasks that require thorough investigation of the flow direction
diverse evaluation perspectives
 involve representation continuity, visual intuition, image contrast, and
color mapping when selecting a set of representative vis. techniques
hybrid timing strategy
 uses two timing schemes (fixed duration / variable duration) to help
reveal subtle differences in vis. effectiveness between techniques
refined statistical analysis
 processes outliers + Ryan REGWQ post-hoc homogeneous subset tests
The major contributions of our work include explicit flow synthesis for reducing data-related bias and implicit task design for suppressing task-related bias.
In addition, we elaborate on diverse evaluation perspectives when selecting a small set of representative visualization techniques.
We also use a hybrid timing strategy to collect test data and use refined statistical analysis methods to investigate the result.
VIS 2011
FlowVUS

12. Experimental Components
2D Flow Visualization User Study Pipeline
Synthetic
Flow
Datasets
Flow
Visualization
Techniques
three fundamental
components of a
typical flow vis.
user study
Synthetic
Flow
Datasets
Visualization
Techniques
Analysis
Tasks
Flow
Analysis
Tasks
Here is a typical pipeline for conducting 2D flow visualization user studies.
Specifically, synthetic flow datasets, flow visualization techniques, and flow analysis tasks are three major experimental components of the pipeline.
VIS 2011
FlowVUS

13. Experimental Components
Synthetic Flow Datasets
A single dataset would introduce learning effect — unacceptable
Synthetic datasets are used for user study in medical imaging
Multiple datasets may incur data-dependent bias (in flow complexity)
Data-dependent bias can be suppressed to an acceptable degree
by synthesizing flows with similar topological complexities
Previous work on flow vis. user study uses implicit flow synthesis
implicit flow synthesis
To deal with the learning effect and data-dependent bias, we need to synthesize flows with similar topological complexities
Previous user studies employ random samples associated with random vectors followed by linear interpolation to create flow fields. We call this implicit flow synthesis.
The resulting pattern and the complexity are not predictable.
samples randomly selected and the associated vectors randomly assigned
a flow field is generated by vector interpolation between the samples
the topology of the resulting flow is unpredictable
— number of critical points, the locations, the types & overall complexity
VIS 2011
FlowVUS

14. Experimental Components
Synthetic Flow Datasets
employs parameterized placement and configuration of critical points
provides great flexibility and control in creating pseudo flow fields
Basis Vector Field (BVF) flow synthesis method by van Wijk (TOG 02)
— a BVF is governed by a critical point with some parameters
— the entire flow results from the combination of multiple BVFs
a survey and initialization-analysis-editing by Zhang et al (TOG 06)
 Explicit Flow Synthesis
FlowVUS BVF
FlowVUS is the first user study to value and apply explicit flow synthesis
based on BVF for fast automatic generation of many synthetic flows
We propose to use explicit flow synthesis by means of parameterized placement and configuration of critical points.
It is based on the basis vector field method. A BVF is governed by a critical point with some control parameters and multiple BVFs are then linearly combined to create an entire flow.
We use force composition and attenuation to define the influence of an Explicitly Specified Critical Point or ESCP on an arbitrary point.
Here an ESCP is either a center or a focus used to describe a BVF. The interaction among centers and foci may give rise to some saddles.
centers and foci — Explicitly Specified Critical Points (ESCPs)
saddles — derived from the interaction among centers and foci
uses a force composition-attenuation method to govern the influence of an
ESCP (with rotational force and radial force) or a BVF on an arbitrary point
VIS 2011
FlowVUS

15. Experimental Components
Synthetic Flow Datasets
This is an interactive ESCP-based flow synthesizer we developed much earlier.
VIS 2011
FlowVUS

16. Experimental Components
Synthetic Flow Datasets
4 pairs of x-symmetric
ESCP placement blocks
4 pairs of y-symmetric
4 pairs of center-symmetric
ESCP placement blocks
blue block: for primary ESCP placement; gray block: for mirror ESCP placement
location
radial force
rotational force
ESCP parameters
force attenuation
sink / source type
clockwise / counter-clockwise orientation
Layout templates
to synthesize flows with diverse structures yet with a relatively balanced
layout of a fixed number of ESCPs + a slightly varying number of saddles
— to maintain nearly the same topological complexity between many flows
to generate x- / y- / center-symmetric and dubiously asymmetric flows
— so as to support our pattern-based implicit flow analysis task design
Here are the ESCP parameters.
We need fast automatic generation of many flows with diverse structures but with similar topological complexities.
In other words, we need to maintain a relatively balanced layout of a fixed number of ESCPs and a slightly varying number of derived saddles.
In addition, we want to generate some symmetric and dubiously asymmetric flows in support of our implicit task design.
So we use three templates for semi-random distribution of ESCPs. Primary ESCPs are randomly placed in light blue blocks and mirror ESCPs are then placed in the gray blocks based on the symmetry type.
By modifying some parameters, we can generate geometrically symmetric but topologically asymmetric flows.
a primary ESCP is randomly placed & configured in each blue block and
its mirror ESCP is placed based on a symmetry type yet with the sink /
source type & clockwise/counter-clockwise orientation possibly different
— they may be geometrically symmetric but topologically asymmetric
VIS 2011
FlowVUS

17. Experimental Components
Synthetic Flow Datasets
symmetric flows versus asymmetric flows
asymmetric x-symmetric asymmetric
center-symmetric asymmetric y-symmetric
Here are some symmetric and asymmetric flows generated using our automatic explicit flow synthesizer.
VIS 2011
FlowVUS

18. Experimental Components
Flow Visualization Techniques
direction — the positive and negative directions tangent to the flow
orientation — the positive direction of the flow only (e.g., oriented LIC)
velocity magnitude — a scalar quantity
Primitive flow characteristics
many flow features (e.g., critical points) visually recognizable from them 
The most important
a vector quantity providing the fundamental info that distinguishes a flow
field from a scalar field and hence governs why / how flow visualization
differs very much from scalar visualization in the working mechanism
how well a flow vis. technique delineates the general, directional info
largely determines its effectiveness in conveying specific flow features
— direction
Flow direction, orientation, and velocity magnitude are usually investigated in a flow vis user study. Here orientation refers to the positive flow direction.
We feel the most important property is flow direction because it distinguishes a flow field from a scalar field and flow vis from scalar vis.
The ability of a technique to delineate this general direction information largely determines the effectiveness in conveying many high-level flow features.
Our work is focused on indirect user-exploration based techniques because they involve many human factors.
An informal classification
Direct Feature-Extraction Based (DFEB) — e.g., topology extraction
Indirect User-Exploration Based ( IUEB) — e.g., flow lines and LIC
from the flow reconstruction or visual analysis perspective
our focus
VIS 2011
FlowVUS

19. Experimental Components
Flow Visualization Techniques
need more user studies than DFEB techniques do due to the human factors
 user exploration  visual analysis  mental reconstruction
IUEB techniques
involve several major visual factors
 representation continuity (e.g., 0D / 1.5D / 2D)  visual intuition
 image contrast  color mapping
— FlowVUS evaluation aspects
54 candidates — 3 families
 hedgehogs  streamlines  LIC
selected through a thorough intra- and inter-family investigation
representative of many geometry-based and texture-based techniques
in terms of the aforementioned four major visual / evaluation aspects
configured via iterative internal tests for optimal visualization results
7 techniques
When selecting a small set of representative techniques, we are particularly interested in four evaluation aspects which are representation continuity, visual intuition, image contrast, and color mapping.
Actually we considered 54 candidate techniques from three families with many possible combinations involving type, size, layout, and color.
After thorough intra- and inter-family comparisons, we selected 7 representative techniques and determined optimal parameters for each.
VIS 2011
FlowVUS

20. Experimental Components
ArrowCM
Experimental Components
ArrowCW
StreamCM
Flow Visualization Techniques
OrientedLIC
EnhancedLIC
BasicLIC
StreamCW
Here are the 7 techniques we finally selected. Arrows and evenly spaced streamlines with rainbow and colorwheel map. Enhanced LIC, basicLIC, and orientedLIC wit rainbow color map.
Here are the images generated using the 7 techniques to visualize the same synthetic flow.
VIS 2011
FlowVUS

21. Experimental Components
Flow Analysis Tasks
impossible & unnecessary to enumerate specific / complex flow features
and then design many flow analysis tasks (how many studies are enough?)
Some essential points
the performance of an average participant in visual flow analysis is
expected to reflect the effectiveness of the IUEB technique (being used)
in conveying the flow direction — the general fundamental information
flow analysis tasks in a user study are not necessarily real or practical
flow analysis tasks are the way instead of (or at least more than) the goal
for example, synthetic tasks are often used for psychological user studies
by devising some seemingly irrelevant yet intrinsically coupled questions
We feel it is impossible and unnecessary to enumerate specific flow features when designing tasks.
Due to the importance of flow direction for understanding high-level flow features, we focus on investigating how well IUEB techniques convey this general direction information.
So flow analysis tasks are not necessarily real or practical because they are the way instead of the goal. One good example is that in psychological user studies, seemingly irrelevant but intrinsically coupled questions are usually designed.
To reduce task-related bias, one important principle is to use an indirect implicit way but take a testable form.
In other words, do NOT directly ask the user to check the flow direction at any single point.
A question needs to be easy to understand but challenging to answer correctly.
in order to reduce task-related bias, flow analysis tasks may take
an indirect / implicit way
and a testable form
(— do not directly ask the user to check the flow direction at a point)
(— questions are easy to understand but challenging to answer correctly)
VIS 2011
FlowVUS

22. Experimental Components
Flow Analysis Tasks
used in previous work and susceptible to bias
a typical example — directly ask to check the flow direction at a point
 the participant is shown a randomly placed circle (of which the center is
hence a random sample) and asked to click on the point along the circle
that a particle advected from the center is to hit
Explicit sample-based tasks
 the complexity of a flow usually varies with the location
 more difficult to do this task in turbulent areas than in laminar areas
 the selection of the circle’s radius may further compound this issue
 mouse pointing & clicking, irrelevant of judgment, affect the test result
Explicit sample-based tasks are used in previous user studies and they are susceptible to bias.
Here we propose to design implicit pattern-based tasks. The basic idea is to indirectly require the user to perform thorough investigation of the flow direction globally or at least around an area..
Two well-known examples are critical point recognition and critical point classification used in previous user studies.
They are good at directing the user to a global or local domain of the flow instead of any single point.
a methodology advocated and formulated in this paper to suppress bias
use a simple form but indirectly require thorough investigation of the flow
 Implicit pattern-based tasks
critical point recognition — detect patterns globally/across the whole domain
critical point classification — match patterns locally/around an area of interest
VIS 2011
FlowVUS

23. Experimental Components
Flow Analysis Tasks
Implicit task design
using specific real tasks
 about well-known flow features
critical point recognition (CPR) critical point classification (CPC)
 involving in-depth flow structures
identification of separatrices identification of periodic orbits
creating general synthetic tasks
 to reduce data-related bias resulting from flow sampling and mouse
point-and-click operations
 to relieve non-expert participants from understanding complex,
possibly domain-specific details
 in the form of easy-to-understand yet difficult-to-answer questions
 requiring intensive analysis of flow directions
Implicit tasks are not necessarily but can be real tasks like critical point recognition, critical point classification and even more complex ones.
In addition, they may be synthetic tasks. Synthetic tasks are good at relieving the non-expert users from understanding complex domain-specific details.
One example is symmetric pattern categorization or SPC used in our user study.
The user needs to examine the flow direction both globally and locally before determining the symmetry type.
 such as symmetric pattern categorization (SPC)
— to examine the flow direction both globally and locally
— to check the entire pattern: x-/y-/z-/center-symmetric or asymmetric
VIS 2011
FlowVUS

24. Experimental Components
Flow Analysis Tasks
Very challenging synthetic tasks
two or three critical points (centers, foci, and saddles) combined with
a variety of configurations to define some Composite Templates (CTs)
 CT-based CPR-like pattern recognition
 CT-based CPC-like pattern classification
checking if flow A and flow B have a CT pattern in common
judging if flow A is a rotational version of flow B
determining if flow A is exactly part of flow B
The selected implicit tasks
CPR + CPC + SPC
integration of 2 real tasks and 1 synthetic task to demonstrate the types
the balance between the overall challenge degree and the test duration
— some synthetic tasks mentioned above would require more test time
Besides SPC, there are more challenging synthetic tasks.
For example, we can combine two or three centers, foci, and saddles in several configurations to define some composite templates or CTs and then design CT-based pattern recognition and CT-based pattern classification.
To balance between the overall challenge degree and the test duration and to integrate the both real and synthetic tasks, we chose CPR, CPC, and SPC in our user study.
VIS 2011
FlowVUS

25. FlowVUS Test Strategy The Input 7M images
generated using the selected 7 techniques to visualize M synthetic flows
involving N x-symmetric, N y-symmetric, N center-symmetric, and
optionally N asymmetric flows — M = 3N or 4N (e.g., N = 30)
depending on the expected complexity and time duration of the test
Ground truth — one record per synthetic flow
symmetry type of the overall pattern
the location and type of every ESCP (center / focus) from the synthesizer
the location of every derived saddle from Newton-Raphson root-finding
Now let’s talk about the test strategy. The input includes a collection of pre-generated images and ground truth, one record for synthetic flow.
Depending on the task type, a session may be made up of 1 CPR task, up tp 30 CPC tasks, or up to 30 SPC tasks.
Task Session
1 CPR task (recognizing ALL critical points from an image) or
<= 30 CPC tasks or
<= 30 (without asymmetric flows) / 40 (with asymmetric flows) SPC tasks
VIS 2011
FlowVUS

26. FlowVUS Test Strategy Task Management VIS 2011
1 set = (1 CPR session + 1 CPC session + 1 SPC session) for one technique
1 cycle = 7 sets (one for each technique)
1 test = 3 cycles
1 session = 1 CPR task or (<= 30) CPC tasks or (<= 30/40) SPC tasks
= 21 sets = 63 sessions for each participant
use 7 techniques thrice to produce 7 × 3 = 21 images (for 21 randomly-
selected flows), with 1 image for each CPC session (3 per technique)
use each technique to produce 30 images (for 30 randomly-selected flows),
with 1 randomly-selected critical point marked per image (with 10 marked
for each critical point type: center, focus, saddle), for each CPC session
use each technique to visualize 30 or 40 randomly-selected flows (creating
10 images for each symmetry / asymmetric type) for each SPC session 
Given the session concept, we define set, cycle, and test.
One user takes a total of 63 sessions, actually 21 sessions for each task type.
For each test, the 63 sessions are created using our TestGen program based on the bank of pre-generated images.
Then the sessions are delivered to the user one by one in random order.
21 CPR sessions + 21 CPC sessions + 21 SPC sessions = 63 sessions
with a bank of images pre-generated for one time, 63 sessions are created
using TestGen upon each test and are then delivered in random order
VIS 2011
FlowVUS

27. FlowVUS Test Strategy Hybrid Timing Effectiveness metrics
the effectiveness of a visualization technique is usually reflected by
answer correctness and response time
a more effective technique allows the user to get a correct answer faster
given a fixed amount of time, more correct answers tend to result from
a more effective technique than from a less effective technique
this hybrid timing strategy helps reveal the subtle differences
in visualization effectiveness that may exist between techniques
Variable-duration session
mouse click positions and response time are recorded for a session
flow analysis (for recognizing a single critical point) is relatively quick
the answer is precision-critical (despite a considerable error tolerance)
 seeks to “curb” the participant from hastiness and excessive inaccuracy
— for CPR
We use a variable duration for each CPR session to curb the participant from hastiness and excessive inaccuracy because the task is relatively quick and precision-critical.
We use a fixed duration for each CPC or SPC session to push the participant to accomplish more tasks because the task is relatively slow and judgment-intensive.
The hybrid timing strategy is intended to reveal the subtle differences between techniques in visualization effectiveness.
Fixed-duration session
as many tasks as possible are presented to the participant one by one
in a fixed amount of time (30s) and radio-button choices are recorded
flow analysis is relatively slow and judgment-intensive
 intended to “push” the participant to accomplish more tasks
— for CPC & SPC (response time on average)
VIS 2011
FlowVUS

28. CPR — Critical Point Recognition
Test Strategy
Here is a CPR session for Enhanced LIC
CPR — Critical Point Recognition
VIS 2011
FlowVUS

29. CPC — Critical Point Classification
Test Strategy
Here is a CPC session for Arrows with color wheel map
CPC — Critical Point Classification
VIS 2011
FlowVUS

30. SPC — Symmetric Pattern Categorization
Test Strategy
Here is an SPC session for evenly spaced streamlines with color wheel map
SPC — Symmetric Pattern Categorization
VIS 2011
FlowVUS

31. FlowVUS Test Results Basic Facts
4 CFD experts + 16 graduate students in science & engineering
expert and non-expert participants were not compared herein
5079 CPR trials CPC trials SPR trials were recorded
the absolute differences in response time for CPR / CPC / SPC
turned out to be small, regardless of the statistical differences
a higher priority assigned to correctness than to response speed
to provide correctness-over-response-sorting (CORS) when evaluating
the seven techniques in the overall visualization effectiveness
Processing Outliers
We recruited 20 participants and recorded 18 thousand tasks.
We provided correctness over speed sorting in the overall visualization effectiveness.
As for response time and CPR location error, we detected outliers based on the histograms and then replaced outliers with appropriate median values.
the response time and the (CPR) location error each showed
a skewed normal distribution in terms of the histogram
outliers were determined case by case by investigating the tails
of the distributions and noting values after conspicuous gaps
each outlier was replaced with the median of the cell’s responses
VIS 2011
FlowVUS

32. FlowVUS Test Results Statistical Analysis
Chi-square tests and ANOVA (univariate analysis of variance)
calculating post-hoc homogeneous subsets using Ryan REGWQ tests
FlowVUS Results
CPR (Critical Point Recognition) — response time
Last but not least --- besides Chi-Square tests and ANOVA analysis, we calculated Post-hoc homogenous subsets using Ryan REGWQ tests.
So we could refine statistical analysis to process the test result more accurately.
Here is the statistical result of CPR on response time. The means with the same letter are not significantly different.
mean time (in seconds) to recognize a critical point (5079 trials, F(6,115.3) = 19.9, p < 0.001)
means with the same letter are not significantly different at p  0.05 (Ryan REGWQ post-hoc hst)
VIS 2011
FlowVUS

33. CORS sorting by CPR effectiveness in decreasing order
Test Results
FlowVUS Results
CPR (Critical Point Recognition) — answer incorrectness
336 errors, χ2(6) = 132, p < 0.001
Here is the statistical result of CPR on answer correctness.
And this is the COSS sorting by CRP effectiveness in decreasing order.
CORS sorting by CPR effectiveness in decreasing order
EnhancedLIC - StreamCM - BasicLIC - OrientedLIC - StreamCW - ArrowCM - ArrowCW
VIS 2011
FlowVUS

34. FlowVUS Test Results FlowVUS Results
CPC (Critical Point Classification) — response time
Similarly, here is the statistical result of CPC on response time.
mean time (in seconds) to classify a critical point (7467 trials, F(6,116.2) = 30.9, p < 0.001)
means with the same letter are not significantly different at p  0.05 (Ryan REGWQ post-hoc hst)
VIS 2011
FlowVUS

35. CORS sorting by CPC effectiveness in decreasing order
Test Results
FlowVUS Results
CPC (Critical Point Classification) — answer incorrectness
753 errors, χ2(6) = 772, p < 0.001
Here is the statistical result of CPC on answer correctness.
And here is the COSS sorting by CPC effectiveness in decreasing order.
CORS sorting by CPC effectiveness in decreasing order
EnhancedLIC - StreamCW - StreamCM - BasicLIC - OrientedLIC - ArrowCW - ArrowCM
VIS 2011
FlowVUS

36. FlowVUS Test Results FlowVUS Results
SPC (Symmetric Pattern Categorization) — response time
Here is the statistical result of SPC on response time.
mean time (in sec.s) to categorize a symmetric pattern (4948 trials, F(6,123.1) = 8.74, p < 0.001)
means with the same letter are not significantly different at p  0.05 (Ryan REGWQ post-hoc hst)
VIS 2011
FlowVUS

37. CORS sorting by SPC effectiveness in decreasing order
Test Results
FlowVUS Results
SPC (Symmetric Pattern Categorization) — answer incorrectness
323 errors, χ2(6) = 70.1, p < 0.001
Here is the statistical result of SPC on answer correctness.
And here is the COSS sorting by SPC effectiveness in decreasing order.
CORS sorting by SPC effectiveness in decreasing order
EnhancedLIC - StreamCM - BasicLIC - OrientedLIC - StreamCW – ArrowCM - ArrowCW
VIS 2011
FlowVUS

38. FlowVUS Test Results EnhancedLIC StreamCM StreamCW BasicLIC
CORS Sorting by
CPR effectiveness
CPC effectiveness
SPC effectiveness
EnhancedLIC
StreamCM
StreamCW
BasicLIC
OrientedLIC
ArrowCM
ArrowCW
color mapping has a considerable influence on a geometry-based flow representation
Here is a summary of the CORS sorting by CPR, CPC, and SPC effectiveness.
A texture-based dense representation with accentuated flow streaks such as Enhanced LIC enables intuitive perception of the flow.
A geometry-based integral representation with uniform density control such as evenly spaced streamlines can exploit visual interpolation to facilitate mental reconstruction of the flow.
In addition, color mapping has a considerable influence on a geometry-based representation of the flow.
a texture-based dense representation with accentuated flow
streaks (EnhancedLIC) enables intuitive perception of the flow
a geometry-based integral representation with uniform density control (StreamCM or
StreamCW) exploits visual interpolation to facilitate mental reconstruction of the flow
VIS 2011
FlowVUS

39. FlowVUS Concluding Remarks Key Points — to reduce data-related bias
— to suppress task-related bias
Explicit flow synthesis
Implicit task design
Two important methodologies / concepts proposed as part of our
anti-bias framework for conducting objective flow vis. user studies
— to reduce data-related bias
template-based parameterized placement & configuration of critical points
automatic synthesis of diverse flows with similar topological complexities
— to suppress task-related bias
pattern-based (real tasks + synthetic tasks) the way more than the goal
— representative techniques
representation continuity visual intuition
image contrast color mapping
variable-duration session fixed-duration session
to reveal the subtle differences in vis. effectiveness between techniques
processes outliers + Ryan REGWQ post-hoc homogeneous subset tests
Explicit flow synthesis
Implicit task design
Diverse evaluation perspectives
Hybrid timing strategy
Refined statistical analysis
Here are the key points of our work, including explicit flow synthesis, implicit task design, diverse evaluation perspectives, hybrid timing strategy, and refined statistical analysis.
In particular, explicit flow synthesis and implicit task design are two important methodologies or concepts that we proposed as part of an anti-bias framework for conducting objective flow vis user studies.
VIS 2011
FlowVUS

40. FlowVUS Concluding Remarks Limitations & Lessons
FlowVUS is bias-resistant but not bias-free
Varying a-priori familiarity with techniques
Varying a-priori familiarity with flow features
Real flows needed for introducing techniques
Care needed for predicting the time duration
bias is pervasive throughout the whole pipeline of a user study and hence
we cannot totally eliminate it while we need to reduce it — cannot let it be
some participants were not familiar with the LICs upon the training session
 more user studies are needed to disseminate the latest vis. techniques
 care needs to be taken when evaluating more sophisticated / current ones
some participants needed extra help with some features during the training session
 many challenges facing an evaluation involving more complex features
synthetic flows are needed for formal tests while real flows (particularly with
contextual boundaries) are needed, besides real flows, for the training session
FlowVUS is bias-resistant but not bias-free. We got some lessons such as the varying a-priori familiarity with the techniques and flow features.
On one hand, this means more user studies are needed to educate the general public. On the other hand, care needs to be taken when designing a user study with more complex configurations.
VIS 2011
FlowVUS

41. FlowVUS Concluding Remarks Future Plans Anti-bias methodologies
controversial
view
Anti-bias methodologies
user studies might otherwise be non-convincing & even worse misleading
probably one way to help you judge between 2 contradicting conclusions
as of now more important than scenarios, techniques, features, conclusions
require much research (e.g., explicit flow synthesis & implicit task design)
— to adopt the conclusions of a user study without necessary anti-bias methods?
neither possible nor necessary to evaluate every existing vis. technique
Evaluation aspects — representative visualization techniques
provide general guidelines for visualization research (algorithm design)
 end users might not care about the underlying working mechanism
 they are interested in the resulting images and the associated visual
aspects (such as image contrast, color map, intuition, continuity, etc)
controversial
view
As for future plans, we would like to continue the work on anti-bias methodologies. Without sufficient research on this topic, the conclusions we draw may be non-convincing and even worse misleading.
So we believe as of right now this is a top priority for flow visualization user study.
In addition, we will improve on the evaluation aspects. This is based on our observation that it is neither possible nor necessary to evaluate every existing visualization technique.
What we need to do is to extract the commonalities of existing techniques in terms of visual effects instead of working mechanisms.
Next we are interested in conducting user studies on streamline placement algorithms, surface flow and volume flow visualization techniques.
Interesting topics
user studies on streamline placement algorithms
user studies on surface flow visualization techniques
user studies on volume flow visualization techniques
VIS 2011
FlowVUS

42. for your time and attention!
Thank you
for your time and attention!
Thank you so much for your time and attention!