HCI Methods for Pathway Visualization Tools Purvi Saraiya, Chris North, Karen Duca* Virginia Tech Dept. of Computer Science, Center for Human-Computer Interaction *Virginia Bioinformatics Institute

What is HCI? All aspects of human interaction with a computer system A discipline concerned with design, implementation, and evaluation of computer systems for human use

Two Perspectives For Pathway Tools Developers: –What tool will be most helpful to users? Users: –Many options, how to select the most appropriate tool?

Software Development Cycle Reqs Analysis Evaluate Design Develop

Requirement Analysis Using Informal Interviews Research Questions: What kinds of tasks do the users perform? How do the tools fit with overall research goals of the users? Outcomes: Usage Scenarios Requirements Method: 1-1 Interviews

Usage Scenarios Scenarios: stories about users and their work activities User-oriented: focus on needs and concerns of users User-perception: how do users perceive their problems?

User Requirements for Pathway Visualization Tools CategoriesRequirementsTasks Pathway Construction 1. Create & Update Collect and link pathways from multiple resources 2. ContextProvide information about pathways 3. UncertaintyShow alternate hypotheses and information reliability 4. CollaborationEnable group work

Questionnaires and Surveys Research Question: Which requirements are more important? Pathway QuestionsStrongly Agree AgreeNeutralDisagree Category: Pathway Assembly R1: Create & Update 1 In my work, the entire pathway(s) is generally not available from a single source.64 R1: Collaborate 2 I collaborate with others and need a tool to let them enter changes from remote sites1144

Focus Groups Research Questions: –What is user feedback for the selected systems? Method: –Group discussion with users No of users: –5 - 10

Systems Evaluated GenMapp, PathwayAssist, Cytoscape, Patika, GScope

Results Example User Responses: –Users were excited about the NLP features provided by PathwayAssist but were skeptical about its reliability –Need more biological context for the pathways End-Users: –Which system should I use for my work?

Longitudinal Studies Research Question: –How are the tools used in actual real world scenarios? –What interactions and features were actually used by the users for their tasks? Method: –Users: Log Keeping –Evaluators: Discuss user logs

Example Log Heatmap + Pathway Visualization in PathwayAssist DateVisualizationInsightValue 9/01Heatmap  A list of genes that are suppressed by smoking but up-regulated by flu. 4 9/12Pathway Visualization  The up-regulation of Mx by flu is suppressed by smoking even though smoking itself did not have an effect on basal Mx activity. 3 Example log for data analysis with PathwayAssist

Empirical Studies Research Question: –Are the tools preferred by users actually better? Method: –Task-Based Method –Insight-Based Method

Task-Based Studies Typical question: –Which visualization is better for defined tasks? Vs. Vis. 1 Vis. 2

Variables Independent Variables: Visualization tool »GenMapp, PathwayAssist, etc. Task type »Find, count, pattern, compare Data size (# of items) »100, 1000, Dependent Variables: Task completion time Errors Subjective satisfaction (survey)

Experiment Protocol Participants may be videotaped Evaluator takes notes

Data Analysis t-test: Compares 1 dep var on 2 treatments of 1 ind var ANOVA: Compares 1 dep var on n treatments of m ind vars Task1: Find the node with max value? Task2Task3 Vis …… Vis 2……… Ind Var 1: Vis. Tool Ind Var 2: Task Type Dep var: user performance times (3 users per cells)

Insight-Based Method Motivation: Task-based methods uses pre-selected tasks Often not representative of the real world visualization tool usage Issues: How to eliminate benchmark tasks? What is Insight? How to codify and quantifiably measure insights across participants?

Insight Characteristics Insight = an individual data observation Observation Time to discover Domain value (importance) Hypotheses generated? Directed vs. unexpected Correctness Category (overview, details) Can be recognized by thinking-aloud

Experiment Design Independent Variables: –Visualization Tools –Datasets –Participant Background Dependent Variables: –Insights –Time at which insights were reported –User feedback

Tool 1Tool 2Tool 3Tool 4 Tool 5 Count of insights Domain value of insights Average time to first insight (in mins)

Data Bias The tool works better for categorical data

Conclusions Developers: Informal 1-1 interviews & questionnaires Focus group meetings Longitudinal Studies Users: Controlled studies to evaluate tools User studies may not always be feasible Insight characteristics can be used as checklists for selecting a tool

Thank you