HCI Methods for Pathway Visualization Tools

1
HCI Methods for Pathway Visualization Tools

• Purvi Saraiya, Chris North, Karen Duca
• Virginia Tech

Dept. of Computer Science, Center for
Human-Computer Interaction Virginia
Bioinformatics Institute
2
What is HCI?

• All aspects of human interaction with a computer
  system
• A discipline concerned with design,
  implementation, and evaluation of computer
  systems for human use

3
Two Perspectives For Pathway Tools

• Developers
• What tool will be most helpful to users?
• Users
• Many options, how to select the most appropriate
  tool?

4
Software Development Cycle
Reqs Analysis
Design
Evaluate
Develop
5
Requirement Analysis Using Informal Interviews

• Research Questions
• What kinds of tasks do the users perform?
• How do the tools fit with overall research goals
  of the users?
• Outcomes
• Usage Scenarios
• Requirements

Method 1-1 Interviews
6
Usage Scenarios

• Scenarios stories about users and their work
  activities
• User-oriented focus on needs and concerns of
  users
• User-perception how do users perceive their
  problems?

7
User Requirements for Pathway Visualization Tools
Categories Requirements Tasks
Pathway Construction 1. Create Update Collect and link pathways from multiple resources
Pathway Construction 2. Context Provide information about pathways
Pathway Construction 3. Uncertainty Show alternate hypotheses and information reliability
Pathway Construction 4. Collaboration Enable group work
8
Questionnaires and Surveys

• Research Question Which requirements are more
  important?

Pathway Questions Strongly Agree Agree Neutral Disagree
Category Pathway Assembly R1 Create Update Category Pathway Assembly R1 Create Update Category Pathway Assembly R1 Create Update Category Pathway Assembly R1 Create Update Category Pathway Assembly R1 Create Update Category Pathway Assembly R1 Create Update
1 In my work, the entire pathway(s) is generally not available from a single source. 6 4
R1 Collaborate R1 Collaborate R1 Collaborate R1 Collaborate R1 Collaborate R1 Collaborate
2 I collaborate with others and need a tool to let them enter changes from remote sites 1 1 4 4
9
Focus Groups

• Research Questions
• What is user feedback for the selected systems?
• Method
• Group discussion with users
• No of users
• 5 - 10

10
Systems Evaluated
GenMapp, PathwayAssist, Cytoscape, Patika, GScope
11
Results

• Example User Responses
• Users were excited about the NLP features
  provided by PathwayAssist but were skeptical
  about its reliability
• Need more biological context for the pathways
• End-Users
• Which system should I use for my work?

12
Longitudinal Studies

• Research Question
• How are the tools used in actual real world
  scenarios?
• What interactions and features were actually used
  by the users for their tasks?
• Method
• Users Log Keeping
• Evaluators Discuss user logs

13
Example Log
Heatmap Pathway Visualization in PathwayAssist
Date Visualization Insight Value
9/01 Heatmap A list of genes that are suppressed by smoking but up-regulated by flu. 4
9/12 Pathway Visualization The up-regulation of Mx by flu is suppressed by smoking even though smoking itself did not have an effect on basal Mx activity. 3
Example log for data analysis with PathwayAssist
14
Empirical Studies

• Research Question
• Are the tools preferred by users actually better?
• Method
• Task-Based Method
• Insight-Based Method

15
Task-Based Studies

• Typical question
• Which visualization is better for defined tasks?

Vs.
Vis. 2
Vis. 1
16
Variables

• Independent Variables
• Visualization tool
• GenMapp, PathwayAssist, etc.
• Task type
• Find, count, pattern, compare
• Data size ( of items)
• 100, 1000, 1000000
• Dependent Variables
• Task completion time
• Errors
• Subjective satisfaction (survey)

17
Experiment Protocol
Participants may be videotaped
Evaluator takes notes
18
Data Analysis
Ind Var 2 Task Type
Task1 Find the node with max value? Task2 Task3
Vis 1 12 32 45
Vis 2
Ind Var 1 Vis. Tool
Dep var user performance times (3 users per
cells)

• t-test
• Compares 1 dep var on 2 treatments of 1 ind var
• ANOVA
• Compares 1 dep var on n treatments of m ind vars

19
Insight-Based Method

• Motivation
• Task-based methods uses pre-selected tasks
• Often not representative of the real world
  visualization tool usage
• Issues
• How to eliminate benchmark tasks?
• What is Insight? How to codify and quantifiably
  measure insights across participants?

20
Insight Characteristics

• Insight an individual data observation
• Observation
• Time to discover
• Domain value (importance)
• Hypotheses generated?
• Directed vs. unexpected
• Correctness
• Category (overview, details)
• Can be recognized by thinking-aloud

21
Experiment Design

• Independent Variables
• Visualization Tools
• Datasets
• Participant Background
• Dependent Variables
• Insights
• Time at which insights were reported
• User feedback

22
Tool 1 Tool 2 Tool 3 Tool 4 Tool 5
Count of insights
Domain value of insights
Average time to first insight(in mins)
23
Data Bias
The tool works better for categorical data
24
Conclusions

• Developers
• Informal 1-1 interviews questionnaires
• Focus group meetings
• Longitudinal Studies
• Users
• Controlled studies to evaluate tools
• User studies may not always be feasible
• Insight characteristics can be used as checklists
  for selecting a tool

25
Thank you