Thinking Interactively with Visualizations

1
Thinking Interactively with Visualizations

• Remco Chang
• UNC Charlotte
• Charlotte Visualization Center

2
Role of Interaction

• Most people in the visualization community
  believe that interactivity is essential for
  visualization and visual analytics
• A visual analysis session is more of a dialog
  between the analyst and the data the
  manifestation of this dialog is the analysts
  interactions with the data representation
  Thomas Cook 2005
• Without interaction, a visualization technique
  or system becomes a static image or autonomously
  animated images Yi et al. 2007

3
Role of Interaction

• However, there has been limited research in
  visualization specific to interaction and
  techniques Yi et al. 2007
• Interaction is often relegated to a secondary
  role in these articles. Interaction rarely is the
  main focus of research efforts in the field,
  essentially making it the little brother of
  Infovis
• The goal of this talk is to consider the role of
  interaction in computer graphics, information
  visualization, and visual analytics.
• First, we think about what interactivity is and
  how to make a visualization interactive.

4
Offline Rendering
5
Offline Rendering

• Masters Thesis --
• Modeling the dynamic motion based on kinematic
  motion
• Jiggling of muscles
• Skinnable Mesh
• Volumetric deformation
• Mass-spring models
• 2nd order constraint
• Approximate finite element method (FEM) with
  implicit integration
• Took all night to render a 500-frame (10 second)
  sequence
• NOT at all interactive
• Key differences between each run had to be
  remembered

6
Thinking about Interactivity in Graphics

• Interactivity
• 12 frames per second appears smooth to most
  people
• Or, render a frame under 0.08 second
• For complex scenes with lots of polygons
  (information)
• Simplify the scene
• Levels of Detail (LOD)

7
Urban Simplification

• (left) Original model, 285k polygons
• (center) e100, 129k polygons (45 of original)
• (right) e1000, 53k polygons (18 of original)

R. Chang et al., Legible simplification of
textured urban models. IEEE Computer Graphics and
Applications, 28(3)2736, 2008. R. Chang et al.,
Hierarchical simplification of city models to
maintain urban legibility. ACM SIGGRAPH 2006
Sketches, page 130 , 2006.
8
Urban Simplification

• Which polygons to remove?

Visually different, but quantitatively similar!
9
Urban Simplification

• The goal is to retain the Image of the City
• Based on Kevin Lynchs concept of Urban
  Legibility 1960
• Paths highways, railroads
• Edges shorelines, boundaries
• Districts industrial, historic
• Nodes Time Square in NYC
• Landmarks Empire State building

10
Urban Visualization with Semantics

• How do people think about a city?
• Describe New York
• Response 1 New York is large, compact, and
  crowded.
• Response 2 The area where I live there has a
  strong mix of ethnicities.

Geometric,
Information,
View Dependent (Cognitive)
11
Urban Visualization

• Geometric
• Create a hierarchy of shapes based on the rules
  of legibility
• Information
• Matrix view and Parallel Coordinates show
  relationships between clusters and dimensions
• View Dependence (Cognitive)
• Uses interaction to alter the position of focus

R. Chang et al., Legible cities Focus-dependent
multi-resolution visualization of urban
relationships. IEEE Transactions on Visualization
and Graphics , 13(6)11691175, 2007
12
Urban Visualization

• Scenario 1 Comparing cities

• Charlotte
• Davidson

13
Urban Visualization

• Scenario 2
• Looking for high Hispanic populations around
  downtown Charlotte.

14
The Role of Interaction in Visualization

• We can use interactions to Yi et al. 2007
• Select mark something as interesting
• Explore show me something else
• Reconfigure show me a different arrangement
• Encode show me a different representation
• Abstract/Elaborate show me more or less detail
• Filter show me something conditionally
• Connect show me related items
• In other words, we can use interactions to think.

15
(1) WireVis Financial Fraud Analysis

• In collaboration with Bank of America
• Looks for suspicious wire transactions
• Currently beta-deployed at WireWatch
• Visualizes 15 million transactions over 1 year
• Uses interaction to coordinate four perspectives
• Keywords to Accounts
• Keywords to Keywords
• Keywords/Accounts over Time
• Account similarities (search by example)

16
(1) WireVis Financial Fraud Analysis
Search by Example (Find Similar Accounts)
Heatmap View (Accounts to Keywords Relationship)
Keyword Network (Keyword Relationships)
Strings and Beads (Relationships over Time)
R. Chang et al., Scalable and interactive visual
analysis of financial wire transactions for fraud
detection. Information Visualization,2008. R.
Chang et al., Wirevis Visualization of
categorical, time-varying data from financial
transactions. IEEE VAST, 2007.
17
(2) Investigative GTD

• Collaboration with U. Marylands DHS Center of
  Excellence START (Study of Terrorism And Response
  to Terrorism)
• Global Terrorism Database (GTD)
• International terrorism activities from 1970-1997
• 60,000 incidents recorded over 120 dimensions
• Visualization is designed to be investigative
  in that it is modeled after the 5 Ws
• Who, what, where, when, and why
• Interaction allows the user to adjust one or more
  of the Ws and see how that affects the other Ws

18
(2) Investigative GTD
R. Chang et al., Investigative Visual Analysis of
Global Terrorism, Journal of Computer Graphics
Forum (Eurovis), 2008.
19
(2) Investigative GTD Revealing Global Strategy
This groups attacks are not bounded by
geo-locations but instead, religious beliefs.
Its attack patterns changed with its
developments.
WHY ?
20
(2) Investigative GTDDiscovering Unexpected
Temporal Pattern
A geographically-bounded entity in the
Philippines.
Domestic Group
The ThemeRiver shows its rise and fall as an
entity and its modus operandi.
21
(3) Analysis of Biomechanical Motion

• Biomechanical motion
  sequences
  (animation) are
  difficult to analyze.
• Watching the movie repeatedly
  does not easily lead
  to insight.
• Collaboration with Brown University and Univ. of
  Minnesota to examine the mechanics of a pig
  chewing different types and amounts of food
  (nuts, pig chow, etc.)
• The data is typically organized by the rigid
  bodies in the model, where each rigid body
  contains 6 variables per frame -- 3 for
  translation, and 3 for rotation.

22
(3) Analysis of Biomechanical Motion
R. Chang et al., Interactive Coordinated
Multiple-View Visualization of Biomechanical
Motion Data, IEEE Vis (TVCG) 2009. To Appear.
23
(3) Analysis of Biomechanical Motion

• Our emphasis is on interactive comparison.
  Following the work by Robertson InfoVis 2008,
  comparisons can be performed using
• Small Multiples
• Side by side comparison
• Overlap
• Between two datasets
• Different cycles in the same data

24
(4) iPCA Interactive PCA

• Quick Refresher of PCA
• Find most dominant eigenvectors as principle
  components
• Data points are re-projected into the new
  coordinate system
• For reducing dimensionality
• For finding clusters
• For many (especially novices), PCA is easy to
  understand mathematically, but difficult to
  understand semantically.

25
(4) iPCA Interactive PCA
R. Chang et al., iPCA An Interactive System for
PCA-based Visual Analytics. Computer Graphics
Forum (Eurovis), 2009.
26
(4) Evaluation iPCA vs. SAS/INSIGHT

• Results
• A bit more accurate
• People dont give up
• Not faster

• Overall preference
• Using letter grades (A through F) with A
  representing excellent and F a failing grade.

27
If (Interactions Thinking)

• What is in a users interactions?
• If (interactions thinking), what can we learn
  from the users interactions?
• Is it possible to extract thinking from
  interactions?

28
What is in a Users Interactions?
Keyboard, Mouse, etc
Input
Visualization
Human
Output
Images (monitor)

• Types of Human-Visualization Interactions
• Text editing (input heavy, little output)
• Browsing, watching a movie (output heavy, little
  input)
• Visual Analysis (closer to 50-50)

29
What is in a Users Interactions?

• Goal determine if there really is thinking in
  a users interactions.

Grad Students (Coders)
Compare! (manually)
Analysts
Strategies Methods Findings
Guesses of Analysts thinking
Logged (semantic) Interactions
WireVis
Interaction-Log Vis
30
Whats in a Users Interactions

• From this experiment, we find that interactions
  contains at least
• 60 of the (high level) strategies
• 60 of the (mid level) methods
• 79 of the (low level) findings

R. Chang et al., Recovering Reasoning Process
From User Interactions. IEEE Computer Graphics
and Applications, 2009. R. Chang et al.,
Evaluating the Relationship Between User
Interaction and Financial Visual Analysis. IEEE
Symposium on VAST, 2009.
31
Whats in a Users Interactions

• Why are these so much lower than others?
• (recovering methods at about 15)
• Only capturing a users interaction in this case
  is insufficient.

32
Lessons Learned

• We have proven that a great deal of an analysts
  thinking in using a visualization is capturable
  and extractable.
• Using semantic interaction capturing, we might be
  able to collect all the thinking of expert
  analysts and create a knowledge database that is
  useful for
• Training many domain specific analytics tasks
  are difficult to teach
• Guidance use existing knowledge to guide future
  analyses
• Verification, and validation go back and check
  to see if everything was done right.
• But not all visualizations are interactive, and
  not all thinking is reflected in the
  interactions.
• A model of how and what to capture in a
  visualization for extracting an analysts
  thinking process is necessary.
• Work is currently in preparation for publication.

33
Conclusion

• Interactions are important for visualization and
  visual analysis
• In considering interactions, one must be aware of
  the necessary speed and frame rate of the
  displays.
• Simplification, LOD, and approximation methods
  can be used, but they must retain the salient
  features in the original data.
• Interactions have been proven to help the
  understanding of complex problems.
• Relevant interactions have been integrated in
  multiple visualizations for different domains and
  demonstrated significant impact.
• Capturing and storing analysts interactions have
  great potential
• They can be aggregated to become a knowledge
  database that has traditionally been difficult
  to create manually.

34
Discussion

• What interactivity is not good for
• Presentation
• YMMV your mileage may vary
• Reproducibility Users behave differently each
  time.
• Evaluation is difficult due to opportunistic
  discoveries..
• Often sacrifices accuracy
• iPCA SVD takes time on large datasets, use
  iterative approximation algorithms such as
  onlineSVD.
• WireVis Clustering of large datasets is slow.
  Either pre-compute or use more trivial binning
  methods.

35
Discussion

• Interestingly,
• It doesnt save you time
• And it doesnt make a user more accurate in
  performing a task.
• However, there are empirical evidence that using
  interactivity
• Users are more engaged (dont give up)
• Users prefer these systems over static
  (query-based) systems
• Users have a faster learning curve
• We would like to find measurements to determine
  the benefits of interactivity

36
Future Work

• Interactive Urban Visualization
• Visualizing a Semantic City. An ideal goal would
  be to have a semantic Google Map that shows
  more than street layouts, but describes
  neighborhood characteristics.
• Further studies on what a cognitive map is and
  how a person gains and maintains spatial
  awareness.
• Applying Interaction in visualizations
• Funded NSF proposal applies visualization to
  studying science policies
• Another funded NSF proposal applies visualization
  to discovering the causes and effects of civil
  strife
• Funded DHS proposal to evaluate the benefits of
  interactions
• Interaction Capturing (Provenance)
• Semi-automatic method for analyzing whats in the
  interaction logs.
• Look to collaborate with PNNL on developing
  generalizable structures for recording provenance.

37

• Thank you!
• rchang_at_uncc.edu
• http//www.viscenter.uncc.edu/rchang

38
Backup Slides
39
Journal Publications (11)

• Visualization
• Urban Visualization
• Remco Chang, Thomas Butkiewicz, Caroline
  Ziemkiewicz, Zachary Wartell, Nancy Pollard, and
  William Ribarsky. Legible simplification of
  textured urban models. IEEE Computer Graphics and
  Applications, 28(3)2736, 2008.
• Thomas Butkiewicz, Remco Chang, Zachary Wartell,
  and William Ribarsky. Visual analysis of urban
  change. Computer Graaphics Forum, 27(3)903910,
  2008.
• Thomas Butkiewicz, Remco Chang, William Ribarsky,
  and Zachary Wartell. Understanding Dynamics of
  Geographic Domains, chapter Visual Analysis of
  Urban Terrain Dynamics, pages 151 169. CRC
  Press/Taylor and Francis, 2007.
• Remco Chang, Ginette Wessel, Robert Kosara, Eric
  Sauda, and William Ribarsky. Legible cities
  Focus-dependent multi-resolution visualization of
  urban relationships. Visualization and Computer
  Graaphics, IEEE Transactions on, 13(6)11691175,
  Nov.-Dec. 2007.
• Visualization and Visual Analytics
• Dong Hyun Jeong, Caroline Ziemkiewicz, Brian
  Fisher, William Ribarsky, and Remco Chang. iPCA
  An interactive system for PCA-based visual
  analytics. Computer Graphics Forum, 2009. to
  appear.
• Remco Chang, Caroline Ziemkiewicz, Tera Marie
  Green, and William Ribarsky. Defining insight for
  visual analytics. IEEE Computer Graphics and
  Applications, 29(2)1417, 2009.
• Remco Chang, Alvin Lee, Mohammad Ghoniem, Robert
  Kosara, William Ribarsky, Jing Yang, Evan Suma,
  Caroline Ziemkiewicz, Daniel Kern, and Agus
  Sudjianto. Scalable and interactive visual
  analysis of financial wire transactions for fraud
  detection. Information Visualization,
  76376(14), 2008.
• Xiaoyu Wang, Erin Miller, Kathleen Smarick,
  William Ribarsky, and Remco Chang. Investigative
  visual analysis of global terrorism database.
  Computer Graphics Forum, 27(3)919926, 2008.
• Provenance
• Wenwen Dou, Dong Hyun Jeong, Felesia Stukes,
  William Ribarsky, Heather Richter Lipford, and
  Remco Chang. Recovering reasoning process from
  user interactions. IEEE Computer Graphics and
  Applications, 2009. to appear
• Graphics, Virtual Reality, and Interface Designs
• Thomas Butkiewicz, Wenwen Dou, Zachary Wartell,
  William Ribarsky, and Remco Chang. Multi-focused
  geospatial analysis using probes. Visualization
  and Computer Graphics, IEEE Transactions on,
  14(6)11651172, Nov.-Dec. 2008.
• Dong Hyun Jeong, Chang Song, Remco Chang, and
  Larry Hodges. User experimentation An evaluation
  of velocity control techniques in immersive
  virtual environments. Springer-Verlag Virtual
  Reality, 13(1)4150, Mar. 2009.

40
Conference/Workshop (19)

• Wenwen Dou, Dong Hyun Jeong, Felesia Stukes,
  William Ribarsky, Heather Richter Lipford, and
  Remco Chang. Comparing usage patterns of domain
  experts and novices in visual analytical tasks.
  In ACM SIGCHI Sensemaking Workshop 2009, 2009.
• Xiaoyu Wang, Wenwen Dou, Rashna Vatcha, Wanqiu
  Liu, Shen-En Chen, Seok-Won Lee, Remco Chang, and
  William Ribarsky. Knowledge integrated visual
  analysis of bridge safety and maintenance. In
  Defense, Security and Sensing 2009, 2009. to
  appear.
• Xiaoyu Wang, Wenwen Dou, William Ribarsky, and
  Remco Chang. Integration of heterogeneous
  processes through visual analytics. In Defense,
  Security and Sensing 2009, 2009. to appear.
• Michael Butkiewicz, Thomas Butkiewicz, William
  Ribarsky, and Remco Chang. Integrating timeseries
  visualizations within parallel coordinates for
  exploratory analysis of incident databases. In
  Defense, Security and Sensing 2009, 2009. to
  appear.
• Thomas Butkiewicz, Dong Hyun Jeong, William
  Ribarsky, and Remco Chang. Hierarchical
  multitouch selection techniques for collaborative
  geospatial analysis. In Defense, Security and
  Sensing 2009, 2009. to appear.
• Dong Hyun Jeong, Remco Chang, and William
  Ribarsky. An alternative definition and model for
  knowledge visualization. In IEEE Visualization
  Workshop on Knowledge Assisted Visualization,
  2008.
• Xiaoyu Wang, Wenwen Dou, Seok won Lee, William
  Ribarsky, and Remco Chang. Integrating visual
  analysis with ontological knowledge structure. In
  IEEE Visualization Workshop on Knowledge Assisted
  Visualization, 2008.
• Dong Hyun Jeong, Wenwen Dou, Felesia Stukes,
  William Ribarsky, Heather Richter Lipford, and
  Remco Chang. Evaluating the relationship between
  user interaction and financial visual analysis.
  In Visual Analytics Science and Technology, 2008.
  VAST 2008. IEEE Symposium on, 2008.
• Ginette Wessel, Remco Chang, and Eric Sauda.
  Towards a new (mapping of the) city Interactive,
  data rich modes of urban legibility. In
  Association for Computer Aided Design in
  Architecture, 2008.
• Ginette Wessel, Remco Chang, and Eric Sauda.
  Visualizing gis Urban form and data structure.
  In Dietmar Froehlick and Michaele Pride, editors,
  Seeking the City Visionaries on the Margins,
  96th Annual Conference of Association of
  Collegiate Schools of Architecture (ACSA), pages
  378384. Association of Collegiate Schools of
  Architecture, 2008.
• Ginette Wessel, Eric Sauda, and Remco Chang.
  Urban visualization Urban design and computer
  visualization. In CAADRIA 2008 Proceedings of the
  13th International Conference on Computer Aided
  Architectural Design Research in Asia, pages
  409416, Chiang Mai, Thailand, April 9-12, 2008.
• Thomas Butkiewicz, Remco Chang, Zachary Wartell,
  and William Ribarsky. Visual analysis for live
  lidar battlefield change detection. volume 6983,
  page 69830B. SPIE, 2008.
• Josh Jones, Remco Chang, Thomas Butkiewicz, and
  William Ribarsky. Visualizing uncertainty for
  geographical information in the global terrorism
  database. volume 6983, page 69830E. SPIE, 2008.
• Alex Godwin, Remco Chang, Robert Kosara, and
  William Ribarsky. Visual analysis of entity
  relationships in the global terrorism database.
  volume 6983, page 69830G. SPIE, 2008.
• Thomas Butkiewicz, Remco Chang, Zachary Wartell,
  and William Ribarsky. Analyzing sampled terrain
  volumetrically with regard to error and geologic
  variation. volume 6495, page 64950O. SPIE, 2007.
• Remco Chang, Mohammad Ghoniem, Robert Kosara,
  William Ribarsky, Jing Yang, Evan Suma, Caroline
  Ziemkiewicz, Daniel Kern, and Agus Sudjianto.
  Wirevis Visualization of categorical,
  time-varying data from financial transactions. In
  Visual Analytics Science and Technology, 2007.
  VAST 2007. IEEE Symposium on, pages 155162, 30
  2007-Nov. 1 2007.
• Remco Chang, Thomas Butkiewicz, Caroline
  Ziemkiewicz, Zachary Wartell, Nancy Pollard, and
  William Ribarsky. Hierarchical simplification of
  city models to maintain urban legibility. In
  SIGGRAPH 06 ACM SIGGRAPH 2006 Sketches, page
  130, New York, NY, USA, 2006. ACM.
• Remco Chang, Robert Kosara, Alex Godwin, and
  William Ribarsky. Towards a role of visualization
  in social modeling. AAAI 2009 Spring Symposium on
  Technosocial Predictive Analytics, 2009. to
  appear.
• Ginette Wessel, Eric Sauda, and Remco Chang.
  Mapping understandingTransforming topographic
  maps into cognitive maps. GeoVis Hamburg
  Workshop, 2009.

41
Grants Awarded (3)

• NSF SciSIP
• A Visual Analytics Approach to Science and
  Innovation Policy.
• PI William Ribarsky, Co-PIs Jim Thomas, Remco
  Chang, Jing Yang.
• 746,567. 2009-2012 (3 years).
• NSF BCS
• Collaborative Project Terror, Conflict
  Processes, Organizations, Ideologies
  Completing the Picture.
• PI Remco Chang
• 100,000. 2009-2010 (2 years).
• DHS International Program
• Deriving and Applying Cognitive Principles for
  Human/Computer Approaches to Complex Analytical
  Problems.
• PI William Ribarsky, Co-PIs Brian Fisher, Remco
  Chang, John Dill.
• 200,000. 2009-2010 (1 year).
• In Submission / Preparation
• 1 other NSF proposal is pending reviews
• 1 NSF and 1 NIH proposals are currently under
  preparation

42
Professional Activities

• Committee / Panelists
• Committee Member AAAI Spring-09 Symposium on
  Technosocial Predictive Analytics
• Panelist 3rd Annual DHS University Summit.
  Panel Research to Reality
• 3rd Annual DHS University Summit. Panel Visual
  Analytics and Discrete Science Integration into
  the DHS Center of Excellence Program
• Invited Talks
• Dec 13, 2006 Google Inc. Simplification of Urban
  Models based on Urban Legibility
• July 6, 2007 Naval Research Lab. Urban
  Visualization
• Oct 4, 2007 Charlotte Viscenter. Urban
  Visualization
• Oct 17, 2007 Charlotte Metropolitan GIS Users
  Group. GIS and Urban Visualization
• Nov 19, 2007 START Center at University of
  Maryland. Integrated Visual Analysis of the
  Global Terrorism Database
• Nov 29, 2007 Charlotte Viscenter. Integrated
  Visual Analysis of the Global Terrorism Database
• Jan 25, 2008 DoD/DHS Social Science Modeling and
  Information Visualization Symposium. Social
  Science and Information Visualization on
  Terrorism and Multimedia
• May 14, 2008 Charlotte Metropolitan GIS User
  Group. Multi-Focused Geospatial Analysis Using
  Probes
• Aug 27, 2008 DoD/DHS Symposium for Overcoming the
  Information Challenge in Federated Analysis From
  Concept to Practice. Roadmap of Visualization
• Mar 19, 2009 DHS University Summit. Panel
  Research to Reality
• Mar 19, 2009 DHS University Summit. Panel Visual
  Analytics and Discrete Science Integration into
  the DHS Center of Excellence Program

43
Introduction

• The common thread between all the journals and
  conference papers, talks, grants, etc. is the use
  of interactivity in graphics, visualization, and
  visual analytics.
• This presentation will focus on select projects
  and publications due to time constraint. For
  more information, see my website
  www.viscenter.uncc.edu/rchang

44
What is in a Users Interactions?

• Approach
• Using WireVis, we captured 10 financial analysts
  interactions in performing fraud detection.
• Financial analysts are from Bank of America,
  (previous) Wachovia, etc.
• Manufactured test dataset that has embedded known
  threat scenarios
• The sessions are captured using screen-capturing,
  video camcorder, and semantic interaction
  logging.
• The sessions were then converted (manually) to a
  text document
• High level Strategies
• Mid Level Methods of implementation
• Low Level Findings
• We then hired 4 graduate students to interpret
  the financial analysts interactions
• Using proprietary visualizations to examine the
  interaction logs

45
In a Tribute to Randy Pausch

• Here are my head fakes
• Visualizations are very useful for analysis of
  complex problems with large datasets.
• The Charlotte Visualization Center is a leader in
  the visual analytics community.