MacVisSTA

1
MacVisSTA Annotation Graphs

• R. Travis Rose, Francis Quek
• Center for Human-Computer Interaction Department
  of Computer Science
• Virginia Tech
• http//www.hci.vt.edu http//vislab.cs.vt.edu

2
Round-trip Conversion of MacVisSTA Annotations
to and from Annotation Graph Format

• Visualization for Situated Temporal Analysis

3
Research Goals

• Multimodal communication and interaction research
• Making new discoveries in patterns of human
  communication and interaction
• Corpus creation management
• Annotation/analysis

4
VISSTA MULTIPLE-LINKED REPRESENTATION
5
MacVisSTA

• Mac OS X port/upgrade from Unix/X-windows system

6
Characterization of MacVisSTA relative to other
selected tools

• Single timeline and multiple tiers
• Multiple-linked representation
• Similar to Praat, Anvil, ELAN, EXMARaLDA, etc.
• Flexible visualization
• Flexible annotation

7
Unique to MacVisSTA

• Native to Mac OS X
• Only tool that was developed natively to support
  coarse-to-fine annotation/analysis
• Supports analysis with a Snap-together
  interface, using drag-and-drop specification
• Has a plugin architecture is easily extendible
• Has audio priority and video priority playback
  modes
• Database support, both embedded and shared
  (MySQL) implementation
• MacVisSTA was developed with direct input from
  real users, annotation experts _at_ U. of Chicago

8
Continuous and Discrete Data

• Continuous signals
• Audio, motion tracking, etc.
• Discrete data
• Fine-grained observations, time-aligned speech,
  etc.

9
Visualizing Continuous Data Co-Temporally with
Video
10
Speech, Gesture, and Gaze
11
What worked well?
12
Sample Annotation File

• Uses Apples plist format
• Annotation Graph (AG) compatible
• Convention each tier is associated with a type
• Caveat multiple types in the same AG file must
  also be separated into individually named tiers

13
Implementation Details

• Converter written in Python (2.3 or higher), uses
  minidom to parse XML files
• minidom lightweight DOM
• Imports/exports execute as a batch process
• Available as stand-alone scripts
• Also packaged with MacVisSTA as an embedded
  resource
• Interface
• Drag-and-drop import/export for AG (.xml) and
  MacVisSTA files

14
Importing from Annotation Graph XML files
15
Common data clip Pear story from ISGS 2005

• Round-trip conversion of Pear story
  annotations worked without loss of information
• Including the use of color tags encoded as a
  feature!
• Other annotations tried
• AFIT gaze annotations
• 5000 gaze transitions for 42 minute meeting
• Converted in 3 minutes on 2.0 GHz Mac

Courtesy of Irene Kimbara Courtesy of Amy
Franklin, Haleema Welji, and others
16
What didnt work well?
17
Notes Notebooks Allow Overlapping Time
Intervals
18
MacVisSTA Project-based System

• Notes/notebooks part of the users project file
• Not translated into Annotation Graph (yet)
• Additional metadata
• Location of the users data space (read/write)
• Location of shared data (extracted features,
  read-only annotations, etc.)
• Location of shared media (audio, video, etc.)
• Not translated

19
Future Work

• New improved interfaces for scrubbing audio
• Freely arranging notes into different
  user-defined groups (using the new brushed-metal
  interface, similar to iTunes)
• Additional interfaces for built-in querying of
  the data, including
• a simple grammar for temporal querying
  (sequences)
• Vertical net to extract overlapping events
• Inter-converting MacVisSTA project and metadata
  details

20
When will the rest of MacVisSTA be ready?

• On or before 7-12-2007 (thesis defense)
• The program source code will be freely
  available on SourceForge for download
• http//sourceforge.net/projects/macvissta

21
Acknowledgments

• AFIT (Air Force Institute of Technology)
• NSF (National Science Foundation)
• Video Analysis and Content Extraction program
• NIST (National Institute of Standards
  Technology)
• VISLab _at_VT
• Vislab _at_UIUC
• Purdue, Univ. of Maryland
• McNeill Lab
• Advisor Prof. Francis Quek
• and others!!

22
(No Transcript)
23
Collaboration
24
Thank You!

• Questions/comments
• Travis Rose
• rtrose_at_vt.edu

Photo courtesy of S. Karjalainen