Drawing Things Together: Integrating Modalities in Dialogue'

1
Drawing Things TogetherIntegrating Modalities
in Dialogue.

• MAGIC Multimodality and Graphics in Interactive
  Communication.
• EPSRC/ESRC PACCIT Initiative People at the
  Centre of CIT.
• Pat Healey, James King, Charlie Peters.
• Information, Media and Communication Research
  Group, QMUL.
• John Lee, Jon Oberlander.
• Human Communication Research Centre, University
  of Edinburgh.
• Simon Garrod, Nick Fay.
• Department of Psychology, University of Glasgow.
• Ichiro Umata, Yasuhiro Katagiri,
• ATR Media Integration and Communications
  Laboratories, Kyoto.

2
Outline

• Drawing-in-interaction ethnographic observations
• turn-taking and topic management
• Graphical Language Games Musical Pictionary
• Experiment 1 Community-specific Graphical
  Languages
• Experiment 2 Mechanisms of Interaction
• The mechanisms of interaction available to people
  directly constrain the form and organisation of
  shared symbol systems
• not just individual cognitive-computational
  abilities
• Augmented Human Interaction.

3
Design for Human-Human Interaction
4
Design for Human-Human Interaction?

• Single User Baton Design

• Multi-User Baton Design
• designed for exchange
• symmetric
• smooth
• centrally balanced

5
Example Architectural Design
6
Example Architectural Design
7
Ethnography of Architectural Design

• 6 staff working on a design competition
• mixed expertise and responsibilities
• Prepare 4 A2 presentation boards addressing
• use of site, environmental concerns, building
  use, open-space use
• 58 project interactions video taped over 4 weeks
• 40 two-party interactions
• 13 three-party interactions
• 5 interactions gt 3-party
• One 12 minute, 3 party interaction transcribed
  for analysis
• coded for overlaps, pauses, run-throughs, stress
  etc.
• coded for gesture and drawing activity

8
Basic Observations

• Complex variety of drawing spaces (in 12 minutes)
• 6 physical drawing spaces
• pieces of paper or regions of a piece
• 3 gestural drawing spaces
• drawing in the air with pen or finger
• verbal spaces

9
Excerpt 1
10
Turn Taking

• Drawing activity does not automatically claim the
  floor
• 1. Drawing activities continue across turn and
  speaker changes
• overlapping speech is avoided (e.g., Levinson,
  1983)
• gestures are used to bid for and maintain floor
  control (Bavelas, et. al. 1995) - but see
  Furuyama.
• 2. Where competition for the floor occurs during
  drawing
• drawing is suspended
• gesture speech used to compete for floor

11
Excerpt 2
12
Topic Management

• Drawing activity is used to manage topic changes
  ceases or shifts location if topic changes
• In excerpt 1 topic changes from rail to space.
• J moves hand to draw in air above the board
• reference to space is harder to resolve
• When topic returns to rail drawing resumes on the
  page
• (Earlier in interaction J draws same space over
  the drawing)

13
Observations

• Drawing, gesture and speech are integrated into
  composite communicative signals (cf. Neilson and
  Lee, 1994)
• often treated as separate channels (e.g.,
  Netmeeting)
• Drawing activities often serve interactional
  functions
• not only representation of domain or
  computational aid.
• use of space to manage topics
• cf. topographic and referential gesture spaces

14
Graphical Dialogue

• 50 of everyday drawings are produced as part of
  interaction
• (van Sommers, 1994)
• Examples of Graphical Dialogue
• Routine interactions
• e.g., sketch maps, explanatory diagrams, games
• Auxiliary mode of Communication
• e.g., cross-linguistic communication, aphasia
• Specialised interactions
• e.g., design interactions
• Does interaction affect representation?

15
Graphical Language Games Musical Pictionary

• What do we do if we dont share a symbol system?
• Criteria for Task
• communication task
• structured, regular, domain
• few (or no) established representational
  conventions
• exclusively graphical interaction
• Typical Set-up
• Subjects seated in separate (soundproof) rooms
• communication via shared whiteboard application
• 30 sec piano piece each
• Task draw picture of target no letters or
  numbers
• SAME or DIFFERENT

16
Room A
Room B

• One piano piece each same or different?
• Communicate by drawing no letters or numbers

17
Music Drawing Types

• 1. Abstract
• Graph-like representation of domain structure
  e.g., pitch, intensity, rhythm
• 2. Figurative
• Ad hoc associations faces, figures, objects or
  situations
• 3. Composite
• Mixture of Abstract and Figurative
• (independent classification by 2 judges Kappa
  0.9, N 287, k 2)

18
Sequence of Figurative Trials
19
Sequence of Abstract Trials
20
Abstract / Figurative Contrast

• Abstract drawings provide a more complex
  representational system for the task.
  Specifically
• Systematicity support direct comparison within
  and between items
• Proto-Compositionality distinct parts of the
  drawing refer to distinct parts of the music
• Figurative drawings are more holistic and more ad
  hoc.

21
What affects use of Abstract or Figurative?

• Previous (non) findings
• No effect of target on drawing type
• genre / tempo / mode
• No effect of medium on drawing type
• stylus vs. mouse
• No difference in drawing effort
• same average quantity of lines and ink

22
Effects of Interaction

• Repetition promotes abbreviation -but only if
  participants can signal understanding
• c.f. grounding in dialogue
• Dialogue partners tend to use drawings of the
  same type.
• c.f. accommodation/entrainment
• 3. Level of communicative interaction
• Concurrent Drawing
• 60 Abstract
• Alternate Drawing
• 60 Figurative

23
Hypotheses

• AUTONOMOUS CO-ORDINATION (truth wins)
• Participants independently migrate toward the
  representational scheme that is most efficient
  for the Concurrent task.
• co-ordination of time-based axis? more
  comparative?
• co-ordination emerges as aggregate individual
  experience
• (e.g., Clark, Lewis)
• COLLABORATIVE CO-ORDINATION
• Participants use the opportunities afforded by
  concurrent interaction to establish a
  co-ordinated 'sub-language'.
• collaborative revision and refinement of
  conventions
• co-ordination emerges through local histories of
  interaction

24
Experiment 1 Community Sub-Languages?

• Does interaction contribute anything, in addition
  to individual expertise, to the co-ordinated use
  of the Abstract drawings?
• Experimental Design
• Phase 1 develop several communities with
  equivalent task experience but different
  interaction histories
• Phase 2 compare interaction within and between
  communities.
• Note community membership is hidden from
  participants.

25
Phase1 Community Development
Subject 1
Subject 6
Round 1 Round 2 Round 3 Round 4
Subject 2
Subject 5
Subject 3
Subject 4
26
Phase 1

• 10 communities (66 people)
• seated round edge of large PC lab
• Music Task one piece each, same or different?
• 4 rounds of 12 trials
• different partner on each round
• During Phase 1
• common interaction history accumulates
• reliable increase in speed 53 sec. to 43 sec.
• reliable increase in accuracy 37 to 52

27
Phase 2 Experimental Manipulation
28
Results
Chi2(2) 19.0, p0.00
29
Results
30
Conclusions Experiment 1

• Cross-group interaction de-stabilises use of
  Abstract drawings
• independently of
• indivdual expertise
• what is being represented
• individual cognitive-computational abilities
• explicit knowledge of community membership
• (cf. Healey 1997).
• Support for Collaborative Co-ordination
  Hypothesis
• localised patterns of interaction lead to
  community-specific (graphical) dialects
• Why?

31
Why does interaction matter?

• Abstract drawings are proto-compositional
  interaction allows participants to co-ordinate
  meaningful elements of each others drawings

• Graphical Interaction Mechanisms
• localisation
• alignment.

32
Hypotheses

• 3. MUTUAL MODIFICATION coordinated use of
  Abstract drawings depends on ability to use
  interaction devices to annotate and modify
  elements of each other's drawings.
• (e.g., circling, underlining, and arrows)
• Experiment 2 interfere with use of interaction
  devices (localisation and alignment) and assess
  effects on communication.

33
Manipulation 1 Block Localisation
Room A Screen
Room B Screen
34
Manipulation 2 Block Alignment
Room A Screen
Room B Screen
Transpose
35
Experimental Design
BLOCKING
- BLOCKING
- TRANSNPOSITON

• Pairs seated in separate soundproof rooms
• Communicate via shared whiteboard- no letters or
  numbers

TRANSPOSITION
Subject As View
Subject Bs View
Subject Bs View
Subject As View
36
Effects of Interference with Interaction
(Blocking Chi2(3) 96.70, p 0.00,
Transposition Chi2(3) 81.61, p 0.00)
37
Conclusions Experiment 2

• Blocking and Transposition cause change in
  graphical conventions
• independently of what is being represented
• Independently of individual cognitive-computationa
  l abilities
• Complexity of emergent symbol system depends on
  mutual modification (localisation, alignment)
• co-ordinated manipulation of external
  representations
• participants ability to modify each others
  representations
• not editing / annotation / revision per se

38
General Conclusions

• Significant parallels between verbal and
  graphical dialogue
• grounding, accommodation, turn-taking,
• modality independent, interactional constraints
  on representation
• How do people co-ordinate despite differences in
  interpretation?
• Local, surface-based, manipulations of external
  representations.
• Verbal and graphical repair mechanisms
• Localisation
• specficity, reprise fragment clarifications
• Alignment
• sequential relevance, embedded repair

39
General Conclusions

• Drawing supports transactional and interactional
  functions
• not just a representational medium
• Not explained by physical / perceptual /
  computational properties per se
• alternate vs. concurrent drawing
• Modalities combine to form composite
  communicative signals applications often treat
  them as separate channels (e.g., net-meeting)
• Generic interaction mechanisms are apparent
  across modalities
• turn-taking, topic management, repair, grounding,
  accommodation
• Augmented interaction mechanisms ? richer more
  expressive languages?

40
Augmented Human InteractionBeyond Face-to-Face

• Capture gesture, expression, attention,
  engagement, understanding
• Provide augmented cues for interaction
• Enable richer, more robust, forms of human
  communication
• gestural languages, musical languages, dance

41
Augmented Human InteractionBeyond Face-to-Face

• Capture gesture, expression, attention,
  engagement, understanding
• Provide augmented cues for interaction
• Enable richer, more robust, forms of human
  communication
• gestural languages, musical languages, dance

42
Augmented Human Interaction Lab

• multi-user, real-time, full body, motion capture.
• integrated 3D audio display

43
Augmented Human Interaction Lab