Automated Generation of Visual Discourse

1
Automated Generation of Visual Discourse

• Michelle X. Zhou
• Department of Computer Science
• Columbia University
• New York, NY 10027
• Supported in part by DARPA Contract
  DAAL01-94-K-0119, New York State Science and
  Technology Foundation, NSF Grant ECD-88-11111,
  and ONR Contract N00014-97-1-0838

2
Automated Visual Design

• Problem
• Designing effective visual presentations is
  difficult and costly
• Designing customized visual presentations in a
  timely manner is more difficult
• Approach
• Develop computer techniques to automate visual
  design process

3
Automated Visual Design
4
Previous Work

• Single displays

APT Mackinlay 86 SAGE Roth Mattis 91 ANDD Marks
91

• A series of displays

APEX Feiner 85 IBIS Seligmann 91 WIP Andre et al.
93
. . .
. . .
5
Our Goal Visual Discourse Design

• A series of connected displays

Open Rotate Scale Move . . .
6
Thesis Work
System Development
General Approach
7
Thesis Work
Visual Discourse Modeling

• Coherence
• Versatility
• Interactivity

System Development
General Approach
8
Visual Discourse Modeling

• Coherence
• Versatility
• Interactivity

Continuity Consistency Unity
Wide range of information Wide variety of visual
media/techniques
Interruptible Responsive
9
Thesis Work
Design Foundation

• Data characterization
• Visual task hierarchy
• Presentation design
• language
• Inference paradigm

Visual Discourse Modeling

• Coherence
• Versatility
• Interactivity

System Development
General Approach
10
Design Foundation Design Process
Input
Presentation Data
Presentation Context
Design Engine
Presentation Intents
Design Knowledge
11
Design Foundation
Input
Data Characterization
Presentation Data
Presentation Context
Design Engine
Presentation Intents
Design Knowledge
12
Data Characterization
Data
Visual Elements
Goal
Previous Work
Characterizing Quantitative data Mackinlay
86 Roth Mattis 90
Presentation-related data properties
Characterizing Qualitative data Arens et al. 93
13
Our Approach Characterizing Heterogeneous Data
Data Dimensions
Domain Type Attribute Relation Role Sense
(Jones (is-a PATIENT) (type ATOMIC)
(property (Form )...) (relation (rel )...)
(role LOCATE) (sense SYMBOL))
Presentation-related data properties
14
Data Characterization Taxonomy
15
Design Foundation
Input
Data Characterization
Presentation Data
Context Modeling
Presentation Context
Design Engine
Presentation Intents
Design Knowledge
16
Context Modeling
Goal
Context Information
Visual Techniques
Previous Work Situation space Friedell 83
Display categories Mackinlay 86
Our Approach Audience Occasion
Environment
17
Design Foundation
Input
Data Characterization
Presentation Data
Context Modeling
Presentation Context
Design Engine
Presentation Intents
Design Knowledge
Intent Modeling
18
Intent Modeling
Goal
Presentation Intents
Visual Techniques
Previous Work
19
Our Approach Visual Task Characterization
Visual Techniques
Presentation Intents
Visual Tasks
(Visual Effects)
Enlarge Highlight Zoom
Search Elaborate
Focus
Abstract Visual Technique
20
Generalize Merge Identify Name Portray
Individualize Profile Locate Position
Situate Pinpoint Outline Rank
Time Reveal Expose Itemize Specify
Separate Switch
Background Associate Colocate Connect
Unite Attach Categorize MarkDistribute Clust
er Outline MarkDistribute Correlate
Plot MarkCompose Compare Differentiate
Intersect Distinguish MarkDistribute
Isolate Emphasize Focus Isolate Reinforce
Encode Label Symbolize Quantify
Iconify Portray Tabulate Plot Trace
Structure Map
Visual Task Taxonomy
21
Design Foundation
Input
Data Characterization
Presentation Data
Context Modeling
Presentation Context
Design Engine
Design Knowledge Modeling
Presentation Intents
Design Knowledge
Intent Modeling
22
Design Knowledge Modeling
Goal Computational representation of design
knowledge
Previous Work Visual formalisms Marks 91
Lohse et al. 94 Visual techniques Friedell
84 Seligmann 93 KellerKeller 94 Visual
design principles WinnHolliday 82 Bertin 83
MulletSano 95 Mackinlay 86 IgnatiusSenay 94
Tufte83, 90, 97
23
Our Approach Presentation Design Language

• Visual objects
• Represent various visual formalisms
• Visual techniques
• Compose/manipulate visual objects
• Visual design principles
• Guide the visual object composition and
  manipulation

24
Visual Object Representation

• Syntax
• Semantics
• Pragmatics

Patterns/compositions
Meanings
human heart
Specific meanings
25
Visual Object Hierarchy
Static/Dynamic
Tables Charts Diagrams
Shape/Color/Size/Orientation/...
26
Visual Techniques
(DesignTableChart (is-a FormationTech)
(operands ?data-obj ?table)
. . .)

• Categorized by function
• Formation
• Transformation
• Camera

(Move (is-a TransformationTech) (operands
?obj) (source) (destination) (startTime)
(endTime))

• Categorized by usage
• Primitive
• Composite

(SetCamera (is-a CameraTech) (operand
?camera) (position) (orientation) . . .)
27
Visual Design Principles

• Expressiveness rules
• Effectiveness rules

• Comprehensiveness distinctiveness
• Generality discreteness
• Integrity

x
28
Visual Design Principles

• Expressiveness rules
• Effectiveness rules

• Comprehensiveness distinctiveness
• Generality discreteness
• Integrity

• Accuracy clarity
• Appropriateness
• Immediacy
• Continuity
• Consistency
• Unity

29
Visual Design Principles

• Representation
• Generality
• Specificity

Usage Conflict resolution
IF (?obj (is-a VIS-OBJ) . . .) (?machine
(graphics NONE)) THEN Wireframe(?obj) IF
(?obj (is-a VIS-OBJ) . . .) (?audience
(type VIP)) THEN Solid(?obj)
Order by specificity
Order by conditions
30
Design Foundation
Input
Data Characterization
Presentation Data
Inference Modeling
Context Modeling
Presentation Context
Design Engine
Design Knowledge Modeling
Presentation Intents
Design Knowledge
Intent Modeling
31
Inference Modeling
Goal Flexible and efficient design method
Previous Work Constructive design Mackinlay
86 Roth Mattis 90 Marks 91 Parametric
design Zdybel et al. 81 Robertson 91
IgnatiusSenay 94
32
Our Approach Hybrid Inference Paradigm

• Constructive synthesis
• Parametric synthesis

Planning presentations from scratch
Efficiently create visual models for atomic data
objects
A least-commitment constraint-based approach
33
Planning Elements Features

• Visual Discourse
• Collection of frames

• Plan
• Collection of actions

Practical hierarchical-decomposition
partial-order planning
34
Extended Features
Dynamic numerical constraints Domain
heuristics
35
Action/Object Decomposition
DesignTableChartlt?domain-obj, ?tablegt
DesignVisReplt?part1,?rep1gt DesignVisReplt?part2,
?rep2gt . . . ?domain-obj lt?part1, ?part2, . .
.gt ?table lt?rep1, ?rep2, . . .gt
DesignHeadinglt?part1,?headgt DesignCelllt?part2,
?cell1gt . . . ?domain-obj lt?part1, ?part2, . .
.gt ?table lt?head, ?cell1, . . .gt
36
Thesis Work
System Design

• Knowledge base
• Inference engine
• Visual realizer
• Interaction handler

Visual Discourse Modeling

• Coherence
• Versatility
• Interactivity

37
System Framework
38
Thesis Work
IMPROVISE

• Computer network
• management
• Patient medical
• record summary

System Design

• Knowledge base
• Inference engine
• Visual realizer
• Interaction handler

Visual Discourse Modeling

• Coherence
• Versatility
• Interactivity

39
IMPROVISE (Illustrative Metaphor PROduction in
VISual Environments)

• Stand-alone graphics system
• Computer network management
• Graphics generator in a multimedia presentation
  system
• Healthcare

40
Architecture
Task Analyzer
Interaction Handler
Presentation Planner
Content Planner
Designer
Chooser Organizer
Stylist Coordinator
Knowledge Base
Converter
Renderer
41
Example Present Patient Information to Nurse
42
Sample Visual Discourse Representation
(Discourse1 (is-a VISUAL-DISCOURSE) (frames
Frame1 Highlight2 . . .))
(Frame1 (is-a VISUAL-FRAME) (frameElements
StructDiag1) (startTime 0.0) (endTime
-1.0))
(Highlight2 (is-a Highlight) (operands
Table1-demo) (style MARKER) (color . .
.) (startTime 5.0) (endTime 8.0) . . .)
(StructDiag1 (is-a STRUCTURE-DIAGRAM) (heading
Table1-Demo) (core Unity1-body)
(elements Unity2-device1 . . .) . . .)
43
Input Presentation Data
(Jones-name (is-a NAME-ATTR) (value S.
Jones) (role IDENTIFY) (sense LABEL). . .)
44
Input Presentation Context
(Context-info (audience (identity NURSE) . .
.) (occasion (presentation ON-LINE) (medium
VISUAL-SPEECH) . . .) (environment (display
(color COLOR) (size . . .) . . .) (platform (cpu
SGI R4400) . . .)))
45
Input Presentation Intent Visual Task Formation
Intent SummarizeltJones-infogt
StructureltJones-infogt
IdentifyltJones-info, Jones-demogt LocateltJones-line
, Jonesgt SymbolizeltJones-linegt
NameltJones-demo, Jones-namegt ItemizeltJones-demogt
NameltJones-line, line-namegt AssociateltJones-line,
line-contentgt
46
Fulfill Visual Tasks
Intent SummarizeltJones-infogt
DesignStructureDiagram ltJones-info, ?diaggt
StructureltJones-infogt
IdentifyltJones-info, Jones-demogt LocateltJones-line
, Jonesgt SymbolizeltJones-linegt
ItemizeltJones-demogt NameltJones-demo, Jones-namegt
NameltJones-line, line-namegt AssociateltJones-line,
line-contentgt
47
Structure Diagram
Patient Info
48
Fulfill Visual Tasks
Intent SummarizeltJones-infogt
DesignStructureDiagram ltJones-info, ?diaggt
StructureltJones-infogt
DesignVisRepltJones-demo, ?hgt DesignVisRepltJones,
?coregt DesignVisRepltJohns-line, ?elgt
IdentifyltJones-info, Jones-demogt LocateltJones-line
, Jonesgt SymbolizeltJones-linegt
ItemizeltJones-demogt NameltJones-demo, Jones-namegt
NameltJones-line, line-namegt AssociateltJones-line,
line-contentgt
49
Demographics
50
Fulfill Visual Tasks
Intent SummarizeltJones-infogt
DesignStructureDiagram ltJones-info, ?diaggt
StructureltJones-infogt
DesignVisRepltJones-demo, ?hgt DesignVisRepltJones,
?coregt DesignVisRepltJohns-line, ?elgt
IdentifyltJones-info, Jones-demogt LocateltJones-line
, Jonesgt SymbolizeltJones-linegt
DesignTableltJones-demo, ?tgt DesignVisRepltJones-nam
e, ?thgt
ItemizeltJones-demogt NameltJones-demo, Jones-namegt
NameltJones-line, line-namegt AssociateltJones-line,
line-contentgt
51
Core Locate
Patient
52
DesignTableltjones-demo, ?tablegt
53
Heading (Table)
Structure Diagram
54
Heading (Table)
Structure Diagram
Name Associate
Core (Unity)
55
Appropriateness Interactivity
Clarity Consistency
STM (Gleicher 95)
56
Contributions

• Definition and modeling of visual discourse
• Methodologies for automated visual discourse
  design
• Approaches to practical system design
• Design and implementation of IMPROVISE

57
Future Work

• Conversational capability
• Explanation capability
• Adaptive capability

58
Thesis Committee MAGIC Group Project Students
Graphics Group Mike Gleicher
Thank You
59
(No Transcript)
60
Exploring Network Entity
(Node1 (is-a NETWORK-NODE) (type COMPOSITE)
(components node-ports) (convey node1-name
node1-loc) (sense SYMBOL) . . .) (Link1
(is-a PHY-CONNECTION) (type COMPOSITE)
(components vpath-segments) (sense SYMBOL)
(source node1) (destination node2) . . .)
61
Exploring Network Link
(Task (Elaborate (link23)))
62
Exploring Network Link
(Task (Elaborate (link23)))
(Elaborate (link23))
63
Exploring Network Entity
Composite Tech Focus (?objX)
Preference low-intersection(?objX)
Preference reduce-intersection(?objX)
Decomposition1 Enlarge(?objX)
Decomposition3 Separate(?objX, ?rest)
Decomposition2 Highlight (?objX)
64
Exploring Network Entity
Composite Tech Expose (?objX)
Preference Openable (?objX)
Preference ok-show-all (?objX)
Decomposition1 Open (?objX)
Decomposition3 SetTransparency (?objX)
Decomposition2 CutAway (?objX)
65
Exploring Network Link
Focus(link23)
Expose (link23)
66
Exploring Network Link
Complete Plan
Move (link23, node1, node5) Scale (link23, node1,
node5) Display (link23-Seg1). . .Display
(relation (link23-Seg1, Capacity)). . .
Align (network1)
Open(link23)
Time
67
Exploring Network Link