Bayesian Networks for Sketch Understanding

1
Bayesian Networks for Sketch Understanding

• Christine AlvaradoMIT Student Oxygen Workshop12
  September 2003

2
Sketching in Design
Mechanical Engineering
Software
3
Computer Design Tools
4
A Challenge In Sketch Understanding

• Noisy Input

There is no one threshold for shapes or
constraints?Interpretation depends on context
5
Naïve Approach

• Why not just try all possibilities?

6
Naïve Approach

• Why not just try all possibilities?

Arrow?
7
Naïve Approach

• Why not just try all possibilities?

Arrow?
8
Naïve Approach

• Why not just try all possibilities?

Arrow?
9
Naïve Approach

• Why not just try all possibilities?

Arrow?
10
Naïve Approach

• Why not just try all possibilities?

Arrow?
11
Naïve Approach

• Why not just try all possibilities?

interpretations
Must consider
n number of strokes/segmentsS set
of shapeski subcomponents in shape Si
12
Naïve Approach

• Why not just try all possibilities?

interpretations
Must consider
n number of strokes/segmentsS set
of shapeski subcomponents in shape Si
And this only considers shapes independently
13
Previous Approaches Use Rigid Segmentation

• Single stroke shapes
• Palm Pilot Graffiti
• Long et. al. 1999
• Explicit Segmentation
• Quickset Cohen et. al. 2001
• Pause between strokes

14
Recognition Using Partial Interpretations

• Recognition
• Build partial interpretations (PIs) as the user
  draws based on easily recognizable low-level
  shapes
• Prune unlikely PIs and use likely PIs to find
  misrecognized low-level shapes
• Evaluating PIs ? Graphical Models
• Missing data unobserved nodes
• Interpretation influenced by top-down and
  bottom-up information

15
BN fragments similar to PRMs, Getoor et. al.
1999
(Define Arrow (Components (Line shaft)
(Line head1)
(Line head2)) (Constraints (connects
shaft.p1 head1.p1) (connects shaft.p1
head2.p1) ( head1.length head2.length) (lt
head1.length shaft.length) (lt (angle head1
shaft) 90) (lt (angle shaft head2) 90) (gt
(angle head1 shaft) 0) (gt (angle shaft head2)
0)))
L1L2L3C1C2C3C4C5C6C7C8
Arrow

L1
L2
L3
C1
C2
C3
C8
Hammond and Davis, 2003
Instantiated and linked together as recognition
proceeds
16
Specifying Conditional Probability Tables
Quadrilateral
Arrow
L1

• P(Child Parent) specified in description
• P(L1 Arrow) 1
• Probability of optional components/constraints
  lower
• Parents combine with noisy-OR (XOR)
• Priors specified for top-level shapes/patterns

17
Primitive shapes/Constraints

• Observation node added when primitive linked to
  stroke
• P(ObsPrim) determined through data collection

L1
Obs
18
Example
0.59
Force-pushes-body
Touches F B
0.95
Body(B)
Force(F)
0.5
Quad
0.95
Arrow
Remaining Arrow Constraints
0.97
0.99
0.99
Connects l1 l2
Line
Line
Line(l1)
Line(l2)
Line(l3)
Sq. error (Stroke a)
Sq. error (Stroke b)
Observation
19
Example
0.59
Force-pushes-body
Touches F B
0.95
Body(B)
Force(F)
0.5
Quad
0.95
Arrow
Remaining Arrow Constraints
0.97
0.99
0.99
Connects l1 l2
Line
Line
Line(l1)
Line(l2)
Line(l3)
Sq. error (Stroke a)
Sq. error (Stroke b)
Sq. error (Stroke c)
Observation
20
Example
0.61
Force-pushes-body
Touches F B
1
Body(B)
Force(F)
0.47
Quad
1
Arrow
Remaining Arrow Constraints
0.97
1
1
Connects l1 l2
0.95
Line
Line
Line(l1)
Line(l2)
Line(l3)
Sq. error (Stroke a)
Sq. error (Stroke b)
Sq. error (Stroke c)
Observation
21
Example
0.95
Force-pushes-body
0.99
0.97
Touches F B
1
Body(B)
Force(F)
0.47
Quad
Observation
1
Arrow
Remaining Arrow Constraints
1
0.99
1
1
Connects l1 l2
1
Ellipse
Line
Line
Line(l1)
Line(l2)
Line(l3)
Sq. error (Stroke d)
Sq. error (Stroke a)
Sq. error (Stroke b)
Sq. error (Stroke c)
Observation
22
Uh oh what about speed?

• Networks get very large (and ugly)
• Solutions
• Prune network
• Assert values (even if not confirmed)
• Approximate inference
• Incremental changes to inference structures

23
Current/Future Work

• Expand domain/include other domains
• Gather sketches from users

24
Conclusion

• Graphical models evaluation Partial
  Interpretations ?Context-guided search
• More drawing freedom More robust recognition
  More natural interfaces (i.e. The goal
  of OXYGEN)