Title: Bayesian Networks for Sketch Understanding
1Bayesian Networks for Sketch Understanding
- Christine AlvaradoMIT Student Oxygen Workshop12
September 2003
2Sketching in Design
Mechanical Engineering
Software
3Computer Design Tools
4A Challenge In Sketch Understanding
There is no one threshold for shapes or
constraints?Interpretation depends on context
5Naïve Approach
- Why not just try all possibilities?
6Naïve Approach
- Why not just try all possibilities?
Arrow?
7Naïve Approach
- Why not just try all possibilities?
Arrow?
8Naïve Approach
- Why not just try all possibilities?
Arrow?
9Naïve Approach
- Why not just try all possibilities?
Arrow?
10Naïve Approach
- Why not just try all possibilities?
Arrow?
11Naïve Approach
- Why not just try all possibilities?
interpretations
Must consider
n number of strokes/segmentsS set
of shapeski subcomponents in shape Si
12Naïve Approach
- Why not just try all possibilities?
interpretations
Must consider
n number of strokes/segmentsS set
of shapeski subcomponents in shape Si
And this only considers shapes independently
13Previous Approaches Use Rigid Segmentation
- Single stroke shapes
- Palm Pilot Graffiti
- Long et. al. 1999
- Explicit Segmentation
- Quickset Cohen et. al. 2001
- Pause between strokes
14Recognition Using Partial Interpretations
- Recognition
- Build partial interpretations (PIs) as the user
draws based on easily recognizable low-level
shapes - Prune unlikely PIs and use likely PIs to find
misrecognized low-level shapes - Evaluating PIs ? Graphical Models
- Missing data unobserved nodes
- Interpretation influenced by top-down and
bottom-up information
15BN fragments similar to PRMs, Getoor et. al.
1999
(Define Arrow (Components (Line shaft)
(Line head1)
(Line head2)) (Constraints (connects
shaft.p1 head1.p1) (connects shaft.p1
head2.p1) ( head1.length head2.length) (head1.length shaft.length) (shaft) 90) (
(angle head1 shaft) 0) ( (angle shaft head2)
0)))
L1L2L3C1C2C3C4C5C6C7C8
Arrow
L1
L2
L3
C1
C2
C3
C8
Hammond and Davis, 2003
Instantiated and linked together as recognition
proceeds
16Specifying Conditional Probability Tables
Quadrilateral
Arrow
L1
- P(Child Parent) specified in description
- P(L1 Arrow) 1
- Probability of optional components/constraints
lower - Parents combine with noisy-OR (XOR)
- Priors specified for top-level shapes/patterns
17Primitive shapes/Constraints
- Observation node added when primitive linked to
stroke - P(ObsPrim) determined through data collection
L1
Obs
18Example
0.59
Force-pushes-body
Touches F B
0.95
Body(B)
Force(F)
0.5
Quad
0.95
Arrow
Remaining Arrow Constraints
0.97
0.99
0.99
Connects l1 l2
Line
Line
Line(l1)
Line(l2)
Line(l3)
Sq. error (Stroke a)
Sq. error (Stroke b)
Observation
19Example
0.59
Force-pushes-body
Touches F B
0.95
Body(B)
Force(F)
0.5
Quad
0.95
Arrow
Remaining Arrow Constraints
0.97
0.99
0.99
Connects l1 l2
Line
Line
Line(l1)
Line(l2)
Line(l3)
Sq. error (Stroke a)
Sq. error (Stroke b)
Sq. error (Stroke c)
Observation
20Example
0.61
Force-pushes-body
Touches F B
1
Body(B)
Force(F)
0.47
Quad
1
Arrow
Remaining Arrow Constraints
0.97
1
1
Connects l1 l2
0.95
Line
Line
Line(l1)
Line(l2)
Line(l3)
Sq. error (Stroke a)
Sq. error (Stroke b)
Sq. error (Stroke c)
Observation
21Example
0.95
Force-pushes-body
0.99
0.97
Touches F B
1
Body(B)
Force(F)
0.47
Quad
Observation
1
Arrow
Remaining Arrow Constraints
1
0.99
1
1
Connects l1 l2
1
Ellipse
Line
Line
Line(l1)
Line(l2)
Line(l3)
Sq. error (Stroke d)
Sq. error (Stroke a)
Sq. error (Stroke b)
Sq. error (Stroke c)
Observation
22Uh oh what about speed?
- Networks get very large (and ugly)
- Solutions
- Prune network
- Assert values (even if not confirmed)
- Approximate inference
- Incremental changes to inference structures
23Current/Future Work
- Expand domain/include other domains
- Gather sketches from users
24Conclusion
- Graphical models evaluation Partial
Interpretations ?Context-guided search - More drawing freedom More robust recognition
More natural interfaces (i.e. The goal
of OXYGEN)