A Joint Conditional Model for Grouping And Labeling Ink Fragments

1
A Joint Conditional Model for Grouping And
Labeling Ink Fragments

• MSRC Internship Talk
• Philip Cowans
• Mentor Martin Szummer
• June 10th 2004

2
Overview

• Task overview
• Description of the model
• Labeling only model
• Labeling and grouping model
• Training and inference
• Examples and results
• Further work

3
The Task Interpreting Ink
Hand-drawn diagram
Machine interpretation
4
Previous Work

• Kumar, Hebert
• Discriminative random fields.
• Qi, Szummer
• Bayes Conditional Networks
• Peña Centeno, Svensén, Bishop
• Variational CRFs.
• Saund
• Perceptual organization.

5
Task Stages

• Divide the input data into ink fragments
• Then
• Group the fragments into perceptually salient
  objects
• And
• Label each object as either a container or
  connector

6
Example - Input
7
Example - Fragmentation
8
Example Grouping / Labeling
9
Fragmentation

• Input data consists of ink strokes, described by
  sampled pen locations.
• Strokes may span multiple objects.
• Divide strokes into straight (within a tolerance)
  fragments.
• Fragments are assumed to lie in a single object.

10
The Model

• Use a Conditional Random Field
• Undirected graphical model conditioned on
  observed data.
• No need to model ink data (which we already
  know).
• For now, just consider labeling.

observation
interaction
11
Potentials

• Observation

• Interaction

• Feature Vectors

• Weight Vectors

12
Nonlinearities

• Exponential
• Probit (Gaussian CDF)
• Epsilon is a noise parameter.

13
(No Transcript)
14
Graph Construction

• Construct a graph with one vertex per ink
  fragment.
• Add edges between nearby fragments.
• Triangulate graph.
• Calculate interaction potentials for all edges.

15
Graph Construction II

• Speed is dependent on the tree-width of the
  graph.
• By constraining the tree-width it should be
  possible to limit the complexity of the algorithm.

16
Features

• Spatial features
• Angles, lengths, distances
• Temporal features
• Stroke ordering, same stroke
• Template features
• T-Junction identification
• 61 observation, 37 interaction features in total

17
Priors

• Three feature categories
• Binary
• Counting
• Continuous
• Continuous features are replaced by histograms
• Use a correlated Gaussian prior on continuous
  features

18
Example Weights
Container
Connector
19
Labeling And Grouping

• Use a slight modification of the model
• Now three interaction categories
• Same object (implies same label)
• Different object, same label
• Different object, different label

combined grouping and labeling
20
Advantages Of This Approach

• Joint labeling
• Makes use of contextual information
• Simultaneous labeling and grouping
• Processes can reinforce each other
• Handles groups directly
• Doesnt define groups via labeling
• Results in a cleaner model

21
Training

• Train by finding MAP weights on example data.
• Training data set was 40 diagrams, from 17
  subjects with a total of 2157 fragments.
• Maximise using quasi-Newton gradient ascent (BFGS
  in MATLAB).
• Evaluate function and gradient using message
  passing.

22
Message Passing

• Just like standard sum-product, but we are
  summing over groupings and labelings together
  rather than just labelings.
• Fortunately efficient computation via message
  passing is still possible.
• Messages are now a list of possible groupings and
  labelings of the separator variables.

23
Message Passing II

• Message update rule
• Marginals

24
Message Passing III
2,9
1,7,8
1,2,3,4
2,3,4,5
x22
4,5,6
25
Inference

• Unseen data is processed by finding the most
  probable grouping and labeling.
• Use the max-product variant of the message
  passing algorithm.
• Messages are now the best you can do given the
  separator configuration.

26
Example - Raw Ink
27
Example Labeling Only
28
Example Labeling And Grouping
29
Example 2 - Raw Ink
30
Example 2 Labeling Only
31
Example 2 Labeling And Grouping
32
MATLAB Demo
33
Grouping Error

• Measured using the metric
• Not really sure how this relates to perceptual
  error.
• Theres potentially a better way of doing this
  via mutual information.

34
Results Accuracy
35
Results Speed

• Median evaluation time for labeling only
• 0.35 seconds.
• Median evaluation time for labeling and grouping
• 13 seconds.
• (And thats not very optimised, using MATLAB!)

36
Conclusions

• Exact inference is possible in reasonable time.
• This approach is capable of providing
  high-quality results.
• Joint labeling improves performance.
• Labeling and grouping simultaneously helps too.

37
Further Work

• Multiclass model
• Dashed lines, text, arrows,
• Relationships
• Which containers are connected together.
• More global features
• Closed paths, convex containers,
• Possibly via re-ordering of N-best lists.
• More data, better features,

38
Finally

• Thanks to everyone who has made my stay at
  Microsoft so enjoyable.

39
Observation Weights
40
Interaction Weights
Same Label, Same Group
Same Label, Different Group