Evaluating the Quality of Image Synthesis and Analysis Techniques

1
Evaluating the Quality of Image Synthesis and
Analysis Techniques

• Matthew O. Ward
• Computer Science Department
• Worcester Polytechnic Institute

2
Evaluation is Important

• Has my modification improved the results?
• Which method works best for my data?
• What are limitations of my technique?
• Is my method better than XXX?

3
Evaluation is Difficult

• What aspects to test?
• How to measure?
• What are limitations of evaluation procedure?
• How to recruit evaluators?

4
Evaluation Often Avoided

• Majority of papers show no substantive evaluation
• Most common approach is subjective, by authors,
  on 1-3 test cases
• Quantitative measures exist for computational
  performance, but not quality of results
• Not much glory in evaluation

5
Case Study 1 Data Visualization
6
The Problem.....

• Lack of rigorous assessment of visualization
  techniques
• Lack of good test cases
• Limited comparison with other techniques
• Lack of guidelines for selection of appropriate
  techniques

7
A Possible Solution.....

• Create a list of goals of visualization
• what is the overall task?
• what is desired/acceptable level of accuracy?
• what are we looking for?
• Locate/create data sets which contain desired
  features
• Test users on a wide range of tasks using
  different visualization methods

8
Goals of Visualization

• Identification - is there some interesting
  feature in the image?
• Classification - what is it?
• Quantification - how many? how big? how close?
• Understanding - are there correlations conveyed
  by the image?
• Comparison - does the image have characteristics
  similar to one generated with a different set of
  data?

9
Advantages of Synthetic Data

• Easy to adjust characteristics
• Less ambiguous than real data
• Easy to create data which contains a single
  structure or phenomena
• Real data can be noisy
• Hard to find real data with desired
  characteristics

10
Advantages of Real Data

• Results using real data are more believable
• Reality is hard to simulate accurately
• Real data has context which can help justify
  usefulness of tasks

11
Our Experiments

• Select two data characteristics of interest
  (outliers and clusters)
• Locate real data sets containing these features
  (validate with statistical analysis)
• Create synthetic data sets containing these
  features (also validate)
• Select three visualization techniques to test
  (scatterplots, parallel coordinates, principal
  components analysis with glyphs)

12
Our Experiments (continued)

• Train subjects on interpreting different display
  techniques
• Train subjects on the desired data
  characteristics
• Test subjects on each characteristic, varying
• number of outliers/clusters
• degree or size
• amount of noise in synthetic sets
• location of outlier/clusters

13
Visualization Techniques Tested
Glyphs
Scatterplot Matrix
Parallel Coordinates
14
Outlier Example
15
Cluster Example
Original
Added Noise
16
Assessing the Results

• Detection - did subjects identify some structure
  in the image?
• Classification - did subjects correctly classify
  structure?
• Measurement -
• number of clusters or outliers
• outlier and cluster degree of separation
• size of cluster
• Errors - false positives, missed structure,
  measurement accuracy

17
Summary of Experiments

• Scatterplot matrix
• best overall
• weak on overlapping clusters, size estimation for
  large clusters, interior outliers
• Glyphs
• best for interior outliers
• good for conveying outlier separation,
  overlapping clusters, measuring cluster size
• poor for differentiating non-outliers
• Parallel coordinates
• generally worse than others
• good for differentiating non-outliers

18
Future Work

• Test alternate data characteristics (e.g.
  repeated patterns)
• Test alternate perceptual tasks (e.g.
  correlation)
• Test other visualization techniques (e.g.
  alternate glyphs, VisDB, dimensional stacking..)
• Create publicly available benchmark suite for
  data sets and analysis tools (submissions from
  other researchers always welcome)
• Compare other multivariate visualization
  assessment methods as they arise.

19
Case Study 2 Image Segmentation
20
Problem Statement

• Image segmentation algorithms traditionally
  classified as model-based or context-free
• Model-based methods highly effective, but
  expensive to design and execute
• Context-free methods are fast, but quality of
  results often poor
• Is there some way to improve the results of
  context-free systems without incurring costs of
  model-based methods?

21
Conjecture

• In most image analysis domains, expectations can
  be placed on the likely occurrence of certain
  shapes, colors, and region/segment sizes.
• Objects in an office scene mostly planar and
  non-specular
• In medical images, boundaries are mostly smooth,
  and regions are usually small or moderate in size
• Outdoor scenes contain a lot of fine texture
• We should be able to use high-level domain
  constraint knowledge to improve the segmentation
  process by
• Selecting a segmentation method likely to produce
  good results
• Set the segmentation parameters to their most
  effective values

22
Defining a Good Segmentation

• All physical object boundaries should be isolated
• False boundaries should be minimized
• Boundary shape should be comparable to internal
  model of object in scene
• Precision in shape and position needed varies
  based on application and importance of individual
  objects to task at hand

23
Defining a Good Evaluation Procedure

• Should be based on real images
• Influence of human subjectivity minimized
• Errors categorized by type, severity, and
  significance
• Magnitude of error should accurately reflect
  difference from ideal
• Tolerance must be permitted

24
Problems with Pixel Counting
2 images with similar error counts, uniform
dilation (left) and bad merge (right).
25
Procedure

• Acquire representative set of images for multiple
  domains
• Approximate constraints on edge/region features
  in domain
• Interactively segment and label edges/region
  tolerance and priority to create ideal
  segmentation
• Compute errors between ideal and algorithmically
  generated segmentation
• If error gt acceptable, adjust parameters (simplex
  algorithm) and recompute errors
• Associate segmentation parameters with domain
  constraints

26
Creating the Ideal Segmentation

• Start with initial region-based segmentation
• Click on a region of interest
• Merge, split, set tolerance level, set priority
  level
• Iterate until all significant regions labeled
• Results are domain and task specific

27
Comparing Ideal to Computed Results
Edge Detection 78 detected essentials
209 oversegmentation
Region Growing 67 detected essentials
120 oversegmentation
28
More Results
Split and Merge 79 detected essentials
73 oversegmentation
Rule-based System 88 detected essentials
93 oversegmentation
29
Summary and Future Work

• Domain constraints produced better segmentations
  than context-free methods (after training)
• Future work includes investigating other types of
  constraints (e.g., texture) and improve the
  tolerance specification and error calculation

30
General Procedure for Assessing Image-Based
Algorithms

• Determine the task to be performed by user of
  image
• Determine image features most relevant to this
  task, and ascertain level of accuracy needed in
  detection, classification, and measuring
• Create benchmark suite of data containing these
  features in varying degrees (real and synthetic
  data)
• Create and administer user tests to evaluate
  effectiveness of algorithm to accurately and
  reliably convey desired data features, or
• Develop image processing algorithms to identify
  desired data features and calculate error types
  and severities in images generated by algorithm
  being assessed

31
Summary of Presentation

• Formal assessment has proven useful in both
  visualization and image processing applications
• Results can be used to guide algorithm
  development and selection
• Quantitative and qualitative approaches can
  provide many insights into effectiveness of image
  analysis and synthesis tasks