Paint Selection

1
Paint Selection

• Jiangyu Liu

University of Science and Technology of China
Jian Sun
Microsoft Research Asia
Heung-Yeung Shum
Microsoft Corporation
2
Interactive image selection
3
Motivation
1.5GHz
lt 1Mp
2004 (Lazy Snapping, GrabCut)
2009 ( ? )
time

4
Related work

• Lazy Snapping Li et al. SIGGRAPH 2004
• UI scribble wait scribble
• Speed slow on high-resolution images (gt 20Mp)

5
Related work

• GrabCut Rother et al. SIGGRAPH 2004
• UI grab scribble
• Speed slow on high-resolution images (gt 20Mp)

6
Related work

• Random Walk Segmentation Grady PAMI 2006
• UI seed-driven multi-label segmentation
• Speed 2.5s for a 256 x 256 image (MATLAB code)

7
Related work

• Geodesic Matting Xue et al. ICCV 2007
• UI scribble-based segmentation/matting
• Speed 2.3s for a 3Mp image (binary segmentation)

8
Related work

• Quick Selection Tool Adobe Photoshop CS 34
• UI preview refinement
• Speed slow on panoramas (gt 100Mp)

24.5Mp
During mouse-dragging
After mouse-up
9
The challenges instant feedback

• Instant accurate feedback on a 20Mp image?
• 20Mp 20,975,120 pixels
• Global method vs. local method?
• How to take advantage of multi-core architecture?

10
The challenges user interface

• Low efficiency to manually correct erroneous input

Scribble error
Inefficient correction
11
The challenges user interface

• Fluctuation effect

12
Our approach Paint Selection
Key observation
Paint Selection
13
Key observation
Image selection is a progressive procedure!
14
System overview
Paint Selection
15
Progressive selection
F existing selection
U background
B user brush
F new selection
F ? F F
16
Progressive selection

• Color
• Fore-colors from S L
• Back-colors from U
• Hard-constraints
• S and dF
• Contrast
• Boykov and Jolly 2001

17
Progressive selection demo
18
Fluctuation effect

• Label changes disconnected from the brush

Existing selection
Fluctuation effect
19
Fluctuation removal

• Reject non-local label changes

20
Fluctuation removal

• Existing selections protected

Before removal
After removal
21
Fluctuation removal demo
Without fluctuation removal
22
Fluctuation removal demo
With fluctuation removal
23
Conflicting scribbles

• Correcting errors / deselecting objects

24
Scribble competition

• Intelligent removal of conflicting scribbles

Background
Existing selection
Existing boundary
Foreground scribble
25
Scribble competition demo
Without scribble competition
With scribble competition
26
Optimizations
27
Progressive multi-level optimization
L
L
L 1
L 1
Progressive
Global Lombaert et al. 2005
28
Adaptive band upsampling
L 1
L
29
Sequential graph-cut

• Dynamic-tree algorithm Boykov and Kolmogorov
  2001
• Best empirical performance in vision and graphics
• Cannot be trivially parallelized

30
Multi-core graph-cut

• Parallelizing the dynamic-tree algorithm for grid
  graphs
• Alternatively partitioning the graph
• Searching augmenting paths in parallel, without
  locking
• Reusing the trees between iterations by adoption

95-98 flow
31
Performance

• 15-20 times faster than Lazy Snapping (Dual-core
  2.8GHz, 2G memory)

Lazy Snapping
Paint Selection
(20Mp image)
32
Demo
33
Performance

• Effectiveness of proposed optimizations

30 Mp image
40 Mp image

• Banded GC
• Banded GC Progressive selection
• Banded GC Progressive selection Dual-core
• Banded GC Progressive selection Dual-core
  Adaptive band upsampling

34
Performance

• Scalability sub-linear growth of response time

35
Performance

• Quick response on a 110Mp panorama

small objects (e.g. vehicle, building)
large region(e.g. sky)
36
Comparison with Quick Selection Tool

• More accurate during mouse dragging

37
Comparison with Quick Selection Tool

• Faster on very large images (gt 100Mp)

38
Demo instant image effects
39
Summary

• A fast and easy-to-use tool for image selection
• Painting-based UI
• Instant accurate feedback
• Scalable to very large images
• Handling conflicting scribbles
• Interchangeable with other selection tools
• Instant image effects