Kim, Seitz, Agrawala

1
Video-Based Document TrackingUnifying Your
Physical and Electronic Desktops

• Jiwon Kim Steven M. Seitz Maneesh Agrawala
• University of Washington Microsoft Research

2
Motivation
3
Unifying physical andelectronic desktops
Video camera

• Record video of paper on physical desktop

Desktop
4
Unifying physical andelectronic desktops
Video camera

• Record video of paper on physical desktop
• Tracking

Desktop
5
Unifying physical andelectronic desktops
Video camera

• Record video of paper on physical desktop
• Tracking
• Recognition

Desktop
6
Unifying physical andelectronic desktops
Video camera

• Record video of paper on physical desktop
• Tracking
• Recognition
• Linking

Desktop
7
Applications
Video camera

Desktop
8
Applications
Video camera

• Find lost document

Desktop
9
Applications
Video camera

• Find lost document
• Browse remote desk

Desktop
10
Applications
Video camera

• Find lost document
• Browse remote desk
• Find electronic version

Desktop
11
Applications
Video camera

• Find lost document
• Browse remote desk
• Find electronic version
• History-based queries

Desktop
12
Example Input Video
13
Demo Remote Desktop
14
Related Work

• Interactive desktops

DigitalDesk Wellner 93
15
Related Work

• Interactive desktops

Self-Organizing Desk Rus et al. 97
DigitalDesk Wellner 93
16
Related Work

• Interactive desktops
• Augmented paper

PADD Guimbretière 03
17
Related Work

• Interactive desktops
• Augmented paper

CyberCode Rekimoto et al. 00
PADD Guimbretière 03
18
Related Work

• Interactive desktops
• Alternative media
• Object tracking recognition

SIFT Lowe 04
19
System Overview
Video camera
Computer
User
Desk
20
System Overview
Video of desk
21
System Overview
Images from PDF
Video of desk
22
System Overview
Images from PDF
Video of desk
Track recognize
23
System Overview
Internal representation
Images from PDF
Video of desk
Track recognize
T
T1
24
System Overview
Internal representation
Images from PDF
Video of desk
Track recognize
T
T1
Scene Graph
25
System Overview
Where is my W-2?
Internal representation
Images from PDF
Video of desk
Track recognize
T
T1
26
System Overview
Where is my W-2?
Answer
Internal representation
Images from PDF
Video of desk
Track recognize
Desk
Desk
T
T1
27
System Overview
Where is my W-2?
Internal representation
Images from PDF
Video of desk
Track recognize
T
T1
28
Tracking Recognition


29
Tracking Recognition
Event
30
Event Types
before
after
Move
31
Event Types
before
after
Move
Entry
32
Event Types
before
after
Move
Entry
Exit
33
Tracking Recognition
Event

Desk
34
Tracking Recognition
Event


Desk
Desk
35
Tracking Recognition
Event
sanders01.pdf
lowe04sift.pdf
tut-article.pdf


objectspaces.pdf
kidd94.pdf
Desk
Desk
36
Assumptions

• Document
• Corresponding electronic copy exists
• No duplicates of same document

37
Assumptions

• Document
• Corresponding electronic copy exists
• No duplicates of same document
• Motion
• 3 event types move/entry/exit
• One document at a time
• Only topmost document can move

38
Non-Assumptions

• Desk need not be initially empty

39
Non-Assumptions

• Desk need not be initially empty
• Stacks may overlap

40
Algorithm Overview
Input Frames


41
Algorithm Overview
Input Frames


Event Detection
before
after
42
Algorithm Overview
Input Frames


Event Detection
before
after
Event Interpretation
A document moved from (x1,y1) to (x2,y2)
43
Algorithm Overview
Input Frames


Event Detection
before
after
Event Interpretation
A document moved from (x1,y1) to (x2,y2)
File1.pdf
Document Recognition
File2.pdf
File3.pdf
44
Algorithm Overview
Input Frames


Event Detection
before
after
Event Interpretation
A document moved from (x1,y1) to (x2,y2)
File1.pdf
Document Recognition
File2.pdf
File3.pdf
Scene Graph Update
Desk
Desk
45
Algorithm Overview
Input Frames


Event Detection
before
after
Event Interpretation
A document moved from (x1,y1) to (x2,y2)
File1.pdf
Document Recognition
File2.pdf
File3.pdf
Scene Graph Update
Desk
Desk
46
Event Detection
47
Event Detection


Frame differences
time
48
Event Detection


Frame differences
time
49
Event Detection


Frame differences
time
50
Event Detection


Frame differences
time
51
Event Detection
Frame differences
time
52
Event Detection
Motion Frames
Event Frames
Threshold
Image motion
time
time
53
Event Detection
Motion Frames
before
after
54
Algorithm Overview
Input Frames


Event Detection
before
after
Event Interpretation
A document moved from (x1,y1) to (x2,y2)
File1.pdf
Document Recognition
File2.pdf
File3.pdf
Scene Graph Update
Desk
Desk
55
Event Interpretation
before
after
Move
Entry
Exit
56
Event Interpretation
before
after
Move
1. Move vs. Entry/Exit
Entry
Exit
57
Event Interpretation
before
after
Move
Entry
2. Entry vs. Exit
Exit
58
Event Interpretation
before
after
Move
1. Move vs. Entry/Exit
Entry
Exit
59
Move vs. Entry/Exit
before
after
60
Move vs. Entry/Exit
before
after
61
Move vs. Entry/Exit
before
after
62
Move vs. Entry/Exit
before
after
63
Event Interpretation

• Use SIFT Lowe 04
• Rotation- and scale-invariant
• Highly distinctive (128-bit vector)

64
Move vs. Entry/Exit
before
after
65
Move vs. Entry/Exit
before
after
66
Move vs. Entry/Exit
before
after
67
Move vs. Entry/Exit
before
after
68
Move vs. Entry/Exit
before
after
69
Move vs. Entry/Exit
before
after
70
Move vs. Entry/Exit
Motion (x,y,?)
before
after
71
Algorithm Overview
Input Frames


Event Detection
before
after
Event Interpretation
A document moved from (x1,y1) to (x2,y2)
File1.pdf
Document Recognition
File2.pdf
File3.pdf
Scene Graph Update
Desk
Desk
72
Document Recognition

• Match against PDF image database

File2.pdf
File3.pdf
File4.pdf
File5.pdf
File6.pdf
File1.pdf
73
Document Recognition

• Performance analysis
• Tested 20 pages against database of 162 pages

74
Document Recognition

• Performance analysis
• Tested 20 pages against database of 162 pages
• 200x300 pixels per document for reliable match

Recognition Rate
Document Resolution
75
Document Recognition

• Performance analysis
• Tested 20 pages against database of 162 pages
• 200x300 pixels per document for reliable match

0.9
Recognition Rate
300
Document Resolution
76
Algorithm Overview
Input Frames


Event Detection
before
after
Event Interpretation
A document moved from (x1,y1) to (x2,y2)
File1.pdf
Document Recognition
File2.pdf
File3.pdf
Scene Graph Update
Desk
Desk
77
Scene Graph Update
Motion (x,y,?)
after
before
Desk
78
Scene Graph Update
Motion (x,y,?)
after
before
Desk
79
Scene Graph Update
Motion (x,y,?)
after
before
Desk
Desk
80
Results

• Input video
• 40 minutes
• 1024x768 _at_ 15 fps
• 22 documents, 49 events
• Running time
• Video processed offline
• No optimization
• A few hours for entire video

81
Demo Paper tracking
82
Photo Sorting Example
83
Photo Sorting Example
84
Demo Photo Sorting
85
Future Work

• Enhance realism
• More applications

86
Future Work

• Enhance realism
• Handle more realistic desktops

87
Moving a stack of documents
88
Documents with no electronic versions
89
Future Work

• Enhance realism
• Handle more realistic desktops
• Real-time performance

90
Future Work

• More applications
• Support other document tasks
• E.g., attach reminder, cluster documents

91
Future Work

• More applications
• Support other document tasks
• E.g., attach reminder, cluster documents
• Beyond documents

92
Future Work

• More applications
• Support other document tasks
• E.g., attach reminder, cluster documents
• Beyond documents

93
Future Work

• More applications
• Support other document tasks
• E.g., attach reminder, cluster documents
• Beyond documents

94
Future Work

• More applications
• Support other document tasks
• E.g., attach reminder, cluster documents
• Beyond documents

95
Acknowledgments

• NSF
• Intel Corp.
• Li Zhang