Contentbased Video Retrieval

1
Content-based Video Retrieval

• Willem Jonker
• Philips Research / Twente University
• IFIP 2.6 meeting, Lausanne, May 15 17, 2002

2
Contents

• Image Retrieval Current Practice
• Content Based Video Retrieval
• Retrieving events from Tennis Video
• Retrieving events from Formula-1 Video

3
Video Content Management
4
Image Retrieval Current Practice
5
Image Retrieval Current Practice
6
Image Retrieval Current Practice
7
Precision Recall
8
Multi-level Modeling
9
Features
background
composition
color
texture
face recognition
motion
text rec. OCR
shape
closed caption
berg-etappe, etappe nr 17
speach rec.
10
Query _at_ feature level
11
QBIC
12
Amore
13
Video Retrieval
14
Video Retrieval Modeling
15
Video Retrieval Modeling
16
Video Retrieval Modeling
17
Video Retrieval Modeling
18
Video Retrieval Query
19
Video Retrieval Query
20
Video Retrieval Query
21
Video Retrieval Query
22
The Semantic Gap
User
Concepts
Semantic gap
Raw Video Data
Data
23
Tennis Goal
Video
Shot segmentation and classification
Shots (1) Playing (2) Close up
(3) Audience and other
24
Tennis HMM History

• 50s - statisticians tried to characterize
  processes with incomplete observations.
• 60s - Baum et al. published HMM theory in a
  series of papers.
• 70s - HMMs were used at IBM and CMU for speech
  recognition.
• 90s - widespread applications (Gesture
  recognition, human actions)

25
Tennis HMM example
26
Tennis HMM example
27
Tennis Pre-processing
initial segmentation (robust M-estimator)
Step 1 Step 2 Step 3
locate player
fit 3D model
final segmentation
28
Tennis Features
Shape Pie features Skeleton features
29
Tennis HMM
Feature extraction
Time
Time
30
Tennis HMM
. .
time
states
31
Tennis Experiments

• Goals
• Determine the right feature set
• Investigate person independence
• Two series of experiments
• Same player in the training and evaluation
• Training with one player, evaluation of strokes
  performed by various players
• Six events Forehand, Backhand, Service, Smash,
  Forehand volley, and Backhand volley

32
Tennis Experiment 1 results
Recognition results ()

• HMM parameters
• Codebook size of 24 symbols
• HMMs with 8 states
• Comparison to related work
• Improvement of 15
• TV videos

33
Tennis System Screen Dump
34
Tennis Some Performance Figures
Dual Pentium II, 350 MHz Preprocessing 6.5 min.
per stroke of 45 frames initial segmentation
95sec fitting 3D models 150 sec final
segmentation 105 sec feature extraction 40
sec Training the model 20 sequences of 45
frames 50 iterations Baum-Welch 4 states 8
symbol codebook 12 sec 48 states 80 symbols 20
min. Inference between 1 and 30 sec depending
on model complexity in our case for 8 states
model with 20 symbols 2 sec
35
Formula-1 Goal
36
Formula-1 Bayesian Networks
A
B
C
D
E
P(X,Y) P(XY) P(Y) P(XY)P(Y)
P(YX)P(X) P(X1,X2,,Xn) P i1..n P(XiPa(Xi))
37
Formula-1 Audio BN
Kword Key Words PRate Pause Rate AV Average
Values DR Dynamic Range MV Maximum Values STE
Short Time Energy MFCC Mel-Frequency Cepstral
Coefficients
38
Formula-1 Audio DBNs
39
Formula-1 Effect of Audio DBNs

40
Formula-1 Example Videos
41
Formula-1 Highlights on Audio
42
Formula-1 Combined Audio-Video
43
Formula-1 Experimental Results
44
Formula-1 System Screen Dump
45
Formula-1 Some Performance Figures
46
Bridging the Gap
User
Concepts
Domain knowledge
Domain features
Features
Raw Video Data
Data
47
DMW projecthttp//wwwhome.cs.utwente.nl/dmw/
48
Some Papers

• M. Petkovic, W. Jonker, Content-Based Video
  Retrieval by Integrating Spatio-Temporal and
  Stochastic Recognition of Events, In Proc. IEEE
  International Workshop on Detection and
  Recognition of Events in Video, Vancouver,
  Canada, July 2001.
• M.Petkovic, W. Jonker, Z. Zivkovic, "Recognizing
  Strokes in Tennis Videos Using Hidden Markov
  Models", International Conference on
  Visualization, Imaging and Image Processing,
  Marbella, Spain, September 3-5, 2001.
• H. E. Blok, M. Windhouwer, R. Zwol, M. Petkovic,
  P. M. G. Apers, W. Jonker, M. Kersten, "Flexible
  and Scalable Digital Library Search", 27th
  International Conference on Very Large Databases,
  Roma, Italy, September 11-14, 2001.
• V. Mihajlovic, M. Petkovic, W. Jonker,
  Content-Based Retrieval of TV Formula 1
  Programs, 2nd International Workshop on
  Multimedia Data Document Engineering (MDDE'02),
  March 24 2002, Prague