Automatic Shot Boundary Detection in the Fschlr Digital Video System

1
Automatic Shot Boundary Detection in the Físchlár
Digital Video System

• Colin OToole
• Dublin City University

2
Contents

• Introduction
• Shots, scenes and video programmes
• Shot boundary detection - shot change types
• Our video test suite
• SBD based on Colour Histograms
• SBD based on Colour Moments
• SBD based on Edge Differences
• SBD based on the Encoded MPEG Bitstream
• The Físchlár Digital Video System

3
Introduction

• Developments in digital video are driven by
  engineering
• ? compression, eg MPEG
• With engineering problems solved, we now turn
  to content
• Content analysis of video programmes shows a
  hierarchy of
• program ? scenes ? shots ? frames
• shot continuous recording by a single camera
• scene set of multiple shots representing a
  logical unit in a program
• Isolating frames is easy, can be re-constituted
  as stills from lossy compression

4
Shot boundaries

• Shot boundary detection can be easy, or
  difficult, depending
• Cut hard boundary, complete change of shot
  between consecutive frames
• Fade fade-out or fade-in, a gradual fade
  to/from a completely black (white ?) frame
• Dissolve simultaneous fade-out and fade-in
• Wipe transition common in Batman and 60s TV
• others
• Each of these post-production techniques make the
  detection of shot boundaries more difficult.

5
Inter-frame differences

• SBD techniques basically compare one frame
  against the next and if there is a significant
  difference, thats a SB.
• Approaches include pixel differences, colour
  histograms, encoding macroblocks, edges, motion
  vectors, etc., and each works well in some
  situations
• Techniques based on colours work well on cuts but
  not fades while edge detection handles wipes and
  dissolves
• In addition, there is no common evaluation test
  suite and published techniques are tried on
  small, tailored test suites
• Work reported here is an evaluation of some
  techniques on a large test suite of broadcast TV
  video

6
Video Test Suite

• 12 June 1998, 8 hours of RTE 1 encoded as MPEG-1
• News, quiz, afternoon TV, cookery program, Dickie
  Rock (!), Touched by an Angel, gardening,
  commercials
• 720,000 frames, 5.4Gbytes
• Shot bounds manually marked up, 5380 cuts and 779
  fades/dissolves, acting as a ground truth
• Evaluation of an implemented SBD algorithm is in
  terms of precision ( boundaries identified that
  are correct) and recall ( total boundaries that
  are identified)

7
SBD based on Colour Histograms

• Compute an overall colour histogram for each
  entire frame
• We use 3 x 64 colour bins and compute
  similarities using cosine similarity measure.
• Our findings show there is no single best
  threshold value for all programme types, e.g.
  visually noisy MTV has high threshold compared to
  visually quiet soaps
• Best results are 88 precision and 85 recall

8
SBD based on Colour Moments

• As a probability distribution is uniquely
  characterised by its moments, we can represent a
  colour distribution by its first three moments.
• The 1st order moment ?i defines the average
  intensity.
• The 2nd order moment ?i defines the variance of
  the intensity.
• The 3rd order moment si defines the skewness of
  the intensity.
• Best results are 87 precision and 80 recall.
• Distance measure between frames I and Q is

9
SBD based on Edge Detection (1)

• Convert frames to gray-scale and use Sobel
  filtering to detect edges.
• Dilate each edge pixel by surrounding with a
  diamond of pixels, to allow for inter-frame
  movement
• Compare dilated edges in one frame with edges in
  adjacent and count the in and out pixels.
• Weve re-scaled frames from 352x288 to 176x144
  and 88x72 pixels and varied the dilation diamond
  to 3x3, 7x7, 13x13
• For adjacent frames we compute- Pout (fraction
  of exiting edge pixels for first frame) and -
  Pin (fraction of entering pixels for second
  frame)

10
SBD based on Edge Detection (2)

• Edge change fraction (ECF) value is maximum of
  Pin and Pout
• We compute the difference values between adjacent
  frames, as opposed to applying thresholds to the
  ECF value.
• Best results are 87 precision and 80 recall.

Max Edge Change fraction Difference values
11
SBD based on encoded MPEG bitstream

• Macroblock ratios
• Macroblock is a 16x16 block of pixels.
• MPEG frames and macroblocks may be of type I, B
  or P.
• Ratio of different types of macroblocks in
  different types of frames can indicate SBs.
• e.g. If a P-frame contains mainly I macroblocks
  -gt Couldnt predict from last frame -gt shot
  boundary present.
• Motion Vectors
• Calculate object/global motion.
• Used as aid to other methods (compensate for
  global motion) or as a stand-alone method

12
The Físchlár Digital Video System

• To demonstrate our research, we have developed a
  web-based record, browse and playback system
  called Físchlár.
• User can select a programme to record by
• Browsing through the TV listings for
  today/tomorrow and selecting a specific program.
• Selecting the exact date, channel, and start/stop
  times.
• Programme is recorded as MPEG-1 and automatically
  segmented by shot boundaries. Currently we use
  the histograms/moments methods but the system is
  modular.
• Content available to browse/watch via
  automatically extracted keyframes.

13
Update TVSchedule
TV Listings
Users PC
Sunsite
Select Programme to Record
PC
Update Record List
Record List
Web Server
Shot Boundary Detection
Recorded MPEG Files
TV Tuner MPEG Encoder
MPEG File Store
14
Users PC
Sunsite
List of Recorded Programmes Keyframe Selection
MPEG File Store
Video Streaming
Web Server
Real Server