Video databases

1
Video databases
2
Reference

• Principles of MM DB Systems,
• V Subrahmanian
• Chapter 7

3
Overview

• Video characteristics
• Frames and their contents
• Video libraries
• Querying libraries
• Indexing
• Frame segment trees
• RS-trees

4
Video characteristics

• A video is a sequence of image frames, often
  together with a synchronised soundtrack
• Typically there are 30 frames per second, and
  videos may last 90 or even 120 minutes.
• Video databases usually concentrate on the images
  for data that support queries.

5
Organizing content

• In deciding how to model the contents, you must
  decide what kind of queries will be supported
• Identify people/objects, properties, and
  activities
• Range of values determine type of queries you can
  support

6
Frames

• The video will be about certain people or objects
  that are involved in activities
• You can model the contents of each frame by
  identifying
• the people and objects in the frame
• the activities that occur in the frame
• the properties they have in the frame

7
Example

• A frame of the film Mary Poppins may contain
• Mary Poppins
• A bicycle, and an umbrella
• And have an activity
• Cycling in the air

8
Video libraries

• Hold several videos
• Contain
• video id
• associated relational data (e.g. name, date
  produced, )
• number of frames,
• links to content and location

9
Queries on video libraries

• Find all video segments that satisfy a specified
  condition
• Find all the people/objects that appear in a set
  of frames of a particular video
• Find all the activities that occur in a range of
  frames of a particular video
• Find all the video segments that have an object
  with a specified property.

10
Segments

• Frames are single images
• Often, several consecutive frames in a video have
  very similar contents
• A segment is a consecutive sequence of frames
  i,j) where i lt j that includes the start frame i
  and all frames up to j, but not including j.

11
Non-overlapping segments

• For each object and activity, we want to find all
  the frames it belongs to by recording all the
  segments it is in
• We look for a collection of segments that dont
  overlap, and hence form a well-ordered sequence.
• Note that a pair of segments i,j) and j,k)
  could be composed into one segment i,k).
• We reduce the data to be represented by
  identifying some larger consecutive frames rather
  than many smaller ones that can join together.

12
Example

• Suppose there are three objects O1,O2,O3.
• O1 appears in frames 0 to 100 and 150 to 250.
• O2 appears in frames 50 to 200
• O3 appears in frames 200 to 300.
• Wed like to find segments for this set of
  objects
• List the start and endpoints of the given
  segments and sort into increasing order

13
Segments for the objects

• 0,100), 150,250), 50,200), 200,300)
• The list of endpoints, in order, is
• 0,50,100,150,200,250,300.
• This gives segments
• 0,50), 50,100), 100,150), 150,200),
  200,250), 250,300).

14
Index structure

• Hold an array for the objects which contains each
  object identifier and a list of segments it
  occurs in.
• Hold a second array for the activities with an
  activity id and a list of segments for each.
• You can use linked lists of reference numbers for
  the segments.

15
Frame segment tree

• A binary tree which has all the leaf nodes at the
  same level and all branches used.
• Each leaf node holds a segment.
• Each non-leaf node holds the segment formed by
  joining its two children nodes.
• The leaf nodes are ordered in increasing order of
  segment x,y) from the left.
• Each node also holds a list of pointers to
  objects in the segment

16
Example

• The frame segment tree for the three objects in
  the last example would have
• Leaf nodes 0,50), 50,100), 100,150),
  150,200),200,250), 250,300), and two dummy
  nodes to fill out the tree 300,301).
• Above the first two nodes, is a node holding
  0,100). Above the next two is one holding
  100,200), above the next, one holding 200,300)
  and finally, one holding the dummy 300,301).
• One level above these are two nodes holding
  0,200) and 200,301).
• The top level holds 0,301).

17
Storing the object lists in the tree

• For each object, you check each of its segments
  to identify which nodes of the tree it is
  associated with.
• You place the object reference in the tree node,
  and link the object array entry to the node of
  the tree.

18
Example

• O1 has 0,100) as a segment.
• This matches one tree node exactly, so you put 1
  in the list at that tree node, and add a
  reference to the node in the object array.
• O1 also has 150,250) that needs two segments
  150,200) and 200,250) from the tree. Both nodes
  have object 1 added, and they are both added to
  the node list in the object array for O1.

19
Support for queries

• The frame segment tree gives an efficient
  retrieval of objects that occur in any given
  segment x,y).
• The object array gives an efficient access to the
  frame segments a particular object occurs in.
• Together these allow more complex conditions in
  the queries.

20
Alternative structures

• The frame segment tree is one way of holding the
  index structure
• An alternative is to use an RS-tree.
• This uses the same arrays for the object and
  activity lists, but uses an R-tree for the frame
  segments treating each segment as a rectangle
  of width 0.
• Advantage its better when using disks for the
  index structure as blocks can hold several
  related nodes.

21
Summary

• Videos are sequences of frames
• Video libraries hold collections of videos
• Look for objects, activities and their properties
  in each frame.
• Want to support queries on these.
• Use frame segments to reduce data stored.
• Frame segment trees or RS-trees can support
  efficient retrievals for queries.