Evolving applications and Interfaces

1
Multimedia Databases
2
Introduction to multimedia databases

• What is an multimedia database?
• A multimedia database can also be known as a
  multimedia database management system (MMDBMS)
• A multimedia database/MMDBMS is a framework that
  manages different types of data potentially
  represented in wide diversity of formats on a
  wide array of media sources
• Database hosts 1 or more primary media file types
  .txt, .jpg, .swf, .mp3

3

• Databases-provide functionalities for easy
  manipulation, query retrieval of relevant info
  from huge collections of stored data
• MMDBs cope with increased usage MM data used in
  various software app
• Provide almost all functionality of traditional
  DB new and enhanced functionality
• Provide unified framework for storing,
  retrieving, transmitting presenting variety of
  media types in a variety of formats (numerical
  constraints)

4

• MMDB difficult and complex to develop because
  they are different to traditional DB
• Data types (large object (LOB)), BFILE (large)
• Manipulation
• Storage and delivery
• Nature and size of MM data and high capacity req
  for delivery-cause problems
• Early app MMDBMS multimedia for presentational
  requirements
• Employee DB include image of Employee
• Sales order processing sys-online catalogue

5

• These sys were imp simply
• storage image files externally to DB
• Storing file reference in DB
• Image retrieved-application process-referenced it
  thru a traditional DB record
• This external data could NOT be manipulated by
  DBMS
• MM applications evolving-people want to exploit
  MM data
• Interrogate, retrieve, manipulate
• Surveillance-search and manipulate pic, sound,
  video to retrieve data needed

6
Features of a MMDBMS

• Ability to uniformly query data (media data,
  textual data) represented in different formats
• Ability to simultaneously query different media
  sources and conduct classical database operations
  (create, read, update and delete etc) across them
• Ability to retrieve media objects from a local
  storage device in a continuous manner
• Ability to take the answer generated by a query
  and develop a presentation of that answer in
  terms of audio-visual media
• Ability to deliver this presentation in a way
  that satisfies various user requirements

7
Review of media types

• Text/Document
• Image
• Video
• Audio
• Classical Data (e.g. relations, flat files,
  object bases etc)

8

• 3 main challenges that arise from MM data that do
  not occur within other data types
• Size-data size affect storage, retrieval,
  transmission MM
• Techniques reduce size crucial
• Time
• Frames in video must run in corr seq at
  acceptable rate
• Real-time nature of MM video-images and sound
  synchronized

9

• Semantic nature of MM
• Add description in wrds to content of image
• Pic means different things to diff people
• Context is important for interpreting img
• In order to manage semantic nature of media
• Interpretations made based on certain features of
  multimedia data stored as meta data
• Meta data-data req to interpret other data as
  meaningful info
• Important-used to retrieve and manipulate MM data

10
Meta data

• Deals with content, structure and semantics of MM
  data
• Manual methods laborious
• Might be necessary to use automatic methods
• Difficult
• Methods used often result in metadata that
  contains too little info
• Trad view meta data-structure DB-tables...
• MMDBMS also used to describe individual
  occurrences (linguistic annotation added to
  attributes)

11
Meta data

• Obj is to allow media to be queried manipulated
  by querying meta data and retrieving actual
  result set
• Same terms must be used to represent same
  occurrences else search criteria will bring back
  incorrect results

12
Review of media types

• Video and audio differ from the other media types
  listed above because of their temporal nature
• Ability to take the answer generated by a query
  and develop a presentation of that answer in
  terms of audio-visual media
• Ability to deliver this presentation in a way
  that satisfies various user requirements
• Video/audio retrievals must appear to be
  continuous, hiccup free presentations
• Video/audio support operations like fast-forward,
  rewind and pause, that were not supported by
  classical data types
• Let us briefly consider how this data could be
  used in a business multimedia scenario

13
Sample multimedia scenario

• Consider current terrorism investigation by the
  USA/UK security bodies
• Investigation may generate the following types of
  data sources
• Video data captured by surveillance cameras that
  record activities at various locations
• Audio data captured by legally authorized
  telephone wiretaps

14
Sample multimedia scenario

• Image data consisting of still photos taken by
  investigators
• Document data seized by the police during raids
  on one or more places
• Structured relational data containing background
  information, bank records etc of the suspects
• Geographical information systems data
• What could we do with this data?
• Answer - raise queries

15
How will users query MM data?

• Relational DB QBE
• Relational template that could be completed with
  sample elements to give examples of the tuples
  (rows) of info to be retrieved
• MM Query
• Often text description of image or audio file
  provided
• Simple QBE must be extended-elements in query
  contain image, video clips ...
• Alternatively provide image/sketch
• Sketch sunflower and provide paintings like it

16
Querying MM data

• Interrogation of MM data through various modes

Query mode
Search mode
Linguistic
Linguistic
Visual
Visual
Evidence suggests users need images to illustrate
text
17
Modes

• Linguisticstandard query languages searching
  meta data which has been stored in the form of
  text in order to locate and retrieve MM data
• Visual mode retrieval by content, e.g. Query
  posted in form of a sketch which user draws, .
  Browser based small selection of images from db
  to use QBE selection. Or use visual thesaurus
  (plant leaf)
• User manipulates basic leaf shape (no. Diff
  attributes) build up visual map with specimen
  leaf. Matching images returned in rank order of
  the attributes
• Facial recognition

18
Modes

• Visual-Linguistic
• Provide example images retrieved by linguistic
  meta data
• Images specified by pixels (sets) with values for
  colour, shape, geometric relations but indexed by
  name, title etc.
• Image inc. In QBE style query-limitations to info
  that can be expressed by user
• Linguistic visual Approach
• Images indexed by visual attributes. User
  expresses query in linguistic form using standard
  query language

19
Query By Image Content

• http//wwwqbic.almaden.ibm.com/
• Images and videos processed to extract features
  (calc numerical values of several image
  descriptors)
• Average colour and colour histogram
• Texture contrast, coarseness, directionality
• Colour layout-positions of colours
• Complex shapes e.g. Draw image QBIC ranks
  numerical values of images in DB to indicate
  their similarity to query image
• Example http//www.hermitagemuseum.org/

20
Usage Scenarios for MMDBMS

• Entertainment Systems
• Request video from catalogue
• User can select video based on textual
  information e.g. Cast etc...
• Users can view video or play randomly selected
  scenes
• Play and pause
• Public protection
• Police use visual information to identify
  people/record scenes of crime for evidence
• UK-everyone arrested photographed, images stored
  with fingerprints, DNA profiles
• Until subject convicted, photographic info
  restricted
• Interrogation of DB may be thr automatic
  fingerprint recognition, DNA matching and face
  recognition
• Video surveillance matched to facial recognition

21
Usage Scenarios for MMDBMS

• Medical Information Systems
• Store visual information such as x-ray, ultra
  sound etc...
• Rules
• Images kept with patient data (unique ID number
  e.g.. NI)
• Image processing such as edge detection and
  feature extraction can be important in diagnosing
  conditions such as tumors tracking growth
• Images may be result of single approach e.g.
  X-ray or result of combination of data (diff
  sources)

22
Example image queries

• I have a photograph/still image e.g.
• I want to know the identity of the person in the
  picture
• The image has a name attribute attached to it
• Query 1 retrieve all images from the image
  library (database) in which the person appearing
  in the currently displayed photograph appears

23
Example image query

• I want to examine pictures of Chris Mayer
• Query 2 retrieve all images from the image
  library in which Chris Mayer appears
• This could be done by either some sort of key
  match or using an image match

24
Issues raised

• If follows that there are two basic kind of
  queries for images
• Image based queries
• Keyword based queries
• In the first query we gave an image as input
  (query image)
• We expect output as a ranked list of images that
  are similar to the query image
• What does similar mean? How confident can we be
  with the result? What action rests on the result?

25
Issues raised

• To support this we need to know what similarity
  means
• We need to know what ranking means
• A multimedia database driven system needs to be
  able to efficiently support these operations

26
Issues raised

• In the 2nd query we gave a keyword as input (name
  of suspect Chris Mayer)
• We want as output those photographs that are
  known to contain an image object whose name
  attribute is Chris Mayer
• To support this we need to know how to associate
  different attributes with images (or parts of
  images)
• We need to index and retrieve images based on
  such attributes

27
Example Audio (sound) query

• An investigation officer is listening to an audio
  surveillance tape
• The tape contains a conversation between
  individual A under surveillance and another
  individual B meeting A
• Query1 Find the identity of individual B given
  that individual A is Chris Mayer

28
Example Audio (sound) query

• Officer wants to review all audio logs that Chris
  Mayer participated in during some specified
  period of time
• Query2 Find all audio tapes in which Chris Mayer
  was a participant

29
Example Text query

• Investigating officer is browsing an archive of
  text documents - newspaper archives, police
  department files on old terrorist cases, witness
  statements etc
• Query Find all documents that deal with the
  Mayer Gangs financial transactions with
  Britannia Building Society

30
Example Video query

• Officer is examining a surveillance video of a
  particular person being assaulted by an hooligan.
  However, the hooligans face is obscured and
  image processing algorithms return very poor
  matches.
• The officer thinks the assault was by someone
  known to the victim
• Query Find all video segments in which the
  victim of the assault appears
• By examining the answer we hope to find other
  people who have previously interacted with the
  victim

31
Simple Textual example

• Query Find all individuals who have been
  convicted of terrorism in the UK and who have had
  electronic fund transfers made into their bank
  accounts from Britannia Building Society
• The answer is problematic
• Determining all people convicted of different
  crimes may require accessing a wide variety of
  databases belonging to different police
  jurisdictions etc
• Britannia may have accounts in hundreds of banks
  worldwide each of which uses different formats
  and different database systems

32
Heterogeneous query

• All queries discussed so far involve one media
  type i.e. image, audio, video or text
• Each query accesses only image or audio or video
  data but does not access a mix of these media
  types
• Complex queries will mix and match data from
  these different media sources
• Mix and match is difficult!

33
Heterogeneous multimedia query

• Query Find all individuals who have been
  photographed with Chris Mayer and who have been
  convicted of security offences in the UK and who
  have recently had electronic fund transfers made
  into their bank accounts from Britannia
• This query requires
• We find all people satisfying the conditions of
  the simple query before

34
Heterogeneous multimedia query

• We access a mug shot database containing names
  and pictures of various individuals
• We access surveillance photograph database of
  still images
• We access a surveillance video database to see if
  a meeting between the suspect and other people
  recorded on the video
• Access image processing algorithms to determine
  who occurs in which video/still

35
Requirements Issues - Queries

• We need a single language within which multimedia
  data of different types can be accessed
• Language must be able to specify combination
  operations across different media types/merge and
  manipulate
• Language must be able to access
• Meta data describing the content of media sources
• Raw data supported by the different media sources

36
Requirements Issues - Queries

• As well as the language we need techniques to
• Optimise queries by planning
• Develop servers that can optimize processing of a
  set of queries

37
Requirements Issues - Content

• What is content of media source? Under what
  conditions can content be described textually and
  under what conditions must it be described
  directly through the original media type?
• How should we extract the content of
• an image?
• an video clip?
• an audio clip?
• a free/structured text document?

38
Requirements Issues - Content

• How should we index the results of this extracted
  content?
• What is retrieval by similarity?
• What algorithms can be used to efficiently
  retrieve media data on the basis of similarity?

39
Requirements Issues - Storage

• How do these storage devices work?
• Disk systems
• CD-ROM and DVD
• Tape systems and tape libraries
• How is data laid out on such devices?
• How to design servers using the above devices
  when they use playbackrewindfast fwd and pause

40
Requirements Issues -Presentations and Delivery

• How do we specify the content of multimedia
  presentations?
• How do we specify the form (layout) of this
  content?
• How to deliver a presentation to users when there
  is the need to
• How to interact with remote (distributed) servers
  and convergence/compatibility issues
• What are the bandwidth issues

41
Questions for you

• Where do you think potential multimedia database
  applications exist?
• Think of local examples and national examples
• Recent examples in the news?
• Multimedia database driven systems
• Multimedia intelligent querying database systems

42
Answers

• Local examples of potential examples
• Football clubs - SCFC/PVFC
• Theatres - New Vic / Regent
• University - SU
• Museums - City
• Newspaper Sentinel
• National examples include
• Airports
• Police databases

43
Directed Reading

• IEEE paper on MMDBMS
• Multimedia Databases Lynne Dunckley
• Chapter 1, 2, 5
• There is one short term loan and one 24 hour loan
  copy
• Computing and Computing Weekly in Thompson
  library
• See if there is any news in this area

44
Exercise 1

• Groups 2-3 Research into 2-3 practical examples
  of the use of multimedia databases
• Write a short report describing the uses of
  multimedia databases in an industry of your
  choice (giving appropriate referencing)
• Discuss your reports with the peers in your group