Television in Words TIWO Round Table Project Launch Meeting EPSRC GRR6719401

1
Television in Words TIWO Round TableProject
Launch Meeting EPSRC GR/R67194/01

• University of Surrey
• 17th December 2001

2
Vision and Language the challenge for computer
science

• Intelligent Multimedia Information Retrieval
• Want intuitive access to multimedia archives
  especially with advent of TV / Internet crossover
• Machine-level view of image/video data is
  low-level ? need to represent high-level
  information about content
• May use collateral text for image/video
  annotation, or use a combination of media for
  communication ?
• ?How to integrate unstructured, non-symbolic
  image and video data with structured, symbolic
  text data in information systems?

3
Vision and Language an interdisciplinary
approach

• Researchers in artificial intelligence borrow
  theoretical constructs and methods from
  philosophy, aesthetics, semiotics, psychology,
  and linguistics to help in the development of
  systems, data structures and algorithms for
  intelligent systems
• The correspondence problem
• How is collateral information related in an
  intelligent system? And, how to convert from
  one form to another? (e.g. via knowledge
  representation formalisms or via neural networks)

4
Vision and Language Work in Progress at Surrey

• Prof. Khurshid Ahmad
• Dr Andrew Salway, Dr Andrew Hippisley, Dr
  Lilian Tang
• RAs Lee Gillam, Bogdan Vrusias
• PhD Students Chris Handy, Mariam Tariq, David
  Boulton, Yan Xu, Elia Tomadaki

5
Research into Images and Texts Work in Progress
at Surrey

• Digital Dance Library
• Scene-of-crime Information System
• Virtual Art Gallery
• Automatically Annotating Medical Images
• Royal Society Archives
• Television in Words

6
Research into Images and Texts Work in Progress
at Surrey

• Characterise a special language used to describe
  and interpret visual information corpus
  linguistics
• Extract information from collateral text in order
  to index image/video data natural language
  processing and object-relational databases
• Model relationship between image-text to provide
  intuitive access to digital libraries containing
  a mix of verbal and visual information data
  modelling and hypermedia

7
Digital Dance Library
8
SOCIS Scene-of-crime Information System

• I can see what appears to be a male laying in the
  prone position on the floor.
• He is wearing a maroon striped shirt with white
  collar and cuffs, blue jeans, and has a pair of
  left and right training shoes which have become
  slightly dis-extended from the foot.
• There appears to be a green tie down by his right
  hand and I can see a possible footwear impression
  in blood on his right hand.
• Surrounding the body there are droplets of blood,
  footwear impression in blood and several pieces
  of broken glass and bottles.

9
Image and Text at TATE Britain
at the extreme right, appears a scandalously
hieratic-looking couple a young mother is
strolling with her little girl dressed in white
with a salmon-colored sash
10
TIWO Aims

• Continuing research into the link between vision
  and language for intelligent multimedia
  information systems
• Develop a computational framework for
  investigating narrative in multimedia systems
• Adapt and apply existing video data models and
  knowledge representation schemes to the task of
  video annotation with audio description scripts
• Adapt and apply existing language processing
  techniques to process audio description scripts

11
TIWO approach

• Gather an audio description corpus
• Analyse language used for audio description
• Evaluate information extraction technology for
  processing audio description scripts
• Evaluate data models and knowledge representation
  formalisms for video data
• Specify, design and prototype AuDesc a system
  to prepare and manage audio description scripts
• ?? Style checking
• ?? Customisation of scripts
• ?? Process scripts into video annotations for
  retrieval

12
TIWO objectives

• A digital library comprising video material of
  diverse programme types, audio description, and
  machine-executable representations of video
  content.
• A specification and prototype of a system
  (AuDesc) to assist in the preparation and
  management of audio descriptions
• Two papers at international conferences (e.g. ACM
  Multimedia and AAAI in 2003/4) and two journal
  submissions (e.g. AI, AI Review, IEEE Multimedia,
  Multimedia Tools and Applications)
• A PhD dissertation
• A major EPSRC research proposal to continue
  research in this area

13
TIWO workplan

• WP1 Adapt and Apply Video Data Modelling and
  Knowledge Representation 6 p.m. (months 1-10)
• WP2 Adapt and Apply Language Engineering
  Techniques 9 person months (months 4-24)
• WP3 Audio Description System 12 p.m. (months
  7-30)
• WP4 Audio Description Round Table 1 p.m.
  (months 1-36)
• WP5 Write up of PhD Dissertation 8 p.m.
  (months 11-12 31-36)
• 1 full-time PhD student Yan Xu
• 1 associated PhD student Elia Tomadaki
• 5 hours per week from AJS (Principal
  Investigator)
• Official Start Date 2 January 2002

14
The role of the TIWO Round Table

• RNIB
• Denise Evans (Head of Broadcasting and Talking
  Images)
• BBC
• Jane Westrop (Audio Description Manager)
• ITFC
• Garry Duguid (Director of Engineering and
  Technology)
• Softel
• Russ Wood (Research Director)

15
The role of the TIWO Round Table

• Data and Expert Knowledge
• Domain understanding and modelling
• Providing sample audio description scripts
• Eliciting audio description in controlled
  scenarios

16
The role of the TIWO Round Table

• Current Issues and Technology
• What current problems should we be addressing?
  User requirements.
• What is the state-of-the-art in audio description
  technology?

17
The role of the TIWO Round Table

• User-feedback
• We would be grateful for continuous feedback on
  the prototype AuDesc system
• demonstrations at Round Table meetings and
  on-site visits
• latest information on TIWO WWW-site

18
The role of the TIWO Round Table

• Technology Transfer and Project Dissemination
• The ideas / technology developed and evaluated in
  the project are made available to the Round Table
  first
• Please tell your colleagues about the project!

19
Television in Words TIWO Round TableProject
Launch Meeting EPSRC GR/R67194/01

• University of Surrey
• 17th December 2001