AMI Project Overview

1
AMI Project Overview

• Steve Renals
• University of Edinburgh
• s.renals_at_ed.ac.uk

2
AMI in a slide

• 15 partners, running from Jan 2004 - Dec 2006,
  second phase (AMIDA project, Sep 2006 - Aug 2009)
• AMI research aims to improve communication in
  business meetings with co-located and remote
  participants
• Not just research training programme, technology
  transfer
• Progress multimodal recognition, content
  abstraction, annotated corpus, demonstrations

3
AMI partners
4
Multimodal processing and meetings
Speech
Facial Expression
Gestures
Attention
Documents
5
A Typical project meeting...

• A few person-weeks of meeting
• 7,000 travel budget
• 24 days away from family and friends
• 24 days backlog of work
• Evenings spent in transit at Schiphol

6
...deserves a better record than this
7
Typical AMI questions

• Next week
• What happened at the work package meeting?
• Next month
• What exactly did they say about the new
  segmentation algorithm?
• Next year
• What was the precise criticism of the error
  measure?
• Today
• How could the CSIRO people have participated?

8
AMI Vision

• Technology supporting human interaction in
  meetings
• richer recording of our time together
• recognize content and interarction events and
  patterns
• enable people to meet and work better more
  satisfaction, efficiency effectiveness
• Application scenarios
• Meeting browser
• Remote meeting assistant
• Multimodal recordings from instrumented meeting
  rooms audio, video, handwriting, slides, ...

9
AMI research

• Browsing meetings (online and archived) requires
• Models of group dynamics
• Audio and video processing and recognition
• Models to combine modalities
• Content extraction
• (As well as meeting user requirements and various
  software technologies)
• And lots of data... well annotated

10
Schematic data flow
11
Instrumented meeting rooms

• Three standardized AMI meeting rooms at IDIAP,
  TNO, Edinburgh
• Standardized collection of 4 person meetings
  using
• 4 close-, 2 wide-view cameras
• 4 headset, 8 array microphones
• data projector capture
• whiteboard capture
• digital pen capture
• extra site-dependent devices (eg second
  microphone array, lapel mics)

12
(No Transcript)
13
AMI data collection

• 100 hour corpus of 4-participant meetings
• Annotated at many different levels (speech,
  dialogue acts, focus of attention, gesture, ...)
• About 70 scenario meetings, 30 real meetings
• Number of specific spoke corpora (eg localization
  and tracking)
• Public release of data and annotations (see
  poster tomorrow... and get the taster DVD)

14
Scenario meetings

• About 70 of the captured meetings are based on a
  scenario design team (role play)
• Series of four meetings
• Real work between meetings
• Advantages of scenario meetings
• Control domain
• easier to understand whats going on
• makes it possible to construct ontologies
• Control knowledge and motivation of participants
• Build in outcome measures

15
Annotation

• Annotation phenomena defined - cater for all the
  key research problems on hub corpus
• Annotations include speech transcription,
  dialogue acts, focus of attention,
  summarization, individual actions
• NITE XML format and toolkit to standardize
  annotations
• See the NITE XML Toolkit in the demo session

16
Signal labelling
17
Dialog act labelling
18
Summarization
19
Meeting browser
20
Audio-video processing

• Defined according to core problems
• What are they saying? (speech recognition)
• How are they saying it? (prosody)
• What are they doing? (action recognition)
• Where are they going? (location tracking)
• How are they feeling? (emotional state)
• Where are they looking? (focus of attention)
• Who are they anyway? (person identification)
• Challenges
• Scalable models and real-time algorithms
• Adaptation to new domains without extensive data
  collection

21
Structure content extraction

• Defined according to application requirements
• Segmentation of multimodal streams
• Structuring by meeting events
• Identification of group activity
• Linguistic and discourse events
• Indexing and retrieval
• Summarization, and generation of textual and
  multimodal summaries

22
AMIDA - remote meetings

• Technology to create archives
• Efficient access (online and offline) to
  multimodal meeting recordings
• Technology to create presence
• Presence as realtime communication of state
• Shared multimodal workspaces
• Technology to create context
• Automatic incorporation of multiple information
  sources during a meeting

23
Presence as sharing state

• Individual state
• who is speaking, for how long
• focus of attention, eye contact
• keyword and topic monitoring
• dominance
• emotion
• Group state
• Discussion/presentation/monologue
• Questions and answers
• Agenda progress, decisions
• Remote meeting assistant

24
Current results from AMI

• Instrumented meeting room infrastructure
• Multimodal corpus of meeting recordings
• Meeting annotation schemes and tools
• Many component technologies speech recognition,
  audio-visual tracking, summarization, ....
• Media file server
• JFerret meeting browser
• Open source software releases NITE XML toolkit,
  TORCH machine learning toolkit

25
Thank you

• More details
• Talks and posters at MLMI
• Some demos in the next session
• http//www.amiproject.org/