Efficient user-centred access to multimedia meeting content

1

• Efficient user-centred access to
  multimedia meeting content

Simon Tucker and Steve Whittaker University of
Sheffield s.tucker, s.whittaker_at_shef.ac.uk
2
AMI Project

• Meetings are a critical way in which knowledge is
  created and shared within organisations
• Most of this knowledge is never recorded
• AMI provides Multimodal Access to Multimedia
  Records of Meetings
• 16 Partners
• Follow on project AMIDA Real Time

3
Sheffield AMI Work

• User Requirements
• Temporal Compression of Speech
• Reducing the amount of time required to listen to
  a meeting recording but still getting the
  important information.
• Dynamic Visual Summarization Techniques
• A number of methods for dynamically presenting
  summary information interactively.
• Temporal Compression of Video
• Audio motivated video compression.

4
Meeting browsers

• The primary means of accessing meeting records is
  via a browser.
• In previous work we segregated browsers into four
  categories according to their focus.
• The focus is either the primary means of
  presentation or navigation that the browser used.
• This segregation allowed us to get a good idea of
  the current browser space.

5
Browser Examples
Audio
Video
Artefact
Discourse
6
User Requirements

• Can make use of two different methods to collect
  user requirements
• Practicecentric
• Examination of current practices.
• Collection through observation.
• Technology-centric
• Exposure to new technology.
• Collection through user opinion.

7
Practice-centric AMI study

• Meetings already generate a large amount of
  information exchange.
• Personal Notes.
• Minutes.
• Post-meeting email discussion.
• Informal meeting discussions.
• Approach taken is to record (where possible) and
  then analyse these records.
• Use this analysis information to determine how
  meeting records are used and what are any
  problems associated with such records.

8
Study details

• We examined the meeting recording practices of
  two firms.
• We studied a core team over a series of meetings.
• Thus we can study the lifecycle of meeting
  documents.
• Meetings in both firms were task oriented rather
  than being about the generation of ideas.
• We collected permission to make recordings from
  each meeting participant
• We also allowed participants to request that the
  recordings be switched off.
• Names were removed from transcripts.

9
Existing Tools and Problems
Type of Record Functions Problems
Public Record (Minutes) Group Todos (actions/decisions) Summary/Gist Group Archive (history) Not timely Lacks context completeness Requires effort to produce
Private Record (Personal Notes) Personal Todos (actions/decisions) (context for actions) Briefing for non-attendees Personal Archive Esoteric Detracts from ability to contribute
10
Analysis of State of the Art Tools

• Important to assess the state of the art.
• Assessed the efficiency of the first generation
  AMI meeting browser in answering typical
  questions about a meeting.
• Generated a number of questions about a single
  meeting.
• Subjects asked to answer these questions using
  the meeting browser.
• Thinkaloud was encouraged and we examined the
  accuracy of the answers.
• The questions were either about specific
  information (what was the total budget?) or were
  more general (what was Eds contribution to the
  meeting?).

11
Tools Analysis Results

• Inefficient for access
• Too much low level detail
• Assumption of large display
• Users need abstraction / summarisation tools

12
Efficient Access to Meeting Data

• There is a clear need for efficient access to
  meeting data.
• Meetings contain a lot of irrelevant information
  (both in general and for specific participants).
• Minutes and notes capture important information
  but lack contextual information.
• State of the art tools lack abstraction
  generally present the raw recordings, unfiltered.
• We focus on lightweight components allowing for
  efficient access to meeting data.

13
Temporal Compression of Speech

• Intended for environments which necessitate
  speech only access.
• e.g. Mobile phone, travelling in car etc.
• Aim is to reduce the length of the recording but
  to retain the important content.
• Two techniques for reducing the length
• Speed Up Play the full clip back at a faster
  rate.
• Excision Remove sections of the recording.

14
Speed Up

• Simplest approach is to directly alter the
  playback rate.
• Has the side effect of altering the pitch of the
  speakers.
• Use an overlap and add algorithm to speed up
  whilst keeping pitch constant.
• Has the problem of not reflecting how speakers
  naturally increase their speech rate.
• Use a variable playback rate to better match how
  human speakers alter their speech rate.

15
Excision

• Simple approach is to remove non-informational
  parts of the recording e.g. silence.
• Limited by the amount of silence.
• Derive measures of word importance and only play
  back the important words missing words are
  mentally replaced.
• Far from natural speech.
• Use larger parts of speech (utterances) and
  locate important utterances and play only those
  back.

16
Examples
17
Experimental Overview

• Initial Exploratory Experiment
• Gain an understanding of the space.
• Informally assessed a large number of techniques.
• Located promising directions for research.
• Follow up detailed study
• Examined a subset of the techniques explored.
• Used a measure of gisting ability to assess
  success.
• Examined short and long meeting clips.
• Also examined effect of a user interface.

18
Measuring Gisting Ability

• A key facet of our techniques is that they
  support the discovery of gist rather than facts.
• Therefore the metrics we have used previously do
  not adequately capture the proposed usage of
  these tools.
• Key components of the performance metric
• Must be quick to assess and to score
  (experimenter and subject time)
• Objective measure

19
Measuring Gisting Ability (2)

• Our solution was to use a hybrid gold standard
  scheme.
• We measure the importance of utterances from the
  transcript and select a number of utterances from
  the full range of importance.
• We then ask judges to rank these utterances in
  order of importance.
• Subjects then listen to the meetings and perform
  the same ranking.
• The objective score is then the difference
  between the gold standard and subject rankings

20
Measuring Gisting Ability
21
Results

• Removing unimportant utterances performed better
  than speed up.
• Listeners understood the gist of a recording
  faster.
• All techniques performed better than applying no
  compression.
• With longer clips understanding was the same.
• Speed up required more interface interactions
  than excision.

22
Dynamic Summarization

• Using summary information to locate points of
  interest within a meeting transcript.
• Traditional summaries can be customized but are
  largely presented statically.
• Underpinned by two concepts
• User is able to dynamically alter the
  summarization level.
• Alteration shown in real time.
• Applying different presentation techniques.

23
Development Procedure

• Using the same process to evaluate as was used
  for the speech work.
• An initial lightweight evaluation of a number of
  UI concepts intended to find promising directions
  of research.
• A follow up study examining the techniques in
  more detail with a more rigorous evaluation
  protocol.

24
Dynamic Summary Display

• Two unit levels examined
• Words
• Utterances
• Two presentation techniques
• Unit shading.
• Unit excision.
• Two hybrid techniques
• Combining the four techniques into one
• An experimental fish-eye view

25
Examples

• Word Excision
• Word Shading

26
Initial results

• Shading works well.
• Operating at the word level is satisfactory.
• Fish-eye was not liked.
• The combinatorial approach did not really offer
  anything novel.

27
Follow Up Study

• Focus solely on the Word Excision and Word
  Shading techniques (highest rated in the previous
  experiment).
• Two questions (one specific, one general) about a
  number of meetings.
• Use the two interfaces (plus a control plain text
  transcript) to answer the questions (one question
  per meeting).
• Measure the time taken to answer, the accuracy
  and the amount of interface actions used when
  answering the questions.
• Collect subjective preference data and user
  comments about each of the techniques.

28
Follow Up Study Results

• Subjects were largely accurate there was no
  effect on interface type on the accuracy
• No effect of interface type on time taken to
  answer i.e. there was no efficiency loss as a
  result of using the dynamic interfaces.

29
Preference and Process Results

• Subjects overwhelmingly preferred the Word
  Excision Condition.
• Subjects scored the Word Excision and Plain
  Transcript conditions equally.
• The Word Shading condition required less
  interface actions than the Word Excision
  condition.
• Specifically users spent more time changing
  compression levels in the Word Excision condition.

30
Video Compression

• The same techniques for audio can also be applied
  to video.
• Compress the audio recording and use this
  compressed version to derive an audio-video
  recording.
• Informal evaluation indicates a different
  modality for video.

31
Video Examples

• Type of video being used
• Word excised video
• The cuts are now much more disconcerting.
• Sped Up video
• More comfortable to watch but disconcerting at
  high compression levels.
• Can also do non-linear compressed video
• Speed up only the non-silent parts.
• Can also e.g. speed up through unimportant parts

32
Summary

• Looking at Interfaces for Browsing Meeting
  Recordings
• Problems with abstraction in current meeting
  recording technology and automatic browsing
  systems
• Temporal Compression of Speech
• Reducing the time required to listen to a speech
  recording but keeping the important information.
• Utterance Excision.

33
Summary

• Dynamic presentation of meeting transcripts
• Real time selection of summary level.
• Word Shading.
• Temporal Compression of Video
• Applying the above to video recordings.
• Speed up more effective.