MAGIC Seen from the Perspective of RAGS

1
MAGIC Seen from the Perspective of RAGS

• Kathleen R. McKeown
• Department of Computer Science
• Columbia University

2
MAGIC

• Multimedia Abstract Generation of Intensive Care
  data
• Collaborators
• Steven Feiner, Desmond Jordan
• Shimei Pan, James Shaw, Michelle Zhou
• Kris Concepcion, Liz Chen, Jeanne Fromer

3
Scenario

• Goal provide post-operative information on
  bypass patients (CABG)
• Prior to completion of surgery and before
  transport to Cardiac Intensive Care Unit (ICU)
• Status needed for ICU nurse, cardiologist
• Time critical

4

5
Issues for Language Generation

• Conciseness Coordinated speech and text that is
  brief but unambiguous
• Coordination with other media Modify wording and
  speech to coordinate references with graphical
  highlighting
• Media specific tailoring
• Produce wording appropriate for spoken language
• Use information from language generation to
  improve quality of synthesized speech

6
Status

• Implemented prototype showing coordination
  between media for limited input
• Text output for large numbers of input cases
• Undergoing evaluation now in ICU
• Runs on live data on a daily basis
• 5-10 error rate
• Continuing research on effects of LG information
  on prosody, partial results

7
(No Transcript)
8
Principles

• Early processes produce media independent
  representations
• Representations use partial orderings in order to
  make early commitments where possible and retain
  flexibility
• Both the speech and graphics content planner may
  add content and ordering constraints
• Constraints on later decisions may be added early
  on (e.g., lexical choice)

9
Data Server and Filter (conceptual)

• Input
• 1825 ltdruggt Drips Norepinephrine
• 1827 ltdruggt Drips Norepinephrine
• 1829 ltdruggt Misc. Magnesium Sulfate
• 1829 ltsurgerygt Cardiac Defibrillated by surgeon
• 183311 100 (BP) 51 (HR)
• 183401 96 52
• Output
• C-inanimate entity -gt C-drug -gt
  C-operating-room-medication -gtC-Drip -gt
  C-Norepinephrine
• Top-level categories
• C-state, C-event, C-entity (abstract, physical,
  organization, math)
• Inferences
• Hypotension time, duration, drugs given

10
General Content Planner - SOAP(Rhetorical,
semantic, conceptual)

• Overview
• Demographics
• Name, Age, MRN, Gender, Doctor, Operation
• Medical history
• Lines
• Therapy
• Devices
• Detail View
• Drips (on leaving)
• Induction info
• Devices
• Lab report
• Timeline
• Inferences
• End values
• Conclusions

11
Speech Content Planner - Satisfying Conciseness

• Speech content planner groups information into
  sentences
• Ms. Jones is an 80 year old, hypertensive
  diabetic female patient of Dr. Smith undergoing
  CABG.
• Ms. Jones is an 80 year old, female patient of
  Dr. Smith undergoing CABG. She has a history of
  diabetes and hypertension.
• To satisfy communicative goal to be concise,
  selects adjectives, prepositional phrases when
  possible.

12
Input to speech content planner -semantic
propositions

• X is-a patient
• X has-property last name Jones
• X has-property age 80 years
• X has-property history hypertension
• X has-property history diabetes
• X has-property gender female
• X has-property surgery CABG
• X has-property doctor Y
• Y has-property last name Smith

13
Forming Sentence Structure(Rhetorical, semantic,
lexical, syntactic)

• ((relation is-a) (arg1 ((item ((class name)
  (last-name Jones)))))
  (arg2 ((item ((class patient))))))
• ((relation is-a) (arg1 ((item ((class name)
  (last-name Jones)))))
  (arg2 ((item ((class patient))
  (premod ((history hypertension))))))

14
3 Types of Aggregation

• Hypotactic aggregation Given a set of
  propositions, can one be realized as a modifier?
• Semantic aggregation if a patient is on multiple
  drips and all devices, a patient has received
  massive cardiotonic therapy
• Paratactic aggregation Combine related
  propositions using conjunction and apposition

15
Coordination across media

• Temporal media
• Coordinate spoken references with highlighting of
  graphical references
• Requires negotiation of ordering and duration of
  media actions

16
Negotiating Ordering

• Spoken language generator has grammatical
  constraints on linear ordering
• Graphics generator has spatial constraints on
  layout
• Individual accounts of these constraints may
  result in an incoherent presentation

17

• Ms. Jones is an 80 year old, diabetic,
  hypertensive female patientof Dr. Smith
  undergoing CABG.

18
Problems for Language Generation Ordering

• When to provide an ordering over references?
• produce a partial ordering after word choice
• How to select an ordering compatible with
  graphics?
• produce several possibilities ordered by
  preference
• How to communicate orderings with graphics?
• maintain a mapping between strings and semantic
  objects

19
Media Negotiation(Conceptual, Semantic, Document)

• Speech components produce candidate partial
  orders
• 1.(lt name age ( diabetes hypertension) gender
  surgeon operation) 10
• 2. (lt name age gender surgeon operation (
  diabetes hypertension) 5
• 3. (lt name age gender ( diabetes hypertension)
  surgeon operation) 4

20
Media Negotiation

• Graphics components produce candidate partial
  orders
• 1. (di (highlight demographics) ((ltm)
  (subhighlight (mrn age gender))(subhighlight
  (medhistory))(subhighlight (surgeon
  operation))) 10
• 2. (di (highlight demographics)( (subhighlight
  (mrn age gender))(subhighlight (medhistory))(subhi
  ghlight (surgeon operation))) 7

21
CTS Architecture
Machine Learning
Prosody model
Speech Corpus
Other
Source
Prosodic
Rules
NLG System
Prosody Realizer
T T S
Text
Input
Sound
Annotated
Structure
Text
22
Focus of Research(Rhetorical, Semantic,
Syntactic, Prosodic)

• Build a prosody model for CTS using prosodic
  features (based on ToBI)
• pitch accent, phrase accent, boundary tone, break
  index.
• Features produced by LG
• Syntactic structure, POS tags, Semantic
  boundaries, Concept
• Informativeness, predictability (statistical
  models)
• Abnormality, unexpectedness, sequential
  rhetorical relation

23
Mapping to RAGS

• Data filter - conceptual
• General Content Planner - rhetorical, semantic,
  conceptual
• Speech Content Planner - rhetorical, semantic
  plus constraints on lexicalization, syntax
• Lexical Chooser - semantic, lexical, syntactic
• Media Coordination - semantic, conceptual,
  document
• Syntactic Realization - semantic, syntactic
• Prosody Realization -rhetorical, semantic,
  syntactic, prosodic

24
Acknowledgments

• This work was funded in part by
• DARPA
• NSF
• ONR
• New York State Center for Advanced Technology
• NLM