Turn-Taking and Diarization

1
Turn-Taking and Diarization

• Julia Hirschberg
• CS 4706

2
Today

• Turn-taking behaviors in human-human conversation
• Task/circumstance dependencies
• Conversational Analysis
• Linguistic/cultural differences
• How do we take and give up turns?
• Diarization Automatic Turn Segmentation

3
Turn-taking Behavior

• Dialogue characterized by turn-taking
• How do speakers know what to say and when to say
  it?
• Conversational partners expect certain patterns
  of behavior in normal conversation
• Pat You got an A? Thats great!
• Chris Yeah, Im really smart you know.
• Chris Well, I was just lucky I happened to read
  the chapter on dialogue systems right before the
  test. Otherwise I never would have squeaked
  through.
• General patterns in ordinary conversation
• Deviation is significant

4

• Children learn turn taking within first 2 years
  (Stern 74)
• General individual differences
• Shy people pause longer and speak less and less
  often (Pilkonis 77)
• Schizophrenics, neurotics, depressed people less
  skilled in turn-taking

5
Expectations of What to Say May Depend on Task at
Hand

• Telephone
• Openings
• Pat Hello?
• Chris Hi, Pat. Its Chris.
• Pat Hi!
• Closings (6-turn)
• Chris Well, I just wanted to see how you were
  doing
• Pat Thanks for calling. We'll have to have lunch
  sometime
• Chris I'd like to
• Pat Okay
• Chris Okay
• Pat See you
• Chris Yeah, see you

6

• Email
• Pat Hi, can we switch lunch to 1230? Im
  running late.
• Chris Sure. 1230.
• Pat Great. See you.
• Service encounters
• Clerk Good morning. Is there something I can
  help you with?
• Pat Hi. Yeah. I wonder if you could show me.
• Meetings
• Boss Today I want to focus on next years goal
  statements. Chris, could you report please.
• Chris
• Boss Pat, now lets hear from you
• Pat
• News broadcasts
• Anchor Chris Smith reports from Rome now on the
  upcoming conclave. Chris?
• Reporter Thanks, Pat.. And now back to Pat
  Jones in New York.

7
Today

• Turn-taking behaviors in human-human conversation
• Task/circumstance dependencies
• Conversational Analysis
• Linguistic/cultural differences
• How do we take and give up turns?
• Diarization Automatic Turn Segmentation

8
Conversational Analysis (Sacks et al 74)

• Can we characterize expectations of what to say
  more generally?
• Rules of turn-taking
• If, during this turn the current speaker has
  selected A as the next speaker, then A must speak
  next
• If the current speaker does not select the next
  speaker, any other speaker may take the next turn
• If no one else takes the next turn, the current
  speaker may take the next turn
• Rules Apply at Transition Relevant Places (TRPs)
  where something allows speaker changes to occur

9
Where Can Speaker Shifts Occur

• Adjacency pairs
• Question/answer
• Greeting/greeting
• Compliment/downplayer
• Dispreferred responses
• Silence
• No to a simple request without explanation
• Changing the topic abruptly without transition
• Important for Spoken Dialogue Systems

10
Today

• Turn-taking behaviors in human-human conversation
• Task/circumstance dependencies
• Conversational Analysis
• Linguistic/cultural differences
• How do we take and give up turns?
• Diarization Automatic Turn Segmentation

11
Cultural Differences in Turn-Taking

• Chinese telephone conversations
• Openings (Zhu 04)
• Mandarin vs. British
• Identification differences
• British self-report
• Chinese callees ask the caller
• Closings (Sun 05)
• 39 female-female telephone conversations
• Closings initiated through matter-of-fact
  statement of intention to end conversation
• Verbalized thanking occurs except in
  mother/daughter closings not the standard
  English model
• Finnish business calls (Halmari 93) vs. American
• Americans get right to the point
• Finns chat

12
Today

• Turn-taking behaviors in human-human conversation
• Task/circumstance dependencies
• Conversational Analysis
• Linguistic/cultural differences
• How do we take and give up turns?
• Diarization Automatic Turn Segmentation

13
Individual Differences British Politicians
(Beattie 82)

• Data 25m televised interviews before 1979
  British General election
• Margaret Thatcher (Tory leader) the Iron Lady
• Jim Callaghan (Prime Minister) Sunny Jim
• Who interrupts?
• Less intelligent, highly neurotic, extroverted
• Men interrupt women
• Interruptions may indicate
• Desire for dominance
• Desire for social approval
• Conveyance of joint enthusiasm, heightened
  involvement

14

• Method
• Identify spkr 2 attempts to take the turn
• Smooth switches no simultaneous speech, spkr 1s
  utterance complete, turn to spkr 2
• Simple interruptions simultaneous speech, spkr 1
  doesnt complete utterance, turn to spkr 2
• Overlap simultaneous speech, spkr 1 completes
  utterance, turn to spkr 2
• Butting-in simultaneous speech but no change of
  turn, spkr 1 keeps the turn
• Silent interruption spkr 1s utterance
  incomplete, no simultaneous speech, turn to spkr
  2

15

• Analyze acoustic/prosodic and gestural
  information
• Turn-yielding behavior
• Pauses
• Speaking rate slows
• Drawl at end of clause
• Drop in pitch or loudness
• Completion of syntactic clause
• Gesture of termination
• Attempt suppression signals
• Filled pauses
• Gestures

16
Results

• Mrs. Thatcher interrupted almost twice as often
  as she interrupts interviewer (19/10) unlike
  Callaghan (14/23)
• Thatcher Starts slow and gets faster, few FPs
  (4)
• Callaghan starts fast and gets slower, many FPs
  (22)
• Public perception Thatcher is domineering in
  interviews and Callaghan is a nice guy
• But Thatcher does not dominate
• Why is Thatcher interrupted?
• Interruptions come at end of syntactic clause
  when drawl on stressed syllable in clause and
  falling intonation

17

• No suppression signals
• Why does she do this?
• Speech training before election?
• Why is she still perceived as domineering?
• When interrupted she doesnt cede the floor
  despite lengthy stretches of simultaneous speech

18
Today

• Turn-taking behaviors in human-human conversation
• Task/circumstance dependencies
• Conversational Analysis
• Linguistic/cultural differences
• How do we take and give up turns?
• Diarization Automatic Turn Segmentation

19
Diarization Automatic Speaker
Identification/Segmentation

• Segment audio corpora (Broadcast News, meetings,
  telephone conversations) into speaker segments
• Speaker segmentation
• Speaker identification
• Speech and music
• Speaker segmentation (Diarization)
• Initial segmentation
• Segment clustering based on acoustic features
• State-of-the-art 8.47 error

20

• Speaker identification
• Linguistic information to identify speaker types
  and speaker names (LIMSI 04)
• Templates (ltnamegt has this report from
  ltlocationgt)
• Results 10.9 error on test set
• But only 10 of segments contain relevant
  patterns
• Estimate 25 error on broadcast news if
  segmentation and clustering is done to id all of
  each speakers segments

21

• ltDOCgt
• ltDOCNOgt CNN19980104.1130.0000 lt/DOCNOgt
• ltDOCTYPEgt MISCELLANEOUS TEXT (automatic initial)
  lt/DOCTYPEgt
• ltDATE_TIMEgt 01/04/1998 113000.00 lt/DATE_TIMEgt
• ltBODYgt
• ltTEXTgt
• lt/TEXTgt
• lt/BODYgt
• ltEND_TIMEgt 01/04/1998 113034.71 lt/END_TIMEgt
• lt/DOCgt
• ltDOCgt
• ltDOCNOgt CNN19980104.1130.0034 lt/DOCNOgt
• ltDOCTYPEgt NEWS STORY lt/DOCTYPEgt
• ltDATE_TIMEgt 01/04/1998 113034.71 lt/DATE_TIMEgt
• ltBODYgt
• ltTEXTgt
• in northern kentucky are forcing 3,000 people in
  two states to flee their
• homes.
• the fire started early this morning at the
  cargill company plant in

22

• officials say all they can do is let the fire
  burn itself out, because
• spraying water on the flames would be too
  dangerous.
• ltTURNgt
• at the current time, our only way of getting it
  under control is to stay
• away from it.
• we've backed everyone off from the fire by about
  a mile and a quarter and
• evacuated homes in that radius and the chief
  threat at this point is a very
• small risk of a very large explosion caused by
  400 tons of ammonia nitrate
• stored in the building.
• ltTURNgt
• foir people have been taken to hospitals.
• one firefighter was injured and treated on the
  scene.
• lt/TEXTgt
• lt/BODYgt
• ltEND_TIMEgt 01/04/1998 113131.00 lt/END_TIMEgt
• lt/DOCgt
• ltDOCgt
• ltDOCNOgt CNN19980104.1130.0091 lt/DOCNOgt
• ltDOCTYPEgt NEWS STORY lt/DOCTYPEgt

23

• caused at least three buildings to collapse.
• it set off a four-alarm fire, which has been
  contained.
• officials tell cnn one person was injured.
• investigators have not determined the cause of
  the incident.
• lt/TEXTgt
• lt/BODYgt
• ltEND_TIMEgt 01/04/1998 113148.11 lt/END_TIMEgt
• lt/DOCgt
• ltDOCgt
• ltDOCNOgt CNN19980104.1130.0108 lt/DOCNOgt
• ltDOCTYPEgt NEWS STORY lt/DOCTYPEgt
• ltDATE_TIMEgt 01/04/1998 113148.11 lt/DATE_TIMEgt
• ltBODYgt
• ltTEXTgt
• unexpected weather conditions are the rule across
  much of the united states
• this weekend.
• angela astore reports.
• ltTURNgt
• ltANNOTATIONgt Reporter lt/ANNOTATIONgt

24

• it was a different scene in the northwest, where
  snow is the story.
• but the winter weather didn't stop this man from
  getting in some warmer
• pursuits.
• and he wasn't bothered by the fact that he
  couldn't see where his golf
• balls landed.
• ltTURNgt
• it's not really where it's going to land that's
  important at this point
• while you are learning.
• once you've learned, then it is.
• we'll worry about that when the snow clears.
• right now, it's probably better that i don't see
  where they land.

25
Next Class

• Errors and Corrections in SDS