CLEF2007 CrossLanguage Speech Retrieval Track Overview

1
CLEF-2007Cross-Language Speech RetrievalTrack
Overview

• CU Pavel Pecina, Jan Hajic, Petra Hoffmannova
• DCU Gareth Jones, Ying Zhang
• UMD Doug Oard, Dagobert Soergel, Scott Olsson
• IBM Bhuvana Ramabhadran
• JHU Bill Byrne (Cambridge), Zak Shafran (OHSU)
• USC Sam Gustman
• UWB Pavel Ircing

2
Speech Retrieval Evaluations

• 1996-1998 TREC SDR
• English broadcast news English queries
• 1997-2004 TDT
• multilingual news query by example
• 2003-2004 CLEF CL-SDR
• English broadcast news Multilingual queries
• 2005-2007 CLEF CL-SR
• English/Czech interviews Multilingual queries
• 2007 CLEF QAST
• English lectures/meetings, English questions

3
Whats New in 2007?

• Czech
• Fixed quickstart time alignment problem!
• 29 training topics, 42 new evaluation topics
• 3 new teams (Brown, Chicago, Charles U)
• English
• 17 relative improvement over 2006 (TD, ASR)
• 4 new teams (Brown, Chicago, Jaen, Amsterdam)
• Same topics and ASR as 2006
• 63 training topics, 33 evaluation topics

4
CLEF?2007Cross-Language Speech Retrieval Track
Overview
5
English ASR
ASR2006A
ASR2004A
ASR2003A
Training 200 hours from 800 speakers
6
ltDOCgt ltDOCNOgtVHF00009-056154.003lt/DOCNOgt ltINTERVIE
WDATAgt Sidonia L... 1927 Shaindl L...
Sydzia lt/INTERVIEWDATAgt ltNAMEgtIssac L..., Cyla
L...lt/NAMEgt ltMANUALKEYWORDgt Shabbat Jewish
identity customs and observances, Jewish
Przemysl (Poland) food Poland 1918 (November
11) - 1939 (August 31) lt/MANUALKEYWORDgt ltSUMMARYgtS
L recounts her daily activities. She notes her
family's Jewish identity and she talks about a
typical Shabbat. SL describes cholent.lt/SUMMARYgt lt
ASRTEXT2003Agtdecember when you talk about what
your daily life was like well after realized i
was a little girl going to public school during
the day after works there were all the lessons
and activities in the winter either an awful lot
of ice skating and i was a very good ice skater
figure i scared that then i would play with
friends on the weekends there are things some of
em now are were my parents were religious that
meant that on saturday i couldn't do very much i
they went to the services i was hanging on
outside was uh the kids making all the noise but
afterwards we had to go home any
ltASRTEXT2004Agtdecember when you talk about what
your daily life was like well after realized i
was a little girl going to public school during
the day after works there were all the lessons
and activities in the winter either an awful lot
of ice skating and i was a very good ice skater
figure i scared that then i would play with
friends on the weekends there are things some of
em now are were my parents were religious that
meant that on saturday i couldn't do very much i
they went to the services i was hanging on
outside was uh the kids making all the noise but
afterwards we had to go home ltASRTEXT2006Agtdecem
ber when you talk about what your daily life was
like well after realized i was a little girl
going to public school during the day after works
there were all the lessons and activities in the
winter either an awful lot of ice skating and i
was a very good ice skater figure i scared that
then i would play with friends on the weekends
there are things some of em now are were my
parents were religious that meant that on
saturday i couldn't do very much i they went to
the services i was hanging on outside was uh the
kids making all the noise but afterwards we had
to go home any ltASRTEXT2006Bgtdecember when you
talk about what your daily life was like well
after realized i was a little girl going to
public school during the day after works there
were all the lessons and activities in the winter
either an awful lot of ice skating and i was a
very good ice skater figure i scared that then i
would play with friends on the weekends there are
things some of em now are were my parents were
religious that meant that on saturday i couldn't
do very much i they went to the services i was
hanging on outside was uh the kids making all the
noise but afterwards we had to go home any
ltAUTOKEYWORD2004A1gt cultural and social
activities customs and observances, Jewish
family life food Shabbat sports and games
education family homes grandmothers
education, Jewish Jewish-gentile relations
schools synagogues Polish (language)
working life photographs (stills) 1930s
Poland 1918 (November 11) - 1939 (August 31)
Poland 1935 (May 13) - 1939 (August 31) Cracow
(Poland) Germany 1918 (November 11) - 1939
(August 31) lt/AUTOKEYWORD2004A1gt ltAUTOKEYWORD2004A
2gt Poland 1918 (November 11) - 1939 (August 31)
customs and observances, Jewish education
cultural and social activities extended family
members education, Jewish family life
Jewish-gentile relations Jewish identity
Hungary 1918 (November 11) - 1939 (August 31)
Shabbat sports and games Budapest (Hungary)
Poland 1941 (June 21) - 1944 (July 21)
synagogue attendance Hungary 1939 (September 1)
- 1944 (March 18) food in the ghettos forced
labor in the ghettos fate of loved ones food
lt/AUTOKEYWORD2004A2gt lt/DOCgt
7
An English Topic
Number 1148 Title Jewish resistance in
Europe Description Provide testimonies or
describe actions of Jewish resistance in Europe
before and during the war. Narrative The
relevant material should describe actions of
only- or mostly Jewish resistance in Europe. Both
individual and group-based actions are relevant.
Type of actions may include survival (fleeing,
hiding, saving children), testifying (alerting
the outside world, writing, hiding testimonies),
fighting (partisans, uprising, political
security) Information about undifferentiated
resistance groups is not relevant.
8
Automatic English TD Runs
Ottawa
DCU
Brown
Chicago
Amsterdam
9
Automatic English TD Runs
AK1 AUTOKEYWORD2004A1, AK2 AUTOKEYWORD2004A2,
ASR03 ASRTEXT2003A, ASR04 ASRTEXT2004A,
ASR06A ASRTEXT2006A, and ASR06B ASRTEXT2006B.
10
Wilcoxon Signed-Rank Test
UO ? DCU The number of nonzero tests is --gt
33 The sum of the signed rank is --gt
86.000000 The 95.0 level of confidence is --gt
184.129807 Methods cannot be separated. UO ?
BLLIP The number of nonzero tests is --gt 33 The
sum of the signed rank is --gt 147.000000 The
95.0 level of confidence is --gt
184.129807 Methods cannot be separated. DCU ?
BLLIP The number of nonzero tests is --gt 32 The
sum of the signed rank is --gt 12.000000 The
95.0 level of confidence is --gt
175.945801 Methods cannot be separated. BLLIP ?
UC The number of nonzero tests is --gt 33 The sum
of the signed rank is --gt 198.000000 The 95.0
level of confidence is --gt 184.129807 Method 1 is
better than method 2. UC ? UVA The number of
nonzero tests is --gt 33 The sum of the signed
rank is --gt 138.000000 The 95.0 level of
confidence is --gt 184.129807 Methods cannot be
separated.
11
Monolingual vs. Cross-lingual (Automatic TD Runs)
12
CLEF 2007 The CL-SR Czech Track

• Pavel Pecina
• pecina_at_ufal.mff.cuni.cz

13
Whats Different in Czech?

• Lack of manual interview annotation
• no topic boundaries (start and stop times)
• no description (summary, assessors' scratchpad)
• English labels (assigned thesaurus terms) not used

• Unknown-boundary relevance assessment
• manual labeling of start and stop times

• Task to identify appropriate replay points
• focus on start time, stop times ignored

• modified mGAP used as the evaluation measure
• penalization for not exact match

14
Interviews

• Czech Holocaust survivors testimonies
• 357 mostly seen speakers, 565 hours
• 35 ASR mean Word Error Rate

2007 Quickstart collection

• 11377 automatically generated overlapping
  passages
• average passage duration 3.75 min, 33 overlap
• fields DOCNO, INTERVIEWDATA, ASRSYSTEM, CHANNEL,
  ASRTEXT
• no thesaurus terms used in 2007

15
Interview Usage

335 Seen
Czech
22 Unseen


800 ASR Train
English
297 IR Eval

16
ltDOCNOgtVHF04106-7401.30lt/DOCNOgt ltINTERVIEWDATAgtTom
my K...-K...lt/INTERVIEWDATAgt ltASRSYSTEMgt2006lt/ASRS
YSTEMgt ltCHANNELgtrightlt/CHANNELgt ltASRTEXTgtPRIVEZLI
VEZNE NA NOSÍTKÁCH A PROSTE MUSEL TADY BYL
SKUTECNE KADÝ KDO TO V TEREZÍNE V TÉ DOBE BYL
ALE STEJNE VÝSLEDEK BYL MYSLÍM E MLUVÍ lt/sgt ltsgt
STOJÍM PROSTORU KDE lt/sgt ltsgt A DO KONCE KVETNA
CTYRICET TRI PRICHÁZELY TRANSPORTY Z BOHUOVIC
TED JETE CHODILI PEKY ALE POTOM U JE TO DEVET
CASTO MOJE TO ZACHOVALA CÁST KOLEJÍ KTERÉ VEDLY
A TAK ZA HAMBURSKÝ KASÁRNA TAM POTOM U OD
KVETNA CTYRICET TRI DOCHÁZELY VECHNY TRANSPORTY
JAK Z VENKU TAKÉ OPACNE KDY TEREZÍNA LY
TRANSPORTY DO TAKOVÉHO lt/sgt ltsgt ZA NÁMI TO BYLA
JSOU HAMBURSKÝ KASÁRNA TAM JSOU HANNOVERSKÝCH
KASÁRNÁCH TADY NA NA TOM O TOM PROSTORU PRED TOU
HROMADOU VECÍ MIMORÁDNE PATNÉ I TADY TREBA MÍSTO
KDE PRICHÁZELY TRANSPORTY KDY CILI DIALOSTECKÝ
DETI DIALOSTECKÝ DETI DVANÁCTSET NEBO KDY POTOM
CHODILY TRANSPORTY Z NEMECKA TAK PROSTE NA TEN NA
TECH KOLEJÍCH SE DELALO JAK TRANSPORTOVÁNI LIDI
CO PRICHÁZELI SEM TAK TRANSPORTOVÁNI TY VECI DÁL
DO OSVETIMI NEJVETÍ TRANSPORTY SE TO ZÁRÍ
CTYRICET CTYRI KDY TAKÉ ODEEL MUJ OTEC A BRATR
TO U BYLO ASI DESET TRANSPORTU PO TISÍCI MUÍCH
TI IDOVTÍ CINNOST KTERÝ SE VELKÁ VETINA
NEVRÁTIL lt/sgt ltsgt TAK SEM PODÍVALA DELÁME NA
BATU TO JE ZAJÍMALI PROSTOR TÍM NEJAKÝM OSOBNÍ
VZPOMÍNKU lt/sgt ltsgt NA TOM NAHORE TO BYLO
ZAHRADNICTVÍ VLÁDNOUT ALE NEJAKÝ DO DOBY NE U
TEN TAM BYL POSTAVENÝ DOMECEK ALE BYLO TAM TAKÉ
MALÝ V ODBOJI HRITE TAM JSME NIKDY NE- NEKDY
PRESNE TO VÍM SEDMADVACÁTÉHO KVETNA CTYRICET
CTYRI JSME TAM HRÁLI FOTBAL UTKÁNÍ SPARTA NE JÁ
VÍM SPARTA TO BYL KLUK Z KLUKU KTERÝ JSME BYDLELI
TADY V HAMBURSKÝCH KASÁRNÁCH TAKE TEN PODVOZEK
ALE NAPRED BYLI MY JSME MELI SVÍCKU NEBO SE
SPARTA A SESTRY STA SEDMNÁCT TY TO BYL DETSKÝ
DOMOV KLUKU TAM MELI SE STALO PRED VNUCKU NE ALE
MY JSME TEHDY DO UTKÁNÍ lt/sgt ltsgt PROHRÁLI TRI
JEDNA ALE TO NENÍ PODSTATNÝ PODSTATNE TO E TEN
ZÁPAS A JÁ TADY DVACÁTÉHO SEDMÉHO KVETNA CTYRICET
CTYRI A E SE NA NE BYL JETE PUJDE OTEC PAK U
ODJEL DÁL A U SEM U NÍ NEVEDEL lt/sgt ltsgt KOUKÁME
SE NÁM ÚZKÁ KASÁRNA TO JSOU KASÁRNA KTERÁ BYLA
DREVENÁ PRO ENY TAK SEM TADY S MAMINKOU HNED V
TOM CTYRICÁTÉM DRUHÉM JSME SEM PRILI TAK JSME
TADY BYLI UBYTOVANÝ NA POKOJ MESTE EST E TO
TYPICKY KASÁRENSKÝCH DOBU KASÁRENSKÝCH BUDOVAL S
DVEMA DVORY VELIKÝM lt/sgt ltsgt A VPRAVO BYLO JSOU
HANNOVERSKÝCH KASÁRNÁCH TA MOJE CERNÁ PEKÁRNA TAM
SE TAK CHLEBA I PRO TEREZÍN TO PRO NÁS A BYL TAM
TAKÉ TAKOVÝ DVUR ODJÍDELI ENY CO VOZILI NA TU
DOBRANSKÝCH VOZECH VECHNO CO BYLO POTREBA
TEREZÍNE DELAL VE SE VOZILO S- VOZY KTERÝ BYLI V
TOM DVORE TAMHLE VZADU MAMINKA TAM BYL ZAMESTNÁN
OSUDU HUNDERTSCHAFT lt/sgt ltsgt JSME UMELY
HAMBURSKÝCH KASÁREN PRED TÍM JSME SE DÍVALI OTEC
PO TOM PRVNÍM DVORE HAMBURSKÝ KASÁRNA KDE BYLI
lt/sgt ltsgt ÚPLNE TEDA VETINOU ENY JÁ SEM ZABIL S
MATKOU PUL ROKU DVAACTYRICET lt/sgt ltsgt BUDE TAKÉ
VIDET MÍSTO KDE JSME BYLI BYLO TO MÍSTO CÍSLO DVE
STE EST S MAMINKOU SPALY SAMOREJME TÍM E MATKA
CHTELAlt/ASRTEXTgt
17
Topics selection
38 2005 English Training
25 2005 English Evaluation
33 2006 English Evaluation
9 Unused
10 2006 Czech Safety
115
28 2006 Czech Evaluation
1 2006 Czech Evaluation
29
18
Topics selection
38 2005 English Training
25 2005 English Evaluation
33 2006 English Evaluation
9 Unused
10 2006 Czech Safety
115
28 2006 Czech Evaluation
1 2006 Czech Evaluation
29
29 2007 Czech Training
29
19
Topics selection
38 2005 English Training
25 2005 English Evaluation
33 2006 English Evaluation
9 Unused
10 2006 Czech Safety
3 2007 Czech Safety
118
28 2006 Czech Evaluation
1 2006 Czech Evaluation
29
29 2007 Czech Training
29
20
Topics selection
40 Possible 2007 Czech Evaluation
10 Possible 2007 Czech Evaluation
(6 or more relevant passages identified during
search-guided assessment)
50
38 2005 English Training
25 2005 English Evaluation
33 2006 English Evaluation
9 Unused
10 2006 Czech Safety
3 2007 Czech Safety
118
28 2006 Czech Evaluation
1 2006 Czech Evaluation
29
29 2007 Czech Training
29
21
Topics selection
34 2007 Czech Evaluation
8 2007 Czech Evaluation
(highly-ranked assessment completed)
42
40 Possible 2007 Czech Evaluation
10 Possible 2007 Czech Evaluation
(6 or more relevant passages identified during
search-guided assessment)
50
38 2005 English Training
25 2005 English Evaluation
33 2006 English Evaluation
9 Unused
10 2006 Czech Safety
3 2007 Czech Safety
118
28 2006 Czech Evaluation
1 2006 Czech Evaluation
29
29 2007 Czech Training
29
22
Evaluation Measure

• based on the mean Generalized Average Precission
• human assessments are binary
• degree of match to the assessments can be partial
• penalization for non 100 match up to 150 sec

0 sec
-75 sec
150 sec
1.0
0.5
0.0

• quantization noise (scores lower than for
  English)
• 15 sec assessment granurality
• quickstart documents begin every 150 sec

23
Relevance Judgements

• performed by 6 relevance assessors in Prague

Search-guided assessment

• completed for 87 topics
• 2156 rel. passages identified in the evaluation
  topics

Highly-ranked assessment

• completed for 42 topics
• pool depth set to 50 start times
• 11896 highly-ranked start times checked
  (284/topic)
• 233 rel. passages identified

24
Relevance Assessment Interface
25
Relevance Judgement Results
26
Relevance Judgement Statistics
27
Participation

• Brown University (BLLIP)
• Matthew Lease, Eugene Charniak
• Charles University (CUNI)
• Pavel Ceka, Pavel Pecina
• University of Chicago (UC)
• Gina-Anne Levow
• University of West Bohemia (UWB)
• Pavel Ircing, Ludek Müller

• total of 15 runs submitted
• required condition automatic queries from Title
  and Description

28
Results
29
Results
30
Results Term normalization

• the effect of term normalization for handling
  Czech morphology is quite significant
• 60-120 relative improvement

31
Alignment Issues in the Quickstart Collection

• 2006 data release (affected 2006 working notes)
• Time mismatch made mGAP uninformative (pauses
  ignored)
• Post-CLEF 2006 evaluation (corrected in 2006
  proceedings)
• Post-hoc start time correction (but missing tapes
  counted as 30 min)
• AUTO and MANUAL KEYWORDS still misaligned
• 2007 data release
• Some additional corrections for ASR timing
• AUTO and MANUAL KEWYORDS removed (too hard to
  fix)
• 2007 evaluation (reported in 2007 working notes)
• Missing-tape timing corrected post-hoc

32
(No Transcript)
33
Test Collection Release

• CLEF CL-SR track test collections
• Package for release
• Independent (cross-site) validation
• Deposit at ELDA
• MALACH ASR training data
• Package English and Czech for release
• With Polish, Russian, Slovak ( maybe Hungarian)
• Deposit at LDC

34
What Did We Learn?

• Searching conversational speech works
• Real user needs, two languages
• Improving ASR helps less than expected
• Error rates vary by speaker
• Ranked retrieval prefers lower error rates
• Automatic classification can help ASR
• At least if error rates are high
• Unsegmented sources bring new challenges
• Cross-sourcealignment
• Evaluation measure design

35
Critiquing the Collection

• Large for ASR is small for IR
• 1,000 hours of speech 20,000 documents
• No manual reference transcription
• Would cost 100,000
• Interviews are just one type of conversation