SMART Final Review Meeting WP2 Sinuhe, MMBT

1
SMART Final Review MeetingWP2 Sinuhe, MMBT

• Craig Saunders
• November 2009

2
Main Contributors

• University of Southampton
• C. Saunders, S. Szedmak, Y. Ni, M. Niranjan
• University College London
• Z. Wang, J. Shawe-Taylor
• University of Helsinki
• J. Rousu, M. Kääriäinen, E. Galbrun
• Jozef Stefan Institute
• B. Fortuna
• University of Bristol
• M. Turchi
• Xerox Research Centre Europe
• N. Cancedda, L. Specia

3
WP2 Machine Learning and SMT

• State of the art Machine Learning
• Large scale optimisation approaches
• Structured prediction
• Application to Statistical Machine Translation
• Improvement of SMT pipeline components
• Novel approaches to SMT

4
Relation to ST objectives

• Directly tackles objective 1
• Machine learning optimisation
• Direct contribution to ER2.1 and ER2.2
• Effectively measure the performance of the
  systems using standard benchmarks
• Good results on (very) ambitious goals

5
Position w.r.t. the State of the Art

• Two novel SMT systems
• Sinuhe
• Scalable, Fast decoding, s.o.t.a. performance
• Machine Learning meets phrase-based SMT
• MMBT
• Structured Prediction approach to SMT
• Very novel truly end-to-end system
• Good performance, approaching s.o.t.a.
• Details are in D2.X deliverables

6
Position w.r.t. the State of the Art

• Other developments
• LSR approach
• Encouraging results, difficult to scale
• L1 structured training of SMT models
• Improvements over Moses, computationally
  expensive
• Max-margin structured reordering
• Improves current state of the art
• Potential Integration into Moses in future

7
Position w.r.t. the State of the Art

• Other developments
• Moses benchmarking
• Examining the learning rates of Moses
• Confidence estimation
• Quality and confidence estimates for MT output

8
Benchmark Results

• Europarl Approx 1.2M sentence pairs
• Results obtained for 10K, 50K, 100K and full
  (1.2M) sentence pair training set sizes
• Full results available on web
• Language pairs en-es, en-fr
• Evaluation NIST (lowercase)

9
Benchmark Results

• French to English (full train, test 10K)
• First four systems show statistical significance
• ER2.2 Very Satisfactory within 5
• MMBT is approaching performance without LM

10
Benchmark Results

• Spanish to English (full train, test 10K)
• Note the improvement differential of new Portage

11
Benchmark Results

• Summary
• Sinuhe within 5-10 of baseline
• SinuheLM within 2-4 of baseline
• MMBT within 10 of baseline
• Language model extension would improve
  performance
• (new) Portage improvement 2
• ER2.1 Incremental improvement lt 5
  unsatisfactory
• In

ER2.2
12
Structured Prediction

• MMS reordering model
• Replaces one component of SMT pipeline
• Training 50K, Test 1K
• Results
• En-Fr Portage 6.47, Moses 6.51, MMS 6.68
• Fr-En Portage 6.51, Moses 6.53, MMS 6.79
• MMS improvement 2-4 over Moses
• Good improvement also shown on En-gtCh

13
Integration and Dissemination

• Integrated into CAT and Wikipedia demos
• Sinuhe and MMBT released under LPGL
• Publications
• Conferences
• 2009 (4) EMNLP, MLSP, 1 ACL-IJCNLP, AISTATS
• 2008 (3) (2) ACL SMT w/shop, Book Chapter
• 2007 (3) NAACL-HLT, ECML, IDA
• Journals (submitted)
• 1 (3) Machine Translation, (1) Computer Speech
  and Language
• Confidence Estimation used with LSP at XRCE

14
Conclusions

• Many scientific approaches to applying state of
  the art in Machine Learning to Statistical MT
• Two novel SMT systems
• New additions for end-users and research
  community
• Several ML module adaptations
• Structured prediction for reordering
• Confidence estimation extension
• Excellent overall performance
• Strong achievements in a 3 year time-frame

15
The SMART Consortium
16
The SMART Consortium
17
User Trials and Demos

• Comparing TER of systems

18
Benchmark Results

• English to French (full train, test 10K)
• For this particular data set, the baseline did
  not perform so well

19
Benchmark Results

• English to Spanish (full train, test 10K)

20
User Trials and Demos

• Sinuhe and MMBT
• Integrated into the Wikipedia demo
• Used in user trials
• Same set of 619 sentences
• CAT Post edit
• MMR Sinuhe with/without post-editing

21
User Trials and Demos

• Comparing TER of systems

22
User Trials and Demos

• Comparing edit distance of systems