Evaluation of Algorithms Over the Internet

1
Evaluation of Algorithms Over the Internet

• Dr. Simon Lucas (University of Essex)
• Joanne Coy (Consignia Research Services)

2
Evaluation of Algorithms Over the Internet

• Why this approach?
• Meaningless performance estimates. This is
  because
• based upon different data sets
• data sets may not be relevant to application
• result evaluation routines
• there is no means of comparing run time
• no control over interference with the
  manipulation of results

3
Evaluation of Algorithms Over the Internet

• Why this approach?
• Large range of input
• Many researchers in image processing
• Geographically disparate
• Cut through the hype
• Researchers problems
• No means of comparison
• Wasted effort on evaluation
• Aim
• Establish a site that is respectable
• Attract researchers use as a development tool
• Visibility of the latest methods in development

4
Evaluation of Algorithms Over the Internet

• An internet based platform for the evaluation and
  effective comparison of algorithms
• Prototype resides at Essex
• Evaluation processes are automated
• Datasets, images and evaluation procedures
  established on the site
• Competition is launched
• Registration includes a non disclosure agreement
• Evaluator code and example solutions can be
  downloaded to build solutions
• Solution uploaded and placed in queue for
  evaluation (Java only)
• League tables of results are produced

5
Evaluation of Algorithms Over the Internet
6
Evaluation of Algorithms Over the Internet

• Trainable OCR algorithm competition November 2000
• 1,176 hits (15 from search engines)
• 15 entries, winner from National Taiwan
  University
• Example hits
• Stanford University
• Department of Computer Science, Edinburgh
  University
• University of Minnesota
• Mannheinz University
• National University of Singapore
• Core Software - A software house
• Motorola

• University of New Jersey
• University of Cambridge
• University of Vaasa, Finland
• Technical University of Vienna
• University of Sait-Etienne, France
• KPNQ West, Finland - An Internet services
  company
• Department of Computer Science, Trinity College,
  Dublin
• Novosibirsk State University, Russia
• University of Jerusalem
• University of Liege, Belgium
• Department of Information Processing, Tokoku
  University, Japan
• University of Queensland, Australia
• Department of Defence, Australia
• Department of Cybernetics, University of Reading
• Department of Computer Science, Southampton
  University

7

• University of Wolverhampton
• AtT Labs Research
• University of Maastricht
• Al Akhawayn University, Morocco
• Yamatte Corporation, Japan
• Assumption University of Thailand
• Centre for Scientific Computing, Berlin
• Kaist University, Korea
• University of Coruñ, Spain
• Department of Computing, Oxford University
• The Chinese University, Hong Kong
• University of Auckland, New Zealand
• University of Technology, Rodes, France
• Douglas College, British Columbia, Canada
• Pattern Recognition Group, Delft University,
  Holland
• Department of Electronic Engineering, Surrey
  University
• Department of Computer Science, Seattle,
  Washington
• School of Psychology, Birkbeck College, London
• Department of Electronic Engineering, University
  of Adelaide

• Chalmers University of Technology, Sweden
• IBM Research
• Bielefeld University, Germany
• Information Systems, Norwich University
• Department of Mathematics, Kings College,
  London
• University of Totu, Poland
• LockheedMartin
• University of Brno, Czech Republic
• Intel
• University of York
• Aston University
• Institute of Technology, Bombay
• Bell Labs
• Gjoevik College, Norway
• Department of Computer Science, University of
  Berlin
• University of Warwick
• University of Zurich
• Nuclear Engineering Department, University of
  Palermo, Spain
• AM University, Texas