Newsgroups: comp.ai.nat-lang
Path: cantaloupe.srv.cs.cmu.edu!bb3.andrew.cmu.edu!newsfeed.pitt.edu!gatech!news.mathworks.com!nntp.primenet.com!news.cais.net!news6.agis.net!agis!news.mxol.com!news.mcb.net!dcs.ed.ac.uk!newsfeed.ed.ac.uk!edcogsci!cnews
From: chrisbr@cogsci.ed.ac.uk (Chris Brew)
Subject: Re: Evaluating MT systems
In-Reply-To: jretief@bafana.epiuse.co.za's message of Wed, 26 Jun 1996
	15:13:04 GMT
X-Nntp-Posting-Host: brodie
Message-ID: <f3t7msnxa7u.fsf@brodie.cogsci.ed.ac.uk>
Sender: chrisbr@brodie.cogsci.ed.ac.uk
Organization: Centre for Cognitive Science, University of Edinburgh
References: <4qrkk4$11ul@inet.up.ac.za>
Date: Tue, 2 Jul 1996 09:42:29 GMT
Lines: 65


In article <4qrkk4$11ul@inet.up.ac.za> jretief@bafana.epiuse.co.za
(Judy Retief) writes:


> We're working on a multi-lingual MT system, and I'm looking for any
> information concerning the evaluation of such a system. What are the
> issues at stake? What linguistic models are used when people say
> "our system translates text 80% correctly" ?  Has any work been done
> to help one determine the weak parts of a translation system (such
> as morphology or concord or whatever)? Has any research been done to
> help one assemble a proper benchmark of sentences with which to
> evaluatate a system?

> Any general comments would be very welcome.


>   Judy Retief epi-use systems Pretoria, South Africa e-mail:
> judy.retief@epiuse.co.za

There are several questions here, all with long and controversial
answers. 

First off, why I know anything about this:

Henry Thompson and I did some work aimed at providing fully automatic
evaluations of translated texts by statistical means:

see
<http://www.ltg.ed.ac.uk/~chrisbr/papers/mt-eval-final/mt-eval-final.html>


Secondly an almost definitive reference: chapter 9 of the book by
Hutchins and Somers "An introduction to machine translation", Academic
Press, 1992, ISBN 0-12-362830-X. Of the references at the end of the chapter, I
particularly liked 

King and Falkedaal (1990) Using Test Suites in
Evaluation of Machine translation systems Coling 1990 Vol 2 pp 211-216
and L

ehrberger and Bourbeau (1988) "Machine Translation: linguistic
characteristics
of MT systems and  general methodology of evaluation" (Lingvisticae
Investigationes: Supplementa Volume 15) John Benjamins, Amsterdam.

You should also take a look at the results of the TSNLP project
see <http://cl-www.dfki.uni-sb.de/tsnlp/objective.html>
for a more general take on the use of test suites, and some
potentially very useful resources.

Best

Chris
-- 
------------------------------------------------------------------
Email: Chris.Brew@edinburgh.ac.uk
Address:  Language Technology Group, 
          HCRC, 2 Buccleuch Place,  Edinburgh EH8 9LW 
          Scotland
Telephone: +44 131 650 4631 
Fax:       +44 131 650 4587
------------------------------------------------------------------


