\mysection{Evaluation}

%% This paragraph could be deleted if it is too obvious or too much
%% like mom-and-apple-pie filler.

In spite of the growing number of commercial and research-oriented MT
systems, evaluation of machine translation is still a topic of much
disagreement.  The confusion stems in part from the diversity of
machine translation applications and the multi-faceted nature of the
research.  Some evaluation methods are relevant for the user, whereas
others are relevant for the system developer.  Some are relevant when
high quality is required, whereas others are relevant when gisting is
satisfactory.  Evaluation by humans is costly, whereas automatic
evaluation may not accurately reflect human judgement.  

In the course of our research we have developed a suite of evaluation
methods that we find informative including end-to-end evaluations of
translation accuracy on a sentence-by-sentence basis; plots of
accuracy as a function of development time; plots of accuracy as a
function of the amount of development data; plots of coverage as a
function of training data (for corpus-based methods); and evaluation
of individual system components against hand-coded gold-standard
output.

Our evaluation methods have been largely accuracy-based
\cite{gates97}, focusing on whether the meaning
of a source language segment is totally and accurately conveyed in the
target language.  This type of evaluation is useful for tracking our
improvement over time.  The measure we use is percent of sentences
that are accurate (we call these {\tt acceptable}) and the percent
that are both accurate and fluent (we call these {\tt perfect}).

Accuracy Based Evaluations (ABE) continue to be useful.  However, the
ability of a user to complete a task (for example, getting a plane
reservation) is higher than would be expected based on an ABE.  For
example, the ABE might be around 70\% acceptable, but the users could
almost always complete the task.  Recently we have begun to design a
Task Based Evaluation (TBE) for speech-to-speech translation
\cite{kavita,lrec00} that measures goal completion.  Goals can be
identified at the dialogue level (e.g., did the user get a hotel room
by the end of the dialogue?) or at the utterance level (e.g., Was the
utterance {\it How much is a double room\/} translated accurately
enough for the interlocutor understand that it was a question about
price and that it concerned a double room?)

Most previous work on TBE has been conducted on human-machine dialogue
(for example \cite{walker97}).  For machine translation, we need a TBE
that is suitable for two humans each expressing communicative goals,
but mediated by a machine. In particular, we have to separate human
clumsiness and error from machine error, because we are not evaluating
the humans, but rather the translation of what they said.  For
example, if one interlocutor says {\it How much does it cost?\/}, and
it is correctly translated, but the other interlocutor gets distracted
and doesn't answer, that should not count as a malfunction of the
system. Additionally, we have to allow for a large and unpredictable
number of communicative goals in each dialogue.  For example,
following the coding scheme we developed for~\ref{lrec00}, the
dialogues we are evaluating each contain over one hundred
communicative goals.  This is in contrast to the types of dialogues
reported in ... that have only ... 

Communicator -- suite of techniques
   efficiency -- length of dialogue
   speech accuracy
   weighting function learned from 
   user satisfaction 



%% Our travel agents knew the scenario.

%% The research issues are:
%%% defining what a communicative goal is

%% Who does the grading?
%%% System developers, if they are allowed to see the test data
%%% Otherwise, college students with a few hours of training
%% How is the intercoder agreement between graders?
%%% Usually less than 10% disagreement.
%% Do we have statistically significant results?  
%%%  No.  It's too expensive. 
%% What is our percent acceptable for C-STAR II and NESPOLE?
%%%  C-STAR II -- around 65% acceptable for speech input,
%%%  75%? acceptable for transcribed input
%%%  NESPOLE results will be available on Friday



