\mysection{Evaluation}

In spite of the growing number of commercial and research-oriented MT
systems, evaluation of machine translation is still a topic of much
disagreement.  The confusion stems in part from the diversity of
machine translation applications and the multi-faceted nature of the
research.  Some evaluation methods are relevant for the user, whereas
others are relevant for the system developer.  Some are relevant when
high quality is required, whereas others are relevant when gisting is
satisfactory.  Evaluation by humans is costly, whereas automatic
evaluation may not accurately reflect human judgement.  

In the course of our research we have developed a suite of evaluation
methods that we find informative including end-to-end evaluations of
translation accuracy on a sentence-by-sentence basis; plots of
accuracy as a function of development time; plots of accuracy as a
function of the amount of development data; plots of coverage as a
function of training data (for corpus-based methods); and evaluation
of individual system components against hand-coded gold-standard
output.

Our evaluation methods have been largely accuracy-based
\cite{gates97}, focusing on whether the meaning
of a source language segment is totally and accurately conveyed in the
target language.  This type of evaluation is useful for tracking our
improvement over time.  The measure we use is percent of sentences
that are accurate (we call these {\tt acceptable}) and the percent
that are both accurate and fluent (we call these {\tt perfect}).

Accuracy Based Evaluations (ABE) continue to be useful.  However, the
ability of a user to complete a task (for example, getting a plane
reservation) is higher than would be expected based on an ABE.  For
example, the ABE might be around 70\% acceptable, but the users could
almost always complete the task.  Recently we have begun to design a
Task Based Evaluation (TBE) for speech-to-speech translation
\cite{kavita}, \cite{lrec00} that measures goal completion.  Goals can be
identified at the dialogue level (e.g., did the user get a hotel room
by the end of the dialogue?) or at the utterance level (e.g., Was the
utterance {\it How much is a double room\/} translated accurately
enough for the interlocutor to understand that it was a question about
price and that it concerned a double room?).  

However, the correct notion of goal completion is difficult to pin
down.  First we must decide what counts as a goal.  For example,
suppose the question {\it How much does a double room cost?\/} is
translated as {\it How much?\/}.  Has a goal of requesting information
about some quantity been met?  Or suppose it is translated as {\it A
double room costs\/}.  Has a goal of talking about the price of a
double room been met? Or has the communicative goal failed completely?
Furthermore, utterances can be rephrased and repeated in the face of
communicative failure.  The repetitions and rephrased sentences should
not count as new communicative goals, but as multiple attempts to
achieve the same communicative goal.  In \cite{lrec00} we formulated a
scoring function for communicative goal completion that took into
account whether each communicative goal ultimately succeeded or failed
and how many times it was attempted before succeeding or failing.
Of course, intercoder agreement on new versus repeated goals then becomes
an issue.

In addition to translation accuracy and goal completion, there
are likely many other factors that are useful for measuring
performance, such as dialogue length.  In human-machine spoken
dialogues, the PARADISE evaluation framework has been successfully
used to understand the relative contribution of a wide variety of such
factors (representing performance dimensions such as task completion,
dialogue quality, dialogue efficiency, and usability) to overall
performance, across many different spoken dialogue systems
\cite{walker97}, \cite{Walkeretal98}, \cite{WKL00}, \cite{Walker01}.
PARADISE models performance as a weighted function of a task-based
success measure and dialogue-based cost measures, where weights are
computed from experimental data by correlating a meaningful external
criterion of usability with performance.

To date, most approaches that go beyond accuracy-based evaluations
have been conducted in the context of human-machine spoken dialogue
systems.  For machine translation, we need an evaluation paradigm that
is suitable for two humans each expressing communicative goals, but
mediated by a machine.  Additionally, we have to allow for a large and
unpredictable number of communicative goals in each dialogue.  For
example, following the coding scheme we developed for~\cite{lrec00},
the dialogues we are evaluating each contain over one hundred
communicative goals.  Our new work on performance evaluation will
extend previous work on task-based and usability-based approaches for
human-machine dialogue, to the context of a computer-mediated
human-human spoken translation system.






