\mysection{Testing and Evaluation}
\label{eval}


Testing and evaluation will focus on robustness, scalability and
portability, as well as impact on users for showcases and
multimodality.  Tests will be conducted on unseen data collected as
described in Section~\ref{data}.  Testing and Evaluation corresponds
to Workpackage~7 in the European proposal.  Carnegie Mellon will be
the leader of this workpackage.

%% \noindent
{\bf Task-Based Evaluation:} This involves identifying the users'
goals in a dialogue and giving the dialogue a score which is a
function of whether each goal succeeded or failed and how many turns
were spent on repairing wrong translations. (The method for deciding
what counts as a goal and the exact nature of the scoring function are
topics for research \cite{walker:97}.)  CMU, with the help of APT and
AETHRA, will conduct several task-based evaluations in Years~2 and~3.

%% \noindent
{\bf Sentence-Based Evaluation of Coverage and Accuracy:} For each
sentence, the source and target are compared by a bilingual human
judge. We evaluate the translation components alone (using
hand-transcribed input), and also evaluate end-to-end performance
using speech-recognition output as the input to the translation
components.  CMU will conduct sentence-based evaluations monthly on
unseen data starting in Year~1.  Results will be plotted over time.

%% \noindent
{\bf Evaluation of individual components:} Analyzers and generators
can be evaluated individually using hand-coded perfect output as a
gold-standard.  Speech recognition can be evaluated using standard
measures such as word error rate.  Multimodality can be evaluated by
comparing the system performances against testbeds obtained from the
data collected during the experimental phase described in
Section~\ref{multimodal}.  Individual components will be evaluated on
an as-needed basis (e.g., for error analysis) by CMU, IRST, CLIPS, and
UKA.

%% \noindent
{\bf Evaluating scalability and portability:} To evaluate scalability
and portability, we will plot coverage and accuracy as a function of
the amount of resources. For knowledge- and IF-based STST systems,
training data, grammars, lexica, and time spent by human developers
will be plotted monthly along with the sentence-based evaluations.
For Example-Based STST, glossary-based STST, and statistical methods,
experiments plotting the amounts of training data, glossaries, etc.,
against coverage and accuracy will be conducted in Year~2.

%% \noindent
{\bf Evaluation of Multi-Engine STST:} We will compare the performance
(probably sentence-based, not task-based) of the multi-engine STST
system to the performance of individual engines. We will also remove
individual engines from the multi-engine system to see how performance
degrades.  This evaluation will take place at CMU in Year~2.

%% \noindent
{\bf Showcases evaluation:} Showcases will be evaluated with respect
to standard measures of usability in the area of human-computer
interaction. Given the demonstration nature of the applications,
subjective measurements will be preferred to objective ones.  From the
beginning of the project, focus groups will be set up and standard
techniques of ``discount usability engineering'' will be applied to
ensure that showcases meet user expectations and needs.  IRST, with
the help of APT and AETHRA, will be responsible for showcase
evaluations.

