\section{Dialogue Monitoring and Modality Management}

In this section we describe our plans for a communication manager
that detects communication problems and deploys recovery strategies.
The communication manager has two main parts, dialogue monitoring and
fall-back management of the modalities (e.g., speech, text messages,
menus) available to the system.  The goal of dialogue monitoring is to
reliably detect failures in the communication and their causes.  The
goal of modality management is to actively assist the user in
overcoming communication difficulties by using alternative modalities.

\subsection{Dialogue Monitoring}

%% Negative results from previous research.
%% Diane:  Did AT+T have any more luck with this than the rest of 
%% us did?

The Enthusiast, Artwork, and Verbmobil projects
\cite{Qu97}, \cite{Qu96}, \cite{rose95}, \cite{reithinger95}
have studied the use of discourse 
information for resolution of ambiguity and improvement of quality of SLT in
task oriented dialogues.  For example, if a plan inference tree of dialogue
goals were available, could it be used to disambiguate the speech act of an
incoming utterance and thereby increase translation accuracy?  These projects
experimented with a variety of techniques including plan inference, finite
state models of speech act sequences, and n-gram models of speech act
sequences.  However, the results were discouraging.  Even with reasonably high
accuracy -- for example, around 70\% correct detection of speech acts -- the
improvement in translation quality was negligible.  Similarly discouraging
results were obtained at the 1997 Johns Hopkins Summer Workshop 
\cite{workshop97} regarding the impact of speech act detection on word
accuracy for speech recognition in non-task-oriented Switchboard dialogues.
\cite{Rose98} showed mildly positive results for using discourse knowledge in
interactive repair.

%% Add Reasons for negative results.
%% The new approach.

While the above results appear to discourage the general utility of attempting
to track the progression of the dialogue for improving translation quality, we
believe that contextual information (such as speech-acts and domain-actions of
previous utterances) can still play an important role in identifying concrete
points within the dialogue, particularly communication failures.  In
\spectrum we plan to focus our attention specifically on monitoring the
dialogue for communication failures.  Discourse knowledge will play only one
part in a suite of confidence measures that will be designed to detect
communication trouble spots. The job of repair will then be handed over to the
Modality Manager.  We plan to develop and integrate a suite of strategies for
detection of translation failure and communication breakdown.  These will
include:

\begin{itemize}
\item {\bf Effective Translation Confidence Measures:} We will develop new
measures for evaluating the confidence in the quality of translation outputs 
of our specific translation engines.  The focus will be on our
interlingua-based engine.  Features that will be taken into account include
the confidence score from the speech recognizer, parser information about
words that were not incorporated into the analysis, the ``distance'' of the
produced paraphrase from the original input, and a new measure for rating
the fitness of produced interlingua representations.  The improved confidence
measures associated with the different translation engines will also be used
to improve the multi-engine integration.

\item {\bf Explicit Detection of Communication Failure:} We will monitor the
progression of the dialogue with the goal of explicitly detecting points of
communication failure.  Our goal will be to find reliable characteristics of
points of communication failure based on a variety of indicators.  These will
include: (1) speech recognition and translation confidence measures for the
current and previous turns; (2) specific domain-actions, and sequences of
domain-actions, that may indicate communication failure; (3) prosody and
intonation features.  We will also attempt to characterize the cause of the
failure: poor speech recognition, poor translation, deviation from the domain, 
etc.  This information will then be passed along to the modality manager.
\end{itemize}

\subsection{Modality Manager}

The Modality Manager will have the task of actively assisting the user in
overcoming communication failures once a failure and the cause of the failure
are identified.  While constrained by the limitations imposed by our mobile
platform, we envision the system will have at least the following modalities
at its disposal: (1) speech translation activated by voice; (2) menus of
common ``phrase-book'' entries for translation; (3) ability to enter short
text messages for translation; and (4) transmission of simple annotated hand
drawings.  

The modality manager will take an active role in suggesting to the
user when to switch from the primary modality of speech translation
activated by voice to any of the other modalities available.  For
example, if we detect that the cause of the failure is poor speech
recognition, text message translation will be proposed as an
alternative.  If on the other hand speech input appears to be
reliable, but the main translation approach is failing, the system can
present a menu of phrase-book entries based on the speech input and
allow the user to select an appropriate entry for translation.  User
studies will play a large role in the investigation of how the
modality manager can effectively assist the user without being
intrusive.

Previous work within the framework of human-machine dialogue systems
has shown positive results using such active dialogue management.  In
a spoken dialogue system for accessing online train schedules, an
adaptive version of the dialogue manager was developed that monitored
ASR confidence scores to incrementally predict whether a user was
having problems as the dialogue progressed.  The system adapted to a
more conservative set of dialogue strategies whenever the predictions
classified the dialogue as problematic.  An experimental evaluation
demonstrated that changing the behavior of the dialogue manager based
on monitoring acoustic confidence scores significantly increased the
task success rate \cite{LP00}.  In the context of the {\sc Janus}
project we have also conducted experiments on multi-modal repair of
speech recognition \cite{Suhm}.  

Based on our recent experience, we believe that active management of
repair modalities is also important for machine translation.
In April 2001, at the conclusion of the Tongues project
\cite{tongues-webpage}, we carried out a field-test of a SLT system running
on a laptop, with regular Army officers and naive Croatian civilians
as users.  While the detailed analysis of the test results is still
underway, our informal observation was that user interaction was
clearly the weakest link.  While ASR and TTS were certainly not
perfect, the majority of the failures to complete tasks successfully
were clearly the result of difficulties the monolingual English
speakers had in assessing whether the MT component had produced
correct translations in the target language.  Assessing the quality of
the translation involved gestures and menus.  A team member who was
bilingual observed that often the English-to-Croatian translation was
reasonably correct, but the English-speaker would abort the
translation because the available feedback suggested that the
translation was not adequate.


