\section{Robust Speech Recognition}

In the context of an autonomous system in a wireless environment, an automatic
speech recognition (ASR) engine is confronted with many challenging
conditions.  These include varying environmental noise levels, bandwidth
limits, varying speaking styles, disfluencies, and speech under stress.  To
obtain acceptable performance, current speech recognition technology must be
adapted to effectively cope with such conditions. This adaptation requires
large amounts of training data that match the working conditions in question.

Until recently most ASR systems have been trained and used under controlled
conditions.  However, when embedded in an immersive environment such as an
autonomous tourist assistant, these limitations are unacceptable since the
system must be easy to set-up and socially acceptable, and should require
little maintenance or attention on the part of the user. In particular, (1)
human users are reluctant to gear up with lapel microphones, let alone
headsets and headgear, or to engage in enrollment phases; (2) users wish to
talk freely without regard to gain settings or start/stop buttons of the
behind-the-scenes recognition technology; (3) environmental noises need to be
filtered out to enhance recognition, but must also be explicitly identified
and tracked in order to model the environment.

We are planning to develop techniques and algorithms that allow for robust
speech recognition in the face of mismatched training and testing conditions
and that build on the robust spoken language technology of the {\sc Janus}
project available at our lab. We propose the following approaches to achieve
the goals of unrestricted open space recognition for multiple languages:

\begin{enumerate}
\item Robust signal enhancement for remote and variable microphone positioning
\item Models of extreme speech, for example, under stress and with 
hyper-articulation
\item Using articulatory features in preparation for
language-independent speech recognition 
\end{enumerate}

\subsection{Robust Signal Enhancement for Remote and Variable Microphone
Positioning}

The key issue in developing accurate recognition of noisy and reverberant
speech is to overcome the mismatch between training and testing
conditions. This problem is usually solved by a separate estimation of
additive and convolutional noise in the spectrum domain, and then a process of
feature compensation or model adaptation through approximation. To this end,
two classes of algorithms have been developed. First is speech signal
enhancement, which can be considered as a form of unsupervised adaptation in
the signal feature domain. This type of algorithm is simple and efficient but
generally makes assumptions that limit their scope of application. Second is
acoustic model adaptation, which is usually accomplished in two ways for
HMM-based modeling. One way is direct adaptation of the HMM parameters. The
other is indirect adaptation of a set of transformation parameters followed by
adaptation of the HMM parameters using these transformations. Both methods are
effective, but are generally slow to adapt or to compute.

Recently, promising new research has been conducted on model adaptation
\cite{Takiguchi01}.  However this approach is computationally costly and 
therefore only applicable for small, low complexity tasks and is not scalable
to LVCSR systems.  In order to overcome these limitations we propose a
combination of signal enhancement and model adaptation algorithms. This goal
can be achieved by extending our Model-combination-based Acoustic Mapping
(MAM) approach. MAM was proven to be very effective for additive noise
\cite{Westphal}, but it only applies a simple cepstral mean normalization for
convolutional noise. However, both kinds of noises interact in a very complex
way on speech features [Pan00] and both are very likely in the \spectrum
scenario (e.g., environmental noises as well as remote microphone conditions,
head movements while speaking, different distances to the microphone
etc.). Therefore, MAM will be extended by using a small set of secondary
acoustic models, which will be used to predict and map the input signal
distortion. Since only the small secondary model is modified to fit the
current signal conditions, this signal compensation method is a rapid on-line
unsupervised model adaptation method.  The large set of core acoustic models
no longer needs to be re-estimated, while the resulting system should be
independent from microphone positioning.  The core acoustic models will be
very robust since they will be trained on various speech databases, like
Broadcast News (various channel conditions), English Spontaneous Scheduling
Task (conversational speech), and Switchboard (telephone speech).


\subsection{Models of Extreme Speech}

To date, considerable progress has been achieved in speech recognition through
techniques such as vocal tract length normalization (VTLN), maximum likelihood
linear regression (MLLR), and speaker adaptive training. However, for highly
accurate dictation applications, it is still necessary to correct errors.  It
is therefore crucial to investigate and understand how users react to
recognition errors.  In previous studies \cite{Oviatt98},\cite{Soltau98}, it
has been shown that humans apply similar recovery strategies in speech
interactions with machines to those that are applied in speech interactions
with other humans.  Speaking style for error correction often becomes more
accentuated in an attempt to more clearly articulate the original mistake.
Unfortunately, contrary to the user's intention, the word error rate increases
significantly under such hyper-articulations.  This well known phenomena is
mostly due to the unmatched speaking style conditions between training and
testing, and to limitations of the current phone based acoustic model
approaches.

We propose to build acoustic models to handle hyper-articulated speech more
robustly and therefore enable our speech components to respond more naturally
to correction situations. We plan to achieve this goal in four steps. In the
first step we will analyze the effects of hyper-articulated speech, i.e.,
which acoustic features (such as speaking rate, pitch contour, formant
frequencies) are effected by hyper-articulation.  In the second step we will
develop a classifier for detecting hyper-articulated speech. To determine
appropriate features for the statistically trained classifier we will take
advantage of the results of the investigation of articulatory features (see
below). In the third step we will train two sets of acoustic models, one for
normal speech, one for hyper-articulated speech (emotion-adaptive
training). In the forth step we plan to do an early integration of the
hyper-articulated effects into the context decision tree by introducing a
``hyper-articulation mode''. This would allow for better parameter sharing,
and thus a very efficient usage of available training data. As a result of
this approach, those acoustic models which are not effected by
hyper-articulation could share all available speech data, and only those
acoustic models which are effected by hyper-articulation would be treated
separately.  This approach had already been applied successfully to handle
dialectal variations
\cite{fuegen}.

The described techniques will be evaluated on various tasks to examine the
robustness against speaking style variations due to stress, emotion,
hyper-articulation as well as the Lombard effect. To carry out the
experiments we will use public available databases like SUSAS (speech under
simulated and actual stress), as well as an in-house collected database which
provides us with parallel data on normal and and hyper-articulated speech.


\subsection{Recognition Based on Articulatory Features and Processes}

Most current speech recognition systems represent speech as a linear sequence
of sound units or phones, with transitions between these units at well-defined
points in time.  During the training phase, the acoustic characteristics of
these phones are learned by aligning the transcription with the speech
waveform, assigning exactly one phone to each part of the utterance
(``beads-on-a-string model'' \cite{Mostendorf}). In reality, these clear-cut
transitions do not exist, as human speech production is a continuous process
due to the nature of our articulatory organs. To compensate for this, a
typical state-of-the-art LVCSR system represents a phone in context of its
neighbors and uses several thousands of these context-dependent units to model
speech.

To improve the system behavior under adverse conditions these models are
adapted to the new conditions. Because these adaptation techniques optimize
the criterion used during the training phase of the recognizer, they usually
improve the performance of the speech engine to a certain extent. However,
these methods do not really lead to robustness against changes in speaking
style (for example, where a realized pronunciation exhibits only certain
articulatory features, which do not correspond to a canonical phoneme).

We therefore propose to model phones no longer as a linear sequence of models,
but as an articulatory feature bundle. Each feature in itself is modeled by a
state sequence, but the transitions do not have to happen time synchronously.
For each point in time the separate probability streams coming from feature
detectors are combined by multiplying their probabilities. This
combination-of-streams approach has already been successfully applied to
noise-robust recognition, where different acoustic models have been trained.
In contrast to systems presented in \cite{KKirchhoff}, our approach
applies this principle to feature detectors and additionally allows the
different streams to segment the data differently.

As a first step, we propose to limit the asynchronicity to one state and
determine the transitions for each stream separately simply by the
optimization of the training criterion (Maximum Likelihood in our case). The
goal is to train feature detectors, to determine suitable features streams as
well as their respective stream weights, and to compare the computational
effort. The completion of this step will show the validity of the concept of
articulatory feature based ASR. We have already conducted initial experiments,
in which we combined six feature streams with a stream that simply consists of
our current 40k vocabulary English single-pass recognizer. The feature
detectors were only trained on a subset of the original training data and they
increased the size of the recognizer from approximately 133k to slightly less
than 140k Gaussian models. The conventional HMM approach has a 14.1\% error
rate and is significantly outperformed by adding the six feature streams,
which has an error rate of 11.8\%. These results were obtained on a 20 minute
in-house test-set comparable to the F0 condition of the Broadcast News
data. Other groups have reported gains on small systems using feature-based
approaches on noisy environments \cite{EEide}.

In a second step, we propose to use the resulting stream weights for speaker
and channel adaptation. We will investigate the effectiveness of existing
adaptation methods with the goal of matching or improving the speech
recognition performance of the proposed articulatory feature stream method on
a broad range of tasks. Training, adapting and testing the systems on
different in-house and external tasks such as Broadcast News and Switchboard
will show whether the proposed method is indeed robust. The proposed system
would enable a much finer modeling, by allowing slow articulators to stay in
place, while requiring fast articulators to make explicit transitions.
Therefore, the confusability with other words should be reduced, improving the
overall performance.
