\section{Robust Speech Recognition}

In the context of a autonomous systems applicable in a wireless
environment the speech recognition engine is confronted with many
challenging conditions, such as varying environmental noise levels,
bandwidth limits, varying speaking styles, disfluencies, and speech
under stress, to name only a few.  To obtain acceptable performance,
current speech recognition systems need to be adapted to those
conditions. This adaptation requires large amounts of training data
that match the working conditions in question.

Until recently most ASR systems have been trained and used under
controlled conditions.  However, in the context of an immersive
environment such as an autonomous tourist assistant, these limitations
are unacceptable since systems should be easy to set-up and socially
acceptable, and should require little maintenance or attention. In
particular, 1) human users would rather not have to gear up with lapel
microphones, let alone headsets and headgear, or enrollment phases; 2)
humans will also want to talk freely without regard to gain settings or
start/stop buttons of the behind-the-scenes recognition technology; 3)
environmental noises should be filtered out to enhance recognition,
but also separately understood and tracked to model the environment.

We are planning to develop techniques and algorithms that allow for
robust speech recognition in the face of mismatched training and
testing conditions and build on the robust spoken language technology
of the {\sc Janus} project underway at our lab. We propose the
following approaches to achieve the goals of unrestricted open space
recognition for multiple languages:

\begin{enumerate}
\item Robust signal enhancement for remote and variable microphone positioning
\item Models of extreme speech, for example, under stress and with 
hyperarticulation
\item Using articulatory features in preparation for
language-independent speech recognition 
\end{enumerate}

\subsection{Robust Signal Enhancement for Remote and Variable Microphone Positioning}

The key issue in developing accurate recognition of noisy and
reverberant speech is to overcome the mismatch between training and
testing conditions. This problem is usually solved by a separate 
estimation of additive and convolutional noise in the
spectrum domain, and then a feature compensation or model
adaptation through approximation. To obtain this, two classes of
algorithms have been developed. First is speech signal enhancement,
which can be considered as a form of unsupervised adaptation in the
signal feature domain. This type of algorithm is simple and efficient
but generally makes assumptions that limit their scope of
application. Second is acoustic model adaptation, which is usually
accomplished in two ways for HMM-based modeling. One way is direct 
adaptation of the HMM parameters. The other is indirect adaptation of a set of
transformation parameters followed by adaptation of the HMM parameters
using these transformations. Both methods are effective, but generally
slow to adapt or to compute.

Recently much promising new research has been conducted on model adaptation
\cite{Takiguchi01}. However the approaches are
computationally costly and therefore only applicable for small, low
complexity tasks and are not scalable to LVCSR systems.
In order to overcome these limitations we propose the combination of
the signal enhancement and the model adaptation algorithms. This goal can
be achieved by extending our Model-combination-based Acoustic Mapping 
(MAM) approach. MAM was proven to be very effective for additive noise 
\cite{Westphal} but it only applies a simple cepstral mean normalization 
for convolutional noise. However, both kinds of noises interact in a very 
complex way on speech features [Pan00] and are both very likely in the 
Spectrum scenario (e.g. environmental noises as well as remote microphone
conditions, head movements while speaking, different distances to the 
microphone etc.). Therefore, MAM will be extended by using a small set of 
secondary acoustic models, which are used to predict and map the input 
signal distortion. Since it is only the small secondary model which is 
modified to fit the current signal conditions, this signal compensation 
method has the advantage of rapid on-line unsupervised model adaptation 
methods. 
The large set of core acoustic models needs no longer to be reestimated,
but the resulting system will be independent from microphone positioning. 
The core acoustic models will be very robust since they will be trained on
various speech databases, like Broadcast News (various channel conditions), 
English Spontaneous Scheduling Task (conversational speech), and 
Switchboard (telephone speech).


\subsection{Models of Extreme Speech}
To date, considerable progress has been achieved in speech recognition through
techniques such as vocal tract length normalization (VTLN), maximum likelihood
linear regression (MLLR), and speaker adaptive training. However, even given
highly accurate dictation applications it is still necessary to correct errors. 
It is therefore crucial to investigate and understand how users react to 
recognition errors. In previous studies \cite{Oviatt98},\cite{Soltau98}, it has 
been shown that human beings apply similar recovery strategies independent from 
whether they interact with other humans or whether they use speech technologies 
to interact with a machine. Most often the speaking style for error correction
becomes more accentuated to disambiguate the original mistake. 
Unfortunately, in contrast to the users intention, the word error rate increases 
significantly in case of these hyperarticulations. 
This well known fact is mostly due to unmatched speaking style conditions for 
training and test, and limitations of the current phone based acoustic model 
approaches.

We propose to build acoustic models to handle hyperarticulated speech more
robustly and therefore enable our speech components to respond more
naturally to correction situations. We are planning to achieve this goal in
four steps. In the first step we will analysis the effects of hyperarticulated 
speech, i.e. which acoustic features (like speaking rate, pitch contour, 
formant frequencies) are effected by hyperarticulation. 
In the second step we will develop a classifier to detect hyperarticulated
speech. To determine appropriate features for the statistically trained classifier 
we will benefit from the results of the investigation of articulatory features
(see below). In the third step we will train two sets of acoustic models, one for
normal speech, one for hyperarticulated speech (emotion-adaptive training). In the
forth step we are planning an early integration of the hyperarticulated effects into 
the context decision tree by introducing a ``hyperarticulation mode''. This would 
allow for a better parameter sharing, and thus a very efficient usage of available 
training data. As a result of this approach those acoustic models which are not
effected by hyperarticulation could share all available speech data, and only those
acoustic models, which are effected by hyperarticulation would be treated separately.
This approach had already been applied successfully to handle dialectal variations 
\cite{fuegen}.

The described techniques will be evaluated on various tasks to examine the robustness
against speaking style variations due to stress, emotion, hyperarticulation as well as
to the Lombard effect. To carry out the experiments we will use public available databases 
like SUSAS (speech under simulated and actual stress), as well as an in-house collected
database which provides us with parallel data on normal and and hyperarticulated speech.


\subsection{Recognition Based on Articulatory Features and Processes}
Most current speech recognition systems represent speech as a linear sequence
of sound units or phones, with transitions between these units at
well-defined points in time.  During the training phase the acoustic 
characteristics of these phones are learned by aligning the transcription 
with the speech waveform, assigning exactly one phone
to each part of the utterance ("beads-on-a-string model"
\cite{Mostendorf}. In reality, these clear-cut transitions do not exist, 
as human speech production is a continuous process due to the
nature of our articulatory organs. To compensate for this, a typical
state-of-the-art LVCSR system represents a phone in context of its neighbors
and uses several thousands of these context-dependent units to model speech.

To improve the system behavior under adverse conditions these models are adapted 
to the new conditions. Because these adaptation techniques optimize the criterion
used during the training phase of the recognizer, they usually improve the
performance of the speech engine to a certain extent. However, these methods
do not really lead to robustness against changes in speaking style for example
where a realized pronunciation exhibits only certain articulatory
features, which do not correspond to a canonical phoneme.

We therefore propose to model phones no longer as a linear sequence of models, but
as an articulatory feature bundle. Each feature in itself is modeled by
a state sequence, but the transitions do not have to happen time synchronously. 
For each point in time the separate probability streams coming from feature
detectors are combined by multiplying their probabilities. This
combination-of-streams approach has already been successfully applied to
noise-robust recognition, where different acoustic models have been trained.
In contrast to systems presented e.g. in \cite{KKirchhoff} our approach 
applies this principle to feature detectors and additionally allows
the different streams to segment the data differently.

As a first step, we propose to limit the asynchronicity to one state and
determine the transitions for each stream separately simply by the
optimization of the training criterion (Maximum Likelihood in our case). The
goal is to train feature detectors, to determine suitable features streams 
as well as their respective stream weights, and to compare the computational
effort. The completion of this step will show the validity of the concept of
articulatory feature based ASR. We have already conducted initial experiments,
in which we combined six feature streams with a stream that simply consists of
our current 40k vocabulary English single-pass recognizer. The features
detectors were trained on a subset of the original training data only and
increased the size of the recognizer from approximately 133k Gaussians to
slightly less than 140k Gaussian models. The conventional HMM approach leads
to 14.1\% error rate and is significantly outperformed by adding the six
feature streams, which achieves 11.8\% error rate. These results were obtained
on a 20 minute in-house test-set comparable to the F0 condition of the
Broadcast News data. Other groups have reported gains on small systems using
feature-based approaches on noisy environments \cite{EEide}.

In a second step, we propose to use the resulting stream weights for speaker and
channel adaptation. We will investigate the effectiveness of existing adaptation methods
with the aim of matching or improving the speech recognition performance of
the proposed articulatory feature stream method on a broad range of
tasks. Training, adapting and testing the systems on different in-house and
external tasks such as Broadcast News and Switchboard will show whether the proposed 
method is indeed robust. The proposed system would enable a much finer modeling, 
by allowing slow articulators to stay in place while requiring fast articulators 
to make explicit transitions. Therefore, the confusability with other words could be
reduced, improving the overall performance.

