\section{Robust Speech Recognition}

In the context of a autonomous systems applicable in a wireless
environment the speech recognition engine is confronted with many
challenging conditions, such as varying environmental noise levels,
bandwidth limits, varying speaking styles, disfluencies, and speech
under stress, to name only a few.  To obtain acceptable performance,
current speech recognition systems need to be adapted to those
conditions. This adaptation requires large amounts of training data
that match the working conditions in question.

Until recently most ASR systems have been trained and used under
controlled conditions.  However, in the context of an immersive
environment such as an autonomous tourist assistant, these limitations
are unacceptable since systems should be easy to set-up and socially
acceptable, and should require little maintenance or attention. In
particular, 1) human users would rather not have to gear up with lapel
microphones, let alone headsets and headgear, or enrollment phases; 2)
humans will also want to talk freely without regard to gain settings or
start/stop buttons of the behind-the-scenes recognition technology; 3)
environmental noises should be filtered out to enhance recognition,
but also separately understood and tracked to model the environment.

We are planning to develop techniques and algorithms that allow for
robust speech recognition in the face of mismatched training and
testing conditions and build on the robust spoken language technology
of the {\sc Janus} project underway at our lab. We propose the
following approaches to achieve the goals of unrestricted open space
recognition for multiple languages:

\begin{enumerate}
\item Robust signal enhancement for remote and variable microphone positioning
\item Models of extreme speech, for example, under stress and with 
hyperarticulation
\item Using articulatory features in preparation for
language-independent speech recognition 
\end{enumerate}

\subsection{Robust Signal Enhancement for Remote and Variable Microphone Positioning}

The key issue in developing accurate recognition of noisy and
reverberant speech is to overcome the mismatch between training and
testing conditions. This problem is usually solved by the separate 
estimation of additive and convolutional noise in the linear and log 
spectrum domain, and then obtain feature compensation or model
adaptation through approximation. To address this, two classes of
algorithms have been developed. First is speech signal enhancement,
which can be considered as a form of unsupervised adaptation in the
signal feature domain. This type of algorithm is simple and efficient
but generally makes assumptions that limit their scope of
application. Second is acoustic model adaptation, which is usually
accomplished in two ways.  One way is direct adaptation of the HMM
parameters.  The other is indirect adaptation of a set of
transformation parameters followed by adaptation of the HMM parameters
using these transformations. Both methods are effective, but generally
slow to adapt or to compute.

Recently much promising new research has been conducted on model adaptation
\cite{Takiguchi01}. However these approaches are
computationlly costly and therefore only applicable for small, low
complexity tasks and are not scalable to LVCSR systems.

To overcome this limitation, we propose a new approach to combine the signal
enhancement and model adaptation algorithms together based on MAM
(model-combination-based acoustic mapping). Baseline MAM \cite{Westphal01} has
proved to be effective for additive noise. However, it only employs a simple
cepstral mean normalization for convolutional noise, which is not very
effective for remote microphone condition (YUE: OR HEAD MOVEMENTS?). In order
to compensate for reverberant speech, new adaptive algorithms need to be
developed. Our idea is to use a small size of secondary acoustic model for
robust signal enhancement. We can also utilize structure parameters in the
secondary model. Based on the secondary model, we can predict and map the
input signal distortion. Because it is a signal compensation methods, we avoid
modify the large size of acoustic models in the core speech recognition system
directly. It also has the advantage of on-line unsupervised model adaptation
methods, since we modify the secondary model to fit the current signal
condition. This adaptation process is fast since the size of secondary model
is small. The resulting system will be independent from microphone
positioning. The large size of acoustic models of core baseline clean speech
system will be trained on several available databases, namely BN (Broadcast
News), ESST (English Spontaneous Speech), and SWB (Telephone Speech)
corpus. Therefore it will be able model as many acoustic conditions as
possible. Although we can't cover all conditions, this will make the baseline
line system very robust and improve the overall speech recognition system
performance.

\subsection{Models of Extreme Speech}

To date, considerable progress has been achieved in speech recognition through
techniques such as vocal tract length normalization (VTLN), maximum likelihood
linear regression (MLLR) or speaker adaptive training. However, even in
dictation applications with 95\% word accuracy (WA), it is often necessary to
correct word errors. Since recognition systems always exhibit some errors, it
is important to examine how users react to recognition errors. In previous
studies \cite{Oviatt98},\cite{Soltau98}, it has been that shown that humans
follow similar recovery strategies as in interaction with humans when they use
recognition technology. In correction situations, speaking style becomes more
accentuated to disambiguate the original mistake. However, contrary to the
users intention, the word accuracy actually decreases significantly in case of
hyperarticulation. This is due to unmatched speaking style conditions and the
limitations of current phone based acoustic model approaches.

We propose to focus on building robust acoustics models to handle
hyperarticulated speech. We will extend our Janus system to build speech
recognizers that react more naturally on extreme speech input. The developed
techniques will be evaluated on different tasks to examine the robustness
against speaking style variations due to stress, emotion, hyperarticulation
and Lombard effect. The experiments will be carried out on different
databases, such as the public available SUSAS database (speech under simulated
and actual stress). An additional database, collected in our laboratories,
allows us to run comparison experiments between normal and hyperarticulated
speech.

We propose the following four steps:
\begin{enumerate}
\item Analysis of hyperarticulated effects \newline
In a first step, we plan to examine what kind of acoustic features (such as
speaking rate, pitch contour, formant frequencies) exhibit differences at
hyperarticulation. For example, the figures show the pitch contour for one
word spoken normally (left) and in a hyperarticulated mode (right).
\item Classifier to detect hyperarticulated speech \newline
In a second step, we plan to train statically classifiers to detect
hyperarticulation. The features for the classifier will be determined using
the results from work package 1.
\item Specialized acoustic models for normal / hyperarticulated speech \newline
After we have a classifier, we will train separate acoustic models for normal
and hyperarticulated speech (emotion-adaptive training). To that end, we can
use our JANUS speech recognition toolkit.
\item Early integration of hyperarticulated effects into context decision 
trees (Modalities) \newline 
To achieve a better parameter sharing, and to more
efficiently use available training data, we will study methods to incorporate
a ``hyperarticulation mode'' already in the context decision tree. The
advantage of such a method would be that exactly those acoustic models, which
exhibit differences at hyperarticulation, could be treated separately and the
phones, which do not need a special treatment, can share all speech data
together. The clustering procedure will determine if a model needs a
hyperarticulation mode automatically based on a cross-likelihood criterion.
\end{enumerate}

\subsection{Recognition Based on Articulatory Features and Processes}

Most current speech recognition systems represent speech as a linear sequence
of sound units, the so-called phones, with transitions between these units at
well-defined points in time.  During the training phase of a speech
recognizer, the acoustic characteristics of these phones are learned by
aligning the transcription with the speech waveform, assigning to each part of
the utterance exactly one phone ("beads-on-a-string model"
\cite{Mostendorf}. In reality, these clear-cut transitions between units do
not exist, as human speech production is a continuous process due to the
nature of our articulatory organs. To compensate for this, a typical
state-of-the-art LVCSR system represents a phone in context of its neighbours
and uses several thousands of these context-dependent units to model speech.

A wealth of compensation and adaptation schemes has been developed to improve
system behavior under adverse conditions such as background noise, cross-talk,
distant microphones, or difficult (narrow-band, distorting) transmission
channels (e.g. telephone or video-conferences) by adapting the models to the
new conditions. Because these adaptation techniques optimize the criterion
used during the training phase of the recognizer, they usually improve the
performance of the speech engine to a certain extent. However, these methods
do not really lead to robustness against changes in speaking style, e.g. from
planned to casual articulation. Initial research on the topic of spontaneous
speech has tried to compensate the effects of sloppy pronunciations by
expanding the dictionary with specific variations of important
entries. However, it is neither easy to generate new entries without
increasing the confusability nor is it clear how to balance variability
modeling through dictionary entries. Moreover, conventional pronunciation
modeling based on phones can not cover effects, where a realized hyper- or
under- articulated pronunciation of phone exhibits only certain articulatory
features, which do not correspond to a canonical phoneme.

We therefore propose to model phones not as a linear sequence of models, but
as an articulatory feature bundle, where each feature is in itself modeled by
a state sequence, but the transitions can occur at different times. For each
point in time the separate probability streams coming from the feature
detectors are combined by multiplying their probabilities. This
combination-of-streams approach has already been successfully applied to
noise-robust recognition, where different acoustic models have been trained
and it is assumed that they are differently affected by noise. Our approach
however applies this principle to feature detectors and additionally allows
the different streams to segment the data differently. This asynchronicity
distinguishes our system from other work e.g. presented in \cite{KKirchhoff}.

As a first step, we propose to limit the asynchronicity to one state and
determine the transitions for each stream separately simply by the
optimization of the training criterion (Maximum Likelihood in our case). The
goal of this phase is to train feature detectors, determine suitable features
streams and their respective stream weights, and to compare the computational
effort. The completion of this step will show the validity of the concept of
articulatory feature based ASR. We have already conducted initial experiments,
in which we combined six feature streams with a stream that simply consists of
our current 40k vocabulary English single-pass recognizer. The features
detectors were trained on a subset of the original training data only and
increased the size of the recognizer from approximately 133k Gaussians to
slightly less than 140k Gaussian models. The conventional HMM approach leads
to 14.1\% error rate and is significantly outperformed by adding the six
feature streams, which achieves 11.8\% error rate. These results were obtained
on a 20 minute in-house test-set comparable to the F0 condition of the
Broadcast News data. Other groups have reported gains on small systems using
feature-based approaches on noisy environments \cite{EEide}.

In a second step, we propose to use these stream weights for speaker and
channel adaptation and test the effectiveness of existing adaptation methods
with the aim of matching or improving the speech recognition performance of
the proposed articulatory feature stream method on a broad range of
tasks. Training, adapting and testing the systems on different in-house and
external tasks such as Broadcast News, Switchboard, and "Meeting" will
establish if the proposed method is indeed robust and works on a number of
tasks. The proposed system would allow for much finer modeling, by allowing
slow articulators to stay in place (alas skip states) while requiring fast
articulators (e.g. the velum) to make explicit transitions, therefore reducing
confusability with other words, which would otherwise increases word error
rate.

