\section{Robust Speech Recognition}

In the context of a autonomous systems applicable in a wireless environment the speech recognition engine is confronted with many challenging conditions, like varying environmental noise levels, bandwidth limits, and various speaking styles, disfluencies, speech under stress, to name only a few.  To obtain acceptable performance, current speech recognition systems need to be adapted to those conditions in order to reach their best performance. This adaptation requires lots of training data, which matches the working conditions in question. So far these limitations mostly resulted in applications trained and used under controlled conditions. 

For systems in the context of an immersive environment, such as an autonomous tourist assistant, these limitations are unacceptable, since systems should be easy to set-up, socially acceptable and should require little maintenance or attention: 1) Human users would rather not have to gear up with lapel microphones, let alone headsets and headgear, or enrollment phases. 2) Humans will also want to talk freely without regard to gain settings, start/stop buttons of the behind-the-scenes recognition technology. 3) Environmental noises should be filtered out to enhance recognition, but separately also understood and tracked to model the environment.

We are planning to develop techniques and algorithms that allow for robust speech recognition in the face of mismatched conditions and build on the robust spoken language technology of the JANUS project underway at our labs. We propose the following approaches to achieve the goals of unrestricted open space recognition for multiple languages: 

\begin{enumerate}
\item Robust signal enhancement for remote and variable microphone positioning
\item Models of extreme speech (stress, disfluencies)
\item Articulary features (language independent - adress multilingual)
\end{enumerate}

\subsection{Robust signal enhancement for remote and variable microphone positioning}
The key issue to develop accurate recognition of noisy and reverberant speech is to overcome the mismatch between training and testing condition. Additive noises' effect on input speech is added in the spectrum domain; convolutional noises' effect on the input speech is multiplied in the spectral domain. The strategy is to estimate additive and convolutional noises separately in linear and log spectrum domain, and then obtain feature compensation or model adaptation through approximation. To address this two classes of algorithms have been developed. Firstly, speech signal enhancement, which can be considered as a form of unsupervised adaptation in signal feature domain. This type of algorithm is simple and efficient but generally makes some (YUE: WHICH?) assumptions that limit their scope of application. Secondly, acoustic model adaptation. For HMM-based speech recognition system, adaptation is usually accomplished in two ways: direct adaptation of the HMM parameters; or indirect adaptation of a set of transformation parameters and then HMM parameters will be adapted through these transformations. These types of methods are effective, but generally slow to adapt or to compute.

Recently many promising new researches have been conducted on model adaptation \cite{Takiguchi01}. However these approaches are computationlly costly and therefore only applicable for tasks with small-size vocabulary and a low complexity language and not scalable to LVCSR systems. 

To overcome this limitation, we propose a new approach to combine the signal enhancement and model adaptation algorithms together based on MAM (model-combination-based acoustic mapping). Baseline MAM \cite{Westphal01} has proved to be effective for additive noise. However, it only employs a simple cepstral mean normalization for convolutional noise, which is not very effective for remote microphone condition (YUE: OR HEAD MOVEMENTS?). In order to compensate for reverberant speech, new adaptive algorithms need to be developed. Our idea is to use a small size of secondary acoustic model for robust signal enhancement. We can also utilize structure parameters in the secondary model. Based on the secondary model, we can predict and map the input signal distortion. Because it is a signal compensation methods, we avoid modify the large size of acoustic models in the core speech recognition system directly. It also has the advantage of on-line unsupervised model adaptation methods, since we modify the secondary model to fit the current signal condition. This adaptation process is fast since the size of secondary model is small. The resulting system will be independent from microphone positioning. The large size of acoustic models of core baseline clean speech system will be trained on several available databases, namely BN (Broadcast News), ESST (English Spontaneous Speech), and SWB (Telephone Speech) corpus. Therefore it will be able model as many acoustic conditions as possible. Although we can't cover all conditions, this will make the baseline line system very robust and improve the overall speech recognition system performance.  

\subsection{Models of extreme speech}
To date, considerable progress has been achieved in speech recognition through techniques such as vocal tract length normalization (VTLN), maximum likelihood linear regression (MLLR) or speaker adaptive training. However, even in dictation applications with 95\% word accuracy (WA), it is often necessary to correct word errors. Since recognition systems always exhibit some errors, it is important to examine how users react to recognition errors. In previous studies \cite{Oviatt98},\cite{Soltau98}, it has been that shown that humans follow similar recovery strategies as in interaction with humans when they use recognition technology. In correction situations, speaking style becomes more accentuated to disambiguate the original mistake. However, contrary to the users intention, the word accuracy actually decreases significantly in case of hyperarticulation. This is due to unmatched speaking style conditions and the limitations of current phone based acoustic model approaches. 

We propose to focus on building robust acoustics models to handle hyperarticulated speech. We will extend our Janus system to build speech recognizers that react more naturally on extreme speech input. The developed techniques will be evaluated on different tasks to examine the robustness against speaking style variations due to stress, emotion, hyperarticulation and Lombard effect. The experiments will be carried out on different databases, such as the public available SUSAS database (speech under simulated and actual stress). An additional database, collected in our laboratories, allows us to run comparison experiments between normal and hyperarticulated speech.

We propose the following four steps:
\begin{enumerate}
\item Analysis of hyperarticulated effects \newline
In a first step, we plan to examine what kind of acoustic features (such as speaking rate, pitch contour, formant frequencies) exhibit differences at hyperarticulation. For example, the figures show the pitch contour for one word spoken normally (left) and in a hyperarticulated mode (right).
\item Classifier to detect hyperarticulated speech \newline
In a second step, we plan to train statically classifiers to detect hyperarticulation. The features for the classifier will be determined using the results from work package 1.
\item Specialized acoustic models for normal / hyperarticulated speech \newline
After we have a classifier, we will train separate acoustic models for normal and hyperarticulated speech (emotion-adaptive training). To that end, we can use our JANUS speech recognition toolkit.
\item Early integration of hyperarticulated effects into context decision trees (Modalities) \newline
To achieve a better parameter sharing, and to more efficiently use available training data, we will study methods to incorporate a "hyperarticulation mode" already in the context decision tree. The advantage of such a method would be that exactly those acoustic models, which exhibit differences at hyperarticulation, could be treated separately and the phones, which do not need a special treatment, can share all speech data together. The clustering procedure will determine if a model needs a hyperarticulation mode automatically based on a cross-likelihood criterion.
\end{enumerate}

\subsection{Recognition based on articulatory features and processes}
Most current speech recognition systems represent speech as a linear sequence of sound units, the so-called phones, with transitions between these units at well-defined points in time.  During the training phase of a speech recognizer, the acoustic characteristics of these phones are learned by aligning the transcription with the speech waveform, assigning to each part of the utterance exactly one phone ("beads-on-a-string model" \cite{Mostendorf}. In reality, these clear-cut transitions between units do not exist, as human speech production is a continuous process due to the nature of our articulatory organs. To compensate for this, a typical state-of-the-art LVCSR system represents a phone in context of its neighbours and uses several thousands of these context-dependent units to model speech.

A wealth of compensation and adaptation schemes has been developed to improve system behavior under adverse conditions such as background noise, cross-talk, distant microphones, or difficult (narrow-band, distorting) transmission channels (e.g. telephone or video-conferences) by adapting the models to the new conditions. Because these adaptation techniques optimize the criterion used during the training phase of the recognizer, they usually improve the performance of the speech engine to a certain extent. However, these methods do not really lead to robustness against changes in speaking style, e.g. from planned to casual articulation. Initial research on the topic of spontaneous speech has tried to compensate the effects of sloppy pronunciations by expanding the dictionary with specific variations of important entries. However, it is neither easy to generate new entries without increasing the confusability nor is it clear how to balance variability modeling through dictionary entries. Moreover, conventional pronunciation modeling based on phones can not cover effects, where a realized hyper- or under- articulated pronunciation of phone exhibits only certain articulatory features, which do not correspond to a canonical phoneme. 

We therefore propose to model phones not as a linear sequence of models, but as an articulatory feature bundle, where each feature is in itself modeled by a state sequence, but the transitions can occur at different times. For each point in time the separate probability streams coming from the feature detectors are combined by multiplying their probabilities. This combination-of-streams approach has already been successfully applied to noise-robust recognition, where different acoustic models have been trained and it is assumed that they are differently affected by noise. Our approach however applies this principle to feature detectors and additionally allows the different streams to segment the data differently. This asynchronicity distinguishes our system from other work e.g. presented in \cite{KKirchhoff}.

As a first step, we propose to limit the asynchronicity to one state and determine the transitions for each stream separately simply by the optimization of the training criterion (Maximum Likelihood in our case). The goal of this phase is to train feature detectors, determine suitable features streams and their respective stream weights, and to compare the computational effort. The completion of this step will show the validity of the concept of articulatory feature based ASR. We have already conducted initial experiments, in which we combined six feature streams with a stream that simply consists of our current 40k vocabulary English single-pass recognizer. The features detectors were trained on a subset of the original training data only and increased the size of the recognizer from approximately 133k Gaussians to slightly less than 140k Gaussian models. The conventional HMM approach leads to 14.1\% error rate and is significantly outperformed by adding the six feature streams, which achieves 11.8\% error rate. These results were obtained on a 20 minute in-house test-set comparable to the F0 condition of the Broadcast News data. Other groups have reported gains on small systems using feature-based approaches on noisy environments \cite{EEide}.

In a second step, we propose to use these stream weights for speaker and channel adaptation and test the effectiveness of existing adaptation methods with the aim of matching or improving the speech recognition performance of the proposed articulatory feature stream method on a broad range of tasks. Training, adapting and testing the systems on different in-house and external tasks such as Broadcast News, Switchboard, and "Meeting" will establish if the proposed method is indeed robust and works on a number of tasks. The proposed system would allow for much finer modeling, by allowing slow articulators to stay in place (alas skip states) while requiring fast articulators (e.g. the velum) to make explicit transitions, therefore reducing confusability with other words, which would otherwise increases word error rate.

\begin{thebibliography}{10}

\bibitem{MOstendorf} M. Ostendorf: "Moving Beyond The Beads-on-a-string Model Of Speech" Proc. ASRU'99, Keystone, Colorado, USA, 1999.
\bibitem{KKirchhoff} K. Kirchhoff: "Integrating Articulatory Features Into Acoustic Models For Speech Recognition" Proc. Phonus 5, Universitaet des Saarlands, Saarbruecken, Germany, 2001
\bibitem{EEide} E. Eide: "Distinctive Features For Use in an Automatic Speech Recognition System" Proc. EuroSpeech 2001, Aalborg, Danmark, 2001.
\bibitem{Pan00} ask Yue
\bibitem{Oviatt98} S. Oviatt: "The CHAM model of hyperarticulated adaptation during human-computer error resolution" Proc. ICSLP 1998, Sydney, Australia, 1998.	
\bibitem{Soltau98} H. Soltau and A. Waibel: "On the influence of hyperarticulated speech on recognition performance" Proc. ICSLP 1998, Sydney, Australia, 1998.	
\bibitem{Takiguchi01} ask Yue
\bibitem{Westphal1999} M. Westphal, A. Waibel, "Towards Spontaneous Speech Recognition for Onboard Car Navigation and Information Systems", Eurospeech 1999.
\bibitem{Westphal01} ask Yue
\end{thebibliography}

