\section{Robust Speech Recognition}

Current speech recognizers need to be adapted to channel conditions, speaker
characteristics, and speaking style in order to reach their best
performance. Current approaches adapt the acoustic models to data mismatched
with respect to all three criteria, usually by pursuing a Maximum Likelihood
approach. However the adaptation is only reasonable under the assumption that
enough data might be available for the condition in question.

In the context of a autonomous system applicable in various environment under 
many situations we adress mainly three problems to overcome this: (1) Robust 
recognition using articulatory features; (2) Robust signal enhancement; and
(3) Discriminative and adaptive training on limited data.


As a basis for this research we will build upon work already done under
several NSF programs including Nespole.  We expect to be able to develop
techniques and algorithms that allow for robust speech recognition in the face
of mismatched conditions such as disfluencies, noise, stress speech, etc. We
will build on the robust spoken language technology of the JANUS project
underway at our labs.

While we have already been working on the general problem of conversational
speech over telephone and in meeting places, several basic limitations still
remain: To obtain acceptable performance, virtually all current speech
recognition and sound processing systems require pure environments.  They are
unable work well outside of environments in which they were trained.  In the
face of mismatched conditions such as disfluencies, noise, stress speech, or
even different microphones; these systems suffer a performance collapse. For
autonomous recognition in the context of an immersive environment, such as an
autonomous tourist assistant, these limitations are unacceptable, since
systems should be easy to set-up, socially acceptable and should require
little maintenance or attention: 1) Human users would rather not have to gear
up with lapel microphones, let alone headsets and headgear. 2) Humans will
also want to talk freely without regard to gain settings, start/stop buttons
of the behind-the-scenes recognition technology. 3) Environmental noises
should be filtered out to enhance recognition, but separately also understood
and tracked to model the environment.

The research to achieve these goals will build on a background of available
sophisticated speech recognition tools developed in the context of
conversational large vocabulary speech recognition. The JANUS recognition
system will run in ``always-on'' mode. For the DARPA Hub-5 system evaluations
we developed segmentation algorithms that can split long telephone
conversations into manageable subsegments [Finke, 1997b; Finke, 1997c;
Fritsch, 1997; Fritsch, 1998]. This procedure will be enhanced to deal with
long silent periods and also occasional room noises, to lead to stable
non-stop operation.. We are already developing noise detection and
classification models that gradually build up an inventory of the usual sounds
in a given environment such as the a meeting room. We expect to correlate
these sounds with locations and people actions or movement, to model human
activity in a room both acoustically and visually.  The identification of the
origin and the originator of sound sources (both visually and acoustically)
and the rejection or elimination of external room noises (phone rings, door
slams, etc.) will also let us begin investigating an as of yet unexplored
dimension of the speech problem: unrestricted open space recognition under
cross-talk conditions. Automatic conversational briefing recognition is a
research topic already under investigation at our lab: under DARPA project
Genoa, we have had the opportunity to investigate conversational speech (which
is sloppy, fast, conversational, which leads to high error rates) and are
working toward improving the accuracies of our recognition systems in this
environment. Our current recording environment, however, still requires
individual participants to wear lapel microphones (thereby reducing the signal
degradation somewhat) and that there is no external speech or noises as part
of the recordings. The proposed research will move this concept further and
attempt to derive sound selectively and track speech in open spaces.

Recognition based on articulatory features and processes Most current speech
recognition systems represent speech as a linear sequence of sound units, the
so-called phones, with transitions between these units at well-defined points
in time.  This approach uses pronunciation dictionaries of the same style as
those used in foreign language learning. For example, the word "sport" is
represented as a sequence /S P AO R T/, while "spot" is written as /S P AA T/.
During the training phase of a speech recognizer, the acoustic characteristics
of these phones are learned using Maximum-Likelihood estimation or Neural
Networks. Here, a transcription of the speech is aligned with the speech
waveform, assigning to each part of the utterance exactly one phone
("beads-on-a-string model" [Mostendorf]), so that the corresponding model can
be trained on this data.  In reality, these clear-cut transitions do not
exist, as human speech production is a continuous process due to the nature of
our articulatory organs. For example, it takes a certain amount of time to
retract the tongue when transitioning from /AO/ to /R/. For this reason, when
modeling phones with Hidden Markov Models, one usually uses several states to
represent one phone, so that /R/ is described by the state sequence [R-b R-m
R-e], [R-b] being the initial part of an /R/, [R-m] being the middle part and
so on. A typical state-of-the-art LVCSR (Large Vocabulary Continuous Speech
Recognition) system uses several thousands of these models for a basic set of
about fifty phones in most cases.

The scheme described above has led to the LVCSR systems, as we know them
today. A wealth of compensation and adaptation schemes has been developed to
improve system behavior under adverse conditions such as background noise,
cross-talk, distant microphones, or difficult (narrow-band, distorting)
transmission channels (e.g. telephone or video-conferences) by adapting the
models to the new conditions without the need for a complete re-training on
matching data, which usually is prohibitively expensive, if at all
feasible. Also, adapting the models to better fit the individual speaker's
characteristics is done in much the same way.  Whenever there is enough data
available to adjust the models, it performs speaker adaptation.

Because these adaptation techniques optimize the criterion used during the
training phase of the recognizer, they usually improve the performance of the
ASR (Automatic Speech Recognition) system under any kind of mismatched
conditions to a certain extent. However, these methods do not really lead to
robustness against changes in speaking style. By this we mean that a speech
recognizer performs much worse on the casual articulation of an utterance in
human-to-human situations such as group meetings in which the speech could be
very fast and difficult to understand even for humans.  In contrast speech in
a dictation situation is a careful reproduction and very easy to
understand. If computers are to produce transcripts or summarizations of
human-to-human meetings, reliable speech recognition under all these
conditions becomes necessary.

Initial research on the topic of spontaneous speech has tried to compensate
the effects of sloppy pronunciations by expanding the dictionary with
specific, reduced variations of important entries. However, it is not easy to
generate new entries without increasing the confusability with existing ones
and it is not clear how to balance variability modeling through dictionary
entries and variability modeling using specialized acoustic models as
described above. Moreover, conventional pronunciation modeling based on phones
can not cover effects, where a realized hyper- or under- articulated
pronunciation of phone exhibits only certain articulatory features, which do
not correspond to a canonical phoneme. In the example above, the /R/ in
"sport" would be a candidate for elimination in sloppy speech, because the
tongue is relatively slow and retracting it to the canonical position for /R/
would require too much time and articulatory effort.  The lips however move
much quicker and will assume a neutral position, allowing them to make the
distinction between /S P AA T/ and /S P AO T/ (the fully reduced version of /S
P AO R T/) which can be made on more evidence than the acoustic quality of the
vowel (/AO/ vs. /AA/) alone. The phone /R/ however is neither fully present
nor fully absent, because one of its "articulatory features", namely the
retracting of the tongue, has disappeared, while another one, the neutral lip
position as opposed to the rounded one for /AO/ and the open one for /AA/ is
still present. These effects cannot be completely described by phone
deletions, substitutions or insertions.

We therefore propose to model phones not as a linear sequence of models, but
as an articulatory feature bundle, where each feature is in itself modeled by
a state sequence, but the transitions can occur at different times. In the
proposed setup, the rounding of the lips is detected by an acoustic model
which is trained on all data with rounded lips (/AO/, /UH/), which leads to a
more robust estimation of the feature. For each point in time, the separate
probability streams coming from the feature detectors are combined by
multiplying their probabilities, which allows the calculation of a probability
for vowels with round lip position (/AO/) etc. This combination- of-streams
approach has already been successfully applied to noise-robust recognition,
where different acoustic models have been trained and it is assumed that they
are differently affected by noise. Our approach however applies this principle
to feature detectors and additionally allows the different streams to segment
the data differently, so that for example the "tongue position" stream would
still be in state [AO-e] while the "lips" stream already is in position
[R-b]. This asynchronicity distinguishes our system from other work
e.g. presented in [KKirchhoff].

Initially, we propose to limit the asynchronicity to one state and determine
the transitions for each stream separately simply by the optimization of the
training criterion (Maximum Likelihood in our case). The goal of this phase is
to train feature detectors suitable for this task and determine suitable
features streams and their respective stream weights and to compare the
computational effort of different efforts. Suitable streams and weights would
be determined by greedy search methods. The completion of this step will show
the validity of the concept of articulatory feature based ASR.

We have already conducted initial experiments, in which we combined six
feature streams with a stream that simply consists of our current 40k
vocabulary English single-pass recognizer.  The features detectors were
trained on a subset of the original training data only and increased the size
of the recognizer from approximately 133k Gaussians to slightly less than 140k
Gaussian models.

Table 1: Preliminary results using articulatory feature streams

System

conventional HMM	
14.1\% error rate
+ articulatory feature streams
11.8\% error rate

These results were obtained on a 20 minute in-house test-set comparable to the
F0 condition of the Broadcast News data. Using on-line incremental ML
adaptation (1 matrix), the error-rate of the baseline system decreased to
13.0\%. Doubling the number of parameters in the baseline recognizers also did
not reduce the error rate significantly so that the system using articulatory
features is currently our top-performing system on this task. Other groups
have reported gains on small systems using feature-based approaches noisy
environments [EEide].

In a second step, we propose to use these stream weights for speaker and
channel adaptation and test the effectiveness of existing adaptation methods
with the aim of matching or improving the speech recognition performance of
the proposed articulatory feature stream method on a broad range of
tasks. Technically, there is no distinction between a phone-model state [AO-m]
and an "articulatory feature"-model state [VOICED-m], so that the adaptation
techniques developed for our existing speech recognizers can be expected to be
useful here as well. Training, adapting and testing the systems on different
in-house and external tasks such as Broadcast News, Switchboard, and "Meeting"
will establish if the proposed method is indeed robust and works on a number
of tasks.

The ultimate goal of this research will be to investigate speaking style
variations as they are exhibited by the asynchronicity of articulatory
movements and use them for speaking style and speaker adaptation. Existing
speech recognizers sometimes use pronunciation and length modeling to improve
recognition performance for fast speech, for example by allowing middle states
to be skipped under certain conditions. The proposed system would allow for
much finer modeling, by allowing slow articulators to stay in place (alas skip
states) while requiring fast articulators (e.g. the velum) to make explicit
transitions, therefore reducing confusability with other words, which would
otherwise increases word error rate.

The proposed approach makes it possible to re-use much of the existing JANUS
infrastructure such as dictionaries, language models and training as well as
adaptation techniques while combining them in a flexible way. As we have
already shown in our initial experiments, the articulatory feature stream
concept can be integrated with existing recognizers, leading to significant
gains in performance even in an early stage of development.

 
Discriminative and adaptive training {I hope for some input from John along
the lines of rapid adaptation on limited data, MMI-SAT}


Robust signal enhancement for remote and variable microphone positioning In
our meeting rooms system, JANUS always listens using (single and multiple)
common tabletop microphones, or microphone arrays. The received speech is
contaminated by background noises (computer noise and air conditioner noise)
and reverberation. The reverberation is a kind of convolutional noise and can
be defined by an impulse response. The effects of remote microphones thus are
the combination of additive and convolutional noise.  These two kinds of
noises interact in a complex way on speech features that are used in
recognition system [Pan00]. Convolutional noise has a larger impact on machine
recognition of speech than human perception of speech intelligibility. In
order to recover the clean speech from reverberant signal, inverse filtering
technique can be applied if the impulse response of acoustic paths is
known. But this requires not only the distant signal but also a reference
signal that is not realistic in practical situations. Moreover the impulse
response is not stationary and can be affected by shape of room, temperature,
humidity and the movement of talker's head etc. When reverberation impulse
responses are short compared with stationary duration of speech, the spectrum
distortions can be approximately estimated from long-term averaging of speech
cepstrum. When the impulse responses are long, they cannot be suppressed by
cepstral mean subtraction. For this research, we will focus more on
compensating the effects of reverberation than removing additive environmental
noise.

The key issue to develop accurate recognition of noisy and reverberant speech
is to overcome the mismatch between training and testing condition. Additive
noises' effect on input speech is added in the spectrum domain; convolutional
noises' effect on the input speech is multiplied in the spectral domain. The
strategy is to estimate additive and convolutional noises separately in linear
and log spectrum domain, and then obtain feature compensation or model
adaptation through approximation. To address this two classes of algorithms
have been developed.  ? Speech signal enhancement which can be considered as a
form of unsupervised adaptation in signal feature domain. This type of
algorithm is simple and efficient but generally makes some assumptions that
limit their scope of application.  ? Acoustic model adaptation. For HMM-based
speech recognition system, adaptation is usually accomplished in two ways:
direct adaptation of the HMM parameters; or indirect adaptation of a set of
transformation parameters and then HMM parameters will be adapted through
these transformations. These types of methods are effective, but generally
slow to adapt or to compute.

Recently many promising new researches have been conducted on model
adaptation. But these approaches are used in tasks of a small-size vocabulary
and a low complexity language, because they are computationally costly and not
scalable for LVCSR (Large Vocabulary Continuous Speech Recognition) system
online adaptation. Model (de)composition [Takiguchi01] one of these
algorithms. It composes new HMMs based on old HMMs decomposition. In their
experiments, the ASR system has only 256 tied-mixture diagonal covariance
HMM's and feature vector is 16 order MFCC's in a 1-frame context window. In
our LVCSR meeting system, more than 100,000 Gaussians are employed and feature
vector is MFCC's in a 15-frame context window, which is further transformed by
LDA(Linear Discriminant Analysis) matrix.

To overcome this, we propose a new approach to combine the signal enhancement
and model adaptation algorithms together based on MAM (model-combination-based
acoustic mapping) Baseline MAM [Westphal 01] has proved to be effective for
additive noise. However, it only employs a simple cepstral mean normalization
for convolutional noise, which is not very effective for remote microphone
condition. In order to compensate for reverberant speech, new adaptive
algorithms need to be developed. There is a close relationship between signal
compensation and model adaptation. Our idea is to use a small size of
secondary acoustic model for robust signal enhancement. State of art model
adaptation methods such as in [Takiguchi01] can be used on this secondary
model; meanwhile the computational cost can be limited in a scalable
extent. We can also utilize structure parameters in the secondary model. Based
on the secondary model, we can predict and map the input signal
distortion. Because it is a signal compensation methods, we avoid modify the
large size of acoustic models in the core speech recognition system
directly. It also has the advantage of on-line unsupervised model adaptation
methods, since we modify the secondary model to fit the current signal
condition. This adaptation process is fast since the size of secondary model
is small.

In summary, we propose to build a new signal enhancement technology and a
robust acoustic model for JANUS recognizer that will fully model the
distant-talking speech for autonomous conversational large vocabulary speech
recognition in meeting rooms. The system will be independent to remote and
variable microphone positioning. The large size of acoustic models of core
baseline clean speech system will be trained on all available databases,
namely BN (Broadcast News), ESST (English Spontaneous Speech), and SWB
(Telephone Speech) corpus.  Therefore it will be able model as many acoustic
conditions as possible. Although we can't cover all conditions, this will make
the baseline line system very robust and improve the overall speech
recognition system performance.  Once the baseline system completes the
offline training, it will keep unchanged until enough new training data for
new conditions is available. In that case, time consuming retraining or
adaptation will always be applied offline.

In the meantime, the proposed on-line signal enhancement method will be
applied for achieving satisfactory performance. Accordingly, the secondary
acoustic model the input signal will be mapped to match the training condition
from current room acoustic conditions. Other modalities can be used to help
fast adaptation of the secondary model, such as speaker position.

