\mysection{Speech and Multimodal Components}

Multimodality corresponds to Workpackage~6 in the European proposal.
This workpackage will be lead by IRST with help from UKA and
participation of CMU and CLIPS.

\mysubsection{Speech Recognition Components}
\label{speech}

The partners of {\sc Nespole!} already have the infrastructure, the research
teams and many years of experience in building large vocabulary speech
recognition systems in the languages of the project.  Since the proposed
effort will focus on enabling STST technologies in practical and commercial
situations, we will focus on the language/translation capabilities.  For
speech recognition components we will therefore leverage as much as possible
from on-going speech research efforts at the partners' laboratories and limit
the speech activity under {\sc Nespole!} to integration and adaptation of our
state-of-the art systems.  This activity will include expanding and adjusting
pronunciation dictionaries and language models to the domains at hand, and
adaptations of the recognition engines to the microphones used, and the
channel requirements of the interconnected systems.   We will also have an
opportunity to evaluate multi-engine combinations of speech systems.
Mono-lingually this technique (also known as ``Rover'') was shown by NIST to
improve results as it combines mutual strengths of different recognizers.  At
CMU we will be able to evaluate a combination of acoustics produced by the
JANUS Speech Recognition Toolkit, ACID (a Neural Net based LV-CSR system),
SPHINX, and various derivatives from all of the above.  Multilingual
integration could also be possible but will be outside of the scope of this
proposal.

\mysubsection{Multimodal Components}
\label{multimodal}

It is an important innovative goal of the {\sc Nespole!} project to achieve
effective cross-lingual communication in the context of multimedia shared
electronic market places.  Cross-lingual speech and language technology will
therefore be integrated and harmonized with different media, images, sound,
text, Web-pages, so as to provide the participants the most effective tools for
communication.  Rapid progress is expected as all the scientific partners have
developed multimodal and multimedia systems, that the consortium can build on.

With respect to STST, multimodality is still a largely unexplored issue, so
care will be taken to study the problem both from the point of view of the
impact on the users of the system, and from the point of view of the impact on
the system itself and its modules.  Drawing on the results of these studies,
we will design and implement solutions for a selected number of relevant
features, which will then be integrated in the final showcase for
demonstration.  Our purpose is not to solve all problems involved, but to
identify viable communication environments and avenues for further research.

We can single out three main problems: the first concerns the impact of
multimodality on the system and its users: reaction by the users, conditions
of use, types of features and facilities which improve the communication.  The
second and third concern a major characteristic of multimodality -- namely,
pointing gestures.  Pointing is the act whereby the user exploits images or
portion thereof for communicative purposes, using them as meaningful
objects. The communicative goal is obtained by associating pointing with
appropriate (referential) linguistic expressions.  We will need to investigate
how pointing is used in our scenario, how it integrates with speech and
language, and how domain and language (in)dependent it is.  Moreover, we will
need to tackle the hard problem of synchronization.  To exemplify, suppose the
customer utters something like "This/the cable is broken" and accompanies the
referring expression with a pointing gesture indicating the relevant cable in
a given image on the screen -- e.g., by circling the cable, crossing it, by a
simple click, etc.  On the other party's side, the system must be able to
reproduce the pointing gesture at the appropriate point within (the utterance
of) the translated sentence.  Translation introduces not only a time delay 
but also linguistic asymmetries, since the linear relationships found in the 
source sentence may not be reproduced by the target one.  We plan on dealing
with the synchronizing pointings and their referential expressions at the IF 
level, where referential expressions will be paired with information relating 
to their corresponding pointing gesture.

\noindent
{\bf Detection of pointing gestures:} Depending on the scenario there
are different types of gestures describing technical problems or
indicating relevant parts of a document, of a picture, etc. Such
pointing gestures can be used to mark regions of interest in a map, to
point to devices, or to select a paragraph in a document. Notice that
we need not address the problem of 'pointing recognition', namely that
of assigning meaning to (possibly different) pointing acts. Meaning is
assigned to pointings by the humans involved in the conversation. We
need only address the simpler problem of detecting particular events
in visual space -namely, pointings. Standard pointing tracking devices
will then be used to this end; they will provide for 'pointing
descriptors'.

\noindent
{\bf Synchronizing audio and visual information:} For simplicity, this
task can be broken down into three steps, each producing a pair (a
link) consisting of the pointing descriptor and an object, X. At
the end of the first step, X is an acoustic event.  At the end of Step
2 the second element of the link is the IF object corresponding to
the acoustic event. Finally, Step 3 substitutes the IF object with
the referring expression in the target language produced by the
generation module. The final link is then interpreted by the (target)
user interface, in such a way that the correct synchronization between
translated sentence and reproduced pointing can be established.

\noindent
{\bf Design and implementation of crosslingual and multimodal
workspaces:} We will explore methods and design issues to support a
rich multimedia environment. One goal is having the system dynamically
propose relevant visual and/or textual information, according to the
evolution of the conversation, the current topic, salient elements,
etc.  This will be obtained by enabling the system to keep track of the
current topic of interest, of the relevant objects mentioned, etc.,
and by making it capable of foreseeing informational needs.

