\section{Scientific Goals and Objectives}

The continuing increase in computational power combined with augmented
connectivity has helped popularise and democratise computing.  The enormous
technological push from these advances and pull from humans eager to interact,
has shrunk computing from institutionalised mainframes and minicomputers to
home PC's in less than two decades, delivering communication and computing
services to every man, woman and child, thereby revolutionizing the way we all
live our daily lives.  The revolution is not over yet, as it gets ready
for the next stage in this progression: the move away from the desktop (a
limiting environment), and on into mobile, wearable and ubiquitous devices.
To serve up content to a growing number of mobile computing devices, broader
access and greater bandwidth will be made available, anytime and anywhere.
3G-wireless, UMTS, wireless LAN's are beginning to be fielded and are being
prepared for deployment.  But who will use them?  Who will require such
capability?  Who needs to be on-line all the time?  And who will pay extra for
it?  Voice alone can already be transmitted without much bandwidth, and
computing intensive data-services are frequently inadequate for mobile use.
Yet an enormous spectrum of possibilities is conceivable, provided content and
services can be brought to the user in a contextually appropriate and natural
manner.  Computing and communication services must be instantiated
specifically with the mobile user in mind, rather than making static desktop
applications smaller and cramming them on wearable devices.  

The proposed \spectrum project aims to exploit the emerging power of
mobility to better serve cross-cultural and cross-linguistic communication and
computing needs. It intends to provide answers to questions such as: What does
a mobile user need and want, and how can it be provided appropriately in a
multilingual and multicultural setting?  How can it be engaged to navigate in
a foreign and multinational environment?  What can we do with computing that
is always on?  Three problem areas serve as focus for \spectrum: 
\begin{itemize}
\item Improved Intercultural Communication in a Mobile Setting - Not only 
data-bandwidth matters in our ability to connect. How do we use mobile,
wireless devices effectively to improve intercultural communication and
cross-lingual understanding?  How can speech-to-speech translation (STST)
systems interpret meaning and intention in face-to-face social situations?
How can emotions and cultural subtleties be communicated inter-culturally?
How can speech translation be provided on a mobile platform, despite the added
complexities of varying noise, changing domains, context and changing
bandwidth? How can it be better ported to new domains? And how can it be
improved in robustness, and provide intelligent fallback strategies that
prevent communication breakdown?

\item Integrating Human-Human and Human-Machine Communication in a mobile 
setting - A wearable device cannot be operated by keyboard, and requires
sophisticated natural multimodal human interfaces. Speech, vision and
handwriting seem natural candidates for human-machine interaction. But how can
a system provide seamless integration between human-machine services and
human-human services? How can the system blend the two, provide assistance and
guidance for a user to access and understand databases and information
resources, but also to serve as a go-between to facilitate the interaction
with other humans or with a user's direct environment.

\item Appropriate Services and Applications for mobile wireless communication 
services - What applications actually make sense in a mobile environment and 
how can they be enabled by the technology we propose to develop?  This 
question cannot be answered in the abstract; it has to be studied in actual 
use. Several prototype systems and services will be developed and combined 
into a mobile information assistant that provides human-human as well as 
human-machine services. The considered scenario will be centred on tourism, 
but business, emergency help and humanitarian uses will also be investigated. 
Deployment will be tested in trial uses.  \spectrum will develop two
showcases, the first 18 month after project start and the second at the end of 
the project, to demonstrate the technical advances, and the feasibility of the 
adopted solutions in the chosen scenarios. 
\end{itemize} 

From a technical/scientific point of view, \spectrum will advance the
underlying component technologies as well as develop new system concepts. In
particular: 
\begin{itemize}
\item Human Language Technologies (HLTs) modules and architectures
that have already been developed by the scientific partners will be improved
and expanded to meet the new requirements and challenges in a wireless, mobile
setting. On the speech side, tolerance to environmental noise, speaking rate,
stress and emotion, have to be studied, as well as the optimal integration of
multilingual recognisers. On the translation side, we expand the development
of translation approaches from Interlingua (Interchange Representation Format,
IRF. Cf. Levin et al., 2000) based methods (already advanced considerably by
the scientific partners of SPECTRUM), to include direct translation methods
(statistical and example based approaches) and automatic learning techniques,
in an effort to facilitate development effort and portability. Integration
into multi-engine systems, and use of fallback and backup strategies will
enhance robustness in actual use.  

\item A flexible design of the telecommunication architecture will be 
undertaken, with careful attention given to the potential and limitations of
foreseeable networking solutions. We will propose architectures that can be
actually deployed in the medium term, but expanded as more advanced
communication systems become available.  

\item We will develop more realistic, and, at the same time, richer 
communicative settings than what is currently available, including: expression 
of emotion and social cues, and exploration of a more (pro)active role of the 
interpreting system in error recovery and communication management.  This may 
lead to the development of communication managers that serve to facilitate in 
the communication process. Such communication managers will then also be 
tasked to facilitate between several human agents via speech translation as 
well as with computer agents that serve up complementary information or 
requested data.  
\end{itemize}

Our measures of success will extend beyond mere word error or translation 
accuracies and include: 
\begin{itemize}
\item The broader issue of communication robustness, defined in terms of the 
capability of the system to actively support achievement of communicative 
goals.
\item Domain robustness and portability of the STST system by exploring a 
fully integrated multi-engine approach to translation and by developing 
advanced learning approaches, and interactive grammar induction.  
\item Adequacy of the platforms and the infrastructure: what is required in 
terms of bandwidth, footprint, and speed for STST to be effective on a 
wireless device? What kind of architectures/infrastructures are more 
appropriate to support multilingual conversation between parties located in 
different parts of the worlds? What kind of networking solutions should be 
adopted, given the current uncertainties about future evolutions?  
\item Effectiveness in multilingual and multimodal use. We aim to achieve 
optimal communication effectiveness in use by allowing for dialog and mixed 
and multimodal initiative.  
\item Improved Overall Effectiveness by identifying an optimal selection and 
integration of technologies/services: translation of speech and text 
(documents and signs-input to the system through a webcam; text-based 
interaction among users, etc.), navigation, information retrieval on location, 
on-line guide books, chatting, meeting rooms, and more.  These are all 
features that could be built around/complement STST, providing a range of 
facilities targeting both multilingual communication and information access. 
Furthermore, at least some of these technologies can provide a more direct 
added value, by playing a major role in securing communication robustness.  
\end{itemize}

In more detail, most current efforts in STST address a rather narrow notion of
robustness, targeting the capability of modules and of the system as a whole
of providing sensible answers even in the presence of corrupted input,
incomplete information, etc. Such a concern is natural as long as STST systems
play a passive role, and their main/unique objective is limited to translation
of isolated messages from one language to another. For actual use, though, it
is crucial to start addressing the question of how STST systems can be
designed that play a more active role in securing achievement of communicative
goals, vis a vis system breakdown, misunderstandings, known HLT modules
limitations, networks problems, etc. Communication robustness encompasses, in
our view, all the ways in which communication can fail: communicative goals
misalignments, misunderstandings among the parties, problems due to the
limitations of the system itself (e.g., HLT modules), or of the underlying
networking, etc. A communication manager must be capable of: monitoring the
conversation, detecting and preventing problems; and deploying recovery
strategies which can creatively rely on the rich communicative environment
provided by the wireless world. Communication robustness involves: 
\begin{itemize}
\item use of context, dialogue information-e.g., the goals, and objectives of 
each party-and domain knowledge to improve disambiguation within the STST 
modules (particularly analysis). This includes identifying and tracking 
requests and responses to requests, and then using this information within, 
e.g., the IRF-based translation components to select a more plausible domain 
action.  
\item management of the modalities available to the system.  This is more of 
an HCI direction of investigation, as it requires studying and modelling the
effectiveness of the various modalities available to the user in addition to
the main STST communication. It relates to the STST system itself, which would
be required to identify translation failures that would then trigger the
suggestion of alternative communication modalities. What is realistic is
resorting to: alternative approaches to translation-e.g., word to word
translation, statistical translation-alternative channels for
communication-e.g., textual messages in a SMS-like mode, phrasebook lookup-and
human-machine dialog to increase the quality of communication, propose and
promote bits of information and new contents or pieces of information
(pictures, drawings, texts).  
\end{itemize}

An important and new topic, which affects the quality of translation, and has
been left untouched by previous and current attempts, concerns the view of
language as a powerful tool for communicating emotional states, in particular
through prosody and intonation. The current interest on affective computing,
and the theoretical/technological results already obtained about the
relationships between prosody and emotion, encourage further efforts on the
prosodic cues that express emotional states and dispositions of speakers, and
on the possibility of 'translating' them. Besides the obvious scientific
interest, the translation of emotions is extremely important for improving the
overall quality of STST, when seen from the point of view of communicative
effectiveness. In ordinary, monolingual conversation, in fact, the detection
of emotional states crucially contributes to relevant inferences about the
other party's attitudes, behavioural dispositions and reactions. This
information feeds the dynamic process leading to the adjustment of own's goals
and conversational strategies. Thus, effective multilingual communication in a
wireless scenario would greatly benefit from the translation of emotions.
