Spectrum: Speech Communication and Translation Under Mobile Environments Proposal for NSF ITR Program Project Summary

The focus of the proposed Spectrum project is communication robustness in spoken language translation (SLT). In addition to conducting scientific research to advance the state-of-the-art in SLT, we will build a prototype of a mobile platform for travel related services. The platform will include spoken language translation for human-human communication as well as spoken humanmachine dialogue for other information and services. Both spoken human-human communication and human-machine communication will be enhanced with fall-back modalities to smooth over communication failures. Evaluation will be based on a suite of metrics related to usability.

Spectrum is designed to be a joint collaborative project between our group at the Language Technologies Institute at Carnegie Mellon and three European research groups at ITC-irst in Italy, Université Joseph Fourier in France and the University of Karlsruhe in Germany. The project consortium will also include Aethra — an Italian telecommunications commercial company. It follows on the footsteps of the Nespole! project currently in progress, involving the same consortium members, and funded by the European Community and NSF under the Multilingual Information Access and Management (MLIAM) program. A partner proposal for Spectrum has been submitted by our European partners to the Fifth Framework program.

Spectrum goes beyond Nespole! in addressing a new round of scientific challenges and in focusing on a broader notion of communication robustness that goes beyond the noise tolerance of parsing and speech recognition. Most current efforts in spoken language translation (SLT) address a rather narrow notion of robustness, targeting the capability of modules and of the system as a whole to provide sensible answers even in the presence of corrupted input, incomplete information, etc. Such a concern is natural as long as SLT systems play a passive role, and their objective is limited to translation of isolated messages from one language to another. For actual use, though, it is crucial to start addressing the question of how SLT systems can be designed that play a more active role in securing achievement of communicative goals. Communication robustness encompasses, in our view, all the ways in which communication can fail: communicative goals misalignments, misunderstandings among the parties, problems due to the limitations of the system itself (e.g., HLT modules), or of the underlying networking, etc.

We identify the following facets of communication robustness: tolerance to noise; adaptability and learning; detection of communication failure; seamless integration of fall-back modalities for repair; seamless integration of services using different modalities (e.g., human-human translation and machine-human navigation assistance); and absorbing information from more modalities (e.g., prosody and emotion). Also, we are moving from evaluations of translation accuracy to evaluations of system usability. Each of the new scientific challenges of Spectrum addresses one of these desiderata.

It is our belief that the investigation of these core research questions identified above can be carried out to a large extent independently, in case European funding is not secured. We believe the research described clearly fits under the ITR focus area of “Augmenting Individuals and Transforming Society”, particularly as it aims to design and develop a multi-lingual system “to serve the needs of multi-national industry, collaborating science teams, or virtual cultural exchanges”.