Part B

Description of scientific/technological objectives and workplan

B1. Title Page

Proposal full title			MULTilinguAl and Multimodal SPoken languagE interaction on the Mobile
Project acronym			MULTAM-SPEM
(Date of preparation)		 
Proposal number			 

B2. Content List (Part B only)

B2.	Content list (Part B only)	2	
		
B3. 	Objectives	4	
		
B4.	Contribution to Programme/ Key Action Objectives.	6	
		
B5.	Innovation 	7	
		
B6.	Project Workplan	13	
		
	B6.A.	Introduction	14	
		
	B6.B.	Project planning and time table	15	
		
	B6.C	Graphical presentation of the project’s components 	16	
		
	B6.D	Detailed project description broken down into workpackages	17	

B3.	Objectives
There is currently much interest in the prospects of wireless communication, triggered by the diffusion of PDAs, the expectations about the forthcoming generations of cellular phones, and the related prospects of technologies providing richer ways for communication among people and with machines.. It is less clear, however, which services might encounter the interest of the wide public, which directions should be pursued in order to give content to the very idea of a wireless world, what technologies better suite foreseeable architectures and infrastructures. 
From the point of view of platforms, there’s growing skepticism for 3G wireless technologies like UMTS, because of the high costs of the required infrastructure, and the consequently narrow window of opportunity for competition. An interesting alternative is provided by wireless networks that offer free “broadband” services on a small geographical scale. Among them, the wireless LANs (WLANs) are the most appealing [Stallings, 1996]. Moreover, network technologies for single-hop wireless LAN are starting to appear on the market at a competitive price. 
Whatever the course of events will be, it is a platitude that the wireless world (WW) will become an interesting and commercially viable reality only if competitive and added-value services will be made available through it. GIUSTIFICARE Concerning human-to-human communication, it is now acknowledged that language can be a serious barrier, which can be overcome by the emerging technology of Speech to Speech Translation (STST). Also, there is consensus that the possibility of managing and mixing different media, hence multimodality, will be crucial. HUMAN-TO-MACHINE??
In the past years, there have been a number of projects exploring the prospects of STST in various scenarios: face-to.face communication (VerbMobil), teleconference (C*-II), internet and the web (NESPOLE!). Some of them also have addressed the issue of multimodal communication, e.g., NESPOLE!. To our knowledge, however, no organised efforts have been devoted to pursue STST-based multilingual and multimodal interaction in the wireless world. 
MULTAM-SPEM aims at multilingual and multimodal communication in the wireless world, by advancing basic technologies, investigating feasibility issues, impact on users, and by exploring new research avenues and application areas. The project will consider four languages: English, French, German and Italian.
The scenario we are targeting is that of an interaction between a human agent and a user/customer, speaking different languages. The agent is equipped with an ordinary PC, while the user exploits a mobile device. The user is visiting the agent’s country, and needs services of the ‘first-aid’ kind: for instance, he/she might want to talk with a doctor to manage some health problems of his/her son, or he/she needs to interact with a car repairer because the car is stuck doesn’t work any more. 
We envisage the production of two showcases, the first at month 18 and the second at the end of the project, to demonstrate the technical advances, and the feasibility of the adopted solutions, in the chosen scenarios.
The technical issues that MULTAM-SPEM will consider are:
•	The improvement of Human Language Technologies (HLTs) modules and architectures already developed by the scientific partners, to meet the requirements and challenges of the WW. Among this, we include development of different approaches to translation: interlingua (Interchange Representation Format, IRF QUOTE) based methods, on which the scientific partners of MULTAM-SPEM have develop a significant experience, direct methods (statistical and example based approaches), and their integration into multi-engine systems.
•	A careful design of the telecommunication architecture, with an eye to the potentialities and limitations of foreseeable networking solutions, and aiming at results which can be actually deployed in the medium run, not only demonstrated.
•	Provision for a more realistic, and, at the same time, richer communicative set than is presently available for other scenarios. This involves issues of affective computing, concerns for a more (pro)active role of the system in error recovery and communication management, moving towards a general communication assistant, and multimodality.
In more detail, the adaptation/improvement of existing STST technologies to the WW involves:
•	Technical questions concerning the features of HLT modules: tolerance to environmental noise, speed, use of prosodic cues to improve language analysis and generation/synthesis—e.g. for differentiating between different dialogue acts (assertions vs. questions), for tracking discourse phenomena (topic/focus), etc. The robustness of the HLT module and of the system as a whole is even more important than in previous application scenarios. Moreover, time is ripe to address the broader issue of communication robustness, defined in terms of the capability of the system to actively support achievement of communicative goals. 
•	Improving the portability to new domains and subdomains of STST systems, by using learning approaches, and interactive grammar induction. 
•	Questions about platforms and infrastructure: what is required in terms of bandwidth, etc. for STST to be effective on a wireless device? What kind of architectures/infrastructures are more appropriate to support multilingual conversation between parties located in different parts of the worlds? What kind of networking solutions should be adopted, given the current uncertainties about future evolutions?
•	Questions about the integration of multilinguality and multimodality in the WW. The WW vision strongly exploits the integration of different media on the same platform. Scientifically, this is supported by results indicating that multimodality is an important asset for STST and multilingual communication in general, for it reduces language complexity and increases the probability of successful interactions. WW, however, requires a careful design of multimodality, because of its physical and technological limitation, and because of the higher demands in terms of software and hardware infrastructure of multimodal communication. We aim at managing these conflicting requirements maximising the benefits of integrating STST/multimodality integration.
•	Questions about the integration of other technologies/services: translation of speech and text (documents and signs—input to the system through a webcam; text-based interaction among users, etc.), navigation, information retrieval on location, on-line guide books, chatting, meeting rooms, etc. These are all features that could be built around/complement STST, providing a range of facilities targeting both multilingual communication and information access. Furthermore, at least some of these technologies can provide a more direct added value, by playing a major role in securing communication robustness.
Most current efforts in STST address a rather narrow notion of robustness, targeting the capability of modules and of the system as a whole of providing sensible answers even in the presence of corrupted input, incomplete information, etc. Such a concern is the natural one as far as STST systems play a passive role, their main/unique objective being to translate messages from one language to another. Time is ripe, however, to start addressing the question of how STST systems can be designed to play a more active role in securing achievement of communicative goals, vis a vis systems breakdowns, misunderstandings, known HLT modules limitations, networks problems, etc. Communication robustness encompasses, in our view, all the ways in which communication can fail: communicative goals misalignments, misunderstandings among the parties, problems due to the limitations of the system itself (e.g., HLT modules), or of the underlying networking, etc. A communication assistant must be capable of: monitoring the conversation, detecting and preventing problems; and deploying recovery strategies which can creatively rely on the rich communicative environment provided by the WW. Communication robustness involves:
•	use of context, dialogue information—e.g., the goals, and objectives of each party—and domain knowledge to improve disambiguation within the STST modules (particularly analysis). This includes identifying and tracking requests and responses to requests, and then using this information within, e.g., the IRF-based translation components to select a more plausible domain action.
•	management of the modalities available to the system. This is more of an HCI direction of investigation, as it requires studying and modelling the effectiveness of various modalities available to the user in addition to the main STST communication. It relates to the STST system itself, which would be required to identify translation failures that would then trigger the suggestion of alternative communication modalities. What is realistic is resorting to: alternative approaches to translation, e.g., word to word translation, statistical translation; alternative channels for communication, e.g., textual messages in a SMS-like mode; more basic capabilities such as phrasebook lookup; to human-machine dialog to increase the quality of communication, propose and promote bits of information and new contents or pieces of information (pictures, drawings, texts).
•	use of information about the interaction to provide minutes and memos, etc., as in smart rooms, but still within a STST framework
An important and new topic, which affects the quality of translation, and has been left untouched by previous and current attempts, concerns the view of language as a powerful tool for communicating emotional states, in particular through prosody and intonation. The current interest on affective computing, and the theoretical/technological results already obtained about the relationships between prosody and emotion, encourage further efforts on the prosodic cues that express emotional states and dispositions of speakers, and on the possibility of ‘translating’ them. Besides the obvious scientific interest, the translation of emotions would be extremely important for improving the overall quality of STST, when seen from the point of view of communicative effectiveness. In ordinary, monolingual conversation, in fact, the detection of emotional states is crucial to permit relevant inferences about the other party’s attitudes, behavioural dispositions and reactions. This information feeds the dynamic process leading to the adjustment of own’s goals and conversational strategies. Clearly, then, effective multilingual communication in the WW requires that these functionalities be supported. Finally, concern for emotional states introduces a new perspective on the ways cultural conventions and differences should be dealt with in multilingual communication systems. STST can be made sensitive to cultural differences, and be enabled to deliver and recognise the culturally appropriate prosodic/emotional cues.
B4.	Contribution to programme/key action objectives
The project addresses the objectives of CPA2 – Multimodal and multisensorial dialogue modes.
The project intends to produce enabling technologies which can help realise the deliverance of high quality information services to European individuals and companies in a user-friendly manner, all these being among the main goals of the IST programme. 
Europe’s language diversity is at the same time a valuable cultural heritage worth preserving, and an obstacle to achieving a more cohesive social and economic development. This situation is reflected in many official EU documents, and has been further stressed as a major challenge in the accompanying document for the Human Language Technology research lines. Improving language communication capabilities is a prerequisite for increasing European industrial competitiveness, this way leading to a sound growth in key economic sectors.
The MULTAM-SPEM project aims at contributing to the promotion of economic growth in the e-commerce and e-service area, at improving access to services by customers in the wireless world, at fostering a better organisation of services by companies and providers, and at promoting a greater control on economic interchanges by customers/users. Improving person-to-person communication between people from different European countries will offer cost-effective and flexible approaches helping to achieve the two strategic objectives of European language policy. 
The project aims at delivering advanced technologies for human-to-human, multilingual and multimodal communication in the wireless world, this way directly contributing to the overall objectives of the IST programme, and in particular to those of CPA2– Multimodal and multisensorial dialogue modes, and Key Action III, research-line III-3.1 (Multilingual Web). Importantly, MULTAM-SPEM has important bearings also on the goals of Key action I, action line I.5.3 (Ambient Intelligence Application Systems for Mobile Users and Travel/Tourism Business). The outcome of the project, in fact, will be a speech-to-speech translation system to serve in this development by making spoken language translation in a wireless world scenario more affordable, robust (even with respect to communicative goals), and easy to deploy. The developed technologies will be demonstrated in a particular domain involving travel/toursim.
At the same time, the concern with cross-domain portability, and the attention given to architectural solutions shows the consortium commitment to device solutions which have a great impact well beyond any particular economic sector, proving beneficial for the overall evolution of services in the wireless world.

B5.	Innovation
MULTAM-SPEM intends to improve over existing speech-to-speech translation technologies, with respect to cross-domain portability, multimedia/multimodality integration, translation of emotions, and both traditional and novel techniques for enforcing robustness. 
Present STST systems are highly domain dependent: in order to improve effectiveness and performances, in fact, both linguistic resources and many of the speech/language engines are often tailored to the particular domain at hand, severely limiting their applicability. Although attempts at addressing scalability and cross-domain portability have already been made in many areas of HLTs, these are rather new concerns in STST. They are crucial, though, to make STST into a viable technological solution. Finding innovative ways to balance between robustness, on the one hand, and scalability and portability, on the other, is one of the main concern of MULTAM-SPEM. To this end we will develop both IRF-based methods—enhancing existing Intermediate Representation Formalism (IRF) to provide for more flexible, maintainable and updatable domain and meaning encoding, and providing solutions to automatise the development and fine-tuning of the required linguistic resources —and direct (statistical, example-based) translation techniques, aiming at integrating them into flexible and effective multi-engines that can dynamically capitalise on the strength of each method while minimising its weaknesses. Importantly, this will also result in extensive comparative analyses and assessments of the respective merits and weakness which will prove beneficial for further research and development effort in this field.
Another important innovative aspect consists in the commitment towards a strict integration of spoken language and images within a multimedia/multimodal setting on a mobile device. Challenges to be met include: integrating the multimodal and multilingual features (how can multimodality affect the STST environment quality?); re-designing graphics, to account for the screen limitations of portable wireless devices; defining types of allowed pen-based gestures, and identify the situations in which they can contribute to the effectiveness of communication; synchronizing speech and visual information (e.g. pen-based gestures, visual feed-backs, etc.); granting flexibility in switching among the input modalities.
Innovation will be pursued also by introducing the novel concern for communication robustness, a notion encompassing all the ways in which communication can fail: communicative goals misalignments, misunderstandings among the parties, problems due to the limitations of the system itself (e.g., HLT modules), or of the underlying networking, etc. The kind of communication assistant we intend to deliver will be capable of: monitoring the conversation, detecting and preventing problems; and deploying recovery strategies which can creatively rely on the rich communicative environment provided by the WW. 
Finally, MULTAM-SPEM will be involved in the new topic of translation of emotions which we judge extremely important for improving the overall quality of translation. This is in line with our overall concern for systems targeting communicative effectiveness. In ordinary, monolingual conversation, in fact, the detection of emotional status is crucial to permit relevant inferences about the other party’s attitudes, behavioural dispositions and reactions. This information feeds the dynamic process leading to the adjustment of own’s goals and conversational strategies. We submit that effective multilingual communication in the WW requires support for these functionalities. Moreover, emotional cues are important not only for the purposes of translation, but for the system itself. For manifestation of stress, dissatisfaction, difficulty of comprehensions, etc., are among the information the system must deal in order to enforce communication robustness.

Finally, let us point out that the very same scenarios (the wireless world) and domain (services for emergency management) MULTAM-SPEM is going to consider constitute an important innovations with respect to those considered by previous and ongoing projects. The integration of sophisticated STST techniques in an environment supporting multimedia presentation and multimodality, in large and complex domains goes well beyond what is currently made available by state-of-the-art technologies (telephone-based call-centres, or by Web-based e-commerce/service), and the concerns of current research projects.
B6.	Project workplan
B6.A	Introduction
The duration of the project is 36 months.
From the point of view of human language processing, MULTAM-SPEM addresses the following main issues: the robustness of STST system with respect to variability due to real users in a WW environment; the integration of different sources of information (translation-mediated spoken language and visual/textual material); the translation of emotions through prosody; portability, by developing both IRF-based and direct approaches to STST, and then integrating them into robust and effective multi-engine systems. From the point of view of the overall quality of the supported interaction, MULTAM-SPEM addresses multimodality—the integration of different sources of information—and communication robustness—namely, the capability of the system to (pro)actively manage communication problems.
Two showcases will be implemented: the first at month 18, demonstrating the results obtained with respect to the new HLT modules (both IRF-based and direct approaches), communication robustness and multimodality; the second, at the end of the project, devoted to multi-engine translation, the translation of emotions, and to further advancements in communication robustness. Taken together, the two showcases aim at demonstrating the advantages and the feasibility of the proposed solutions for multilingual and multimodal communication in a wireless scenario.
The translation of emotion issue will be dealt, in the first place, by means of careful and detailed studies aiming at isolating a set of emotions which is relevant both for the domain, and the kind of interaction scenario envisaged, and whose expression at the level of prosody can be dealt with. During this phase, particular emphasis will be placed on investigating ways to manage cross-linguistic and cross-cultural differences in the expression of emotions. Then technological solution will be studied and implemented which, when validated, will eventually be integrated in the final showcase.
Communication robustness will require investigation of the kind of information—about the context, the dialogue, and the domain—the system can rely on in order to play a more active role. At the same time, a typology of situations in which the system’s active intervention is required need be isolated—translation failures, communication breakdown, problems related to the HLT modules or to the underlying infrastructure, etc. With these in place, ways to manage problems will be hypothesised, implemented and tested. We are currently considering: use of alternative approaches to translation, use of alternative channels (text, images); promotion of information and contents, etc. The activities will be carried in two successive steps: a first phase leading to a demonstration of the obtained results in the first showcase, and a second step devoted to refinements and improvements, with results to be demonstrated in the second showcase.
Multimodality will be dealt with by capitalising on previous experiences, in order to design and implement the integration of multimedia, multimodal and multilingual features. We plan to have relevant results already available for demonstration in the first showcase. These will be submitted to extensive user-oriented studies, leading to improvements which will be shown in the final showcase. User-oriented studies will be exploited both during planning/design, to isolate relevant features, and during the evaluation of the two showcases, to test the effectiveness of the proposed solutions, and feed successive improvements.
The project workplan is as follows. We have four major sets of activities spanning the whole temporal extent of the project: a) the study, development and evaluation of HLT modules (speech recognition/synthesis, IRF-based and direct translation methods, multiengine approaches); b) the activities related to the multimedia/multimodality issues; c) the activities targeting communication robustness; d) the translation of emotions. The first three lines of investigation and development will converge on the realisation of two showcases: the first, due by month 18, will demonstrate (communication) robust multilingual and multimedia-based conversation in the ‘traveller first-aid’ domain. It will be evaluated on real data in order to stress and evaluated robustness. The results will feed the second phase, whereby improvements on all the relevant research lines will be pursued which will be demonstrated in the second showcase, due by month 36. The second showcase will also integrate the results concerning the translation of emotions. 
W1 - Management
A smooth organisation of works requires a good management, co-ordination and interfacing of the different activities. In MULTAM-SPEM the management is even more crucial as the consortium activity has to maintain a close integration with the American partner. Management will take care of the coordination of technical aspects, will supervise the project’ schedule, assess project progress, secure a proper level of communication among the partners, the respect of the work-flow, and will organise project meetings. Evaluation procedures for the whole project will be developed and implemented. Reviews, verifications and demonstrations during the work will be organised, and quality controls will be enforced. Reports and deliverables will be collected, and their quality will be assessed before submitting them to the European Commission.
Information exchange will be guaranteed also through a project WEB site where reports and drafts will be made available. The WEB site will also serve for dissemination purposes. 
In order to exploit possible synergies with other HLT projects working on equal or analogous themes, some activity will be devoted to concertation and clustering. To this end, the following activities will be necessary:
•	Actively participate in general cluster meetings (min. 2/year, typically by the technical coordinator).
•	Contribute and participate to cluster special events such as seminars, workshops, open days, etc.
•	Collaborate in designated work groups (typically by interested and competent RTD staff). 
•	Make available project-related information, including technical documentation, to other cluster participants and/or work group members. 
•	Contribute content to print and/or electronic cluster publications, demonstrations and showcases.
•	Support the designated cluster organs (coordinator and secretariat) in successfully implementing the agreed cluster activities and to respond to their requests. 
•	Report progress to the cluster coordinator.
The project management will also take care of all administrative procedures, including contractual matters between partners. The manager(s) will act as a first point of contact in liaising with the funding agencies, the partners and the user-group. Concerning financial matters, they will prepare accounts, provide for payments and so on.
W2 - Requirements specification
System requirements falls in two main categories: application dependent and domain dependent requirements. The former includes hardware and software requirements addressing the desired functionalities rather than the domain characteristics. The latter focus on the peculiarities of the scenarios and domains.
Domain requirements
The choice of the application domains—i.e. services for the traveller—follows general consideration about the tendencies and needs of the international markets. Tourism and travel are among the economical sectors with more growth chances, and among those which will take the greatest advantage of WW. Tourism is involved in an important change consisting in a shift from the current broker-supported market (agencies, tour operators) to a situation in which the customer directly contacts and negotiates with local representatives of service providers the so called destinations. At the same time, destinations are more and more turning into providers of broad spectrum services, not only limiting themselves to traditional tasks such as the care of lodging. Among the services the traveller expects, those related to emergencies are among the most requested ones. Helps, directions, advices in case of illness and other small health problem, about such inconveniences as car breakdowns, can be highly problematic when the traveller does not speak/understand the language of the country he/she is in. On the provider side, this requires the capability of negotiating and supplying personalised solutions, engaging in an interaction with the customer, with the aim of meeting the demands and needs of customers. The customer, in turn, will have the possibility of explaining his/her situation and problems, his/her expectations, and will enjoy a true guidance towards finding optimal solutions.
Addressing such an emerging scenario requires that we resolutely improve over existing STST prototypes by addressing highly complex communicative exchanges.. Within MULTA-SPEM the customer interacts and negotiates on a great range of topics with the service provider using his/her own native language. He/she must be enabled to describe his/her problem, possibly resorting to exemplificative pictures, providing details about his/her travel plans, receiving punctual and reassuring information. The destination, on the other hand, needs a system capable of assisting him/her in managing data-bases of pictures and images describing hotels availability, sporting resorts, cultural events, etc., as well as textual information of various kinds. All this will enable the destination to react to the customer choice by proposing optimal solutions.
Scenarios requirements
The scenario we envisage integrates STST for four languages (English, French, German and Italian) with multimedia and multimodal interaction in a WW environment. A careful and thorough study of the role of the interacting partners in the ‘first-aid’ scenario is needed, leading to a detailed description of the system functionalities.
Architectural requirements. 
Currently, there’s growing skepticism for 3G wireless technologies like UMTS, because of the high costs of the required infrastructure, and the consequently narrow window of opportunity for competing. An interesting alternative, at least as far as experimentation with the relevant services is concerned, is provided by wireless networks that offer free “broadband” services on a small geographical scale. Among them, the wireless LANs (WLANs) are the most appealing [Stallings, 1996], and network technologies for single-hop wireless LAN are starting to appear on the market at a competitive price. 
Given the complexity of the involved issues, and the speed of evolution of technology, much care will be needed to carefully design the architectural solutions, prioritizing those which permit to effectively experiment with, and assess innovative services, while at the same time do not prejudice integration in future platforms.
For our project, we plan to first focus on two WLAN technologies: Bluetooth [BLUE] and IEEE 802.11 [IEEE802]. Extended WLANs, in fact, can guarantee the full coverage of a territory—e.g., city centres, university campuses, airports, highways, etc. Moreover, hardware supporting both the Bluetooth and IEEE802.11 standard are also appearing [EWDI], these products also aiming at solving the co-existence problems that plague wireless networks. 
MULTAM-SPEM will constantly monitor the technological evolution of platforms, with a particular attention paid to multimedia cellular phone of third generation. If there will be relevant evidence that UMTS will take off, the second show case will be deployed as a UMTS services. In order to do this a service provider partner, or terminal producer will be involved at the end of a negotiating phase which will last around 1 year after the project start.
HLT modules and resources specification. 
This will include: requirements and specifications for the HLT modules, the IRF, the linguistic resources, data and corpora collection and annotation, with a particular emphasis on prosody and emotions.
Specification of the communication robustness features
This includes the details and the specifications for the activity aiming at the features and functionalities needed to turn the system into a true, multilingual communication assistant supporting communication robustness.
Testing and evaluation specification
A detailed plan for the various testing and evaluation activities will be provided, including objectives, thresholds, procedures, etc. This will cover all the aspect under investigation, starting from the more traditional ones—evaluation of single HLT modules, comparative evaluation of whole STST systems, evaluation of usability and impact on communication quality for multimodal features—till the more innovative aspects the project is concerned with—evaluation of the emotional quality of translation and evaluation of communication robustness. With respect to the latter, the topic of better and more effective task-based evaluation metrics becomes crucial to assess the effectiveness of switching between modalities and strategies, hence to evaluate the degree of communication robustness attained.
W3 - Showcase development
In both showcases, the scenario considers an agent of a ‘service-provider’ and a ‘customer (the tourist/traveller), speaking different languages. Both parties use thin terminals and communicate using voice and pens (or other pointing devices). The agent is equipped with an ordinary PC, while the customer exploits a mobile device, e.g., a last generation PDA with:
•	multimodal input : voice, pen, touch screen, camera;
•	wireless communication based on 802.11 standard;
•	emerging mobile operating systems (Windows CE, Mobile Lynux, …)
•	Netmeeting-like communication application
The user is visiting the agent’s country, and needs services of the ‘first-aid’ kind: in particular he/she is might need to talk to a doctor to manage health problems of his/her son, and/or to talk with a car repairer because of car failures. 
This workpackage will take care of all the activities concerning the development of the relevant architecture/infrastucture, and the integration of all the modules. In more detail:
•	Set-up, test and validate the hardware and software platform supports. The communication architecture will be based on a number of HOTSPOTS (one for each language)—that is, WLANs covering restricted areas such as airports, buildings or downtown areas, linked to Internet through a high bandwidth connection. Users can move within HOTSPOTS; whereas agents are based at a Web Call Centres, linked with the HOTSPOTS by a high bandwidth connection. 
Recently, WLAN solutions, based on IEEE 802.11 standard have appeared on the market, and, software packages based on H323 standard and Netmeeting running on PDA have also been presented—e.g., Spectrum 24 by Symbol Technologies. Also very interesting is the iPAQ Pocket PC’s ability to talk to wireless LANs using a PC Card, and the prospects offered by the Voice over Internet Protocol (VoIP).
•	MULTAM-SPEM will explore the suitability of system architecture based on thin terminals for the users. They will: input speech, images and gestures; encode and send them to the server(s); output the same information when this is made available. All the modules implementing the multilingual and multimodal features of the system will reside on geographically distributed servers. In this respect, we intend to capitalise and improve on solutions the technological partner has already developed. 
•	Integration of the STST, multimedia and multimodal functionalities. Our scenarios integrate visual cues and images with multilingual speech. Thus, both the agent and the customer might want to be able to use images, movies, and graphics, while commenting on them by means of spoken language, and acting on them through gestures.
W4 - Translating Emotions
The translation of emotions expressed through prosody is a topic, that, to our knowledge, has never be addressed before. It is of the utmost importance, in general, and for the WW in particular. In ordinary, monolingual conversation, in fact, the detection of emotional states is crucial to permit relevant inferences about the other party’s attitudes, behavioural dispositions and reactions. This information feeds the dynamic process leading to the adjustment of own’s goals and conversational strategies. Clearly, then, effective multilingual communication in the WW requires that these functionalities be supported.
Emotional cues are also very important for the system itself in view of communication robustness: stress, dissatisfaction, difficulty of comprehensions, etc., as manifested through prosody, are among the information it could use to monitor the status of the interaction, and plan repair strategies. 
We intend to proceed as follows:
•	although we plan to build on existing results concerning the relationships between prosody and emotion (cite XX), the concern with STST translation is new, so preliminary empirical studies are needed to isolate a set of emotional states which is relevant for both the domain and the kind of interaction scenario envisaged, and whose expression at the level of prosody can be dealt with. 
•	once the relevant set of emotional states has been identified, we will turn to the adaptation of the speech recognisers and synthesers to allow them to recognise and produce the relevant cues, and to the adaptation of the translation system. As to the latter, we plan to prioritise the IRF-based approach. This method, in fact, seems to provide for a more perspicuous encoding of emotional indicators, by having them directly included into the IRF expressions coming from the analysis chain, and feeding the synthesis/generation one. Studies will be performed to identify the right level at which to include emotional information—e.g., at the level of dialogue acts, at some lower level (concepts and features), or at both. Experiments will be conducted also with direct translation methods, though we do not commit ourselves to results in these respects.
•	We plan to have results stemming from these activities available and demonstrable in the final showcase..
W5 - Communication robustness
Communication robustness requires the design, implementation and test of a communication manager—that is, a module which monitors the status of the interaction, detects and anticipates communication failures, and deploys appropriate strategies to repair/prevent them. 
To this end, we intend to:
•	investigate the kind of information—about the context, the dialogue, and the domain—the system can rely on in order to play a more active role;
•	isolate a typology of situations in which the system’s active intervention is required—translation failures, communication breakdown, problems related to the HLT modules or to the underlying infrastructure, etc.;
•	define the set of strategies the system will deploy to enforce communication robustness;
•	individuate the kind of resources the communication manager can rely on to implement its repair/prevention strategies. Currently, we are considering: alternative approaches to translation, alternative channels (text, images); active promotion of information and contents, etc.;
•	design and implement the repair/prevention strategy themselves.
The activities will be carried in two successive steps: a first phase leading to a demonstration of the obtained results in the first showcase, and a second step devoted to refinements and improvements, with results to be demonstrated in the second showcase.
W6 - HLT development
At the core of the project there will be the development of new state-of-the-art Human Language Technology (HLT) components, which will be integrated into complete STST systems. While we plan to build upon the technology and scientific advances achieved by previous and current major STST efforts such as C-STAR, VERBMOBIL and NESPOLE!, our main research goal will be to significantly advance the technological capabilities of STST systems in terms of robustness, scalability and portability to new domains, and prosodic processing.
Because of the spontaneous nature of spoken language, effective STST systems require a design that is robust to disfluent and unpredictable input. Such phenomena must be appropriately dealt with at every level, from that of the acoustic signal up to that of conversational structures. Moreover, the broadness and complexity of the domains to be addressed by MULTAM-SPEM suggest that it be broken down modularly into smaller sub-domains. This, however, requires a complex architecture of STST components that can support modular development and can effectively integrate the separate sub-domain knowledge sources. As a consequence, portability and scalability acquire a particular relevance.
A major objective of MULTAM-SPEM is the development and test of different approaches to STST. Thus, we will continue to pursued IRF-based approaches, but will address also direct methods: statistical and example-based translation. 
The IRF-based approach targets a series of modules, each addressing one of the considered language, and consisting of two separate chains: an analysis chain, which maps input (spoken) utterances to IRF representations, and a synthesis chain, mapping the latter into output (spoken) utterances. One of the great merits of this approach is that given n languages, only n (analysis and generation) modules are required for translating between all the possible language pairs. At the same time, the integration is very easy, simply requiring that each module be capable of producing/understanding IRF-strings. To enhance cross-domain portability, we intend to focus on learning approaches for both analysis and generation (the latter being a quite new topic), and on interactive grammar induction
Direct methods, on the other hand, directly and immediately take advantage of data and, for this reason, they provide a simpler solution to the portability and scalability problem. Also, in comparative tests on simple domains (QUOTE Ney), they seem to perform better than competing methods, though it is acknowledged that they might be inferior as to quality of translation and robustness.
Our aim is not only to improve on existing technologies for all the mentioned approaches, and provide comparative evaluation along the relevant dimensions (portability, robustness, quality of translation). We have the more ambitious goal of addressing the integration of those different techniques within a unique translation engine. In this, we are encouraged by preliminary results obtained in projects such as Pangloss, DIPLOMAT and NESPOLE!, which show that a multi-engine approach has the capability of dynamically making the best of the strengths of each techniques, while minimising the respective weaknesses. 
Corpora acquisition and annotation.
Data for both the ‘first-aid’ domain will be collected and annotated in the various languages considered, at all the relevant levels, including the prosodic and emotional one. They will be used to develop and test the language knowledge sources of the translation components (grammars, lexica, IRF, etc.), train HLT modules, including the language models of acoustic recognisers, and feed the activities of W4. They will also be used to perform comparative evaluations among different approaches.
Speech Recognition and Synthesis Components
The partners of MULTAM-SPEM already have the infrastructure, the research teams and many years of experience in building large vocabulary speech recognition and synthesis systems in the languages of the project. Besides expanding and adjusting pronunciation dictionaries and language models to the domains at hand, starting from their state-of-the art systems, we plan to: 
•	adapt the recognition engines to the channel and environmental (noise) requirements of the WW and of the chosen scenario;
•	address prosody, both on the recognition and on the synthesis side. This will target: more information, and of better quality, for sentence level processing (disambiguating among different dialogue acts; provide cues for understanding the informational structure of sentence (topic/focus distribution), etc.); the detection and expression of relevant emotional states, as identified in W4.
Translation Components
- IRF-based Analysis and Generation Engines
Our main goal in pursuing further the IRF-based approach is to deepen the understanding, and sharpen learning approaches to analysis and generation (the latter a quite new topic of research), and comparing the results with those obtained with existing modules for IRF-based translation.
Besides this, work will be needed to adapt the IRF to the new domain.
Direct translation engines	
We will consider the following techniques:
•	Example-based Translation, using generalised EBMT engines capable of producing good results from a relatively small bilingual corpus. 
•	Direct Statistical Translation
One main objective is to compare direct methods with the IRF-based one with respect to robustness, scalability and domain portability. 
Multi-engine Translation Approaches
The experience and insights gained with single-engine approaches will feed our effort to address multi-engine STST. Such a paradigm combines several different translation modules into a single architecture, with the goals of capitalising on their respective strengths, and minimising their weaknesses, in terms of robustness, scalability, domain portability and translation quality. To this end, we plan to build upon the results of projects such as Pangloss, DIPLOMAT and NESPOLE!. The main new research challenges in this context are: (1) how to extend the multi-engine framework to incorporate the IRF-based and direct translation approaches; and (2) the development of a significantly improved statistical framework for selecting an optimal combination result from the multiple engines.
W7 - Multimodality
Traditional keyboards and mouse interfaces are unpractical on small portable devices. Most researchers and technology providers agree on the idea that the next generation of mobile networked users will use speech and gesture to enter and retrieve information. Recent studies in HCI indicate that multiple input modalities can assure greater communication naturalness and higher system effectiveness. In particular some laboratory studies found that the combination of spoken and pen-based input yields significantly faster task performance, if compared with unimodal interfaces, with significantly fewer errors and disfluences.  In addition, experimental users often switches modes to correct errors and prefer to interact multimodally than unimodally if they can choose. Multimodal interaction seems to provide the user with the ability to capitalize on the advantages of all the input modalities and to overcome their weaknesses.
Building on the experience of such projects as NESPOLE!, the aim of the W7 is to design, implement, and integrate multimedia/multimodal features with multilingual ones on the base of user-oriented studies, taking into account the physical and technological constraints and the software and hardware limitations.
Experimental studies, as well as non-experimental usability evaluations, will be performed in order to define guidelines for, and to evaluate the performances of the system as to:
•	Re-designing graphics. On the small screen of portable wireless devices, graphical information should be re-formatted. In addition, usability evaluations for all icons and dialogue boxes is needed in order to assure to users an immediate comprehension of the function of all the elements displayed on the screen.
•	Defining types of allowed pen-based gestures. Answers to the following questions should be given: which kind of messages could/should be supported by pen-based gestures? which are the relevant properties of the pointing gestures that the system should recognize to improve the quality of communication?
•	Synchronizing speech and visual information (e.g. pen-based gestures, visual feed-backs…).  How can the system deal with time delays and reproduce the original and/or natural sequence of spoken and pen-based input?
•	Granting a minimum level of task performance even when the computer is disconnected from the network. Mobile devices connected to a wireless network suffer wide variations in network conditions. Which are the system functionalities that should be granted even in case of disconnection from the network?
•	Granting flexibility in switching among the input modalities. The user should feel free to choose which modality to use for each input.
•	Integrating the multimodal and multilingual features. How can multimodality affect the STST environment quality? Can multimodality be an added value even in a wireless multilingual environment?
W8 - Assessment and evaluation
Testing and evaluation will focus on robustness, scalability and portability, as well as impact on users for showcases and multimodality. To these ends, data reflecting the real tasks will be used. Every partner will assess all the modules it develops.
1.	 Task-Based Evaluation: This involves identifying the users’ goals in a dialogue and giving the dialogue a score which is a function of whether each goal succeeded or failed and how many turns were spent on repairing wrong translations. The method for deciding what counts as a goal and the exact nature of the scoring function are topics for research, especially in view of the importance of this type of evaluation for communication robustness.
2.	 Sentence-Based Evaluation: The goal here is to evaluate coverage and accuracy. For each sentence the source and target are compared by a bilingual human judge. We evaluate the translation components alone (using hand-transcribed input), and also evaluate end-to-end performance using speech-recognition output as the input to the translation components. This covers both IRF-based methods and direct approaches.
3.	 Evaluation of individual components: Individual components can be evaluated by comparing that component’s output to hand-coded ‘perfect’ output. For example, In the IRF-approach, the parser is evaluated by comparing its output on a set of sentences to hand-coded IRFs for those sentences, and generators by comparing their outputs with sentences produced by a human from the same input IRFs. Speech recognition can be evaluated using standard measures such as word error rate. Multimodality can be evaluated by comparing the system performances against testbeds obtained from the data collected during the experimental phase described in W6.
4.	Evaluating scalability and portability: In order to evaluate scalability and portability, we will plot coverage and accuracy as a function of the amount of resources. For IRF-based STST systems, the resources are training data, grammars and lexica, and time spent by human developers. For Example-Based STST, and statistical methods, the resources are training data.
5.	Evaluation of Multi-Engine STST: We will compare the performance (probably sentence-based, not task-based) of the multi-engine STST system to the performance of individual engines. We will also remove individual engines from the multi-engine system to see how performance degrades.
6.	Evaluation of the emotional quality of translation. This will resort both to traditional methods, similar to those mentioned at point 2 above and at point (7) below.
7	Showcases evaluation. Showcases will be evaluated with respect to standard measures of usability in the area of human-computer interaction. Given the demonstration nature of the applications, subjective measurements will be preferred to objective ones. Since the beginning of the project, focus groups will be set up and standard techniques of the so-called “discount usability engineering” will be applied to ensure that showcases meet the user expectations and needs.
W9 – Dissemination
To fully disseminate and exploit the possibilities of the foreseen technology of spoken translation other potential sectors, not represented by the project’s users, will be considered, i.e. car sellers, furniture companies, insurance companies, banks, Internet service providers, etc. In order to pursue this goal, a user group will be set up. The composition of this group will be as heterogeneous as possible in order to reach different potential users and/or system integrators. Many other fields of interested in services offered through the WW will be contacted and invited to participate to the users group. 
Another target of dissemination activities will be the scientific community. Papers describing the project approaches and results will be submitted to scientific (SpeechCom, IEEE, ACM, CL), and to wide diffusion journals (Science et Vie, Scientific American, La Recherche), to major international congresses (Eurospeech, ICSLP, IEEE, Coling, ACL), national congress (JEP, TALN, IHM), etc. Moreover, they will be shown in demonstration stands (ErgoIA, CHI, Interact), on the Web and will be advertised in the industrial world.
A third important way to disseminate the results will be the formation of a user group consisting of relevant players of the scientific and technological community. Such a user group will have the task of commenting and advising on relevant technologies, orienting choices, etc. Representatives of this user group might be invited to participate in scientifically/technologically relevant events organised by the consortium. Besides representativeness with respect to the scientific world, this user group will also aim at covering languages not present in the consortium, with a particular emphasis on Asian ones.
A Web site for the project will be made available, with different access permissions for the consortium members and the rest of the world. At the appropriate time, it will feature a simple demonstration showing the main functionalities of the systems. With similar contents, a CD will be produce with the goal of promoting the project and its results.
At the end of the project a workshop will be organised in order to present the projects results, along with the showcases, the user group outcomes and the future exploitation activities.
Finally, the consortium will take care of exploitation matters, by performing a detailed study of the economical and social relevance and potentials of the developed technologies.

B6.B	Project planning and time table

B6.C	Graphical presentation of the project’s components

B6.D	Detailed Project description broken down into workpackages
9.2 Workpackage list

B1.	Workpackage list	
								
Work-package No.	Workpackage title	Lead contractor No.	Person-months	Start
month	End month	Phase	Deliverable No.	
W1	Project Management	P1	82	m0	m36		D1	
W2	Requirements	P3	72	m0	m23		D2, D12	
W3	Showcases development	P4	80	m4	m36		D10, D17	
W4	Translation of Emotions	P1	75	m4	m34		D7, D14	
W5	Communication Robustness	P5	104	m4	m34		D8, D15 	
W6	HLT development 	P2	230	m4	m34		D6, D13, D14, 	
W7	Multimodality	P1	87	m4	m34		D9, D16	
W8	Assessment and Evaluation	P5	56	m16	m36		D11, D18	
W9	Dissemination and Implementation	P1	24		m36		D3, D4, D5, D19, D20	
								
	TOTAL		810					

B2.	Deliverable list

Del. no.	Del. name	WP no.	Lead participant	Estimated person-months 	Del. type**	Security*	Delivery
(project
month)	
D1	Evaluation and Assessment Plan	W1	P1	2	D	Int	m3	
D2	Requirements for 1st shc	W2	P3	4	D	Int	m4	
D3	Project Web site	W9	P1	2	P	Int	m4	
D4	Project Presentation	W9	P1	1	D	Pub	m5	
D5	Dissemination and Use Plan	W9	P1	2	D	Publ	m6	
D6	Annotated Data for Showcase1	W6	P2	4	D, O	Publ	m12	
D7	Emotional states and prosody	W4	P1	4	D	Publ	m17	
D8	Communication Robustness	W5	P5	4	D	Publ	m17	
D9	Definition of Multimodal Environment	W7	P1	4	D	Publ	m17	
D10	Showcase 1	W3	P4	27	P, D	Pub	m18	
D11	Evaluation of Showcase1	W8	P5	4	D	Publ	m20	
D12	Requirements for 2nd shc	W2	P3	4	D	Int	m23	
D13	Annotated Data for Showcase2	W6	P2	4	O, D	Publ	m30	
D14	HLT modules	W4, W7	P2-P1	52	O, D	Publ	m35	
D15	Communication Manager 	W5	P5	14	P, D	Publ	m35	
D16	Multimodal Environment	W7	P1	20	P, D	Publ	m35	
D17	Showcase2	W3	P4	26	P, D	Publ	m36	
D18	Evaluation of Showcase2	W8	P5	4	D	Publ	m36	
D19	Technical Implementation Plan	W9	P1	3	D	Int	m36	
D20	Audio-Video Presentation	W9	P1	6	O	Publ	m18	

*Int.	Internal circulation within project (and Commission Project Officer if requested)
  Rest.	Restricted circulation list (specify in footnote) and Commission PO only
  IST	Circulation within IST Programme participants
  FP5	Circulation within Framework Programme participants
  Pub.	Public document
**D	Document 
   P	Program
  O	Other	

B3.	Workpackage description

Workpackage number :	W1 - Management	
Start date or starting event:	m0	
Participant:	P1	P2	P3	P4	P5			
Person-months per participant:	30	12	10	10	20			

Objectives 
Devoted to overall management, co-ordination and interfacing activities in the project. The following tasks will be accomplished:
• Organise and supervise the project schedule, by assessing the project progress, managing communication, work-flow and meetings;
• Collect reports and deliverables, assess their quality and submit them to the Commission;
• Take care of contractual matters between partners;
• Liaise with Commission, partners and subcontractors, acting as first point of contact;
• Manage all financial matters (prepare accounts; provide payments etc.);
• Ensure quality control and enforce best practices;
• Develop and implement Project Evaluation; 
• Organise peer reviews, verification and demonstration;
• Provide technical co-ordination;
  Liase with cluster activities.

Description of work 
Exchange information among partners; ensure participants have visibility of the whole project by managing the project Web-site; ensure the smooth integration of the different activities carried on within the project. 

Deliverables 
• Progress reporting
• Financial reporting
   D1: Evaluation and Assessment Plan	

Milestones and expected result 

	

Workpackage number :	W2 - Requirements	
Start date or starting event:	m0	
Participant number:	P1	P2	P3	P4	P5			
Person-months per participant:	15	15	15	12	15			

Objectives 
• Definition of the ‘first-aid’ domain. Study of traveller/tourist needs.
• Definition of the hardware and software architecture for the showcases.
• Definition of the assessment procedures for both showcases and for the single modules.
• STST modules and multimedial environment specification, including multimedial data-bases. This also includes definitions and specification for prosody processing and the translation of emotions.
• Definition of and specifications for, the ‘communication assistant’ features.
• Definition of and specifications for the multimodal features.

Description of work 
T2.1	Hardware and software platform specification
T2.2	Domain definition and specification
T2.3	STST modules and multimodal environment specification (including the translation of emotions)
T2.4	Communication assistant specification
T2.5	Assessment procedures.

Deliverables 
• D2 Document reporting on: Domain definition for the first showcase (first-aid). STST modules specifications; hardware and software platform specification; multimedial and multimodal environment. Assessment procedures for the first showcase.
• D12 Document reporting on the domain definition for the second showcase (expanded first-aid) and assessment procedures for the second showcase. It also contains the specification for the translation of emotions, and possible changes to the platform as suggested by the results of the evaluation of the first showcase

Milestones and expected result 
M1 at m4: requirements for the first showcase
M4 at m21: requirements for the second showcase.

Workpackage number :	W3 - Showcase development	
Start date or starting event:	m4	
Participant	P1	P2	P3	P4	P5			
Person-months per participant:	16	10	8	24	16			

Objectives 
We intend to: develop and test the hardware and software architecture for the showcases; integrate the STST modules; integrate the multimedia and the multimodality modules, integrate the modules implementing the ‘communicative assistant’ features, and translation of emotions.

Description of work 
T3.1 Construction, testing and validation of the hardware and software platform supports
T3.2 Realisation of the first showcase
T3.3 Realisation of the second showcase

Deliverables 
• D10 First showcase + Documentation
• D17 Second showcase + Documentation

Milestones and expected result 
M3 at m18: First showcase
M6 at m36: Second showcase

Workpackage number :	W4 – Translating Emotions	
Start date or starting event:	M4	
Participant number:	P1	P2	P3	P4	P5			
Person-months per participant:	24	15	13		24			

Objectives 
• Analyse prosodic properties of the languages involved (English, French, German and Italian);
• Isolate a relevant set of emotional states together with their prosodic cues;
• Develop models of the relationships between emotional states and prosody;
• Adapt HLT modules (speech recognisers and synthesisers, translation modules) to the needs of the translation of emotions. 

Description of work 
T4.1	Study and experimentation on the relationships between emotional states and prosody
T4.2	Adaptation of the HLT modules to the needs of the translation of emotions

Deliverables 
• D7. Document containing the results of T4.1 
• D14. HLT modules + documentation. 	

Milestones and expected result 

M5 at m35: New HLT modules enforcing the translation of emotion.

Workpackage number :	W5 – Communication Robustness	
Start date or starting event:	m3	
Participant number:	P1	P2	P3	P4	P5			
Person-months per participant:	24	15	12		24			

Objectives
• investigate the kind of information the system can rely on in order to enforce communication robustness;
• isolate a typology of situations in which the system’s active intervention is required;
• define the set of strategies the communication manager will deploy to enforce communication robustness;
• individuate the kind of resources the communication manager can rely on to implement its repair/prevention strategies;
• design and implement the repair/prevention strategy, and the relevant resources.

Description of work 
T5.1	Definition of the relevant information, typology of situations, strategies and basic resources needed for communication robustness.
T5.2	Implementation of the communication manager, and of the relevant resources.

Deliverables 
D8 Document containing the result of T5.1: specifications for the communication manager.
D15 Communication manager + documentation

Milestones and expected result 
M2 at m12: modules for the first showcase
M5 at m24: modules for the second showcase

Workpackage number :	W6 – HLT modules	
Start date or starting event:	m3	
Participant number:	P1	P2	P3	P4	P5			
Person-months per participant:	60	60	30		60			

Objectives
• organise and perform data collection and annotation in the four languages; 
• organise and perform comparative tests between the various single-engine approaches and the multi-engine one;
• production of the STST systems for showcase1. Update of the STST systems, and production of multi-engines for showcase2. 

Description of work 
T6.1	Data collection and annotation
T6.2	Development and updating of STST systems for showcase1
T6.3	Development of STST systems for showcase2

Deliverables 
D6 Annotated data for first showcase
D13 Annotated data-2 for second showcase
D14 HLT modules for STST + Documentation (same software and same documentation as for the second deliverable for W4)

Milestones and expected result 
M2 at m12: modules for the first showcase
M5 at m24: modules for the second showcase

Workpackage number :	W7 - Multimodality	
Start date or starting event:	 M4	
Participant number:	P1	P2	P3	P4	P5			
Person-months per participant:	26	20	15		26			

Objectives 
• Study of interaction in a multilingual, and multimodal environments. 
• Define features for graphics re-designing
• Define the types of allowed pen-based gestures. 
• Specifications for the synchronisation between speech and visual .
• Development of the multimodal features and their integration with multilingual ones. 

Description of work 
T7.1	Study of the multimodal, multilingual setting.
T7.2	Definition of multimodal environment.
T.6.3	Design and implementation of multimodal workspaces, and their integration.

Deliverables 

D9 Document containing the results of the study concerning T7.1.
D16 Multimodal environment + Documentation.

Milestones and expected result 
M2 at m12: multimodal modules for the first showcase;
M5 at m24: Multimodal modules for the second showcase

Workpackage number :	W8 - Testing and evaluation	
Start date or starting event:	 m16	
Participant number:	P1	P2	P3	P4	P5			
Person-months per participant:	12	12	8	8	16			

Objectives 
We intend to test and evaluate single modules and whole STST systems, including multimodal spaces, along the dimensions of: (communication) robustness, portability, quality of translation (including the translation of emotions). 

Description of work 
T8.1	Evaluation of modules, STST systems and multimodal spaces for the first showcase
T8.2	Evaluation of modules, STST systems and multimodal spaces for the second showcase	

Deliverables 
D11 Document reporting on the results of the evaluation of: the single HLT modules, STST systems and multimodal spaces for the first showcase.
D18 Document reporting on the results of the evaluation of: the single HLT modules, STST systems and multimodal spaces for the second showcase.

Milestones and expected result 
M3 at m20: Evaluation of the single HLT modules, of the STST systems, and multimodal spaces for the first showcase
M6 at m36: Evaluation of the single HLT modules, of the STST systems, and multimodal spaces for the second showcase

Workpackage number :	W9 - Dissemination	
Start date or starting event:	m0	
Participant number:	P1	P2	P3	P4	P5			
Person-months per participant:	9	5	3	3	2			

Objectives 
• Plan adequate dissemination and exploitation actions.
• Publications in international congress
• Publications in national congress
• Publications in large diffusion journals
• Publications in scientific journals
• Participation to demonstration stands
• Set up and maintenance of the project Web site
• Set up an audio-visual presentation
• Set up a public demo
• Set up and manage two working groups, one targeting companies and the commercial/economic world, and the other addressing the scientific community.
• Organisation of a final event
• Develop policies for the protection of intellectual rights
• Plan exploitation policies

Description of work 

	

Deliverables 
D3 Project’s Web site
D4 Project presentation
D5 Dissemination and Use Plan
D19 Technical Implementation Plan
D20 Audio-Video Presentation

Milestones and expected result 
M6 at m36: Technical implementation plan

Milestone number	Milestone title	Milestone date	
M1	First showcase requirements	m6	
M2	Modules for the first showcase	m15	
M3	First showcase and its assessment	m20	
M4	Second showcase requirements	m24	
M5	Modules for the second showcase	m33	
M6	Second showcase and its assessment	m36	

Part C
Description of Contribution to EC Policies, Economic Development, Management and Participants
C1.	Title Page

Proposal full title			XX
Project acronym			XX
(Date of preparation)		 
Proposal number			 

C2. Content List (Part C only)

c2.	Content list (Part C only)	2	
		
c3. 	Community Added Value and Contribution to EC Policies	4	
		
c4.	Contribution to Community Social Objectives.	6	
		
C5.	Project Mangament 	7	
		
C6.	Description of the Consortium	13	
		
C7.	Description of the Participants	14	
		
C8.	Economic Development and Scientific and Technological Prospects	15	

C3.	Community Added Value and Contribution to EC Policies
C4.	Contribution to Community Social Objectives
C5.	Project management
The main goals of the management of this project are:
-	to organise the project as a whole, initiate its different activities, perform all the necessary administrative tasks; maintain the contacts and report to the Commission, the NSF and the partners;
- 	to supervise the technical progress of the project, and give technical advice;
-	to assure that the project adheres to its scientific goals and satisfies user needs, in order to achieve a high exploitation potential of the project’s results.
An important fact to be considered is the “transatlantic” nature of the NESPOLE! project. Thus the consortium will have to report to two funding agencies, the EC and the NSF, which differ in many important respects: cost monitoring practices, overall control on projects achievements, etc. At present, it is not clear how the two funding agencies are going to harmonise the requirements for jointly funded projects. This suggests a rather conservative approach to management organisation. Thus, the management structure we are going to describe is very much European-like; we are ready, however, to modify it to comply with updated managerial guidelines from the EC and the NSF. 
Management of the project will be assured by the partners through the following structures:
Administrative Director
Technical Directors (PD)
Project Management Committee (PMC)
Project Technical Committee (PTC)
Project Exploitation Manager (PEM)
Project Managers
Workpackage leaders
In addition to the classical structure defining managerial and technical committees, we add the following concerns:
* Knowledge protection;
* Exploitation preparation;
* Dissemination of results;
* Quality Assurance;
* Establishing and interacting with a User Group.
PROJECT ADMINISTRATIVE DIRECTOR
The Project Administrative Director will be responsible for:
*	chairing the PMC;
*	preparing and managing the management reports;
*	handling all communications with the EC and the NSF;
*	monitoring project costs;
*	creating and maintaining the conditions necessary for successful and effective collaboration;
*	proposing and implementing the quality assurance procedures;
*	creating and coordinating the user groups;
*	planning and implementing the project's contribution to IST/HLT project clusters;
*	representing the project (or delegating the project representation to the appropriate project staff) in the occasion of cluster events and meetings.
PROJECT TECHNICAL DIRECTORS (PROJECT COORDINATORS)
Given the important role of the US partner, the Consortium will have two Technical Project Directors, one for the European side and the other for the US one; they will be nominated by the (European) Prime Contractor, and by the US partner, respectively. They will be responsible for:
*	co-chairing the PTC meetings;
*	managing the progress reports;
*	monitoring the time schedule and the timing of the related activities;
*	recommending appropriate actions to rectify delays;
*	ensuring that all project deliverables are available on time;
*	ensuring that all the resources consumed in the performance of the work are actually relevant to the specific work involved;
*	representing the project (or delegating the project representation to the appropriate project staff) in concertation with other scientific events.
The coordinating partner will operate a Project Secretariat office for the duration of the project. The office will support the project by:
*	maintaining a central archive of all documents produced within the project;
*	distribution of information inside and outside the project;
*	maintaining the Project Plan and producing consolidated reports on efforts, results, schedule, and resource consumption.
PROJECT MANAGEMENT COMMITTEE (PMC). This committee will be formed by one key person of each full contracting partner (Project Managers) involved in the project and by the Project Exploitation Manager. The role of the PMC is to:
*	assist the Project Directors when carrying out their duties;
*	make sure that the activities and results thereof conform to the proposed quality standards;
*	approve all official deliverables;
*	approve all significant changes in the project workplan;
*	approve the Exploitation Plan;
*	establish Knowledge Protection policies;
*	assign specific responsibility to the most suitable partner representative, when new events require it.
Meetings of the PMC could be, also, attended by representatives of the User Group if needed.
CONFLICT RESOLUTION. The decision making procedure is organised as follows: each full contracting partner has a vote. Decisions will normally be taken by seeking consensus. For controversies the majority vote will be chosen.
PROJECT TECHNICAL COMMITTEE (PTC). This committee will be formed by one representative of each full contracting partner (Technical Project managers). The role of the PTC is to:
*	monitor the technical direction of the project;
*	approve all major technical decisions;
*	propose to the PMC reviewing and/or amending of the workplan;
*	propose to the PMC reviewing and/or amending the cost or time schedule under the EC Contract;
*	propose to the PMC reviewing and/or amending the termination of the EC Contract;
* 	lay down procedures for publications and press releases with regard to the project.
PROJECT EXPLOITATION PLANNING MANAGER (PEM). The PEM will be responsible for coordinating the overall project exploitation planning strategies and actions. He will, also, coordinate the preparation of a detailed Exploitation Plan. This Plan will be defined throughout the project in order to be able to support, effectively, the project operation and exploitation phase.
USER GROUPS (UG). Two User Groups will be created and actively involved in the following project activities: (i) user requirements; (ii) system validation; (iii) system demonstration, and (iv) system exploitation. The Project Administrative Director will be directly responsible for creating and coordinating these two user groups.
ADMINISTRATIVE PROJECT MANAGERS. Each contracting organisation will appoint an Administrative Project Manager (APM). All official communications will be addressed to him. He will attend the PMC meetings and also liaise with it to ensure the alignment between the organisation’s objectives and the direction of the project. He will also be responsible for ensuring that the organisation provides resources to the level specified in the project. In addition, he will provide to the project Director all the needed information regarding his organisation for the preparation of the management reports.
TECHNICAL PROJECT MANAGERS. Each contracting organisation will appoint a Technical Project Manager (TPM). He will be responsible for ensuring that the organisation respect the planned schedule, both with respect to activities and their results. He/she will provide to the project Director all the needed information regarding his organisation for the preparation of the advancement reports.
WORKPACKAGE LEADERS. The Workpackage leader is responsible for the coordination of the activities carried out by his Workpackage. He reports to the Project Technical Committee.
INFORMATION FLOW. The preparation process of a deliverable is the following: a project deliverable is prepared under the responsibility of the person appointed by the responsible organisation for a specific Task. The deliverable is sent to the project Director, who submits it PMC to the Commission.
Reports
Periodic Reports. The Coordinator (ITC-irst) will supply a full report on a quarterly, semestrial and/or annual basis, detailing the progress of the work, any problems encountered, actual expenditures (of money and manpower) versus plan, and plans for the coming year.

Annual Public Reports. Whereas the previous report is for use by the Commission services for project monitoring, a public report is due for each calendar year, for broad information purposes. The aim is to document the main results obtained and promote the objectives of the project so as to invite the outside world to make contact with the consortium. Designed for Web publishing, in HTML format, based on a template to be provided by the Commission.

Final Report. The Final Report, covering all the work, objectives, results and conclusion, will be prepared in a form suitable for publication. It will include sufficient information on new developments to enable third parties in the Union and Associated States to become aware of opportunities to request a licence for the technology developed within the project. In case the consortium needs to provide confidential information so as to give a complete picture of the work, this may be given in a confidential annex, or in a non-public version of the report.

Technology Implementation Plan. NESPOLE! will provide, along with the Final Report, a Technology Implementation Plan (TIP) which shall indicate all potential foreground rights and exploitation intentions, including a timetable for exploitation.

Table: Overview of reports 
*Int.	Internal circulation within project and Commission Project Officer only
  Rest.	Restricted circulation list (specify in footnote) and Commission PO only 
Rep. No.	Report Name	WP No.	Lead participant	Del. type	Security*	Delivery
(project month)	
R1	Progress reports	1	ITC-irst	R	Int	Semestrial	
R2	Annual Reports	1	ITC-irst	R	Pub	Calendar years	
R3	Progress Report	1	ITC-irst	R	Int 	Quaterly	
R4

	Final Report	1	ITC-irst	R	Pub	Project end	

  IST	Circulation within IST Programme participants
  FP5	Circulation within Framework Programme participants
  Pub.	Public document
NB: Templates will be provided by the IST programme.

C6.	Consortium description

NESPOLE! will be pursued by a consortium including leading European and American institutions, as well as users and technology providers. The consortium consists of: ITC-irst, UJF, University of Karslruhe (UKA),  Carnegie Mellon University-ILT, Azienda per la Promozione Turistica di Trento (APT),
The scientific partners of NESPOLE! — ITC-irst, UJF, University of Karlsruhe (UKA), Carnegie Mellon University-ILT — are amongst the major players in the speech and natural language processing community. All these institutions have been co-operating since 1994 within an international consortium for spoken translation: C-STAR II. For this reason, they have a long and well-established experience of joint work and of successful cooperation. Interestingly, the expertise acquired by the partners while cooperating inside C-STAR II has extended to such languages as Korean and Japanese, strengthening the position of the NESPOLE! consortium with respect to multilinguality. Such a background will provide a firm basis for the project, reducing the risk of failures due to cooperation problems, lack of understanding of other partners goals, and diverging work methodologies. The international relationships of the consortium will be important when setting up the user group, and in the dissemination and exploitation phases.
The expertises of the scientific partners of NESPOLE! nicely complement themselves. Indeed, despite the long previous cooperation they have developed different approaches to STST which will prove of the greatest importance for the present project. Moreover, each scientific partner will address its own language, by agreeing to develop all the HLT modules specific for its mothertongue. 
Starting from a shared methodology and technology the project will involve both users —Azienda per la Promozione Turistica di Trento (APT) and— and a technology provider — itself— in order to develop and demonstrate the feasibility and advantages for e-commerce, e-service of human-to-human communication through translation in a rich, multimodal environment. 
APT is a typical “tourism destination”. Information Technologies and the Web are introducing far reaching changes in tourism: destinations will be able to manage directly most of the segments of customer marketing. The trend is similar to other services, i.e. flight ticketing, telemarketing and so on. Such reorganisation is centred on the set up of call centres integrated with the Web.
AETHRA, a medium-size enterprise involved in the production and commercialisation of video-conference equipments and an e-commerce/service supplier, will play the role of technology provider and user at the same time. As a matter of fact, will provide both technology for video-conference and the relevant expertise in video-call centre supply. Presently, their services are addressed to the company’s own users: help-desk and tele-training are the main services, diffused all over the world: South America, Asia, Europe and North America. manages also services in outsourcing for other companies.
In addition to these partners, which form the NESPOLE! consortium, a user-group will be set up since the beginning of the project, to act as a: consultant for strategic choices, exploitation, external validation, dissemination.
The role of the various partners is as follows: ITC-irst, UKA, CMU-ILT and UJF will provide the human language technologies for Italian, German, English and French languages, and the technologies for multimedia and multimodality. APT will provide requirements of a typical destination in tourism, and will play a major role in validation. will provide both the video-call centre technology and the user expertise of a help-desk managing organisation on a global level. will also be involved in the validation activities. In conclusion all the partners have a well defined role, making the best out of their expertise, specialisation and interests.

C7.	Description of the participants
Participant’s name: 	Istituto Trentino di Cultura - Istituto per la Ricerca Scientifica e Tecnologica (ITC-irst)
Participant’s address:	via Santa Chiara
				38100 Trento, Italy
Director:			Oliviero Stock
The Istituto Trentino di Cultura (created 1962 by the Autonomous Province of Trento) has as its objective both scientific excellence and innovation and technology transfer to companies and public services. In its areas of competence, ITC collaborates with the main actors in world-wide research and it works in syntony with the European Union Programs. The total budget is currently about 17 M Euro.
Research activities are carried in scientific and technological areas, advanced computer science, microelectronics, physics, mathematical sciences and in human sciences. 
ITC-irst, ITC Centre for Scientific and Technological Research, is a point of reference in the international scientific community and, at the same time, a hub for the development of technologies and applications with social and economical impact. Personnel at IRST is about one hundred people on a permanent basis, and about 50 people on “soft” money.
Altogether ITC-irst budget amounts to about 10 MEuro. Half of ITC-irst direct costs are covered by industrial contracts and European and National contracts. So far over 40 European contracts of diverse kind have been carried on by ITC-irst.
A substantial portion of ITC-irst activities are in information technology (mostly in user -friendly and intelligent systems), with projects organised in three Divisions. Other areas of activity are microsystems (facilities include a clean room, the speciality is innovative microsensors), and in some applied physics areas. Altogether the activity is organised in six Divisions: Interactive Sensory Systems (ISS), Cognitive and Communications Technologies (CCT), Automatic Reasoning Systems (ARS), Medical Biophysics (MBP), Microsystems (MS), Physics-Chemistry of Surfaces and Interfaces (PCS) and a Tele-medicine Laboratory (TeleMed). CCT and ISS are directly involved in the present proposal. Altogether about 40 scientists and some 15 junior researchers are involved in the two divisions.
Research activities of CCT: Natural language-based dialogue; automatic generation of texts and spoken utterances; information extraction from texts; development and maintenance of linguistic resources; multimedia and multimodality. 
Research activities of ISS: video and document analysis and indexing by content, spoken language technology for translation, audio archive management, telephone-based applications and noisy environment; automatic learning; software architectures.
The CCT and ISS divisions have been cooperating for a long time on integrating their technologies and approaches towards common objectives 
Relevant European project references
ITC-irst has been active in many successful Third, Fourth and Fifth Framework projects, in some case with the role of coordinating partner. Among the relevant ones:
FACILE (LE), GIST (LRE); HIPS (Esprit), SPEECHDATCAR (LE); SPEEDATA (LE); TAMIC (MLAP), TAMIC-P (LE), TRANSTERM (LRE), VODIS2 (LE), CHARADE (Esprit); CARICA (Esprit), NESPOLE (5th FP), M-PIRO (5th FP), VICO (5th FP), ECO (5th FP)
Key persons
Gianni Lazzari. Degree on Electronic Engineering in 1977. From 1985 involved in a research activity in the field of Artificial Intelligence, mainly spoken language systems and intelligent systems. Involved in many internal, industrial and European projects implementing intelligent systems, like automating reporting by speech, spoken data entry and mobile robots for services applications. Presently, he is vice-director of ITC-irst and head of Interactive System Division. Responsible, from 1994 for the research strategy and the management of the division Consultant for the Autonomous Province of Trento since 1984 for technical/scientific evaluation of industrial research projects and technology transfer project. Evaluator for HLT projects of the EU. His research interests are now in the field of multilingual multimodal human machine interaction. During the last years he led the Italian side of the international project on speech-to-speech translation: C-STAR
Fabio Pianesi. Degree in Psychology in 1980. Specialisation in Computer Science in 1986. He has been with ITC-irst since 1988, in the Cognitive and Communication Technologies (CCT). He is a Senior Research Scientist, and since 1997 he is serving as acting head of CCT. He has been involved in many EC-funded projects, acted as manager of GIST-LRE and is technical director of NESPOLE! (5th FP). He coordinated the activities on the Italian generator for C*-II. His research interests focus on formal linguistics (syntax and semantics), computational linguistics and natural language processing (generation, parsing, linguistic resources), cognitive sciences. He is the author and co-author of many works on linguistic theory, computational linguistics, formal ontology.

Participant’s name:	University of Karlsruhe - Interactive Systems Laboratories
Director:			Alex Waibel
Participant’s Address:	Universität Karlsruhe, Fakultät für Informatik
				76131 Karlsruhe, Germany
Phone:				+49-721-608-4730
FAX:				+49-721-60-7721

The Interactive Systems Laboratory (ISL) at the University of Karlsruhe has maintained a strong and tight collaborative relationship with its sister laboratory at Carnegie Mellon University for almost a decade. The labs are organised as twin laboratories located in both places, at University of Karlsruhe and at Carnegie Mellon. The collaboration is facilitated by the fact that both laboratories are directed by Prof. Waibel, a Professor at both institutions who works and lives in both places. A benefit of this joint arrangement has been the effective joint design of speech translation systems in the European and US American contexts, as well as extensive experience with Multimodal, Multimedia delivery and manipulation of information across sites. The laboratories already work and live day by day in a distributed and in a multilingual fashion. Projects frequently span across the two sites and researchers are used to operating in a distributed fashion. We frequently hold joint videoconferenced seminars and lectures in which students and researchers on both sides participate. It comes natural therefore for the labs to collaborate effectively and to pursue research on improved multilingual and distributed human-human workspaces. We believe that this multinational constellation of our labs is unique in Europe and the US. Around our lab, University of Karlsruhe provides an excellent environment for this research. The Computer Science Department (of which ISL is a part) was ranked as the best CS department in Germany for three years. It maintains an excellent computing infrastructure and attracts some of Germany's best students. The lab and the University provide a perfect background for the proposed research by way of already existing human resources, equipment, language skills, expertise and experience in multilingual, multimodal cross-continental and cross-cultural research.
The ISL at the University of Karlsruhe is also one of Germany's leading speech and language research centres. It has been one of the founding members of C-STAR, the international Consortium for Speech Translation Advanced Research. The first German-English-Japanese speech translation system (JANUS) was developed there in 1991. The consortium was then expanded and joined by other European partners, ITC-irst and UJF for inner-European speech translation collaboration. All three partners currently form the European team in C-STAR which has developed a first inner-European spontaneous-speech German-English-Italian-French translation arrangement. The ISL has also been instrumental in the organisation of Verbmobil a national speech translation initiative in Germany that focuses on German-Japanese-English translation in the travel domain. This activity has further broadened the labs' reach toward robust spoken translation in Germany.
The JANUS speech translation system was one of the first systems to demonstrate (in ‘91) that speaker- independent, continuous speech-to-speech translation is possible. After initial versions limited in vocabulary and speaking style, JANUS-III has been developed 93-99, which handles ill- formed spontaneous, conversational spoken dialogues and an open vocabulary but in restricted domains of discourse. The robust parsing and language processing algorithms provide robust spoken language understanding and dialogue processing also for human-computer applications (database queries, car navigation, etc.), in the presence of recognition and speaking errors. Our efforts in large vocabulary (60,000+ words) speech recognition aim at greater robustness, rapid deployment and application to new domains and languages (current languages: English, German, Spanish, Korean, Japanese, Chinese (2), French, Italian, Portuguese, Swedish, Serbo-Croatian, Russian, Turkish, Arabic, Tamil). Another emphasis is our effort to handle sloppy, conversational speech under noise and/or cross-talk, such as in the car, on the telephone or in conference rooms. The JANUS recognition toolkit was applied and ranked first in the official ‘96 and ‘97 DARPA Hub-5 benchmarks (conversational telephone speech) and all official German Verbmobil benchmark tests in ‘94, ‘95, ‘96, and '98.
The laboratories also work on improving human-computer interfaces by processing and combining multiple communication modalities known to be helpful in human communicative situations. Among others, we seek to derive a better model of where a person is in a room, who he/she might be talking to, and what he/she is saying despite the presence of jamming speakers and sounds in the room (the cocktail party effect). We are also working to interpret the joint meaning of gestures and handwriting in conjunction with speech, so that computer agents can carry out intended actions more robustly and naturally and in more flexible ways. One particular focus is error repair, permitting the system to respond efficiently to a user's corrections and change of mind. Several human-computer interaction tasks are explored to see how automatic gesture, speech and handwriting recognition, face and eye tracking, lip-reading and sound source localisation can all help to make human-computer interaction easier and more natural. The base tools and software components already developed in these projects provide an excellent backdrop, from which we will be able to explore how effective multilingual and multimodal cross-cultural collaboration can be achieved.
Key persons
Alex Waibel. Prof. Waibel is a Principal Research Scientist at Carnegie Mellon University, Pittsburgh and University Professor of Computer Science at University of Karlsruhe (Germany). He directs the Interactive Systems Laboratories at both Universities with research emphasis in speech and handwriting recognition, language processing, speech translation, machine learning and multimodal and multimedia interfaces. At Carnegie Mellon, he has also served as Associate Director of the Language Technology Institute, Director of the Language Technology PhD program, and on the steering committee of the Human Computer Interaction Institute (HCII). Dr. Waibel was one of the founders of C-STAR, the international consortium for speech translation research and now serves as its chairman. He is also co-director of Verbmobil, the German national speech translation initiative. His lab's efforts in speech recognition and speech translation led to the development of JANUS, one of the first, and most advanced speech-to-speech translation systems to date. He was awarded the IEEE best paper award in 1990 and the Alcatel SEL research prize for technical communication in 1994. Dr. Waibel received his B.S. in Electrical Engineering from the Massachusetts Institute of Technology in 1979, and his M.S. and Ph.D. degrees in Computer Science from Carnegie Mellon University in 1980 and 1986.
Ivica Rogina was born on Sep. 29th, 1964, in Zapresic (Croatia). Since 1969 he has been living in Germany. 
Education:  1975-1984: Gymnasium in Philippsburg; 1985-1990: studied computer science at Karlsruhe, (Diploma Nov. 5, 1990); June 1997: PhD in computer science at the University of Karlsruhe. Work experiences: Nov. 1990 - Mar. 1991: teacher of theoretic computer science classes at Karlsruhe; Apr. 1991 - Aug. 1991: porting of Janus to German in Karlsruhe; Aug. 1991 - Aug. 1992: research programmer at CMU, Pittsburgh; since Sep. 1992: working on the Janus acoustic modelling in Karlsruhe.

Participant’s name			Carnegie Mellon University
Participant’s address		Carnegie Mellon University
					School of Computer Science
					Pittsburgh, PA 15221
					Phone: +1-412-268-7676
					FAX: +1-412-268-5578
Located in Pittsburgh, Pennsylvania, Carnegie Mellon is a national research university of about 7,500 students and 3,000 faculty, research and administrative staff. The university consists of seven colleges and schools, the Carnegie Institute of Technology (engineering), the College of Fine Arts, the College of Humanities and Social Sciences, the Mellon College of Science, the Graduate School of Industrial Administration, the School of Computer Science and the H. John Heinz III School of Public Policy and Management. 
Carnegie Mellon's School of Computer Science (SCS) is one of the top ranking academic organisation in the US devoted to the study of computers. Its four degree-granting departments --- the Computer Science Department, Robotics Institute, Human-Computer Interaction Institute, and Language Technologies Institute --- include over 200 faculty, 300 graduate students, and a 200-member professional technical staff. Two new units, the Centre for Automated Learning and Discovery and the Entertainment Technology Centre, opened in 1997. SCS also collaborates with other University Research Centres, including the DoD-funded Software Engineering Institute (SEI); the NSF-sponsored Pittsburgh Supercomputing Centre (PSC), the Information Networking Institute, and the Institute of Complex Engineered Systems (ICES). SCS has a reputation for developing innovative computers, devices, networks, and systems that benefit diverse applications. 
The Language Technologies Institute (LTI) is part of the School of Computer Science (SCS) at Carnegie Mellon University. Created in 1986 as the “Centre for Machine Translation”, its scientific scope broadened to encompass Computational Linguistics, Information Retrieval, Speech Recognition, Text Mining and related fields and is the largest degree granting research institute in language technologies in the US. Currently the LTI has about 15 faculty, 35 graduate students, and another 20-or-so researchers and visiting scholars. Active research areas in the LTI include:
	Machine Translation (text-to-text and speech-to-speech)
	Speech Recognition and Synthesis
	Information Retrieval (including Translingual IR)
	Robust Parsing Technologies
	Machine Learning Methods for text categorisation and mining
	Text Summarisation, Clustering, Analysis
	Language Tutoring
Major recent accomplishments include:
	First speech-to-speech MT system (JANUS)
	First industrially-deployed knowledge-based MT system (KANT)
	Birthplace of Lycos web-search/spider engines
	Translingual IR "Best Paper" in IJCAI-97
	Top-performance in multiple US. government evaluations (in text summarisation, speech recognition, ...)
Our research in speech translation can look back on a long and strong transatlantic cooperation with several European partners. We have carried out a very active joint research program between CMU and University of Karlsruhe since 1991 (by way of Dr. Waibel's dual (alternating) appointments at the Interactive Systems Labs with both Universities), and have integrated our speech translation systems with those of ITC-irst and UJF through collaboration under C-STAR.
Key Persons
Alex Waibel. Prof. Waibel is a Principal Research Scientist at Carnegie Mellon University, Pittsburgh and University Professor of Computer Science at University of Karlsruhe (Germany). He directs the Interactive Systems Laboratories at both Universities with research emphasis in speech and handwriting recognition, language processing, speech translation, machine learning and multimodal and multimedia interfaces. At Carnegie Mellon, he has also served as Associate Director of the Language Technology Institute, Director of the Language Technology PhD program, and on the steering committee of the Human Computer Interaction Institute (HCII). Dr. Waibel was one of the founders of C-STAR, the international consortium for speech translation research and now serves as its chairman. He is also co-director of Verbmobil, the German national speech translation initiative. His lab's efforts in speech recognition and speech translation led to the development of JANUS, one of the first, and most advanced speech-to-speech translation systems to date. He was awarded the IEEE best paper award in 1990 and the Alcatel SEL research prize for technical communication in 1994. Dr. Waibel received his B.S. in Electrical Engineering from the Massachusetts Institute of Technology in 1979, and his M.S. and Ph.D. degrees in Computer Science from Carnegie Mellon University in 1980 and 1986.
Lori Levin. Dr. Lori Levin is a Senior Research Scientist at the Language Technologies Institute at Carnegie Mellon University. She received a B.A. in Linguistics from the University of Pennsylvania in 1979 and a Ph.D. in Linguistics from the Massachusetts Institute of Technology in 1986. Since 1990 she has co-directed several projects including Pangloss (knowledge based translation of newspaper text), ALICE (computer-assisted instruction of Japanese), Enthusiast (machine translation of spoken Spanish dialogues in a limited semantic domain), Clarity (spoken dialogue understanding of Spanish), and the machine translation aspects of the JANUS project (multi-lingual spoken language translation). Dr. Levin has co-authored many papers about machine translation, linguistic theory, and computer-assisted language instruction.
Dr. Rober Frederking received a B.S. in Computer Engineering in 1977 from Case Western Reserve University, and an M.S. (1981) and Ph.D. (1986) in Computer Science (speciality in Artificial Intelligence) from Carnegie Mellon University. He is currently a Senior Systems Scientist in the Language Technologies Institute at CMU, and directs the DIPLOMAT project (rapid-deployment speech-to-speech translation). He was previously the system integrator and co-Principal investigator for the PANGLOSS project (translation of newspaper text) during which time he developed large-scale implementations of Example-Based Machine Translation and Multi-Engine Machine Translation.
Dr. Alon Lavie is a Research Scientist at the Language Technologies Institute at Carnegie Mellon University. He received his B.S in Computer Science from The Technion - Israel Institute of Technology in 1987, and his M.S. (1993) and Ph.D. (1996) in Computer Science from Carnegie Mellon University. Dr. Lavie is a faculty member of the JANUS speech translation group, a key participant in the C-STAR speech translation consortium. Together with Dr. Lori Levin, he oversees the development of the translation components of the JANUS speech translation system, and the integration of linguistic knowledge sources. With a particular area of expertise in parsing algorithms, he spearheads the research on robust language analysis using a combination of symbolic and statistical approaches. Dr. Lavie is a co-PI of Clarity, a DOD funded project on automatic identification of discourse structure in spoken language dialogues. Dr. Lavie teaches and advises graduate students at the Language Technologies Institute.

Participant’s name:		UJF
Participant’s address: 	UJF
				Laboratory 385
				rue de la Bibliothèque
				B.P. 53 38041 Grenoble
				Cedex 9 
				Phone: +33 4 76 51 46 34
				Fax: +33 4 76 44 66 75

Director:			Yves CHIARAMELLA 
				E-mail address: Yves.Chiaramella@imag.fr

The UJF laboratory is an academic institution linked to CNRS (Centre National de la Recherche Scientifique) and University Joseph Fourier at Grenoble (France). UJF deals with themes related to human-computer interfaces, interactive systems, multimedia systems, virtual realities. The research focuses are : 
1	The "Natural Languages, Translation and Dialogues" axis : natural languages as topic, and also as a communication vector for human-computer dialogue or automatic translation. Computer-based tools for the Automatic Processing of Natural Languages (C. Boitet). Speech and Dialogue (J. Caelen)
2	The "Interaction Systems" axis : interaction systems (multimodal interfaces, virtual realities, tele-presence, etc.) for specific contexts: (artistic creation, computer-aided design, etc.). Multi-sensorial interaction and representations (J. Coutaz, J. Caelen), Engineering for human-computer interaction (J. Coutaz)
3	The "Multimedia Systems" axis: multimedia systems (knowledge-based systems, information retrieval systems using natural languages, hypermedia systems, etc.) providing models and basic tools for Multimedia information retrieval (M.F. Bruandet); Hypermedia application design environments (J.P. Peyrin)
The research themes require academic research as well as understanding uses and needs (design, evaluation, software pre-development, etc.): thus the laboratory integrates an usability lab (MultiCom), which is open to the socio-economic world. 
European Project References
CATS (Computer Aided Theatrical Score: design special tools for speech processing in the theatrical area), IT IV, RTD project, Review report, CNR, 1996, Final Report CNR, 1998. SAM & SAM-A (Speech Assessment Methodology : automatic annotation of large speech corpora) Esprit I, II & III, BRA projects #1541 & 2589 & 6819, Final Reports, Multilingual Speech Input/Output Assessment, Methodology and Standardisation, UCL, 1989, 1991, 1993. AMODEUS II (Human-computer interface design and multimodal interface), Esprit III, BRA project #7040, Final Report, UCL, 1993. MULTIWORKS, Esprit II, R&D project, Final report, OLIVETTI, 1992. (MULTImedia WORKStation : design of a multimedia workstation including speech synthesis, speech recognition, human-machine dialogue in addition to standard multimedia functionalities).COCOS (Computer and Component Software), Esprit I, R&D project, Final report, BULL, 1989. 
Others Project References
UJF is partner in the C-STAR II project (International Consortium for Speech Translation)
Key persons
Hervé Blanchon (main correspondent for the project). Hervé Blanchon defended his Ph. D. Thesis in Computer Science on January 94 on the definition and study of the "Dialogue-Based Machine Translation for monolingual authors" concept. He focused on the global architecture of such systems and on the interactive disambiguation process. Then, he spent one year at ATR-ITL (Japan) as an associate researcher implementing and evaluating interactive disambiguation of English. He received the best paper award at NLPRS'95 for this work that has also been presented at IJCAI-97 and HCI-97. Since September 95 he is an Associate Professor at the University Pierre Mendès-France (Grenoble). He carries out his research, in the framework of interactive disambiguation and clarification, within the GETA team of the CLIPS lab. In 98, he took the lead of the CLIPS++ group (CLIPS, LIRMM, LATL, LAIP), a partner member of the C-STAR II consortium (ETRI, ATR-ITL, ITC-irst, CMU, UKA).

Jean Caelen. Born in 1947, Jean CAELEN is a CNRS senior researcher and vice-director of CLIPS laboratory. He manages since 1992, the French Man-Machine Communication Program supported by French government to stimulate scientific activities across French research laboratories in the domain of Man-Machine Communication (speech, natural language, vision and HCI). He participated as partner in COCOS Esprit I project, MULTIWORKS Esprit II Project, SAM Esprit I and Esprit II projects, CATS RTD, IT IV project and as associated partner in AMODEUS BRA Esprit II project. He is ACM, IEEE, ESCA, ACL, AFCET, AFIA member. The field of his scientific skill is automatic speech recognition and multimodal dialogue. His present thematic interest concerns automatic recognition in natural and multimodal spontaneous dialogue. He published about 130 papers in the domain of the various aspects of speech processing (recognition, understanding and dialogue).

Christian Boitet. Born in 1947, Ch. Boitet presented his State Doctoral Thesis (on several mathematical and algorithmic problems related to MT) in 1976. In 1977, he became associated professor of computer science at Université Joseph Fourier (Grenoble 1).. He is full professor since 1987. He has been in charge of several research contracts aiming at reaching the operational stage (Russian-French, 1980-86) and the industrial stage (French National MAT Project, 1981-87). He has also been involved in and/or in charge of GETA’s participation in several cooperative research efforts (pre-Eurotra (1978-82), Eurotra (1984-87), EuroLang research track on DBMT (1992-95), ATR-CNRS MIDDIM (1993-96), UNL (1996-), CSTAR (1996-)), and in bilateral cooperation programs (Canada (1971-72, 1982, 1985-87), Soviet Union (1973, 1975-), Malaysia (1979-), Thailand (1981-87, 95-), etc. Tchecoslovakia (1981, 1986, 1992), USA (1984-85), Morocco (1989-94), Hungary (1975, 1978, 1991-), China (1981-83, 87-)). He has been invited researcher at KDD Research Laboratories (1983) and ATR Interpreting Telephony Research Laboratories (1988, 1991, 1992-93 for a sabbatical year, 1994, 1995). His current interests include personal dialogue-based MT for monolingual authors (GETA’s LIDIA project and the UNL project), speech translation (CSTAR project), machine helps to translators and interpreters, portable & readable encodings for multilingual documents, integration of speech processing in MT, multilingual lexical data bases, and computer tools (specialised languages and environments) for lingware engineering and linguistic research. 
Jean-Philippe Guilbaud. As a CNRS Research Engineer, Jean-Philippe Guilbaud is a linguist and developer of MT linguistic applications. His know-how encompass methods and techniques of developing grammars and dictionaries required to perform automated analysis and synthesis (of French, German, English and Spanish), and use of Specialised Languages for Linguistic Programming allowing for declarative grammatical specification and description : Q-Systems, ARIANE-G5. He worked on the Dialogue-Based Machine Translation framework implementing a non-deterministic multisolution French analyser allowing for interactive disambiguation. He also developed grammars and dictionaries for generating French, German and English. He is currently working on a French analyser producing Interchange Format Structures from transcripts of dialogues in the domain of travel planning and scheduling for the CLIPS++ group in the C-STAR II consortium.
Laurent Besacier. He defended his PhD thesis in Computer Science in April 1998 on “A parallel model for automatic speaker recognition” at the University of Avignon (France). Then he spent one and a half year at IMT (Switzerland) as an associate researcher working on M2VTS European project (Multimedial Person Authentication). Since Sptember 1999 he is an associate professor at the University Joseph Fourier (Grenoble). He carries out research on automatic speech and speaker recognition, within the GEOD tem at CLIPS Lab. He published about 30 papers on various aspect of speech recognition. He is in the board of GFPC, a special interest group of ISCA.
Damien Genthial. Damien Genthial defended his Ph. D. thesis in Computer Science in January 91. He is an Associate Professor at the University Pierre Mendès-France (Grenoble). As a researcher in man-machine communication in the CLIPS Laboratory, he is an active member of the PILAF project of the TRILAN team: software linguistic tools for French written texts, working on tools for morphological parsing and generation, for syntactic parsing (construction of dependency trees), for detection and correction of errors. He is currently involved in the international consortium CSTAR-II.

Participant’s name:		AETHRA  S.r.l. 

Participant short name: 	AETHRA

AETHRA is an independent, privately owned, Italian company based in Ancona since 1972 and with offices in Rome, Milan, Bologna, Turin and Venice. There are also several world-wide branches: Inc. in Miami-Florida (USA), GULF in Dubai (United Arab Emirates), SUR in Santiago (Chile) and Beijing (China).
The leading European company in telecommunications and videoconferencing, we are engaged in the design, manufacture and distribution of a wide range of telecommunications products currently covering more than 85% of the Italian videoconferencing market and also establishing a strong presence on the international market with a network of partners and resellers already active in many countries. 
 AETHRA is the supplier of several Telecom Companies world-wide for Network Terminations (NT1, NT1PLUS, NT IP), Test Instruments as well as videoconferencing products and videosurveillance solutions.
A 25 year background and experience in telecommunications gives significant advantages over competing designers of video communications systems.
With a large installed base world-wide, we provide a full range of Customer support services for our products, either directly or through associated companies, to ensure a high level of customer satisfaction.
The facility in Ancona employs more than 350 people, a high percentage of whom are qualified to professional standards; all design, procurement, testing and quality control activities are carried out in-house to the highest standards. (In 1990 implemented a Quality Assurance system which has obtained the ISO 9001 certification).
 product line comprises three basic strands :
	(1) ISDN Products and Instruments (NT1, NT1 PLUS, NT IP, D1080)
	(2) Video and Multimedia Communications systems;
	(3) Videosurveillance.
ISDN Products and Instruments
For over 25 years, the core of the company’s activity has been the design and manufacture of network equipment and measuring instruments for telecommunication networks. The company is still playing an important role in this industry with the manufacture of ISDN network terminations, a basic product for the development of the ISDN infrastructure with several projects carried out in Italy, Belgium, Finland, Spain, Chile and others in collaboration with the main Telecom organisations. 
The standard NT1 is produced in significant quantities together with the NT1 Plus which adds to the features of the NT1 a dual interface to analogue terminal for connection to POTS networks (standard phones, fax G3 and so on). This is particularly useful in countries where ISDN is just being introduced as users are not forced to completely change the existing office equipment.
The recently introduced NT-IP combines all the features of the NT1 PLUS with an RS 232 interface to provide a fully digital, fast and reliable connection to a PPP terminal server such as an Internet Server Provider. In this case a fully digital data service is provided by the NT-IP straight to the COM port of a PC with an unrivalled performance and no additional hardware and software are required.
Test Equipment is also included in the portfolio for the testing of analogue and digital lines: D1080, an ISDN analyser for BRI, PRI and U Loop with superior performance and additional capabilities, and the latest D2000, with a new user friendly interface, pre-defined tests and protocol analysis.
Video and Multimedia communications systems
 is the only manufacturer that offer a complete line of H.320 and T.120 conform videoconferencing systems from the videophone and desktop to a wide range of rollabout solutions incorporating the most advanced multimedia applications and MCU capabilities. The systems fully meet the H.243 Multipoint criteria and have the capability to work at the highest quality also with alternative leased interfaces such as V.35-RS366, X.21, G.703, RS449.
The product line includes VEGA 384, the most affordable high quality and high performance set top solution on the market, MAIA videophones, the pioneer solutions in the single user segment, VOYAGER, a unique videoconferencing solution in a briefcase.
The rollabout low-end segment includes HERMES 384, the most cost-effective solution for small groups videoconferencing while the high –end segment is covered by a full line of single and dual monitor multimedia rollabout solutions.
Relevant European Projects: TELEINSULA, TELEREGION SUN2.
Key Persons
Roberto Giamagli. Degree in Computer Science at the University of Pisa 1984/1989. At since 1988. He served in AETHRA’s ‘R&D department for software development’ from 1988 till 1995. From 1996 up to present time, he has been serving as the head of the ‘System Engineering Department’ where he is co-ordinating several different activities including help-desk and post sales.
Marco Domizio. Degree in Computer Science at the University of Pisa 1984/1989. He is at since 1994. He is the Manager of AETHRA’s Service Centre (help desk - audioconference – multivideoconference)

Participant’s name	Azienda Provinciale per il Turismo (The Trentino Tourist Board)
Participant’s address
The Trentino Tourist Board/APT Trentino, established in 1987, by means of the provincial law L.P. 21/1986, is the provincial body in charge of promoting tourism in Trentino both at national and international level. In this capacity it assists and informs present and potential tourists using various tools such as information technologies and publications, advertising and fair stands, workshops, organisation of events etc. In order to promote and boost tourism in Trentino, the APT Trentino also conducts studies and research on the organisation and enhancement of the province’s tourist resources.
The Trentino Tourist Board forms, together with the other 15 local tourist offices in the province, a system which creates a communication flow at institutional level that can be defined as “make-oriented”, as well as a communication network at account level, namely “product-oriented”, focusing on products such as motorcycle tourism, gastronomic tourism, cultural tourism etc.
Key person
Ernesto Rigoni is the Director-General of the Trentino Tourist Board /APT TRENTINO.

C8.	Economic Development and Scientific and Technological Prospects

3. Participant list

Participant
name	Participant short name	Country	Status*	Date enter project	Date exit project	
 Istituto Trentino di Cultura - Istituto per la Ricerca Scientifica e Tecnologica		ITC - irst	 Italy	 C	 month 0	 month 30	
 University of Karlsruhe - Interactive Systems Laboratories		UKA	 Germany	 P	 month 0	 month 30	
 Carnegie Mellon University		CMU	 USA	 **	 month 0	 month 30	
 UJF		UJF	 France	 P	 month 0	 month 30	
 AETHRA S.r.l.		AETHRA	 Italy	 P	 month 0	 month 30	
 Azienda Provinciale per il Turismo (The Trentino Tourist Board)		APT	 Italy	 P	 month 0	 month 30	

*C = Coordinator (or use C-F and C-S if financial and scientific coordinator roles are separate)
  P - Principal contractor
 A - Assistant contractor
** The US research Institution Carnegie Mellon University (CMU) is expected to contribute to the project under a forthcoming NSF grant within the framework of the EU-US Science and Technology Cooperation Agreement, without being contractually part of the IST consortium. Their contribution is described in the workplan (chapter 9 of this Annex-I). A letter of intent to this effect is included in Annex-IV. Appropriate working agreements will be concluded in due course between the Consortium and CMU.

9.7 Provisional Meeting Schedule
Travels to conferences, workshops, exhibitions, etc. using NESPOLE! project funding should be coordinated on the project management level and outlined in the dissemination and use plan. Travel outside the EU member states (except to partners' sites) is subject to prior EC approval.

Kick-off Meeting			m0

Plenary Consortium Meetings:	m3, m9, m17, m23

Workpackage meetings: 		to be determined

User groups meetings:		m3, m17, m30

Provisional Conference List

ACL 2002, 2003, 2004
ICASSP 2002, 2003, 2004
COLING 2002, 2004
ICSLP 2002, 2004
EUROSPEECH ???
ICNLG 2000, 2002 ????
IJCAI 2003
IIIA ????
ENTER

11. Dissemination and exploitation
11.1 Dissemination and use plan
NESPOLE! is a research project whose main purpose is to show the feasibility of spoken language translation in the context of future services in the field of e-commerce and e-service. This will be accomplished by developing and testing with users and technology providers two showcases in the domain of competence of the users partners, i.e. tourism and video call centre for help desk.
The two user partners are already managing customer services through call centre. In the field of tourism APT manages every day many national and international telephone calls and Web visits. Specialised human operators and automatic operators take care of the call centre. Customers, however, are becoming more and more demanding; they want to know about the places they are going to visit, see how the landscape looks, explore new opportunities, etc.. Being able to satisfy such user needs means that a tourism destination can gain new market sectors and increase  competitiveness. For this reasons APT is going to install automatic operator functions, and is currently looking for new technologies to improve human-to-human communication. Video call centres with spoken language translation facilities will be tested during the project and the two showcases will be evaluated by APT. From the point of view of APT, the evaluation is not limited to assessing the quality of spoken translation technologies, but crucially extends to an understanding of organisational impacts in terms of number of calls managed, skills required by the operator, user interface (multimedia communication), user skills and behaviour, reorganisation of information services. At the end of the project APT will be in a position to decide how to exploit the project results for improving its own business.
Concerning video call centres supported help desk, AETHRA is a telecommunication company working in Videoconference technologies since 1989. Its activity focuses on the development of terminals and applications for videocomunication market, covering all the market sectors, from videophones through group-systems, conference rooms, multipoint units, vertical applications such as telemedicine, tele-teaching , design and management of big Service centres.
 AETHRA R&D department is actively involved in the development of solutions and technologies for video call centres. AETHRA is also one of the first users of video call centre. Moreover, it set up a video call centre facility in his central office in Ancona, Italy, in 1995, which now manages more than  50 internationals video calls per day. The video call centre consists of 8 operators enabled to manage audio,video, and data calls through ISDN and IP network. The services offered by the Video Call Centre are: customer service, technical support, telemarketing on videoconference products, demo and commercial assistance. Help desks, audioconferencing services, audiografic services and Multipoint service for videoconferencing are also available.
The market globalisation and the diffusion of the videoconferencing systems makes it necessary that they be capable of handling different languages through simultaneous translation. This will provide an important added value which will be complemented by more fully exploiting the already existing possibilities of using video, images and data as communication media. In this perspective, the results of NESPOLE! will be of the utmost importance to create news opportunities for the growth of video call centre market, and for improving the quality and diversity of the offered services. Thus, is firmly intentioned to exploit the results of NESPOLE!, this way improving its position over competitors.
ITC-irst is interested in exploiting the results of the NESPOLE! project, mainly in the field of spoken language technologies and multimodality. ITC-irst has a long experience in deploying research results in the field of HLT: speech recognition for dictation, call centre, data entry, etc.
11.2 Dissemination tasks
The NESPOLE! partners undertake to promote and publicise the project within a wider circle of interested IST actors encompassing enterprises, administrations, research institutions, professional associations, industrial and business groups, etc., vis-à-vis the general public, i.e. European consumers, the media, and the sponsors, i.e. European institutions including Commission and Parliament. To this effect, they will undertake the following tasks, each giving rise to a contractual deliverable:
establish a project website, covering objectives, approach, benefits, expected results, consortium details, events, etc. It will be available as elsewhere established, and updated in the course of the project, at least every 6 months. The website will host relevant public results from the project, including reports, presentations, research papers, press releases, audio-visual materials and software demonstrations, etc. to be added as they become available. 
produce a professionally designed, executable audio-visual presentation of the intended or realised project results, around project mid-term. This presentation is typically a movie object based on state of the art multimedia technologies, meeting the following requirements: targeted toward non-specialists, including business executives, decision and policy makers, and the media; presents the project result and weaves a “story” around it, linking the result with concretely felt needs and highlighting its potential for exploitation. It will comprise views of the demonstrator (see below) and will last approximately 3-5 minutes. It will be deployed on the Web, DVD, and CD-ROMs alike and will feature commentary in English, other language versions being the subject of determination by the consortium. 
provide a public demo version (“showcase”) of the project result, runnable on the Web, explaining the project objectives in technical terms. The demo will be a derivative of ongoing development work, for display on relevant websites and at technical forums, especially IST cluster events. The demo might take the form of animated screencams, with commentary overlaid in voice or textual roll-overs. Details to be agreed during contract negotiation. 
In addition, consortia will participate in conferences, workshops, seminars and other forums and events taking place at national, European or international level, deemed to be most relevant for the project’s information dissemination and awareness activities, on the basis of a rolling meeting plan communicated to the EC services through regular reporting.
In order to maximise the impact of the results, and gain precious insights and comments two user groups will be settled.
The first user group will consist of service providers from different fields : tourism, manufacturers corresponding to the target groups of NESPOLE!, publishers and multimedia companies, electronic sales, etc. The purpose is to stimulate users to think how their services, market reach and global business can be improved by the adoption of the technologies developed by NESPOLE!. At the same time, the consortium members will gain important insights and suggestions as to the requirements, solutions and performance of the showcases produced by the project. 
The second user group will consist of technology providers operating in the field of HLT, communication infrastructure and network coding. The goal of this user group is first of all to enlarge the number of languages involved in the project, by discussing peculiarities and requirements that could impact on the definition and deployment of the IRF. For these purposes, considerations of languages such as Chinese, Japanese, Korean, etc. will be crucial to guarantee that the delivered IRF will be able to adapt and cope with different linguistic and domain requirements. The second goal is to get advise about adopted solutions, with respect to such matters as coding of multimodal information  communication infrastructure, HLT technologies, etc..
The two user groups will play an important role with respect to dissemination and exploitation. As a matter of fact systems of the same class as NESPOLE! will be mostly useful for communication from and to European-like languages, and Asian and East European ones. Quite generally, future e-commerce development relies on the capabilities of reaching Japanese, Chinese, Korean, Russian, etc. markets, with important prospects for STST. 

 12. Clustering
Projects identified as participating in a cluster of research projects should detail their planned interactions with the other projects in the cluster, including a listing of these other projects, expected interactions/input/output, and detailed planning for the interaction including management plans/structure, along the following lines.
Inter-project cooperation in the IST programme is mainly realised through project clusters. Clusters are a paradigmatic way to improve focus and impact of RTD activities. They contribute to visibility and help to assess progress towards the cluster’s goal in a measurable way. Furthermore, clustering RTD activities will provide economy of scale through sharing and exchanging of tools, language resources, technologies, development platforms and architectures. Clusters will also contribute to improve the pace of progress by furthering standardisation and supporting a fast adoption of best practices. Lastly, by improving impact and visibility, and furthering standardisation and best practices, clusters will contribute to an improved dissemination and take-up of RTD results. 
Clusters will be set up by the EC services following the launch of the first batch of HLT projects. Coordination and support is expected to be provided by a dedicated action. Cluster activities are expected to cover a priori any of the following: Cluster network for information exchange; joint market analysis; joint technology watch; joint awareness campaigns; joint showcase; joint standards and best practice activities; joint language resources development/purchasing; common API/development platform(s)/system architecture; complementary or competitive development of technologies/applications; joint technology evaluation (possibly open to participants in other relevant clusters); joint verification; etc.
While detailed, agreed cluster workplans will only become available in the constitutional phase of clusters, the consortium commits itself to help prepare, discuss and adopt such workplans, to support and implement them, and to make sufficient project resources available. In particular, each consortium assigned to a cluster is to 
Actively participate in general cluster meetings (min. 2/year, typically by the technical coordinator).
Contribute and participate to cluster special events such as seminars, workshops, open days, etc.
Collaborate in designated work groups (typically by interested and competent RTD staff). 
Make available project-related information, including technical documentation, to other cluster participants and/or work group members. 
Contribute content to print and/or electronic cluster publications, demonstrations and showcases.
Support the designated cluster organs (coordinator and secretariat) in successfully implementing the agreed cluster activities and to respond to their requests. 
Report progress to the cluster coordinator.

13. Other contractual conditions
Protection of knowledge. The consortium intends to adequately protect the knowledge and technical result produced by requesting patents where relevant. To this end, the budget include 10KEuro.