Study on Multimodality - Part 2
Research Plan Proposal

Erica Costantini
September, 6th 2002

1.	Research Objectives

The main objective of the “Study on Multimodality – Part 2” is to answer the following question:

  Which kind of interaction is supported by our multilingual, multimedia and multimodal system? 

The idea is to have (again) some novice users using the system to perform a task. Since for this study we don’t aim at testing hypothesis, there is no need to have an experimental plan to make comparisons between different conditions (as we did in the last year MM experiment – SO condition and MM condition). What we need is to describe in a detailed way “what is happening” at the communication level (language features, gestures used, speech-gesture integration patterns, communication effectiveness). With a non-experimental plan we are almost free to choose the number of dialogues to collect (if we had to run statistical tests to compare conditions, we would have had constraints concerning the minimum number of dialogues). 

2.	Research Plan

The research is a non-experimental user-study. The main features of the plan proposal are the following (see also table 1):

-	Low number of dialogues (5+1?) involving novice users (possibly paid - $10?) in the role of the customers and experienced APT agents;
-	Detailed transcription and annotation of the linguistic phenomena and of the dialogue (Nespole! conventions for words, syntax and spontaneous events - TransEdit tool; turns classification and dialogue effectiveness – see deliverable D5: "Study on Multimodality Part 1")
-	Goal-based (manual) annotation of gestures (see gesture annotation conventions in the Nespole! web site)
-	Deep analysis of the speech-gestures integration patterns.

The proposal is that all dialogues are collected using only one language pair and that they will be all ITA to ENG. 

The original audio should be disabled. The device for gestures should be the optical pen + tablet. 
We should avoid a push-to-talk modality, if possible  - let’s wait for the final results of the technical tests for LangTech (the same is for the decision concerning the video transmission).

The usability questionnaire will be circulated the week before the data collection. It will be similar to that of the monolingual data collection 2002 with some additional items concerned with the translation.

The task will be probably travel planning; the scenario is Winter holidays in Val di Fiemme, enriched with new developed topics (a bit enlarged with respect to that used last year). The instructions will be similar to those for the “Multimodal Condition” of the last year.

Question	Which kind of interaction is supported by our multilingual, multimedia and multimodal system? (dialogue description)	
Research Design	User study (non experimental design; no comparisons…)	
System version	Frozen Showcase 2A (with Multimodality)	
Language	Italian to English 	
Involved partners	-	Discussion: all
-	Data collection: ITC-irst, CMU, APT (only agents)
-	Transcriptions and annotations: ITC-irst; CMU (?)
-	Data analysis and deliverable production: ITC-irst; CMU?	
No. of dialogues	6 (5+1)	
Recording date	1st week October (2 dialogues per day, on Tuesday, Wednesday and Thursday)	
Participants	Travel agents: 3 actual (Italian) APT operators 
Customers: 6 (possibly paid - $10?) volunteers, uninvolved with Nespole! (and possibly uninvolved with speech labs or system designing, development or evaluation): students? People from administration?	
Recordings	Audio, video, maps, log files (a detailed list will follow)	
Transcriptions
& annotations	Level 2 + 3 + gestures (see table XXX):
-	Nespole! conventions for words, syntax and spontaneous events; TransEdit tool
-	Transcriptions alignment (orig_agent + syn_agent + orig_customer + syn_customer)
-	Annotation for turn success ? (successful, partially successful, non-successful turns) and for turn repetitions
-	discourse analysis (turn classification on the base of the turn function – e.g. using speech acts; ambiguities; illegal questions)
-	Gesture annotation (Nespole! conventions: manual annotation form videos) + partially automatized annotation???	
Push-to-talk?	No (if possible)	
Original audio	disabled	
Devices	-	Pen + tablet	
Video transmission	?	
Measured variables	-	disfluences
-	dialogue length
-	number of word tokens and word types
-	number of turns
-	number of gestures
-	type of gesture
-	speech-gesture integration patterns
-	turn success (?)
-	turn repetitions
-	turn function (speech acts?)
-	(ambiguities)
-	(illegal questions)
-	user satisfaction (questionnaire)	
Task	Scenario from MMexp 2001 + some new developed topics ?	
Questionnaires	-	New questionnaire (modified version of that used in the data collection 2002): asking about single features of the Nespole system	

Table 1. Research plan.
3.	
Recordings

A document with details concerning file to record and formats will follow. More or less they will be the same of those of the last year MM experiment.

4.	Transcriptions and Annotations

The possible transcriptions level are described in the following table.

Transcription levels	Description	
Level 0	just transcription of words	
Level 1	transcription with annotation for spontaneous events (Trans Edit) - .txt file	
Level 2	aligned transcriptions; annotation for turn repetitions and turn success (successful, partially successful, non-successful turns); ambiguities; illegal questions - .xls file?	
Level 3	discourse analysis (turn classification on the base of the turn function – e.g. using speech acts)	

Table 2. Possible transcription levels

Level one is referring to the transcription standards in Nespole!, which includes annotation of interruptions and spontaneous events (see Verbmobil and Nespole! conventions). For the aim of effectiveness and discourse analysis, level 1 is not necessary: we can in fact obtain levels 2 and 3 even from level 0 (but we would not obtain data concerning disfluences). 
In any case, level 1 is suggested as minimum level of transcription for the second part of the user study on multimodality, since a decision was taken last year in order to have the same format for all transcriptions in Nespole!. Levels 2 and 3 are referring to annotation of additional phenomena for a more complete dialogue description; for details concerning transcription alignment, turn repetitions, turn success, turn function, ambiguities, illegal questions see “D5 – Study on Multimodality, part 1”. 

Figure 1 is about the turn classification used for the last year MM experiment (details in D5).

Figure 1. Turn classification (MM experiment - summer 2001)

For the last experiment turns were classified by one single judge (me). We did not run any statistics on that. For the next study we can use the same classification. This would enable us to compare percentages for each category with the percentages of last year. We can in fact assume that a better system leads to higher communication quality and that higher quality means higher percentages of turns focussed on information exchange. 

Alternatively, we can have some “IF” experts annotating speech acts and we can find out some statistics concerning speech act frequencies (Lory’s proposal). We need to discuss further the matter, evaluating the resources we need and the possible outcomes for both alternatives (a final decision should be taken even after the data are collected). 

The proposal is that gesture annotation is done following the gesture annotation conventions in the Nespole! web site (http://nespole.itc.it/internal/multimodal/doc/gestannotation.doc).

Partners involved in transcriptions and annotations (proposal). The proposal assumes that  we choose to classify the turns on using the speech acts instead of the classification in figure 1.

Production of:	Partner	
Transcriptions and annotations of spontaneous events (.trl files, using TransEdit)	ITC-irst for Italian, CMU for English	
Gesture annotation 	ITC-irst (transcribers) for both languages (?)	
Aligned Transcriptions (single file with English and Italian original speech and synthesis)	ITC-irst (transcribers)	
Translation files (with all contributions in English and annotation for turn repetition and turn success)	ITC-irst (me + eventual judges for successfulness)	
Turn classification (using speech acts)	CMU	

Table 3. Possible transcription levels

5.	Schedule 

September:
11th (Wednesday): deadline for comments on this proposal
13th (Friday): circulating new proposal
20th (Friday): agreement on general plan and recordings 
27th (Friday): agreement task and instructions

Week: September 27th – October 4th: 
One day for dialogue tests, 3 days for data collection (2 dialogues per day). We should make some effort to be ready for this week, because probably the following week (October, 7th-11th) there could be some problems at irst.

N.B. September 26th and 27th: demo at LangTech in Berlin

6.	Additional human-to-human dialogues

Additional “human-to-human” dialogues could be collected to answer the following question: 

•	(how) is the interaction different form a “natural” human-to-human direct interaction?
We could collect additional 6 dialogues (same task) at the APT desk between an APT agent and a person acting as customer (without using any machine). 
This data collection extension is optional and would probably involve only ITC-irst and APT.