END TO END EVALUATION: SOME NOTES 
Erica Costantini
2002, October, 31

1. EVALUATION DATA COLLECTION DESIGN

1.1 LANGUAGES AND NUMBER OF DIALOGUES

FROM \ TO	ITA	ENG	FRE	GER	
ITA	Age ita-ita	Age ita-eng	Age ita-fre	Age ita-ger	
ENG	Cli eng-ita	Cli eng-eng	-	-	
FRE	Cli fre-ita	-	Cli fre-fre	-	
GER	Cli ger-ita	-	-	Cli ger-ger	

Is this table right? Are we going to evaluate the Italian agents turns towards all the four languages and the eng, ger, fre clients turns in monolingual situation and towards Italian?

Assuming that the answer to those questions is “yes”,  we will have to judge:
-	4 translations for each of the 4 italian dialogues                                => 16 dialogue files
-	2 translation for each of the non-italian dialogues (4 dial. * 3 lang.) => 24 dial. files
(the number of dialogues to grade is already high)

In case we will have to evaluate both on manually transcribed input and on actual speech recognition of the original audio, we will have:
-	4 * 4 (see above) * 2  => 32 dialogue files
-	2 * 4 * 3 * 2               => 48 dialogue files

32 + 48 =>  80 dialogue files

Finally, if we want all dialogues graded by three different judges and each of the judges evaluating all the 4 dialogues of each languages in all conditions (e.g. 3 bilingual eng-ita evaluating the 4 English translations into English and Italian, for both inputs) the number of  “judged dialogues” will be:
 
3 (judges for dialogue) * 80 (dialogue files) => 240 dialogue judgments

And if we want the judgments to be done in two different times (for intracoder agreement), the number of  judgments will be:

240 * 2 (times) => 480 dialogue judgments!!!!!

(… this is really too much!)

We can reduce the number of dialogues in different ways, e.g.:
-	using only one input (manually transcribed);
-	measuring intracoder agreement over the dialogues (among turns in the same grading session; there should be different ways of doing it) instead of measuring it over time;
-	using a non-complete design. See the tables below for examples.

	G1	G2	G3	
D1	+	+	+	
D2	+			
D3		+		
D4			+	

In this table each of the 3 bilingual (e.g. ita-eng) judges grades 2 dialogues:  one common dialogue, D1 plus another one (different among the others), both in monolingual situation (ita to ita) and crosslingual one (ita to eng). In this way we have one common dialogue graded by all the judges, for intercoder agreement. The number of judgments is much lower. One of the bigger problems of this design if that we cannot calculate majority vote per each dialogue.

	G1	G2	G3	G4	
D1	+			+	
D2	+	+			
D3		+	+		
D4			+	+	

In this other case we will not have a common dialogue, but each dialogue is graded twice per each condition (ita-eng, ita-ita,…).
Other designs are possible.
-	other ways for reducing the number???

Decisions should be based on the evaluation main objectives: there are a lot of different aspects we can focus on (e.g.: differences across conditions or across languages), and we should try to list our objectives starting with the most important until the less relevant before deciding.
In addition, we should know for sure if all the 4 languages are involved in the evaluation.

NB In any case, all the possible designs need BALANCING (or casual assignment) of the dialogues: each of the graders grading the same dialogues will have to grade them in different orders, to reduce the order effects and the experience effects.

1.2 DATA COLLECTION DESIGN PROPOSAL

1.2.1 ITALIAN DIALOGUES

ITALIAN	G1	G2	G3	
D1	+	+	+	
D2	+			
D3		+		
D4			+	

The judges are people with a good knowledge in all the 4 involved languages (almost “4-lingual”: it is not impossible to find 3 of them at the translators/interpreters school in Trieste).
Each of the judges grades 2 dialogues in all the 4 target Languages, all with manual transcriptions as input. This results in 8 dialogues per judge, which is already an high number of dialogues and we should find a way to pay the judges.

In case we want each judge to grade all the dialogues,  the number of dialogues to grade would be:
4 (dialogues) * 4 (languages) => 16 
I think that grading 16 dialogues is a very hard work…

Other designs could be considered.

1.2.2 ENGLISH, FRENCH AND GERMAN DIALOGUES

	G1	G2	G3	
D1	+	+	+	
D2	+	+	+	
D3	+	+	+	
D4	+	+	+	

In this case each dialogue has only 2 target languages. My suggestion is that we use three groups of judges, one for english-italian, one for german-italian and one for french-italian; each group is composed by three bilingual persons (best will be from the school for translators/interpreters), grading all the dialogues in both crosslingual and monoligual condition. The total number of graded dialogues per person will be 8 in this case (again: it can be faced, but only if we can pay the subjects). 

1.2.3 DESIGN PROPOSAL: comments and conclusions

The proposed design requires 12 paid persons willing to grade 8 dialogues each (plus training; at least 2 or 3 appointments per person). Some of the variables we would like to investigate cannot be included (e.g: different inputs) but there is no  way in my opinion to include other variables (and there is not way to have more appointments). It is already too much work for each grader. The only way for reducing further the amount of work is to have each partner working on its language, like last year. At least in my opinion…

2. TWO DISCUSSED ISSUES

2.1.GRADING

I think that both the used “three point” grading and the suggested binary grading schemes in its discussed version (decision on fluent/grammatical only for SDU’s whose meaning is preserved in traslation) have some problems.

THREE POINT GRADING
The problems concerning the three-point grading have already been discussed. They are mainly related to joint meaning/fluency judgment, and to collapsing 3 (perfect, good, bad) into 2 (acceptable vs. non acceptable), right?

BINARY GRADING
The suggested binary scheme does not solve all the problems. The main problem with this solution in my opinion is that in this case we are forced to ask the question about meaning before than that about grammar, and I think that the opposite order (grammar – meaning) would be better (it is the more frequent in psycholinguistic studies, as some colleagues told me, because it seams to ensure a greater independence between the two judgments). 
In addition for the aim of data description/analysis it would be much better to have information about all the 4 possible cells (see the following table).

		FLUENCY	
		YES	NO	
MEANING	YES	perfect	ok	
	NO	bad	?	

THE “THIRD WAY”
I think that one possible solution to the above mentioned problems could be having the two separate judgements for all the SDUs:
-	first the “grammar/fluency” question, 
-	then the “meaning” question. 
Perhaps this solution would introduce further problems, but I have not found them yet…
I would strongly suggest to consider this option.

2.2 AVERAGING SCORE VERSUS MAJORITY SCORES

I think that the decision could be postponed, and also that we could calculate both scores.