RESULTS

Scores obtained are summarised for FR-FR, FR-IT and IT-FR translation. Only acceptable scores (p+k) and perfect scores (p) are detailed in % for each grader and for each dialog. 

The manual transcription of all the French four dialogs were concatenated into a single file called "4FrenchDialogsConcatenated.txt". For this file, the SDUs were obtained automatically from IF analysis. 
All the evaluation (FR-FR and FR-IT) were run with this file as source. The grade files were then separated in order to evaluate each dialog separately. 

Please note that sometimes, the total number of SDUs is not exactly the same on the "scores" files of different graders, due to graders who forgot an SDU. The scores for all dialogues are calculated with the number of SDU. 

FR-FR
3 graders : besacier, rossato, vaufreydaz

Scores on translation from manually transcribed file
%acceptable (%perfect)	Dial. A1 (01)
60 utterances	Dial. A2 (15)
74 utterances	Dial. C3 (10)
64 utterances	Dial. C4 (26)
37 utterances	All dialogues
235 utterances	
Grader 1 (lb)	75.22 (55.75)	54.84 (33.06)	51.04 (35.41)	70.27 (41.89)	62.40 (41.52)	
Grader 2 (sr)	74.10 (57.14)	60.97 (47.96)	56.25 (42.70)	72.97 (64.86)	65.76 (52.46)	
Grader 3 (dv)	68.14 (44.25)	51.67 (28.33)	45.83 (28.12)	65.75 (39.72)	57.46 (34.82)	
Average	72.48 (52.38)	55.82 (36.45)	51.04 (37.84)	69.66 (48.82)	61.87 (42.93)	

Scores on translation from automatically transcribed file (from ASR)
%acceptable (%perfect)	Dial. A1 (01)
60 utterances	Dial. A2 (15)
74 utterances	Dial. C3 (10)
64 utterances	Dial. C4 (26)
37 utterances	All dialogues
235 utterances	
Grader 1 (lb)	59.29 (41.59)	34.14 (19.51)	42.10 (31.58)	62.50 (40.27)	48.13 (32.25)	
Grader 2 (sr)	58.04 (46.43)	36.20 (25.86)	41.93 (35.48)	56.16 (50.68)	47.46 (38.58)	
Grader 3 (dv)	59.46 (38.74)	35.29 (17.64)	42.55 (26.60)	52.70 (35.13)	46.99 (28.99)	
Average	58.93 (42.25)	35.21 (21.00)	42.19 (31.22)	57.12 (42.03)	47.56 (33.27)	

FR-IT
2 graders : alvalitu (IRST), anneclaire

Scores on translation from manually transcribed file
%acceptable (%perfect)	Dial. A1 (01)
60 utterances	Dial. A2 (15)
74 utterances	Dial. C3 (10)
64 utterances	Dial. C4 (26)
37 utterances	All dialogues
235 utterances	
Grader 1 (al)	69.91 (50.44)	50.80 (41.13)	46.31 (40.00)	59.46 (50.00)	56.65 (45.07)	
Grader 2 (an)	65.48 (56.64)	52.84 (40.65)	51.04 (40.62)	69.18 (57.35)	58.75 (48.00)	
Average	67.69 (53.54)	51.82 (40.89)	48.67 (40.31)	64.32 (53.67)	57.70 (46.53)	

Scores on translation from automatically transcribed file (from ASR)
%acceptable (%perfect)	Dial. A1 (01)
60 utterances	Dial. A2 (15)
74 utterances	Dial. C3 (10)
64 utterances	Dial. C4 (26)
37 utterances	All dialogues
235 utterances	
Grader 1 (al)	53.27 (36.45)	33.91 (26.96)	39.08 (32.18)	60.56 (50.70)	45.52 (35.26)	
Grader 2 (an)	50.90 (43.64)	29.03 (25)	42.71 (35.42)	56.52 (53.62)	43.11 (37.59)	
Average	52.08 (40.04)	31.47 (25.98)	40.89 (33.80)	58.54 (52.16)	44.31 (36.42)	

Hypothesis* means translation eval from automatically (with ASR) transcribed sentences on these Hypothesis* files, 1 one grader (alvalitu) used the ASR hypothesis string as reference whereas the other grader (anneclaire) used the manually transcribed string as reference for translation eval

IT-FR
2 graders : anneclaire, federico

Scores on translation from manually transcribed file
%acceptable (%perfect)	ita024co3
22 utterances	ita060co1
33 utterances	ita806co1
10 utterances	ita911co1
25 utterances	All dialogues
90 utterances	
Grader 1 (fe)	35.82 (31.34)	40.27 (36.11)	30.00 (25.00)	47.50 (30.00)	39.20 (32.16)	
Grader 2 (an)	47.82 (40.57)	45.83 (38.89)	45.00 (30.00)	50.00 (35.00)	47.26 (37.81)	
Average	41.82 (35.95)	43.05 (37.50)	37.50 (27.50)	48.75 (32.50)	43.23 (34.98)	

Scores on translation from automatically transcribed file (from ASR)
%acceptable (%perfect)	REC_ita506co1
10 utterances	REC_ita511co1
25 utterances	All dialogues
35 utterances	
Grader 1 (fe)	15.00 (15.00)	37.50 (20.00)	30.00 (18.33)	
Grader 2 (an)	16.67 (16.67)	33.33 (23.08)	28.07 (21.07)	
Average	15.83 (15.83)	35.41 (21.54)	29.03 (19.70)