\subsection{Preliminary Experimental Results}\label{experiment}

The above algorithms have been implemented and run on a corpus of 141 noun phrases and sentences.  The training set consists of structurally relatively simple noun phrases and short sentences.  Two examples from the training set are "the very tall man" and "The family ate dinner." The training corpus was translated into German and word-aligned by hand.  A fully inflected target language dictionary as described above was used.  No other target language information was given. This section presents preliminary results that indicate that Seed Generation, Compositionality, and Seeded Version Space Learning can indeed effectively be used to infer transfer rules, and the resulting transfer rules can be used to translate, albeit not without making mistakes.  

We report results for both training set accuracy (i.e. the system was trained on all 141 examples and evaluated on all of them), and for each split of a 10-fold cross-validation.  
To evaluate translation power, we used an automatic evaluation metric. Namely, we simply compared the resulting translations against the reference translation.  If the grammar could be used to produce the exact reference translation, the sentence counts as correctly translated. No partial matching was done.  Note that this is a very harsh evaluation metric - it is well known that a sentence can correctly be translated into many different surface forms in the target language.  For this reason, MT evaluation generally performs user studies or matches partial outputs to reference translations.  In this test, our system was graded on a harder scale, while in future evaluations, we will consider other metrics as well.

Further, our system considers a sentence as correctly translated if one of the possible translations matches the reference translation. This set accuracy measure should be judged in conjunction with the number of possible translations per sentence (see Table~\ref{ResultsSummary}). For instance, if the grammar can produce the reference translation, but only as one of a hundred possible translations, then the system may still perform poorly, as it may not pick as its top translation the perfect translation.  The current version of the transfer engine does not rank outputs.  Therefore, it is important to report the average number of possible translations per rule in order to put into perspective the translation (set) accuracy. The above are the two metrics with the highest priority:  the system will always be measured by its translation accuracy (which is desired to be high) and by the average number of possible translations per sentence (which is desired to be low).  The results of this evaluation can be seen in Table~\ref{ResultsSummary} below.

\begin{table}
%\begin{tabular}{@{} |l|c|c| @{}} \hline
\begin{tabular}{|l|c|c|} \hline
\textbf{Evaluation Set} & \textbf{Translation Accuracy} & \textbf{Ave Num Possible Translations} \\ \hline
Training & 0.716 & 4.936\\ \hline
TestAve & 0.625 & 4.460\\ \hline
CVSplit1 & 0.4 & 2.533\\ \hline
CVSplit2 & 0.714 & 7\\ \hline
CVSplit3 & 0.714 & 3.286\\ \hline
CVSplit4 & 0.643 & 6.643\\ \hline
CVSplit5 & 0.714 & 2.357\\ \hline
CVSplit6 & 0.786 & 4.857\\ \hline
CVSplit7 & 0.786 & 3.214\\ \hline
CVSplit8 & 0.5 & 5.643\\ \hline
CVSplit9 & 0.429 & 2.929\\ \hline
CVSplit10 & 0.571 & 6.143\\ \hline 
\end{tabular}
\caption{Translation Set Accuracy and Number of Possible Translations for the Training Set as well as 10 Cross-validation Sets.}
\label{ResultsSummary}
\end{table}

Because of the simplicity and small size of the training set, it is clear that much work remains to be done, both in refining the algorithms and making them scale to more complex examples and to bigger sets of data.  The initial results, however, are promising and suggest that future exploration will very much be worthwhile. 