Meteor Universal: Evaluation for Any Language

Meteor now supports language-specific evaluation for any target language for which there is enough data to build a standard phrase-based machine translation system. To build language-specific resources (paraphrase table and function word list), run the new_language.py script with your parallel data and a Moses-format phrase table:

python scripts/new_language.py out-dir corpus.f corpus.e phrase-table.gz [target-corpus.e]
Paraphrases will be extracted matching the target corpus (this can be a collection of relevant dev sets). If no target corpus is provided, the first 10,000 lines of the English corpus will be used (in practice this works adequately). Meteor can then be run with these files:
java -Xmx2G -jar meteor-*.jar test reference -new out-dir/meteor-files
Data should be pre-tokenized. Meteor will lowercase all data for evaluation (-new implies -lower). A universal parameter set will be used. These parameters are tuned on over 100,000 binary ranking judgments across 8 language directions and encode the following general properties: