Meteor 1.5: Automatic Machine Translation Evaluation System

Code by Michael Denkowski

WebsiteGithub

Table of Contents:

1. About
2. Supported Languages
3. Running Meteor
4. Meteor Options
5. Standalone Meteor Aligner
6. Standalone Word Stemmer
7. Integrating Meteor with your Software
8. Meteor X-ray
9. Training Meteor Parameters
10. Meteor Universal: Evaluation for Any Language
11. Special Thanks

1. About

Meteor consists of two major components: a flexible monolingual word aligner and a scorer. For machine translation evaluation, hypothesis sentences are aligned to reference sentences. Alignments are then scored to produce sentence and corpus level scores. Score and alignment information can also be used to visualize word alignments and score distributions using Meteor X-ray. For detailed information on Meteor word alignment and scoring, see Denkowski and Lavie, 2011. This paper also details the flexible matching support that allows Meteor to align words and phrases with differing surface forms.

This release includes the following software:

Meteor is released under the GNU Lesser General Public License (LGPL) and includes some files subject to the (compatible) WordNet license. See the included COPYING files for details.

2. Supported Languages

Language support is divided into two groups. Fully supported languages include flexible word and phrase matching (at least one type of match other than exact) and language-specific parameters tuned to maximize correlation between Meteor scores and human judgments of translation quality. Partially supported languages include flexible word matching and use language-independent parameters chosen to generalize well across known languages.

Fully supported languages:

LanguageExact MatchStem MatchSynonym MatchParaphrase MatchTuned Parameters
EnglishYesYesYesYesYes
ArabicYesNoNoYesYes
CzechYesNoNoYesYes
FrenchYesYesNoYesYes
GermanYesYesNoYesYes
SpanishYesYesNoYesYes

Partially supported languages:

LanguageExact MatchStem MatchSynonym MatchParaphrase MatchTuned Parameters
DanishYesYesNoNoLI
DutchYesYesNoNoLI
FinnishYesYesNoNoLI
HungarianYesYesNoNoLI
ItalianYesYesNoNoLI
NorwegianYesYesNoNoLI
PortugueseYesYesNoNoLI
RomanianYesYesNoNoLI
RussianYesYesNoNoLI
SwedishYesYesNoNoLI
TurkishYesYesNoNoLI

Paraphrase capability can also be added to unsupported languages. If your MT system has a bilingual phrase table, you can use Parex to build paraphrases tables and use them with Meteor. For example, if you want to evaluate a system that translates into Danish and build a paraphrase table named paraphrase-da.gz, you can use:

java -Xmx2G -jar meteor-*.jar test reference -l da \
-a paraphrase-da.gz -m 'exact stem paraphrase' -w '1.0 0.5 0.5'
This tells Meteor to use the paraphrase table (-a paraphrase-da.gz) add the paraphrase module (-m 'exact stem paraphrase') and add a weight for paraphrases (-w '1.0 0.5 0.5').

Other Languages:

Meteor is capable of scoring UTF-8 encoded data for any language. Specifying language "other" will automatically select exact matches only for alignment and language-independent scoring parameters. Remember to pre-segment, tokenize, and lowercase text as needed.

java -Xmx2G -jar meteor-*.jar test reference -l other

3. Running Meteor

To call Meteor, run the following:

java -Xmx2G -jar meteor-*.jar
Running Meteor with no arguments prints the following help message:
Meteor version 1.4

Usage: java -Xmx2G -jar meteor-*.jar <test> <reference> [options]

Options:
-l language                     Fully supported: en cz de es fr ar
                                Supported with language-independent parameters:
                                  da fi hu it nl no pt ro ru se tr
-t task                         One of: rank util adq hter li tune
                                  util implies -ch
-p 'alpha beta gamma delta'     Custom parameters (overrides default)
-m 'module1 module2 ...'        Specify modules (overrides default)
                                  Any of: exact stem synonym paraphrase
-w 'weight1 weight2 ...'        Specify module weights (overrides default)
-r refCount                     Number of references (plaintext only)
-x beamSize                     (default 40)
-s wordListDirectory            (if not default for language)
-d synonymDirectory             (if not default for language)
-a paraphraseFile               (if not default for language)
-f filePrefix                   Prefix for output files (default 'meteor')
-q                              Quiet: Segment scores to stderr, final to stdout,
                                  no additional output (plaintext only)
-ch                             Character-based precision and recall
-norm                           Tokenize / normalize punctuation and lowercase
                                  (Recommended unless scoring raw output with
                                   pretokenized references)
-lower                          Lowercase only (not required if -norm specified)
-noPunct                        Do not consider punctuation when scoring
                                  (Not recommended unless special case)
-sgml                           Input is in SGML format
-mira                           Input is in MIRA format
                                  (Use '-' for test and reference files)
-vOut                           Output verbose scores (P / R / frag / score)
-ssOut                          Output sufficient statistics instead of scores
-writeAlignments                Output alignments annotated with Meteor scores
                                  (written to <prefix>-align.out)

Sample options for plaintext: -l <lang> -norm
Sample options for SGML: -l <lang> -norm -sgml
Sample options for raw output / pretokenized references: -l <lang> -lower

See README file for additional information
The simplest way to run Meteor is as follows:
java -Xmx2G -jar meteor-*.jar test reference -l en -norm
This tells Meteor to score the file "test" against "reference", where test and reference are UTF-8 encoded files that contain one sentence per line. The "-l en" option tells Meteor to use settings for English. The -norm flag tells Meteor to apply language-specific text normalization before scoring. These are the ideal settings for which language-specific parameters are tuned.

Important note: If you are scoring text in a partially supported language, do not use the -norm flag, as Meteor has no normalization rules for these languages. Instead, use your own tools for segmenting, tokenizing, and lowercasing (if desired) the test and reference text prior to scoring. Meteor will warn if the -norm flag is used with unsupported languages. For example, to score Danish text, pre-tokenize the files and run:

java -Xmx2G -jar meteor-*.jar test.da.tok reference.da.tok -l da

To score the example files included with Meteor, use the following:

java -Xmx2G -jar meteor-*.jar example/xray/system1.hyp example/xray/reference -l en -norm
You should see the following output:
Meteor version: 1.4

Eval ID:        meteor-1.4-wo-en-norm-0.85_0.2_0.6_0.75-ex_st_sy_pa-1.0_0.6_0.8_0.6

Language:       English
Format:         plaintext
Task:           Ranking
Modules:        exact stem synonym paraphrase
Weights:        1.0 0.6 0.8 0.6
Parameters:     0.85 0.2 0.6 0.75

Segment 1 score:        0.447752250844953
Segment 2 score:        0.4284116369815996
Segment 3 score:        0.2772888474043816
Segment 4 score:        0.39587671218995263
Segment 5 score:        0.34983532103052495
.
.
.
Segment 2485 score:     0.29553941444479426
Segment 2486 score:     0.27829272093582047
Segment 2487 score:     0.2825995999223381
Segment 2488 score:     0.32037812996981163
Segment 2489 score:     0.33120147321343485

System level statistics:


           Test Matches                  Reference Matches
Stage      Content  Function    Total    Content  Function    Total
1            16268     20842    37110      16268     20842    37110
2              485        26      511        489        22      511
3              820       119      939        845        94      939
4             3813      3162     6975       3954      2717     6671
Total        21386     24149    45535      21556     23675    45231

Test words:             61600
Reference words:        62469
Chunks:                 20118
Precision:              0.6767347074578696
Recall:                 0.6500539115850005
f1:                     0.663126043401952
fMean:                  0.6539211143997783
Fragmentation penalty:  0.5099053526424513

Final score:            0.3204832379614146
The output contains the following in order:

4. Meteor Options

For the majority of scoring scenarios, only the -l and -norm options should be used. For more advanced usage, the full list of options follows.

Language: -l lang
Use settings for specified language. Lang can be either the language name or two letter code. See the supported language list.

Task: -t task
Use a different pre-defined set of parameters for scoring (currently limited to English):

Parameters: -p 'alpha beta gamma delta'
Set parameters manually. Parameter string should be quoted.

Modules: -m 'module1 module2 ...'
Set modules manually. Options are: exact stem synonym paraphrase. See supported languages. Module string should be quoted.

Weights: -w 'weight1 weight2 ...'
Set weights for each match type manually. Parameter string should be quoted.

Reference Count: -r refCount
Specify N, the number of reference sentences for each hypothesis. For N references, it is assumed that the reference file will be N times the length of the test file, containing sets of N references in order. For example, if N=4, reference lines 1-4 will correspond to test line 1, 5-8 to line 2, etc.

Beam Size: -x
This number, set to 40 by default, is used to limit the beam size when searching for the highest scoring alignment. As parameters are tuned for a beam size of 40, simply increasing this number does not necessarily produce more reliable scores.

Word List Directory: -s wordListDirectory
This option should only be used to test external function word lists. By default, the included function word lists will be used.

Synonymy Directory: -d synonymDirectory
This option should only be used to test external synonymy databases. By default, the included synonymy database will be used.

Paraphrase File: -a paraphraseFile
This option should only be used to test external synonymy databases. By default, the included paraphrase tables will be used. To build your own paraphrase tables, use Parex.

File Prefix: -f filePrefix
If alignments are to be written, they are written to <prefix>-align.out. In SGML mode, files produced will be <filePrefix>-seg.scr, <filePrefix>-doc.scr, <filePrefix>-sys.scr. The default prefix is "meteor".

Quiet: -q
Sentence scores to stderr, one per line. Corpus score to stdout, one line total. No additional output.

Character-based -ch
Calculate character-based precision and recall. Alignment is still word and phrase-level. Fragmentation penalty is still word and phrase-level.

Normalize: -norm
Tokenize and lowercases input lines, normalize punctuation to improve scoring accuracy. This option is highly recommended unless scoring raw system output against pretokenized references.

Lowercase: -lower
Lowercase input lines (not required if -norm also specified). This is most commonly used scoring cased, tokenized outputs with pretokenized references.

Ignore Punctuation: -noPunct
If specified, punctuation symbols will be removed before scoring. This is generally not recommended as parameters are tuned with punctuation included.

SGML: -sgml
This specifies that input is in SGML format. In addition to summary output, the following files are produced: The prefix can be changed with the -f option.

Stdio Format: -stdio
Input is from stdin using the format described below. Stats and scores written to stdout. Use "-" for test and reference files. Input lines are of two types, SCORE and EVAL.
SCORE ||| reference 1 words ||| reference n words ||| hypothesis words
Scores hypothesis against one or more references and returns line of sufficient statistics.
EVAL ||| stats
Calculates final scores using output of SCORE lines. Meteor exits on end-of-file.

Verbose Output: -vOut
Output verbose scores (Precision, Recall, Fragmentation, Score) in place of regular scores.

Sufficient Statistics: -ssOut
This option outputs sufficient statistics in place of scores and omits all other output. Statistics for a single hypothesis/reference instance are:
tstLen refLen stage1tstTotalMatches stage1refTotalMatches
stage1tstWeightedMatches stage1refWeightedMatches s2tTM s2rTM s2tWM
s2rWM s3tTM s3rTM s3tWM s3rWM s4tTM s4rTM s4tWM s4rWM chunks lenCost

Write Alignments: -writeAlignments
Write alignments between hypotheses and references to meteor-align.out or <prefix>-align.out when file prefix is specified. Alignments are written in Meteor format, annotated with Meteor statistics:
Title precision recall fragmentation score
sentence1
sentence2
Line2Start:Length Line1Start:Length Module Score
...

5. Standalone Meteor Aligner

Meteor includes a monolingual word aligner that can be run independently of the scorer. To run the aligner, use:

java -Xmx2G -cp meteor-*.jar Matcher
Running the aligner with no arguments shows the help message:
Meteor Aligner version 1.4
Usage: java -Xmx2G -cp meteor-*.jar Matcher <test> <reference> [options]

Options:
-l language                     One of: en da de es fi fr hu it nl no pt ro ru se tr
-m 'module1 module2 ...'        Specify modules (overrides default)
                                  One of: exact stem synonym paraphrase
-t type                         Alignment type (coverage vs accuracy)
                                  One of: maxcov maxacc
-x beamSize                     Keep speed reasonable
-d synonymDirectory             (if not default)
-a paraphraseFile               (if not default)

See README file for examples
Most options are the same as in the Meteor scorer. The additional option is -t, which specifies whether alignments should maximize coverage (comparable to recall) or accuracy (comparable to precision).

Sentences are read from test and reference files, one per line, and alignments are written to stdout using the Meteor format:

Alignment <line N>
sentence1
sentence2
Line2Start:Length	Line1Start:Length	Module		Score
...

Important note: the Meteor Aligner does not apply any normalization to input text. Text should be segmented, tokenized, and lowercased as desired prior to Meteor alignment.

6. Standalone Word Stemmer

Meteor includes a standalone word stemmer for supported languages. To run the stemmer, use:

java -cp meteor-*.jar Stemmer
Running the stemmer with no arguments shows the help message:
Snowball stem some text in a supported language
Languages: en da de es fi fr hu it nl no pt ro ru se tr
Usage: Stemmer lang < in > out
The stemmer reads lines from stdin and writes to stdout. Each word in the input is stemmed using the Snowball stemmer for the specified language.

Important note: the Meteor Stemmer does not apply any normalization to input text. Text should be segmented, tokenized, and lowercased as desired prior to Meteor alignment.

7. Integrating Meteor with your Software

The simplest way to integrate Meteor with your software involves using the -stdio option:

java -Xmx2G -jar meteor-*.jar - - -l en -norm -stdio
This tells Meteor to use the English settings, normalize text, and use stdin/stdout. You can then write lines of the following form to Meteor's stdin:
SCORE ||| reference 1 words ||| reference n words ||| hypothesis words
This scores a hypothesis against one or more references and returns a line of sufficient statistics.
EVAL ||| stats
This reads a line of sufficient statistics and produces a final score. Meteor exits on end-of-file.

Languages such as C++, Python, and Perl can open an external process and communicate with its stdin and stdout. For more information, see the documentation for process control for your language.

If your software is written in Java, you can use the Meteor API directly:

import edu.cmu.meteor.scorer.MeteorConfiguration;
import edu.cmu.meteor.scorer.MeteorScorer;
import edu.cmu.meteor.util.Constants;

MeteorConfiguration config = new MeteorConfiguration();
config.setLanguage("en");
config.setNormalization(Constants.NORMALIZE_KEEP_PUNCT);
MeteorScorer scorer = new MeteorScorer(config);
double score = scorer.getMeteorStats("test string", "reference string").score;
Remember to add meteor-*.jar to your classpath. See the source files for MeteorConfiguration and MeteorScorer for additional information.

8. Meteor X-ray

X-ray visualizes alignments and scores of one or more MT systems against a set of reference translations. When scoring translation hypotheses with Meteor, use the -writeAlignments option to produce alignment files annotated with Meteor statistics. X-Ray uses these files to produce graphical representations of alignment matrices and score distributions via XeTeX and Gnuplot. Final output is in PDF form with intermediate LaTeX and plot files preserved for easy inclusion in reports and presentations.

Requirements:

For example, on Ubuntu Linux, install the following packages:
sudo apt-get install python texlive-full gnuplot unifont

Setup:

If XeTeX and Gnuplot are installed somewhere other than /usr/bin, edit xray/Generation.py to include the correct locations:

xelatex_cmd = '/usr/bin/xelatex'
gnuplot_cmd = '/usr/bin/gnuplot'

Usage:

Run X-ray with the following:

python xray/xray.py
Running X-Ray with no arguments shows the help message:
MX: X-Ray your translation output
Usage: xray.py [options] <align.out> [align.out2 ...]

Options:
  -h, --help            show this help message and exit
  -c, --compare         compare alignments of two result sets (only first 2
                        input files used)
  -n, --no-align        do not visualize alignments
  -x MAX, --max=MAX     max alignments to sample (default use all)
  -p PRE, --prefix=PRE  prefix for output files (default mx)
  -l LBL, --label=LBL   optional system label list, comma separated:
                        label1,label2,...
  -u, --unifont         use unifont (use for non-western languages)

Example usage: score and visualize the hypotheses from system1 and system2 in the example/xray directory.

Score system1 with Meteor using the following options:

java -Xmx2G -jar meteor-*.jar example/xray/system1.hyp example/xray/reference \
-norm -writeAlignments -f system1
-norm: tokenize and normalize before scoring
-writeAlignments: write out sentence alignments used to calculate Meteor scores
-f system1: write alignments to system1-align.out

Visualize alignments and scores of system1 with Meteor X-Ray:

python xray/xray.py -p system1 system1-align.out
-p system1: prefix output files with 'system1'
system1-align.out: output from Meteor

Files produced:

Score system2 with Meteor:

java -Xmx2G -jar meteor-*.jar example/xray/system2.hyp example/xray/reference \
-norm -writeAlignments -f system2
Compare performances of system1 and system2:
python xray/xray.py -c -p compare system1-align.out system2-align.out
-c: compare two Meteor outputs
-p compare: prefix output with 'compare'

Files produced:

Additional systems:

To compare any number of systems, score each with Meteor (as above) and pass the align.out files to X-Ray. Without the -c flag, X-Ray will generate individual alignment matrices for each system and a single score PDF with score distributions for all systems. This is useful for comparing many configurations of the same system.

9. Training Meteor Parameters

Meteor parameters can be optimized to maximize agreement with human judgments of translation quality. The most frequently used evaluation task is ranking, where metrics should replicate human preferences between multiple translation hypotheses. Training Meteor to ranking data requires the following:

These files can exist for multiple language pairs with the same target language. For example, in WMT English, we have fr-en.sys1, fr-en.sys2, fr-en.ref, fr-en.rank, de-en.sys1, de-en.sys2, de-en.ref, de-en.rank, ... Meteor will consider all data files in the training directory when optimizing parameters. Meteor outputs the names of processed files to verify the data being used.

To prepare for training, convert the input files to SGML format where needed. (Plaintext would be fine in most cases, but datasets distributed as SGML aren't required to have segments in consistent order, which can create problems for older data.) The following example uses the included data in example/train:

mkdir my-train-dir
python scripts/sgmlize.py t < example/train/fr-en.sys1 > my-train-dir/fr-en.sys1.sgm
python scripts/sgmlize.py t < example/train/fr-en.sys2 > my-train-dir/fr-en.sys2.sgm
python scripts/sgmlize.py r < example/train/fr-en.ref > my-train-dir/fr-en.ref.sgm
cp example/train/fr-en.rank my-train-dir

Since parallel training requires loading several copies of Meteor into memory, filter the paraphrase table to minimize memory usage:

java -cp meteor-*.jar FilterParaphrase data/paraphrase-en.gz filtered.gz \
example/train/fr-en.ref

Meteor trains parameters by calculating sufficient statistics for all hypotheses and running an exhaustive grid search over rescorings. Every point explored is written out. To run the Trainer directly on one cpu:

java -cp meteor-*.jar Trainer rank my-train-dir -a filtered.gz > train.out

To find the best training point, sort the output:

sort -gr train.out > train.out.sort

The point with the highest correlation is the first line of the sorted file.

Running on multiple cpus greatly improves the speed of Meteor training. To run the grid search in parallel, use the meteor_shower script:

python scripts/meteor_shower.py meteor-*.jar en 4 rank my-train-dir work-dir 8 -a `pwd`/filtered.gz

This will keep 8 trainers running in parallel. Make sure to specify an absolute path for the paraphrase file. The results will be written to work-dir, along with a script for sorting the results.

10. Meteor Universal: Evaluation for Any Language

Meteor now supports language-specific evaluation for any target language for which there is enough data to build a standard phrase-based machine translation system. To build language-specific resources (paraphrase table and function word list), run the new_language.py script with your parallel data and a Moses-format phrase table:

python scripts/new_language.py out-dir corpus.f corpus.e phrase-table.gz [target-corpus.e]
Paraphrases will be extracted matching the target corpus (this can be a collection of relevant dev sets). If no target corpus is provided, the first 10,000 lines of the English corpus will be used (in practice this works adequately). Meteor can then be run with these files:
java -Xmx2G -jar meteor-*.jar test reference -new out-dir/meteor-files
Data should be pre-tokenized. Meteor will lowercase all data for evaluation (-new implies -lower). A universal parameter set will be used. These parameters are tuned on over 100,000 binary ranking judgments across 8 language directions and encode the following general properties:

11. Special Thanks

Authors of previous Meteor versions:

Cotributors to previous Meteor versions: