Rule Induction Toolkit

Description:

Synchronous grammar rule learning for a Syntax based machine translation. For more details of the algorithms used inside the toolkit please refer to the papers below :

Vamshi Ambati and Alon Lavie: Improving Syntax Driven Translation Models by Re-structuring Divergent and Non-isomorphic Parse Tree Structures : Student Research Symposium at the The Eighth Conference of the Association for Machine Translation in the Americas (AMTA 2008),Waikiki, Hawai'i. (pdf)
Alon Lavie, Alok Parlikar, Vamshi Ambati: Syntax-driven Learning of Sub-sentential Translation Equivalents and Translation Rules from Parsed Parallel Corpora: In Proceedings of Syntax and Structure in Statistical Translation (SSST ) Workshop at ACL 2008. (pdf)

Download:

Please download a copy of your jar file from - rulelearner toolkit from here.
You may need a copy of antlr parsing libraries for Java included in your CLASSPATH. You can download a latest version from their website . A version that works for me can also be downloaded from - rulelearner toolkit from here.

To run:

java -jar ruleinduction.jar <configfile>

A sample config file format can be seen here.

Formats and Details:

More details on the parameters and formats of files follows:

Input Files:

CORPUS_FILE : Elicitation format Parallel Data with Alignment information. More description below
SPARSE_FILE : Source side parsetrees, one per line (Should be same in number and properly aligned with number of entries in corpus file
TPARSE_FILE : Target side parsetrees, one per line (Should be same in number and properly aligned with number of entries in corpus file

Output Files:

GRAMMAR_FILE : Output file that stores all the rules produced
PTABLE_FILE : Syntactic phrase table obtained is saved to this file
LEXICON_FILE: Word level lexicon created is saved here . These are one-one aligned, part-of-speech level one-word entries.

INPUT_MODE :

T2S - Projection based (Tree on one side, String on other)
T2T - Tree node alignment based (Tree on both sides required )
T2T2S - Tree on both sides, but Target trees are modified such that best of both T2T, T2S is obtained

OUTPUT_MODE:

S2S - Always use this to obtain Flattend form of rules (Synchronous-CFG style). Avenue formalism output - used in all Xfer runs.
T2S - Use this to obtain rules such that the Tree form is preserved on Source side, but Target side rules are all flattened
T2T - This format produces rules where tree structures are retained on both sides (Synchronous-TIG style )

OUTPUT_MODE: Samples

Flattened CFG style rules on both sides (S2S): Eg:
{NP,1}
NP::NP [ DET N] -> [DET N]
Tree on one side and flattened rule on other side: Eg:
{NP,1}
(NP (NP (DET N))) -> [DET N]
Tree structures on both sides (T2T): Eg:
{NP,1}
(NP (NP (DET N))) -> (NP (DET N))

CORPUS_FILE Format:

Any number of sentences can be given as input in this format. Each sentence should be separated by a new line. Anything starting with a semicolon is a comment. Sample example for one sentence is below

;; This is a comment
SentenceIndex:1
SL: Resumption of the session
TL: reprise de la session
Alignment:((1,1),(2,2),(3,3),(4,4))
Type: S

Config file:

Depending upon the mode of operation (T2S) or (T2T) , some of the TPARSE_FILE may be optional. Everything else is required. A sample config file is given below.

###START OF SAMPLE CONFIG#########
VRULES_ROOT=C:/rulelearner/

#### RULE LEARNING MODES ####
INPUT_MODE=T2T
OUTPUT_MODE=T2T

##### Parallel Treebank
CORPUS_FILE=C:/rulelearner/ger/ec.txt
SPARSE_FILE=C:/rulelearner/ger/en1.parsed
TPARSE_FILE=C:/rulelearner/ger/de1.parsed

GRA_FILE=C:/rulelearner/ger/grammar.gra
PTABLE_FILE=C:/rulelearner/ger/phrases.phr
LEXICON_FILE=C:/rulelearner/ger/lexicon.lex

# Output modes can be 'NONE, 'AVENUE' or 'ONELINE'
GRA_FORMAT=ONELINE
PTABLE_FORMAT=ONELINE
LEXICON_FORMAT=ONELINE

# Output Case - LOWER, or TRUE (Default is the true case)
OUTPUT_CASE=LOWER

# (Optional Parameter) . if not included, runs on entire corpus
STOPAT=100000

TOOL_MODE=AVENUE

# Some extra parameters you may not need, but just set for now 
TOOL_MODE=AVENUE
MAX_RULE_SIZE=10
OUTPUT_CASE=MIXED
LEXICALIZATION=NONE
MARKOVIZATION=NONE

###END OF SAMPLE CONFIG#########