This is a README for evaluating FrameNet 1.5 full text annotation models Dipanjan Das dipanjan@cs.cmu.edu 2/18/2012 ======================================================================== Below, ${temp} is a temporary directory that you provide to the Perl script. Run the following in this directory to evaluate. scoring/fnSemScore_modified.pl -c ${temp} -l \ -n \ -t \ -v \ framesSingleFile.xml \ frRelationModified.xml \ file.gold.xml \ file.predict.xml The flags over here gives verbose output and uses partial matching. (You can change those for evaluating different parts of a frame-semantic parse.) framesSingleFile and frRelationModified.xml are versions of similar files released as part of SemEval 2007. file.gold.xml and file.predict.xml are sample gold and automatic files respectively. To evaluate, you have to create these files from the *.frame.elements, *.tokenized and *.lemma.tags files whose format I have discussed in: http://www.ark.cs.cmu.edu/SEMAFOR/data/README. To convert from the *.frame.elements and *.tokenized files to the above XML format, you can use the following class from SEMAFOR 2.0 (just an example using the training data): # preprocessing fefile=cv.train.sentences.frame.elements tokenizedfile=cv.train.sentences.tokenized processedfile=cv.train.sentences.all.lemma.tags cat ${fefile} | awk '{print "0""\t"$0}' > cv.train0.sentences.frame.elements end=`wc -l ${tokenizedfile}` end=`expr ${end% *} #conversion java -classpath ${classpath} -Xms1000m -Xmx1000m edu.cmu.cs.lti.ark.fn.evaluation.PrepareFullAnnotationXML \ testFEPredictionsFile:cv.train0.sentences.frame.elements \ startIndex:0 \ endIndex:${end} \ testParseFile:${processedfile} \ testTokenizedFile:${tokenizedfile} \ outputFile:file.gold.xml