info.ephyra.trec
Class EphyraTREC13To16

java.lang.Object
  info.ephyra.OpenEphyra
      info.ephyra.trec.OpenEphyraCorpus
          info.ephyra.trec.EphyraTREC13To16

public class EphyraTREC13To16
extends OpenEphyraCorpus

Runs and evaluates Ephyra on the data from the TREC 13-16 QA tracks.

This class extends OpenEphyraCorpus.

Version:

2008-02-07

Author:

Nico Schlaefer

Field Summary

protected static float	FACTOID_ABS_THRESH Absolute threshold for factoid question scores.
protected static int	FACTOID_MAX_ANSWERS Maximum number of factoid answers.
private static boolean	factoidLog Load answers to factoid questions from log file?
private static TRECPattern[]	factoidPatterns Patterns for factoid questions (optional).
private static java.util.ArrayList<java.lang.Float>	factoidQuestionScores Scores for the factoid questions within a target.
private static java.util.ArrayList<java.lang.Float>	factoidTargetScores Factoid scores for the targets.
private static java.lang.String	inputLogFile Log file used as a source for answers to some of the questions.
protected static float	LIST_REL_THRESH Relative threshold for list question scores (fraction of top score).
private static boolean	listLog Load answers to list questions from log file?
private static TRECPattern[]	listPatterns Patterns for list questions (optional).
private static java.util.ArrayList<java.lang.Float>	listQuestionScores Scores for the list questions within a target.
private static java.util.ArrayList<java.lang.Float>	listTargetScores List scores for the targets.
private static java.lang.String	logFile Log file for the results returned by Ephyra.
private static boolean	otherLog Load answers to "Other" questions from log file?
private static java.lang.String	runTag Tag that uniquely identifies the run (also used as output file name).
private static TRECTarget[]	targets Target objects containing the TREC questions.

Fields inherited from class info.ephyra.OpenEphyra

dir, NORMALIZER

Constructor Summary

EphyraTREC13To16()

Method Summary

Result[]	askOther(java.lang.String question) Asks Ephyra an 'other' question.
Result[]	askOther(TRECTarget target) Asks Ephyra an 'other' question, making use of the target description and previous questions and answers.
private static boolean[]	evalFactoidQuestion(java.lang.String qid, Result[] results, float absThresh) Calculates the score for a single factoid question.
private static float	evalFactoidTarget() Calculates the factoid score for the current target.
private static float	evalFactoidTotal(float absThresh) Calculates the total score for the factoid component and logs the score.
private static boolean[]	evalListQuestion(java.lang.String qid, Result[] results, float relThresh) Calculates the score for a single list question.
private static float	evalListTarget() Calculates the list score for the current target.
private static float	evalListTotal(float relThresh) Calculates the total score for the list component and logs the score.
protected void	initOther() Initializes the pipeline for 'other' questions.
static void	main(java.lang.String[] args) Runs Ephyra on the TREC questions.
private static void	runAndEval() Initializes Ephyra, asks the questions or loads the answers from a log file, evaluates the answers if patterns are available and logs and saves the answers.

Methods inherited from class info.ephyra.trec.OpenEphyraCorpus

askFactoid, askList, initFactoidCorpus, initFactoidWeb

Methods inherited from class info.ephyra.OpenEphyra

askFactoid, commandLine, getDir, initFactoid, readLine, runPipeline

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

FACTOID_MAX_ANSWERS

protected static final int FACTOID_MAX_ANSWERS

Maximum number of factoid answers.

See Also:

Constant Field Values

FACTOID_ABS_THRESH

protected static final float FACTOID_ABS_THRESH

Absolute threshold for factoid question scores.

See Also:

Constant Field Values

LIST_REL_THRESH

protected static final float LIST_REL_THRESH

Relative threshold for list question scores (fraction of top score).

See Also:

Constant Field Values

targets

private static TRECTarget[] targets

Target objects containing the TREC questions.

runTag

private static java.lang.String runTag

Tag that uniquely identifies the run (also used as output file name).

logFile

private static java.lang.String logFile

Log file for the results returned by Ephyra.

inputLogFile

private static java.lang.String inputLogFile

Log file used as a source for answers to some of the questions.

factoidLog

private static boolean factoidLog

Load answers to factoid questions from log file?

listLog

private static boolean listLog

Load answers to list questions from log file?

otherLog

private static boolean otherLog

Load answers to "Other" questions from log file?

factoidPatterns

private static TRECPattern[] factoidPatterns

Patterns for factoid questions (optional).

listPatterns

private static TRECPattern[] listPatterns

Patterns for list questions (optional).

factoidQuestionScores

private static java.util.ArrayList<java.lang.Float> factoidQuestionScores

Scores for the factoid questions within a target.

listQuestionScores

private static java.util.ArrayList<java.lang.Float> listQuestionScores

Scores for the list questions within a target.

factoidTargetScores

private static java.util.ArrayList<java.lang.Float> factoidTargetScores

Factoid scores for the targets.

listTargetScores

private static java.util.ArrayList<java.lang.Float> listTargetScores

List scores for the targets.

Constructor Detail

EphyraTREC13To16

public EphyraTREC13To16()

Method Detail

evalFactoidQuestion

private static boolean[] evalFactoidQuestion(java.lang.String qid,
                                             Result[] results,
                                             float absThresh)

Calculates the score for a single factoid question.

Parameters:

qid - ID of the question

results - the results from Ephyra

absThresh - absolute confidence threshold for results

Returns:

for each answer a flag that is true iff the answer is correct

evalListQuestion

private static boolean[] evalListQuestion(java.lang.String qid,
                                          Result[] results,
                                          float relThresh)

Calculates the score for a single list question.

Parameters:

qid - ID of the question

results - the results from Ephyra

relThresh - relative confidence threshold for results

Returns:

for each answer a flag that is true iff the answer is correct

evalFactoidTarget

private static float evalFactoidTarget()

Calculates the factoid score for the current target.

Returns:

the score or -1 if there are no factoid questions

evalListTarget

private static float evalListTarget()

Calculates the list score for the current target.

Returns:

the score or -1 if there are no list questions

evalFactoidTotal

private static float evalFactoidTotal(float absThresh)

Calculates the total score for the factoid component and logs the score.

Parameters:

absThresh - absolute confidence threshold for results

Returns:

the score or -1 if there are no factoid questions

evalListTotal

private static float evalListTotal(float relThresh)

Calculates the total score for the list component and logs the score.

Parameters:

relThresh - relative confidence threshold for results

Returns:

the score or -1 if there are no list questions

runAndEval

private static void runAndEval()

Initializes Ephyra, asks the questions or loads the answers from a log file, evaluates the answers if patterns are available and logs and saves the answers.

main

public static void main(java.lang.String[] args)

Runs Ephyra on the TREC questions.

Parameters:

args - argument 1: questionfile
[argument 2: tag=runtag (uniquely identifies the run, also used as output file name, if not set an unambiguous tag is generated automatically)]
[argument 3: log=logfile (if not set an unambiguous file name is generated automatically)]
[argument 4: lin=input_logfile (specifies a separate logfile that is used as a source for answers to some of the questions, if not set the standard log file is used)]
[argument 5: lflags=[f][l][o] (answers to these types of questions are loaded from the log file instead of querying Ephyra, e.g. "flo" for factoid, list and other questions)]
[argument 6: fp=factoid_patternfile]
[argument 7: lp=list_patternfile]

initOther

protected void initOther()

Initializes the pipeline for 'other' questions.

askOther

public final Result[] askOther(java.lang.String question)

Asks Ephyra an 'other' question.

Parameters:

question - other question

Returns:

array of results

askOther

public Result[] askOther(TRECTarget target)

Asks Ephyra an 'other' question, making use of the target description and previous questions and answers.

Parameters:

target - the target the 'other' question is about

Returns:

array of results