In this study, we compared performance of multiple versions of four
planners (labeled for this section with W, X, Y and Z, with larger
version numbers indicating subsequent versions). We considered two
criteria for improvement: outcome of planning and computation time for
solved problems. The outcome of planning is one of: solved, failed or
timed-out. On each criterion, we statistically analyzed the data for
superior performance of one of the versions. The outcome results for
all the planners are summarized in Table 7. As
the table shows, rarely does a new version result in more problems
being solved. Only Z improved the number of our test
problems solved in subsequent versions.
Version performance: counts of outcome and change in number
To check for whether the differences in outcome are significant, we ran 2x3
tests with planner version as independent variable and outcome as
dependent. Table 8 summarizes the results of the
analysis. For Z, we compared each version to its successor only. The
differences are significant except for Y and the transition from Z 2 to 3 (this
was expected because these two versions were extremely similar).
results comparing versions of the same planner.
Another planner performance metric, which we evaluated, was the speed
of solution. For this analysis, we limited the comparison to just those
problems that were solved by both versions of the planner. We then
classified each problem by whether the later version solved the problem
faster, slower, or in the same time as the preceding version. From the
results in Table 9, we see that all of the
planners improved in the average speed of solution for subsequent versions, with the
exception of Z (transition from the 1 to 2 versions). However,
Z did increase the number of problems solved between those versions.
Improvements in execution speed across versions. The
Faster column counts the number of cases in which the new version
solved the problem faster; Slower specifies those cases in which
the new version took longer to solve a given problem.