Figure 9 describes the speed comparisons that can be made between the fully-automated planners according to the Wilcoxon test. It can be observed that FF is significantly consistently faster than the other fully-automated planners at the STRIPS and NUMERIC levels (the significance of each of the arrows in the figure is sufficient to support transitive reasoning). Indeed, at the STRIPS and NUMERIC levels there is an interesting linear ordering between FF, LPG, MIPS and VHPOP (three of which were the prize-winners amongst the fully-automated planners) which is maintained between LPG, MIPS and VHPOP at the SIMPLETIME level. Despite the observation that SIMPLANNER was faster, in a significant proportion of the STRIPS problems than MIPS, there is no significant Wilcoxon relationship between them so that the four prize-winners comprise a spine of performance around which the other planners competing at these levels are clustered. The relationship breaks down at the TIME level because only LPG, MIPS, SAPA and TP4 participated. In this data set it can be seen that MIPS and SAPA are indistinguishable, with respect to the Wilcoxon test, but LPG is significantly consistently faster than both.
|
|
For comparing the hand-coded planners the competition used a collection of small problems and a collection of large problems at each problem level. The large problems were beyond the capabilities of any of the fully-automated planners. Interestingly, the hand-coded planners behaved differently in the small and large problem collections. This is most marked in Figure 10, at the STRIPS level, where the performances of TLPLAN and TALPLANNER are inverted in the small and large problem sets. In the small SIMPLETIME and TIME problems TLPLAN is consistently faster than either TALPLANNER or SHOP2, with TALPLANNER and SHOP2 being statistically indistinguishable in these data sets. TLPLAN is also consistently faster than TALPLANNER, which is in turn consistently faster than SHOP2, in the large TIME problems.