The HARDNUMERIC Satellite problem instances contained logical goals that are almost all trivial. For example, in most cases the problems involved simply ensuring that each of the satellites target a specific observation site at the end of the plan. However, the plan metric used to evaluate the plans was far more informative: the plans were evaluated according to the data collected by the satellites during their execution. Simply satisfying the explicit goals would generate a correct plan, but a worthless one in terms of the plan metric. Of the fully-automated planners, only MIPS and FF tackled these problems. TLPLAN and SHOP2 were the hand-coded planners that attempted these problems. It is instructive to compare the qualities of the plans produced by all four of these planners on this problem set. Figure 2 shows that the quality of the plans produced by the hand-coded planners is significantly higher than the quality of the plans generated by fully-automated planners. Indeed, FF generates plans that satisfy only the logical goals and minimise the plan size required to achieve that, so do not lead to any data collection. With some careful adjustments, MIPS has been able to generate plans that collect some data, but this is clearly a rather limited result. The closeness of the results generated by TLPLAN and SHOP2 suggest that they are both solving the data collection problem at a level close to optimal, or are applying similar heuristic approaches to the problem and generating similar locally optimal solutions. This domain very clearly highlights an advantage of hand-coded planners in their exploitation of the knowledge that their human domain-engineer can bring to bear.
|