To evaluate the sensitivity to representation, ten permutations of each problem in the AIPS2000 set were generated, resulting in 4510 permuted problems. The permutations were constructed by randomly reordering the preconditions in the operator definitions and the order of the definitions of the operators within the domain definition.

We limited the number of problems in this study because ten permutations of all problems would be prohibitive. We selected the AIPS2000 problems for attention because this was the most recently developed benchmark set. Even within that set, not all of the domains were permuted because some would not result in different domains under the transformation we used. For the purposes of this investigation, we limited the set of modifications to permutations of preconditions and operators because these were known to affect some planners and because practical considerations limited the number of permutations that could be executed. Finally, for expediency, we ran the permutations on a smaller number of faster platforms because it expedited throughput and computation time was not a factor in this study.

To analyze the data, we divided the performance on the permutations of the problems into three groups based on whether the planner was able to solve all of the permutations, none of the permutations or only a subset of the permutations. If a planner is insensitive to the minor representational changes, then the subset count should be zero. From the results in Table 5, we can see that all of the planners were affected by the permutation operation. The susceptibility to permuting the problem was strongly planner dependent ( , ), demonstrating that some planners are more vulnerable than others.

By examining the number in the Subset column, one can assess the degree of susceptibility. All of the planners were sensitive to reorderings, even those that relied on Graphplan methodology. The most sensitive were E, F, I and J (which included some Graphplan based planners and in which 40% of the problems had mixed results on the permutations) with C and L being least sensitive (3-4% were affected).