This section summarizes our experiments and analysis of results. We evaluate our ideas with several experiments, described in the next subsections. Section 5.1 evaluates the impact of the implementation enhancements on the planner's performance. Section 5.2 focuses on the effect of macro-operators in the system used in the competition. In these two subsections, the benchmarks that we competed in as part of IPC-4 are used for experimental evaluation: Promela Dining Philosophers - ADL (containing a total of 48 problems), Promela Optical Telegraph - ADL (48 problems), Satellite - STRIPS (36 problems), PSR Middle Compiled - ADL (50 problems), Pipesworld Notankage Nontemporal - STRIPS (50 problems), Pipesworld Tankage Nontemporal - STRIPS (50 problems), and Airport - ADL (50 problems). MACRO-FF took the first place in Promela Optical Telegraph, PSR, and Satellite.
Section 5.3 compares the two abstraction techniques discussed in this article using STRIPS domains with static facts. We provide more details later in this section. Section 5.4 contains an empirical analysis of how CA-ED macros affect heuristic state evaluation and depth of goal states. All experiments reported in this article were run on a AMD Athlon 2 GHz machine, with the limits of 30 minutes and 1 GB of memory for each problem.