Next: Metric Assumption 1: Does
Up: Assumptions of Direct Comparison
Previous: Planner Assumption 3: Are
Most comparisons emphasize the number of problems solved and the CPU time
to completion as metrics. Often, the problems are organized in increasing
difficulty to show scale-up. Comparing based on these metrics leaves a lot
open to interpretation. For example, some planners are designed to find the
optimal plan, as measured by number of steps in either a parallel or
sequential plan. Consequently, these planners may require more
computation. Thus, by ignoring plan quality, these planners may be unfairly
judged. We also hypothesize that the hardware and software
platform for the tests can vary the results. If a planner is developed for
a machine with 1GB of memory, then likely its performance will degrade
with less. A key issue is whether the effect is more or less uniform
across the set of planners.
In this section, we examine these two issues: execution platform and
effect of plan quality.
©2002 AI Access Foundation and Morgan Kaufmann
Publishers. All rights reserved.