We are happy with the outcomes of the first probabilistic track of the International Planning Competition. In addition to bringing attention to this important set of planning challenges, it appears to have helped spur the community to use uniform comparison problems by providing a domain language and a set of benchmarks (Yoon, Fern, & Givan, 2005).
In spite of the success, we feel there are changes that could be made in future competitions that would increase their value to the community. First, on the competition logistics side, our server logged outcomes of the interactions between planners and domains, but did not keep an exhaustive record of the actions taken and timing information. In retrospect, such information would have been helpful in identifying how the planners addressed the domains and whether they took suboptimal actions or just got unlucky. In addition, our server had no provisions for security. A simple password and/or reservation system would have helped the evaluations go much more smoothly as it would have prevented inadvertent access to the server by one group when another was assigned an evaluation slot.
On the domain side, we hope future competitions are able to focus on more interesting domains. We found that simply adding noisy action failures to a deterministic domain was not enough to produce interesting probabilistic problems—for such domains, straightforward replanning can be very effective. The non-Blocksworld domains we created were not mastered by any of the planners and we hope that they are retained in some form in future evaluations.
Like the progression of competitions in the classical track, we hope future competitions in the probabilistic track move toward domains grounded in real-life data and real-world problems including the handling of partially observability and time. A second competition is slated to be held in conjunction with IPC in 2006 and we urge interested members of the planning community to participate to help keep the competition moving in a productive direction for the benefit of the field.
Håkan L. S. Younes