Most previous efforts in generating intelligent multi-media presentations have focused on coordinating natural language and graphical depictions of real world devices (e.g., military radios (Feiner and McKeown, 1991) and coffee makers (Wahlster et al., 1993)) for generating instructions about their repair or proper use. These projects tackled important problems such as apportioning content to media and generating cross references between them. Research has also focused on issues regarding the generation of coordinated presentations in applications where the graphics are familiar, or possess an obvious mapping between the dataset and a graphical image (e.g., weather maps (Kerpedjiev, 1992) and network diagrams (Marks and Reiter, 1990)).

Our work differs from these projects in two ways. The first difference concerns the type of data that our system deals with. Unlike the presentations generated by the systems mentioned above, presentations generated by sage are usually based on abstract or relational information (e.g., census reports, logistics data, hospital administration data, real estate sales data), lacking any obvious graphical depiction. Second, although our long term goal is to generate coordinated multi-media explanations using informational graphics and natural language, our focus in this paper was on generating effective natural language explanations about the graphical presentations. In order to do this, the system had to explicitly reason about the perceptual complexity of the presentation. Generating such captions is an important component of constructing multi-media explanations involving integrative graphical displays.

The PostGraphe system (Fasciano, 1996; Fasciano and Lapalme, 1996) is the closest related research effort. As in our work, PostGraphe generates statistical graphics and accompanying captions. However, the issues considered in our work differ from those in PostGraphe in several ways and both the text and the graphics generated by PostGraphe emphasize aspects orthogonal to the ones considered in our project. For instance, PostGraphe can take as input a list of aspects that should be conveyed by the presentation. (These goals are represented in the system as a pre-defined set of templates, such as, "show the evolution of <attribute-name-1> with respect to <attribute-name-2>".) This information is then used by PostGraphe to not only generate an appropriate type of diagram (e.g., a line chart), but also to generate a caption that explicitly captures the specific aspects of interest, such as: "The profits were at their highest in 1975 and lowest in 1974, with about half their 1975 value." This is in contrast to our system, which does not reason about trends or relationships between different data points shown in the graphic. Instead, our work has focused on describing complex data to grapheme mappings and deriving metrics for perceptual complexity. This is due, in part, to the nature of the graphical presentations that the two systems can design. (\sc sage ), for instance, is capable of designing novel graphical presentations for very complex datasets, using techniques such as multiple grapheme composition and space alignment to facilitate cross-attribute comparisons. The range of graphical capabilities in PostGraphe is more limited. Combined with the fact that the graphics are generated in response to an explicit user goal, user comprehension problems in PostGraphe are less likely than in our system. Perhaps in light of this, PostGraphe does not need to explicitly analyze its graphic presentations for potential ambiguities or perceptual complexities, and the captions accompanying the graphic do not take these factors into account.

However, our current implementation, described in the paper, should not be confused with our long term research agenda; it was designed as a framework to evaluate more sophisticated capabilities. These include some of the capabilities that PostGraphe has, particularly those dealing with the generation of information about trends and patterns. We plan to extend the approach used by PostGraphe to take into account both the writer's goals and domain- and data-specific aspects. To this end, we are developing a language to express presentation intentions, taking into account both our experiences as well as the language used in PostGraphe. Furthermore, whereas the sequence of presentation goals to be achieved are part of the input to PostGraphe, our new framework generates these dynamically by integrating a data analysis module with a discourse planner. The data analysis module is being designed to identify all possible relevant aspects of the data based on the domain specification and an analysis task. The planner can use a variety of strategies to select and organize these aspects into complex arguments that can be realized as presentations combining both text and graphics (see (Kerpedjiev et al., 1997) for further details on our new framework).