Describing Complex Charts in Natural Language: A Caption Generation System

Vibhu O. Mittal*
University of Pittsburgh

Johanna D. Moore†‡
University of Pittsburgh

Giuseppe Carenini
University of Pittsburgh

Steven Roth§
Carnegie Mellon University

Graphical presentations can be used to communicate information in relational data sets succinctly and effectively. However, novel graphical presentations that represent many attributes and relationships are often difficult to understand completely until explained. Automatically generated graphical presentations must therefore either be limited to generating simple, conventionalized graphical presentations, or risk incomprehensibility. A possible solution to this problem would be to extend automatic graphical presentation systems to generate explanatory captions in natural language, to enable users to under-stand the information expressed in the graphic. This paper presents a system to do so. It uses a text planner to determine the content and structure of the captions based on: (1) a representation of the structure of the graphical presentation and its mapping to the data it depicts, (2) a framework for identifying the perceptual complexity of graphical elements, and (3) the structure of the data expressed in the graphic. The output of the planner is further processed regarding issues such as ordering, aggregation, centering, generating referring expressions and lexical choice. We discuss the architecture of our system and its strengths and limitations. Our implementation is currently limited to 2-D charts and maps, but, except for lexical information, it is completely domain independent. We illustrate our discussion with figures and generated captions about housing sales in Pittsburgh.


* Current affiliation: Just Research, 4616 Henry Street, Pittsburgh, PA 15213; and Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA 15213; e-mail:

Intelligent Systems Program

LRDC and Computer University of Pittsburgh Science Department}

§ Robotics Institute and Computer Science Department


Paper Sections:

     To Part 1: Introduction
     To Part 2: SAGE: A System for Automatic Graphical Explanations
     To Part 3: Discourse Strategies for Generating Captions
     To Part 4: Graphical Complexity: The Need for Clarification
     To Part 5: Generating Explanatory Captions
     To Part 6: System Implementation and Evaluation
     To Part 7: Related Work
     To Part 8: Conclusions and Future Work
     To Appendix A
     To Acknowledgements

Some large images may have suffered degredation in bringing their sizes down to web standards. In this case, double-clicking on the images will download the original, larger images. Download time may be longer due to increased file size.
    [RESEARCH]     [SAMPLES]     [PAPERS]     [PEOPLE]     [HOME]