Generating Visual Arguments: a Media-independent Approach

Nancy Green, Stephan Kerpedjiev, Steven F. Roth

School of Computer Science

Carnegie Mellon University

Pittsburgh, PA 15231 USA

ngreen,kerpedji,roth@cs.cmu.edu

Giuseppe Carenini, Johanna Moore

Intelligent Systems Program

University of Pittsburgh

Pittsburgh, PA 15260 USA

carenini,jmoore@cs.pitt.edu

Introduction

The research reported here is part of our ongoing effort (Kerpedjiev et al. 1997b; 1997a; Green, Carenini, & Moore 1998; Green et al. 1998; Kerpedjiev et al. 1998) to design systems that can automatically generate integrated text and information graphics presentations of complex, quantitative data. In this paper, we take the position that certain types of arguments that can be presented visually in information graphics (e.g., bar charts and scatter plots) can be gener ated from an underlying media-independent represen tation of a presentation. In support of this claim, first we briefly describe the architecture we are developing for the generation of integrated text and information graphics presentations. In this architecture, media- independent communicative acts are transformed into user task specifications which are the basis for the automatic design of the presentation's graphics. Then we present an example showing correspondences between the media-independent representation of an argument and the tasks that would be used to design a graphic expressing the argument.

Architecture

We are investigating the integration of two complemen tary approaches to automatic generation of integrated text and graphics presentations: hierarchical planning to achieve communicative goals and task-based graphic design. Researchers in natural language processing (Mittal et al. 1995; Moore 1995; Wahlster et al. 1993) have modeled presentation design as a process of hi erarchical planning to achieve communicative goals. Researchers in graphics have emphasized the need to design presentations that support the perceptual and logical tasks a user must perform (Casner 1991; Roth & Mattis 1990; Beshers & Feiner 1993).

In our hybrid approach, shown in Figure 1, the con tent and organization of a presentation is rst planned at a media-independent level using a hierarchical plan ner (Young 1994), resulting in a presentation plan. The presentation plan describes the intentional and infor mational structure of the presentation (Moser, Moore, & Glendening 1995; Moore & Pollack 1992), as well as what low-level media-independent communicative acts are to be performed by the system to achieve the presentation's goals. A media allocation component decides which parts of the presentation plan to realize in which media. Two media-specic generators (text, graphics) then realize their assigned parts of the plan.

The text generator converts its assigned part of the plan to functional descriptions of sentential units, which are subsequently realized by a general-purpose sentence generator (FUF/SURGE) (Elhadad & Robin 1996). (The complex process of converting the plan to text is beyond the scope of this paper.) Graph ics generation is performed in two stages. First, the graphics generator converts the parts of the plan as signed to it by the media-allocation component to a se quence of user tasks that will enable the presentation's goals to be achieved. (Previous integrated text and graphic generation systems, e.g., (Andre & Rist 1994; Fasciano & Lapalme 1996; Feiner & McKeown 1991; Maybury 1991; McKeown et al. 1992; Wahlster et al. 1993) have not attempted to automatically de rive user tasks from a media-independent presenta tion plan.) The task sequence is then input to the SAGE graphic design system (Roth & Mattis 1990; Roth et al. 1994), which automatically creates a graphic designed to enable the user to perform these tasks. All design decisions are made by SAGE, from the type of graphic (e.g., a bar chart), to specic prop erties of a graphic (e.g., the choice of horizontal as opposed to vertical bars). In this way, graphic de sign is tailored to a presentation's goals. (For details on our approach to graphics generation, including the derivation of tasks in our system, see (Kerpedjiev et al. 1998).)

Expressing an Argument in Graphics

In this section, we give an analysis of an argument and its representation in a media-independent presentation plan. We describe the user tasks which would be de rived from the media-independent communicative acts of the plan in our current approach, and then suggest some ways in which the structure of the discourse may also contribute to the design of effective graphics, as well as its in uence on media allocation.

Figure 1:Integrated Text-Graphics Generation Architecture

Analysis of example

The data used in this paper is fictitious.

The goal of the example presentation is for the user to accept the belief that a certain local newspaper, the Post-Gazette (PPG), has more readers than the total number of readers of all other newspapers that are subscribed to in some region. The user is currently ignorant of this fact, but probably would not accept it just on the basis of a simple assertion by the system, due to his current beliefs. In particular, the user knows that the New York Times (NYT) has more readers than the Wall Street Journal (WSJ) in the region, and erroneously believes that because of this, the New York Times must have the largest number of readers of all newspapers in the region. However, the latter belief is incompatible with the belief which it is the goal of the presentation to get the user to accept.

To simplify discussion, let us abbreviate the propositions playing a role in the example as follows:

• Q: the number of readers of NYT exceeds the number of readers of WSJ
• R: the number of readers of NYT exceeds the number of readers of any other paper in the region
• T: the number of readers of PPG exceeds the total number of readers of all other papers in the region.

• Goal: (BMB User T), i.e., the goal is that the User believe that it is mutually believed with the system that T, (see footnote)
• Constraints:
  • (Bel System (Bel User Q)), the system believes that the user believes that Q,
  • (Bel System (Bel User (Entails Q R))), the system believes that the user believes that Q entails R.
  • (Bel System (Incompatible R T)), the system believes that R and T cannot both be true,
  • (Bel System Q), the system believes that Q,
  • (Bel System T), the system believes that T, and
  • (Bel System (not R)), the system believes that R is false,
• Subgoals:
  • (BMB User Q), i.e., make sure that the User knows that the System is aware of Q,
  • (BMB User (not (Entails Q R)))
  • (BMB User (Bel System T))

• (Acknowledge System User Q), which achieves (BMB User Q).
• (Assert System User (not R)), which achieves (BMB User (not (Entails Q R))) provided that (BMB User Q).
• (Assert System User T), which achieves (BMB User (Bel System T)).

Figure 2:Discourse Plan

In summary, the intentional structure of this argument can be represented in the plan shown in Figure 2. (The figure shows only the hierarchical relations among the communicative acts and the core-contributor distinction among acts; unlabelled acts are contributors.) Such a plan might be realized in text as follows: Although the New York Times is read by more people in Western PA than the Wall Street Journal, the New York Times does not have the highest number of readers in the region. The Post-Gazette has more readers than the total number of readers of all other newspapers in the five-county Western PA region.

Realization in Graphics

A graphic realizing this argument is shown in Figure 3. The core of the argument, the assertion that T holds, is expressed by enabling the user to perform the task of comparing the upper bar's length (which represents the number of readers of PPG) to the lower bar's length (which represents the total number of readers of the other newspapers). (In general, an assertion that some quantity is greater than another quantity would be transformed by our graphics generator into a comparison task; for details on the process of deriving a task sequence, see (Kerpedjiev et al. 1998).)

The contributor given in support of T is expressed in the same graphic, although less prominently. Its core consists of the assertion (not R), which is expressed in the graphic indirectly by falsifying R. That is, by enabling the user to perform the task of comparing the length of the segment of the lower bar labelled NYT to the length of the upper bar, the user can see that there is one newspaper (PPG) with more readers than NYT, which falsifies R. The concession Q (contributing to the acceptance of the assertion that R does not hold), is expressed within the lower bar by enabling the user to perform the task of comparing the length of the segment labelled NYT (representing the number of NYT readers) to the length of the segment labelled WSJ (representing the number of WSJ readers).

Figure 3:Graphic realizing the argument

Thus, in our current approach each the three low- level communicative acts of this plan would be tansformed into the comparison tasks described above. The tasks would then be used by SAGE to design a graphic such as the one shown in Figure 3 to support these tasks. Note that more than one graphic may be designed by SAGE to support the tasks. An interesting open question that we are investigating is how the intentional structure of an argument should influence graphic design. In this graphic, for example, the assertion corresponding to the core of the argument is more visually prominent than the other information since the graphic contains only two horizontal bars, one for each of the entities compared in the core assertion. Also, the quantities compared in the core assertion are encoded differently (i.e. as horizontal bars) from the other quantities (i.e. as segments of a stacked bar). The user may interpret the difference as conversation ally implicating an important distinction in the two sets of quantities (Marks & Reiter 1990).

A related issue is the role of discourse structure in media allocation. For example, if the graphic shown in Figure 3 is accompanied only by text realizing the core of the argument (e.g., The Post-Gazette has more readers than the total number of readers of all other news papers in the five-county Western PA region), then the text would contribute to the user's recognition of the main point of the graphic. On the other hand, if the same graphic is accompanied only by text realizing one of the other acts of the plan (e.g., The New York Times has more readers than the Wall Street Journal), then the text might impede the user's recognition of the main point of the graphic. In future work, we hope to address these open issues.

Acknowledgments

This project was supported by DARPA, contract DAA- 1593K0005.

References

Andre, E., and Rist, T. 1994.

Referring to world objects with text and pictures. In Proceedings of COLING, 530-34.

Beshers, C., and Feiner, S. 1993.

AutoVisual: Rule-based design of interactive multivariate visualization. IEEE Computer Graphics and Applications 41-49.

Casner, S. M. 1991.

A task-analytic approach to the automated design of information graphic presentations. ACM Transactions on Graphics 10(2):111-151.

Clark, H., and Marshall, C. 1981.

Definite reference and mutual knowledge. In Joshi, A. K.; Webber, B.; and Sag, I., eds., Elements of discourse understanding. Cambridge: Cambridge University Press.

Elhadad, M., and Robin, J. 1996.

An overview of SURGE: A reusable comprehensive syntactic realization component. Technical Report Technical Report 96-03, Dept of Mathematics and Computer Science, Ben Gurion University, Beer Sheva, Israel.

Fasciano, M., and Lapalme, G. 1996.

PostGraphe: a System for the Generation of Statistical Graphics and Text. In Proceedings of the 8th International Natural Language Generation Workshop, 51-60.

Feiner, S., and McKeown, K. 1991.

Automating the generation of coordinated multimedia explanations. IEEE Computer 24(10):33-40.

Green, N.; Carenini, G.; Kerpedjiev, S.; Roth, S.; and Moore, J. 1998.

A media-independent content language for integrated text and graphics generation. Submitted.

Green, N.; Carenini, G.; and Moore, J. 1998.

Generating attributive descriptions for integrated text and information graphics presentations. In Proceedings of the Ninth International Workshop on Natural Language Generation. To appear.

Kerpedjiev, S.; Carenini, G.; Roth, S.; and Moore, J. 1997a.

AutoBrief: a multimedia presentation system for assisting data analysis. Computer Standards and Interfaces 18:583-593.

Kerpedjiev, S.; Carenini, G.; Roth, S. F.; and Moore, J. D. 1997b.

Integrating planning and task-based design for multimedia presentation. In International Conference on Intelligent User Interfaces (IUI '97), 145-152. Association for Computing Machinery.

Kerpedjiev, S.; Carenini, G.; Green, N.; Roth, S. F.; and Moore, J. D. 1998.

Saying it in graphics: from intentions to visualizations. Submitted.

Marks, J., and Reiter, E. 1990.

Avoiding unwanted conversational implicatures in text and graphics. In Proceedings of the National Conference on Articial Intelligence, 450-455.

Maybury, M. T. 1991.

Planning multimedia explanations using communicative acts. In Proceedings of the Ninth National Conference on Articial Intelligence,61-66.

McKeown, K.; Feiner, S.; Robin, J.; Seligmann, D.; and Tanenblatt, M. 1992.

Generating cross-references for multimedia explanation. In Proceedings of AAAI, 9-16.

Mittal, V.; Roth, S.; Moore, J.; Mattis, J.; and Carenini, G. 1995.

Generating explanatory captions for information graphics. In Proceedings, 1276-1283. Montreal: International Joint Conference on Artificial Intelligence.

Moore, J. D., and Pollack, M. E. 1992.

A problem for RST: The need for multi-level discourse analysis. Computational Linguistics 18(4):537-544.

Moore, J. D. 1995.

Participating in Explanatory Dialogues. MIT Press.

Moser, M.; Moore, J. D.; and Glendening, E. 1995.

Instructions for coding explanations: Identifying segments, relations and minimal units. Technical report, Learning Research and Development Center, University of Pittsburgh, Pittsburgh, Pennsylvania.

Roth, S., and Mattis, J. 1990.

Data characterization for intelligent graphics presentation. In Proceedings of the Conference on Human Factors in Computing Systems (SIGCHI '90), 193-200.

Roth, S. F.; Kolojejchick, J.; Mattis, J.; and Goldstein, J. 1994.

Interactive graphic design using automatic presentation knowledge. In Proceedings of the Conference on Human Factors in Computing Systems (SIGCHI '94), 112-117.

Wahlster, W.; Andre, E.; Finkler, W.; Protlich, H.J.; and Rist, T. 1993.

Plan-based integration of natural language and graphics generation. Articial Intelligence 63:387-427.

Young, M. R. 1994.

A developer's guide to the long-bow discourse planning system. Technical Report ISP TR Number: 94-4, University of Pittsburgh, Intelligent Systems Program.