SAGE is a knowledge-based presentation system that designs graphical displays of combinations of diverse information (e.g., quantitative, relational, temporal, hierarchical, categorical, geographic). The inputs to SAGE include: (1) sets of data represented as tuples in a relational database, (2) a characterization of the properties of the data that are relevant to graphic design, and (3) an optional set of design specifications, expressing a user's preferences for visualizing the data set.

SAGE’s output consists of one or more coordinated sets of 2-D information graphics that use a variety of graphical techniques to integrate multiple data attributes in a single display. SAGE integrates multiple attributes in three ways:

• by representing them as different properties of the same set of graphical objects. For example, both the left and the right edges of the bars in the left most chart in Figure 2 are used to map attributes (asking-price and selling-price respectively)

• by assembling multiple graphical objects into groups that function as units to express data. For example, the interval bar and the mark in the left most chart in Figure 2 are used to show different types of price-related attributes: asking-price, selling-price and the agency-estimate.

• by aligning multiple charts and tables together with respect to a common axis. For example, the three charts in Figure 2 are aligned on the Y-axis which indicates the house.

Creating a graphic that integrates data in this way is partly an encoding process in which the values of data attributes are converted into graphical values of properties of objects (e.g., color, shape, and spatial position of polygons). Interpreting the information in a graphic is a decoding process, where people must translate visual symbols back into data values. SAGE creates graphics that enable people to perform efficiently information-seeking tasks (e.g., searching for clusters of data values that are different from the rest and looking up other facts to understand what makes them different). However, in designing graphics, SAGE only considers how effectively attributes can be mapped to graphical properties to support a task. For example, a requirement to be able to search for particular values by name might result in the relevant attribute being arranged along an axis in lexicographic order; on the other hand, if it is important to find the maximum and minimum values in a set, SAGE might order these values in terms of magnitude. SAGE, like other automated presentation systems (Casner 1991, Mackinlay 1986,), does not take into account perceptual complexities associated with the resulting graphic. For instance, SAGE does not explicitly reason about the difficulties users may have in translating bi-color saturation scales to exact numerical values

SAGE can also design complex presentations that have overlapping objects, or use cluster composition to define a novel combination or grouping of graphical objects in the presentation. This can make under-standing some of the graphics that SAGE generates quite difficult. Fortunately, the picture representation used in SAGE contains a complete declarative representation of the content and structure of the graphic in a form that can be used for reasoning by other components. Thus, this representation can be used to reason about possible sources of user confusion arising from mappings that are either complex or ambiguous to the user.

SAGE 's representation serves three functions in explanation generation. First, it helps define what a viewer must understand about a graphic in order to obtain useful information from it. It does this by defining the elements of a graphic and the way they combine to express facts (i.e., how they map to data). Second, the representation describes the structure of both the graphical presentation and the data it presents, so that they can be explained coherently. Finally, the representation helps derive judgments of complexity for specifying graphical elements needing text explanation. To understand these three functions, we briefly review the representation.

A graphic generated by SAGE to illustrate the use of encoders.

Graphemes are the basic building blocks for constructing pictures. Marks, text, lines, and bars are some of the different grapheme classes available in SAGE. Each grapheme type consists of a definition of the parameters that control the appearance of all graphemes of that type; different grapheme sub-types can be created by varying specific parameters. Individual graphemes can be generated by providing appropriate values for all the input parameters. For instance, individual marks can be generated by providing values for the parameters: x coordinate, y coordinate, shape, size and color to an instance of a mark class encoder; individual line segments can be generated by providing values for the coordinates x1, y1, x2 and y2, thickness and color to the line class encoder.

Symbol classes are used to organize graphemes into structures that express facts in the data set. A labeled mark, an interval bar and a bar with an attached label are some of the more familiar symbol classes available in SAGE . Each symbol class consists of a definition of the spatial relationship among a set of graphemes and the correspondence between the parameters of this set and attributes types in a data-set. A labeled-mark, for instance, would be defined as a combination of a mark and a text label and the spatial relationship between them. Consider the labeled-marks in the chart shown in Figure 3. The spatial position of the label is dependent on the position of the mark: it is offset slightly to the right and above the mark.

Encoders used in mapping attributes to a labeled-mark in the next chart shown in Figure 3.

Symbol classes in SAGE can be either pre-defined (some of the more common ones, such as a labeled-mark have already been defined), or created by the system based on rules about combining different graphemes into clusters.

Encoders are used to relate specific data values and graphical values to each other. Horizontal/Vertical axes, color keys, size keys and shape keys are some of the different encoders available in SAGE. Each encoder class consists of a definition of the relation between a family of data set attributes and a particular graphical type. SAGE can then use this information to map data values to graphical values in designing a picture, and provide a frame of reference (e.g., axes, keys, etc.) that can be used to visually interpret specific values in the picture. For instance, a color encoder could map data values "less than 5" to the graphical value "blue" and others to "red." A schematic of the encoders used in the chart shown in Figure 3 is shown in Figure 4.

In addition to this knowledge about graphemes, symbols and encoders, SAGE also uses knowledge of the characteristics of data relevant to graphic design (Roth and Mattis, 1990, Roth and Hefley 1993), including knowledge of data types and scales of measurement (e.g., quantitative, interval, ordinal, or nominal data sets), structural relationships among data (e.g., the relation between the end-points of ranges or between the two coordinates of a 2-D geographic location), and the functional dependencies among attributes in database relations (e.g., one:one, one:many, many:many). As we will show later, the latter is an important factor in selecting a high-level discourse strategy for generating explanatory captions.

Finally, SAGE has a library of graphical techniques, knowledge of the appropriateness of the techniques for different data and tasks, and design knowledge for assembling these techniques into composites that can integrate information in a single display. SAGE uses this graphic design knowledge together with the data characterization knowledge to generate displays of information.

To summarize, the portion of SAGE’s knowledge base that is most relevant for generating explanatory captions is its graphical syntax and semantics. The syntax includes a definition of the graphical constituents that convey information: spaces (e.g., charts, maps, tables), (graphemes) (e.g., labels, marks, bars) , their properties (e.g. color, shape), and encoders--the frames of reference that enable their properties to be interpreted/translated back to data values--(e.g., axes, graphical keys.). The syntax also defines the ways in which graphemes can be combined to form symbols--composites that integrate multiple data attributes (e.g., a label attached to a mark). The syntactic structure of a graphical display, like the linguistic structure of text, can provide guidance for creating structurally coherent explanations.

The representation of the semantics of graphics conveys the way data is mapped to the syntactic elements of displays. It also provides guidance for organizing explanatory captions by grouping graphical elements that express data attributes that form a coherent group. The data characterization provides knowledge of the structure of the data and therefore also influences the structure of the explanation.