School of Computer Science
(C) 1993 IEEE. Reprinted with permission, from Proceedings 1993 IEEE Symposium on Visual Languages, August 24-27, 1993, Bergen, Norway, pp. 256-260.
Permission to copy without fee all or part of this material is granted provided that the copies are not made or distributed for direct commercial advantage, the IEEE copyright notice and the title of the publication and its date appear, and notice is given that copying is by permission of the Institute of Electrical and Electronics Engineers. To copy otherwise, or to republish, requires a fee and specific permission.]
Despite numerous innovations in visual languages, relatively little effort has been devoted to integrating visual languages with each other and with textual languages so each can be used for those purposes for which it is best suited. This paper advances a model for multiparadigm language environments which utilizes text in two roles, visual program organization and compiler input. The paper further shows that the Andrew User Interface System is a convenient framework for building language systems on this model. The approach is exemplified with a visual language for specification of string patterns.
In 1991, Glinert, Blattner, and Frerking noted that "the hope is to develop 'general purpose' visual languages which combine textual and graphical elements."  The result should be a multiparadigm language environment [5; 18]. Indeed, Borges  had already reported one such effort, although limited to a specific set of visual components. In such an environment, "visual sublanguages" are employed where appropriate; for instance, decision tables for logic, query-by-example for data base access, modification diagrams for data structure operations, and branching trees to specify keystroke sequences. By working within a common environment, the effort to produce each individual sublanguage is reduced and the value is multiplied by synergy with other sublanguages.
The present paper suggests an open systems model wherein multiple visual components can be combined to produce both general and special purpose languages. Text streams serve two key roles in organizing the environment. In one role, text organizes the visual components of the program for human perusal; in the other, a text collects the rendering of the visual images for sequential input to a compiler. The paper goes on to show that the Andrew User Interface System is an ideal vehicle for the environment because it offers strong support for text and especially because of its architecture for recursive embedding of objects. This architecture provides that objects of arbitrary variety may be nested to arbitrary depth; for instance, an equation in a drawing in a table in a text.
A multiparadigm language environment for visual sublanguages must support not only graphical interaction but also file storage, printing, compilation, and execution. Since multiple sublanguages are to be supported, the environment must provide for management and integration of images from multiple classes. It may be adequate to display each sublanguage element in a separate window, but this runs the risk of chaotic imagery among which it is difficult to discern the relationships among program fragments . Instead, the author should have the power to organize the program fragments for perusal by the reader. The organization itself, together with commentary, aids the reader in comprehending the program. Of course, organizing the program is an extra effort and responsibility for the author, but the result is more maintainable.
Some languages may be expressible with text and a few lines or icons; others may require intricate diagrams and colors. The graphical component of the environment must support both and must provide for creating, revising, viewing and storing the images. For interactive graphics, experience has shown that the environment should be built on an object oriented programming environment. In some cases, it may be desirable for the user to view the text generated to represent the image--for instance to learn how to program the most simple tasks more quickly than by drawing pictures. In other cases, the text would be meaningless. To provide this option, the environment should permit multiple views of objects.
Some sublanguages such as declarations and flow charts may translate readily to code in some traditional programming language; others like data base queries may require the services of a specialized interpreter. Some may generate little code and others enormous quantities. The multiparadigm environment must support a range of code generation options and must support a scheme to coordinate execution of the various components.
The above requirements can be met by a model comprised of two streams of general text, one for the author/reader and a second for compiler input. The second of these may be pure ASCII, but the first must support embedding of objects.
The author/reader text organizes the program within the framework of the text of a programming language; either a traditional language or a new one. Visual sublanguage components of the program are instantiated as objects embedded within this text stream. Ideally, the object environment supports recursive embedding of objects so that sublanguage components may themselves contain text written in either the base language or still other sublanguages.
For presentation to a compiler, a processor scans the author/reader text and generates a second text by requesting each embedded object to render itself as text for compilation. The result is compiled and executed by standard methods or within some suitable debugging environment.
The ability of objects to render thmeselves as text may have other advantages. In some cases, the textual form may be considerably more compact than the picture. By permitting the user to review the textual form generated for a sublanguage object, the system can offer the user the opportunity to learn how to create programs more quickly. In developing new sublanguages, it will be convenient to debug them if their intermediate output is in text. Moreover, text is machine independent so the resulting systems will be more portable.
Objects may adopt different strategies in rendering executable text. The simplest sublanguages will generate text directly in the base language. More complex sublanguages can generate a load module defining the appropriate function; in this case the textual representation need be little more than a call on the function. In other cases, the implementor may choose to add a library or augment the base language itself.
Text is attractive for the author/reader component because it is a natural vehicle for descriptive commentary and because there are many existing tools like editors and cross reference generators. Nonetheless, substrates other than text can also be accomodated in the model. For instance, rather than a linear text, the overall program could be displayed as a picture containing textual commentary and a graph linking the various program objects . Such organizations are interesting experiments in themselves, so the recursive embedding environment should support embedding of objects in substrates other than text.
Support of the model requires an object-oriented environment wherein objects can be embedded in text and recursively within one another. The environment must provide mechanisms so sections in various sublanguages can share screen space, user input devices, data stream, the printed page, and cut/copy/paste operations. Within such an environment there will be a superclass of objects that serve to store expressions in sublanguages. This superclass specifies two methods which must be implemented by subclasses:
Based on the RenderDeclarations, the superclass implements an additional method called RenderEnvironment which merges the tables from an object and all surrounding objects, thus producing a list of all accessible variables. Neither this nor RenderDeclarations are usually needed during compilation. However, these routines are available for semantic checks while creating the visual image.
Corresponding to classes storing sublanguage expressions are other classes responsible for providing the visible view of these objects. These are primarily ordinary graphical objects, but need one additional method to control the form of the image:
3. Andrew satisfies the model
The only system available at this writing that meets the requirements
above is the Andrew Toolkit (ATK) component of the Andrew User Interface
System [15; 1; 17]. In particular, Andrew fully supports recursive
embedding of objects and permits text to be embellished with all the
usual typographic styles. When an object is embedded, it retains its
interactive interface and can continue to be edited. Consider Figure
1 where various objects are embedded in a spreadsheet which is itself
in a text.
Figure 1. Recursively embedded insets.
Many other graphical user interface systems offer excellent support for graphical operations but do not support recursive embedding of images; for instance, Garnet , Interviews , MacApp, and NeXTStep. Systems in development by Taligent and Microsoft may offer such support, but are not yet available.
The ATK mechanisms for recursive embedding of objects are described in more detail in . In addition to a graphical toolkit, Andrew offers complete applications such as a help system, sophisticated mail management, programmable system interface, extension language, and editors for text, rasters, drawings, table/spreadsheets, fonts, preferences and so on.
A quick review will demonstrate that a multiparadigm language environment for visual sublanguages based on Andrew meets the requirements italicized in section 1:
Provide for creating, revising, viewing and storing objects: All of these are inherent in the ATK object design.
Multiple views of objects: ATK pioneered the separation of data from view; it has always supported multiple views of objects.
Code generation options: Because ATK supports full formated text, there is no difficulty generating any desired text.
Coordinate execution: Execution of a single text means a single compilation and execution. For those cases where the sublanguage object generates a separate executable, it can be generated into a directory from which dynamically loaded objects are fetched and will thus be available during execution.
In addition to these mandatory characteristics, ATK has a number of other relevant characteristics, many of which will aid in support of visual sublanguages. These include a change propagation mechanism, window system independence, widget sets, and provision for input of gestures. ATK's Ness extension language  can serve as a base language into which to embed sublanguage objects. Implementation of Ness required development of parsing tools which have themselves become part of ATK. Among these are the tlex token scanner  which is linked to an an augmented Bison parser generator  to create an integrated system where tokens need be specified only once. A symbol table object, sym, provides for the needs of languages with or without nested scopes.
As befits a user interface system, there are already a number of visual elements in programming with the Andrew User Interface System. Indeed, the system has already served as the basis for at least one visual programming system . Other simple examples include the ADEW interface builder , automatic transformation of key and mouse sequences into macros in the Ness language, arbitrary styles and embedded objects in comments in C programs and in both comments and strings in Ness programs.
Implementation of the model for multiparadigm languages is done in Andrew by building on the existing object hierarchy. The two principal classes are dataobject and view. Data objects manage data state information and provide methods for transfer between memory and a data stream. Views access information from data objects, display it on the screen, and provide the user interface operations for modifying it.
Text and textv are subclasses of dataobject and view, respectively, and provide for general styled text with embedded objects. To implement sublanguages the classes sublang and sublangv are defined as subclasses of text and textv. Each individual sublanguage is then implemented as subclasses of sublang and sublangv. Of these, sublang defines the RenderXxx methods which are then overriden by the sublanguage implementation. The text in a sublang object will usually be empty until RenderToText has been called. Thereafter, the text contents are accessed via the methods of the text superclass. The symbol table object, sym, is used for implementing the tables of declared variables required for RenderDeclarations and RenderEnvironment.
In addition to text, ATK offers several other substrates which could be used to organize sublanguage components. Layout provides for scattering objects in a rectangle. Figure does the same and allows the rectangle to be decorated with arbitrary drawing. Table permits rectangular arrays of objects. The page object provides for switching among an arbitrary set of images. Since all of these support recursive embedding, a program could be, for instance, a page object containing a sequence of figures.
4. Example: Pattern Specification
To illustrate Andrew as a multiparadigm language environment, we created an object that extends the Ness language with a visual object for specifying regular expression patterns. See Figures 2 and 3. Coding of the demonstration took about one day. In addition to using text for the base language and the rendered value, the object uses the figure inset to display the actual image. The bulk of the coding effort was to write a parser of regular expressions which generates the appropriate figure objects. With this as a basis, user interactions change the string and the changes are reflected in the image for feedback to the user.
for p := each
in readfile("data") do
replace(start(p), " ")
for s := each (TAB)|(SPACE) in p do
Figure 2. A visual sublanguage in use.
The image is integral to the program.
for p := each
in readfile("data") do
replace(start(p), " ")
for s := each (TAB)|(SPACE) in p do
Figure 3. Text form of a visual sublanguage. A click on the title bar of the regular expression sublanguage in Figure 2 has caused it to display the textual representation shown here.
String processing often specify a sequence of patterns and for each a resulting transformation of the input or sequence of operations. In a typical if-then-else language the patterns become dispersed among the various predicates so it is not easy to read them for similarities. The actions are dispersed among the -then- and -else- clauses so they also cannot be readily compared. The appropriate layout is instead a two column table with predicates in the left column and actions in the right. In Andrew, pattern-action tables can be implemented as a subclass of the table inset. Because recursive embedding is fully supported, the predicate cells can be filled with either text or the pattern object above. Moreover, rendering of the patter-action table into code can combine the patterns to achieve a more optimum scanner. Figure 4 shows how the resulting visual sublanguage appears for a portion of a program to normalize C preprocessor conditional statements. (The first row of the table specifies the subject of the comparison.) For an if-then-else version of this program, see .
Figure 4. Recursive embedding of language constructs. In this mockup, a table specifies the normalization of a C preprocessor line to remove extraneous blanks and add comment delimiters. When the pattern before the arrow is detected, the line is replaced with the text after the arrow, substituting appropriately for <text>. (Five lines of the table have been elided.)
This paper has described a multiparadigm language environment model for integrating visual languages to create a more general language. The model employs text for organizing the visual images and also as the target for rendering sublanguage elements into executable form. That this model is viable has been demonstrated by showing that the Andrew User Interface System well supports the model and further by displaying two sublanguages implemented within Andrew. Use of Andrew as a basis for experiments in visual languages can free researchers from reinventing graphics toolkits so they can concentrate on progress in the human aspects.
Acknowledgements: Brad Myers made helpful comments and Andrew
Plotkin coded the regular expression sublanguage object. I am grateful
for their help.
 Borges, Jose A. and R. E. Johnson, Multiparadigm Visual Programming Languages, Proc. 1990 IEEE Workshop on Visual Languages, Oct 4-6, 1990, Skokie, IL, IEEE Computer Society Press (Los Alamitos, CA, 1990), 233-240.
 Donnelly, C. and R. Stallman, BISON: The YACC-compatible Parser Generator, Free Software Foundation, (Cambridge, 1992).
 Glinert, Ephraim P., Meera M. Blattner, Christopher J. Frerking, Visual Tools and Languages: Directions for the '90s, Proc. 1991 IEEE Workshop on Visual Languages, IEEE Computer Society Press (Los Alamitos, CA, 1991), 89-95.
 Hailpern, B., Multiparadigm Languages and Environments, IEEE Software, 3(1), Jan. 1986, 6-9.
 Hansen, Wilfred J., Enhancing documents with embedded programs: How Ness extends insets in the Andrew Toolkit, Proceedings of IEEE Computer Society 1990 International Conference on Computer Languages, March, 1990, New Orleans, IEEE Computer Society Press (Los Alamitos, CA) 23-32.
 Hansen, W. J., A model for recursive embedding of widgets based on experience with the Andrew Toolkit, Andrew Consortium, Carnegie Mellon Univ., May, 1992.
 Hansen, W. J., Subsequence References: First Class Values for Substrings, ACM Trans. Prog. Lang. and Sys. 14, 4, Oct. 1992.
 Gentlex - a generator of lexical analyzers for parser input, Andrew Consortium, Carnegie Mellon Univ., January, 1993.
 Clayton Lewis and Gary M. Olson, "Can Principles of Cognition Lower the Barriers to Programming?" Report on an informal workshop, University of Colorado, July, 1986.
 Linton, M. A., J. M. Vlissides, P. R. Calder, Composing user interfaces with interviews, IEEE Computer 22(2), February, 1989, 8-22.
 Myers, B. A. and B. Vander Zanden, Environment for rapidly creating interactive design tools, The Visual Computer V. 8(2), Springer Verlag, (Feb, 1992) 94-116.
 Meyers, S., S. P. Reiss, A system for multiparadigm development of software systems, Proc. of the 6th International Workshop on Software Specification and Design, Como, Italy; 25-26 Oct. 1991; IEEE Comput. Soc. Press (Los Alamitos, CA, 1991) 202-9.
 Neuendorffer, Thomas P., ADEW: The Andrew Development Environment Workbench: An Overview, presented at the X Conference, Boston, MA, 1989.
 Palay, Andrew J., Wilfred J. Hansen, et al., The Andrew Toolkit - An Overview, presented at the Usenix Conference, Dallas, TX, January, 1988.
 Rogers, G., The GRClass visual programming system, Proc. of the 1990 IEEE Workshop on Visual Languages, Skokie, IL, 4-6 Oct. 1990, IEEE Comput. Soc. Press, (Los Alamitos, CA, 1990) 48-53.
 Sherman, Mark, D. Anderson, W. J. Hansen, T. P. Neuendorffer, A. J. Palay, Z. Stern, "Allocation of User-Interface Resources in the Andrew Toolkit,", Proceedings of the International Conference on Multimedia Information Systems, (Singapore) McGraw-Hill, January, 1991.
 Zave, P., A compositional approach to multiparadigm programming,
IEEE Software 6(5), (Sept, 1989), 15-25.