Wilfred J. Hansen

"Open source" software available in source form now augments the burgeoning set of COTS (Commercial-off-the-shelf) software products already available. Both introduce new complexities to configuration management. Other projects such as Adele [2] and PCL [9] have concerned themselves with the problems of modeling and managing the complexity arising from externally initiated, asynchronous upgrades; this paper explores the impacts on the task of describing system construction and deployment. Occasionally, deployment can be done with a simple file transfer, but usually it also requires updates to system information, emplacement of icons, initiation of background processes, and the like. We will consider all these deployment difficulties here in the light of some experience with creating scripts for GEE, a pedagogical prototype at the Software Engineering Institute. In particular, this paper will focus on three languages needed to express aspects of the deployment task—languages for expressing environmental conditions, dependencies, and actions.

The main "theme" of the effort to create the construction/deployment system for GEE was Avoidance of Duplication. The goal was that each piece of information be specified in one of a few central definition files and derived from there wherever it was needed. All of the following and more were centralized:

• GEE version number
• directory and file locations
• URLs, host names, and port numbers
• Java CLASSPATH
• command sequences for compilation and testing
• platform dependent flags for system commands

While there is certainly some saving in not repeating lengthy command sequences, the real benefit of Avoiding Duplication is that changes can be made in the definition files and will be consistently reflected throughout the system.

The bulk of the utilization of centralized information was for the generation of the construction/deployment scripts in the form of Makefiles. Most configuration managers offer some mechanism for automatic generation of Makefiles, but descriptions of these systems do not make clear how this occurs. From our experience, no simple scheme where processing is dependent strictly on file extensions is likely to be satisfactory; indeed, seven of the nine Makefiles for GEE components required GEE-specific processing above and beyond compilation and installation. COTS software was accommodated in the system sources by additional directories with idiosyncratic Makefile provisions for

• installation
• addition of libraries to the Java CLASSPATH
• server initiation
• connection between GEE and the COTS product

In what follows, the deployment problems will be largely expressed in terms of deployment of GEE, although the same considerations will apply to any COTS or open source system.

This paper will use the term transmittal for an incoming package containing software to be deployed. A transmittal includes the files to be deployed, instructions for their deployment, and various user documentation such as a description of the features and improvements in the new software.

To simplify the recipient's task, the instructions for deployment should be automated by software which we will call a deployment processor. Its input is the newly released transmittal; its result is an operational system on the target computer. Use of the deployment processor avoids manual command entry, a process which is tedious, time consuming, error-prone and difficult to document due to the abundance of options and the diversity of possible target environments. Indeed, we suspect that creating deployment processor instructions may be easier than writing accurate and comprehensive, easily-followed instructions for manual deployment.

A critical part of a transmittal is the control files to instruct the deployment processor. Hall, Heimbigner, and Wolf [6] (hereafter, "HHW") identify five categories of such information—assert constraints, dependency constraints, artifacts, configuration, and activities—and have implemented a system supporting them [5]. Assert constraints describe dependencies on pre-existing COTS software at the site; dependency constraints do the same but additionally specify actions that can be taken to ameliorate unresolved constraints. The artifacts describe the files in the distribution. The configuration defines system file changes needed to describe the installation. Finally the activities category is a catchall for arbitrary additional activities.

The HHW categories are mainly descriptive/declarative rather than executable. As such they offer opportunities for automatic processing to aid extensively in building and understanding the installation of the transmittal. However, a purely descriptive interface is insufficient for distribution of software in open source form; there must be greater resort to the executable specifications in the "activities" category. Examining the five HHW categories more closely we can find three sublanguages in which portions are written:

Requirements for these sublanguages form the last section of this paper.

Environment Description Sublanguage definitions have been the recent work of two initiatives, Software MIF (now superseded by CIM[1]) and OSD [11]. HHW paints an unpromising picture of these, demonstrating that both are inadequate, even without the added complexity of open source distributions. For the vast majority of users, the issue is moot; they will continue to receive transmittals in object file format and the considerations of HHW will continue to dominate the problem. However, some recipients—we can call them adjunct developers--will receive the package in source form, modify it if necessary, build it, and install the result. The source form of the package must carry with it all the information to enable building and testing the system in addition to deploying it.

Deployment of a COTS product is described by HHW as a life cycle with eight activities:

• Release - The transmittal is assembled by the vendor, typically as an archived collection of files.
• Install - The files are unpacked from the archive and inserted into appropriate directories on the destination system.
• Activate - System files are modified as necessary to allow the user to utilize the product. Icons are added to menus. Initiation of background processes may be arranged.
• Deactivate - Remove the modifications made by Activation.
• Deinstall - Remove files inserted by Installation.
• Derelease - The world is notified of the withdrawal of the product, although sites may continue to use it.
• Update - The transmittal is a set of revised files. Appropriate subsets of Install and Activate are performed to incorporate the new files into the system.
• Adapt – Definition files are modified to tailor the product to local needs and desires. Files such as document boilerplate may be modified or added.

Open source distributions still require all the above activities, but three new ones must be added: source modification, system build, and test. Rather than try to describe these in a life cycle, I have expressed them in the flow diagram of Figure 1, where the data being operated on are named explicitly.

In this Figure, ovals are sets of files, round rectangles are manual processes, and rectangles are activities carried out by the deployment processor under control of its instructions (which are part of the Source Files). Corresponding to each rectangle is another activity which undoes the action of the rectangle: Build is undone by Clean, Install by Deinstall, Test by Exit, Activate by Deactivate, and Update by Revert. These additional activities are omitted here to reduce complexity.

The most comprehensive viewpoint on Figure 1 is that of a vendor developer who has been asked to add a feature to the system. He or she begins by extracting from a source manager the sources for the subsystem to be changed. The steps continue thus:

Fig. 1. Deployment Process. The two "release process" hexagons are instances of the process shown on the right.

Figure 1 shows separate instances of "Release Process" for source and object files; both proceed through the steps shown on the right in the Figure. The ExtractChanges activity determines what files have changed. The Release activity takes those files, adds human readable explanations, and creates the transmittal, most likely as a web site or CD-ROM. Finally, the Update activity modifies the collection of sources or objects at the destination.

A second viewpoint on Figure 1 is that of a typical destination site. Here there are no Source Files, no Changes, and no activities are performed for Modify, Build, Test, Extract Changes, or Release. The Release Process reduces to receiving the Transmittal and Updating (or initially creating) the Object Files. The administrator then goes through the activities of Adaptation, Installation, and Activation to arrive at a working system. Subsequent Updates are triggered by the arrival of a transmittal package containing either a new system or replacement files.

The last viewpoint is that of destination sites having one or more local adjunct developers who modify the Source Files to improve the package or adapt it for local requirements. At such sites all the blocks of Figures 1 are active. The developer may even Extract Changes from the sources and create a Transmittal for return to the vendor. (The version control issues are enormous!) The local developer uses the Test activity to do tests independently from the working system, just as a vendor developer would do.

Since deployment is not a new problem, a number of existing systems provide at least partial solutions. We consider them here in order of increasing capability and complexity.

Projects. Least complex are the various code development environments which support "projects." In Java versions, each file in the project is a Java source file and the project is built by compiling all those files. This mechanism is completely inadequate for serious projects written in multiple source languages or utilizing generator programs to produce sources for later parts of the build. For instance, there is no way to preprocess Java files. This is not supported in the Java philosophy, but is sometimes necessary.

InstallShield for Microsoft Windows [7] copies files to various destinations in the system and makes appropriate changes to certain well known operating system files. This is satisfactory for object file releases, but useless for source releases. Creation of the delivery package utilized by InstallShield is inherently a manual process since it is driven by a GUI user interface. This renders the InstallShield generator unsuitable for the most general case since it cannot be driven from a script or Makefile

Software Dock [5] provides a complete architecture for deployment. A "field dock" at each deployment site keeps track of all software installed thereon. A "release dock" is created for each deployable system. When a user wishes to deploy a piece of software advertised by a release dock, he or she initiates an agent which negotiates between the local field dock and the desired release dock. The two collaborate to install the software and arrange for notification of future update availability. The current implementation has no provision for building from an open source distribution.

Scripting languages. IBM systems have long had the REXX macro language which provides very expressive string handling; Unix systems have various scripting/macro languages like TCL and Perl. All these systems lack an explicit notion of dependencies, but can simulate dependencies with if-then-else. These systems have the same power to solve the deployment problems as any other programming language; their one advantage is in close ties to the shell so program initiation can readily be incorporated in script execution.

When the make process is invoked, the target-file is checked to see if it is older than one or more of the listed source-file(s). If so, the actions are executed and are assumed to produce a newer version of target-file. Since the actions are passed to a shell, absolutely any action can be taken, so make is very general in this respect and has been used for many tasks other than compilation initiation. For instance in a deployment process based on make would there would be a target for each of the activities shown in Figure 1.

Despite its power, make suffers a number of shortcomings:

1. At least one process and often two are initiated for each action. This overhead becomes non-trivial when many actions are needed.
2. The supported dependency mechanism is ill-equipped to deal with the large variety of dependencies required in practice.
3. It may be necessary to specify different variants of the actions depending on the host platform.
4. Makefiles can be tediously repetitious due to its limited mechanisms for avoiding repeated actions for multiple target files.

We will see below that various alternative systems solve one or more of these problems, but there is no comprehensive solution to them all.

Qed/qef. The most general alternative to make is the qed/qef system [10]. As with imake, make’s shortcoming (d), repetition, is solved with macro expansion of a tiny control file in each source directory. A further capability of qed/qef is the provision of several small languages for text processing. These are all interpreted by one process, eliminating much of the system call overhead for actions in make’s shortcoming (a).

The GEE project—the name stands for Generic Enterprise Ensemble—is an ensemble of COTS products and application code that implements a model enterprise-level information system. Customers are expected to study GEE to understand how the COTS products can be used together and may then subsume some generic portions of GEE into their own systems. Accordingly, GEE is distributed in open source form. In a sense, GEE provides a sort of fill-in-the-blanks enterprise information system; we called it a "genotype" because in addition to the functions of a prototype it also serves as a framework for building a real-life system. COTS products used GEE include Java, Netscape servers and browsers, the Visibroker CORBA tool, an Oracle data base, and Live Software's Jrun tool for Java servlets.

As part of GEE, we implemented a system for build and install that supports both local development and eventual distribution. It addresses many needs:

• Multiple versions. Old versions must remain buildable and installed in order to maintain our history and resolve issues raised by users of those versions.
• Partial build. Permit independent builds of components so we can test local changes without rebuilding the entire system.
• Independent testing. Allow each developer to test a newly revised component without installing it into a full-blown version of the system.
• Developer full build. Let each developer have a separate full installation of the system, if desired.
• Full build and install. Rebuild the system every night to ensure that the released version is current.
• Multiple platforms. Support building on different hardware and operating systems.
• Build at foreign site. Enable a distribution recipient to build and install the system.

In the make tradition, build and install are driven from a single control file. The install operation checks first that all object files are up-to-date and, if not, does any necessary compilations

On the basis of our experience with GEE and other projects we can suggest a number of attributes that the three Description Sublanguages must have. In support of each suggestion, we recount some of our experiences with constructing the GEE build/deployment system.

Most of what is needed for the Environment Description Sublanguage is covered by CIM [1]. It provides a rich set of attributes and an object oriented structure. Version numbers, for example, are described as three integers separated with decimal points. For GEE itself we support multiple versions and need multiple versions of Java, so CIM provides attributes that could be tested if GEE and Java were described in CIM.

A single system-wide description of the environment will not suffice. The GEE testing regime attempts to allow each developer a separate environment so there can be independent testing. When two developers are using different versions of Java, their environments are best described in separate environment descriptions. Other tester specific environment information that should be (but sometimes isn’t) allowed in GEE include distinct Visibroker ports, web page areas, Java class paths, servlet collections, and data bases.

New Java distributions were not painless for GEE. Package name changes meant that Java source files using those packages had to be revised. The vendor omitted sending an automated tool for this task and it was small enough for us to do by hand. These changes, however, require us to retain the several Java distributions used by various GEE versions.

In order to be able to utilize environment descriptions, there must be well-known sets of attributes names and for each an expected range of values so dependency descriptions can know what information will be available and the precise meaning for each value. It would be too bad if a deployment looked for Netscape and the environment description said that Mozilla (Netscape's internal name for its browser) was available. The definitions of attributes, expected values, and hierarchies must be ever evolving as new technologies and products appear. Since this evolution must be recorded and maintained by some standards organization, it will be interesting to see how this activity is financed and how that financing affects its stability.

In make, dependencies are expressed in terms of the relative timestamps of source and target files. Clearly the dependency language must also allow expressions referring to attributes and their values from the Environment Description. In addition to the usual arithmetic expressions, the Dependency Description Sublanguage should also offer tests of set membership and numeric range inclusion. Some additional requirements follow.

Multi-file compilation. Java supports simultaneous compilation of any number of source files, which is essential to resolve circularities that cannot be resolved with one-at-a-time compilation. This does not mesh well with the standard make approach which assumes single-source-file compilation. For GEE, we implemented a little program which checks a list of Java class files and determines that the corresponding source file needs to be recompiled if it is more recent. If a change alters a class definition any client class which is not recompiled will encounter a run-time error. Fortunately, this is infrequent because developers usually remember to change all clients when a class definition changes and changes to those source files will ensure their recompilation.

Platform description. It may be necessary to specify different variants of the actions depending on the host platform. This is not the case with Java, but is an important consideration for systems including components written in C or C++.

Recursive component processing dependencies. Java provides a javadoc function to produce a set of web pages describing the interfaces to packages. In preparation for running this command, a list of all source files to be processed is needed. Running javadoc for a single directory is fairly easy, but to get full documentation of all packages, a single invocation of javadoc over the entire set of sources is needed. The full documentation for GEE had to be produced from the build system, and it was also necessary to be able to create the web pages for a single component or subtree of components. Providing all these capabilities under make’s limited dependency language proved to be a challenge. It required five rules in the template file, another rule for each Java package, a separate make target to build a list of Java source files, and two variables defined in each recursive invocation of make.

One large challenge in design of the Dependency Description Sublanguage is to arrange to permit the sort of precedence ordering done by make. As far as possible, the user should not have to sequence the instructions in the description file so as to produce an appropriate order of execution for builds and installs. This and other processing are simplified if the Dependency Description Sublanguage is declarative rather than procedural.

Doubtless, the Action Description Sublanguage can be as trivial as that in make wherein every action is the invocation of a program; indeed, this capability must be made available as the ultimate fallback. We consider this option below after first describing a number of other less general, but more useful facilities.

Built-in Actions. Rather than initiate a process for every action—as is done in make—it seems preferable that the deployment processor provide some set of actions that can be performed directly. These would be available in any environment and would always have the same semantics. A prime example of a desirable built-in action is that of installing a file into a directory. Not only is this capability lacking from make, but it is provided in Unix by two programs both called install but having different command line syntaxes. Thus install had to become one of the GEE properties; it is specified the imake template file. Other useful specialized commands would modify system configuration information as for specifying icons, specifying MIME types, and updating the Environment Description.

Since the C preprocessor, make, the shell, and awk have overlapping quote characters, some tasks were tricky to code. For example, to set the ownership and group of a file it was necessary to write:

Arbitrary programs. As an ultimate fallback, execution of arbitrary programs is necessary in any build/install language, but there are disadvantages. The program must be found and executed, usually requiring more system resources of memory and processes. There is also a heavy reliance on details of the deployment environment, which may be altered even by such peculiarities as individual deployer's idiosyncratic definitions of system commands. In GEE it proved necessary to use the full path name of every system command in order to avoid such definitions. Having no tools except arbitrary programs leads to tiny helper programs which must themselves be source-managed, built, and made executable for at least the duration of the deployment. Each such tool increases the risk of conflict with some such tool introduced for another product.

Smart Executor. Our problems in GEE with interference from developer definitions leads to the notion that a deployment system, or even the full system, can offer a "smart executor." This tool would be cognizant of the environment description and would chose the appropriate version of a tool for each requestor. Among the other GEE problems that this would solve are that one tool, the Visibroker idl2java, needs to execute under Java 1.1 while the rest of GEE is designed for Java 1.2. The smart executor would notice this from the description of idl2java. As another instance, GEE introduced special commands gj, gjc, and vbstart for running Java, compiling Java, and starting Visibroker, respectively. These were necessary to incorporate all the environment information from gee.properties into the execution; environment information which should have been in an Environment Description were one available.

COTS and the advent of open source distributions place new requirements on the deployment processor and its sublanguages for describing the environment, dependencies, and actions. The paper has illustrated these problems with examples from the GEE project undertaken at the Software Engineering Institute.

We have shown that the deployment process itself must be extended to include building and testing the software. These must be provided for by either the deployment processor or an adjunct which serves the role of make. Alternatively, the entire deployment may be conducted by the latter, as it is in GEE. In any case, the processor will permit deployment with a minimum of manual processing, leading to less effort, fewer errors, and more seamless operations.

The main theme of the GEE scripts effort was to Avoid Duplication; as far as possible specifications and data were moved to a few central definition files from which all other instances of these values are derived. This produced a flexible construction system, but one in which names are bound too early. Based on the GEE script effort the paper has pointed out a number of characteristics that would be of value in the description sublanguages for environments, dependencies, and actions. Among other factors, we discovered a great diversity in the requirements imposed by COTS systems: They differed in their mechanisms for installation and interconnection to the application and other COTS products and all added Java libraries that had to be in the CLASSPATH. This complexity would be reduced if all were deployed by a common mechanism.

Do all these experiences suggest that it is time to abandon make? It certainly would be pleasant to avoid some of make’s quoting nightmares. However, make is well known and widely available and still provides enormous flexibility, so it is difficult to see what advantages a new system can offer to wean-away make’s existing users.

Acknowledgements I am indebted to the other members of the GEE team: Scott Hissom, Fred Long, and Robert Seacord under the leadership of Kurt Wallnau.