ISSUE: Library interchange format

REVISION HISTORY: Rob MacLachlan, 2 February 94

STATUS: Open

RELATED ISSUES: Headers for Dylan Source Files
		Library Descriptions
		Evaluation of Top-Level Forms 

CATEGORY: Addition

PROBLEM DESCRIPTION:

"Library Descriptions" says:
 -- The mechanism by which the library-name in a library-use-clause is
    associated with the corresponding library description is
    implementation-defined.
 -- The association between library names and related persistent data such as
    source text [...] is implementation defined so as not to require any
    particular representation of this format.

The intention is that these associations of source code with libraries will be
maintained by the programming environment, probably using some non-file-based
organizational framework.  

However, this requires some extra work when a program is moved between Dylan
implementations, since there is no standard interchange format for this
information.  It is not possible to simply unpack a source distribution and
build the system.  Users would presumably have to consult some human-readable
documentation containing this structural information, and would then use an
implementation-specific means to enter the structural information.


PROPOSAL:

This proposal has two related parts:
 -- A set of naming and source-tree organization conventions which allow
    easy interchange of source trees between different filesystems, and
 -- A formatted library interchange definition (LID) file which structures
    this source tree into Dylan libraries.

A LID file is a file with newline separated entries in the following format:
    LIBRARY INTERCHANGE DEFINITION<1: constant text, case not significant>
    <2: one line comment, possibly blank>
    <3: full Dylan library name>
    <4: Dylan library and module definition file>
    <5: first Dylan executable code file>
    <6: second Dylan executable code file>
    ...

1,2:
    The header[1] and comment line[2] provide a file format tag and space
    for version information.

  3: 
    The full Dylan library name (as used in the "define library" form) is
    next.

  4:
    The name of the file containing all "define library" and
    "define module" forms should appear next.

5,6,...:
    Following lines are source files which contain executable code.

The order of executable files in the LID file determines the initialization
order across the files within the defined library (see "Evaluation of
Top-Level Forms".)

It is assumed that there is some "type" associated with all files which allows
LID files to be distinguished from all other files that appear in a Dylan
source distribution.  In the systems where the "type" is a text string, this
type should be "lid".

The file names that appear in the LID file may be any number of mixed-case
alphanumeric characters.  These names must be unique in the first eight
characters (disregarding case).  To obtain the actual file name in the
distribution tree, these names are truncated to eight characters and
case-folded to the implementation's canonical case (such as lowercase for
Unix.)  Permitting longer names in the LID file allows the full name to be
automatically recovered.

Note that:
 -- Since file names don't contain any "directory" part, all of the source
    files directly associated with a library should be placed in the same
    directory as the LID file.
 -- Since file names don't contain any "type" part, this must be defaulted by
    the implementation.  Whenever possible, implementations should use the
    type "dyl" as the interchange source file type. 

The name of the LID file is not significant, and in particular need not be the
same as the library name.  Hierarchical directory structure can be used to
organize multi-library systems as long as the files directly associated with
each library are in a single directory.

In practice, importing a source distribution into a Dylan system involves
unpacking the source distribution into its own subtree and then informing the
Dylan implementation of the location of the tree root.  The implementation
walks the entire subtree locating LID files (by their type) and building the
association between library definitions, library names and source files.


RATIONALE:

This proposal:
 -- Avoids any explicit use of directory names or file types, and
 -- Uses short file names with a restricted character set.

This achieves the portability goal by allowing transport of Dylan programs to
most (all?) filesystem without requiring significant user intervention (such
as file renaming.)  If file renaming is necessary, only the LID files need be
updated, since only LID files contain file names.

This proposal also:
 -- Gives an interchange format that is highly human-readable
    and easy to manually maintain.  File-based Dylan environments could
    permanently maintain source code in interchange format.
 -- Allows automated generation of interchange format by
    non-file-based environments.  Since there is no relation between any of
    the file names and Dylan identifiers, it would be possible to use
    meaningless (but unique) file names.  In reality, environments would
    probably want to make the file names as meaningful as possible.

During development, file-based environments might want to allow long filenames
and directories.  An export tool would create a tree which truncates names to
eight characters and flattens out directories.  The reason for requiring
pre-truncation of interchange file names is that this avoids triggering
unknown renaming heuristics in the transport mechanism.


EXAMPLES:

Because user code must be placed in a user-defined module, and this can only
be done by the Module: file header option, at least three files are required
for interchange format:
 -- The Library Interchange Definition (LID) file, containing a file list.
 -- The library/module definition file.
 -- One or more files containing source code.

File: fact.lid
________________________________________________________________
Library interchange definition

Factorial
factdef
fact
________________________________________________________________


File: factdef.dyl
________________________________________________________________

define library factorial
  use dylan;
  export factorial-module;
end;

define module factorial-module
  export fact;
end;
________________________________________________________________


File: fact.dyl
________________________________________________________________
Module: factorial-module

define generic fact(n);

define method fact(n == 0)
  1;
end;

define method fact(n)
  n * fact(n - 1);
end;
________________________________________________________________


COST TO IMPLEMENTORS: 
    Slight additional implementation effort to locate and parse LID files.

COST TO USERS:
    LID file syntax might need to be understood by some users.

PERFORMANCE IMPACT: none

BENEFITS: portability

AESTHETICS:

FUTURE FEATURES: 
    Doesn't seem to preclude any non-file-based implementation strategies, 
    but programmers might conceivably need to specify some additional
    information in order to allow the interchange format to be generated,
    especially if the file names are to be human-readable.