Issue:        MACRO-CACHING
Forum:	      Compiler
References:   8.2 Macro Expansion (CLtL pp151-152),
	      Issues PACKAGE-CLUTTER, LISP-SYMBOL-REDEFINITION, 
	      CONSTANT-MODIFICATION,
	      and MACRO-ENVIRONMENT-EXTENT
Category:     Clarification
Edit history: 31-Jan-89, Version 1 by Pitman
	      11-Mar-89, Version 2 by Loosemore (add discussion)
	      30-May-89, Version 3 by Loosemore (simplify, rewrite)
Status:	      proposal DISALLOW passed June 89

Problem Description:

 The description of *MACROEXPAND-HOOK* in CLtL states that its purpose
 is "to facilitate various techniques for improving interpretation
 speed by caching macro expansions".  However, there is no portable way
 to correctly perform such caching.

 Caching by displacement won't work because the same (EQ) macro call
 form may appear in distinct lexical contexts.  In addition, the macro
 call form may be a read-only constant.

 Caching by table lookup won't work because such a table would have to
 be keyed by both the macro call form and the environment, and proposal
 MACRO-ENVIRONMENT-EXTENT:DYNAMIC (passed at the March 1989 meeting)
 states that macro environments are permitted to have only dynamic
 extent.

 Caching by storing macro call forms and expansions within the
 environment object itself would work, but there are no portable
 primitives that would allow users to do this.


Proposal (MACRO-CACHING:DISALLOW):

 (1) Remove the suggestion that *MACROEXPAND-HOOK* be used for caching
     macroexpansions.  Instead, suggest that it might be used for
     debugging purposes.

 (2) Clarify that although there is no correct portable way to use
     *MACROEXPAND-HOOK* to cache macro expansions, there is no
     requirement that an implementation call the macro expansion
     function more than once for a given form and lexical environment.

 Rationale:

  Item (1) fixes the description of what *MACROEXPAND-HOOK* is for, from
  the point of view of a user.  Item (2) allows implementors to use
  other, correct but nonportable techniques for caching macro
  expansions.

Proposal (MACRO-CACHING:DEPRECATE):

 This is the same as DISALLOW, but also deprecate *MACROEXPAND-HOOK*.

 Rationale:

  Since *MACROEXPAND-HOOK* has now been shown to be unusable for its
  original stated purpose, it is of questionable usefulness.


Test Case:

 ;; #1: File compiling this definition in some implementations will produce
 ;;     a definition that returns read-only list structure. The call to EVAL
 ;;     on the result must not try to modify the read-only structure during
 ;;     macroexpansion. [See issue CONSTANT-MODIFICATION.]

 (DEFUN READ-ONLY-FOO () '(MACROLET ((FOO (&WHOLE FORM) (+ 1 1))) (FOO)))

 (EVAL (READ-ONLY-FOO))
 => 2

 ;; #2: This constructs a form and then uses it in two places in another
 ;;     constructed form. Each of the uses is in a different lexical
 ;;     contour, so must be expanded differently.

 (LET ((FOO (LIST 'FOO)))
   (EVAL `(LIST (MACROLET ((FOO (&WHOLE FORM) '(+ 1 1))) ,FOO)
		(MACROLET ((FOO (&WHOLE FORM) '(+ 1 2))) ,FOO))))
 => (2 3)  

 ;; #3: This is effectively the same thing but involves a MACROLET
 ;;     shadowing a DEFMACRO rather than two MACROLETs, since some
 ;;     implementations might only be caching expansions that come
 ;;     from DEFMACRO.

 (DEFMACRO FOO (&WHOLE FORM) '(+ 1 1))

 (LET ((FOO (LIST 'FOO)))
   (EVAL `(LIST ,FOO (MACROLET ((FOO (&WHOLE FORM) '(+ 1 2))) ,FOO))))
 => (2 3)


Current Practice:

 Symbolics Genera does not use displacing or table caching in either
 the interpreter or compiler.

 Symbolics Cloe, a compiled only implementation, uses table caching
 to boost compilation by a little. Running the test cases above turned
 up a bug (in test case #3), which is now in the process of being fixed.
 [The fact that a bug was turned up in code written by a CL implementor
  is an existence proof that the potential for trouble was not imagined.]

 The TI Explorer evaluator does displacement of macros, but is careful
 to correctly handle the cases exemplified in test cases #1 and #2.  
 It does not do the right thing for #3, but that is a bug that can
 fairly easily be fixed.

Cost to Implementors:

 This proposal is upward compatible with correct implementations.

Cost to Users:

 There is no cost to users, unless they were using semantically invalid
 or nonportable caching techniques.  Nonportable caching techniques might
 continue to work in some implementations.

Cost of Non-Adoption:

 Continued confusion about the purpose of *MACROEXPAND-HOOK* and the
 validity of macro caching techniques.

Benefits:

 The misleading description of *MACROEXPAND-HOOK*'s purpose is
 removed.

Aesthetics:

 Most people agree that macro caching techniques are only supposed to
 improve speed without affecting semantics.  This proposal is only
 intended to underscore that necessary truth.  Insofar as this is only
 a clarification, it presumably has no significant aesthetic impact.

Discussion: