AutoTest.DOC
Shelved on the 21st of December 1987
Updated on the 20th of February 1988 with the final comment on testing
for correct instantiation.


                                AUTOTEST


HOW TO USE IT:

There are two main predicates: '$run'(F) and '$run_test'(F).


'$run'(F) expects F  to be an atom naming a  steering file. The steering
file  names  other files,  the  so-called  "test-files". Each  test-file
contains goals to be auto-tested.

'$run_test'(F) expects F  to name a test-file, and not  a steering file.
It tests the goals in F.


STEERING FILE

The steering file  must contain at least no Prolog  terms. Each term can
be:
    (1) log_to( Atom )  or  log_to( List_of_atoms )
    (2) log_all_tests(on)  or  log_all_tests(off)
    (3) The atom 'stay'
    (4) Another atom
If the term is 'log_to(_)', it resets the log files - see later. If it's
log_all_tests,  it   alters  whether  successful  tests   are  recorded.
Otherwise,  the atom  specifies a  filename, that  of a  so-called 'test
file'. If any filename is 'stay', then  the rest of the steering file is
assumed to be the one and only test file.


TEST FILES

Each test file must contain at least  no Prolog terms. Each term must be
one of:
    (1) A structure of the form Goal :: Spec
    (2) log_to( Atom )  or  log_to( List_of_atoms )
    (3) log_all_tests(on)  or  log_all_tests(off)
    (4) Anything else.
:: is an xfx, 255 operator declared in AutoTest.PL.

If the  term is 'log_to(_)' or  'log_all_tests(_)', it behaves  like the
same term in a steering file.

If the term  is not of the  form Goal :: Spec, then  AutoTest will write
it to the log file(s). Terms of these forms are treated specially:
    (a) A <> B      - recursively write A and B
    (b) A ... B     - equivalent to A <> ' ' <> B
    (c) nl          - emit a newline
    (d) '$'(S)      - assume S is a list of character codes, and convert
                      it to an atom before writing.
<> and ... are xfy, 40 operators.


GOALS AND SPECIFICATIONS

If the term is a structure, Goal::Spec, then the Spec must be one of
    (a) The atom 's' or 'succeeds'
    (b) The atom 'f' or 'fails'
    (c) The atom 'c' or 'crashes'
    (d) A structure, 'c(ErrorKind)' or 'crashes(ErrorKind)'
    (e) Anything else

(a) to  (d) specify whether the  Goal should succeed, or  fail, or crash
with some error. (e) specifies some  more tests, given as a Prolog goal,
which must succeed. Usually, they  will contain tests on variables bound
by Goal.

Examples might be:
    (a)  append( [1], [2], [1,2] ) :: s.
    (b)  member( x, [] ) :: f.
    (c)  see( V ) :: c.
    (d)  see( 'Not a filename' ) ::
         crashes( error('-RMS-E-FNF, file not found') ).
    (e)  member( X, [1,2,3,4] ) :: X == 4.

When testing a  Goal::Spec term, AutoTest calls the Goal  (and then cuts
it, to prevent backtracking into it). It notes whether the Goal:
    (a) Succeeded
    (b) Failed
    (c) Caused an error
If (c), AutoTest tries to determine the error-code, if any.

AutoTest then compares the Goal's effect  with its Spec. If they differ,
it  writes a  message  stating what  did happen,  and  what should  have
happened. In doing this:
    Goal's success obviously matches 's' or 'succeeds'
    Goal's failure obviously matches 'f' or 'fails'
    Goal's crashing matches 'c' or 'crashes' or 'c(_)' or 'crashes(_)'
    - except  that, if AutoTest  can determine  an error code,  and Spec
      specified a  DIFFERENT code,  then AutoTest  will write  a message
      saying so.
    If  the  Spec is a  Prolog  goal,  AutoTest calls it. If it fails or
    causes an error, AutoTest reports that.

Normally, AutoTest will not log successful  tests. You can make it do so
by including the term 'log_all_tests(on)'  in the steering or test file.
This will cause all tests, whether  or not they succeeded, to be logged.
A subsequent  'log_all_tests(off)' will revert AutoTest  to logging only
failed tests.


TRAPPING ERRORS

Note that the way errors  are detected is implementation-dependent. This
version of  AutoTest can't  detect them, because  it relies  on standard
Edinburgh Prolog. You will have to modify the predicate
    '$call_giving_sfe'
to  trap errors,  and  return  a structure 'e(ErrorCode)' in its  second
argument; or the atom 'e' if you can't find out the error code.


LOGGING

When  AutoTest   writes  messages,   it  does   so  via   the  predicate
'$writef_to_log',  which  sends  to  the  log  files  specified  by  the
assertion '$log_files' below.

'$writef_to_log' writes the same message to each file in turn. The files
are closed when AutoTest finishes.

If you give a 'log_to(FL)' in the steering file, FL must be one atom, or
a list  of atoms, specifying  the names of  the new log  files. AutoTest
closes any log  files which are not  named in FL, and makes  those in FL
the new files. They are not opened until first written to.


PREDICATE NAMES

In order to avoid interfering with predicates under test, all predicates
in  AutoTest.PL have  names starting  with $  (dollar). There  are three
infix operators declared: "::", "<>", and "...".


OPEN FILES

If an error occurs while AutoTest is  reading a file, then that file may
not  be closed  properly. This  means that  the next  call of  '$run' or
'$run_test' may start part of the way down the file.


EXAMPLES

There now  follows a sample steering  file, between (not  including) the
line of equals signs.

========================================================================
/*  AUTOTEST.STEER  */

/*  Steering file for AutoTest.PL  */

'autotest.tst'.
========================================================================

That file would cause the contents of AUTOTEST.TST to be tested.


Here is AUTOTEST.TST, again between equals signs.

========================================================================
/*  AUTOTEST.TST  */

/*  SAMPLE INPUT TO AUTOTEST.PL

    This input contains some terms which should provoke messages
    from AutoTest.

    This is the expected output:

Line 1 of AutoTest.TST
Goal 5 = 5 should have failed but suceeded
Goal 6 = 6 should have failed but suceeded
Goal 7 \= 7 should have suceeded but failed
Goal 8 \= 8 should have suceeded but failed
Goal 9 = 9 should have crashed but suceeded
Goal 10 = 10 should have crashed but suceeded
Goal 11 \= 11 should have crashed but failed
Goal 12 \= 12 should have crashed but failed
Goal 13 = 13 should have crashed with error 137 but suceeded
Goal 14 = 14 should have crashed with error 137 but suceeded
Goal 15 \= 15 should have crashed with error 137 but failed
Goal 16 \= 16 should have crashed with error 137 but failed
Goal functor(f(1, 2), f, 2) should have passed tests f = g , 2 = 3 but failed
Final line of AutoTest.TST

*/

'Line 1 of' ...
'AutoTest.TST'<>nl.             /*  Display  */

1 = 1 :: s.                     /*  OK  */

2 = 2 :: succeeds.              /*  OK  */

3 \= 3 :: f.                    /*  OK  */

4 \= 4 :: fails.                /*  OK  */

5 = 5 :: f.                     /*  Message  */

6 = 6 :: fails.                 /*  Message  */

7 \= 7 :: s.                    /*  Message  */

8 \= 8 :: succeeds.             /*  Message  */

9 = 9 :: c.                     /*  Message  */

10 = 10 :: crashes.             /*  Message  */

11 \= 11 :: c.                  /*  Message  */

12 \= 12 :: crashes.            /*  Message  */

13 = 13 :: c(137).              /*  Message  */

14 = 14 :: crashes(137).        /*  Message  */

15 \= 15 :: c(137).             /*  Message  */

16 \= 16 :: crashes(137).       /*  Message  */

17 = X :: X = 17.               /*  OK  */

functor( f(1,2), F, A ) ::
    F = f, A = 2.               /*  OK  */

functor( f(1,2), F, A ) ::
    F = g, A = 3.               /*  Message  */

'Final line of'<>
' AutoTest.TST'<>nl.            /*  Display  */
========================================================================

Here is another steering file:

========================================================================
/*  DEMO.STEER  */
/*  Steering file for AutoTest.PL  */


'autotest.tst'.             /* Log to files specified in AutoTest.PL */
log_to( 'fred' ).           /* Log to FRED.                          */
'test2.pl'.                 /* TEST2.PL is the next test file.       */
log_to( [bert,joe,me] ).    /* Log to BERT., JOE., ME.               */
'[-]TEST3.TST'.             /* [-]TEST3.TST is the next test file.   */
log_to( user ).             /* Now log just to the terminal.         */
stay.                       /* Future test input comes from here.    */
fail::s.
log_all_tests(on).          /* Report future tests, even if OK.      */
true::f.
'Final line of steering file.'<>nl.
========================================================================


NOTE ON TESTING FOR PROPER INSTANTIATION

Suppose we want to test the predicate 'append', which joins two lists to
make a third. The obvious way to do this is:

    append( [1,2,3,4], [5,6,7,8], [1,2,3,4,5,6,7,8] ) :: s.

Now, suppose that our 'append' works correctly if the third argument is
instantiated at call, so
    ?- append( [1,2,3,4], [5,6,7,8], [1,2,3,4,5,6,7,8] ).
succeeds. Suppose  further that it doesn't  quite work correctly  if the
third argument is uninstantiated. Instead, it gives a wrong solution; if
you backtrack into it, it then gives the correct solution:

    ?- append( [1,2,3,4], [5,6,7,8], X ).
    X = []
    More ? (yes/no)
    yes
    X = [1,2,3,4,5,6,7,8]

(This is actually unlikely to happen with the way that most people write
'append', but it's  a good illustration of what can  happen when testing
other predicates).


If you autotest 'append' with  the third argument instantiated as above,
this bug will not be invoked. If you test it like this:

    append( [1,2,3,4], [5,6,7,8], X ), X = [1,2,3,4,5,6,7,8] :: s.

the bug will be invoked. However, although the first value for X will be
wrong, the goal X=[1,2,3,4,5,6,7,8] will force backtracking into append;
append will  then provide  a correct value  for X;  the whole  test will
succeed; so the bug won't be noticed.


Get round this by using this test:
    append( [1,2,3,4], [5,6,7,8], X ) :: X = [1,2,3,4,5,6,7,8].
As soon as 'append' returns with its first (wrong) value for X, AutoTest
will cut it, so 'append' can never return the second (correct) value for
X. The post-test X=[1,2,3,4,5,6,7,8] will fail, and AutoTest will report
this, hence making the bug obvious.
