Newsgroups: comp.lang.scheme,comp.lang.lisp
Path: cantaloupe.srv.cs.cmu.edu!das-news2.harvard.edu!news2.near.net!news.mathworks.com!news.alpha.net!uwm.edu!math.ohio-state.edu!jussieu.fr!univ-lyon1.fr!swidir.switch.ch!scsing.switch.ch!news.dfn.de!news.dkrz.de!news.rrz.uni-hamburg.de!news.Hanse.DE!wavehh.hanse.de!cracauer
From: cracauer@wavehh.hanse.de (Martin Cracauer)
Subject: Re: Benchmarking results for modern Lisps and Schemes
Message-ID: <1995Mar7.111833.10559@wavehh.hanse.de>
Organization: The Internet
References: <1995Mar7.074834.18661@ptolemy-ethernet.arc.nasa.gov>
Date: Tue, 7 Mar 95 11:18:33 GMT
Lines: 75
Xref: glinda.oz.cs.cmu.edu comp.lang.scheme:12262 comp.lang.lisp:16972

At one time (last May?) I tried to collect a useful benchmark suite
for Common Lisp. I found that most `real-world' programs (as compared
to benchmark-only programs in the gabriel benches) are not portable
enough between CLs, at least not when highly optimized. Not to speak
of a CL <-> Scheme comparison.

The only useful benchmark suite I found myself to be able to collect
would be a collection of Screamer code. Screamer is ported and
optimized on a wide range of platforms and Screamer applications are
not implementation-specific anymore, while they still show different
strengths of given architectures. But this is not a useful benchmark
to predict performance of random CL programs. I could dig up my old
work if you really want such a Screamer-based suite.


To go some deeper, I found that performance is much more
application-dependent for Lisp programs as it is for other languages.

Example: While gcl generally reaches about 1/2 of the speed as my
native compilers (Lispworks and CMU) for my `real work', it needs
about 100 times the runtime for a isolated CLOS-message-call benchmark
I did for inter-language comparison. This is probably because CMU and
Lispworks recognize message calls and implement them in machine
language, while simple PCL implementations have Lisp code running
(don't really know). I bet that gcl can be tuned to be at least 10
times faster by a few days hacking. In a word: Whatever you benchmark,
it is not likely to reflect the quality of the overall implementation.

I have been disappointed by overall CL performance anyway and don't use
Lisp variants for performance-oriented applications anymore. Maybe
that is another reason why I lost interest in collecting a benchmark
suite, I don't need a fast Lisp anymore.

k p c <kpc@ptolemy.arc.nasa.gov> writes:

>I looked (*) for benchmarking results for recent versions of Lisp and
>Scheme implementations on modern machines and did not find any.

>Has anybody done informal or formal (**) speed or space tests for any
>two of the following Lisps, preferably under Solaris (***)?

In general, I think the big machine-code-generating CL compilers
(Lucid, CMU, Allegro, Lispworks) can blow other implementations away
if you really go into optimization and have code that can be tuned
(much low-level operations). gcl is generally faster than clisp (but
not that much), while clisp is more comfortable. gcl should be easier
to tune for a specific problem.

I didn't use Scheme that much, but I found elk to be faster and more
stable than scm and stk. But performance is not that greatest
difference between them.

>(**) I would definitely be satisfied with times for the Gabriel Lisp
>benchmarks (referenced in the MK FAQ) with as many variables as
>possible controlled.

The garbiel benchmarks are useful to tune an implementation. They
don't give an idea about the performance of real applications.

BTW, you can easily try performance of CL variants out. All but the
commercials are ftp-able and you can get evaluation copies of
Lispworks, Lucid (from Harlequin, too) and Allegro easily.

If you choose a commercial implementation, you can call Support to
help you with implementation-sepcific optimization. That can be more
important than the general quality of the optimizer. CMUCL seems to be
the fastest when it comes to things like processing large
floatig-point vectors.

If I really needed a fast Lisp, I would look at the underlying
technology of the compiler, so that I could make an assumption about
it's potential. And then choose the fastest I can get source for so I
can make it reach it's potential.
-- 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Private email Martin.Cracauer@wavehh.hanse.de Fax +4940 522 8536. No NeXTMail!
