Newsgroups: comp.lang.smalltalk
Path: cantaloupe.srv.cs.cmu.edu!rochester!udel!delmarva.com!newsfeed.internetmci.com!howland.reston.ans.net!spool.mu.edu!torn!nott!cunews!dbuck
From: dbuck@superior.carleton.ca (Dave Buck)
Subject: Smalltalk vs C vs C++ benchmark results
X-Nntp-Posting-Host: superior.carleton.ca
Message-ID: <DEDsF1.3I8@cunews.carleton.ca>
Sender: news@cunews.carleton.ca (News Administrator)
Organization: Carleton University, Ottawa, Canada
Date: Mon, 4 Sep 1995 12:37:49 GMT
Lines: 111


There are several threads on this newsgroup talking about how fast or
how slow Smalltalk is compared to C and C++, but it seems that nobody
has done any benchmarks to measure the differences.  Some time ago, I
needed to make a case for Smalltalk as a development language, so I
developed a number of benchmarks which tested several programming
tasks in Smalltalk and in C and C++ where applicable.  Here's a
summary of the results with some caveats and quid pro quo's.

First, it's extremely difficult to perform realistic benchmarks in any
language.  All tests below were run 10 times in a loop that iterated
100,000 to 1,000,000 times.  Hopefully, this is enough to measure the
speed of the operation with negligible impact from the loop itself.
All measurements were made based on elapsed time.  It's not accurate,
but it's the best I could do.  For the C and C++ tests, I hand
examined (buit didn't modify) the assembly code created to insure that
the compiler hadn't optimized out the code I was trying to test.  In
many cases, the code was written to prevent static opimization by the
compiler.  I want to test typical runtime speeds, not how clever the
compiler can be to eliminate the runtime code.

The following results came from a 66 MHz PC running OS/2.  The
Smalltalk was VisualWorks and the C++ compiler was the IBM CSet++
compiler.  Similar tests were run with Borland C++ which usually
performed worse than the CSet++ compiler.  The Borland results aren't
available here.  All measurements are elapsed times in seconds.  For
C++ collections tests, the Tools.h++ package from RogueWave Software
were used.  Footnotes are shown in brackets (1).


Test                 C (sec)   C++ (sec)  Smalltalk (sec)
--------------------------------------------------------
alloc/dealloc (1)     5.406      5.78           2.5
1,000,000 objects
of 10 pointers each.
--------------------------------------------------------
function calls(2)     2.187      3.219          4.25
towers of Hanoi
28 disks
--------------------------------------------------------
integer math (3)      1.718        -            10.16
3 million iterations
of 2 adds and
2 multiplies
- 12 million operations
--------------------------------------------------------
float math            2.282        -            50.5
3 million iterations
of 2 adds and 2 multiplies.
12 million operations
--------------------------------------------------------
string comparison (4)  -         10.375         13.9
1 million string
compares
--------------------------------------------------------
array indexing       0.687       21.343         10.0
10 million array
accesses
--------------------------------------------------------
Ordered Collection (5) -          6.25           4.5
accessing
1 million accesses
--------------------------------------------------------
Dictionary access      -          6.625          1.9
100,000 accesses
--------------------------------------------------------
OrderedCollection (6)  -          63.33         66.6
iteration
1 million iterations
over a 10 element
collection


(1)  Smalltalk doesn't have to explicitly deallocate, so to test
Smalltalk, I first forced a generation scavenge then within the timing
loop I ran the allocations and forced another generation scavenge.
The second scavenge was included in the performance time but not the
first.  The C time used malloc/free and the C++ time used new/delete
without constructors.

(2) For this test, the C++ program used a virtual member function.  In
the Borland test, Smalltalk outran Borland C++.

(3) Integer math in C++ is the same code as for C so it wasn't
measured separately.  Smalltalk has no integer overflow problems as
C does.  In the Borland test, Smalltalk out-performed Borland.  All
integers were long signed integers.

(4) In the C test, the compiler completely optimized out my comparison
and I had no time to correct it.  The C++ test used the String class
from RogueWave Software.  Again, the Borland version ran slower than
Smalltalk.

(5) C has no OrderedCollection class.  In both the C++ (RogueWave
Software) and Smalltalk versions, the collection was a static 10
element collection.  Only the time to access the collection was timed.

(6) RogueWave's Iterators were used in the C++ case compared to
Smalltalk's do:.  In the Borland test, the Borland time was almost
twice the Smalltalk time to run the test.

Your mileage may vary.  I encourage you to perform your own tests.

David Buck
dbuck@ccs.carleton.ca

_________________________________
| David K. Buck                 |
| dbuck@ccs.carleton.ca         |
| The Object People             |
|_______________________________|
