summary of shootout results

[Ken Anderson <address@hidden>]

Here is a summary of performance data from:
http://www.bagley.org/~doug/shootout/

The site compares 31 language implementations with 25 benchmarks,
which typically are a page in length.  I chose 15 of the languages,
which are the types we've been talking about on this list:

C: gcc, g++
Java
Functional: ocaml, ocamlb (byte coded), milton, smlnj, haskell
Lisp: bigloo, cmucl, guile
scripting: perl, njs (java script), python, ruby

The exhibits below show order statistics of the rank that the
language implementation occurs in for each benchmark, for cpu time
(cpu) and lines of code (loc).  So for example, cpu0 is the minimum
rank, cpu50 is the median rank, and cpu100 is the maximum rank.
In both tables, the languages are ordered by the median.

Exhibit 1: CPU rank sorted by cpu50:
language cpu0 cpu25 cpu50 cpu75 cpu100
      gcc    1     1     1     2     10
    ocaml    1     2     3     4      9
      g++    1     3     4     6     20
   milton    1     3     4     7     10
    cmucl    3     5     8    10     26
   bigloo    2     6     8     9     14
    smlnj    1     7     9    12     27
  haskell    1     9    11    23     26
   ocamlb    6     9    12    14     21
     java    6    11    12    15     31
     perl    6    11    14    21     26
      njs    6    14    17    20     28
   python   13    15    19    22     25
     ruby   12    17    20    23     28
    guile    5    20    23    25     29

While gcc comes in first half the time, gcc, ocaml and milton are
always in the top 10. Ocaml, milton, smlnj, and haskell are faster
than gcc in at least one benchmark.  Cmucl and bigloo are not far
behind.

If your favorite language is in the first 15 out of 31, half the time,
i'd consider it reasonably fast.  Performance across the benchmarks
is difficult to predict.  For example, while guile is among the
slowest, it manages to come in 5th in a threading benchmark.  This
observation is consistent with "Timing Trials, or, the Trials of
Timing: Experiments with Scripting and User-Interface Languages"
http://cm.bell-labs.com/cm/cs/who/bwk/interps/pap.html.  Some of the
code in the shootout benchmarks came originally from this paper.

I'm not sure if the benchmarks are done in a way that would let the
Java HotSpot compiler do its optimizations, so its performance might
be a bit low.

Exhibit 2: Lines of code rank, sorted by loc50.
language loc0 loc25 loc50 loc75 loc100
    ocaml    1     1     2     9     22
   ocamlb    1     1     2     9     22
     ruby    1     3     4     8     17
     perl    1     4     6    10     24
  haskell    1     1     7    17     30
   python    1     6     8    10     20
    guile    1     4     8    13     23
    cmucl    1     7    11    19     26
      njs    1     8    14    19     23
   bigloo    4    14    17    21     22
     java    7    14    17    21     28
      g++    3    16    19    22     28
      gcc    7    19    21    23     28
   milton    1    19    23    27     30
    smlnj   14    22    25    28     30

Ocaml also does great in LOC, as does ruby, perl, haskell and python.
Lisp languages tend to be in the middle.  I haven't looked to hard
into why milton and smlnj are worse at loc.  I suspect that part of it
is a need to define nonstandard library functions.

So while speed is important, less code is probably more so, and sometimes 
you can have both.