Re: summary of shootout results

[Paul Graham <address@hidden>]

I'd be pretty dubious about the LOC metrics.  I suspect what
they are really measuring is the presence or absence of
libraries, not the expressive power of the core language.  

It's hard to write LOC benchmarks that really test the latter.
For one (inadvertant) example see:

http://www.paulgraham.com/accgen.html

--Ken Anderson wrote:
> Here is a summary of performance data from:
> http://www.bagley.org/~doug/shootout/
> 
> The site compares 31 language implementations with 25 benchmarks,
> which typically are a page in length.  I chose 15 of the languages,
> which are the types we've been talking about on this list:
> 
> C: gcc, g++
> Java
> Functional: ocaml, ocamlb (byte coded), milton, smlnj, haskell
> Lisp: bigloo, cmucl, guile
> scripting: perl, njs (java script), python, ruby
> 
> The exhibits below show order statistics of the rank that the
> language implementation occurs in for each benchmark, for cpu time
> (cpu) and lines of code (loc).  So for example, cpu0 is the minimum
> rank, cpu50 is the median rank, and cpu100 is the maximum rank.
> In both tables, the languages are ordered by the median.
> 
> Exhibit 1: CPU rank sorted by cpu50:
> language cpu0 cpu25 cpu50 cpu75 cpu100
>       gcc    1     1     1     2     10
>     ocaml    1     2     3     4      9
>       g++    1     3     4     6     20
>    milton    1     3     4     7     10
>     cmucl    3     5     8    10     26
>    bigloo    2     6     8     9     14
>     smlnj    1     7     9    12     27
>   haskell    1     9    11    23     26
>    ocamlb    6     9    12    14     21
>      java    6    11    12    15     31
>      perl    6    11    14    21     26
>       njs    6    14    17    20     28
>    python   13    15    19    22     25
>      ruby   12    17    20    23     28
>     guile    5    20    23    25     29
> 
> While gcc comes in first half the time, gcc, ocaml and milton are
> always in the top 10. Ocaml, milton, smlnj, and haskell are faster
> than gcc in at least one benchmark.  Cmucl and bigloo are not far
> behind.
> 
> If your favorite language is in the first 15 out of 31, half the time,
> i'd consider it reasonably fast.  Performance across the benchmarks
> is difficult to predict.  For example, while guile is among the
> slowest, it manages to come in 5th in a threading benchmark.  This
> observation is consistent with "Timing Trials, or, the Trials of
> Timing: Experiments with Scripting and User-Interface Languages"
> http://cm.bell-labs.com/cm/cs/who/bwk/interps/pap.html.  Some of the
> code in the shootout benchmarks came originally from this paper.
> 
> I'm not sure if the benchmarks are done in a way that would let the
> Java HotSpot compiler do its optimizations, so its performance might
> be a bit low.
> 
> Exhibit 2: Lines of code rank, sorted by loc50.
> language loc0 loc25 loc50 loc75 loc100
>     ocaml    1     1     2     9     22
>    ocamlb    1     1     2     9     22
>      ruby    1     3     4     8     17
>      perl    1     4     6    10     24
>   haskell    1     1     7    17     30
>    python    1     6     8    10     20
>     guile    1     4     8    13     23
>     cmucl    1     7    11    19     26
>       njs    1     8    14    19     23
>    bigloo    4    14    17    21     22
>      java    7    14    17    21     28
>       g++    3    16    19    22     28
>       gcc    7    19    21    23     28
>    milton    1    19    23    27     30
>     smlnj   14    22    25    28     30
> 
> Ocaml also does great in LOC, as does ruby, perl, haskell and python.
> Lisp languages tend to be in the middle.  I haven't looked to hard
> into why milton and smlnj are worse at loc.  I suspect that part of it
> is a need to define nonstandard library functions.
> 
> So while speed is important, less code is probably more so, and sometimes 
> you can have both.
>