4.2 Benchmarks

We will present a table with the performance of three scheme systems on a number of benchmarks: interpreted SCM, byte-compiled VSCM and hobbit-compiled code. The upper 13 benchmarks of the table are the famous Gabriel benchmarks (originally written for lisp) modified for scheme by Will Clinger. The lower five benchmarks of the table are proposed by other people. Selfcompile is the self-compile time of Hobbit.

Hobbit performs well on most of the benchmarks except CPSTAK and CTAK: CPSTAK is a closure-intensive tailrecursive benchmark and CTAK is a continuations-intensive benchmark. Hobbit performs extremely well on these benchmarks which essentially satisfy the criterias for well-optimizable code outlined in the section 6 above.

FFT is real-arithmetic-intensive.

All times are in seconds.

SCM 4c0(U) and 1.1.5*(U) (the latter is the newest version of VSCM) are compiled and run by Matthias Blume on a DecStation 5000 (Ultrix). VSCM is a bytecode-compiler using continuation-passing style, and is well optimized for continuations and closures.

SCM 4e2(S) and Hobbit4b(S) compiled (with ‘cc -xO3’) and run by Tanel Tammet on a Sun SS10 (lips.cs.chalmers.se). Hobbit is a Scheme-to-C compiler for SCM, the code it produces does not do any checking. SCM and hobbit are not optimized for continuations. Hobbit is not optimized for closures and proper mutual tailrecursion.

SCM and Hobbit benchmarks were run giving ca 8 MB of free heap space before each test.

Benchmark       |SCM 4c0(U) 1.1.5*(U)| SCM 4e2(S)  Hobbit4b(S)
----------------|------------------------------------------------
Deriv           | 3.40      3.86     |  2.9            0.18
Div-iter        | 3.45      2.12     |  2.6            0.083
Div-rec         | 3.45      2.55     |  3.5            0.42
TAK             | 1.81      1.71     |  1.4            0.018
TAKL            |14.50     11.32     | 13.8(1.8 in gc) 0.13
TAKR            | 2.20      1.64     |  1.7 1.5        0.018
Destruct        | ?          ?       |  7.4(1.8 in gc) 0.18
Boyer           | ?          ?       | 27.(3.8 in gc)  1.9
CPSTAK          | 2.72      2.64     |  2.0 1.92       3.46(2.83 in gc)
CTAK            |31.0       4.11     | memory          memory
CTAK(7 6 1)     | ?          ?       |  0.83           0.74
FFT             |12.45     15.7      | 11.4 10.8       1.0
Puzzle          | 0.28      0.41     |  0.46(0.22 gc)  0.03
----------------------------------------------------------------
(recfib 25)     | ?          ?       |  4.1            0.079
(recfib 30)     | ?          ?       | 55. (10.in gc)  0.87
(pi 300 3)      | ?          ?       |  7.4            0.46
(hanoi 15)      | ?          ?       |  0.68           0.007
(hanoi 20)      | ?          ?       | 31. (9. in gc)  0.2
----------------------------------------------------------------