test-manager/ - An automatic unit-testing framework for MIT Scheme or Guile
(load "test-manager/load.scm")
(in-test-group
simple-stuff
(define-test (arithmetic)
"Checking that set! and arithmetic work"
(define foo 5)
(assert-= 5 foo "Foo should start as five.")
(set! foo 6)
(assert-= 36 (* foo foo)))
; Each of these will become a separate anonymous one-form test
(define-each-test
(assert-= 4 (+ 2 2) "Two and two should make four.")
(assert-= 6 (+ 2 2 2))
(assert-= 2147483648 (+ 2147483647 1) "Addition shouldn't overflow.")
(assert-equal '(1 2 3) (cons 1 '(2 3))))
(run-registered-tests)
This test framework defines a language for specifying test suites and a simple set of commands for running them. A test suite is a collection of individual tests grouped into a hierarchy of test groups. The test group hierarchy serves to semantically aggregate the tests, allowing the definition of shared code for set up, tear down, and surround, and also partition the test namespace to avoid collisions.
The individual tests are ordinary procedures, with some associated bookkeeping. A test is considered to pass if it returns normally, and to fail if it raises some condition that it does not handle (tests escaping into continuations leads to unspecified behavior). The framework provides a library of assertions that can be invoked in tests and have the desired behavior of raising an appropriate condition if they fail.
All tests are grouped into a hierarchy of test groups. At any point in the definition of a test suite, there is an implicit ``current test group'', into which tests and subgroups can be added. By default, the current test group is the top-level test group, which is the root of the test group hierarchy.
Define a test named name that consists of the given expressions,
and add it to the current test group. When the test is run, the
expressions will be executed in order, just like the body of any
procedure. If the test raises any condition that it does not handle,
it is considered to have failed. If it returns normally, it is
considered to have passed. Usually, tests will contain assertions
from the list below, which raise appropriate conditions when they
fail. In the spirit of Lisp docstrings, if the first expression in
the test body is a literal string, that string will be included in the
failure report if the test should fail.
This is the most verbose and most expressive test definition syntax. The three test definition syntaxes provided below are increasingly terse syntactic sugar for common usage patterns of this syntax.
Define an explicitly anonymous test. I can't see why you would want to do this, but it is provided for completeness.
Define a one-expression anonymous test. The single expression will be
printed in the failure report if the test fails. This is a special
case of define-each-test, below.
Define a one-expression anonymous test for each of the given expressions. If any of the tests fail, the corresponding expression will be printed in that test's failure report.
If you have many simple independent assertions you need to make and you don't want to invent names for each individual one, this is the test definition syntax for you.
Locate (or create) a test subgroup called name in the current test
group. Then temporarily make this subgroup the current test group,
and execute the expressions in the body of in-test-group. This
groups any tests and further subgroups defined by those expressions
into this test group. Test groups can nest arbitrarily deep. Test
groups serve to disambiguate the names of tests, and to group them
semantically. In particular, should a test fail, the names of the
stack of groups it's in will be displayed along with the test name
itself.
Defines a sequence of expressions to be run before every test in the current test group. Clobbers any previously defined set up for this group.
Defines a sequence of expressions to be run after every test in the current test group. Clobbers any previously defined tear down for this group.
Defines a sequence of expressions to be run surrounding every test in
the current test group. Inside the define-surround, the identifier
run-test is bound to a nullary procedure that actually runs the
test. Clobbers any previously defined surround for this group.
Defines a sequence of expressions to be run once before running any test in the current test group. Clobbers any previously defined group set up for this group.
Defines a sequence of expressions to be run once after running all tests in the current test group. Clobbers any previously defined group tear down for this group.
Defines a sequence of expressions to be run once surrounding running
the tests in the current test group. Inside the
define-group-surround, the identifier run-test is bound to a
nullary procedure that actually runs the tests in this group.
Clobbers any previously defined group surround for this group.
The following procedures are provided for running test suites:
Looks up the test indicated by name-stack in the current test group,
runs it, and prints a report of the results. Returns the number of
tests that did not pass. An empty list for a name stack indicates the
whole group, a singleton list indicates that immediate descendant, a
two-element list indicates a descendant of a descendant, etc. For
example, (run-test '(simple-stuff harder)) would run the second
test defined in the example at the top of this page.
Runs all tests registered so far, and prints a report of the results.
Returns the number of tests that did not pass. This could have been
defined as (run-test '()).
The following assertions are provided for writing tests. The
message arguments to the assertions are user-specified messages to
print to the output if the given assertion fails. The assert-proc
assertion requires a message argument because it cannot construct a
useful output without one, and because it is not really meant for
extensive direct use. The message is optional for the other
assertions because they can say something at least mildly informative
even without a user-supplied message. In any case, the message can be
either a string or a promise (as created by delay) to produce a
string. The latter is useful for assertions with dynamically computed
messages, because that computation will then only be performed if the
test actually fails.
Passes iff the given procedure, invoked with no arguments, returns a
true value. On failure, arranges for the given message to appear in
the failure report. This is a primitive assertion in whose terms
other assertions are defined.
Passes iff the given value is a true value (to wit, not #f).
Passes iff the given value is a false value (to wit, #f).
Passes iff the given actual value is equivalent, according to the
corresponding predicate, to the expected value. Produces a
reasonably helpful message on failure, and includes the optional
message argument in it if present. When in doubt, use
assert-equal to compare most things; use assert-= to compare
exact numbers like integers; and use assert-in-delta, below, for
inexact numbers like floating points.
Are aliases for assert-equal and assert-=, respectively.
This is intended as a tool for building custom assertions. Returns an
assertion procedure that compares an expected and an actual value with
the given predicate and produces a reasonable failure message.
assert-equal and company could have been defined in terms of
assert-equivalent as, for example, (define assert-equal
(assert-equivalent equal? "equal?")).
Passes iff the given regular expression matches the given string.
Passes iff the given regular expression does not match the given string.
Passes iff the given actual value differs, in absolute value, from
the given expected value by no more than delta. Use this in
preference to assert-= to check sameness of inexact numerical
values.
This unit testing framework is a work in progress. The assertion library is quite impoverished, the test groups do not support as much shared set up code among their tests as I would like, and the language for explicit test group handling is ill-specified and undocumented (peruse test-group.scm if interested). Suggestions are welcome.
Alexey Radul, axch@mit.edu