Randoop for Java

Randoop for Java is the Java version of Randoop, a feedback-directed random test generator.

Randoop is a command-line program that generates unit tests for Java classes (in JUNit format). The input to Randoop is a set of classes under test. The output of Randoop is a JUnit test suite.

Index

Part 1: Download and build
Part 2: Using Randoop
Part 3: List of Randoop Commands
- genfailures
  Attempts to generate contract-violating unit tests for a set of classes.
- genregressions
  Attempts to generate regression unit tests for a set of classes.
- help
  Displays a help message for a given command.
Part 4: If you get stuck

Part 1: Download and build

Randoop is available as a gzipped tar file. Download the following file: dist.tgz. The file contains the source code for Randoop (open source MIT license).

You can compile Randoop in two ways. One is via the pretty simple Makefile provided, by doing make build. this work on Linux; no Windows support). Another is using Eclipse. The file dist.tgz contains an Eclipse project that you can import and build.

Part 2: Using Randoop

Randoop can generate two kinds of unit tests:

Contract-violating tests are tests that fail (some assertion is violated). The failure suggests an error in one or more classes under test.
Regression tests are tests that pass (no assertions are violated). Regression tests record the behavior of the classes under test, and can be run on a different (for example, a newer) version of the classes to discover regression errors.

Command-line interface

Randoop's command-line interface is self-documenting. To get the top-level help screen, run Randoop via its main class randoop.main.Main, with no arguments, or with help as its sole argument. The classpath should contain two things: (1) the bin directory where Randoop classes are compiled to, and (2) the lib/junit-4.3.1.jar jar (or equivalent). For example:

Under the top-level Randoop directory:
java -classpath bin:lib/junit-4.3.1.jar randoop.main.Main help

As specified in the top-level help screen, to get help on a specific command, run Randoop with `help command-name' as arguments. For example:

java -classpath bin:lib/junit-4.3.1.jar randoop.main.Main help genfailures

Generating contract-violating tests

Contract-violating tests are test that fail when executed on the classes under test. A contract-violating test shows a specific use of a class under test that leads to a violation of an API contract. A contract is a property that should hold (e.g. an object invariant, or a method postcondition), and a contract violation suggests an error. Randoop checks a number of default contracts, such as:

Given a non-null object o, the call o.equals(o) should always return true.
The hashCode() and toString() methods should not throw an exception.
If none of its parameters is null, a method should not throw a NullPointerException.

This section shows an example use of Randoop with the goal of generating contract-violating test cases. Suppose you want to test the class java.util.Collections, a utility class that defines several methods for manipulating collections.

The first important thing to keep in mind is that Randoop will only generate tests using the classes you specify. In order to effectively test Collections, You should probably also specify some helper classes, including a class that generates collections; let's add java.util.TreeSet to the mix.

Invoke Randoop as follows.

java -classpath bin:lib/junit-4.3.1.jar randoop.main.Main genfailures --testclass=java.util.TreeSet --testclass=java.util.Collections --timelimit=10

Alternatively, you can create a file that lists the names of the classes under test, and use the --classlist option:

java -classpath bin:lib/junit-4.3.1.jar randoop.main.Main genfailures --classlist=myclasses.txt --timelimit=10

Where the contents of myclasses.txt are as follows.

myclasses.txt:

java.util.Collections
java.util.TreeSet

After 10 seconds, Randoop stops generating tests (if you omit the --timelimit option, Randoop's default behavior is to generate tests for 60 seconds). The last thing Randoop reports are the name of the JUnit files containing the tests it generated. You should see a message similar to the following:

Created file: my/home/directory/RandoopTest0.java
Created file: my/home/directory/RandoopTest.java
done.

The main test driver is in class RandoopTest. You can now compile and run the tests.

javac -classpath bin:lib/junit-4.3.1.jar RandoopTest*.java
java -classpath .:bin:lib/junit-4.3.1.jar junit.textui.TestRunner RandoopTest

JUnit will report several tests as failures, and the rest as errors.

A test case that reveals an error

Take a look at the generated unit tests. Open the file RandoopTest0.java. Each unit test consists of a snippet of code that use the classes under test, along with assertions stating properties that fail to hold. For example, one of the tests might look as follows (actually, it will probably be longer, but we show a short test for simplicity).

public static void test1() {
  LinkedList var0 = new LinkedList();
  Object var1 = new Object();
  var0.addFirst(var1);
  TreeSet var2 = new TreeSet(var0);
  Set var3 = Collections.synchronizedSet(var2);
  //EqualsNotReflexive
  Assert.assertTrue(var3.equals(var3) == true);
}

This test shows a scenario that makes it possible to create a set that does not preserve reflexivity of equality, a property specified in the API for java.lang.Object. If you spent some time with the debugger, you would discover that the erroneous behavior arises because the constructor call new TreeSet(var0) fails to throw a ClassCastException, which it should because the element in var0 is not Comparable (see the API for java.util.TreeSet). This is an error in the TreeSet constructor. Later on, somewhere deep inside the call to var3.equals(var3), a ClassCastException is thrown, but it is caught, and the equals method ends up returning true. This tests reveals at least one error, and possibly two, that could be fixed as follows:

The constructor to TreeSet should throw an exception when given a single-element list with a non-comparable element.
The first statement in the equals method in SynchronizedSet should be
```
        if (o == this) return true;
```
In general, all implementations of equals should include this check.

A test case that does not reveal an error

Not all tests that Randoop generates may reveal errors. Some tests may exhibit behavior that represents normal operation of the classes under test. For example, consider the output of Randoop on the class java.util.Formatter.

java -classpath bin:lib/junit-4.3.1.jar randoop.main.Main genfailures --testclass=java.util.Formatter --timelimit=10

Randoop generates many test that fail when executed. Here is an example test generated by Randoop:

public void test0() throws Throwable {
  Formatter var0 = new Formatter();
  Locale var1 = var0.locale();
  var0.close();

  //ToStringThrowsException java.util.FormatterClosedException 
  if (var0 != null) {
   var0.toString();
  }
}

JUnit reports an error on this test, because the last call to toString() throws a FormatterClosedException. In other words, this test fails the Randoop built-in contract that stipulates that no call of toString() should result in an exception. However, in this case the exception represents normal behavior of the class, and this behavior is specified in the API for the formatter class.

Future versions of Randoop will try to be smarter and avoid reporting tests like this one, and will let the user of the tool specify methods for which specific contracts do not hold.

Writing your own contracts

You can specify contracts that Randoop should check in addition to its default contracts. Do this via the --checkerclass=classname option, which takes the fully-qualified name of a class.

I will write a more detailed tutorial later, but for now, you can look at examples of a contract-checking class by looking at the classes in src/randoop/contract/unary/. For example, the class src/randoop/contract/unary/EqualsNotReflexive.java implements the contract that checks reflexivity of equality.

Let me know if you have any questions on writing contracts.

NOTE: only unary contracts (those checking the property of a single object) are currently enabled.

Generating regression tests

Contract-violating tests attempt to reveal errors in the current implementation of the classes under test. Regression tests different. They actually pass when executed on the classes under test. A regression test records the behavior of the classes when used in a specific scenario, and can be used to ensure that the behavior is preserved in later versions of the software. For example, to generate regreession tests for TreeSet and Collections, invoke Randoop with the genregressions command:

java -classpath bin:lib/junit-4.3.1.jar randoop.main.Main genregressions --testclass=java.util.TreeSet --testclass=java.util.Collections --timelimit=10

This time, instead of generating JUnit tests that fail when executed, Randoop will generate tests that pass when executed.

Non-reproducible tests. Occasionally, a regression test generated by Randoop will fail instead of pass. This can happen if the test uses global (i.e. static) state of the classes under test, or is nondeterministic. You should remove these kinds of tests from your regression test suite (future versions of the tool will do this automatically).

Part 3: Randoop Commands

This section describes Randoop's commands. The information presented here is the same as that which you can get via Randoop's help command.

genfailures

Usage: genfailures OPTIONS

Where: At least one class is specified via `--testclass' or `--classlist'.

Summary. Attempts to generate JUnit tests that violate an API contract. Randoop generates tests using feedback-directed random test generation. If no tests are generated that violate any contracts, this command will not output any tests (see `genregressions' command if you want to generate unit tests that do not violate contracts).

Input: One or more names of classes to test. A class to test can be specified via the `--testclass=' or `--classlist=' options.

Output: A JUnit test suite (as one or more Java source files). The tests in the suite will fail when executed using the classes under test.

Example use:

java -jar randoop.jar genfailures --testclass=java.util.Collections  --testclass=java.util.TreeSet

Notes.

Randoop executes the code under test, with no mechanisms to protect your system from harm. If random execution of your code could have undesirable effects (e.g. deletion of files, opening network connections, etc.) make sure you execute Randoop in a sandbox machine.
Randoop will only use methods from the classes that you specify for testing. If Randoop is not generating tests for a particular method, make sure that you are including classes for the types that the method requires. Otherwise, Randoop may fail to generate tests due to missing input parameters.
Randoop is designed to be deterministic when the code under test is itself deterministic. This means that two runs of Randoop on deterministic code will generate the same tests. To get variation across runs, use the --randomseed option.

Options

--nochecks=boolean
Default: true
Disable assertions and checkRep methods.
--joetestrun=boolean
Default: false
Perform (expensive) checks that tests Joe (for Joe developers).
--senderr=globals$errorstreamassigner
Default: no default.
Pipe stderr to the given file.
--testclass=string
Default: []
Specify the fully-qualified name of a class under test.
--classlist=string
Default: no default.
Specify the name of a file that contains a list of classes under test. Eachclass is specified by its fully qualified name on a separate line.
--inputlimit=int
Default: 1000000
Used to determine when to stop test generation. Generation stops when either the time limit (--timelimit=int) OR the input limit (--inputlimit=int) is reached. Note that the number of tests output may be smaller than then number of inputs created, because redundant and illegal inputs may be discarded.
--timelimit=int
Default: 100
Used to determine when to stop test generation. Generation stops when either the time limit (--timelimit=int) OR the input limit (--inputlimit=int) is reached.
--testsperfile=int
Default: 500
Maximum number of tests to write to each JUnit file.
--junitclass=string
Default: RandoopTest
Name of the JUnit file containing Randoop-generated tests.
--randomseed=int
Default: 0
The random seed to use in the generation process
--maxsize=int
Default: 100
Do not generate tests with more than statements
--null-allowed=boolean
Default: false
Allow Randoop to use null as input to methods. IMPORTANT: even if this option is set to true, null is only used if there is no non-null values available.
--checkerclass=string
Default: []
Specify the fully-qualified name of a checker class to install.

genregressions

Usage: genregressions OPTIONS

Where: At least one class is specified via `--testclass' or `--classlist'.

Summary. Attempts to generate JUnit tests that capture the behavior of the classes under test. Randoop generates tests using feedback-directed random test generation. By construction, regression tests always pass. If you want Randoop to generate tests that uncover errors, see the `genfailures' command.

Input: One or more names of classes to test. A class to test can be specified via the `--testclass=' or `--classlist=' options.

Output: A JUnit test suite (as one or more Java source files). The tests in the suite will pass when executed using the classes under test.

Example use:

java -jar randoop.jar genregressions --testclass=java.util.Collections  --testclass=java.util.TreeSet

Notes.

Randoop executes the code under test, with no mechanisms to protect your system from harm. If random execution of your code could have undesirable effects (e.g. deletion of files, opening network connections, etc.) make sure you execute Randoop in a sandbox machine.
Randoop will only use methods from the classes that you specify for testing. If Randoop is not generating tests for a particular method, make sure that you are including classes for the types that the method requires. Otherwise, Randoop may fail to generate tests due to missing input parameters.
Randoop is designed to be deterministic when the code under test is itself deterministic. This means that two runs of Randoop will generate the same tests. To get variation across runs, use the --randomseed option.

Options

--testclass=string
Default: []
Specify the fully-qualified name of a class under test.
--classlist=string
Default: no default.
Specify the name of a file that contains a list of classes under test. Eachclass is specified by its fully qualified name on a separate line.
--inputlimit=int
Default: 1000000
Used to determine when to stop test generation. Generation stops when either the time limit (--timelimit=int) OR the input limit (--inputlimit=int) is reached. Note that the number of tests output may be smaller than then number of inputs created, because redundant and illegal inputs may be discarded.
--timelimit=int
Default: 100
Used to determine when to stop test generation. Generation stops when either the time limit (--timelimit=int) OR the input limit (--inputlimit=int) is reached.
--testsperfile=int
Default: 500
Maximum number of tests to write to each JUnit file.
--junitclass=string
Default: RandoopTest
Name of the JUnit file containing Randoop-generated tests.
--randomseed=int
Default: 0
The random seed to use in the generation process
--maxsize=int
Default: 100
Do not generate tests with more than statements
--null-allowed=boolean
Default: false
Allow Randoop to use null as input to methods. IMPORTANT: even if this option is set to true, null is only used if there is no non-null values available.

help

Usage: help

Summary. Displays a help message for a given command.

Input: None (for the general help message), or the name of a command (for command-specific help).

Output: A help message is printed to stdout.

Part 4: If you get stuck...

If you have any questions, comments, or bug reports, don't hesitate to contact Carlos Pacheco via email: