Randoop 1.2

The Randomized Unit Test Generator for Java

Home

Publications

Download

User Manual

Developer Notes

Index

Running Randoop
Generating JUnit Tests
Interpreting the Results
Randoop commands
- gentests
  Generates unit tests for a set of classes.
- help
  Displays a help message for a given command.

Running Randoop

The easiest way to run Randoop is by adding to your classpath the file randoop.jar, provided with the Randoop distribution. Run Randoop by invoking its main class randoop.main.Main. Randoop's interface is command-based: it expects a specific command as the first argument, followed by command arguments. (Currently, Randoop includes only two commands, help and gentests.) For example, to run the help command, do:

java -classpath randoop.jar randoop.main.Main help

Randoop will print out something like this:

Randoop  is a command-line tool that creates unit tests for Java.
It accepts one of the commands listed below. For the user manual,
please visit http://people.csail.mit.edu/cpacheco/randoop/

Type `help' followed by a command name to see documentation.

Commands:

gentests -- Generates unit tests for a set of classes.
help -- Displays a help message for a given command.

As the above message states, to get help on a specific command, run Randoop with `help command-name' as arguments. For example:

java -classpath randoop.jar randoop.main.Main help gentests

Randoop will output the syntax, description and options related to the gentests command.

Generating JUnit tests

This section shows an example use of Randoop with the goal of generating JUnit test cases. Imagine we want to generate tests for the class java.util.Collections, a utility class that defines several methods for manipulating collections.

The first important thing to keep in mind is that Randoop will only generate tests using the classes you specify. In order to effectively test Collections, You should probably also specify some helper classes, including a class that generates collections. For this example, we will add java.util.TreeSet to the mix.

Invoke Randoop as follows (all in a single line):

java -classpath randoop.jar randoop.main.Main gentests
   --testclass=java.util.TreeSet
   --testclass=java.util.Collections
   --timelimit=10

Alternatively, you can create a file that lists the names of the classes under test, and use the --classlist option:

java -classpath randoop.jar randoop.main.Main gentests
   --classlist=myclasses.txt
   --timelimit=10

Where the contents of myclasses.txt are as follows.

myclasses.txt:

java.util.Collections
java.util.TreeSet

After 10 seconds, Randoop stops generating tests (if you omit the --timelimit option, Randoop's default behavior is to generate tests for 1 minute). The last thing Randoop prints out is the name of the JUnit files containing the tests it generated. You should see a message similar to the following:

Created file: my/home/directory/RandoopTest0.java
Created file: my/home/directory/RandoopTest.java
done.

The main test driver is in class RandoopTest. You can now compile and run the tests. Don't forget to include randoop.jar in the classpath when you compile and run the tests. In this example, you should also add the current directory (".") when you run the tests.

javac -classpath randoop.jar RandoopTest*.java
java -classpath .:randoop.jar junit.textui.TestRunner RandoopTest

JUnit will execute the generated tests. Most will pass, and some will fail. The next section describes the kinds of tests that Randoop generates, and the reason why they may fail.

Interpreting the Results

Randoop generates two kinds of unit tests, regression tests and contract-violating tests.

Regression tests

Most of the tests that Randoop generates will probably pass. They are regression tests: they record the current behavior of the classes under test. Regression tests are useful because they can alert you in the future if you make a change to your code that changes the external behavior of the classes. Here is an example of a regression test for the TreeSet class we generated tests for in the previous section.

// This test passes when executed
public void test10() throws Throwable {

  java.util.TreeSet var0 = new java.util.TreeSet();
  java.lang.Short var1 = new java.lang.Short((short)100);
  boolean var2 = var0.contains(var1);
    
  // Regression assertion (captures the current behavior of the code)
  assertTrue(var2 == false);
}

This test will pass when you run it right after executing it. But notice that it captures an important behavior of the method TreeSet.contains: it returns false if the set is empty. If later, as the developers of this class, we introduced an error that caused contains to return true on an empty set, the test would fail and thus alert us to the error.

Contract-violating tests

Some of the tests that Randoop generated may fail when you first run the tests, indicating that an assertion was violated. These are contract-violating tests. The failure suggests a potential error in one or more classes under test. A contract-violating test shows a specific use of a class under test that leads to a violation of an API contract. A contract is a property that should hold (e.g. an object invariant, or a method postcondition), and a contract violation suggests an error. Currently, Randoop checks that the classes under test exhibit the following properties:

Given a non-null object o, o.equals(o) should always return true, and o.equals(null) should always return false.
The hashCode() and toString() methods should not throw an exception.

Take a look at the generated unit tests. Open the file RandoopTest0.java. Each unit test consists of a snippet of code that use the classes under test, along with assertions stating properties that fail to hold. For example, one of the tests might look as follows (actually, it will probably be longer, but we show a short test for simplicity).

// This test fails when executed
public static void test1() {
  LinkedList var0 = new LinkedList();
  Object var1 = new Object();
  var0.addFirst(var1);
  TreeSet var2 = new TreeSet(var0);
  Set var3 = Collections.synchronizedSet(var2);
  // Checks the contract:  var3.equals(var3)
  Assert.assertTrue(var3.equals(var3) == true);
}

This test shows a scenario that makes it possible to create a set that does not preserve reflexivity of equality, a property specified in the API for java.lang.Object. If you spent some time with the debugger, you would discover that the erroneous behavior arises because the constructor call new TreeSet(var0) fails to throw a ClassCastException, which it should because the element in var0 is not Comparable (see the API for java.util.TreeSet). This is an error in the TreeSet constructor. Later on, somewhere deep inside the call to var3.equals(var3), a ClassCastException is thrown, but it is caught, and the equals method ends up returning true. This tests reveals at least one error, and possibly two, that could be fixed as follows:

The constructor to TreeSet should throw an exception when given a single-element list with a non-comparable element.
The first statement in the equals method in SynchronizedSet should be
```
        if (o == this) return true;
```

In general, all implementations of equals should include this check.

Note that not all failing tests that Randoop generates may reveal errors. Some tests may exhibit behavior that represents normal operation of the classes under test. For example, consider the output of Randoop on the class java.util.Formatter.

Randoop Commands

This section describes Randoop's commands. The information presented here is the same as that which you can get via Randoop's help command.

gentests

Usage: gentests OPTIONS

Where: At least one class is specified via `--testclass' or `--classlist'.

Summary. Attempts to generate JUnit tests that capture the behavior of the classes under test and/or find contract violations. Randoop generates tests using feedback-directed random test generation.

Input: One or more names of classes to test. A class to test can be specified via the `--testclass=' or `--classlist=' options.

Output: A JUnit test suite (as one or more Java source files). The tests in the suite will pass when executed using the classes under test.

Example use:

java randoop.main.Main gentests --testclass=java.util.Collections  --testclass=java.util.TreeSet

Notes.

Randoop executes the code under test, with no mechanisms to protect your system from harm resulting from arbitrary code execution. If random execution of your code could have undesirable effects (e.g. deletion of files, opening network connections, etc.) make sure you execute Randoop in a sandbox machine.
Randoop will only use methods from the classes that you specify for testing. If Randoop is not generating tests for a particular method, make sure that you are including classes for the types that the method requires. Otherwise, Randoop may fail to generate tests due to missing input parameters.
Randoop is designed to be deterministic when the code under test is itself deterministic. This means that two runs of Randoop will generate the same tests. To get variation across runs, use the --randomseed option.

Options

--check-object-contracts=boolean
Default: true
Check java.lang.Object contracts, e.g. equals(Object) is reflexive, hashCode() throws no exceptions, etc.
--testclass=string
Default: []
Specify the fully-qualified name of a class under test.
--classlist=string
Default: no default.
Specify the name of a file that contains a list of classes under test. Eachclass is specified by its fully qualified name on a separate line.
--methodlist=string
Default: no default.
Specify the name of a file that contains a list of methods under test. Eachmethod is specified on a separate line.
--inputlimit=int
Default: 1000000
Used to determine when to stop test generation. Generation stops when either the time limit (--timelimit=int) OR the input limit (--inputlimit=int) is reached. Note that the number of tests output may be smaller than then number of inputs created, because redundant and illegal inputs may be discarded.
--timelimit=int
Default: 100
Used to determine when to stop test generation. Generation stops when either the time limit (--timelimit=int) OR the input limit (--inputlimit=int) is reached.
--testsperfile=int
Default: 500
Maximum number of tests to write to each JUnit file.
--junit-classname=string
Default: RandoopTest
Name of the JUnit file containing Randoop-generated tests.
--junit-package-name=string
Default:
Name of the package that the generated JUnit files should have.
--junit-output-dir=string
Default: no default.
Name of the directory to which JUnit files should be written.
--randomseed=int
Default: 0
The random seed to use in the generation process
--maxsize=int
Default: 100
Do not generate tests with more than statements
--forbid-null=boolean
Default: true
Forbid Randoop to use null as input to methods. IMPORTANT: even if this option is set to true, null is only used if there is no non-null values available.
--null-ratio=double
Default: 0.0
Use null with the given frequency. [TODO explain]
--dont-output-tests=boolean
Default: false
Run Randoop but do not create JUnit tests (used in research experiments).
--log=filewriter
Default: no default.
Log lots of information to a given file name (`filewriter' is the name of a file).

help

Usage: help

Summary. Displays a help message for a given command.

Input: None (for the general help message), or the name of a command (for command-specific help).

Output: A help message is printed to stdout.

Comments, questions, bug reports?

Last modified: July 20 2008 21:06:37.