Download: PDF.
“Finding Errors in .NET with Feedback-directed Random Testing”
by Carlos Pacheco, Shuvendu K. Lahiri, and Thomas Ball.
In ISSTA 2008: International Symposium on Software Testing and
Analysis, (Seattle, Washington), July 20-24, 2008.
A previous version appeared as Microsoft Research technical report
MSR-TR-2008-29, (Redmond, WA), Feb. 2008.
We present a case study in which a team of test engineers at Microsoft applied a feedback-directed random testing tool to a critical component of the .NET architecture. Due to its complexity and high reliability requirements, the component had already been tested by 40 test engineers over five years, using manual testing and many automated testing techniques. Nevertheless, the feedback-directed random testing tool found errors in the component that eluded previous testing, and did so two orders of magnitude faster than a typical test engineer (including time spent inspecting the results of the tool). The tool also led the test team to discover errors in other testing and analysis tools, and deficiencies in previous best-practice guidelines for manual testing. Finally, we identify challenges that random testing faces for continued effectiveness, including an observed decrease in the technique's error detection rate over time.
Download: PDF.
BibTeX entry:
@inproceedings{PachecoLB2008,
author = {Carlos Pacheco and Shuvendu K. Lahiri and Thomas Ball},
title = {Finding Errors in .NET with Feedback-directed Random Testing},
booktitle = {ISSTA 2008: International Symposium on Software Testing
and Analysis},
address = {Seattle, Washington},
month = {July~20--24,},
year = {2008}
}