Replicating the RIFL Developer Study

This page contains materials for replicating the developer study presented in our paper, Robust Programs with Filtered Iterators. The complete artifact is available online.

Experimental Design

Section 4 of the paper presents the design, analysis, and results of the study. Here we present the instrumentations that we developed for conducting the experiments.

• Starter files
  • Control: Tutorial Task1 Task2 Task3
  • Inspect: Tutorial Task1 Task2 Task3
  • Sample inputs (same for both groups): Tutorial Task1 Task2 Task3
• Virtual machines with RIFL implementations (download links)
  • Control: VM
  • Inspect: VM
• Reading materials
  • Consent form
  • Control: Manual Instructions Solutions
  • Inspect: Manual1 Manual2 Instructions Solutions

Raw Data

Here we present the raw data that we collected from the experiments.

• Program files
  • Control: c1 c2 c3 c4 c5
  • Inspect: i1 i2 i3 i4 i5
• Screencasts for both groups (download link): screencasts
• Booklets
  • Control: c1 c2 c3 c4 c5
  • Inspect: i1 i2 i3 i4 i5

Analysis

As discussed in the paper, we analyze the thumbnail programs in files named prog3.stu. Section IV of the paper presents how to measure defects and program complexity from the source code. Here we present test inputs and program outputs.

• Test inputs (same for both groups)
• Program outputs
  • Control: c1 c2 c3 c4 c5
  • Inspect: i1 i2 i3 i4 i5