StudyCaster

In recent years, online crowdsourcing markets like oDesk and Amazon Mechanical Turk have made it easy to recruit human subjects for online user studies, anonymously and in large numbers. A major limitation of online user studies lies in the difficulty of actually observing the actions of the subject as he or she solves the tasks presented, and much work may be necessary on the part of the experimenter in order to instrument the system under test or develop questionnaires that can shed light on the subject's approach to the problem at hand. StudyCaster is a Java-based tool that allows subjects to stream complete recordings of their computer screens to an experimenter's server with a minimum of effort. A novel feature is anonymization, which blurs any window on the screen that does not match predefined criteria.

Tech Note

StudyCaster: A Tool for Automatic and Anonymized Recording of Online User Studies (2.4MB, PDF file)

(Note: The paper above is a little out of date with respect to the latest implementation version of the system. In particular, the current version supports deployment of the client application as a Java Applet, the collection of window system metadata, and the configurable display of user study instructions in multiple wizard-style pages on the client-side.)

Sample Output

The following video shows the experimenter's view of a user study recorded with the StudyCaster client, at 10x the original speed. The StudyCaster client window, which provides the user with instructions and control over the progress of the study, can be seen on the right. The output video is overlaid with additional metadata captured by the StudyCaster client, such as the name and location of windows and focusable components on the screen, the server-synchronized timestamp at which each frame occurred, what part of the study the user is working on, and whether the user is currently providing mouse or keyboard input ("U" in the lower left-hand corner). Also note that the user's web browser window is automatically blurred on the client-side before uploading, for anonymization purposes.

Note: You can click the "HD" button to watch the video in higher quality (on Vimeo).

Feature List

The StudyCaster system is a web-launchable screen recording application with various features that make it suitable for doing online user studies. Features:

• Simple user interface; the user who participates in a user study clicks a link to launch the application, and then follows step-by-step instructions in a popup window. In the end, the user gets a "confirmation code" that s/he can hand back to the experimenter (a common idiom on Amazon Mechanical Turk).
• The system continuously uploads screencast data in a custom codec through firewall-friendly HTTP post requests in small chunks, and properly deals with a wide variety of errors that can happen at various points during user studies which may be up to several hours long.
• Screencasting framerate is throttled to not use more than a certain maximum CPU time fraction. Mouse cursor movements are captured at a high frequency regardless of the screenshot frame rate.
• If running on Windows, the client uses the Win32 API to capture data about windows on the screen, their title, focus etc, and stores this data together with the video data so that it can be used to calculate accurate user study timing data for research purposes. For instance, in a previous study (that of the video above), I calculated the time each user spent looking at the instructions for one particular question, while having a particular PDF open and in focus in a PDF reader, and while subtracting "idle" time spent without using the keyboard or mouse for more than 5 seconds.
• Screencast data is anonymized on the client by blurring screen areas outside of certain windows of interest. This simplifies the Institutional Review Board (IRB) requirements for doing federally funded research studies on human subjects.
• A report generator correlates cookies, IP addresses, and other data to detect multiple submissions that come from potentially the same research subject (it's common for Mechanical Turk workers to sometimes use multiple accounts, which is bad for user studies).
• Transcode to H.264/MP4 files with the various captured metadata overlaid (see the example video above).
• Fully configurable via an XML configuration format.
• Runs on Amazon Elastic Beanstalk or any Java web server.
• Continuous upload of log/console output data for debugging purposes.

Source Code

You can find the source code for StudyCaster on GitHub. Some source files of interest for the technically curious:

• studyconfig.xml: An example configuration file containing two user studies and some other examples.
• ServerContext: The client side of the protocol that is used to upload/stream screencasts from the client.
• NonBlockingOutputStream: A buffer stream that is used to queue screencasting data that still needs to be sent to the server without blocking.
• Win32DesktopLibrary: A class encapsulating the collection of window/OS metadata when the client is run on MS Windows.
• CaptureEncoder/CaptureDecoder, ScreenCastImage: The custom codec written to compress/decompress screen recording video data.
• Quilt: A class representing the information about what rectangular patches on screen need to be anonymized. The algorithm/interface (i.e. Quilt.getPatchRun() ) is designed such that there is practically no performance overhead for the CaptureEncoder.
• CaptureScheduler: Handles the CPU throttling of framerate for screenshot capturing, mouse cursor capturing, and desktop metadata capturing.
• APIServlet: The server side of the protocol.

Build and Server Installation Instructions

(Written down from memory, so beware of documentation bugs...)

1. Install the Java Development Kit for building the server-side application (any recent version should be fine).
2. Install the Java Development Kit 5.0 Update 22 for building the client-side application. This is an old version of the JDK, but StudyCaster needs to be compiled with it to be compatible with as many client machines as possible.
3. Install the NetBeans IDE (download a version that includes Java EE).
4. Clone the studycaster GitHub repository.
5. Open the sc_client and sc_server projects in NetBeans.
6. In NetBeans, configure the sc_server project to use your standard JDK and sc_client to use your installation of JDK 1.5.
7. Configure NetBeans with a Tomcat server and point the sc_server project to use it. NetBeans might already have configured one running locally for you, which is useful for testing.
8. Set up a MySQL or JavaDB server. If you want to run one locally, click the "Services" tab in NetBeans and start the JavaDB one.
9. In sc_server/nbproject/project.properties, set studycaster.jdbc.url to the JDBC URL of your database, including the username and password. For instance, a JavaDB instance started by NetBeans will have the URL jdbc:derby://localhost:1527/sc_devel;create=true.
10. In sc_server/nbproject/project.properties, set studycaster.storage.path to the directory where you want the server to store compressed video data and other file data (the "storage directory"). You can leave it at ../devel_storagedir for testing.
11. If you want GeoIP support (automatic recording of geographic regions associated with IP addresses where clients are run), put an updated GeoLiteCity.dat file in the storage directory (this file can be downloaded from MaxMind).
12. Clean and build the sc_client project.
13. Clean and build the sc_server project.
14. Run the sc_server project. This will deploy the server to the Tomcat container you specified earlier, and open up the StudyCaster admin page.
15. From the StudyCaster admin page, create the database schema by providing the JDBC URL and a new administrator password to be created (that you will use to log in subsequently), and click "Create Schema" (or something like that).
16. Stop and run the sc_server project again. If all is well, you should this time be prompted for a password to log on to the StudyCaster admin page. When you are logged on, the status display should display "OK" for both the database, the storage directory, and the GeoIP data file.
17. Try to run the StudyCaster, make a screencast, and retrieve a confirmation code. Go back to the StudyCaster admin page and look at the "subject report" (or something like that). Your session with the StudyCaster client should be recorded in the report. Also look in the storage directory; there should be a folder for the confirmation code you received from the client.
18. To converted the recorded screencast in the storage directory into a playable MP4 video, run the Converter command-line utility that can be found in the no.ebakke.studycaster.applications package in the sc_client project.
19. To configure what the user sees when opening the client (including custom instructions etc.), edit no/ebakke/studycaster/resources/studyconfig.xml and recompile the sc_client, then sc_server. There is a way to take the studyconfig.xml file directly from the storage directory instead, but I disabled it somewhere in the code for security reasons once I started signing the application (see below).
20. To actually deploy this thing on the Internet, you likely need to sign the client with a code signing certificate (nowadays, no Java plugin allows applets or Java Web Start applications with elevated priveleges like StudyCaster to be run without being signed). Depending on the format of your certificate, you may need to convert it from PKCS12 format to JKS format, as described here. Then you can point the sc_client project to the JKS file (project properties→Application→Web Start→Signing).

(Phew! I never said it was easy for the experimenter...)