Projects
Deep Shot
A user task often spans multiple heterogeneous devices, e.g., working on a PC in the office and continuing the work on a laptop or a mobile phone while commuting on a shuttle. However, there is a lack of support for users to easily migrate their tasks across devices. To address this problem, we created Deep Shot, a framework for capturing the user’s work state that is needed for a task (e.g., the specific part of a webpage being viewed) and resuming it on a different device. In particular, Deep Shot supports two novel and intuitive interaction techniques, deep shooting and deep posting, for pulling and pushing work states, respectively, using a mobile phone camera.
Related publication: Tsung-Hsiang Chang, and Yang Li. "Deep Shot: A Framework for Migrating Tasks Across Devices Using Mobile Phone Cameras." CHI 2011.
Sikuli Test
The tasks of testing a GUI’s visual behavior typically requires human testers to interact with the GUI and to observe whether the expected results of interaction are presented. Sikuli Test is a new approach to GUI testing using computer vision for testers to automate their tasks. Testers can write a visual test script that uses images to specify which GUI components to interact with and what visual feedback to be observed. Testers can also generate visual test scripts by demonstration. By recording both input events and screen images, it is possible to extract the images of components interacted with and the visual feedback seen by the demonstrator, and generate a visual test script automatically.
Related publication: Tsung-Hsiang Chang, Tom Yeh, and Robert C. Miller. "GUI Testing Using Computer Vision." CHI 2010.
Sikuli Script
Sikuli Script is a new way to
automate GUI interactions using screenshot patterns
to direct mouse and keyboard events.
You can programmatically control a web page, a desktop application running on Windows/Linux/Mac OS X,
or even an iphone application running in an emulator
as long as you can see them on the screen.
Introduction Video
Sikuli Script is avaiable for downloading at http://sikuli.org. You also can get more information about how Sikuli Script works and how to use it in your everyday life there.
Related publication: Tom Yeh, Tsung-Hsiang Chang, and Robert C. Miller. "Sikuli: Using GUI Screenshots for Search and Automation." UIST 2009, pp. 183-192.
Sikuli Search
Sikuli Search allows users to take a screenshot of a GUI
element (such as a toolbar button, icon, or dialog box)
and query a document database using the screenshot instead
of textual descriptions.
Introduction Video
Related publication: Tom Yeh, Tsung-Hsiang Chang, and Robert C. Miller. "Sikuli: Using GUI Screenshots for Search and Automation." UIST 2009, pp. 183-192.