Research Projects

Bare Hand Continuous Gesture Recognition for Natural Multimodal Interaction

I continue to develop intelligent gesture input interfaces for my PhD research. However, instead of using a coloered glove, I'm using a Kinect for tracking bare hands. I am developing a unified framework for real-time continuous gesture recognition that handles path and pose gestures seamlessly, and responds to discrete and continuous flow gestures promptly and appropriately. The framework is based on hidden Markov models (HMMs). Using this framework, I am creating a browser-based gesture-controlled presentation interface.

[platform: Linux, Windows 7] [programming languages: Java, MATLAB, Ruby, CoffeeScript, C#] [source code]

Intelligent Multimodal Interface for Natural Interaction

This is my Master thesis project. I developed a new multimodal tabletop interface for natural interaction, based on a new technology capable of real-time tracking of 3D hand postures. We focus on natural gestures, i.e., those encountered in spontaneous interaction, rather than a set of artificial gestures designed for the convenience of recognition. We will also use an off-the-shelf speech recognition engine for keyword spotting, then combine speech and deictic gestures.

Poster presentation at the ACM International Conference on Interactive Tabletops and Surfaces 2009 [video] [abstract]

[platform: Windows XP] [programming languages: Java, MATLAB, C++] [code metrics]

DIVAs - Gestural Control of DIgital Ventriloquized Actors

This is a project I worked on when I was an undergraduate research assistant at the Media and Graphics Interdisciplinary Centre (MAGIC) at UBC in the summer of 2008. It is about using hand gestures to produce synthetic vocal expression. I developed software modules for streaming data from Cyberglove and Polhemus tracker efficiently and robustly. I also helped with developing the user interface for the gesture training session and the vocal performance session. The final system is a real-time, compact and wearable system such that a person can wear and walk around to make an artistic performance. Here is a video of Marguerite Witvoet, a vocal artist, performing using the system at Ritsumeikan University in Japan in January 2009.

[video] [platform: Mac OS X] [programming languages: C, Max/MSP]

Personal Projects

Lango

Lango is an Android application I developed with Lei Zhang for the MIT IAP 6.570 Mobile App Competition in 2013, and we got 4th place. It is a foreign language learning tool for the MIT Japanese courses. It allows students to play and record audio exercises conveniently. We are currently working with the Japanese language teachers to further improve the app.

[platform: Android] [programming languages: Java] [source code]

ProtoM

ProtoM is a web application I developed with Jeremy Scott and Andrew Correa for the MIT IAP 6.470 Web Programming Competition in 2012. We got Honorable Mention for the most advanced functionality. We made a online machine learning tool for people to learn and apply practical machine learning more easily. It provides interactive courses where user can select data sets and see the results of different machine learning methods visually. We used WEKA in the backend to provide different machine learning implementations.

[framework: Ruby on Rails] [programming languages: Ruby, Javascript] [source code]

PwnTime

PwnTime is a time and productivity management web application based on the well known Pomodoro Technique. It consists a Chrome extension for creating tasks and starting and stopping a timer, and a web interface for viewing personal productivity analytics.

[framework: Ruby on Rails] [programming languages: Ruby, Javascript]

Clutter - a Chrome Extension

Clutter is a Chrome extension that allows users to view multiple websites in the same tab. It is especially useful for ChromeOS notebook where you can only have one window, but want to view several websites at the same time. In general, it is very useful for viewing and comparing websites side by side when preparing for a trip, doing online shopping or making a personal dashboard. I was involved in developing the interface using HTML5 and CSS3. Our team won the top prize in the Chrome hackathon.

Check out the screencast I made!

Nextlang.com

Nextlang is an language exchange website that combines social network experience, flashcards, and instant messaging to make practicing a foreign language more enjoyable and more efficient. The instant messaging text editor has a novel feature that hightlights the words that are unknown to conversation partner as one types. In this way, users can choose words at the right level for the learners. We originally built the website in 2 days as an entry for the Rails Rumble 2010 competition. All the expert judges gave us 4 or 5 out of 5 points in innovation and usefulness. I conceived the idea, and worked on both the frond-end and back-end of the website.

[framework: Ruby on Rails] [programming languages: Ruby, Javascript]

Course Projects

SwingSpotter

SwingSpotter is a Kinect based golf instruction system that my team developed for the MIT 15.390 New Enterprises course. My team members include Nathaniel Stewart, Eunsuk Kang and Vinicius Ferriani. Jess Struebing, director of golf instruction at MIT was our mentor. We use computer vision algorithms to detect the player's swing plane and gives personalized feedback for how to improve golf swings.

[platform: Windows 7] [programming language: C#]

Scorched Earth 3D

This is a forth-year course project for the Computer Graphics course EECE478 at UBC. Our team won the overall best game award. The game is a 3D verison of the classic 2D Scortched Earth. In the game, two players on a randomly generated terrain take turns in launching attacks against one another until only one remains. The team members are Neema Teymory, Andrew Wong, Lane Feltis and me. The program is based on OpenGL and SDL libraries.

[video] [platform: Windows XP] [programming language: C++]

Multi-Server File System

This is a sequence of individual lab assignments for the graduate course 6.824 Distributed Systems at MIT. We built a multi-server file system called Yet-Another File System (YFS) in the spirit of Frangipani, using replicated state machine for fault tolerance and Paxos consensus algorithm for agreeing on master election. The last part of the lab is a project which we can add whatever extensions to the file system. I added read/write locking and peer-to-peer data transfer to improve the concurrency of the YFS.

[report] [platform: Linux] [programming language: C++]

Gnutella Crawler

This is a course project for EECE 411 Design of Distributed Software Applications at UBC. We built a Gnutella network crawler with 50+ crawler workers deployed on PlanetLab nodes to collect information about the nodes participating in the network and the files shared. We used a master/worker approach with non-blocking I/O on the worker side. We also used dynamic work distribution to maximize performance and promote scalability.

[report] [platform: Windows XP, Linux] [programming language: Java, Python]

iFit - Personal Fitness Advisor

This is a project we did for course EECE 375 (Computational Instrumentation Design Laboratory), a project-based course at UBC. We built a set of accelerometer and Bluetooth based devices including a digital caliper for measuring body fat, an activity monitor for measuring daily calories burnt, and a software package for keeping track of the data.

[report] [platform: AVR microcontroller, Windows XP] [programming language: C, Java]