Robert Wang

Robert Y. Wang

My e-mail
[Scholar] [LinkedIn]

Hi! I support a computer vision and machine perception team at Facebook Reality Labs / Oculus developing and shipping egocentric hand-tracking, body-tacking and and human understanding technology for augmented reality and virtual reality. Prior to that I had co-founded a small company, Nimble VR (acquired by Facebook) that built skeletal hand-tracking software. Drop me a line if you're interested in computer vision for the future of interaction.

I received my Ph.D. in EECS while working in CSAIL with Jovan Popović. My research interests are in computer graphics, computer vision and human computer interaction.

I attended Carnegie Mellon University as an undergraduate where I worked with Doug James and Jessica Hodgins.

6D Hands: Markerless Hand-Tracking for Computer Aided Design

Hand-tracking proof of concept videos

For more information and videos, check out the project page.


Decoding Surface Touch Typing from Hand-Tracking

marker-based hand-tracking Mark Richardson, Matt Durasoff, Robert Wang
to appear in ACM User Interface Software and Technology (UIST), 2020
[paper and video]

MEgATrack: Monochrome Egocentric Articulated Hand-Tracking for Virtual Reality

marker-based hand-tracking Shangchen Han, Beibei Liu, Randi Cabezas, Christopher D. Twigg, Peizhao Zhang, Jeff Petkau, Tsz-Ho Yu, Chun-Jung Tai, Muzaffer Akbay, Zheng Wang, Asaf Nitzan, Gang Dong, Yuting Ye, Lingling Tao, Chengde Wan, Robert Wang
in ACM Transactions on Graphics,
(Proc. SIGGRAPH 2020), 39(4).
[paper] [video]

Lightweight Multi-View 3D Pose Estimation Through Camera-Disentangled Representation

marker-based hand-tracking Edoardo Remelli, Shangchen Han, Sina Honari, Pascal Fua, Robert Wang
in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020
[paper] [video]

Going Deeper in Spiking Neural Networks: VGG and Residual Architectures

spiking neural network Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, Kaushik Roy
in Frontiers in Neuroscience, 2019

Online Optical Marker-based Hand Tracking with Deep Labels

marker-based hand-tracking Shangchen Han, Beibei Liu, Robert Wang, Yuting Ye, Christopher Twigg, Kenrick Kin
in ACM Transactions on Graphics,
(Proc. SIGGRAPH 2018), 37(4).
[paper] [main video] [datasets and pre-trained networks]

DodecaPen: Accurate 6DoF Tracking of a Passive Stylus

dodecapen Po-Chen Wu, Robert Wang, Kenrick Kin, Christopher Twigg, Shangchen Han, Ming-Hsuan Yang, Shao-Yi Chien
in ACM User Interface Software and Technology (UIST), 2017
[paper] [main video]

Structural Optimization of 3D Masonry Buildings

structural optimization Emily Whiting, Hijung Shin, Robert Wang, John Ochsendorf and Fr├ędo Durand
in ACM Transactions on Graphics,
(Proc. SIGGRAPH Asia 2012), 31(6).
[paper] [main video]

6D Hands: Markerless Hand Tracking for Computer Aided Design

markerless hand-tracking Robert Y. Wang, Sylvain Paris, and Jovan Popović
in ACM User Interface Software and Technology (UIST), 2011
[paper] [main video] (mp4) [supplementary video] (mp4)

Practical Color-Based Motion Capture

upper body tracking Robert Y. Wang, Sylvain Paris, and Jovan Popović
in ACM/Eurographics Symposium on Computer Animation (SCA), 2011
also CSAIL Technical Reports, MIT-CSAIL-TR-2010-044
[paper] [main video] [supplementary video]

Seafloor Image Compression with Large Tilesize Vector Quantization

underwater stingray Chris Murphy, Robert Y. Wang, Hanumant Singh
in Proceedings of IEEE AUV Conference, 2010

Real-Time Hand-Tracking with a Color Glove

hand-tracking system Robert Y. Wang and Jovan Popović in ACM Transactions on Graphics,
(Proc. SIGGRAPH 2009), 28(3).
[project] [paper] [video] [slides] [fast forward]

Real-Time Hand-Tracking as a User Input Device

many hands Robert Y. Wang
Doctoral Symposium, ACM Symposium on User Interface Software and Technology (UIST), 2008.

Real-Time Enveloping with Rotational Regression

training examples for rotational regression Robert Y. Wang, Kari Pulli, and Jovan Popović in ACM Transactions on Graphics,
(Proc. SIGGRAPH 2007), 26(3).
[paper] [video 1] [video 2] [supplemental work] [slides]

Mesh Ensemble Motion Graphs: Data-driven Mesh Animation with Constraints

irises Doug L. James, Christopher D. Twigg, Andrew A. Cove, and Robert Y. Wang
in ACM Transactions on Graphics, 26(4), 2007.
[paper] [project]

Side Projects

While in grad school, I also built a few websites using Ruby on Rails.

RentMonkey: (requires MIT certificates)

rent monkey screen shot 1 rent monkey screen shot 2 rent monkey screen shot 3

I wrote RentMonkey in my fourth year, while I was serving on the Graduate Student Council. At the time, there wasn't an efficient way to look for off-campus housing in Cambridge / Boston suitable for MIT students.

RentMonkey was designed to track listings exclusively provided by MIT students, for MIT students. The website authenticates each student visiting the site. The listings on the site are suitable to MIT students because they typically have been occupied by the students who posted them. Furthermore, students can indicate the rent they paid at a residence historically, so that future tenants can negotiate with the landlord with confidence.

Since its launch in Spring 2008, RentMonkey has been used by 4,700 unique MIT students, who have posted over 500 listings and information about 1,600 residences. RentMonkey has become a clearing house for rental listings in the MIT community. (December 2009)


rent monkey screen shot 1 rent monkey screen shot 2 rent monkey screen shot 3

In Spring 2009, I worked with Feng Zhang to create a high quality expression cloning tool for molecular biology. We optimized our site for both private collaboration and publishing. Because EveryVector is completely on the web, users do not need to install or upgrade software, or worry about data backups. This model helps biologists focus on biology rather than IT.

We launched EveryVector in September 2009. Today, over 400 users have created more than 6,000 vectors on our site. The site is gaining users steadily, and we hope that EveryVector will become a platform for molecular biology analysis. (December 2009)