Robert Wang

Robert Y. Wang

My e-mail
[Scholar] [LinkedIn]

Hi! I support a computer vision and machine perception team at Reality Labs / Oculus / Meta developing and shipping egocentric hand-tracking, body-tacking, object-tracking and human understanding technology for augmented and virtual reality. Prior to that I had co-founded a small company, Nimble VR (acquired by Facebook) that built skeletal hand-tracking software. Drop me a line if you're interested in computer vision for input and interaction.

I received my Ph.D. in EECS while working in CSAIL with Jovan Popović. My research interests are in computer graphics, computer vision and human computer interaction.

I attended Carnegie Mellon University as an undergraduate where I worked with Doug James and Jessica Hodgins.


UmeTrack: Unified multi-view end-to-end hand tracking for VR

hands clasping, hand interaction Shangchen Han, Po-chen Wu, Yubo Zhang, Beibei Liu, Linguang Zhang, Zheng Wang, Weiguang Si, Peizhao Zhang, Yujun Cai, Tomas Hodan, Randi Cabezas, Luan Tran, Muzaffer Akbay, Tsz-Ho Yu, Cem Keskin, Robert Wang
to appear in ACM Transactions on Graphics,
(Proc. SIGGRAPH Asia 2022)

[website / data / code] [paper]

Neural Correspondence Field for Object Pose Estimation

inferring object correspondences from an RGB image Lin Huang, Tomas Hodan, Lingni Ma, Linguang Zhang, Luan Tran, Christopher Twigg, Po-Chen Wu, Junsong Yuan, Cem Keskin, Robert Wang
in European Conference on Computer Vision (ECCV), 2022

[code] [paper]

Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities

Assembly 101 dataset Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, Angela Yao
in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

[website / data / code] [paper]

DeltaCNN: End-to-End CNN Inference of Sparse Frame Differences in Videos

DeltaCNN teaser Mathias Parger, Chengcheng Tang, Christopher D. Twigg, Cem Keskin, Robert Wang, Markus Steinberger
in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022
[website / code] [paper]

EM-POSE: 3D Human Pose Estimation from Sparse Electromagnetic Trackers

understanding occlusion Manuel Kaufmann, Yi Zhao, Chengcheng Tang, Lingling Tao, Christopher Twigg, Jie Song, Robert Wang, Otmar Hilliges
in International Conference on Computer Vision (ICCV), 2021
[website / data / code] [paper]

UNOC: Understanding Occlusion for Embodied Presence in Virtual Reality

understanding occlusion Mathias Parger, Chengcheng Tang, Yuanlu Xu, Christopher D. Twigg, Lingling Tao, Yijing Li, Robert Wang, Markus Steinberger
in IEEE Transactions on Visualization and Computer Graphics (TVCG), 2021
[paper] [data]

Decoding Surface Touch Typing from Hand-Tracking

typing on a flat surface Mark Richardson, Matt Durasoff, Robert Wang
in ACM User Interface Software and Technology (UIST), 2020
[paper and video]

MEgATrack: Monochrome Egocentric Articulated Hand-Tracking for Virtual Reality

monochrome-based hand-tracking Shangchen Han, Beibei Liu, Randi Cabezas, Christopher D. Twigg, Peizhao Zhang, Jeff Petkau, Tsz-Ho Yu, Chun-Jung Tai, Muzaffer Akbay, Zheng Wang, Asaf Nitzan, Gang Dong, Yuting Ye, Lingling Tao, Chengde Wan, Robert Wang
in ACM Transactions on Graphics,
(Proc. SIGGRAPH 2020), 39(4).
[paper] [video]

Lightweight Multi-View 3D Pose Estimation Through Camera-Disentangled Representation

multi-view pose estimation Edoardo Remelli, Shangchen Han, Sina Honari, Pascal Fua, Robert Wang
in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020
[paper] [video]

Going Deeper in Spiking Neural Networks: VGG and Residual Architectures

spiking neural network Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, Kaushik Roy
in Frontiers in Neuroscience, 2019

Online Optical Marker-based Hand Tracking with Deep Labels

marker-based hand-tracking Shangchen Han, Beibei Liu, Robert Wang, Yuting Ye, Christopher Twigg, Kenrick Kin
in ACM Transactions on Graphics,
(Proc. SIGGRAPH 2018), 37(4).
[paper] [main video] [datasets and pre-trained networks]

DodecaPen: Accurate 6DoF Tracking of a Passive Stylus

dodecapen Po-Chen Wu, Robert Wang, Kenrick Kin, Christopher Twigg, Shangchen Han, Ming-Hsuan Yang, Shao-Yi Chien
in ACM User Interface Software and Technology (UIST), 2017
[paper] [main video]

Structural Optimization of 3D Masonry Buildings

structural optimization Emily Whiting, Hijung Shin, Robert Wang, John Ochsendorf and Fr├ędo Durand
in ACM Transactions on Graphics,
(Proc. SIGGRAPH Asia 2012), 31(6).
[paper] [main video]

6D Hands: Markerless Hand Tracking for Computer Aided Design

markerless hand-tracking Robert Y. Wang, Sylvain Paris, and Jovan Popović
in ACM User Interface Software and Technology (UIST), 2011
[paper] [main video] (mp4) [supplementary video] (mp4)

Practical Color-Based Motion Capture

upper body tracking Robert Y. Wang, Sylvain Paris, and Jovan Popović
in ACM/Eurographics Symposium on Computer Animation (SCA), 2011
also CSAIL Technical Reports, MIT-CSAIL-TR-2010-044
[paper] [main video] [supplementary video]

Seafloor Image Compression with Large Tilesize Vector Quantization

underwater stingray Chris Murphy, Robert Y. Wang, Hanumant Singh
in Proceedings of IEEE AUV Conference, 2010

Real-Time Hand-Tracking with a Color Glove

hand-tracking system Robert Y. Wang and Jovan Popović in ACM Transactions on Graphics,
(Proc. SIGGRAPH 2009), 28(3).
[project] [paper] [video] [slides] [fast forward]

Real-Time Hand-Tracking as a User Input Device

many hands Robert Y. Wang
Doctoral Symposium, ACM Symposium on User Interface Software and Technology (UIST), 2008.

Real-Time Enveloping with Rotational Regression

training examples for rotational regression Robert Y. Wang, Kari Pulli, and Jovan Popović in ACM Transactions on Graphics,
(Proc. SIGGRAPH 2007), 26(3).
[paper] [video 1] [video 2] [supplemental work] [slides]

Mesh Ensemble Motion Graphs: Data-driven Mesh Animation with Constraints

irises Doug L. James, Christopher D. Twigg, Andrew A. Cove, and Robert Y. Wang
in ACM Transactions on Graphics, 26(4), 2007.
[paper] [project]

6D Hands: Markerless Hand-Tracking for Computer Aided Design

Hand-tracking proof of concept videos

For more information and videos, check out the project page.

Side Projects

While in grad school, I also built a few websites using Ruby on Rails.

RentMonkey: (requires MIT certificates)

rent monkey screen shot 1 rent monkey screen shot 2 rent monkey screen shot 3

I wrote RentMonkey in my fourth year, while I was serving on the Graduate Student Council. At the time, there wasn't an efficient way to look for off-campus housing in Cambridge / Boston suitable for MIT students.

RentMonkey was designed to track listings exclusively provided by MIT students, for MIT students. The website authenticates each student visiting the site. The listings on the site are suitable to MIT students because they typically have been occupied by the students who posted them. Furthermore, students can indicate the rent they paid at a residence historically, so that future tenants can negotiate with the landlord with confidence.

Since its launch in Spring 2008, RentMonkey has been used by 4,700 unique MIT students, who have posted over 500 listings and information about 1,600 residences. RentMonkey has become a clearing house for rental listings in the MIT community. (December 2009)


rent monkey screen shot 1 rent monkey screen shot 2 rent monkey screen shot 3

In Spring 2009, I worked with Feng Zhang to create a high quality expression cloning tool for molecular biology. We optimized our site for both private collaboration and publishing. Because EveryVector is completely on the web, users do not need to install or upgrade software, or worry about data backups. This model helps biologists focus on biology rather than IT.

We launched EveryVector in September 2009. Today, over 400 users have created more than 6,000 vectors on our site. The site is gaining users steadily, and we hope that EveryVector will become a platform for molecular biology analysis. (December 2009)