I am a Research Scientist at NVIDIA Seattle Robotics Lab led by Professor Dieter Fox. I obtained my Ph.D. degree in Computer Science at MIT CSAIL in 2022 under the supervision of Professor Wojciech Matusik in the Computational Design and Fabrication Group (CDFG). I obtained my bachelor's degree from Department of Computer Science and Technology at Tsinghua University with honor in 2016. During my undergraduate period, I worked with Professor Shi-Min Hu in the Tsinghua Graphics & Geometric Computing Group and with Professor Stelian Coros as an exchange student at CMU.

My research mainly focuses on the intersection of Robotics, Simulation, and Machine Learning. Specifically, I am interested in the following topics: Robotics Control, Reinforcement Learning, Differentiable Physics-based Simulation, Robotics Control and Design Co-Optimization, and Sim-to-Real.

Jie Xu

I am a Research Scientist at NVIDIA Seattle Robotics Lab led by Professor Dieter Fox. I obtained my Ph.D. degree in Computer Science at MIT CSAIL in 2022 under the supervision of Professor Wojciech Matusik in the Computational Design and Fabrication Group (CDFG). I obtained my bachelor's degree from Department of Computer Science and Technology at Tsinghua University with honor in 2016. During my undergraduate period, I worked with Professor Shi-Min Hu in the Tsinghua Graphics & Geometric Computing Group and with Professor Stelian Coros as an exchange student at CMU.

My research mainly focuses on the intersection of Robotics, Simulation, and Machine Learning. Specifically, I am interested in the following topics: Robotics Control, Reinforcement Learning, Differentiable Physics-based Simulation, Robotics Control and Design Co-Optimization, and Sim-to-Real.

News

01/2025: TacSL is accepted to T-RO.
05/2024: One paper accepted to ICML 2024 and two papers accepted to RSS 2024.
08/2024: One paper accepted to CoRL 2023 (Oral).
01/2024: We release the code for our Tactile Simulation Paper (CoRL 2022).
09/2023: RVT is accepted to CoRL 2023.
10/2022: SHAC is covered by Two Minute Papers.
10/2022: I started as a Research Scientist at Nvidia Seattle Robotics Lab.
09/2022: Our work on differentiable tactile simulation has been accepted by CoRL 2022.
08/2022: I successfully defended my Ph.D. thesis and obtained the Ph.D. degree.
08/2022: Our work on general assembly path planning has been conditionally accepted by SIGGRAPH Asia 2022 (Journal Track).
08/2022: Our work about 3d printing control by reinforcement learning is covered by MIT News.
07/2022: We officially release the code of SHAC, please check it out.
06/2022: Our work about UAV grammar for UAV co-design is accepted by IROS 2022.
05/2022: Our tactile manipulator design pipeline is covered by MIT News.
03/2022: A paper is conditionally accepted by SIGGRAPH 2022 (journal track).
02/2022: DiffCloth is accepted by TOG.
01/2022: Two papers are accepeted by ICRA 2022.
01/2022: Our work on using differentiable simulation for policy learning on complex control tasks is accepted by ICLR 2022.
12/2021: EvoGym is covered by MIT News.
11/2021: Our work on challenging in-hand re-orientation wins Best Paper Award at CoRL 2021.
09/2021: Our work EvoGym, a large-scale co-design benchmark is accepted by NeurIPS 2021.
09/2021: Our work on challenging in-hand manipulation is accepeted as an oral paper (6.5%) by CoRL 2021.
07/2021: DiffHand is covered by MIT News.
06/2021: Join NVIDIA simulation team as a research intern.
05/2021: Our work about differentiable simulation for robot co-design is accepted by RSS 2021.
03/2021: A paper on multi-objective robot topology search is accepted by ICRA 2021.
11/2020: RoboGrammar is covered by MIT News.
+ more

Selected Publications (Show All)

TacSL: A Library for Visuotactile Sensor Simulation and Learning
Iretiayo Akinola*, Jie Xu*, Jan Carius, Dieter Fox, Yashraj Narang
IEEE Transactions on Robotics (T-RO, 2025)
[Project Page] | [Paper]

MeMo: Meaningful, Modular Controllers via Noise Injection
Megan Tjandrasuwita, Jie Xu, Armando Solar-Lezama, Wojciech Matusik
Conference on Neural Information Processing Systems (NeurIPS 2024)
[Paper] | [Code]

Adaptive Horizon Actor-Critic for Policy Learning in Contact-Rich Differentiable Simulation
Ignat Georgiev, Krishnan Srinivasan, Jie Xu, Eric Heiden, Animesh Garg
International Conference on Machine Learning (ICML 2024)
[Project Page] | [Paper] | [Code]

AutoMate: Specialist and Generalist Assembly Policies over Diverse Geometries
Bingjie Tang, Iretiayo Akinola, Jie Xu, Bowen Wen, Ankur Handa, Karl Van Wyk, Dieter Fox, Gaurav S. Sukhatme, Fabio Ramos, Yashraj Narang
Robotics: Science and Systems (RSS 2024)
[Project Page] | [Paper]

RVT-2: Learning Precise Manipulation from Few Examples
Ankit Goyal, Valts Blukis, Jie Xu, Yijie Guo, Yu-Wei Chao, Dieter Fox
Robotics: Science and Systems (RSS 2024)
[Project Page] | [Code]

AdaDemo: Data-Efficient Demonstration Expansion for Generalist Robotic Agent
Tongzhou Mu, Yijie Guo, Jie Xu, Ankit Goyal, Hao Su, Dieter Fox, Animesh Garg
International Symposium of Robotics Research (ISRR 2024)
[Paper]

Signatures meet dynamic programming: Generalizing Bellman equations for trajectory following
Motoya Ohnishi, Iretiayo Akinola, Jie Xu, Ajay Mandlekar, Fabio Ramos
6th Annual Learning for Dynamics & Control Conference (L4DC 2024)
[Paper]

RVT: Robotic View Transformer for 3D Object Manipulation
Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, Dieter Fox
Conference on Robot Learning (CoRL 2023)(Oral: 6.5%)
[Project Page] | [Arxiv] | [Code] | [Video]

Efficient Tactile Simulation with Differentiability for Robotic Manipulation
Jie Xu, Sangwoon Kim, Tao Chen, Alberto Rodriguez, Pulkit Agrawal, Wojciech Matusik, Shinjiro Sueda
Conference on Robot Learning (CoRL 2022)
[Project Page] | [Paper] | [Video] | [Code] | [Poster]

Accelerated Policy Learning with Parallel Differentiable Simulation
Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos, Wojciech Matusik, Animesh Garg, Miles Macklin
International Conference on Learning Representations (ICLR 2022)
[Project Page] | [Paper] | [Video] | [Code] | [Talk] | [Poster] | [Two Minute Papers]

Assemble Them All: Physics-Based Planning for Generalizable Assembly by Disassembly
Yunsheng Tian, Jie Xu, Yichen Li, Jieliang Luo, Shinjiro Sueda, Hui Li, Karl D.D. Willis, Wojciech Matusik
ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2022)
[Project Page] | [Paper] | [Video]

Automatic Co-Design of Aerial Robots Using a Graph Grammar
Allan Zhao, Tao Du, Jie Xu, Josie Hughes, Juan Salazar, Pingchuan Ma, Wei Wang, Daniela Rus, Wojciech Matusik
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2022)
[Paper]

Closed-Loop Control of Direct Ink Writing via Reinforcement Learning
Michal Piovarči*, Michael Foshey*, Jie Xu, Timothy Erps, Vahid Babaei, Piotr Didyk, Szymon Rusinkiewicz, Wojciech Matusik, Bernd Bickel
ACM Transactions on Graphics (Proc. SIGGRAPH 2022)
[Project Page] | [Paper] | [Supp] | [Code] | [MIT News]

An Integrated Design Pipeline for Tactile Sensing Robotic Manipulators
Lara Zlokapa, Yiyue Luo, Jie Xu, Michael Foshey, Kui Wu, Pulkit Agrawal, Wojciech Matusik
IEEE International Conference on Robotics and Automation (ICRA 2022)
[Project Page] | [Code] | [MIT News]

Graph Grammar-Based Automatic Design for Heterogeneous Fleets of Underwater Robots
Allan Zhao, Jie Xu, Juan Salazar, Wei Wang, Pingchuan Ma, Daniela Rus, Wojciech Matusik
IEEE International Conference on Robotics and Automation (ICRA 2022)
[Paper]

DiffCloth: Differentiable Cloth Simulation with Dry Frictional Contact
Yifei Li, Tao Du, Kui Wu, Jie Xu, Wojciech Matusik
ACM Transactions on Graphics 2022
[Project Page] | [Arxiv]

An End-to-End Differentiable Framework for Contact-Aware Robot Design
Jie Xu, Tao Chen, Lara Zlokapa, Michael Foshey, Wojciech Matusik, Shinjiro Sueda, Pulkit Agrawal
Robotics: Science and Systems (RSS 2021)
[Project Page] | [Paper] | [Supp] | [Sim Doc] | [Code] | [Video] | [Poster] | [Talk] | [MIT News]

A System for General In-Hand Object Re-Orientation
Tao Chen, Jie Xu, Pulkit Agrawal
Conference on Robot Learning (CoRL 2021) (Best Paper Award) (Oral: 6.5%)
[Project Page] | [Paper] | [Supp] | [Arxiv] | [Video] | [BibTeX] | [MIT News]

Evolution Gym: A Large-Scale Benchmark for Evolving Soft Robots
Jagdeep Singh Bhatia, Holly Jackson, Yunsheng Tian, Jie Xu, Wojciech Matusik
Conference on Neural Information Processing Systems (NeurIPS 2021)
[Project Page] | [Paper] | [Supp] | [Code] | [Video] | [BibTeX] | [MIT News]

Multi-Objective Graph Heuristic Search for Terrestrial Robot Design
Jie Xu, Andrew Spielberg, Allan Zhao, Daniela Rus, Wojciech Matusik
IEEE International Conference on Robotics and Automation (ICRA 2021)
[Project Page] | [Paper] | [Arxiv] | [Video] | [Talk]

Prediction-Guided Multi-Objective Reinforcement Learning for Continuous Robot Control
Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, Wojciech Matusik
International Conference on Machine Learning (ICML 2020)
[Project Page] | [Paper] | [Supp] | [Code] | [Video] | [Talk]

RoboGrammar: Graph Grammar for Terrain-Optimized Robot Design
Allan Zhao, Jie Xu, Mina Konaković Luković, Josephine Hughes, Andrew Spielberg, Daniela Rus, Wojciech Matusik
ACM Transactions on Graphics 39(6) (Proc. SIGGRAPH Asia 2020)
[Project Page] | [Paper] | [Video] | [Code] | [MIT News] | [Two Minute Papers]

Learning to Fly: Computational Controller Design for Hybrid UAVs with Reinforcement Learning
Jie Xu, Tao Du, Michael Foshey, Beichen Li, Bo Zhu, Adriana Schulz, Wojciech Matusik
ACM Transactions on Graphics 38(4) (Proc. SIGGRAPH 2019)
[Project Page] | [Paper] | [Video] | [Code] | [MIT CSAIL News]

Interactive Design Space Exploration and Optimization for CAD Models
Adriana Schulz, Jie Xu, Bo Zhu, Changxi Zheng, Eitan Grinspun, Wojciech Matusik
ACM Transactions on Graphics 36(4) (Proc. SIGGRAPH 2017)
[Project Page] | [Paper] | [Supplementary] | [Code] | [Video]

View Suggestion for Interactive Segmentation of Indoor Scenes
Sheng Yang, Jie Xu, Kang Chen, Hong-Bo Fu
Computational Visual Media, Vol. 3, Issue 2, pp 131-146, June 2017
[Paper] | [Video] | [Supplementary]

Extracting Sharp Features from RGB-D Images
Yan-Pei Cao, Tao Ju, Jie Xu, Shi-Min Hu
Computer Graphics Forum, Vol. 35, No. 8, pp. 138-174, 2017
[Paper]

Thesis

Towards Computational Design of Shape and Control for Rigid Robots
Thesis Advisor: Wojciech Matusik
Committee: Daniela Rus, Pulkit Agrawal
Ph.D. Thesis, Massachusetts Institute of Technology
[Thesis] | [Video]


Learning to Fly: Computational Controller Design for Hybrid UAVs with Reinforcement Learning
Advisor: Wojciech Matusik
Master Thesis, Massachusetts Institute of Technology


Interactive Point Cloud Segmentation (in Chinese)
Advisors: Song-Hai Zhang, Shi-Min Hu
Undergraduate Thesis, Tsinghua University

Professional Experience

Research Intern at NVIDIA
Mentor: Miles Macklin

June 2021 - Aug 2021

Research Intern at Snap Research
Mentors: Guru Krishnan, Shree Nayar

June 2020 - Aug 2020

Awards & Honors

Best Paper Awards at CoRL 2021

Nov 2021

Gold Medal in the ACM-ICPC World Finals 2015
by Association for Computing Machinery

May 2015

China Computer Federation Outstanding Undergradate Award
by China Computer Federation

Oct 2015

Outstanding Graduate in Tsinghua University
by Tsinghua University

June 2016

Miscellaneous

MIT Table Tennis Team Member

Sept 2018 - Aug 2022

Championship in 2018 MIT Open U1700 Division (USATT rated)
Singles

Oct 2018

Championship in NCTTA Upper New England Division in season 2018-2019
MIT Table Tennis Co-ed Team

Feb 2019

Third place in NCTTA Upper New England Division Men's Singles
Singles

Feb 2022