Jun-Yan Zhu; photo credit: Prof. Ren Ng, UC Berkeley

Jun-Yan Zhu

Postdoctoral researcher

Computer Science and Artificial Intelligence Laboratory

Department of EECS

Massachusetts Institute of Technology

Email: junyanz at mit dot edu


CV | Google Scholar | GitHub | Thesis | Teaching

Software | Papers | Talks | Events | Awards | Arxiv


I am a postdoctoral researcher at MIT, working with William T. Freeman, Josh Tenenbaum, and Antonio Torralba. I obtained my Ph.D. from UC Berkeley after spending five wonderful years at CMU and UC Berkeley with Alexei A. Efros. I received my B.E from Tsinghua University. I study computer vision, computer graphics, and machine learning with the goal of building intelligent machines, capable of recreating our visual world.


[Online Demo] SPADE/GauGAN demo for creating photorealistic images from user sketches.

[Code] PyTorch implementation for CycleGAN and pix2pix (with PyTorch 0.4+).

[CatPapers] Cool vision, learning, and graphics papers on Cats.


Learning to Synthesize and Manipulate Natural Images

December, 2017

ACM SIGGRAPH Outstanding Doctoral Dissertation Award.

David J. Sakrison Memorial Prize for outstanding doctoral research, by the UC Berkeley EECS Dept.


Thesis | Talk | News | Cover


Learning the Signatures of the Human Grasp Using a Scalable Tactile Glove

Subramanian Sundaram, Petr Kellnhofer, Yunzhu Li, Jun-Yan Zhu, Antonio Torralba, and Wojciech Matusik

Nature, 569 (7758), 2019


See the Economist article and BBC Radio

Project | Paper | Code | BibTex

Connecting Touch and Vision via Cross-Modal Prediction

Yunzhu Li, Jun-Yan Zhu, Russ Tedrake, Antonio Torralba

CVPR 2019

See CNN News


Project | Code | Paper | BibTex

Semantic Image Synthesis with Spatially-Adaptive Normalization

Taesung Park, Ming-Yu Liu, Ting-Chun Wang, Jun-Yan Zhu

CVPR 2019


Project | Code | Paper | Youtube
GTC 2019 Demo | BibTex

GAN Dissection: Visualizing and Understanding Generative Adversarial Networks

David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, Antonio Torralba

ICLR 2019


Project | Paper | Demo | Code
Video | Slides | BibTex

Propagation Networks for Model-Based Control Under Partial Observation

Yunzhu Li, Jiajun Wu, Jun-Yan Zhu, Joshua B. Tenenbaum, Antonio Torralba, Russ Tedrake

ICRA 2019


Project | Paper | Video | BibTex

Dataset Distillation

Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, Alexei A. Efros

arXiv 2018


Project | Paper | Code | BibTex

Visual Object Networks: Image Generation with Disentangled 3D Representation

Jun-Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Joshua B. Tenenbaum, William T. Freeman

NeurIPS 2018


Project | Paper | Code | BibTex

3D-Aware Scene Manipulation via Inverse Graphics

Shunyu Yao*, Tzu-Ming Harry Hsu*, Jun-Yan Zhu, Jiajun Wu, Antonio Torralba, William T. Freeman, Joshua B. Tenenbaum

NeurIPS 2018


Project | Paper | Code | BibTex

Video-to-Video Synthesis

Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, Bryan Catanzaro

NeurIPS 2018

See our driving game demo.


Project | Code | Full Paper | arXiv | Youtube | BibTex

CyCADA: Cycle-Consistent Adversarial Domain Adaptation

Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Alexei A. Efros, and Trevor Darrell

ICML 2018


Paper | Code | BibTex

High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs

Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, Bryan Catanzaro

CVPR 2018

Featured in GTC 2018 Keynote.


Project | Code | Paper | Youtube | Slides | BibTex

Spatially Transformed Adversarial Examples

Chaowei Xiao*, Jun-Yan Zhu*, Bo Li, Mingyan Liu, and Dawn Song

ICLR 2018


Paper | BibTex

Generating Adversarial Examples with Adversarial Networks

Chaowei Xiao, Bo Li, Jun-Yan Zhu, Mingyan Liu, and Dawn Song

IJCAI 2018


Paper | BibTex

Toward Multimodal Image-to-Image Translation

Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A. Efros, Oliver Wang, and Eli Shechtman

NIPS 2017

Mentioned in the New Yorker article on image generation.


Project | Code | Paper | Youtube | Poster | BibTex

Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks

Jun-Yan Zhu*, Taesung Park*, Phillip Isola, and Alexei A. Efros

ICCV 2017

Mentioned in the NY Times article on GANs.


Project | PyTorch | Torch | Paper
Spotlight Talk | Slides | BibTex

Image-to-Image Translation with Conditional Adversarial Nets

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros

CVPR 2017

See Distill blog and the Economist article | Also see neat uses of #pix2pix on Twitter.

Project | PyTorch | Torch | Paper | Slides | BibTex


Real-Time User-Guided Image Colorization with Learned Deep Priors

Richard Zhang*, Jun-Yan Zhu*, Phillip Isola, Xinyang Geng, Angela S. Lin, Tianhe Yu, and Alexei A. Efros


Project | UI Code | PyTorch Training | Youtube | Video
Paper | Slides | Talk | BibTex | Fastforward

Light Field Video Capture Using a Learning-Based Hybrid Imaging System

Ting-Chun Wang, Jun-Yan Zhu, Nima Khademi Kalantari, Alexei A. Efros, and Ravi Ramamoorthi


Project | GitHub | Youtube | Training code
Paper | Talk | Video | Data (18GB) | BibTex


Generative Visual Manipulation on the Natural Image Manifold

Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros

ECCV 2016

See Distill blog and article in California Magazine


Project | YouTube | GitHub | Paper
Slides | Video | BibTex


A 4D Light-Field Dataset and CNN Architectures for Material Recognition

Ting-Chun Wang, Jun-Yan Zhu, Ebi Hiroaki, Manmohan Chandraker, Alexei A. Efros, and Ravi Ramamoorthi

ECCV 2016


Paper | Data (thumbnail) | Full data (15.9G)
Supplement | Poster | BibTex


Learning a Discriminative Model for the Perception of Realism in Composite Images

Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros

ICCV 2015


Project | Paper | GitHub | Slides | Poster | BibTex


Mirror Mirror: Crowdsourcing Better Portraits

Jun-Yan Zhu, Aseem Agarwala, Alexei A. Efros, Eli Shechtman, and Jue Wang

SIGGRAPH Asia 2014


Project (code) | Paper | Data | Slides | Supplement | BibTex

AverageExplorer: Interactive Exploration and Alignment of Visual Data Collections

Jun-Yan Zhu, Yong Jae Lee and Alexei A. Efros



See article in The New Yorker

Project | YouTube | Paper | Slides | Supplement | BibTex


MILCut: A Sweeping Line Multiple Instance Learning Paradigm for Interactive Image Segmentation

Jiajun Wu*, Yibiao Zhao*, Jun-Yan Zhu, Siwei Luo and Zhuowen Tu

CVPR 2014


Project | Paper | Poster | BibTex


Reverse Image Segmentation: A High-Level Solution to a Low-Level Task

Jiajun Wu, Jun-Yan Zhu, and Zhuowen Tu

BMVC 2014


Paper | BibTex


Unsupervised Object Class Discovery via Saliency-Guided Multiple Class Learning

Jun-Yan Zhu, Jiajun Wu, Yan Xu, Eric Chang and Zhuowen Tu

TPAMI 2015 | CVPR 2012


Project | Paper | Supplement | Poster | BibTex


Multiple Clustered Instance Learning for Histopathology Cancer Image Classification, Segmentation and Clustering

Yan Xu*, Jun-Yan Zhu*, Eric I-Chao Chang and Zhuowen Tu

CVPR 2012 | Medical Image Analysis 2014


Project | GitHub | Paper | BibTex | Poster


Motion-Aware Gradient Domain Video Composition

Tao Chen, Jun-Yan Zhu, Ariel Shamir and Shi-Min Hu

TIP 2013


Paper | YouTube | Video | BibTex


VON: Code for synthesizing textured 3D objects.

gandissect: Pytorch-based tools for visualizing and understanding the neurons of a GAN.

vid2vid: High-resolution (e.g., 2048x1024) photorealistic video-to-video translation.

CYCADA: Pytorch implementation of cycle-consistent adversarial domain adaptation.

pix2pixHD: 2048x1024 image synthesis with conditional GANs.

BicycleGAN: multimodal image-to-image translation.

Interactive Deep Colorization: real-time interface for user-guided colorization.

PyTorch Colorization: PyTorch code for training interactive colorization models.

Light Field Video: light field video applications (e.g. video refocusing, changing aperture and view).

CycleGAN: Torch implementation for learning an image-to-image translation without input-output pairs.

pix2pix: Torch implementation for learning a mapping from input images to output images.

pytorch CycleGAN & pix2pix: PyTorch implementation for both unpaired and paired image-to-image translation.

iGAN: a deep learning software that easily generates images with a few brushstrokes.

RealismCNN: code for predicting and improving visual realism in composite images.

MCILBoost: a boosting-based Multiple Instance Learning (MIL) software.

MirrorMirror: an expression training App that helps users mimic their own expressions.

SelectGoodFace: a program for selecting attractive/serious portraits from a personal photo collection.

FaceDemo: a simple 3D face alignment and warping demo.


Learning to Generate Images

SIGGRAPH Dissertation Award Talk (2018)

Unpaired Image-to-Image Translation

CVPR Tutorial on GANs (2018)

Learning to Synthesize and Manipulate Natural Photos

MIT, HKUST CSE Departmental Seminar, ICCV Tutorial on GANs, O'Reilly AI, AI with the best, Y Conf, DEVIEW, ODSC West (2017)

On Image-to-Image Translation

Stanford, MIT, Facebook, CUHK, SNU (2017)

Interactive Deep Colorization

SIGGRAPH, NVIDIA Innovation Theater, Global AI Hackathon (2017)

Visual Manipulation and Synthesis on the Natural Image Manifold

Facebook, MSR, Berkeley BAIR, THU, ICML workshop "Visualization for Deep Learning" (2016)

Mirror Mirror: Crowdsourcing Better Portraits

SIGGRAPH Asia (2014)

What Makes Big Visual Data Hard?

SIGGRAPH Asia invited course "Data-Driven Visual Computing" (2014)

AverageExplorer: Interactive Exploration and Alignment of Visual Data Collections



[Tutorial] CVPR 2019 Tutorial on Map Synchronization.

[Tutorial] CVPR 2018 Tutorial on Generative Adversarial Networks.

[Tutorial] ICCV 2017 Tutorial on Generative Adversarial Networks.

[Workshop] ICML 2017 Workshop on Visualization for Deep Learning.

[Course] SIGGRAPH Asia 2014 invited Course on Data-Driven Visual Computing.


Co-instructor, Deep Learning at Udacity.

Guest Lecturer, Advances in Computer Vision (6.819/6.869) at MIT.

Teaching Assistant, Image Manipulation and Computational Photography (CS 194-26) at UC Berkeley.


ACM SIGGRAPH Outstanding Doctoral Dissertation Award (2018)

David J. Sakrison Memorial Prize for Outstanding Doctoral Research, by Berkeley EECS (2018)

NVIDIA Pioneer Research Award (2018)

Facebook Fellowship (2015)

Outstanding Undergraduate Thesis in Tsinghua University (2012)

Excellent Undergraduate Student in Tsinghua University (2012)

National Scholarship, by the Ministry of Education of China (2009 and 2010)

Singapore Technologies Engineering China Scholarship (2010, 2011, and 2012)


Photo of my cat Aquarius and my dog Arya.