Jun-Yan Zhu; photo credit: Prof. Ren Ng, UC Berkeley

Jun-Yan Zhu

Postdoctoral researcher

Computer Science and Artificial Intelligence Laboratory

Department of EECS

Massachusetts Institute of Technology

Email:


junyanz at mit dot edu


 

Software | Google Scholar | GitHub | Arxiv

Papers | Talks | Awards | Thesis

 

I am a postdoctoral researcher at MIT CSAIL, working with Prof. Antonio Torralba, Prof. William T. Freeman, and Prof. Josh Tenenbaum. I obtained my Ph.D. from UC Berkeley in 2017 after spending five wonderful years at CMU and Berkeley with Prof. Alexei A. Efros. I study computer vision, computer graphics, and machine learning with a goal of building machines capable of understanding and recreating our visual world.


My Ph.D. was supported by a Facebook Fellowship. My dissertation won the 2018 ACM SIGGRAPH Outstanding Doctoral Dissertation Award from SIGGRAPH and 2017-18 David J. Sakrison Memorial Prize for outstanding doctoral research from the UC Berkeley EECS Department.


I received my B.E in Computer Sciences from Tsinghua University in 2012, where I worked with Prof. Zhuowen Tu and Dr. Eric Chang at Microsoft Research Asia and worked with Prof. Shi-Min Hu at Tsinghua's Graphics Group.


News & Events

[Code] PyTorch implementation for CycleGAN and pix2pix (with PyTorch 0.4+).

[Teaching] CycleGAN/pix2pix lecture at Udacity deep learning nanodegree program.

[Service] SIGGRAPH Asia 2018, Technical Papers Committee member.

[Tutorial] CVPR 2018 Tutorial on Generative Adversarial Networks.

[Tutorial] ICCV 2017 Tutorial on Generative Adversarial Networks.

[Workshop] ICML 2017 Workshop on Visualization for Deep Learning.

[Course] SIGGRAPH Asia 2014 invited Course on Data-Driven Visual Computing.

[CatPapers] Cool vision, learning, and graphics papers on Cats.


Publications

Video-to-Video Synthesis

Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, Bryan Catanzaro

In arXiv, 2018

 

Project | Code | Full Paper | arXiv | Youtube | BibTex

CyCADA: Cycle-Consistent Adversarial Domain Adaptation

Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Alexei A. Efros, and Trevor Darrell

In International Conference on Machine Learning (ICML), 2018

 

Paper | Code | BibTex

High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs

Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, Bryan Catanzaro

In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018

See GTC 2018 Keynote.

 

Project | Code | Paper | Youtube | Slides | BibTex

Spatially Transformed Adversarial Examples

Chaowei Xiao*, Jun-Yan Zhu*, Bo Li, Mingyan Liu, and Dawn Song

In International Conference on Learning Representations (ICLR), 2018

 

Paper | BibTex

Generating Adversarial Examples with Adversarial Networks

Chaowei Xiao, Bo Li, Jun-Yan Zhu, Mingyan Liu, and Dawn Song

In International Joint Conference on Artificial Intelligence (IJCAI), 2018

 

Paper | BibTex

Toward Multimodal Image-to-Image Translation

Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A. Efros, Oliver Wang, and Eli Shechtman

In Advances in Neural Information Processing Systems (NIPS), 2017

 

Project | Code | Paper | Youtube | Poster | BibTex

Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks

Jun-Yan Zhu*, Taesung Park*, Phillip Isola, and Alexei A. Efros

In IEEE International Conference on Computer Vision (ICCV), 2017

Mentioned in NY Times article on GANs.

 

Project | PyTorch | Torch | Paper
Spotlight Talk | Slides | BibTex

Image-to-Image Translation with Conditional Adversarial Nets

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros

In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017

See Distill blog and the Economist article | Also see neat uses of #pix2pix on Twitter.

Project | PyTorch | Torch | Paper | Slides | BibTex

 

Real-Time User-Guided Image Colorization with Learned Deep Priors

Richard Zhang*, Jun-Yan Zhu*, Phillip Isola, Xinyang Geng, Angela S. Lin, Tianhe Yu, and Alexei A. Efros

In ACM Transactions on Graphics (SIGGRAPH), 2017

Project | UI Code | PyTorch Training | Youtube | Video
Paper | Slides | Talk | BibTex | Fastforward

Light Field Video Capture Using a Learning-Based Hybrid Imaging System

Ting-Chun Wang, Jun-Yan Zhu, Nima Khademi Kalantari, Alexei A. Efros, and Ravi Ramamoorthi

In ACM Transactions on Graphics (SIGGRAPH), 2017

Project | GitHub | Youtube | Training code
Paper | Talk | Video | Data (18GB) | BibTex

 

Generative Visual Manipulation on the Natural Image Manifold

Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros

In European Conference on Computer Vision (ECCV), 2016

See Distill blog and article in California Magazine

 

Project | YouTube | GitHub | Paper
Slides | Video | BibTex

 

A 4D Light-Field Dataset and CNN Architectures for Material Recognition

Ting-Chun Wang, Jun-Yan Zhu, Ebi Hiroaki, Manmohan Chandraker, Alexei A. Efros, and Ravi Ramamoorthi

In European Conference on Computer Vision (ECCV), 2016

 

Paper | Data (thumbnail) | Full data (15.9G)
Supplement | Poster | BibTex

 

Learning a Discriminative Model for the Perception of Realism in Composite Images

Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros

In IEEE International Conference on Computer Vision (ICCV), 2015

 

Project | Paper | GitHub | Slides | Poster | BibTex

 

Mirror Mirror: Crowdsourcing Better Portraits

Jun-Yan Zhu, Aseem Agarwala, Alexei A. Efros, Eli Shechtman, and Jue Wang

In ACM Transactions on Graphics (SIGGRAPH Asia), 2014

 

Project (code) | Paper | Data | Slides | Supplement | BibTex

AverageExplorer: Interactive Exploration and Alignment of Visual Data Collections

Jun-Yan Zhu, Yong Jae Lee and Alexei A. Efros

In ACM Transactions on Graphics (SIGGRAPH), 2014

 

See article in The New Yorker

Project | YouTube | Paper | Slides | Supplement | BibTex

 

MILCut: A Sweeping Line Multiple Instance Learning Paradigm for Interactive Image Segmentation

Jiajun Wu*, Yibiao Zhao*, Jun-Yan Zhu, Siwei Luo and Zhuowen Tu

In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014

 

Project | Paper | Poster | BibTex

 

Reverse Image Segmentation: A High-Level Solution to a Low-Level Task

Jiajun Wu, Jun-Yan Zhu, and Zhuowen Tu

In British Machine Vision Conference (BMVC), 2014

 

Paper | BibTex

 

Unsupervised Object Class Discovery via Saliency-Guided Multiple Class Learning

Jun-Yan Zhu, Jiajun Wu, Yan Xu, Eric Chang and Zhuowen Tu

In IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2015

(an expanded journal version of our CVPR 2012 paper)

 

Project | Paper | Supplement | Poster | BibTex

 

Multiple Clustered Instance Learning for Histopathology Cancer Image Classification, Segmentation and Clustering

Yan Xu*, Jun-Yan Zhu*, Eric I-Chao Chang and Zhuowen Tu

In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012

(See an expanded journal version at Medical Image Analysis (MIA), 2014

 

Project | GitHub | Paper | BibTex | Poster

 

Motion-Aware Gradient Domain Video Composition

Tao Chen, Jun-Yan Zhu, Ariel Shamir and Shi-Min Hu

In IEEE Transactions on Image Processing (TIP), 2013

 

Paper | YouTube | Video | BibTex


Software

vid2vid: High-resolution (e.g., 2048x1024) photorealistic video-to-video translation.

CYCADA: Pytorch implementation of cycle-consistent adversarial domain adaptation.

pix2pixHD: 2048x1024 image synthesis with conditional GANs.

BicycleGAN: multimodal image-to-image translation.

Interactive Deep Colorization: real-time interface for user-guided colorization.

PyTorch Colorization: PyTorch code for training interactive colorization models.

Light Field Video: light field video applications (e.g. video refocusing, changing aperture and view).

CycleGAN: Torch implementation for learning an image-to-image translation without input-output pairs.

pix2pix: Torch implementation for learning a mapping from input images to output images.

pytorch CycleGAN & pix2pix: PyTorch implementation for both unpaired and paired image-to-image translation.

iGAN: a deep learning software that easily generates images with a few brushstrokes.

RealismCNN: code for predicting and improving visual realism in composite images.

MCILBoost: a boosting-based Multiple Instance Learning (MIL) software.

MirrorMirror: an expression training App that helps users mimic their own expressions.

SelectGoodFace: a program for selecting attractive/serious portraits from a personal photo collection.

FaceDemo: a simple 3D face alignment and warping demo.


Talks

Learning to Generate Images

SIGGRAPH Dissertation Award Talk (2018)

Unpaired Image-to-Image Translation

CVPR Tutorial on GANs (2018)

Learning to Synthesize and Manipulate Natural Photos

MIT CSAIL, HKUST CSE Departmental Seminar, ICCV Tutorial on GANs, O'Reilly AI, AI with the best, Y Conf, DEVIEW, ODSC West (2017)

On Image-to-Image Translation

Stanford, MIT CSAIL, Facebook, CUHK, SNU (2017)

Interactive Deep Colorization

SIGGRAPH, NVIDIA Innovation Theater, Global AI Hackathon (2017)

Visual Manipulation and Synthesis on the Natural Image Manifold

Facebook, MSR, Berkeley BAIR, THU, ICML workshop "Visualization for Deep Learning" (2016)

Mirror Mirror: Crowdsourcing Better Portraits

SIGGRAPH Asia (2014)

What Makes Big Visual Data Hard?

SIGGRAPH Asia invited course "Data-Driven Visual Computing" (2014)

AverageExplorer: Interactive Exploration and Alignment of Visual Data Collections

SIGGRAPH (2014)

Discovering Objects and Harvesting Visual Concepts via Weakly Supervised Learning

Berkeley Visual Computing Lab (2014)


Awards

ACM SIGGRAPH Outstanding Doctoral Dissertation Award (2018)

David J. Sakrison Memorial Prize for Outstanding Doctoral Research, by Berkeley EECS (2018)

NVIDIA Pioneer Research Award (2018)

Facebook Fellowship (2015)

Outstanding Undergraduate Thesis in Tsinghua University (2012)

Excellent Undergraduate Student in Tsinghua University (2012)

National Scholarship, by the Ministry of Education of China (2009 and 2010)

Singapore Technologies Engineering China Scholarship (2010, 2011, and 2012)


MISC

Photo of my cat Aquarius and my dog Arya.