Jun-Yan Zhu's portrait

Jun-Yan Zhu

Assistant Professor

School of Computer Science

Carnegie Mellon University

Email: junyanz at cs dot cmu dot edu


CV | Google Scholar | GitHub | Arxiv

Papers | Talks | Events | Teaching


I am an Assistant Professor with The Robotics Institute in the School of Computer Science of Carnegie Mellon University. I also hold affiliated faculty appointments in the Computer Science Department and Machine Learning Department. I study computer vision, computer graphics, machine learning, and computational photography, with the goal of building intelligent machines, capable of recreating our visual world.

Prior to joining CMU, I was a Research Scientist at Adobe Research. I did a postdoc at MIT CSAIL, working with William T. Freeman, Josh Tenenbaum, and Antonio Torralba. I obtained my Ph.D. from UC Berkeley, under the supervision of Alexei A. Efros. I received my B.E. from Tsinghua University, working with Zhuowen Tu, Shi-Min Hu, and Eric Chang.

Software & News

[Code] Try "pip install clean-fid". An FID calculation repo with proper image resizing and quantization steps.

[Code] Interactively editing a Conditional NeRF. Changing the color and shape of 3D regions with sparse scribbles.

[Code] GAN Ensembling: use generative models for test-time data augmentation.

[Code] Swapping Autoencoder for various image editing tasks: texture swapping, local editing, and latent code vector arithmetic.

[Code] Don't forget to use data augmentation for your GANs training. The code supports StyleGAN2-PyTorch/TF and BigGAN-PyTorch.

[Code] Anycost GAN can accelerate StyleGAN2 inference by 6-12x on diverse hardware. Try it on your laptop.

[Code] Contrastive Learning for unpaired image-to-image translation. Faster and lighter training compared to CycleGAN.

[Code] for model rewriting. You can now interactively edit the network weights, instead of training with a data set.

[Code] PyTorch and Tensorflow implementation of the Hessian Penalty, a regularization term that encourages disentanglement in neural networks.

[Code]: Projecting and transforming a real image into class-conditional GAN (BigGAN).

[Code] for compressing pix2pix, CycleGAN, and GauGAN by 9-21x.

[Code] PyTorch implementation and Google Colab for CycleGAN and pix2pix.

[Software] Photoshop introduces "Neural Filters", which are partly built on our work iGAN (ECCV 2016), ideepcolor (SIGGRAPH 2017), and CycleGAN (ICCV 2017).

[Demo] SPADE/GauGAN demo for creating photorealistic images from user sketches.

[Tutorial] CVPR 2020 Tutorial on Neural Rendering

[Tutorial] Eurographics 2020 STAR on Neural Rendering

[Workshop] ICCV 2019 Workshop on Image and Video Synthesis.

[CatPapers] Cool vision, learning, and graphics papers on Cats.


· George Cazenavette (MSR)

· Kangle Deng (PhD, joint with Deva Ramanan)

· Nupur Kumari (MSR)

· Gaurav Parmar (MSR)

· Sheng-Yu Wang (PhD)


16-824: Visual Learning and Recognition (Fall 2021): If you are on the waiting list, please contact me via email.

16-726: Learning-based Image Synthesis (Spring 2021)

Deep Learning at Udacity (Co-instructor).


On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation

Gaurav Parmar, Richard Zhang, Jun-Yan Zhu

arXiv 2104.11222, 2021

Installation: pip install clean-fid


Code | Project | Paper | BibTex


Editing Conditional Radiance Fields

Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu, Bryan Russell

arXiv 2105.06466, 2021


Code | Project | Paper | Colab | BibTex


Ensembling with Deep Generative Views

Lucy Chai, Jun-Yan Zhu, Eli Shechtman, Phillip Isola, Richard Zhang

CVPR 2021


Code | Project | Paper | Colab | BibTex


Anycost GANs for Interactive Image Synthesis and Editing

Ji Lin, Richard Zhang, Frieder Ganz, Song Han, Jun-Yan Zhu

CVPR 2021


Code | Project | Paper | Video | BibTex


Understanding the Role of Individual Units in a Deep Neural Network

David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, Antonio Torralba

PNAS 2020


Project | Code | Paper | Arxiv | BibTex
CNN Dissection Colab | GAN Dissection Colab


Swapping Autoencoder for Deep Image Manipulation

Taesung Park, Jun-Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei A. Efros, and Richard Zhang

NeurIPS 2020


Project | Code | Video | Paper | BibTex


Differentiable Augmentation for Data-Efficient GAN Training

Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, Song Han

NeurIPS 2020


Project | Code | Paper | BibTex


Contrastive Learning for Unpaired Image-to-Image Translation

Taesung Park, Alexei A. Efros, and Richard Zhang, Jun-Yan Zhu

ECCV 2020


Project | Code | Video
Paper | Talk | BibTex


Rewriting a Deep Generative Model

David Bau, Steven Liu, Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba

ECCV 2020


Project | Code | Paper | Video
Colab | Talk | BibTex


The Hessian Penalty: A Weak Prior for Unsupervised Disentanglement

William Peebles, John Peebles, Jun-Yan Zhu, Alexei A. Efros, Antonio Torralba

ECCV 2020


Project | Code | Paper | Video | Talk | BibTex


Transforming and Projecting Images into Class-conditional Generative Networks

Minyoung Huh, Richard Zhang, Jun-Yan Zhu, Sylvain Paris, Aaron Hertzmann

ECCV 2020


Project | Code | Paper | BibTex


GAN Compression: Efficient Architectures for Interactive Conditional GANs

Muyang Li, Ji Lin, Yaoyao Ding, Zhijian Liu, Jun-Yan Zhu, Song Han

CVPR 2020


Code | Project | Paper | Video
Demo Video | Slides | BibTex


Diverse Image Generation via Self-Conditioned GANs

Steven Liu, Tongzhou Wang, David Bau, Jun-Yan Zhu, Antonio Torralba

CVPR 2020


Project | Code | Paper | BibTex


State of the Art on Neural Rendering

Ayush Tewari*, Ohad Fried*, Justus Thies*, Vincent Sitzmann*, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, Rohit Pandey, Sean Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Shechtman, Dan B Goldman, Michael Zollhöfer

Eurographics 2020 (STAR Report)


Paper | Project | BibTex
CVPR Tutorial | Eurographics Tutorial


Seeing what a GAN Cannot Generate

David Bau, Jun-Yan Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, Antonio Torralba

ICCV 2019


Project | Code | Colab | Paper | BibTex


Semantic Photo Manipulation with a Generative Image Prior

David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, Antonio Torralba



Project | Demo | Paper | Video | BibTex


Learning the Signatures of the Human Grasp Using a Scalable Tactile Glove

Subramanian Sundaram, Petr Kellnhofer, Yunzhu Li, Jun-Yan Zhu, Antonio Torralba, and Wojciech Matusik

Nature, 569 (7758), 2019


See the Economist article and BBC Radio

Project | Paper | Code | BibTex


Connecting Touch and Vision via Cross-Modal Prediction

Yunzhu Li, Jun-Yan Zhu, Russ Tedrake, Antonio Torralba

CVPR 2019

See CNN News


Project | Code | Paper | BibTex


Semantic Image Synthesis with Spatially-Adaptive Normalization

Taesung Park, Ming-Yu Liu, Ting-Chun Wang, Jun-Yan Zhu

CVPR 2019 (Best Paper Finalist)

SIGGRAPH 2019 Real-time Live Demo "GauGAN" (with Chris Hebert and Gavriil Klimov)

Won "Best in Show Award" and "Audience Choice Award" in SIGGRAPH 2019 Real-time Live.


Project | Real-time Live | Code | Paper
Youtube | GTC 2019 Demo | BibTex


GAN Dissection: Visualizing and Understanding Generative Adversarial Networks

David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, Antonio Torralba

ICLR 2019


Project | Paper | Demo | Code
Video | Slides | BibTex


Propagation Networks for Model-Based Control Under Partial Observation

Yunzhu Li, Jiajun Wu, Jun-Yan Zhu, Joshua B. Tenenbaum, Antonio Torralba, Russ Tedrake

ICRA 2019


Project | Paper | Video | BibTex


Dataset Distillation

Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, Alexei A. Efros

arXiv 2018


Project | Paper | Code | BibTex


Visual Object Networks: Image Generation with Disentangled 3D Representation

Jun-Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Joshua B. Tenenbaum, William T. Freeman

NeurIPS 2018


Project | Paper | Code | BibTex


3D-Aware Scene Manipulation via Inverse Graphics

Shunyu Yao*, Tzu-Ming Harry Hsu*, Jun-Yan Zhu, Jiajun Wu, Antonio Torralba, William T. Freeman, Joshua B. Tenenbaum

NeurIPS 2018


Project | Paper | Code | BibTex


Video-to-Video Synthesis

Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, Bryan Catanzaro

NeurIPS 2018

See our driving game demo.


Project | Code | Full Paper | arXiv | Youtube | BibTex


CyCADA: Cycle-Consistent Adversarial Domain Adaptation

Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Alexei A. Efros, and Trevor Darrell

ICML 2018


Paper | Code | BibTex


High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs

Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, Bryan Catanzaro

CVPR 2018

Featured in GTC 2018 Keynote.


Project | Code | Paper | Youtube | Slides | BibTex


Spatially Transformed Adversarial Examples

Chaowei Xiao*, Jun-Yan Zhu*, Bo Li, Mingyan Liu, and Dawn Song

ICLR 2018


Paper | BibTex


Generating Adversarial Examples with Adversarial Networks

Chaowei Xiao, Bo Li, Jun-Yan Zhu, Mingyan Liu, and Dawn Song

IJCAI 2018


Paper | BibTex


Toward Multimodal Image-to-Image Translation

Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A. Efros, Oliver Wang, and Eli Shechtman

NeurIPS 2017


Project | Code | Paper | Youtube | Poster | BibTex


Learning to Synthesize and Manipulate Natural Images

December, 2017

ACM SIGGRAPH Outstanding Doctoral Dissertation Award.

David J. Sakrison Memorial Prize for outstanding doctoral research, by the UC Berkeley EECS Dept.


Thesis | Talk | News | Cover

Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks

Jun-Yan Zhu*, Taesung Park*, Phillip Isola, and Alexei A. Efros

ICCV 2017


Project | PyTorch | Torch | Paper
Spotlight Talk | Slides | BibTex


Image-to-Image Translation with Conditional Adversarial Nets

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros

CVPR 2017

See Distill blog | Also see neat uses of #pix2pix on Twitter.

Project | PyTorch | Torch | Paper | Slides | BibTex


Real-Time User-Guided Image Colorization with Learned Deep Priors

Richard Zhang*, Jun-Yan Zhu*, Phillip Isola, Xinyang Geng, Angela S. Lin, Tianhe Yu, and Alexei A. Efros


Photoshop Element 2020 ColorizePhoto is based on our work

Project | UI Code | PyTorch Training | Youtube | Video
Paper | Slides | Talk | BibTex | Fastforward


Light Field Video Capture Using a Learning-Based Hybrid Imaging System

Ting-Chun Wang, Jun-Yan Zhu, Nima Khademi Kalantari, Alexei A. Efros, and Ravi Ramamoorthi


Project | GitHub | Youtube | Training code
Paper | Talk | Video | Data (18GB) | BibTex


Generative Visual Manipulation on the Natural Image Manifold

Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros

ECCV 2016

See Distill blog and article in California Magazine


Project | YouTube | GitHub | Paper
Slides | Video | BibTex


A 4D Light-Field Dataset and CNN Architectures for Material Recognition

Ting-Chun Wang, Jun-Yan Zhu, Ebi Hiroaki, Manmohan Chandraker, Alexei A. Efros, and Ravi Ramamoorthi

ECCV 2016


Paper | Data (thumbnail) | Full data (15.9G)
Supplement | Poster | BibTex


Learning a Discriminative Model for the Perception of Realism in Composite Images

Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros

ICCV 2015


Project | Paper | GitHub | Slides | Poster | BibTex


Mirror Mirror: Crowdsourcing Better Portraits

Jun-Yan Zhu, Aseem Agarwala, Alexei A. Efros, Eli Shechtman, and Jue Wang

SIGGRAPH Asia 2014


Project (code) | Paper | Data | Slides | Supplement | BibTex


AverageExplorer: Interactive Exploration and Alignment of Visual Data Collections

Jun-Yan Zhu, Yong Jae Lee and Alexei A. Efros



See article in The New Yorker

Project | YouTube | Paper | Slides | Supplement | BibTex


MILCut: A Sweeping Line Multiple Instance Learning Paradigm for Interactive Image Segmentation

Jiajun Wu*, Yibiao Zhao*, Jun-Yan Zhu, Siwei Luo and Zhuowen Tu

CVPR 2014


Project | Paper | Poster | BibTex


Reverse Image Segmentation: A High-Level Solution to a Low-Level Task

Jiajun Wu, Jun-Yan Zhu, and Zhuowen Tu

BMVC 2014


Paper | BibTex


Unsupervised Object Class Discovery via Saliency-Guided Multiple Class Learning

Jun-Yan Zhu, Jiajun Wu, Yan Xu, Eric Chang and Zhuowen Tu

TPAMI 2015 | CVPR 2012


Project | Paper | Supplement | Poster | BibTex


Multiple Clustered Instance Learning for Histopathology Cancer Image Classification, Segmentation and Clustering

Yan Xu*, Jun-Yan Zhu*, Eric I-Chao Chang and Zhuowen Tu

CVPR 2012 | Medical Image Analysis 2014


Project | GitHub | Paper | BibTex | Poster


Motion-Aware Gradient Domain Video Composition

Tao Chen, Jun-Yan Zhu, Ariel Shamir and Shi-Min Hu

TIP 2013


Paper | YouTube | Video | BibTex


Learning to Generate Images

SIGGRAPH Dissertation Award Talk (2018)

Unpaired Image-to-Image Translation

CVPR Tutorial on GANs (2018)

Learning to Synthesize and Manipulate Natural Photos

MIT, HKUST CSE Departmental Seminar, ICCV Tutorial on GANs, O'Reilly AI, AI with the best, Y Conf, DEVIEW, ODSC West (2017)

On Image-to-Image Translation

Stanford, MIT, Facebook, CUHK, SNU (2017)

Interactive Deep Colorization

SIGGRAPH, NVIDIA Innovation Theater, Global AI Hackathon (2017)

Visual Manipulation and Synthesis on the Natural Image Manifold

Facebook, MSR, Berkeley BAIR, THU, ICML workshop "Visualization for Deep Learning" (2016)

Mirror Mirror: Crowdsourcing Better Portraits

SIGGRAPH Asia (2014)

What Makes Big Visual Data Hard?

SIGGRAPH Asia invited course "Data-Driven Visual Computing" (2014)

AverageExplorer: Interactive Exploration and Alignment of Visual Data Collections



[Tutorial] CVPR 2020 Tutorial on Neural Rendering

[Tutorial] Eurographics 2020 STAR on Neural Rendering

[Journal] IJCV Special Issue on Generative Adversarial Networks for Computer Vision (2019-2020)

[Workshop] ICCV 2019 Workshop on Image and Video Synthesis.

[Tutorial] CVPR 2019 Tutorial on Map Synchronization.

[Tutorial] CVPR 2018 Tutorial on Generative Adversarial Networks.

[Tutorial] ICCV 2017 Tutorial on Generative Adversarial Networks.

[Workshop] ICML 2017 Workshop on Visualization for Deep Learning.

[Course] SIGGRAPH Asia 2014 invited Course on Data-Driven Visual Computing.


· My cat Aquarius and my dog Arya's photo and its Ukiyo-e style.

· You can eat Greek yogurt when wearing a mask. See how Arya does it.