Zhiqing Sun

For a more complete list, see the publications page, or my google scholar page.

(*=equal contribution)

2023

1. NeurIPS (Spotlight)

   Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision

   Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan

   In Thirty-seventh Conference on Neural Information Processing Systems, 2023

   Bib HTML Code

   @inproceedings{sun2023principle,
     title = {Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision},
     author = {Sun, Zhiqing and Shen, Yikang and Zhou, Qinhong and Zhang, Hongxin and Chen, Zhenfang and Cox, David and Yang, Yiming and Gan, Chuang},
     booktitle = {Thirty-seventh Conference on Neural Information Processing Systems},
     year = {2023},
     url = {https://openreview.net/forum?id=p40XRfBX96},
   }

2. NeurIPS (Spotlight)

   DIFUSCO: Graph-based Diffusion Solvers for Combinatorial Optimization

   Zhiqing Sun, and Yiming Yang

   In Thirty-seventh Conference on Neural Information Processing Systems, 2023

   Bib HTML Code

   @inproceedings{sun2023difusco,
     title = {{DIFUSCO}: Graph-based Diffusion Solvers for Combinatorial Optimization},
     author = {Sun, Zhiqing and Yang, Yiming},
     booktitle = {Thirty-seventh Conference on Neural Information Processing Systems},
     year = {2023},
     url = {https://openreview.net/forum?id=JV8Ff0lgVV},
   }

3. ICML

   A Neural PDE Solver with Temporal Stencil Modeling

   Zhiqing Sun, Yiming Yang, and Shinjae Yoo

   In Proceedings of the 40th International Conference on Machine Learning, 2023

   Abs Bib HTML PDF Code

   Numerical simulation of non-linear partial differential equations plays a crucial role in modeling physical science and engineering phenomena, such as weather, climate, and aerodynamics. Recent Machine Learning (ML) models trained on low-resolution spatio-temporal signals have shown new promises in capturing important dynamics in high-resolution signals, under the condition that the models can effectively recover the missing details. However, this study shows that significant information is often lost in the low-resolution down-sampled features. To address such issues, we propose a new approach, namely Temporal Stencil Modeling (TSM), which combines the strengths of advanced time-series sequence modeling (with the HiPPO features) and state-of-the-art neural PDE solvers (with learnable stencil modeling). TSM aims to recover the lost information from the PDE trajectories and can be regarded as a temporal generalization of classic finite volume methods such as WENO. Our experimental results show that TSM achieves the new state-of-the-art simulation accuracy for 2-D incompressible Navier-Stokes turbulent flows: it significantly outperforms the previously reported best results by 19.9% in terms of the highly-correlated duration time, and reduces the inference latency into 80%. We also show a strong generalization ability of the proposed method to various out-of-distribution turbulent flow settings, as well as lower resolution or 1-D / 3-D settings. Our code is available at https://github.com/Edward-Sun/TSM-PDE .

   @inproceedings{pmlr-v202-sun23o,
     title = {A Neural {PDE} Solver with Temporal Stencil Modeling},
     author = {Sun, Zhiqing and Yang, Yiming and Yoo, Shinjae},
     booktitle = {Proceedings of the 40th International Conference on Machine Learning},
     pages = {33135--33155},
     year = {2023},
     editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan},
     volume = {202},
     series = {Proceedings of Machine Learning Research},
     publisher = {PMLR},
     url = {https://proceedings.mlr.press/v202/sun23o.html},
   }

4. ICLR

   Recitation-Augmented Language Models

   Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou

   In The Eleventh International Conference on Learning Representations, 2023

   Bib HTML Code

   @inproceedings{sun2023recitation,
     title = {Recitation-Augmented Language Models},
     author = {Sun, Zhiqing and Wang, Xuezhi and Tay, Yi and Yang, Yiming and Zhou, Denny},
     booktitle = {The Eleventh International Conference on Learning Representations},
     year = {2023},
   }

2022

1. NeurIPS

   Dimes: A differentiable meta solver for combinatorial optimization problems

   Ruizhong Qiu*, Zhiqing Sun*, and Yiming Yang

   Advances in Neural Information Processing Systems, 2022

   HTML Code
2. ICLR

   Sparse attention with learning to hash

   Zhiqing Sun, Yiming Yang, and Shinjae Yoo

   In International Conference on Learning Representations, 2022

   Bib HTML Code

   @inproceedings{sun2021sparse,
     title = {Sparse attention with learning to hash},
     author = {Sun, Zhiqing and Yang, Yiming and Yoo, Shinjae},
     booktitle = {International Conference on Learning Representations},
     year = {2022}
   }

2021

1. ICCV

   Rethinking transformer-based set prediction for object detection

   Zhiqing Sun*, Shengcao Cao*, Yiming Yang, and Kris M Kitani

   In Proceedings of the IEEE/CVF international conference on computer vision, 2021

   HTML Code

2020

1. ACL

   MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices

   Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou

   In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020

   Bib HTML Code

   @inproceedings{sun2020mobilebert,
     title = {MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices},
     author = {Sun, Zhiqing and Yu, Hongkun and Song, Xiaodan and Liu, Renjie and Yang, Yiming and Zhou, Denny},
     booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
     pages = {2158--2170},
     year = {2020},
   }

2. ICML

   An EM approach to non-autoregressive conditional sequence generation

   Zhiqing Sun, and Yiming Yang

   In International Conference on Machine Learning, 2020

   Bib HTML Code

   @inproceedings{sun2020approach,
     title = {An EM approach to non-autoregressive conditional sequence generation},
     author = {Sun, Zhiqing and Yang, Yiming},
     booktitle = {International Conference on Machine Learning},
     pages = {9249--9258},
     year = {2020},
     organization = {PMLR}
   }

2019

1. NeurIPS

   Fast structured decoding for sequence models

   Zhiqing Sun*, Zhuohan Li*, Haoqing Wang, Di He, Zi Lin, and Zhihong Deng

   Advances in Neural Information Processing Systems, 2019

   HTML Code
2. ICLR

   RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space

   Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang

   In International Conference on Learning Representations, 2019

   Bib HTML Code

   @inproceedings{sun2018rotate,
     title = {RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space},
     author = {Sun, Zhiqing and Deng, Zhi-Hong and Nie, Jian-Yun and Tang, Jian},
     booktitle = {International Conference on Learning Representations},
     year = {2019},
   }