Zhiqing Sun

孙之清

avatar.jpg

PhD Student

CMU LTI (GHC 5507)

zhiqings [at] cs.cmu.edu

Hey there, welcome!

I am a Ph.D. student in Language Technologies Institute, Carnegie Mellon University. I am fortunately advised by Prof. Yiming Yang. I received my M.S. degree in May 2021 and anticipate completing my Ph.D. studies in 2024. My academic journey began at Peking University, where I received a B.S. in Computer Science, advised by Prof. Zhi-Hong Deng.

Research Interests

My research focuses on neural-symbolic reasoning, which entails synergistically leveraging the strengths of machine learning systems with symbolic systems such as knowledge graphs and combinatorial optimization solvers.

More recently, my research interests have been oriented towards the alignment of Large Language Models (LLMs) and Large Multimodal Models (LMMs), with a special emphasis on improving reliability through scalable oversight (minimal human supervision) such as human-defined principles or factual feedback from real-world interactions.

I have also conducted research in related fields, including efficient sequence generation (non-autoregressive generation and sparse attention), model compression (knowledge distillation), and AI4Science (neural PDE solvers).

Education

  Language Technologies Institute, Carnegie Mellon University
  • Aug. 2019 - Present, Ph.D. in Language Technologies
  School of Electrical Engineering & Computer Science (EECS), Peking University
  • Sept. 2015 - July 2019, B.S. in Computer Science (Summa Cum Laude)

Experience

  MIT-IBM Watson AI Lab
  Google Brain
  Google Brain
  Microsoft Research Asia
  Mila - Quebec Artificial Intelligence Institute & University of Montreal

Selected Publications

For a more complete list, see the publications page, or my google scholar page.

(*=equal contribution)

2023

  1. NeurIPS (Spotlight)
    Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision
    Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan
    In Thirty-seventh Conference on Neural Information Processing Systems, 2023
  2. NeurIPS (Spotlight)
    DIFUSCO: Graph-based Diffusion Solvers for Combinatorial Optimization
    Zhiqing Sun, and Yiming Yang
    In Thirty-seventh Conference on Neural Information Processing Systems, 2023
  3. ICML
    A Neural PDE Solver with Temporal Stencil Modeling
    Zhiqing Sun, Yiming Yang, and Shinjae Yoo
    In Proceedings of the 40th International Conference on Machine Learning, 2023
  4. ICLR
    Recitation-Augmented Language Models
    Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou
    In The Eleventh International Conference on Learning Representations, 2023

2022

  1. NeurIPS
    Dimes: A differentiable meta solver for combinatorial optimization problems
    Ruizhong Qiu*, Zhiqing Sun*, and Yiming Yang
    Advances in Neural Information Processing Systems, 2022
  2. ICLR
    Sparse attention with learning to hash
    Zhiqing Sun, Yiming Yang, and Shinjae Yoo
    In International Conference on Learning Representations, 2022

2021

  1. ICCV
    Rethinking transformer-based set prediction for object detection
    Zhiqing Sun*, Shengcao Cao*, Yiming Yang, and Kris M Kitani
    In Proceedings of the IEEE/CVF international conference on computer vision, 2021

2020

  1. ACL
    MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
    Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou
    In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020
  2. ICML
    An EM approach to non-autoregressive conditional sequence generation
    Zhiqing Sun, and Yiming Yang
    In International Conference on Machine Learning, 2020

2019

  1. NeurIPS
    Fast structured decoding for sequence models
    Zhiqing Sun*, Zhuohan Li*, Haoqing Wang, Di He, Zi Lin, and Zhihong Deng
    Advances in Neural Information Processing Systems, 2019
  2. ICLR
    RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space
    Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang
    In International Conference on Learning Representations, 2019