Jielin Qiu
jielinq [at] cs (dot) cmu (dot) edu

I am a PhD student in the Computer Science Department at School of Computer Science, Carnegie Mellon University. I also work closely with Prof. Bo Li and Prof. Douglas Weber. My research interests lie in Multimodal Machine Learning, I am interested in designing scalable inference and learning algorithms to connect language, perception, and actions for robust multimodal interaction. My current research lies in the foundations of multimodal learning with applications in multimedia, computer vision, natural language processing, cognition, and healthcare. My research is generously supported by DARPA, NSF, Adobe, Allegheny Health Network, and Cleveland Clinic.

Before coming to CMU, I received my B.Eng. from Shanghai Jiao Tong University, advised by Prof. Bao-Liang Lu.

Email  /  Google Scholar  /  LinkedIn

profile photo
Update
  • If you're an undergraduate or master student interested in RA/summer internship, please feel free to contact me. Our lab is looking for several students for research projects.
News
  • [2023-06] One paper accepted by ICML 2023 Workshop on Interactive Learning with Implicit Human Feedback (spotlight).
  • [2023-06] Two paper accepted by ICML 2023 Workshop on Machine Learning for Multimodal Healthcare Data.
  • [2023-05] Start a research internship at Meta.
  • [2023-05] One paper accepted by ACL Findings 2023.
  • [2023-04] One paper accepted by ICML 2023.
  • [2023-04] Invited talk at Microsoft Research Cambridge.
  • [2023-02] One paper accepted by CVPR 2023.
  • [2023-02] One paper accepted by ICASSP 2023.
  • [2023-01] Start a research internship at Microsoft.
  • [2023-01] One paper accepted by EACL Findings 2023.
  • [2023-01] One paper accepted by AISTATS 2023.
  • [2022-10] One paper accepted by WACV 2023.
  • [2022-10] One paper accepted by NeurIPS 2022 Workshop on Distribution Shifts.
  • [2022-10] Top Reviewers in NeurIPS 2022.
  • [2022-06] One paper accepted by MLHC 2022.
  • [2022-05] Start a research internship at AWS AI.
  • [2022-05] One paper accepted by ICML 2022 workshop on Principles of Distribution Shift.
  • [2022-04] One paper accepted by ICLR 2022 Workshop on Socially Responsible Machine Learning.
  • [2021-09] Receive a gift funding from Adobe. Thanks, Adobe!
  • [2021-05] Start a research internship at Adobe research.
Preprints

* marked as equal contribution

Are Multimodal Models Robust to Image and Text Perturbations?
Jielin Qiu, Yi Zhu, Xingjian Shi, Florian Wenzel, Zhiqiang Tang, Ding Zhao, Bo Li, Mu Li
Under Review / arXiv / Project Webpage / code

MultiSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos
Jielin Qiu, Jiacheng Zhu, William Han, Aditesh Kumar, Karthik Mittal, Claire Jin, Zhengyuan Yang, Linjie Li, Jianfeng Wang,
Bo Li, Ding Zhao, Lijuan Wang
Under Review / arXiv / Project Webpage / Dataset / code

Converting ECG Signals to Images for Efficient Image-text Retrieval via Encoding
Jielin Qiu*, Jiacheng Zhu*, Shiqi Liu, William Han, Jingqi Zhang, Chaojing Duan, Michael Rosenberg, Emerson Liu,
Douglas Weber, Ding Zhao
Under Review / arXiv

Selected Publications

* marked as equal contribution

Embodied Executable Policy Learning with Language-based Scene Summarization
Jielin Qiu, Mengdi Xu*, William Han*, Seungwhan Moon, Ding Zhao
ICML 2023 Workshop on Interactive Learning with Implicit Human Feedback (spotlight) / arXiv

Multimodal Representation Learning of Cardiovascular Magnetic Resonance Imaging
Jielin Qiu*, Peide Huang*, Makiya Nakashima, Jaehyun Lee, Jiacheng Zhu, Wilson Tang, Pohao Chen, Christopher Nguyen, Byung-Hak Kim, Debbie Kwon, Douglas Weber, Ding Zhao, David Chen
ICML 2023 Workshop on Machine Learning for Multimodal Healthcare Data / arXiv

An Empirical Exploration of Cross-domain Alignment between Language and Electroencephalogram
William Han*, Jielin Qiu*, Jiacheng Zhu, Mengdi Xu, Douglas Weber, Bo Li, Ding Zhao
ICML 2023 Workshop on Machine Learning for Multimodal Healthcare Data / arXiv / code

Semantics-Consistent Cross-domain Summarization via Optimal Transport Alignment
Jielin Qiu, Jiacheng Zhu, Mengdi Xu, Franck Dernoncourt, Trung Bui, Zhaowen Wang, Bo Li, Ding Zhao, Hailin Jin
ACL 2023 Findings / arXiv / Press

Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models?
Jielin Qiu*, William Han*, Jiacheng Zhu, Mengdi Xu, Michael Rosenberg, Emerson Liu, Douglas Weber, Ding Zhao
EACL 2023 Findings / Paper / arXiv / code

Cardiac Disease Diagnosis on Imbalanced Electrocardiography Data Through Optimal Transport Augmentation
Jielin Qiu*, Jiacheng Zhu*, Mengdi Xu, Peide Huang, Michael Rosenberg, Douglas Weber, Emerson Liu, Ding Zhao
ICASSP 2023 / Paper / arXiv

Interpolation for Robust Learning: Data Augmentation on Geodesics
Jiacheng Zhu, Jielin Qiu, Aritra Guha, Zhuolin Yang, XuanLong Nguyen, Bo Li, Ding Zhao
ICML 2023 / arXiv

Align and Attend: Multimodal Summarization with Dual Contrastive Losses
Bo He, Jun Wang, Jielin Qiu, Abhinav Shrivastava, Trung Bui, Zhaowen Wang
CVPR 2023 / arXiv / code

Benchmarking Robustness under Distribution Shift of Multimodal Image-Text Models
Jielin Qiu, Yi Zhu, Xingjian Shi, Zhiqiang Tang, Ding Zhao, Bo Li, Mu Li
NeurIPS 2022 Workshop on Distribution Shifts / Paper / Press / code

LiveSeg: Unsupervised Multimodal Temporal Segmentation of Long Livestream Videos
Jielin Qiu, Franck Dernoncourt, Trung Bui, Zhaowen Wang, Ding Zhao, Hailin Jin
WACV 2023 / Paper / arXiv / Press / code (coming soon)

Group Distributionally Robust Reinforcement Learning with Hierarchical Latent Variables
Mengdi Xu, Peide Huang, Yaru Niu, Visak Kumar, Jielin Qiu, Chao Fang, Kuan-Hui Lee, Xuewei Qi, Henry Lam, Bo Li, Ding Zhao
AISTATS 2023 / arxiv / code

GeoECG: Data Augmentation via Wasserstein Geodesic Perturbation for Robust Electrocardiogram Prediction
Jiacheng Zhu*, Jielin Qiu*, Zhuolin Yang, Douglas Weber, Michael Rosenberg, Emerson Liu, Bo Li, Ding Zhao
MLHC 2022 / Paper / arXiv

Data Augmentation via Wasserstein Geodesic Perturbation for Robust Electrocardiogram Prediction
Jiacheng Zhu*, Jielin Qiu*, Zhuolin Yang, Michael Rosenberg, Emerson Liu, Bo Li, Ding Zhao
ICLR 2022 Workshop on Socially Responsible Machine Learning (SRML) / Paper

Comparing Recognition Performance and Robustness of Multimodal Deep Learning Models for Multimodal Emotion Recognition
Wei Liu, Jielin Qiu, Wei-Long Zheng, Bao-Liang Lu
IEEE Transactions on Cognitive and Developmental Systems 2021 / Paper / arXiv / code

Visual Sequence Learning in Hierarchical Prediction Networks and Primate Visual Cortex
Jielin Qiu, Ge Huang, Tai Sing Lee
NeurIPS 2019 / Paper

Investigating Sex Differences in Classification of Five Emotions from EEG and Eye Movement Signals
Lan-Qing Bao, Jielin Qiu, Hao Tang, Wei-Long Zheng, Bao-Liang Lu
EMBC 2019 / Paper / code

Approximation Gradient Error Variance Reduced Optimization
Weiye Zhao, Yang Liu, Xiaoming Zhao, Jielin Qiu, Jian Peng
AAAI-RLG 2019 / Paper

Multi-view Emotion Recognition Using Deep Canonical Correlation Analysis
Jielin Qiu, Wei Liu, Bao-Liang Lu
ICONIP 2018 / Paper / code

Teaching
Services
  • Reviewer: NeurIPS 2023, CVPR 2023, ICML 2023, ICCV 2023, KDD 2023, EACL 2023, MICCAI 2023, ICASSP 2023, WACV 2023, AISTATS 2023, MLHC 2023, CHIL 2023, NeurIPS 2022, CVPR 2022, ICML 2022, ECCV 2022, ACM MM 2022, MLHC 2022, CHIL 2022, ICML 2021.
  • PC Member: ACL 2023, AAAI 2023, EMNLP 2022, AAAI 2022, AAAI 2021.
  • Committee: NeurIPS 2022 virtual deep-dive session chair, CMU RISS Committee.

Design and source code from Jon's and Zhijian's website