Po-Yao (Bernie) Huang

Greetings! I am Po-Yao (Bernie) Huang. I am a research scientist of Facebook AI Research (FAIR). I obtained my Ph.D. degree in the Language Technologies Institute (LTI) of School of Computer Science (SCS) at Carnegie Mellon University (CMU) in 2021. My research interest is multimodal machine learning. I am particularly interested in bridging computer vision and natural language processing for the tasks of multimodal machine translation, cross-modal search and retrieval, and large-scale multimodal data mining and analysis.

Education

Work Experience

Publication

• Towards Multilingual Vision-Language Models
  Po-Yao Huang
  Ph.D. Thesis, 2021. [slides]

• VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding
  Hu Xu, Gargi Ghosh, Po-Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, Luke Zettlemoyer
  Findings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL-Findings), 2021.

• Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models
  Po-Yao Huang*, Mandela Patrick*, Junjie Hu, Graham Neubig, Florian Metze, Alexander Hauptmann
  The 2021 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2021.

• Support-set bottlenecks for video-text representation learning
  Mandela Patrick*, Po-Yao Huang*, Yuki Asano*, Florian Metze, Alexander Hauptmann, João Henriques, Andrea Vedaldi
  International Conference on Learning Representations (ICLR), 2021.

• Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting
  Po-Yao Huang, Junjie Hu, Xiaojun Chang, Alexander Hauptmann
  Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020.

• Forward and Backward Multimodal NMT for Improved Monolingual and Multilingual Cross-Modal Retrieval
  Po-Yao Huang, Xiaojun Chang, Alexander Hauptmann, Eduard Hovy
  Proceedings of the 2019 on ACM International Conference on Multimedia Retrieval (ACM ICMR), 2020.

• Structural Analysis and Optimization of Convolutional Neural Networks with a Small Sample Size
  Rhett N D’souza, Po-Yao Huang, Fang-Cheng Yeh
  Scientific Reports, Jan 2020.

• Multi-Head Attention with Diversity for Learning Grounded Multilingual Multimodal Representations
  Po-Yao Huang, Xiaojun Chang, Alexander Hauptmann
  Proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2019.

• Annotation Efficient Cross-Modal Retrieval with Adversarial Attentive Alignment.
  Po-Yao Huang, Guoliang Kang, Wenhe Liu, Xiaojun Chang, Alexander G Hauptmann
  Proceedings of the 27th ACM International Conference on Multimedia (ACM MM), 2019.

• Improving What Cross-Modal Retrieval Models Learn through Object-Oriented Inter- and Intra-Modal Attention Networks
  Po-Yao Huang, Vaibhav, Xiaojun Chang, Alexander Hauptmann
  Proceedings of the 2019 on ACM International Conference on Multimedia Retrieval (ACM ICMR), 2019.

• RCAA: Relational Context-Aware Agents for Person Search
  Xiaojun Chang, Po-Yao Huang, Yi-Dong Shen, Xiaodan Liang, Yi Yang, Alexander Hauptmann
  Proceedings of the European Conference on Computer Vision (ECCV), 2018.

• Multimodal Filtering of Social Media for Temporal Monitoring and Event Analysis
  Po-Yao Huang, Junwei Liang, Jean-Baptiste Lamare, Alexander Hauptmann
  Proceedings of the 2018 on ACM International Conference on Multimedia Retrieval (ACM ICMR), 2018.

• Video Representation Learning and Latent Concept Mining for Large-scale Multi-label Video Classification
  Po-Yao Huang, Ye Yuan, Zhenzhong Lan, Lu Jiang, Alexander Hauptmann
  YouTube 8M Workshop at IEEE Conference on Computer Vision and Pattern Recognition (YT8M@CVPR), 2017.

• Synchronization for Multi-Perspective Videos in the Wild
  Junwei Liang, Po-Yao Huang, Jia Chen, Alexander Hauptmann
  The 42nd IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017.

• An Event Reconstruction Tool for Conflict Monitoring in Social Media
  Junwei Liang, Desai Fan, Han Lu, Po-Yao Huang, Jia Chen, Lu Jiang and Alexander Hauptmann
  The Thirty-First AAAI Conference on Artificial Intelligence (AAAI), 2017.

• Attention-based Multimodal Neural Machine Translation
  Po-Yao Huang, Frederick Liu, Sz-Rung Shiang, Jean Oh, Chris Dyer
  ACL 2016 First Conference on Machine Translation (WMT16), 2016.

• Entity Hierarchy Embedding
  Zhi-Ting Hu, Po-Yao Huang, Yuntian Deng, Yingkai Gao, and Eric Xing [supp]
  The 53rd Annual Meeting of the Association for Computational Linguistics (ACL), 2015.

• Cognitive Vertical Handover in Heterogeneous Networks
  S.M. Cheng, Po-Yao Huang
  The 11th EAI International Conference on Heterogeneous Networking for Quality, Reliability, Security and Robustness (QSHINE), 2015.

• A Cognitive CSMA-based Multichannel MAC for Cognitive Radio Network
  Po-Yao Huang and K.C. Chen
  Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA-ASC) , 2009.

Awards

• 1st Place in the TREC Incident Streams challenge 2019
• 1st Place in the TRECVID ActEV (Activities in Extended Videos) challenge 2019
• 2nd Place in the Kinetics-800 Action Recognition challenge in ActivityNet 2019
• 1st Place in the ActEV challenge in ActivityNet 2019
• 2nd Place in the TRECVID Ad-hoc Video Search challenge 2018
• 2nd Place in the Youtube 8M challenge 2017
• Named Siebel Scholar (Top CS/Business students in top graduate schools worldwide) 2016

Scholarship

• ACL, ICMR, AAAI, ACM MM, CVPR travel awards/grants, 2015-2019
• NSF travel awards, 2015-2019
• CMU Research Fellowship 2014-2019
• Taiwan's Study Abroad Scholarship, 2016
• Siebel Scholarship, 2016
• Din-Jing Memorial Scholarship, 2009