Junwei Liang 梁俊卫

			Junwei Liang
			(Ph.D. Candidate)      
			梁俊卫
		
			Language Technologies Institute
		
			School of Computer Science
		
			Carnegie Mellon University
		
			Office: Gates and Hillman Centers 5513
		
			Email: junweil@cs.cmu.edu
		
		My academic blog on Medium.

Publications Awards Research Code Talks

Introduction

Greetings! I'm a 3rd year Ph.D. student under the supervision of Prof. Alexander Hauptmann at Carnegie Mellon University.
My research interests are large-scale computer vision and video analytics.
I am also doing research/building tools for event reconstruction from unorganized social media videos for public safety projects.

News

[07/2020] Awarded "AI Rising Star" at the World AI Conference.
[07/2020] SimAug paper accepted by ECCV 2020.
[06/2020] Multiverse code and dataset are released! [blog] [知乎] [code]
[04/2020] A vision-based Social Distancing Early Forecasting system is open-sourced. Project received Google Cloud COVID-19 Research Grant.
[12/2019] Multiverse paper is out! Accepted by CVPR 2020.
[12/2019] Received Baidu Scholarship (10 recipients globally). Press Coverage: , , , ,
[09/2019] Our Shooter Localization System won Best Demo award at CBMI2019. [Project Site]
Press Coverage: , , , , ,
[06/2019] Presented Future Prediction paper at CVPR 2019. [Tweets]
[04/2019] Our CMU team's (INF & MUDSML) system achieved the best performance on the activity detection challenge (Cached) in surveillance videos hosted by NIST & IARPA. We have released our code and model for Object Detection & Tracking here.
[02/2019] Future activity & path prediction paper reported by the media with 30k+ views in a week.
[12/2018] MemexQA paper accepted by TPAMI.
[06/2018] Presented MemexQA paper at CVPR 2018. [Spotlight Talk]
[03/2017] Two papers accepted by ICASSP 2017.
[02/2017] Two demo papers accepted by AAAI 2017.
[11/2016] Best performer in the NIST TRECVID 2016 Ad-hoc Video Search Challenge (no annotation track).
[02/2016] One oral paper accepted by IJCAI 2016.

Educations

Ph.D. in Language and Information Technology
2017 - present

School of Computer Science, Carnegie Mellon University

Advisor: Alexander Hauptmann
M.S. in Language Technologies
2015 - 2017

School of Computer Science, Carnegie Mellon University

Advisor: Alexander Hauptmann
B.S. in Computer Science
2011 - 2015

School of Information, Renmin University of China

Advisor: Qin Jin

Awards

Rising Star (云帆奖-明日之星), World AI Conference
2020
Baidu Scholarship
2019
Best Demo Award at CBMI2019
2019
Yahoo! Fellowship
2016-2018
Winner, TRECVID ActEV Challenge
2019
Winner, TRECVID Ad-hoc Video Search Challenge, no annotation track
2016
CMU LTI Student Research Symposium Best Paper Honorable Mentions
2018
Google Cloud COVID-19 Research Grant
2020

Selected Media

CBS. Researchers At Carnegie Mellon University Develop Video System To Locate Mass Shooters Using Smartphones, November 20, 2019.
post-gazette. CMU develops video system that can locate mass shooter, November 20, 2019.
GIZMODO. Smartphone Videos Can Now Be Analyzed and Used to Pinpoint the Location of a Shooter, November 21, 2019.
DailyMail. Active shooters can be located within minutes by new software that analyzes smartphone video from the scene and can even identify the type of gun, November 20, 2019.
Techspot. Researchers develop system that can pinpoint a shooter's location using smartphone videos, November 21, 2019.
Baidu. 乘风破浪的AI技术青年——首届WAIC云帆奖名单公布, July 11, 2020.
China.com.cn. 人大高瓴人工智能学院“高屋建瓴-青年说”首期开讲, Jan 6, 2020.
Baidu. AI界的中国力量！百度奖学金助力中国AI人才绽放光芒！, Jan 5, 2020.
量子位. 李飞飞团队造出”窥视未来”新AI:去哪干啥一起猜, 准确率压倒老前辈, received 30k+ views in a week, Feb 13, 2019.
机器之心. 遇见未来！李飞飞等提出端到端系统Next预测未来路径与活动, Feb 14, 2019.

Selected Publications [Google Scholar]

The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction
Junwei Liang, Lu Jiang, Kevin Murphy, Ting Yu, Alexander Hauptmann

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

[Paper] [BibTex] [Demo Video] [Project Page/Code/Model] [blog] [知乎] [Google Research]
SimAug: Learning Robust Representations from Simulation for Trajectory Prediction
Junwei Liang, Lu Jiang, Alexander Hauptmann

In Proceedings of the European conference on computer vision (ECCV), 2020.

[Paper] [BibTex] [Project Page/Code/Model]
Peeking into the Future: Predicting Future Person Activities and Locations in Videos
Junwei Liang, Lu Jiang, Juan Carlos Niebles, Alexander Hauptmann, Li Fei-Fei

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. (Translated and reported by multiple Chinese media (量子位 & 机器之心, 02/13/2019), with 30k+ views in a week.)

[Paper] [BibTex] [Demo Video] [Project Page/Code/Model] [Google Research]
Shooter Localization Using Social Media Videos
Junwei Liang, Jay Aronson, Alexander Hauptmann

In Proceedings of the 27th ACM International Conference on Multimedia, 2019.
(Press coverage: , , , , , , , , , )

[Paper] [BibTex] [Demo Video] [Project Page]
Focal Visual-Text Attention for Memex Question Answering
Junwei Liang, Lu Jiang, Liangliang Cao, Yannis Kalantidis, Li-Jia Li, and Alexander Hauptmann

In IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2019.

[Paper] [BibTex] [Demo Video] [Code/Model/Dataset] [Google Research]
Focal Visual-Text Attention for Visual Question Answering
Junwei Liang, Lu Jiang, Liangliang Cao, Li-Jia Li, and Alexander Hauptmann

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. (Spotlight Paper, 6.8% acceptance rate)

[Paper] [BibTex] [Code/Model] [Presentation] [Google Research]
An Event Reconstruction Tool for Conflict Monitoring Using Social Media
Junwei Liang, Desai Fan, Han Lu, Poyao Huang, Jia Chen, Lu Jiang, and Alexander Hauptmann

In AAAI Conference on Artificial Intelligence (AAAI), 2017.

[Paper] [BibTex] [Demo Video] [Project Page]
Webly-Supervised Learning of Multimodal Video Detectors
Junwei Liang, Lu Jiang, and Alexander Hauptmann

In AAAI Conference on Artificial Intelligence (AAAI), 2017.

[Paper] [BibTex] [Demo Video] [Poster]
Learning to Detect Concepts from Webly-Labeled Video Data
Junwei Liang, Lu Jiang, Deyu Meng, and Alexander Hauptmann

In International Joint Conference on Artificial Intelligence (IJCAI), 2016.

[Paper] [BibTex] [Demo Video]
Video Description Generation using Audio and Visual Cues
Qin Jin, and Junwei Liang

In ACM International Conference on Multimedia Retrieval (ICMR), 2016.

[Paper] [BibTex]
Temporal Localization of Audio Events for Conflict Monitoring in Social Media
Junwei Liang, Lu Jiang and Alexander Hauptmann

In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2017.

[Paper] [BibTex] [Demo Video]

Research Experience

Research Assistant at Carnegie Mellon University
2015 - present

Worked on Large-scale Video Analysis and Retrieval. Studied unsupervised learning of video concept detectors from the Internet. Also participated in the development of event reconstruction tool. The project is for Synchronization and localization of noisy user-generated videos to reconstruct the event scene and timeline from unorganized social media videos, affiliates with CMU Center for Human Rights Science. I'm also the major contributor to the government-funded projects: PSCR by NIST (2017-2020), DIVA by IARPA (2017-2021) and ALADDIN by IARPA (2017). Advised by Prof. Alexander Hauptmann.
Research Intern at Google Cloud AI
May 2020 - Aug 2020

Worked on viewpoint equivariant representation learning for activity recognition. Advised by Ting Yu, Xuehan Xiong and Prof. Liangliang Cao.
Research Intern at Google AI
May 2019 - Aug 2019

Worked on future person activity and trajectory prediction in videos. Integrated research models to a Google Cloud product. Used 3D simulator (carla.org) to collect multi-modal future behavioral data. Advised by Lu Jiang and Prof. Kevin Murphy.
Student Researcher at Google Cloud AI
May 2018 - Dec 2018

Worked on activity recognition and prediction in multi-perspective streaming videos. Studied principal computer vision and high-level semantic reasoning models for interperson and person-object interaction to help AI better understand human activities. Advised by Lu Jiang and Prof. Juan Carlos Niebles.
Research Assistant at Renmin University of China
2013 - 2015

Studied semantic concept annotation on user-generated videos using audio. Participated HUAWEI semantic concept annotation of UGC videos grand challenge 2014 and ranked 3rd in the evaluation. Also worked on natural language description generation for images and videos with deep models. Ranked 1st in ImageCLEF 2015 “image to sentence” subtask in the evaluation. Advised by Prof. Qin Jin.

Teaching & Talks

Research Presentation: The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction
2020

Presented internally at Waymo's machine learning reading group with about 60 attendees.
Contributed Talk: Shooter Localization Using Social Media Videos [Talk]
2020

At AI for Social Good, Carnegie Mellon University
Guest Lecture: Introduction to Machine Learning in Computer Vision [Slides]
2020

11-775 Large-scale Multimedia Analysis, hosted by Prof. Alex Hauptmann and Prof. Rita Singh, Carnegie Mellon University
Invited Talk: Real-time Video Analytics for Situation Awareness [Talk-2018] [Talk-2019]
2019

At the Public Safety Communications Research annual conference hosted by NIST PSCR.
Oral Talk: Focal Visual-Text Attention for Memex Question Answering
2018

At CMU LTI Student Research Symposium 2018. Received best paper honorable mentions award.
Spotlight Talk: Focal Visual-Text Attention for Visual Question Answering [Video]
2018

At CVPR 2018 in Salt Lake City
Oral Talk: Learning to Detect Concepts from Webly-Labeled Video Data
2016

At IJCAI 2016 in New York City.
Teaching Assistant
2018

LTI Colloquium, hosted by Prof. Alex Hauptmann, Carnegie Mellon University
Teaching Assistant
2014

Practical Course on Speech Synthesis and Building Synthetic Voices by Prof. Alan Black, Summer School of Renmin University of China

Academic Service

Journal Reviewer
IEEE Transactions on Image Processing (TIP)

IEEE Transactions on Multimedia

IEEE Access

IEEE Transactions on Circuits and Systems for Video Technology (TCSVT)

ACM TOMM

Neurocomputing

Defense Technology
Conference Reviewer
ACM Multimedia 2017/2019/2020

ACL 2020 Student Research Workshop

CVPR 2020 AI for Content Creation Workshop

Web Application Portfolio

Shooter Localization from Social Media Videos
Widely reported by news media. Presented at CES 2020.
Attendance Management System
Used within CMU LTI for a course with over a hundred students every semester since 2017.
Major CMS websites for organizations
Hillhouse Academy, Annual Computer Design Competition in China (>2000 users annually)