Junwei Liang / 梁俊卫

			Junwei Liang
			(Ph.D. Candidate)      
			梁俊卫
		
			Language Technologies Institute
		
			School of Computer Science
		
			Carnegie Mellon University
		
			Office: Gates and Hillman Centers 5513
		
			Email: junweil@cs.cmu.edu
		
		My academic blog on Medium.

Publications Awards Media Coverage Projects Talks

Introduction

Greetings! I'm a fourth year Ph.D. student under the supervision of Prof. Alexander Hauptmann at Carnegie Mellon University.
My research interests are large-scale computer vision and video analytics.
My mission: develop AI technologies for social good.

News

[09/2020] We won the Automated Streams Analysis for Public Safety Challenge with a $30k prize.
[08/2020] Our paper has been accepted by WACV 2021 (one strong-accept) and reported by CMU news:
[08/2020] Analyzed videos for journalist from the Washington Post on a major news.
[07/2020] Awarded "AI Rising Star" at the World AI Conference.
[07/2020] SimAug paper accepted by ECCV 2020.
[06/2020] Multiverse code and dataset are released! [blog] [知乎] [code]
[04/2020] A vision-based Social Distancing Early Forecasting system is open-sourced. Project received $6200 Google Cloud Research Grant.
[12/2019] Multiverse paper is out! Accepted by CVPR 2020.
[12/2019] Received Baidu Scholarship (10 recipients globally). Press Coverage: , , , ,
[09/2019] Our Shooter Localization System won Best Demo award at CBMI2019. [Project Site]
Press Coverage: , , , , ,
[06/2019] Presented Future Prediction paper at CVPR 2019. [Tweets]
[04/2019] Our CMU team's (INF & MUDSML) system achieved the best performance on the activity detection challenge (Cached) in surveillance videos hosted by NIST & IARPA. We have released our code and model for Object Detection & Tracking here.
[02/2019] Future activity & path prediction paper reported by the media with 30k+ views in a week.
[12/2018] MemexQA paper accepted by TPAMI 2019.
[06/2018] Presented MemexQA paper at CVPR 2018. [Spotlight Talk]
[03/2017] Two papers accepted by ICASSP 2017.
[02/2017] Two demo papers accepted by AAAI 2017.
[11/2016] Best performer in the NIST TRECVID 2016 Ad-hoc Video Search Challenge (no annotation track).
[02/2016] One oral paper accepted by IJCAI 2016.

Awards

Rising Star (云帆奖-明日之星), World AI Conference
2020
Baidu Scholarship (10 Ph.D. student worldwide)
2019
Winner, Automated Streams Analysis for Public Safety Challenge - $30k prize
2020
Best Demo Award at CBMI2019
2019
Yahoo! Fellowship
2016-2018
Winner, TRECVID ActEV Challenge
2019
Winner, TRECVID Ad-hoc Video Search Challenge, no annotation track
2016
CMU LTI Student Research Symposium Best Paper Honorable Mentions
2018
Google Cloud COVID-19 Research Grant - $6200
2020

Projects

Trajectory/Activity Forecasting [See global paper rankings on PaperWithCode]
2018 - present

SimAug - Multi-view Adversarial Learning

Multiverse - 3D Simulation

Next-Prediction

COVID-19 Project - Social Distancing Early Forecasting [Awarded $6200 GCP research credits]

Object Detection and Tracking in Videos
Public Safety / AI for Social Good [sponsored by NIST] [ASAPS challenge winner]
2017 - present

Gunshot Detection & Shooter Localization

3D Event Reconstruction

Video Analytic Toolkit
Multimodal Question Answering
2016 - 2019

MemexQA - the First Personal Multimedia Collection QA dataset

Dual Attention Network
Weakly Supervised Learning
2015 - 2017

Webly-labeled Learning

Video Semantic Features

Selected Publications [Google Scholar]

The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction
Junwei Liang, Lu Jiang, Kevin Murphy, Ting Yu, Alexander Hauptmann

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

[Paper] [BibTex] [Demo Video] [Project Page/Code/Model] [blog] [知乎] [Google Research] [读芯术学术报告]
Peeking into the Future: Predicting Future Person Activities and Locations in Videos
Junwei Liang, Lu Jiang, Juan Carlos Niebles, Alexander Hauptmann, Li Fei-Fei

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. (Translated and reported by multiple Chinese media (量子位 & 机器之心, 02/13/2019), with 30k+ views in a week.)

#1 Tensorflow-based code on PaperWithCode in Trajectory Prediction task.

[Paper] [BibTex] [Demo Video] [Project Page/Code/Model] [Google Research]
SimAug: Learning Robust Representations from Simulation for Trajectory Prediction
Junwei Liang, Lu Jiang, Alexander Hauptmann

In Proceedings of the European conference on computer vision (ECCV), 2020.

[Paper] [BibTex] [Project Page/Code/Model] [Google Research]
Shooter Localization Using Social Media Videos
Junwei Liang, Jay Aronson, Alexander Hauptmann

In Proceedings of the 27th ACM International Conference on Multimedia, 2019.
(Press coverage: , , , , , , , , , )

[Paper] [BibTex] [Demo Video] [Project Page]
Focal Visual-Text Attention for Memex Question Answering
Junwei Liang, Lu Jiang, Liangliang Cao, Yannis Kalantidis, Li-Jia Li, and Alexander Hauptmann

In IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2019.

[Paper] [BibTex] [Demo Video] [Code/Model/Dataset] [Google Research]
Focal Visual-Text Attention for Visual Question Answering
Junwei Liang, Lu Jiang, Liangliang Cao, Li-Jia Li, and Alexander Hauptmann

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. (Spotlight Paper, 6.8% acceptance rate)

[Paper] [BibTex] [Code/Model] [Presentation] [Google Research]
An Event Reconstruction Tool for Conflict Monitoring Using Social Media
Junwei Liang, Desai Fan, Han Lu, Poyao Huang, Jia Chen, Lu Jiang, and Alexander Hauptmann

In AAAI Conference on Artificial Intelligence (AAAI), 2017.

[Paper] [BibTex] [Demo Video] [Project Page]
Webly-Supervised Learning of Multimodal Video Detectors
Junwei Liang, Lu Jiang, and Alexander Hauptmann

In AAAI Conference on Artificial Intelligence (AAAI), 2017.

[Paper] [BibTex] [Demo Video] [Poster]
Learning to Detect Concepts from Webly-Labeled Video Data
Junwei Liang, Lu Jiang, Deyu Meng, and Alexander Hauptmann

In International Joint Conference on Artificial Intelligence (IJCAI), 2016.

[Paper] [BibTex] [Demo Video]
Video Description Generation using Audio and Visual Cues
Qin Jin, and Junwei Liang

In ACM International Conference on Multimedia Retrieval (ICMR), 2016.

[Paper] [BibTex]
Temporal Localization of Audio Events for Conflict Monitoring in Social Media
Junwei Liang, Lu Jiang and Alexander Hauptmann

In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2017.

[Paper] [BibTex] [Demo Video]

Educations

Ph.D. in Artificial Intelligence
2017 - present

School of Computer Science, Carnegie Mellon University

Advisor: Alexander Hauptmann
M.S. in Artificial Intelligence
2015 - 2017

School of Computer Science, Carnegie Mellon University

Advisor: Alexander Hauptmann
B.S. in Computer Science
2011 - 2015

School of Information, Renmin University of China

Advisor: Qin Jin

Selected Media

Carnegie Mellon University News. Amateur Drone Videos Could Aid in Natural Disaster Damage Assessment, August 28, 2020.
AZO Robotics. New AI System Helps Detect Damage Caused to Buildings by Hurricanes, August 31, 2020.
Washington Post. Lewd cheerleader videos, sexist rules: Ex-employees decry Washington’s NFL team workplace (featured in the video analytics), August 26, 2020.
CBS. Researchers At Carnegie Mellon University Develop Video System To Locate Mass Shooters Using Smartphones, November 20, 2019.
post-gazette. CMU develops video system that can locate mass shooter, November 20, 2019.
GIZMODO. Smartphone Videos Can Now Be Analyzed and Used to Pinpoint the Location of a Shooter, November 21, 2019.
DailyMail. Active shooters can be located within minutes by new software that analyzes smartphone video from the scene and can even identify the type of gun, November 20, 2019.
Techspot. Researchers develop system that can pinpoint a shooter's location using smartphone videos, November 21, 2019.
New York Times. Who Killed the Kiev Protesters? A 3-D Model Holds the Clues (featured in the video analytics), May 30, 2018.
读芯术. 卡内基梅隆大学梁俊卫：视频中行人的多种未来轨迹预测, August, 2020.
Baidu. 乘风破浪的AI技术青年——首届WAIC云帆奖名单公布, July 11, 2020.
China.com.cn. 人大高瓴人工智能学院“高屋建瓴-青年说”首期开讲, Jan 6, 2020.
Baidu. AI界的中国力量！百度奖学金助力中国AI人才绽放光芒！, Jan 5, 2020.
量子位. 李飞飞团队造出”窥视未来”新AI:去哪干啥一起猜, 准确率压倒老前辈, received 30k+ views in a week, Feb 13, 2019.
机器之心. 遇见未来！李飞飞等提出端到端系统Next预测未来路径与活动, Feb 14, 2019.

Research Experience

Research Assistant at Carnegie Mellon University
2015 - present

Worked on Large-scale Video Analysis and Retrieval. Studied unsupervised learning of video concept detectors from the Internet. Also participated in the development of event reconstruction tool. The project is for Synchronization and localization of noisy user-generated videos to reconstruct the event scene and timeline from unorganized social media videos, affiliates with CMU Center for Human Rights Science. I'm also the major contributor to the government-funded projects: PSCR by NIST (2017-2020), DIVA by IARPA (2017-2021) and ALADDIN by IARPA (2017). Advised by Prof. Alexander Hauptmann.
Research Intern at Google Cloud AI
May 2020 - Aug 2020

Worked on viewpoint equivariant representation learning for activity recognition. Advised by Dr. Ting Yu, Dr. Xuehan Xiong and Prof. Liangliang Cao.
Research Intern at Google AI
May 2019 - Aug 2019

Worked on future person activity and trajectory prediction in videos. Integrated research models to a Google Cloud product. Used 3D simulator (carla.org) to collect multi-modal future behavioral data. Advised by Dr. Lu Jiang and Prof. Kevin Murphy.
Student Researcher at Google Cloud AI
May 2018 - Dec 2018

Worked on activity recognition and prediction in multi-perspective streaming videos. Studied principal computer vision and high-level semantic reasoning models for interperson and person-object interaction to help AI better understand human activities. Advised by Dr. Lu Jiang and Prof. Juan Carlos Niebles.
Research Assistant at Renmin University of China
2013 - 2015

Studied semantic concept annotation on user-generated videos using audio. Participated HUAWEI semantic concept annotation of UGC videos grand challenge 2014 and ranked 3rd in the evaluation. Also worked on natural language description generation for images and videos with deep models. Ranked 1st in ImageCLEF 2015 “image to sentence” subtask in the evaluation. Advised by Prof. Qin Jin.

Teaching & Talks

Research Presentation: The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction
2020

Presented internally at Waymo's machine learning reading group with about 60 attendees.
Research Presentation: The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction [Link]
2020

Live presentation for CMU graduate students at LTI Summer Seminar.
Research Presentation: The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction [Talk]
2020

Live presentation for hundreds of Chinese college students.
Contributed Talk: Shooter Localization Using Social Media Videos [Talk]
2020

At AI for Social Good, Carnegie Mellon University
Guest Lecture: Introduction to Machine Learning in Computer Vision [Slides]
2020

11-775 Large-scale Multimedia Analysis, hosted by Prof. Alex Hauptmann and Prof. Rita Singh, Carnegie Mellon University
Invited Talk: Real-time Video Analytics for Situation Awareness [Talk-2018] [Talk-2019]
2019

At the Public Safety Communications Research annual conference hosted by NIST PSCR.
Oral Talk: Focal Visual-Text Attention for Memex Question Answering
2018

At CMU LTI Student Research Symposium 2018. Received best paper honorable mentions award.
Spotlight Talk: Focal Visual-Text Attention for Visual Question Answering [Video]
2018

At CVPR 2018 in Salt Lake City
Oral Talk: Learning to Detect Concepts from Webly-Labeled Video Data
2016

At IJCAI 2016 in New York City.
Teaching Assistant
2018

LTI Colloquium, hosted by Prof. Alex Hauptmann, Carnegie Mellon University
Teaching Assistant
2014

Practical Course on Speech Synthesis and Building Synthetic Voices by Prof. Alan Black, Summer School of Renmin University of China

Academic Service

Journal Reviewer
IEEE Transactions on Image Processing (TIP)

Pattern Recognition

IEEE Transactions on Intelligent Transportation Systems (ITS)

IEEE Transactions on Multimedia

IEEE Access

IEEE Transactions on Circuits and Systems for Video Technology (TCSVT)

ACM TOMM

Neurocomputing

Defense Technology
Conference Reviewer
AAAI 2021

WACV 2021

ACM Multimedia 2017/2019/2020

ACL 2020 Student Research Workshop

CVPR 2020 AI for Content Creation Workshop

Web Application Portfolio

Shooter Localization from Social Media Videos
Widely reported by news media. Presented at CES 2020.
Attendance Management System
Used within CMU LTI for a course with over a hundred students every semester since 2017.
Major CMS websites for organizations
Hillhouse Academy, Annual Computer Design Competition in China (>2000 users annually)