Shu Kong

I'm a postdoc at RI | CMU supervised by Deva Ramanan. I earned my PhD from ICS | UCI advised by Charless Fowlkes. I strive to create intelligent vision systems to benefit humanity. My research interests span computer vision and machine learning (CV/ML), and their applications to autonomous vehicles and interdisciplinary research.

• Computer Vision & Machine Learning. My current research focus is best summarized as "visual perception and learning in an open world", on which I expand briefly in the content of my book (in progress). My recent paper that addresses the open world received Best Paper / Marr Prize Honorable Mention at ICCV 2021.
• Interdisciplinary Research. My recent interdisciplinary endeavor includes applying CV/ML to palynology (what&why?), which is explained in detail in my PNAS paper. The National Science Foundation featured our work as that "opens new era of fossil pollen research", "greatly enhances the use of pollen data in ecological and evolutionary research".

I am applying for faculty positions this season.

Contact

• Email: aimerykong [at] gmail [dot] com
• Office: EDSH 101, 5000 Forbes Ave, Pittsburgh, PA, 15213

Links

• Book: Visual Perception and Learning in an Open World
• Workshop:
  • Visual Perception and Learning in an Open World (VPLOW) at CVPR'22
  • Dealing with the Novelty in Open Worlds (DNOW) at WACV'22
  • Open-World Vision at CVPR'21
• Challenge: Open-World Image Classification Challenge
• Research Center: CMU Argo AI Center for Autonomous Vehicle Research
• others: Github, Google Scholar, Project Collection...

Recent Updates

• Our workshop "Visual Perception and Learning in an Open World" will be held in conjunction with CVPR'22 (12/24/2021)

• Our paper "OpenGAN: Open-Set Recognition via Open Data Generation" received Best Paper / Marr Prize honorable mention at ICCV'21 . Watch this 12min video (10/12/2021)

• Our in-person workshop Dealing with the Novelty in Open Worlds will be held on Jan 4, 2022, in conjunction with WACV'22 (8/27/2021)

• Our challenge Open-World Image Classification is online now! The challenge will be held in conjunction with our Open World Vision workshop and CVPR'21 (5/14/2021)

• Our paper "Camera Pose Matters: Improving Depth Prediction by Mitigating Pose Distribution Bias" is accepted for oral presentation by CVPR'21 . Read more from Yunhan Zhao (03/3/2021)

• Our virtual workshop Open-World Vision will be held in conjunction with CVPR'21 (12/11/2020)

• Our work is published on "Improving the Taxonomy of Fossil Pollen using Convolutional Neural Networks and Superresolution Microscopy", and featured by the NSF. (09/14/2020)

Students via Research Supervision / Mentorship

• current
• Shaden N Alshammari (MIT; joint with Deva Ramanan)
• Zhiqiu Lin (CMU; joint with Deva Ramanan)
• Jeet Kanjani (CMU; joint with Deva Ramanan)
• Shubham Gupta (CMU; joint with Deva Ramanan)
• Zelin Ye (CMU; joint with Deva Ramanan)
• Samia Shafique (UCI; joint with Charless Fowlkes)
• Jennifer Feng (UIUC, joint with Surangi Punyasena)
• Marc-Elie Adaime (UIUC, joint with Surangi Punyasena)
• Francis Yu (UIUC; joint with Yu-Xiong Wang)
• past
• Yunhan Zhao (UCI; joint with Charless Fowlkes)
• Neehar Peri (UMD, now at CMU; joint with Deva Ramanan)
• Yi-Ting Chen (CMU, now at UMD; joint with Deva Ramanan)
• Jinghao Shi (CMU, now at Aibee; joint with Deva Ramanan)
• Ingrid Carolina Romero Valero (UIUC, now at MoreheadState; joint with Surangi Punyasena)
• Derek Haselhorst (UIUC, now at UT Austin; joint with Surangi Punyasena)
• Linfeng Wang (UCI, now at Amazon; joint with Charless Fowlkes)
• Zhiyuan Fang (ASU; joint with Yezhou Yang)

Papers

• S. Kong, D. Ramanan, "OpenGAN: Open-Set Recognition via Open Data Generation", ICCV, 2021
  [webpage] [paper] [github] [poster] [slides] [watch 12min video presentation]
  Best Paper / Marr Prize, Honorable Mention

• Yi-Ting Chen*, Jinghao Shi*, Christoph Mertz, S. Kong, D. Ramanan, "Multimodal Object Detection via Bayesian Fusion", Technical Report, 2021 arXiv:2104.02904
  [project page] [github] [video demo]

• S. Punyasena*, D. Haselhorst*, S. Kong*, C. Fowlkes, J. Moreno, " Automated identification of diverse palynological samples using convolutional neural networks", 2021. (in submission)
  [page] [code]

• Y. Zhao, S. Kong, C. Fowlkes, "Camera Pose Matters: Improving Depth Prediction by Mitigating Pose Distribution Bias ", CVPR, 2021
  [project page] [arxiv] [github] [slides]

• I. Romero, S. Kong, C. Fowlkes, C. Jaramillo, M. Urban, F. Oboh-Ikuenobe, C. D'Apolito, S. Punyasena, "Improving the Taxonomy of Fossil Pollen using Convolutional Neural Networks and Superresolution Microscopy", Proc. of the National Academy of Sciences (PNAS), 2020.
  [paper][code] [NSF news]

• Y. Zhao, S. Kong, D. Shin, C. Fowlkes, "Domain Decluttering: Simplifying Images to Mitigate Synthetic-Real Domain Shift and Improve Depth Estimation", CVPR, Seattle, 2020
  [project page] [arxiv] [slides] [poster] [github]

• Linfeng Wang, S. Kong, Zachary Pincus, C. Fowlkes, "Celeganser: Automated Analysis of Nematode Morphology and Age ", CVMI@CVPR, Seattle, 2020
  [project page] [preprint] [slides] [poster] [github]

• F. Zhou, S. Kong, C. Fowlkes, T. Chen, B. Lei, "Fine-Grained Facial Expression Analysis Using Dimensional Emotion Model", Neurocomputing, 2020.
  [project page] [arxiv] [demo] [models] [github]

• Z. Fang, S. Kong, Z. Wang, C. Fowlkes, Y. Yang, "Weakly-Supervised Temporal-Language Association with Referring Attention", arXiv:2006.11747, 2020
  [project page] [arxiv]

• S. Kong, C. Fowlkes, "Multigrid Predictive Filter Flow for Unsupervised Learning on Videos", arXiv:1904.01693, 2019.
  [project page] [arxiv] [github] [demo] [slides] [poster]

• Zhiyuan Fang, S. Kong, C. Fowlkes, Yezhou Yang, "Modularized Textual Grounding for Counterfactual Resilience", CVPR, Long Beach, CA, June 2019.
  [paper] [project page] [github]

• S. Kong, C. Fowlkes, "Image Reconstruction with Predictive Filter Flow", arXiv:1811.11482, 2018.
  [project page] [high-res paper (44MB)] [github] [slides] [poster]

• S. Kong, C. Fowlkes, "Pixel-wise Attentional Gating for Scene Parsing", WACV, Hawaii,2019.
  [project page] [arxiv] [github] [slides] [ROB Entry of Depth Est.] [ROB Entry of Segm.]

• S. Kong, C. Fowlkes, "Recurrent Pixel Embedding for Instance Grouping", CVPR, Salt Lake City, UT, 2018 .
  [project page] [arxiv] [demo] [models] [github] [poster] [slides]

• S. Kong, C. Fowlkes, "Recurrent Scene Parsing with Perspective Understanding in the Loop", CVPR, Salt Lake City, UT, 2018.
  [project page] [technical report] [demo] [model] [poster] [slides]

• S. Kong, C. Fowlkes, "Low-rank Bilinear Pooling for Fine-Grained Classification", CVPR, Honolulu, HI, 2017.
  [project page] [technical report] [abstract] [demo] [model] [poster] [slides]

• S. Kong, X. Shen, Z. Lin, R. Mech, C. Fowlkes, "Photo Aesthetics Ranking Network with Attributes and Content Adaptation", ECCV, Amsterdam, the Netherlands, (Oct. 2016).
  [project page] [paper] [code&demo] [dataset&model] [bibtex] [poster] [AMT instruction] [patent filed]

• S. Kong, S. Punyasena, C. Fowlkes, "Spatially Aware Dictionary Learning and Coding for Fossil Pollen Identification", CVPR CVMI Workshop, Los Vegas, NV, (July 2016).
  [project page with code&demo] [paper] [bibtex] [talk] [poster]

• Shu Kong, Zhuolin Jiang, Qiang Yang, "Modeling Neuron Selectivity over Simple Mid-Level Features for Image Classification", IEEE Trans. on Image Processing, 2015
  [paper]

• Yuetan Lin, Shu Kong, Donghui Wang, Yueting Zhuang, "Saliency Detection within a Deep Convolutional Architecture", AAAI'14 Workshop on Cognitive Computing for Augmented Human Intelligence, Quebec, Canada, 2014.
  [paper]

• Shu Kong*, Donghui Wang* "A Classification-Oriented Dictionary Learning Model: Explicitly Learning the Particularity and Commonality Across Categories", Pattern Recognition, 2014.
  [paper] [code]

• Shu Kong, Donghui Wang, "Learning Exemplar-Represented Manifolds in Latent Space for Classification", ECML/PKDD, Prague, Czech, 2013.
  [paper] [code]

• Donghui Wang, Xikui Wang, Shu Kong, "Integration of Multi-Feature Fusion and Dictionary Learning for Face Recognition", Image and Vision Computing (IVC), 2013.
  [paper] [code]

• Shu Kong, Donghui Wang, "Learning Individual-Specific Dictionaries with Fused Multiple Features for Face Recognition", FG, Shanghai, China, 2013.
  [paper]

• Shu Kong, Xikui Wang, Donghui Wang, "Multiple Feature Fusion for Face Recognition", FG, Shanghai, China, 2013.
  [paper] [code]

• Shu Kong, Donghui Wang, "A Dictionary Learning Approach for Classification: Separating the Particularity and the commonality", ECCV, Firenze, Italy, 2012.
  [paper] [code]

• Shu Kong, Donghui Wang, "Transfer Heterogeneous Unlabeled Data for Unsupervised Clustering", ICPR, Tsukuba Science City, Japan, 2012.
  [paper] [code]

• Shu Kong, Donghui Wang, "A Multi-task Learning Strategy for Unsupervised Clustering via Explicitly Separating the Commonality", ICPR, Tsukuba Science City, Japan, 2012.
  [paper]

• Donghui Wang, Shu Kong, "Learning Class-Specific Dictionaries for Digit Recognition from Spherical Surface of a 3D Ball", Machine Vision and Applications (MVA), 2012.
  [paper] [SingleBall_dataset (288MB)] [MultiBall_dataset (121MB)]

• Donghui Wang, Shu Kong, "Feature Selection from High-Order Tensorial Data via Sparse Decomposition", Pattern Recognition Letters, 2012.
  [paper] [code]

Abstract/Workshop Papers

• Zhiyuan Fang, Shu Kong, Charless Fowlkes ,Yezhou Yang, " Modularized Textual Grounding for Counterfactual Resilience", Language And Vision workshop joint with CVPR, 2019.

• Surangi W. Punyasena, Shu Kong, Charless C. Fowlkes, "Improving the taxonomic accuracy and precision of fossil pollen identifications", North American Paleontological Convention, Riverside, USA, 2019.

• Ingrid Romero, Shu Kong, Charless C. Fowlkes, Michael A. Urban, Surangi W. Punyasena, "Automated Neotropical Fossil Pollen Fabaceae Analysis Using Convolutional Neural Networks", GSA Annual Meeting in Indianapolis, Indiana, USA, 2018.

• Zhiyuan Fang, Shu Kong, Tianshu Yu, Yezhou Yang, "Weakly Supervised Attention Learning for Textual Phrases Grounding", Language and Vision Workshop jointwith CVPR, 2018.

• Shu Kong, Charless C. Fowlkes, "Low-rank Bilinear Pooling for Fine-Grained Classification", the Fourth Workshop on Fine-grained Visual Categorization joint with CVPR, 2017.

• Shu Kong, Charless C. Fowlkes, "Recurrent Scene Parsing with Perspective Understanding in the Loop", Southern California Machine Learning Symposium, 2017.

• Ingrid Romero, Shu Kong, Charless C. Fowlkes, Michael A. Urban, Carlos D'Apolito, Carlos Jaramillo, OBOH-IKUENOBEA, Francisca E. Oboh-Ikuenobea, Surangi W. Punyasena, "NOVEL MORPHOLOGICAL ANALYSIS OF A FOSSIL FABACEAE POLLEN TYPE, STRIATOPOLLIS CATATUMBUS (TRIBE DETARIAE)", GSA, 2017.
  

• Romero, I.C., S. Kong, C.C. Fowlkes, M.A. Urban, C.A. D'Apolito, C. Jaramillo, F. Oboh-Ikuenobe, and S.W. Punyasena, "Cenozoic biogeography of Striatopollis catatumbus (Fabaceae Detariae)", AASP-The Palynological Society, 2017.
  

• Derek S. Haselhorst, Shu Kong, Charless C. Fowlkes, J. Enrique Moreno, David K. Tcheng, Surangi W. Punyasena, "Automating tropical pollen counts using convolutional neural nets: from image acquisition to identification", the iDigBio inaugural conference, 2017.
  

• Surangi W. Punyasena, Shu Kong, Charless C. Fowlkes, and Stephen P. Jackson, "Reconstructing the extinction dynamics of Picea critchfieldii - the application of computer vision to fossil pollen analysis ", the iDigBio inaugural conference, 2017.
  

• Shu Kong, Charless C. Fowlkes, "Low-rank Bilinear Pooling for Fine-Grained Classification", Southern California Machine Learning Symposium, 2016.
  

Patents

• Utilizing deep learning to rate attributes of digital images, US 2018 / 0268535 A1
• UTILIZING DEEP LEARNING FOR RATING AESTHETICS OF DIGITAL IMAGES, US 20170294010
• Method and Apparatus for Image Content Recognition, CN 201410350987.X
• Method and Apparatus for Image Feature Extraction, CN 201410223300.6

Teaching

• Guest Lectures in Collegiate Courses

  • CMU, 16-720, Graduate Computer Vision, Spring 2022
  • UIUC, CS 446 / ECE 449, Machine Learning, Fall 2021
  • UMich, EECS 442, Computer Vision, Fall 2021
  • CMU, Computer Vision Seminar, Summer 2020
  • UCI, AI & Machine Learning Seminar, Spring 2018

• Outreach Lectures

  • "Visual Perception and Learning in an Open World", Living U, Feb 2022
  • "What is a robot?", Steel City Kindergarten, December 20, 2021
  • "Autonomous Vehicles: from Visual Perception to the Final Autonomous Stack", Living U, March 5, 2021

• Teaching assistant: Big Data Image Processing & Analysis Course Information (2017Fall), Computational Photography and Vision (2017Spring), Big Data Image Processing & Analysis Course Information (2016Fall), Graph Algorithms (2016Spring), Machine Learning and Data Mining (2015Winter), Introduction to Graphic Models (2015Fall), Graph Algorithms (2015Spring), Machine Learning and Data Mining (2014Winter), Introduction to Artificial Intelligence (2013Spring), Computer Vision (2012Fall), Logic and Computer Design Fundamentals (2011Fall).

Services

Organizer

• Visual Perception and Learning in an Open World (VPLOW) at CVPR'22

• Dealing with the Novelty in Open Worlds at WACV'22

• Open-World Vision Workshop at CVPR'21

Reviewer / (Senior) Program Committee

• Conference: CVPR, ICCV, ECCV, ICLR, NeuriPS, ICML, UAI, AAAI, BMVC.

• Journal: IEEE PAMI, IJCV, IEEE TIP, RA-L, IEEE JBHI, IEEE TKDE, PLOS ONE, IEEE THMS, IEEE CYB, JVLC, Palaeo Electronica, PRLetters, IEEE Access, MVAP, DSP, IEEE SPLetters.

Mentorship Program

• RISS, RI, CMU, 2020-2021

• Capstone Advisor, MSCV Program, 2020-2021

• CMU AI Mentoring Program mentor, 2020

• Undergrad GradSchool Q&A Panel (2017), UROP (2015), MDP (2015), Individual Study CompSci299 (2015~2019)

Department/School/University Service

• MSCV Admissions Committee RI-CMU, 2021

• Organizer of CVPR'21 internal review, Robotics Institute, CMU, 2020

• Student Committee of Faculty Hiring CS-ICS-UCI: 2018, 2019

• Graduate Open House Host: 2018, 2019

• Panelist@ASUCI Research Mobilization Commission, 2019

Consulting

• RGG (2020), Trace (2018-2019), US Cabinets Online (2018), Paralian Tech (2017)

Awards

• Best Paper Award / Marr Prize Honorable Mention, 2021
• Bob & Barbara Kleist Endowed Graduate Fellowship, 2019
• CVPR PhD Consortium, 2019
• WACV PhD Consortium, 2019
• Google Graduate Student Award, 2017
• Career Development at Janelia Junior Scientist Workshop 2016
• Multidisciplinary Design Program Grant 2014-2015
• ECCV Travel Award 2012