Shu Kong

I'm a postdoc at RI | CMU supervised by Deva Ramanan. I earned my PhD from ICS | UCI advised by Charless Fowlkes.

My research is motivated by a desire to create intelligent vision systems that benefit humanity. My research focuses on computer vision and machine learning, with an emphasis on real-world applications and interdisciplinary research. My current endeavor is best summarized as "visual perception and learning in an open world", on which I expand briefly in this content of my book (in progress).

Contact

• Email: aimerykong [at] gmail com
• Office: EDSH 101, 5000 Forbes Ave, Pittsburgh, PA, 15213

Links

• Book: Visual Perception and Learning in an Open World
• Workshop: Dealing with the Novelty in Open Worlds (DNOW) at WACV'22, Open-World Vision@CVPR'21
• Challenge: Open-World Image Classification Challenge
• Research Center: CMU Argo AI Center for Autonomous Vehicle Research
• others: Github, Google Scholar, Project Collection...

Recent Updates

• Our in-person workshop Dealing with the Novelty in Open Worlds will be held on Jan 4, 2022, in conjunction with WACV'22 (8/27/2021)

• Our paper "OpenGAN: Open-Set Recognition via Open Data Generation" is accepted for oral presentation at ICCV'21 . Code will be released to github (7/22/2021)

• Our challenge Open-World Image Classification is online now! The challenge will be held in conjunction with our Open World Vision workshop and CVPR'21 (5/14/2021)

• Our paper "Camera Pose Matters: Improving Depth Prediction by Mitigating Pose Distribution Bias" is accepted for oral presentation by CVPR'21 . Read more from Yunhan Zhao (03/3/2021)

• Our virtual workshop Open-World Vision will be held in conjunction with CVPR'21 (12/11/2020)

• Our work is published on "Improving the Taxonomy of Fossil Pollen using Convolutional Neural Networks and Superresolution Microscopy". See highlight in NSF news. (09/14/2020)

Students via Research Supervision / Mentorship

• current
• Shaden N Alshammari (MIT; joint with Deva Ramanan)
• Jeet Kanjani (CMU; joint with Deva Ramanan)
• Shubham Gupta (CMU; joint with Deva Ramanan)
• Zelin Ye (CMU; joint with Deva Ramanan)
• Samia Shafique (UCI; joint with Charless Fowlkes)
• Jennifer Feng (UIUC, joint with Surangi Punyasena)
• Marc-Elie Adaime (UIUC, joint with Surangi Punyasena)
• Francis Yu (UIUC; joint with Yu-Xiong Wang)
• past
• Yunhan Zhao (UCI; joint with Charless Fowlkes)
• Neehar Peri (UMD, now at CMU; joint with Deva Ramanan)
• Yi-Ting Chen (CMU, now at UMD; joint with Deva Ramanan)
• Jinghao Shi (CMU, now at Aibee; joint with Deva Ramanan)
• Ingrid Carolina Romero Valero (UIUC, now at MoreheadState; joint with Surangi Punyasena)
• Derek Haselhorst (UIUC, now at UT Austin; joint with Surangi Punyasena)
• Linfeng Wang (UCI, now at Amazon; joint with Charless Fowlkes)
• Zhiyuan Fang (ASU; joint with Yezhou Yang)

Papers

• S. Kong, D. Ramanan, "OpenGAN: Open-Set Recognition via Open Data Generation", ICCV, 2021
  [paper] [github] [poster]

• Yi-Ting Chen*, Jinghao Shi*, Christoph Mertz, S. Kong, D. Ramanan, "Multimodal Object Detection via Bayesian Fusion", Technical Report, 2021 arXiv:2104.02904
  [project page] [github] [video demo]

• Y. Zhao, S. Kong, C. Fowlkes, "Camera Pose Matters: Improving Depth Prediction by Mitigating Pose Distribution Bias ", CVPR, 2021
  [project page] [arxiv] [github] [slides]

• I. Romero, S. Kong, C. Fowlkes, C. Jaramillo, M. Urban, F. Oboh-Ikuenobe, C. D'Apolito, S. Punyasena, "Improving the Taxonomy of Fossil Pollen using Convolutional Neural Networks and Superresolution Microscopy", Proc. of the National Academy of Sciences (PNAS), 2020.
  [paper][code] [NSF news]

• Y. Zhao, S. Kong, D. Shin, C. Fowlkes, "Domain Decluttering: Simplifying Images to Mitigate Synthetic-Real Domain Shift and Improve Depth Estimation", CVPR, Seattle, 2020
  [project page] [arxiv] [slides] [poster] [github]

• Linfeng Wang, S. Kong, Zachary Pincus, C. Fowlkes, "Celeganser: Automated Analysis of Nematode Morphology and Age ", CVMI@CVPR, Seattle, 2020
  [project page] [preprint] [slides] [poster] [github]

• F. Zhou, S. Kong, C. Fowlkes, T. Chen, B. Lei, "Fine-Grained Facial Expression Analysis Using Dimensional Emotion Model", Neurocomputing, 2020.
  [project page] [arxiv] [demo] [models] [github]

• Z. Fang, S. Kong, Z. Wang, C. Fowlkes, Y. Yang, "Weakly-Supervised Temporal-Language Association with Referring Attention", arXiv:2006.11747, 2020
  [project page] [arxiv]

• S. Kong, C. Fowlkes, "Multigrid Predictive Filter Flow for Unsupervised Learning on Videos", arXiv:1904.01693, 2019.
  [project page] [arxiv] [github] [demo] [slides] [poster]

• Zhiyuan Fang, S. Kong, C. Fowlkes, Yezhou Yang, "Modularized Textual Grounding for Counterfactual Resilience", CVPR, Long Beach, CA, June 2019.
  [paper] [project page] [github]

• D. Haselhorst, S. Kong, C. Fowlkes, S. Punyasena, " Automated identification of diverse palynological samples using convolutional neural nets", 2019. (on the way)
  [page] [code]

• S. Kong, C. Fowlkes, "Image Reconstruction with Predictive Filter Flow", arXiv:1811.11482, 2018.
  [project page] [high-res paper (44MB)] [github] [slides] [poster]

• S. Kong, C. Fowlkes, "Pixel-wise Attentional Gating for Scene Parsing", WACV, Hawaii,2019.
  [project page] [arxiv] [github] [slides] [ROB Entry of Depth Est.] [ROB Entry of Segm.]

• S. Kong, C. Fowlkes, "Recurrent Pixel Embedding for Instance Grouping", CVPR, 2018 .
  [project page] [arxiv] [demo] [models] [github] [poster] [slides]

• S. Kong, C. Fowlkes, "Recurrent Scene Parsing with Perspective Understanding in the Loop", CVPR, 2018.
  [project page] [technical report] [demo] [model] [poster] [slides]

• S. Kong, C. Fowlkes, "Low-rank Bilinear Pooling for Fine-Grained Classification", CVPR, 2017.
  [project page] [technical report] [abstract] [demo] [model] [poster] [slides]

• S. Kong, X. Shen, Z. Lin, R. Mech, C. Fowlkes, "Photo Aesthetics Ranking Network with Attributes and Content Adaptation", ECCV, Amsterdam, the Netherlands, (Oct. 2016).
  [project page] [paper] [code&demo] [dataset&model] [bibtex] [poster] [AMT instruction] [patent filed]

• S. Kong, S. Punyasena, C. Fowlkes, "Spatially Aware Dictionary Learning and Coding for Fossil Pollen Identification", CVPR CVMI Workshop, Los Vegas, NV, (July 2016).
  [project page with code&demo] [paper] [bibtex] [talk] [poster]

• Shu Kong, Zhuolin Jiang, Qiang Yang, "Modeling Neuron Selectivity over Simple Mid-Level Features for Image Classification", IEEE Trans. on Image Processing, 2015
  [paper]

• Yuetan Lin, Shu Kong, Donghui Wang, Yueting Zhuang, "Saliency Detection within a Deep Convolutional Architecture", AAAI'14 Workshop on Cognitive Computing for Augmented Human Intelligence, 2014.
  [paper]

• Shu Kong*, Donghui Wang* "A Classification-Oriented Dictionary Learning Model: Explicitly Learning the Particularity and Commonality Across Categories", Pattern Recognition, 2014.
  [paper] [code]

• Shu Kong, Donghui Wang, "Learning Exemplar-Represented Manifolds in Latent Space for Classification", ECML/PKDD, 2013.
  [paper] [code]

• Donghui Wang, Xikui Wang, Shu Kong, "Integration of Multi-Feature Fusion and Dictionary Learning for Face Recognition", Image and Vision Computing (IVC), 2013.
  [paper] [code]

• Shu Kong, Donghui Wang, "Learning Individual-Specific Dictionaries with Fused Multiple Features for Face Recognition", FG, 2013.
  [paper]

• Shu Kong, Xikui Wang, Donghui Wang, "Multiple Feature Fusion for Face Recognition", FG, 2013.
  [paper] [code]

• Shu Kong, Donghui Wang, "A Dictionary Learning Approach for Classification: Separating the Particularity and the commonality", ECCV, 2012.
  [paper] [code]

• Shu Kong, Donghui Wang, "Transfer Heterogeneous Unlabeled Data for Unsupervised Clustering", ICPR, 2012.
  [paper] [code]

• Shu Kong, Donghui Wang, "A Multi-task Learning Strategy for Unsupervised Clustering via Explicitly Separating the Commonality", ICPR, 2012.
  [paper]

• Donghui Wang, Shu Kong, "Learning Class-Specific Dictionaries for Digit Recognition from Spherical Surface of a 3D Ball", Machine Vision and Applications (MVA), 2012.
  [paper] [SingleBall_dataset (288MB)] [MultiBall_dataset (121MB)]

• Donghui Wang, Shu Kong, "Feature Selection from High-Order Tensorial Data via Sparse Decomposition", Pattern Recognition Letters, 2012.
  [paper] [code]

Abstract/Workshop Papers

• Zhiyuan Fang, Shu Kong, Charless Fowlkes ,Yezhou Yang, " Modularized Textual Grounding for Counterfactual Resilience", Language And Vision workshop joint with CVPR, 2019.

• Surangi W. Punyasena, Shu Kong, Charless C. Fowlkes, "Improving the taxonomic accuracy and precision of fossil pollen identifications", North American Paleontological Convention, Riverside, USA, 2019.

• Ingrid Romero, Shu Kong, Charless C. Fowlkes, Michael A. Urban, Surangi W. Punyasena, "Automated Neotropical Fossil Pollen Fabaceae Analysis Using Convolutional Neural Networks", GSA Annual Meeting in Indianapolis, Indiana, USA, 2018.

• Zhiyuan Fang, Shu Kong, Tianshu Yu, Yezhou Yang, "Weakly Supervised Attention Learning for Textual Phrases Grounding", Language and Vision Workshop jointwith CVPR, 2018.

• Shu Kong, Charless C. Fowlkes, "Low-rank Bilinear Pooling for Fine-Grained Classification", the Fourth Workshop on Fine-grained Visual Categorization joint with CVPR, 2017.

• Shu Kong, Charless C. Fowlkes, "Recurrent Scene Parsing with Perspective Understanding in the Loop", Southern California Machine Learning Symposium, 2017.

• Ingrid Romero, Shu Kong, Charless C. Fowlkes, Michael A. Urban, Carlos D'Apolito, Carlos Jaramillo, OBOH-IKUENOBEA, Francisca E. Oboh-Ikuenobea, Surangi W. Punyasena, "NOVEL MORPHOLOGICAL ANALYSIS OF A FOSSIL FABACEAE POLLEN TYPE, STRIATOPOLLIS CATATUMBUS (TRIBE DETARIAE)", GSA, 2017.
  

• Romero, I.C., S. Kong, C.C. Fowlkes, M.A. Urban, C.A. D'Apolito, C. Jaramillo, F. Oboh-Ikuenobe, and S.W. Punyasena, "Cenozoic biogeography of Striatopollis catatumbus (Fabaceae Detariae)", AASP-The Palynological Society, 2017.
  

• Derek S. Haselhorst, Shu Kong, Charless C. Fowlkes, J. Enrique Moreno, David K. Tcheng, Surangi W. Punyasena, "Automating tropical pollen counts using convolutional neural nets: from image acquisition to identification", the iDigBio inaugural conference, 2017.
  

• Surangi W. Punyasena, Shu Kong, Charless C. Fowlkes, and Stephen P. Jackson, "Reconstructing the extinction dynamics of Picea critchfieldii - the application of computer vision to fossil pollen analysis ", the iDigBio inaugural conference, 2017.
  

• Shu Kong, Charless C. Fowlkes, "Low-rank Bilinear Pooling for Fine-Grained Classification", Southern California Machine Learning Symposium, 2016.
  

Patents

• Utilizing deep learning to rate attributes of digital images, US 2018 / 0268535 A1
• UTILIZING DEEP LEARNING FOR RATING AESTHETICS OF DIGITAL IMAGES, US 20170294010
• Method and Apparatus for Image Content Recognition, CN 201410350987.X
• Method and Apparatus for Image Feature Extraction, CN 201410223300.6

Services

Organizer

• Dealing with the Novelty in Open Worlds @ WACV2022

• Open-World Vision Workshop @ CVPR2021

Reviewer / (Senior) Program Committee

• Conference: CVPR, ICCV, ECCV, ICLR, NeuriPS, ICML, UAI, AAAI, BMVC.

• Journal: IEEE PAMI, IJCV, IEEE TIP, RA-L, IEEE JBHI, IEEE TKDE, PLOS ONE, IEEE THMS, IEEE CYB, JVLC, Palaeo Electronica, PRLetters, IEEE Access, MVAP, DSP, IEEE SPLetters.

Mentorship Program

• RISS, RI, CMU, 2020-2021

• Capstone Advisor, MSCV Program, 2020-2021

• CMU AI Mentoring Program mentor, 2020

• Undergrad GradSchool Q&A Panel (2017), UROP (2015), MDP (2015), Individual Study CompSci299 (2015~2019)

Department/School/University Service

• MSCV Admissions Committee RI-CMU, 2021

• Student Committee of Faculty Hiring CS-ICS-UCI: 2018, 2019

• Graduate Open House Host: 2018, 2019

• Panelist@ASUCI Research Mobilization Commission, 2019

Consulting

• RGG (2020), Trace (2018-2019), US Cabinets Online (2018), Paralian Tech (2017)

Teaching

• Big Data Image Processing & Analysis Course Information (2017Fall), Computational Photography and Vision (2017Spring), Big Data Image Processing & Analysis Course Information (2016Fall), Graph Algorithms (2016Spring), Machine Learning and Data Mining (2015Winter), Introduction to Graphic Models (2015Fall), Graph Algorithms (2015Spring), Machine Learning and Data Mining (2014Winter), Introduction to Artificial Intelligence (2013Spring), Computer Vision (2012Fall), Logic and Computer Design Fundamentals (2011Fall).

Funding/Support

• CMU Argo AI Center for Autonomous Driving Research, 2020 --
• Bob & Barbara Kleist Endowed Graduate Fellowship 2019 - 2020
• NIA R01AG057748 2019
• CVPR PhD Consortium, 2019
• IIS-1253538 2016-2020
• NSF DBI-1262547 2015-2020
• WACV PhD Consortium, 2019
• Google Graduate Student Award, 2017
• Hardware donation from NVIDIA, 2016
• Janelia Junior Scientist Workshop Travel Grant 2016
• Adobe Research Gift 2015
• Multidisciplinary Design Program Grant 2014-2015
• ECCV Travel Award 2012