Human-Computer Interaction Institute Thesis Defense

  • Remote Access - Zoom
  • Virtual Presentation - ET
  • Ph.D. Student
  • Human-Computer Interaction Institute
  • Carnegie Mellon University
Thesis Orals

Practical Privacy Preserving Ambient Sensing

We have entered a new age of computing where the computer is tied not only to a person's body but may also be present in their environment. The ambient presence of sensors enables unprecedented opportunities to build smart environments that adapt to user needs, tracks activities, enables interactions and assists the user in their daily tasks. Even though ambient sensing exists, its scale until recently was limited by the available hardware and computing power. And even despite some recent advancements in their capabilities, ambient sensing techniques generally tend to be privacy intrusive.

In this dissertation, I identify key challenges of robust ambient sensing i.e., the ability to track users and activities via sensors present in the environment. First, there may be multiple users present in the same environment. The need to build reliable novel approaches that detect multiple users and activities from the same sensor stream and identify each user performing those activities presents a unique technical challenge. Additionally, these machine-learning powered techniques require a large amount of training data that posits another challenge of data collection and labeling. Lastly, managing the privacy expectations of all users in a shared environment is a socio-technical challenge that influences the design of those approaches.

In my thesis, I focus on two ambient sensors: cameras and mmWave radar. While mmWave radar is inherently a privacy-preserving sensor, the cameras are regarded as highly intrusive. Thus, I first present a mixed-methods approach to understand the privacy preferences of users for cameras being used as sensors in a range of environments. This work highlights how using privacy preserving techniques to sense activities and clearly communicating how it works may instill trust in a user. Next, I discuss three systems I built that tackle the aforementioned challenges of ambient sensing.

  1. The ability to sense multiple activities: I showcase a camera-based exercise detection and tracking system that can sense different exercise types and count the number of repetitions for 100s of users at the same time.
  2. The ability to identify individual users in the same environment: I present a hybrid camera-IMU approach that uses motion correspondence from both modalities to identify individual users in a scene.
  3. The ability to collect and label data for new sensors: I discuss a novel domain adaptation approach that leverages existing labeled IMU datasets to train a mmWave radar sensor for activity recognition.

I have also conducted appropriate evaluations in unconstrained and semi-constrained environments to underscore the practicality of these approaches. Finally, I also outline how all systems tackle a different challenge of ambient sensing and their impact on the privacy of the user.

Thesis Committee:
Mayank Goel (Chair)
Jodi Forlizzi
Scott Hudson
Thomas Ploetz (Georgia Institute of Technology)

Additional Information

Zoom Participation. See announcement.

For More Information, Please Contact: