Computer Science Thesis Proposal

  • Gates Hillman Centers
  • Traffic21 Classroom 6501
  • JUNJUE WANG
  • Ph.D. Student
  • Computer Science Department
  • Carnegie Mellon University
Thesis Proposals

Scaling Wearable Cognitive Assistance

Wearable Cognitive Assistance has emerged as a new genre of applications that pushes the boundaries of augmented cognition. These applications continuously process data from body-worn sensors and provide just-in-time guidance to help a user complete a specific task. For example, an IKEA Lamp assistant has been built to assist the assembly of a table lamp. To use the application, a user wears head-mounted smart glasses that continuously captures her actions and surroundings from a first-person viewpoint. In real-time, the camera stream is analyzed to identify the state of the assembly. Audiovisual instructions are generated based on the detected state. The instructions either demonstrate a subsequent procedure or correct a mistake. To meet the latency requirements of these applications, previous research leverages edge computing and offloads computation to a cloudlet. Researchers have developed an application framework for wearable cognitive assistance, named Gabriel, that leverages cloudlets, optimizes for end-to-end latency, and eases application development.

While previous research has demonstrated the technical feasibility of wearable cognitive assistants, many practical concerns have not been addressed. First, previous work operates the wireless networks and cloudlets at low utilization to meet application latency. The economics of practical deployment preclude operation at such low utilization. Second, previous work does not address the most time-consuming parts of creating a wearable cognitive assistance application. Experience has shown that developing computer vision modules that analyze video feeds is a time-consuming and painstaking process that requires special expertise. Developer tools that alleviate the time and the expertise needed can significantly facilitate the creation of these applications.

My thesis will address these challenges. In particular, I claim that: Two critical challenges to the widespread adoption of wearable cognitive assistance are 1) the need to operate cloudlets and wireless network at low utilization to achieve acceptable end-to-end latency 2) the level of specialized skills and the long development time needed to create new applications. These challenges can be effectively addressed through system optimizations, functional extensions, and the addition of new software development tools to the Gabriel platform.

Thesis Committee:
Mahadev Satyanarayanan (Chair)
Daniel Siewiorek
Martial Hebert
Roberta Klatzky
Padmanabhan Pillai (Intel Labs)

Copy of Proposal Summary

For More Information, Please Contact: 
Keywords: