Human-Computer Interaction Thesis Proposal

  • Ph.D. Student
  • Human-Computer Interaction Institute
  • Carnegie Mellon University
Thesis Proposals

Accessible User-Generated Social Media for People with Vision Impairments

Social media platforms are becoming less accessible to people with vision impairments as the prevalence of user-generated images and videos increases. For example, over 25% of content on Twitter contains visual media, but I have found that only 0.1% of images contain descriptions for people with vision impairments. Through interviews with the few users who do write descriptions, I’ve uncovered that poor feature design and a lack of user education is stymieing efforts to increase accessible content on social media platforms. I built and deployed Twitter A11y to demonstrate and evaluate methods to address this problem at scale using multiple methods including text recognition, automatic image captioning, and human crowdsourcing. Participants with vision impairments who used Twitter A11y saw a drastic increase in accessible content on their accounts, with every image having a description and majority being high-quality. Social media platforms include unique categories of images that are hard to make accessible, such as memes. I explored alternative methods to describe memes without losing their humor and built a system to make these accessible by re-using templates created by online volunteers. By combining rich human descriptions and automatic methods, my work seeks to make visual media on social media platforms accessible at scale.

To complete this dissertation, I will explore how to improve human-provided descriptions through the following:

  1. My work on improving the accessibility of memes on Twitter has revealed that animated GIFs are a common accessibility barrier for blind Twitter users. I will explore designs for accessible alternatives, including alt text, audio descriptions, and the original audio from the source material used to create the GIF. Through interviews with blind users, I will determine if these methods may make the visual content of GIFs accessible.
  2. Even the relatively few images on Twitter with alt text often have low-quality descriptions. Using language and image features, I will develop an automated approach to estimate the quality of alt text. I will present this automated quality measure as a feedback mechanism to image description authors, to explore if real-time feedback helps them write better descriptions.

Thesis Committee:
Jeffrey P. Bigham (Co-chair) (HCII)
Kris M. Kitani (Co-chair) (Robotics)
Chieko Asakawa
Patrick Carrington
Meredith Ringel Morris (Microsoft Research)

Additional Proposal Information

For More Information, Please Contact: