Twenty-First Century Captioning Technology, Metrics and Usability (Captioning DRRP)

ID: 3889
School: Research Center/Lab
Program: Technology Access Program (TAP)
Status: Ongoing
Start date: October 2018
End Date: September 2023


Captioned video is essential for the 36 million Americans who are deaf or hard of hearing. Access to captioned video has a direct impact on participation in society. In the twenty-first century, video is everywhere: entertainment, news, political engagement, government, schools, postsecondary education, at-home learning, social engagement, and much more. However, captioning has not kept up with the shift from broadcast TV to video that can be produced by anyone. The technology and processes for creating captions are fundamentally the same as in the 1980s and 1990s, and do not serve the needs of consumers today. Today, our personal devices have high-quality screens and can support customized captions. At the same time, automatic speech recognition has much potential to both improve caption quality and the availability of captions for us. We are in the middle of a disruptive transition to captions that can be viewed anywhere, anytime. These new technologies create different types of caption errors, compared with human captioning techniques that have evolved over 40 years. As a result, there has been much consumer frustration. With these new technologies, it is critical to understand how caption errors impact consumers who rely on captioned video. We need a way to measure whether captions on a video are good enough for consumers, or not. We also need to understand how modern consumer electronics could support better caption usability and viewer experiences. The Twenty-First Century Captioning Usability & Metrics project has two goals to support the technology transition: First, to develop consumer-focused metrics for caption quality. Second, to improve caption usability on all devices. We have formed a deaf-led five-year research partnership between Gallaudet University, Rochester Institute of Technology and AppTek to achieve these goals. Our approach embraces the perspective of a diverse range of stakeholders, including consumers, caption providers, broadcasters, and other video distributors.

Principal investigators

Additional investigators

Priorities addressed

Funding sources

Approved Products


Datta, P., Balchandani, A., Jakubowicz, P., Glasser, A. & Kushalnagar, R. (2019). Impact of Punctuation Quality on User Experience in Closed Captioning. Poster presentation given at REU AICT 2019, Gallaudet University. Online:

Telecommunications for the Deaf and Hard of Hearing, Inc. (TDI),National Association of the Deaf (NAD),Hearing Loss Association of America (HLAA),Association of Late-Deafened Adults (ALDA),Cerebral Palsy and Deaf Organization (CPADO),Deaf Seniors of America (DSA),Deaf/Hard of Hearing Technology Rehabilitation Engineering Research Center (DHH-RERC),Twenty-First Century Captioning Disability and Rehabilitation Research Project (Captioning DRRP),Rehabilitation Engineering Research Center on Universal Interface & Information Technology Access (IT-RERC),National Technical Institute for the Deaf (2019). Petition for Rulemaking Regarding Live Captioning Quality Metrics and Use of Automated Speech Recognition. Filed with the FCC, CG Docket 05-231, July 31, 2019

Vogler, C. & Williams, N. (2019). Browser-based renderer for live closed captions.

Vogler, C. (2019). Key Performance Indicators for Accessibility – how do we measure accessibility? Presentation at the M-Enabling Summit, Arlington, VA, June 2019.


Berke, L, Seita, M, and Huenerfauth, M (2020). Deaf and hard-of-hearing users’ prioritization of genres of online video content requiring accurate captions. Proceedings of the 17th International Web for All Conference (W4A ’20) (pp. 1-12). New York, NY: ACM.

Glasser, A, Riley, E. M., Weeks, K. and Kushalnagar, R. (2019). Mixed Reality Speaker Identification as an Accessibility Tool for Deaf and Hard of Hearing Users. ACM Symposium on Virtual Reality Software and Technology (VRST '19). New York, NY: ACM.

Kafle, S and Huenerfauth, M (2020). Usability Evaluation of Captions for People who are Deaf or Hard of Hearing. SIGACCESS Accessibility and Computing, 122, N/A.

Kafle, S, Alm, Co. O., and Huenerfauth, M (2019). Fusion Strategy for Prosodic and Lexical Representations of Word Importance. Proceedings of the 20th Annual Conference of the International Speech Communication Association. Graz, Austria: International Speech Communication Association.

Li, J, Luettgen, M, Huenerfauth,M, Al-khazraji, S, Bailey, R, and Alm, C. O. (2020). Gaze Guidance for Captioned Videos for DHH Users. Journal on Technology and Persons with Disabilities, 8, N/A.

Vogler, C (2019). Comments on TDI et al. Petition for Declaratory Ruling and/or Ruling Making on Live Closed Captioning Quality Metrics and the Use of Automatic Speech Recognition Technologies. FCC.


A. A. Amin, S. Hassan, M. Huenerfauth. 2021. Caption-Occlusion Severity Judgments across Live-TelevisionGenres from Deaf and Hard-of-Hearing Viewers. In Proceedings of the 18th International Web for All Conference(W4A’21). ACM, New York, NY, USA. DOI:

Akhter Al Amin, Abraham Glasser, Raja Kushalnagar, Christian Vogler, Matt Huenerfauth. 2021 “Preferences of Deaf or Hard of Hearing Users for Live-TV Caption Appearance.” Universal Access in Human-Computer Interaction. Lecture Notes in Computer Science. Springer, Cham.

Akhter Al Amin, Saad Hassan, Matt Huenerfauth. 2021. “Effect of Occlusion on Deaf and Hard of Hearing Users’ Perception of Captioned Video Quality.” Universal Access in Human-Computer Interaction. Lecture Notes in Computer Science. Springer, Cham.

Datta, P., Jakubowicz, P., Vogler, C., & Kushalnagar, R. (2020). Readability of Punctuation in Automatic Subtitles.In N/A (Ed.), International Conference on Computers Helping People with Special Needs (pp. 195-201). N/A: Springer.

J. Li, M. Luettgen, S. Al-khazraji, M. Huenerfauth, R. Bailey, C. O. Alm. Gaze Guidance for Captioned Videos for DHH Users. 2020. In Proceedings of the 35th Annual Assistive Technology Conference. Journal on Technology and Persons with Disabilities.

M. Seita and M. Huenerfauth. 2020. Deaf Individuals' Views on Speaking Behaviors of Hearing Peers when Using an Automatic Captioning App. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA '20). Association for Computing Machinery, New York, NY, USA, 1–8. DOI:

Yeung P., Alonzo O., Huenerfauth M. (2020) Interest and Requirements for Sound-Awareness Technologies Among Deaf and Hard-of-Hearing Users of Assistive Listening Devices. In: Antona M., Stephanidis C. (eds) Universal Access in Human-Computer Interaction. Applications and Practice. HCII 2020. Lecture Notes in Computer Science, vol 12189. Springer, Cham.