MeetDot: Videoconferencing with Live Translation Captions

The current pandemic made videoconferencing an indispensable part of our working lives.

In order to help people, who speak different languages, effectively communicate, a recent paper on arXiv.org proposes a videoconferencing solution with live translation captions.

Image credit: Mbrickn via Wikimedia (CC BY 4.0)

There, participants can see an overlaid translation of other participants’ speech in their preferred language. The incoming speech signal is processed in a streaming mode, transcribed in the speaker’s language, and used as input to a machine translation system. The researchers use several features to enable a better user experience as smooth pixel-wise scrolling of the captions or fading text that is likely to change.

A comprehensive evaluation suite is implemented to accurately compute metrics like latency, caption flicker, and accuracy and motivate fast development according to these metrics.

We present MeetDot, a videoconferencing system with live translation captions overlaid on screen. The system aims to facilitate conversation between people who speak different languages, thereby reducing communication barriers between multilingual participants. Currently, our system supports speech and captions in 4 languages and combines automatic speech recognition (ASR) and machine translation (MT) in a cascade. We use the re-translation strategy to translate the streamed speech, resulting in caption flicker. Additionally, our system has very strict latency requirements to have acceptable call quality. We implement several features to enhance user experience and reduce their cognitive load, such as smooth scrolling captions and reducing caption flicker. The modular architecture allows us to integrate different ASR and MT services in our backend. Our system provides an integrated evaluation suite to optimize key intrinsic evaluation metrics such as accuracy, latency and erasure. Finally, we present an innovative cross-lingual word-guessing game as an extrinsic evaluation metric to measure end-to-end system performance. We plan to make our system open-source for research purposes.

Research paper: Arkhangorodsky, A., “MeetDot: Videoconferencing with Live Translation Captions”, 2021. Link: https://arxiv.org/abs/2109.09577