Real-Time Bidirectional Speech Translation With Automated Note Generation: A Hybrid Approach Using Whisper AI And Neural Machine Translation

3 Feb
Posted on Author

Authors: Lalit Sharma, Khushi Rathore, Vishakha Bisen, Dr. Nidhi Dahale

Abstract: This paper presents a novel web-based system for real-time bidirectional speech translation coupled with automated note generation. The system integrates OpenAI's Whisper for offline speech recognition, Google Translate API for neural machine translation, and a React-based frontend for user interaction. Unlike conventional translation systems, our approach includes intelligent text analysis for action item extraction, question detection, and contextual memory to maintain translation coherence across conversation segments. The system achieves an average transcription accuracy of 94% with Whisper's small model and provides sub-2-second latency for real-time translation. Experimental results demonstrate the system's effectiveness in educational settings, business meetings, and cross-cultural communication scenarios.

DOI: http://doi.org/10.5281/zenodo.18466600