Hands-Free Virtual Mouse with Gesture, Eye, and Speech Recognition Using Machine Learning Algorithms

Authors- Assistant Professor Mr. M. Prathap, Birru Ganesh Vardhan Yadav, Mogadala Prem Kumar, Bemuni Palli Naga Suvarna, Tankasala Anjali

Abstract-The advancement of human-computer interaction (HCI) has led to the development of hands-free control systems, enabling accessibility and convenience in digital environments. This paper presents a Hands-Free Virtual Mouse utilizing gesture, eye, and speech recognition powered by machine learning algorithms to enhance user experience and accessibility. The system integrates multiple recognition techniques to simulate conventional mouse functions. Gesture recognition is achieved using computer vision and deep learning models, allowing users to perform cursor movements and clicks through predefined hand gestures. Eye tracking is implemented via facial landmark detection and gaze estimation, providing an intuitive pointer control mechanism. Additionally, speech recognition employs natural language processing (NLP) models to execute commands such as “click,” “scroll,” and “drag.”Machine learning techniques, including Convolutional Neural Networks (CNNs) for gesture detection, Support Vector Machines (SVMs) for gaze estimation, and Deep Neural Networks (DNNs) for speech command classification, are used to ensure high accuracy and responsiveness. The system is trained on extensive datasets to improve adaptability across different users and environments. This hands-free virtual mouse can significantly benefit individuals with physical disabilities, virtual reality (VR) applications, and smart home interfaces, reducing the dependency on traditional input devices. The results demonstrate that combining multiple recognition modalities enhances usability, making the system a robust alternative to conventional pointing devices.

DOI: /10.61463/ijset.vol.13.issue2.239