SignMate: Sign Language Detection System

Authors: Om More Mrinaalini Shankar, Yogesh Pawar, Mrudula Kotgire, Vihar Motghare, Riyah More

Abstract: This paper outlines the designing and improvement of a real-time sign language recognition system that can precisely translate hand movements from a typical webcam. The basis of the system is YOLOv5 (Ultralytics), a cutting-edge object detection model, utilizing the PyTorch deep learning framework for implementation and training. Data acquisition entailed capturing personal Indian Sign Language (ISL) gestures with a webcam under normal illumination. These movements were carefully annotated and exported with Roboflow to produce a high-quality, YOLO-ready dataset. The system initially supported 7 sign classes. Through dramatic enhancements, the dataset grew to 15 classes: "Bye," "Congratulations," "How_Are_You," and "No_Worries," with 50-100 images taken per class to maximize diversity and balance. The YOLOv5s model was trained for 30 epochs with 16 batch size and an image input of 416×416 and achieved a remarkable average mAP@0.5 of more than 98%. Real-time inference is conducted using the detect.py script with accurate bounding box predictions and confidence scores. The whole project was implemented in Python, using Visual Studio Code and Anaconda for managing the environment, and is run locally on a CPU with OpenCV for video processing. The system thus shows much better real-world applicability, diversity of classes, and usability than its previous version and represents a great leap toward more effective gesture-based communication. Future projects involve building an Android app through a Flask API, enhancing the graphical user interface, testing out more larger YOLOv5 models, and incorporating temporal tracking for video-based identification.

DOI: http://doi.org/