A REAL TIME INDIAN SIGN LANGUAGE RECOGNITION USING TENSORFLOW
DOI:
https://doi.org/10.63458/ijerst.v2i4.98Keywords:
Sign Language Recognition, Machine Learning, Transfer Learning, Tensor Flow, Real-time Systems, Communication Barrier, Deaf Communication, Dumb Communication, Indian Sign Language, Gesture Recognition, Text-to-Audio Translation, Accessibility, Assistive Technology, Communication Aid, Human-Computer Interaction.Abstract
Communication is the exchange of information, ideas, or emotions, typically through spoken or written language. However, for individuals who are deaf or mute, traditional communication methods may not be effective. Instead, they rely on sign language—a visual form of communication using gestures and movements. Unfortunately, many people are unfamiliar with sign language, creating a barrier between those who use it and those who do not. Machine learning offers a promising solution to this challenge. By training a model to recognize and translate sign language gestures into spoken or written language, we can bridge this communication gap. This study proposes a real-time Sign Language Recognition (SLR) system using transfer learning with TensorFlow. Our approach involves capturing Indian Sign Language (ISL) gestures via a webcam and continuously training a deep learning model for accurate, real-time recognition. To enhance usability, we integrate a text-to-audio translation feature, converting recognized gestures into spoken language. This functionality allows individuals proficient in sign language to communicate seamlessly with those who are not, fostering inclusivity and accessibility. By leveraging machine learning, our system aims to break communication barriers and create a more inclusive society.
References
Kapur, R.: The Types of Communication. MIJ. 6, (2020).
Suharjito, Anderson, R., Wiryana, F., Ariesta, M.C., Kusuma, G.P.: Sign Language Recognition Application Systems for Deaf-Mute People: A Review Based on Input Process-Output. Procedia Comput. Sci. 116, 441–448 (2017). https://doi.org/10.1016/J.PROCS.2017.10.028
Konstantinidis, D., Dimitropoulos, K., Daras, P.: Sign Language Recognition Based on Hand and Body Skeletal Data. 3DTV-Conference (2018). https://doi.org/10.1109/3DTV.2018.8478467
Dutta, K.K., Bellary, S.A.S.: Machine Learning Techniques for Indian Sign Language Recognition. Int. Conf. Curr. Trends Comput. Electr. Electron. Commun. CTCEEC2017, 333–336 (2018). https://doi.org/10.1109/CTCEEC.2017.8454988
Bragg, D., Koller, O., Bellard, M., Berke, L., Boudreault, P., Braffort, A., Caselli, N., Huenerfauth, M., Kacorri, H., Verhoef, T., Vogler, C., Morris, M.R.: Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective. 21st Int. ACM SIGACCESS Conf. Comput. Access. (2019). https://doi.org/10.1145/3308561
Rosero-Montalvo, P.D., Godoy-Trujillo, P., Flores-Bosmediano, E., Carrascal-Garcia, J., Otero-Potosi, S., Benitez-Pereira, H., Peluffo-Ordonez, D.H.: Sign Language Recognition Based on Intelligent Glove Using Machine Learning Techniques. 2018 IEEE 3rd Ecuador Tech. Chapters Meet. ETCM (2018). https://doi.org/10.1109/ETCM.2018.8580268
Zheng, L., Liang, B., Jiang, A.: Recent Advances of Deep Learning for Sign Language Recognition. DICTA 2017 - Int. Conf. Digit. Image Comput. Tech. Appl. (2017). https://doi.org/10.1109/DICTA.2017.8227483
Rautaray, S.S.: A Real-Time Hand Tracking System for Interactive Applications. Int. J. Comput. Appl. 18, 975–8887 (2011).
Zhang, Z., Huang, F.: Hand Tracking Algorithm Based on Super-Pixels Feature. Proc. - 2013 Int. Conf. Inf. Sci. Cloud Comput. Companion, ISCC-C 2013, 629–634 (2014). https://doi.org/10.1109/ISCC-C.2013.77
Lim, K.M., Tan, A.W.C., Tan, S.C.: A Feature Covariance Matrix with Serial Particle Filter for Isolated Sign Language Recognition. Expert Syst. Appl. 54, 208–218 (2018). https://doi.org/10.1016/J.ESWA.2016.01.047
Lim, K.M., Tan, A.W.C., Tan, S.C.: Block-Based Histogram of Optical Flow for Isolated Sign Language Recognition. J. Vis. Commun. Image Represent. 40, 538–545 (2016). https://doi.org/10.1016/J.JVCIR.2016.07.020
Gaus, Y.F.A., Wong, F.: Hidden Markov Model-Based Gesture Recognition with Overlapping Hand-Head/Hand-Hand Estimated Using Kalman Filter. Proc. - 3rd Int. Conf. Intell. Syst. Model. Simulation, ISMS 2012, 262–267 (2012). https://doi.org/10.1109/ISMS.2012.67
Nikam, A.S., Ambekar, A.G.: Sign Language Recognition Using Image-Based Hand Gesture Recognition Techniques. Proc. 2016 Online Int. Conf. Green Eng. Technol. IC-GET2016 (2016). https://doi.org/10.1109/GET.2016.7916786
Mohandes, M., Aliyu, S., Deriche, M.: Arabic Sign Language Recognition Using the Leap Motion Controller. IEEE Int. Symp. Ind. Electron., 960–965 (2014). https://doi.org/10.1109/ISIE.2014.6864742
Enikeev, D.G., Mustafina, S.A.: Sign Language Recognition Through Leap Motion Controller and Input Prediction Algorithm. J. Phys. Conf. Ser. 1715, 012008 (2021). https://doi.org/10.1088/1742-6596/1715/1/012008
Cheok, M.J., Omar, Z., Jaward, M.H.: A Review of Hand Gesture and Sign Language Recognition Techniques. Int. J. Mach. Learn. Cybern. 10, 131–153 (2017). https://doi.org/10.1007/S13042-017-0705-5
Wadhawan, A., Kumar, P.: Sign Language Recognition Systems: A Decade Systematic Literature Review. Arch. Comput. Methods Eng. 28, 785–813 (2019). https://doi.org/10.1007/S11831-019-09384-2
Camgöz, N.C., Koller, O., Hadfield, S., Bowden, R.: Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 10020–10030 (2020). https://doi.org/10.1109/CVPR42600.2020.01004
Cui, R., Liu, H., Zhang, C.: A Deep Neural Framework for Continuous Sign Language Recognition by Iterative Training. IEEE Trans. Multimed. 21, 1880–1891 (2019). https://doi.org/10.1109/TMM.2018.2889563
Bantupalli, K., Xie, Y.: American Sign Language Recognition Using Deep Learning and Computer Vision. Proc. - 2018 IEEE Int. Conf. Big Data, Big Data 2018, 4896–4899 (2019). https://doi.org/10.1109/BIGDATA.2018.8622141
About - OpenCV (2017). Poster of the Manual Alphabet in ISL | Indian Sign Language Research and Training Center, Government of India. Transfer Learning and Fine-Tuning | TensorFlow Core (2017).
Wu, S., Yang, J., Wang, X., Li, X.: IoU-Balanced Loss Functions for Single-Stage Object Detection (2020).
