SilentSpeak: A Proposed Multilingual Deep Learning Frame work for Lip Reading in English, Hindi, and Marathi to Assist Individuals with Speech and Hearing Impairments | IJCT Volume 13 – Issue 6 | IJCT-V13I6P93

IJCT
International Journal of Computer Techniques
ISSN 2394-2231 · Peer-Reviewed · Open Access
📚 Volume 13, Issue 6
📅 November 16, 2026
📄 Pages 786–791
🔖 ID: IJCT-V13I6P93

SilentSpeak: A Proposed Multilingual Deep Learning Frame work for Lip Reading in English, Hindi, and Marathi to Assist Individuals with Speech and Hearing Impairments

Author(s)

Priyanka Pramod Kurde, Mahek Shaukat Patel, Sakshi Ganesh Muley, Mrs. Aarti Patil

Abstract

Lip reading, or visual speech recognition, can recover spoken content when audio is missing, unusable or inaudible, which is valuable for people with hearing or speech impairments. Most published research targets Eng lish, and the authors’ literature search found no publicly documented word-level lip-reading dataset or system for Hindi or Marathi. This paper proposes SilentSpeak, a conceptual framework for word-level lip reading in English, Hindi and Marathi that is designed to run on a standard webcam without an internet connection. The proposed pipeline locates the lips with MediaPipe Face Mesh, describes each frame with a small convolutional network, and models the mouth movement over time with a bidirectional long short-term memory network. An English base model trained on the GRID corpus is to be adapted to Hindi and Marathi through transfer learning, by freezing the convolutional layers and retraining the recurrent and output layers on small pilot vocabularies. The framework also proposes confidence scoring with a top-three fallback, an emergency-keyword alert and offline speech output. The paper reviews the rele vant literature, identifies the research gap, specifies the methodology and system architecture, and outlines a plan for evaluation. This is a proposed research direction, and no experimental results are claimed in this paper.

Keywords

Assistive technology, Hindi, Lip reading, Marathi, Transfer learning, Visual speech recognition

Conclusion

This paper proposed SilentSpeak, a multilingual framework for word-level lip reading in English, Hindi and Marathi. It combines MediaPipe-based lip extrac tion, a compact CNN–BiLSTM model, transfer learn ing from English to two regional languages, confi dence scoring, an emergency-keyword alert and offline speech output. The review of the literature showed a gap in resources for Hindi and Marathi and little atten tion to practical features such as urgent-word handling. The methodology, architecture and evaluation plan are specified so that the framework can be implemented and tested. This is a proposed research direction, and no experimental results are claimed in this pa per.

References

1] World Health Organization, World Report on Hearing. Ge neva, Switzerland: World Health Organization, 2021, ISBN 978-92-4-002048-1.
[2] S. Varshney, “Deafness in India,” Indian J. Otol., vol. 22, no. 2, pp. 73–76, 2016, doi: 10.4103/0971-7749.182281.
[3] W. H. Sumby and I. Pollack, “Visual contribution to speech intelligibility in noise,” J. Acoust. Soc. Amer., vol. 26, no. 2, pp. 212–215, 1954, doi: 10.1121/1.1907309.
[4] Y. M. Assael, B. Shillingford, S. Whiteson, and N. de Freitas, “LipNet: End-to-end arXiv:1611.01599, sentence-level 2016. https://arxiv.org/abs/1611.01599 [Online]. lipreading,” Available:
[5] J. S. Chung and A. Zisserman, “Lip reading in the wild,” in Proc. Asian Conf. Comput. Vis. (ACCV), 2016, ser. Lecture Notes in Computer Science, vol. 10112, pp. 87–103, doi: 10.1007/978-3-319-54184-6_6.
[6] J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman, “Lip reading sentences in the wild,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, doi: 10.1109/CVPR.2017.367.
[7] T. Stafylakis and G. Tzimiropoulos, “Combining residual net works with LSTMs for lipreading,” in Proc. Interspeech, 2017, pp. 3652–3656, doi: 10.21437/Interspeech.2017-85.
[8] B. Martinez, P. Ma, S. Petridis, and M. Pantic, “Lipreading using temporal convolutional networks,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), 2020, pp. 6319–6323, doi: 10.1109/ICASSP40776.2020.9053841.
[9] S. Fenghour, D. Chen, K. Guo, B. Li, and P. Xiao, “Deep learning-based automated lip-reading: A survey,” IEEE Ac cess, vol. 9, pp. 121184–121205, 2021, doi: 10.1109/AC CESS.2021.3107946.
[10] T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisser man, “Deep audio-visual speech recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 12, pp. 8717–8727, Dec. 2022, doi: 10.1109/TPAMI.2018.2889052.
[11] P. Ma, S. Petridis, and M. Pantic, “Visual speech recognition for multiple languages in the wild,” Nature Mach. Intell., vol. 4, no. 11, pp. 930–939, Nov. 2022, doi: 10.1038/s42256-022 00550-z.
[12] S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Trans. Knowl. Data Eng., vol. 22, no. 10, pp. 1345–1359, Oct. 2010, doi: 10.1109/TKDE.2009.191.
[13] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How trans ferable are features in deep neural networks?,” in Advances in Neural Information Processing Systems, vol. 27, 2014, pp. 3320–3328.
[14] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural net works from overfitting,” J. Mach. Learn. Res., vol. 15, no. 56, pp. 1929–1958, 2014.
[15] C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” J. Big Data, vol. 6, Art. no. 60, 2019, doi: 10.1186/s40537-019-0197-0.
[16] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Comput., vol. 9, no. 8, pp. 1735–1780, 1997, doi: 10.1162/neco.1997.9.8.1735.
[17] M. Schuster and K. K. Paliwal, “Bidirectional recurrent neural networks,” IEEE Trans. Signal Process., vol. 45, no. 11, pp. 2673–2681, Nov. 1997, doi: 10.1109/78.650093.
[18] C. Lugaresi et al., “MediaPipe: A framework for building per ception pipelines,” arXiv:1906.08172, 2019. [Online]. Avail able: https://arxiv.org/abs/1906.08172
[19] Y. Kartynnik, A. Ablavatski, I. Grishchenko, and M. Grund mann, “Real-time facial surface geometry from monocular video on mobile GPUs,” arXiv:1907.06724, 2019. [Online]. Available: https://arxiv.org/abs/1907.06724 M. Cooke, J. Barker, S. Cunningham, and X. Shao, “An audio visual corpus for speech perception and automatic speech recognition,” J. Acoust. Soc. Amer., vol. 120, no. 5, pp. 2421 2424, Nov. 2006, doi: 10.1121/1.2229005.
[21] M. Abadi et al., “TensorFlow: A system for large-scale ma chine learning,” in Proc. 12th USENIX Symp. Oper. Syst. De sign Implement. (OSDI), Savannah, GA, USA, Nov. 2016, pp. 265–283.
[22] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On cali bration of modern neural networks,” in Proc. 34th Int. Conf. Mach. Learn. (ICML), vol. 70, 2017, pp. 1321–1330.

📋 How to Cite This Paper

Priyanka Pramod Kurde, Mahek Shaukat Patel, Sakshi Ganesh Muley, Mrs. Aarti Patil (2026). SilentSpeak: A Proposed Multilingual Deep Learning Frame work for Lip Reading in English, Hindi, and Marathi to Assist Individuals with Speech and Hearing Impairments. International Journal of Computer Techniques, 13(6), 786–791. ISSN: 2394-2231. DOI: https://doi.org/10.5281/zenodo.24286304
© 2026 International Journal of Computer Techniques (IJCT). All rights reserved. · ijctjournal.org
Submit Your Paper