
SilentSpeak: A Proposed Multilingual Deep Learning Frame work for Lip Reading in English, Hindi, and Marathi to Assist Individuals with Speech and Hearing Impairments | IJCT Volume 13 – Issue 6 | IJCT-V13I6P93
IJCT
International Journal of Computer Techniques
ISSN 2394-2231 · Peer-Reviewed · Open Access
📚 Volume 13, Issue 6
📅 November 16, 2026
📄 Pages 786–791
🔖 ID: IJCT-V13I6P93
Table of Contents
ToggleSilentSpeak: A Proposed Multilingual Deep Learning Frame work for Lip Reading in English, Hindi, and Marathi to Assist Individuals with Speech and Hearing Impairments
Author(s)
Priyanka Pramod Kurde, Mahek Shaukat Patel, Sakshi Ganesh Muley, Mrs. Aarti Patil
Abstract
Lip reading, or visual speech recognition, can recover spoken content when audio is missing, unusable or inaudible, which is valuable for people with hearing or speech impairments. Most published research targets Eng lish, and the authors’ literature search found no publicly documented word-level lip-reading dataset or system for Hindi or Marathi. This paper proposes SilentSpeak, a conceptual framework for word-level lip reading in English, Hindi and Marathi that is designed to run on a standard webcam without an internet connection. The proposed pipeline locates the lips with MediaPipe Face Mesh, describes each frame with a small convolutional network, and models the mouth movement over time with a bidirectional long short-term memory network. An English base model trained on the GRID corpus is to be adapted to Hindi and Marathi through transfer learning, by freezing the convolutional layers and retraining the recurrent and output layers on small pilot vocabularies. The framework also proposes confidence scoring with a top-three fallback, an emergency-keyword alert and offline speech output. The paper reviews the rele vant literature, identifies the research gap, specifies the methodology and system architecture, and outlines a plan for evaluation. This is a proposed research direction, and no experimental results are claimed in this paper.
Keywords
Assistive technology, Hindi, Lip reading, Marathi, Transfer learning, Visual speech recognition
Conclusion
This paper proposed SilentSpeak, a multilingual framework for word-level lip reading in English, Hindi and Marathi. It combines MediaPipe-based lip extrac tion, a compact CNN–BiLSTM model, transfer learn ing from English to two regional languages, confi dence scoring, an emergency-keyword alert and offline speech output. The review of the literature showed a gap in resources for Hindi and Marathi and little atten tion to practical features such as urgent-word handling. The methodology, architecture and evaluation plan are specified so that the framework can be implemented and tested. This is a proposed research direction, and no experimental results are claimed in this pa per.
References
1] World Health Organization, World Report on Hearing. Ge neva, Switzerland: World Health Organization, 2021, ISBN 978-92-4-002048-1.
[2] S. Varshney, “Deafness in India,” Indian J. Otol., vol. 22, no. 2, pp. 73–76, 2016, doi: 10.4103/0971-7749.182281.
[3] W. H. Sumby and I. Pollack, “Visual contribution to speech intelligibility in noise,” J. Acoust. Soc. Amer., vol. 26, no. 2, pp. 212–215, 1954, doi: 10.1121/1.1907309.
[4] Y. M. Assael, B. Shillingford, S. Whiteson, and N. de Freitas, “LipNet: End-to-end arXiv:1611.01599, sentence-level 2016. https://arxiv.org/abs/1611.01599 [Online]. lipreading,” Available:
[5] J. S. Chung and A. Zisserman, “Lip reading in the wild,” in Proc. Asian Conf. Comput. Vis. (ACCV), 2016, ser. Lecture Notes in Computer Science, vol. 10112, pp. 87–103, doi: 10.1007/978-3-319-54184-6_6.
[6] J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman, “Lip reading sentences in the wild,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, doi: 10.1109/CVPR.2017.367.
[7] T. Stafylakis and G. Tzimiropoulos, “Combining residual net works with LSTMs for lipreading,” in Proc. Interspeech, 2017, pp. 3652–3656, doi: 10.21437/Interspeech.2017-85.
[8] B. Martinez, P. Ma, S. Petridis, and M. Pantic, “Lipreading using temporal convolutional networks,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), 2020, pp. 6319–6323, doi: 10.1109/ICASSP40776.2020.9053841.
[9] S. Fenghour, D. Chen, K. Guo, B. Li, and P. Xiao, “Deep learning-based automated lip-reading: A survey,” IEEE Ac cess, vol. 9, pp. 121184–121205, 2021, doi: 10.1109/AC CESS.2021.3107946.
[10] T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisser man, “Deep audio-visual speech recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 12, pp. 8717–8727, Dec. 2022, doi: 10.1109/TPAMI.2018.2889052.
[11] P. Ma, S. Petridis, and M. Pantic, “Visual speech recognition for multiple languages in the wild,” Nature Mach. Intell., vol. 4, no. 11, pp. 930–939, Nov. 2022, doi: 10.1038/s42256-022 00550-z.
[12] S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Trans. Knowl. Data Eng., vol. 22, no. 10, pp. 1345–1359, Oct. 2010, doi: 10.1109/TKDE.2009.191.
[13] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How trans ferable are features in deep neural networks?,” in Advances in Neural Information Processing Systems, vol. 27, 2014, pp. 3320–3328.
[14] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural net works from overfitting,” J. Mach. Learn. Res., vol. 15, no. 56, pp. 1929–1958, 2014.
[15] C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” J. Big Data, vol. 6, Art. no. 60, 2019, doi: 10.1186/s40537-019-0197-0.
[16] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Comput., vol. 9, no. 8, pp. 1735–1780, 1997, doi: 10.1162/neco.1997.9.8.1735.
[17] M. Schuster and K. K. Paliwal, “Bidirectional recurrent neural networks,” IEEE Trans. Signal Process., vol. 45, no. 11, pp. 2673–2681, Nov. 1997, doi: 10.1109/78.650093.
[18] C. Lugaresi et al., “MediaPipe: A framework for building per ception pipelines,” arXiv:1906.08172, 2019. [Online]. Avail able: https://arxiv.org/abs/1906.08172
[19] Y. Kartynnik, A. Ablavatski, I. Grishchenko, and M. Grund mann, “Real-time facial surface geometry from monocular video on mobile GPUs,” arXiv:1907.06724, 2019. [Online]. Available: https://arxiv.org/abs/1907.06724 M. Cooke, J. Barker, S. Cunningham, and X. Shao, “An audio visual corpus for speech perception and automatic speech recognition,” J. Acoust. Soc. Amer., vol. 120, no. 5, pp. 2421 2424, Nov. 2006, doi: 10.1121/1.2229005.
[21] M. Abadi et al., “TensorFlow: A system for large-scale ma chine learning,” in Proc. 12th USENIX Symp. Oper. Syst. De sign Implement. (OSDI), Savannah, GA, USA, Nov. 2016, pp. 265–283.
[22] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On cali bration of modern neural networks,” in Proc. 34th Int. Conf. Mach. Learn. (ICML), vol. 70, 2017, pp. 1321–1330.
[2] S. Varshney, “Deafness in India,” Indian J. Otol., vol. 22, no. 2, pp. 73–76, 2016, doi: 10.4103/0971-7749.182281.
[3] W. H. Sumby and I. Pollack, “Visual contribution to speech intelligibility in noise,” J. Acoust. Soc. Amer., vol. 26, no. 2, pp. 212–215, 1954, doi: 10.1121/1.1907309.
[4] Y. M. Assael, B. Shillingford, S. Whiteson, and N. de Freitas, “LipNet: End-to-end arXiv:1611.01599, sentence-level 2016. https://arxiv.org/abs/1611.01599 [Online]. lipreading,” Available:
[5] J. S. Chung and A. Zisserman, “Lip reading in the wild,” in Proc. Asian Conf. Comput. Vis. (ACCV), 2016, ser. Lecture Notes in Computer Science, vol. 10112, pp. 87–103, doi: 10.1007/978-3-319-54184-6_6.
[6] J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman, “Lip reading sentences in the wild,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, doi: 10.1109/CVPR.2017.367.
[7] T. Stafylakis and G. Tzimiropoulos, “Combining residual net works with LSTMs for lipreading,” in Proc. Interspeech, 2017, pp. 3652–3656, doi: 10.21437/Interspeech.2017-85.
[8] B. Martinez, P. Ma, S. Petridis, and M. Pantic, “Lipreading using temporal convolutional networks,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), 2020, pp. 6319–6323, doi: 10.1109/ICASSP40776.2020.9053841.
[9] S. Fenghour, D. Chen, K. Guo, B. Li, and P. Xiao, “Deep learning-based automated lip-reading: A survey,” IEEE Ac cess, vol. 9, pp. 121184–121205, 2021, doi: 10.1109/AC CESS.2021.3107946.
[10] T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisser man, “Deep audio-visual speech recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 12, pp. 8717–8727, Dec. 2022, doi: 10.1109/TPAMI.2018.2889052.
[11] P. Ma, S. Petridis, and M. Pantic, “Visual speech recognition for multiple languages in the wild,” Nature Mach. Intell., vol. 4, no. 11, pp. 930–939, Nov. 2022, doi: 10.1038/s42256-022 00550-z.
[12] S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Trans. Knowl. Data Eng., vol. 22, no. 10, pp. 1345–1359, Oct. 2010, doi: 10.1109/TKDE.2009.191.
[13] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How trans ferable are features in deep neural networks?,” in Advances in Neural Information Processing Systems, vol. 27, 2014, pp. 3320–3328.
[14] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural net works from overfitting,” J. Mach. Learn. Res., vol. 15, no. 56, pp. 1929–1958, 2014.
[15] C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” J. Big Data, vol. 6, Art. no. 60, 2019, doi: 10.1186/s40537-019-0197-0.
[16] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Comput., vol. 9, no. 8, pp. 1735–1780, 1997, doi: 10.1162/neco.1997.9.8.1735.
[17] M. Schuster and K. K. Paliwal, “Bidirectional recurrent neural networks,” IEEE Trans. Signal Process., vol. 45, no. 11, pp. 2673–2681, Nov. 1997, doi: 10.1109/78.650093.
[18] C. Lugaresi et al., “MediaPipe: A framework for building per ception pipelines,” arXiv:1906.08172, 2019. [Online]. Avail able: https://arxiv.org/abs/1906.08172
[19] Y. Kartynnik, A. Ablavatski, I. Grishchenko, and M. Grund mann, “Real-time facial surface geometry from monocular video on mobile GPUs,” arXiv:1907.06724, 2019. [Online]. Available: https://arxiv.org/abs/1907.06724 M. Cooke, J. Barker, S. Cunningham, and X. Shao, “An audio visual corpus for speech perception and automatic speech recognition,” J. Acoust. Soc. Amer., vol. 120, no. 5, pp. 2421 2424, Nov. 2006, doi: 10.1121/1.2229005.
[21] M. Abadi et al., “TensorFlow: A system for large-scale ma chine learning,” in Proc. 12th USENIX Symp. Oper. Syst. De sign Implement. (OSDI), Savannah, GA, USA, Nov. 2016, pp. 265–283.
[22] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On cali bration of modern neural networks,” in Proc. 34th Int. Conf. Mach. Learn. (ICML), vol. 70, 2017, pp. 1321–1330.
📋 How to Cite This Paper
Priyanka Pramod Kurde, Mahek Shaukat Patel, Sakshi Ganesh Muley, Mrs. Aarti Patil (2026). SilentSpeak: A Proposed Multilingual Deep Learning Frame work for Lip Reading in English, Hindi, and Marathi to Assist Individuals with Speech and Hearing Impairments. International Journal of Computer Techniques, 13(6), 786–791. ISSN: 2394-2231. DOI: https://doi.org/10.5281/zenodo.24286304









