A Comparative Study of Machine Translation Tools for English-to-Marathi Language Translation | IJCT Volume 12 – Issue 5 | IJCT-V12I5P62

The rapid advancement of artificial intelligence has led to the development of highly efficient machine translation systems such as Google Translate, ChatGPT and Gemini. This study presents a comparative evaluation of three machine translation systems — Google Translate, ChatGPT and Gemini — based on both manual and automated assessment methods. Manual evaluation focused on accuracy, fluency, and adequacy, while automated evaluation was conducted using the BLEU (Bilingual Evaluation Understudy) score, calculated through Python programming. A One-Way ANOVA test revealed a statistically significant difference among the BLEU scores (F(2,147) = 10.26109, p < 0.05), indicating that translation quality varied across systems. Tukey’s post-hoc test showed that Gemini performed significantly better than both Google and ChatGPT, while no significant difference was observed between Google and ChatGPT. Error analysis further supported these results, with Gemini showing minimal grammatical errors, Google displaying moderate lexical and idiomatic issues, and ChatGPT exhibiting varied misformation and semantic errors. Overall, Gemini demonstrated the highest translation accuracy and fluency, validating its superior performance among the three systems.

This paper presented a comprehensive comparative analysis of Google Translate, ChatGPT, and Gemini for English-to-Marathi machine translation using both manual and automated evaluation methods. Manual assessment based on accuracy, fluency, and adequacy revealed that Gemini consistently produced more natural and contextually appropriate translations, followed by ChatGPT and Google Translate. Automated evaluation using the BLEU score, computed through Python, supported these findings. The one-way ANOVA test yielded an F-value of 10.26109, indicating statistically significant differences among the mean BLEU scores of the three systems. Subsequent Tukey’s post-hoc pairwise analysis showed that while the difference between Google and ChatGPT was not significant, both Google vs. Gemini and ChatGPT vs. Gemini comparisons were statistically significant. This confirms Gemini’s superior translation quality in terms of automated evaluation metrics. The error analysis further revealed that Gemini produced the fewest linguistic and semantic errors, while ChatGPT exhibited frequent lexical and grammatical misformations. Google Translate demonstrated balanced performance but occasionally struggled with idiomatic and context-dependent phrases. Overall, the findings highlight that Gemini outperforms Google Translate and ChatGPT in both manual and automated assessments for English-to-Marathi translation. The study emphasizes the growing reliability of advanced AI-driven systems like Gemini and ChatGPT in handling low-resource regional languages, while also identifying areas for future improvement in context understanding and idiomatic accuracy.

[1]M. Aiken and Sh. Balan, “An analysis of Google Translate accuracy,” Translation Journal, vol. 16, no. 2, 2011. [2]H. Ghasemi and M. Hashemian, “A comparative study of Google Translate translations: An error analysis of English-to-Persian and Persian-to-English translations,” English Language Teaching, vol. 9, no. 3, pp. 13–17, 2016. [3]K. Papineni, S. Roukos, T. Ward, and W. J. Zhu, “BLEU: A method for automatic evaluation of machine translation,” in Proc. 40th Annual Meeting on Association for Computational Linguistics (ACL), 2002, pp. 311–318. [4]R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” in Proc. 54th Annual Meeting of the Association for Computational Linguistics (ACL), 2016, pp. 1715–1725. [5]P. Goyal and G. S. Lehal, “Hindi–Marathi machine translation system using rule-based approach,” Journal of Computing and Information Technology, vol. 16, no. 3, pp. 225–232, 2008. [6]M. Joshi, S. Singh, and S. Saini, “Comparative study of machine translation tools for Indian languages,” Int. J. Adv. Comput. Sci. Appl. (IJACSA), vol. 12, no. 3, pp. 45–52, 2021. [7]M. Popović, “A comparative analysis of evaluation metrics for machine translation,” Language Resources and Evaluation, vol. 49, no. 3, pp. 679–705, 2015. [8]E. Loffler, “Error typologies for machine translation quality assessment,” Linguistica Antverpiensia, vol. 20, pp. 230–245, 2021. [9]P. Rao, L. M. McGee, and C. A. Seideman, “A comparative assessment of ChatGPT vs. Google Translate for the translation of patient instructions,” J. Med. Artif. Intell., vol. 6, no. 11, 2023. [10]M. M. Obeidat, A. S. Haider, S. Abu Tair, and Y. Sahari, “Analyzing the performance of Gemini, ChatGPT, and Google Translate in rendering English idioms into Arabic,” FWU J. Soc. Sci., vol. 18, no. 4, pp. 1–18, 2024. [11]R. Ranathunga, R. Sennrich, et al., “Overview and challenges of machine translation for contextually-rich, low-resource languages,” PMC Survey, 2023. [12]B. Haddow, R. Bawden, A. V. Miceli Barone, J. Helcl, and A. Birch, “Survey of low-resource machine translation,” ACL Anthology, 2022.

Call for Paper (Hub) Fast Paper Publication IJCT Indexing & Databases Submit Manuscript Call for Paper: Sep–Oct 2025 Call for Paper: Publish Research