GenAI as a Learning Buddy for Non-English Majors: Effects on Listening and Writing Performance

Background/purpose. Despite the rapid assimilation of generative artificial intelligence (GenAI) into education, and initial evidence suggesting its potential in language learning, rigorous empirical investigations into its efficacy for cultivating specific language skills remain limited. This study aimed to address this lacuna by examining the impact of guided chatbot interactions on English as a Foreign Language (EFL) learners’ listening and writing skills.

Materials/methods. A quasi-experimental design was employed, involving 93 undergraduates enrolled in mandatory EFL courses at a public university. Participants were randomly allocated to an experimental group (n=48), engaging in 10 weekly researcher-designed GenAI-mediated listening and writing activities, or a comparison group (n=45) following a traditional curriculum with optional chatbot use. Data were collected at three time points: pre-test, immediate post-test, and a delayed follow-up. Additionally, post-test qualitative reflections on chatbot use were garnered.

Results. While both groups demonstrated improvement over time, the experimental group reached significantly greater gains in both listening and writing at the immediate post-test compared to the control group. However, these advantages were not maintained at the follow-up assessment. Thematic analysis revealed that students valued the personalized and immediate feedback offered by the chatbots, yet expressed concerns about inconsistent content quality, occasional repetition, and the need for clearer task structures.

Conclusion. Overall, the findings suggest that targeted GenAI-mediated EFL activities can facilitate short-term improvements in listening and writing performance. Future research is advised to investigate approaches for sustaining these gains, optimizing content generation, and refining the user experience to better support second language learners’ long-term development.

Keywords: Artificial intelligence, chatbots, ChatGPT, EFL students, Gemini.

References

Alier, M., Pereira, J., Garcia-Peñalvo, F. J., Casañ, M. J., & Cabré, J. (2025). LAMB: An open-source software framework to create artificial intelligence assistants deployed and integrated into learning management systems. Computer Standards and Interfaces, 92, 103940. https://doi.org/10.1016/j.csi.2024.103940

Chen, A., Jia, J., Li, Y., & Fu, L. (2024). Investigating the effect of role-play activity with GenAI agent on EFL students’ speaking performance. Journal of Educational Computing Research. Advance online publication. https://doi.org/10.1177/07356331241299058

Crowther, D., & Isbell, D. R. (2023). Second language speech comprehensibility: A research agenda. Language Teaching. Advance online publication. https://doi.org/10.1017/s026144482300037x

Escalante, J., Pack, A., & Barrett, A. (2023). AI-generated feedback on writing: insights into efficacy and ENL student preference. International Journal of Educational Technology in Higher Education, 20, 57. https://doi.org/10.1186/s41239-023-00425-2

Foung, D., Lin, L., & Chen, J. (2024). Reinventing assessments with ChatGPT and other online tools: Opportunities for GenAI-empowered assessment practices. Computers and Education Artificial Intelligence, 6, 100250. https://doi.org/10.1016/j.caeai.2024.100250

García-López, I. M., González, C. S. G., Ramírez-Montoya, M., & Molina-Espinosa, J. (2024). Challenges of implementing ChatGPT on education: Systematic literature review. International Journal of Educational Research Open, 8, 100401. https://doi.org/10.1016/j.ijedro.2024.100401

Guo, K., Di Zhang, E., Li, D., & Yu, S. (2024). Using AI‐supported peer review to enhance feedback literacy: An investigation of students’ revision of feedback on peers’ essays. British Journal of Educational Technology. Advance online publication. https://doi.org/10.1111/bjet.13540

Han, L. (2024). Metacognitive writing strategy instruction in the EFL context: Focus on writing performance and motivation. Sage Open, 14(2). https://doi.org/10.1177/21582440241257081

Imran, M., & Almusharraf, N. (2024). Google Gemini as a next generation AI educational tool: A review of emerging educational technology. Smart Learning Environments, 11, 22. https://doi.org/10.1186/s40561-024-00310-z

Ironsi, C. S., & Ironsi, S. S. (2024). Experimental evidence for the efficacy of generative AI in improving students’ writing skills. Quality Assurance in Education. Advance online publication. https://doi.org/10.1108/qae-04-2024-0065

Jackaria, P. M., Hajan, B. H., & Mastul, A. H. (2024). A comparative analysis of the rating of college students’ essays by ChatGPT versus human raters. International Journal of Learning Teaching and Educational Research, 23(2), 478–492. https://doi.org/10.26803/ijlter.23.2.23

Kalniņa, D., Nīmante, D., & Baranova, S. (2024). Artificial intelligence for higher education: Benefits and challenges for pre-service teachers. Frontiers in Education, 9, 1501819. https://doi.org/10.3389/feduc.2024.1501819

Karataş, F., Abedi, F. Y., Gunyel, F. O., Karadeniz, D., & Kuzgun, Y. (2024). Incorporating AI in foreign language education: An investigation into ChatGPT’s effect on foreign language learners. Education and Information Technologies, 29(15), 19343–19366. https://doi.org/10.1007/s10639-024-12574-6

Kim, J., & Hwang, Y. (2024). How are personality factors connected to EFL learners’ oral performance? A psychological network analysis. System, 127, 103516. https://doi.org/10.1016/j.system.2024.103516

Kim, J., Yu, S., Detrick, R., & Li, N. (2024). Exploring students’ perspectives on Generative AI-assisted academic writing. Education and Information Technologies. Advance online publication. https://doi.org/10.1007/s10639-024-12878-7

Kohnke, L., Moorhouse, B. L., & Zou, D. (2023). Exploring generative artificial intelligence preparedness among university language instructors: A case study. Computers and Education: Artificial Intelligence, 5, 100156. https://doi.org/10.1016/j.caeai.2023.100156

Law, L. (2024). Application of generative artificial intelligence (GenAI) in language teaching and learning: A scoping literature review. Computers and Education Open, 6, 100174. https://doi.org/10.1016/j.caeo.2024.100174

Li, B., Lowell, V. L., Wang, C., & Li, X. (2024). A systematic review of the first year of publications on ChatGPT and language education: Examining research on ChatGPT’s use in language learning and teaching. Computers and Education Artificial Intelligence, 7, 100266. https://doi.org/10.1016/j.caeai.2024.100266

Li, X., Li, B., & Cho, S. (2023). Empowering Chinese language learners from low-income families to improve their Chinese writing with ChatGPT’s assistance afterschool. Languages, 8(4), 238. https://doi.org/10.3390/languages8040238

Lim, W. M., Gunasekara, A., Pallant, J. L., Pallant, J. I., & Pechenkina, E. (2023). Generative AI and the future of education: Ragnarök or reformation? A paradoxical perspective from management educators. International Journal of Management Education, 21(2), 100790. https://doi.org/10.1016/j.ijme.2023.100790

Marzuki, Widiati, U., Rusdin, D., Darwin, & Indrawati, I. (2023). The impact of AI writing tools on the content and organization of students’ writing: EFL teachers’ perspective. Cogent Education, 10(2), 2236469. https://doi.org/10.1080/2331186x.2023.2236469

Mora, J. C., Mora‐Plaza, I., & Miranda, G. B. (2024). Speaking anxiety and task complexity effects on second language speech. International Journal of Applied Linguistics, 34(1), 292–315. https://doi.org/10.1111/ijal.12494

Parker, L., Carter, C., Karakas, A., Loper, A. J., & Sokkar, A. (2024). Graduate instructors navigating the AI frontier: The role of ChatGPT in higher education. Computers and Education Open, 6, 100166. https://doi.org/10.1016/j.caeo.2024.100166

Rasul, T., Nair, S., Kalendra, D., Balaji, De Oliveira Santini, F., Ladeira, W. J., Rather, R. A., Yasin, N., Rodriguez, R. V., Kokkalis, P., Murad, M. W., & Hossain, M. U. (2024). Enhancing academic integrity among students in GenAI Era: A holistic framework. International Journal of Management Education, 22(3), 101041. https://doi.org/10.1016/j.ijme.2024.101041

Shahsavar, Z., Kafipour, R., Khojasteh, L., & Pakdel, F. (2024). Is artificial intelligence for everyone? Analyzing the role of ChatGPT as a writing assistant for medical students. Frontiers in Education, 9, 1457744. https://doi.org/10.3389/feduc.2024.1457744

Song, C., & Song, Y. (2023). Enhancing academic writing skills and motivation: Assessing the efficacy of ChatGPT in AI-assisted language learning for EFL students. Frontiers in Psychology, 14, 1260843. https://doi.org/10.3389/fpsyg.2023.1260843

Tafazoli, D. (2024). Exploring the potential of generative AI in democratizing English language education. Computers and Education Artificial Intelligence, 7, 100275. https://doi.org/10.1016/j.caeai.2024.100275

Wang, C., Zou, B., Du, Y., & Wang, Z. (2024). The impact of different conversational generative AI chatbots on EFL learners: An analysis of willingness to communicate, foreign language speaking anxiety, and self-perceived communicative competence. System, 103533. https://doi.org/10.1016/j.system.2024.103533

Xia, Q., Weng, X., Ouyang, F., Lin, T. J., & Chiu, T. K. (2024). A scoping review on how generative artificial intelligence transforms assessment in higher education. International Journal of Educational Technology in Higher Education, 21, 40. https://doi.org/10.1186/s41239-024-00468-z

Yusuf, A., Pervin, N., Román‐González, M., & Noor, N. M. (2024). Generative AI in education and research: A systematic mapping review. Review of Education, 12(2), e3489. https://doi.org/10.1002/rev3.3489

Zhai, X. (2024). Transforming teachers’ roles and agencies in the era of generative AI: Perceptions, acceptance, knowledge, and practices. Journal of Science Education and Technology. Advance online publication. https://doi.org/10.1007/s10956-024-10174-0

Announcement

EDUPIJ Citation Metrics

EDUPIJ News!

► Educational Process International Journal has changed to publish in article number order instead of in page range order beginning with Volume 14 (2025).