The Effect of AI-Assisted Learning on EFL Writing Proficiency: Quasi-Experimental and Cluster Analysis

Background/purpose. The study investigates whether the AI-assisted learning leads to a significant difference in student scores across the measured writing skills, and it also provides valuable insights for teachers and students on utilizing ChatGPT effectively. Specifically, the study applies a quasi-experimental design to assess five core dimensions of writing proficiency: content, organization, vocabulary, language use, and mechanics.

Materials/methods. The research was conducted with a sample of 10 high school students in Dhahran, who were selected through convenience sampling. Over a three-week period, the participants engaged in sessions, practicing how to write while using ChatGPT as a supplementary aid simultaneously. The AI tool provided real-time feedback on grammar, vocabulary, and coherence. The study utilized pre-test and post-test assessments to measure improvements in five specific competencies, namely, content, organization, vocabulary, language use, and mechanics. Participants were at CEFR levels A2–B1. Writing performance was evaluated using Jacobs et al.’s (1981) analytical scoring rubric. Statistical analyses included the Wilcoxon Signed Rank Test to assess significance and Two-Step Cluster Analysis to examine learner response patterns.

Results. The results revealed a substantial increase in scores for content, language use, and vocabulary from pre-test to post-test phases. However, the advancement in scores for organization and mechanics was minuscule. The findings suggest that incorporating AI tools for writing, such as ChatGPT, into EFL instruction can augment certain aspects of writing proficiency. Statistically significant gains were observed in content (p < .01), language use (p < .05), and vocabulary (p ≈ .05), while organization and mechanics did not show significant changes. Cluster analysis identified two distinct student groups, with one demonstrating stronger performance gains.

Conclusion. The increases in the average scores for content, vocabulary, language use, and the total scores indicate that the AI-assisted training was practical in helping EFLs enhance their writing proficiency. Overall, AI-assisted learning shows much promise in improving the writing proficiency of EFL students, but continued research is imperative to explore its full potential while addressing its limitations concurrently. Future research should also consider potential confounding variables, such as age, gender, first language, and cultural background. Accommodating the above factors will ensure accurate and comprehensive results.

Keywords: AI-assisted learning, EFL writing proficiency, ChatGPT, AI-driven learning, AI-aided instructional intervention, technological pedagogical content knowledge

References

Al-khresheh, M. H. (2024). The Future of Artificial Intelligence in English Language Teaching: Pros and Cons of ChatGPT Implementation through a Systematic Review. Language Teaching Research Quarterly, 43, 54-80. https://eric.ed.gov/?id=EJ1457241

Alzahrani, F. K. J. & Alotaibi, H. H. (2024). The impact of artificial intelligence on enhancing EFL writing skills among high school students. Journal of Educational and Human Sciences, 34, 226–240. https://doi.org/10.33193/JEAHS.34.2024.482

Aryanti, R. D., & Santosa, M. H. (2024). A Systematic Review on Artificial Intelligence Applications for Enhancing EFL Students’ Pronunciation Skill. The Art of Teaching English as a Foreign Language (TATEFL), 5(1), 102-113.

Chang, Y. T., Choe, A. T., Holden, D., & Isbell, D. R. (2024). Making each point count: Revising a local adaptation of the Jacobs et al.’s (1981) ESL COMPOSITION PROFILE rubric. Language Testing, 41(2), 443-455.

Chanpradit, T. (2025). Generative artificial intelligence in academic writing in higher education: A systematic review. Edelweiss Applied Science and Technology, 9(4), 889-906.

Cleophas, T. J., & Zwinderman, A. H. (2016). SPSS for Starters and 2nd Levelers (pp. 35-40). Cham, Switzerland: Springer.

Corder, G. W., & Foreman, D. I. (2011). Nonparametric statistics for non‐statisticians.

Crompton, H., Edmett, A., Ichaporia, N., & Burke, D. (2024). AI and English Language Teaching: Affordances and Challenges. British Journal of Educational Technology, 55(6), 2503–2529. https://doi.org/10.1111/bjet.13460

Dugosija, T. (2024). Benefits and challenges of artificial intelligence in English language teaching. Knowledge International Journal, 62(2), 275–280. https://ikm.mk/ojs/index.php/kij/article/view/6590

Eden, C. A., Chisom, O. N., & Adeniyi, I. S. (2024). Integrating AI in Education: Opportunities, Challenges, and Ethical Considerations. Magna Scientia Advanced Research and Reviews, 10(2), 006–013.

Edmett, A., Ichaporia, N., Crompton, H., & Crichton, R. (2023). Artificial Intelligence and English Language Teaching: Preparing for the Future. British Council.

Feng, Haiying, Kexin Li, and Lawrence Jun Zhang. "What does AI bring to second language writing? A systematic review (2014-2024)." (2025). https://scholarspace.manoa.hawaii.edu/items/c86f4480-2a2b-45b5-80ea-576d1fb990da

Field, A. (2018). Discovering Statistics Using IBM SPSS Statistics. Sage Publications.

Fleckenstein, J., Liebenow, L. W., & Meyer, J. (2023). Automated feedback and writing: A multi-level meta-analysis of effects on students' performance. Frontiers in Artificial Intelligence, 6, 1162454. https://www.frontiersin.org/journals/artificial-intelligence/articles/10.3389/frai.2023.1162454/full

George, D., & Mallery, P. (2019). IBM SPSS statistics 26 step by step: A simple guide and reference. Routledge.

Haswell, R. (2005). Researching Teacher Evaluation of Second Language Writing. In T. Silva &P. K. Matsuda (Eds.), Second Language Writing Research: Perspectives on the Process of Knowledge. Construction (pp.105−120). London: Lawrence Erlbaum Associates.

Jiang, R. (2022). How Does Artificial Intelligence Empower EFL Teaching and Learning Nowadays? A Review on Artificial Intelligence in the EFL Context. Frontiers in Psychology, 13, 1049401.

King, R. S. (2015). Cluster analysis and data mining: An introduction. Mercury Learning and Information.

Kushmar Lesia, V., Vornachev, A. O., Korobova, I. O., & Kaida, N. O. (2022). Artificial Intelligence in Language Learning: What Are We Afraid Of. Arab World English Journal.

Mahmoudi, A. (2017). Effect of planning on Iranian intermediate EFL learners' mastery of writing skill. Theory and Practice in Language Studies, 7(3), 219. https://core.ac.uk/download/pdf/266995161.pdf

Marzuki, Widiati, U., Rusdin, D., Darwin, & Indrawati, I. (2023). The impact of AI writing tools on the content and organization of students' writing: EFL teachers' perspective. Cogent Education, 10(2), 1–18. https://doi.org/10.1080/2331186X.2023.2236469

Meyers, L. S., Gamst, G. C., & Guarino, A. J. (2013). Performing data analysis using IBM SPSS. John Wiley & Sons.

Mikeladze, T., & Meijer, P. C. (2024). Exploring the Application of Artificial Intelligence in Foreign Language Education within School Settings: Systematic Literature Review. Journal of Education in Black Sea Region, 10(1). https://jebs.ibsu.edu.ge/jms/index.php/jebs/article/view/328

Pratama, M. D. & Hastuti, D. P. (2024). The use of artificial intelligence to improve EFL students' writing skills. English Learning Innovation, 5(1), 13–25. https://doi.org/10.22219/englie.v5i1.30212

Rahman, A., Raj, A., Tomy, P., & Hameed, M. S. (2024). A comprehensive bibliometric and content analysis of artificial intelligence in language learning: Tracing between the years 2017 and 2023. Artificial Intelligence Review, 57(4), 107.

Setyowati, L., Sukmawan, S., & El-Sulukiyyah , A. (2020). Exploring the Use of ESL Composition Profile for College Writing in the Indonesian Context. International Journal of Language Education, 4(2), 171-18. https://doi.org/10.26858/ijole.v4i2.13662

Shivakumar, A., Shukla, S., Vasoya, M., Kasrani, I. M., & Pei, Y. (2019). AI-Enabled Language Speaking Coaching for Dual Language Learners. IADIS International Journal on WWW/Internet, 17(1).

Son, J.-B., Ružić, N. K., & Philpott, A. (2023). Artificial Intelligence Technologies and Applications for Language Learning and Teaching. Journal of China Computer-Assisted Language Learning, 0.

Song, C. & Song, Y. (2023). Enhancing academic writing skills and motivation: Assessing the efficacy of ChatGPT in AI-assisted language learning for EFL students. Frontiers in Psychology, 14, 1–14. https://doi.org/10.3389/fpsyg.2023.1260843

Tkaczynski, A. (2017). Segmentation using two-step cluster analysis. Segmentation in social marketing: Process, methods and application, 109-125.

Tseng, Y., & Lin, Y. (2024). Enhancing English as a foreign language (EFL) learners’ writing with ChatGPT: A university-level course design. The Electronic Journal of e-Learning, 22(2), 78–97. https://doi.org/10.34190/ejel.21.5.3329

Widiati, U., Rusdin, D., & Indrawati, I. (2023). The impact of AI writing tools on the content and organization of students' writing: EFL teachers' perspective. Cogent Education, 10(2), 1–18. https://doi.org/10.1080/2331186X.2023.2236469

Woo, J. H., & Choi, H. (2021). Systematic Review for Ai-Based Language Learning Tools. arXiv Preprint arXiv:2111.0445

Announcement

EDUPIJ Citation Metrics

EDUPIJ News!