Abstract

Background/purpose – ChatGPT, a recent form of AI-based language model, have garnered interest among people from diverse backgrounds with its immersive capabilities. Using ChatGPT to support or generate scientific research has also created an ongoing debate over its advantages versus risks. The present study aimed to conduct an AI-enabled research process using ChatGPT so as to evaluate its potential to generate an accurate, clear, concise, and unbiased information as these are essential elements of rigorous scientific work.

Materials/methods – To achieve this aim, we worked on emergency remote teaching (ERT), which garnered significant interest due to its wide-spread use during the COVID-19 pandemic, and created opposing views, particularly in comparison to online teaching. We conducted a simultaneous query on ChatGPT-3.5 and 4 on five basic themes: (1) the definition and emergence of ERT, (2) the appropriateness of ERT for different grade levels, (3) a comparison between ERT and online teaching, (4) the possible outcomes of ERT, (5) the future prospects and uses of ERT, and we performed a comparative evaluation of these responses with regard to accuracy, clarity, conciseness, and potential bias. We also used Cohen’s kappa to assess inter-rater agreement in our analysis.

Results – The results indicated that both versions were capable of generating accurate information without significant bias although the responses lacked depth and insight with being somewhat repetitive. As the level of judgment required by the query increased, the performance of ChatGPT-4 was much better; it provided clearer and more concise responses with a more synthesized and detailed categories of information on ERT.

Conclusion – Based on our results, we state that the cooperation of human and artificial intelligence is still warranted to ensure an accurate and reliable output from AI-based scientific queries. If ChatGPT is a plane with innovative technologies, there still needs to be a pilot in the cockpit to make use of these technologies in the best way so as to fly the plane safely to its destination.

Keywords: Artificial intelligence, ChatGPT, emergency remote teaching, OpenAI, generative AI, AI in education, chatbot

Aczel, B., and Wagenmakers, E. 2023. Transparency guidance for ChatGPT usage in scientific writing. PsyArXiv, Preprint.

Adam N.L., Alzahri F.B., Cik S.S., Abu Bakar N., and Mohamad Kamal N.A. 2017. Self-regulated Learning and Online Learning: A Systematic Review. In Advances in Visual Informatics. Lecture Notes in Computer Science. Edited by H. B. Zaman, P. Robinson, A. F. Smeaton, T. K. Shih, S. Velastin, T. Terutoshi, A. J. Nazlena, and M. Ali. New York, NY: Springer, vol. 10645, pp.143-154.

Alkaissi, H., and McFarlane, S. I. 2023. Artificial hallucinations in ChatGPT: Implications in scientific writing. Cureus 15(2). https://doi.org/10.7759/cureus.35179

Aydın, Ö., and Karaarslan, E. 2022. OpenAI ChatGPT generated literature review: digital twin in healthcare. In Emerging Computer Technologies 2. Edited by Ö. Aydın. İzmir: İzmir Akademi Dernegi, pp. 22-31. http://dx.doi.org/10.2139/ssrn.4308687

Barbour, M. K., LaBonte, R., Hodges, C.B., Moore, S., Lockee, B.B., Trust, T. Bond, M.A., Hill, P., and Kelly, K. (2020). Understanding pandemic pedagogy: Differences between emergency remote, remote, and online teaching. Canadian eLearning Network. https://vtechworks.lib.vt.edu/handle/10919/101905  

Ben-David, A. 2008. Comparison of classification accuracy using cohen’s weighted kappa. Expert Systems with Applications 34(2): 825-832.

Borji, A. A. 2023. Categorical archive of ChatGPT failures. arXiv:2302.03494.

Bozkurt, A., and Sharma, R. C. 2020. Emergency remote teaching in a time of global crisis due to coronavirus pandemic. Asian Journal of Distance Education 15(1): i-vi.

Cahan, P., and Treutlein, B. 2023. A Conversation with ChatGPT on the role of computational systems biology in stem cell research. Stem Cell Reports 18(1): 1-2.

Checcucci, E., Verri, P., Amparore, D., Cacciamani, G.E., Fiori, C., Breda, A., Porpiglia, F. 2023. Generative pre-training transformer chat (chatgpt) in the scientific community: The train has left the station. Minerva Urology and Nephrology  75(2): 131-133. https://doi.org/10.23736/S2724-6051.23.05326-0

Chen, T. J. 2023. ChatGPT and other artificial intelligence applications speed up scientific writing. Journal of the Chinese Medical Association 86: 351-353. https://doi.org/10.1097/JCMA.0000000000000900.

Darvas, P. 2017. Liberia - Education Sector - Ebola Recovery and Reconstruction Project: P154880 - Implementation Status Results Report : Sequence 02 (English)  http://documents.worldbank.org/curated/en/171961488217795401/pdf/ISRisclosableP154880-02-27-2017-1488217785586.pdf

Delen, E., and Liew, J. 2016. The use of interactive environments to promote self-regulation in online learning: A literature review. European Journal of Contemporary Education 15(1): 24-33. https://eric.ed.gov/?id=EJ1095976

Dowling, M., & Lucey, B. (2023). ChatGPT for (finance) research: The bananarama conjecture. Finance Research Letters 53: 103662. https://doi.org/10.1016/j.frl.2023.103662

Ferri, F., Grifoni, P., and Guzzo, T. 2020. Online learning and emergency remote teaching: opportunities and challenges in emergency situations. Societies 10(4): 86-95. https://doi.org/10.3390/soc10040086

Gilat, R., and Cole, B. J. 2023. How will artificial intelligence affect scientific writing, reviewing and editing? The Future is Here…. Arthroscopy 39(5): 1119-1120.

Gordijn, B., and Have, H. T. (2023). ChatGPT: Evolution or revolution?. Medicine, Health Care and Philosophy, 1-2.https://doi.org/10.1007/s11019-023-10136-0

Greene, J. 2022. Will ChatGPT make lawyers obsolete? (Hint: Be afraid). Reuters (Dec 09, 2022).

Hallgarten, J. 2020. Evidence on efforts to mitigate the negative educational impact of past disease outbreaks K4D Helpdesk Report 793. Reading, UK: Education Development Trust. https://opendocs.ids.ac.uk/opendocs/handle/20.500.12413/15202

Hodges, C. B., Moore, S., Lockee, B. B., Trust, T., and Bond, M. A. 2020. The difference between emergency remote teaching and online learning. Available at: https://er.educause.edu/articles/2020/3/the-difference-between-emergency-remote-teaching-andonline-learning

Karakose, T. 2021. Emergency remote teaching due to COVID-19 pandemic and potential risks for socioeconomically disadvantaged students in higher education. Educational Process: international journal 10(3): 53-62. https://dx.doi.org/10.22521/edupij.2021.103.4

Karakose, T. 2021b. The impact of the COVID-19 epidemic on higher education: Opportunities and implications for policy and practice. Educational Process: International Journal 10(1): 7-12. https://doi.org/10.22521/edupij.2021.101.1

Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., ... and Kasneci, G. 2023. ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103: 102274. https://doi.org/10.1016/j.lindif.2023.102274

Landis, J.R., and Koch, G.G. 1977. The measurement of observer agreement for categorical data. Biometrics 33: 159-174

Liebrenz, M., Schleifer, R., Buadze, A., Bhugra, D., & Smith, A. (2023). Generating scholarly content with ChatGPT: ethical challenges for medical publishing. The Lancet Digital Health 5(3): e105-e106.

Lund, B. D., and Wang, T. 2023. Chatting about ChatGPT: How may AI and GPT impact academia and libraries?. Library Hi Tech News, ahed-of-print. https://doi.org/10.1108/LHTN-01-2023-0009

OpenAI (2023). GPT-4 technical report. Available at: https://cdn.openai.com/papers/gpt-4.pdf

Sabzalieva, E., and Valentini, A. 2023. Chat GPT and artificial intelligence in higher education: A quick start guide. France: UNESCO

Sallam, M. 2023. ChatGPT utility in healthcare education, research, and practice: Systematic review on the promising perspectives and valid concerns. Healthcare (11)6. https://doi.org/10.3390/healthcare11060887

Smith, W. C. 2021. Consequences of school closure on access to education: Lessons from the 2013–2016 Ebola pandemic. International Review of Education 67(1-2): 53-78.

Stokel-Walker, C., and Van Noorden, R. 2023. What ChatGPT and generative AI mean for science. Nature 614: 214–216.

Strubell, E., Ganesh, A., and McCallum, A. 2019. Energy and policy considerations for deep learning in NLP. Proceedings of the Annual Meeting of the Association for Computational Linguistics, 57: 3645-3650.

Sun, S. 2011. Meta-analysis of Cohen’s kappa. Health Services and Outcomes Research Methodology 11: 145-163. http://doi.org/10.1007/s10742-011-0077-3

Taecharungroj, V. 2023. “What Can ChatGPT Do?”; Analyzing early reactions to the innovative AI Chatbot on Twitter. Big Data and Cognitive Computing 7: 35-49.

van Dis, E.A.M., Bollen, J., Zuidema, W., van Rooij, R., Bockting, C.L. 2023. ChatGPT: Five priorities for research. Nature 614: 224–226.

Vieira, S. M., Kaymak, U., and Sousa, J. M. 2010, July. Cohen's kappa coefficient as a performance measure for feature selection. In International conference on fuzzy systems (pp. 1-8), July, 18-23, 2010 - CCIB, Barcelona, Spain.

Wang, S., Scells, H., Koopman, B., and Zuccon, G. 2023. Can ChatGPT write a good Boolean query for systematic review literature search?. arXiv 2302.03495.

Warner, J., 2023. Biblioracle: Will artificial intelligence like ChatGPT bring the end for all writers? Chicago Tribune (Jan 07, 2023).

Warrens, M. J. 2015. Five ways to look at Cohen's kappa. Journal of Psychology & Psychotherapy 5(4): 1. http://dx.doi.org/10.4172/2161-0487.1000197

Whittle, C., Tiwari, S., Yan, S., and Williams, J. 2020. Emergency remote teaching environment: A conceptual framework for responsive online teaching in crises. Information and Learning Sciences 121(5/6): 311-319.