AI evidence extraction

ChatGPT for Education and Research: Opportunities, Threats, and Strategies

Md. Mostafizer Rahman, Yutaka Watanobe · Applied Sciences · 2023

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

6/10

Relevance

3/4

Quality (LMQS)

Evidence

904

Citations

32.01

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.3390/app13095783

Methodology & findings

Study design

Mixed methods approach combining: (1) literature review of peer-reviewed and preprint articles on ChatGPT in education; (2) controlled experiments with ChatGPT on coding tasks (code generation from problem descriptions validated via Aizu Online Judge system, pseudocode generation, code debugging, and code optimization); (3) surveys with undergraduate, Master's, and doctoral students (n=62 based on survey percentages) and teachers regarding ChatGPT support for programming learning and teaching..

Sample

N = 67, 8 groups

Primary method

Descriptive statistics including percentages, frequency distributions, and accuracy rates. Validation of generated code on Aizu Online Judge (AOJ) platform using online judge compilation and basic compiler. No inferential statistical tests (t-tests, ANOVA, chi-square, etc.) are reported.

Main result

The study found that ChatGPT demonstrates significant capabilities for programming education and support. For code generation from problem descriptions, "the correctness rate of the generated code based on the basic compiler is approximately 95.83%; the correctness rate based on AOJ compilation is approximately 75%". In the survey, "78.8% of the students answered 'Yes'" to using ChatGPT for solving programming problems, and "86.7% of the students answered 'Yes'" when asked if ChatGPT suggestions help in solving programming problems. Additionally, "92.9% of the students answered 'Yes'" regarding the usefulness of ChatGPT for learning programming, and concerning student satisfaction, "most students were satisfied with the programming support provided by ChatGPT" with 36.6% rating 4/5 and 20% rating 5/5.

Reports effect sizes.

Research paradigm

Mixed paradigm (empirical-interpretive)

Author conclusions

The authors conclude: "ChatGPT and other AI LLMs have the potential and can be supporting tools for educational and research work. ChatGPT is a revolutionary LLM that can maintain humanlike conversations and generate human-like text for any natural language query that is nearly indistinguishable. The model can be used to answer questions, write essays, solve problems, explain complex topics, provide virtual tutoring, practice languages, learn programming, teach, and support research." They further state: "Our surveys and experimental results show that ChatGPT is useful not only for programming education but also for education and research. However, although ChatGPT is a powerful tool that can generate impressive responses on a variety of topics, it still has certain limitations, such as a lack of common sense, potential bias, difficulty with complex reasoning, and inability to process visual information."

Risk of bias

Selection bias: Survey respondents self-selected (those using ChatGPT more likely to participate); Limited sample size: Surveys conducted with students and teachers of unspecified institution(s); Temporal bias: Experiments conducted during ChatGPT's active development phase with stated limitations; Non-representative sample: Heavy skew toward undergraduate students (61.3%) with limited programming experience (80.6% rated 1-3/5); Publication bias in literature review: Authors noted 'few peer-reviewed scholarly papers' available; many reviewed preprints; Confirmation bias: Authors may have selected experiments and survey questions favoring ChatGPT's capabilities; Selection bias in survey respondents (self-selected students and teachers); Social desirability bias in survey responses (respondents may overstate satisfaction); Limited problem set in coding experiments (8 random problems); Lack of randomization or control group in survey component; Temporal specificity - experiments conducted on ChatGPT in active development phase; Selection bias in survey participation (self-selected student and teacher participants); Limited sample size for teacher survey (n=10); Timing bias due to ChatGPT being in active development during experiments; Problem selection bias (eight random problems from AOJ used for code generation experiments); Potential recall bias in survey responses about ChatGPT usage

Limitations

The authors state: "It is worth noting that our experiments were conducted on ChatGPT, which is currently in an active development phase
During the experiments, we obtained significant results in code generation, error checking and debugging, and optimization of the solution code
The results may vary for the following reasons: (i) the release of a new version of ChatGPT may lead to different results
(ii) asking different questions than those presented in this study
(iii) results may vary for different problem descriptions
(iv) code-optimization results may vary for different solution codes." Furthermore, "Although ChatGPT generates about 85.42% correct code, it still has the limitations of generating code based on the description

Open questions raised

No comprehensive survey previously conducted on ChatGPT's support for programming learning and teaching
Limited research addressing opportunities, threats, and strategies of ChatGPT for education, research, and programming education
Need for further research to fully understand the impact of AI LLMs such as ChatGPT and strategies for combating misuse
Further research required to design academic curricula, question-and-answer patterns, assignments, and exams to address challenges raised by ChatGPT
Need for new technologies (e.g., AI-based plagiarism detectors) to distinguish between AI-generated and human-generated texts
Research needed on strategies for determining whether responses and programming codes are generated by ChatGPT

Data: "The experimental dataset, including all problem descriptions and erroneous source code, was collected from the Aizu Online Judge System. The web links are: http://developers.u-aizu.ac.jp/index (accessed on 15 March 2023) and https://onlinejudge.u-aizu.ac.jp/ (accessed on 15 March 2023)."; source; description; urls; The experimental dataset including all problem descriptions and erroneous source code was collected from the Aizu Online Judge System. Web links provided: http://developers.u-aizu.ac.jp/index (accessed on 15 March 2023) and https://onlinejudge.u-aizu.ac.jp/ (accessed on 15 March 2023)Code: No code repositories explicitly mentioned or provided.; No code repositories explicitly mentioned in the paper.Extracted from: pdfAgreement 45%

Explore related topics

AI Manuscript Writing Academic Writing Mixed Methods