AI evidence extraction

Teaching Research Integrity through Verification of AI-Generated References: An Activity for Upper-Level Chemistry Courses

Yulia V. Sevryugina, Diego Vargas · Journal of Chemical Education · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1021/acs.jchemed.5c01620

Methodology & findings

Study design

Mixed-methods activity-based study involving 69 students who used AI chatbots to generate bibliographies, followed by systematic validation of citations against scholarly tools, analysis of 456 AI-generated references, survey administration, and in-class discussion analysis..

Sample

N = 69, 3 groups

Primary method

Descriptive statistical analysis of citation validation (counts and percentages); survey analysis and qualitative thematic analysis of in-class discussions. Specific software tools not mentioned in abstract.

Main result

The study found that "on average a student essay contained 51.3% real and correctly cited references" and that "of the original 456 AI-generated citations, only 58% remained in students' final essays, underscoring ongoing difficulties with both citation accuracy and relevance." Additionally, "students developed a greater awareness of research integrity and gained practical skills for critically assessing scholarly sources."

Reports effect sizes.

Research paradigm

Mixed-methods (quantitative analysis of citations + qualitative assessment of student learning)

Author conclusions

The authors conclude that "this activity underscores the importance of fostering both AI literacy and research integrity in chemistry education and offers actionable recommendations to help students use AI responsibly in academic writing." They emphasize that "students directly encountered the challenges and pitfalls of relying on AI-generated sources."

Risk of bias

Selection bias: Students self-selected into upper-level biochemistry and bioinorganic chemistry courses; No control group mentioned for comparison of AI-literacy gains; Survey responses may reflect social desirability bias regarding research integrity awareness; Lack of pre/post comparison design to isolate intervention effects; Selection bias: participant sample limited to students enrolled in specific upper-level chemistry courses; Potential social desirability bias in self-reported survey responses about research integrity awareness; Lack of control group for comparison of AI literacy development; Single institution study may limit generalizability; Selection bias: participants were self-selected upper-level chemistry students; No control group to compare against students not receiving the intervention; Potential social desirability bias in survey responses; No information on blinding or standardization of assessment

Open questions raised

The paper identifies the need for actionable recommendations to help students use AI responsibly in academic writing and emphasizes the importance of domain-specific knowledge in evaluating sources in disciplinary chemistry contexts.

Data: not_statedCode: not_statedExtracted from: pdfAgreement 61%

Explore related topics

Research Integrity Empirical Study