AI evidence extraction

College students' credibility assessments of GenAI‐generated information for academic tasks: An interview study

Wonchan Choi, Hyerin Bak, Jiaxin An, Yan Zhang, Besiki Stvilia · Journal of the Association for Information Science and Technology · 2024

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

10.87

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1002/asi.24978

Methodology & findings

Study design

Semistructured interview study with qualitative content analysis.

Sample

N = 25, 2 groups

Primary method

Qualitative content analysis using NVivo 14 software. Iterative coding process: initial codebook development based on prior research and interview guide, team coding of transcript subsets, consultation of audio recordings for unclear transcriptions, collective coding sessions to discuss codes and address disagreements, validation phase with team members reviewing code subsets, and multiple group meetings to consolidate codebook.

Main result

The study identified various types of academic tasks for which students used ChatGPT, including writing, programming, and learning. Six factors influencing students' motivation and ability to assess the credibility of GenAI-generated information were identified (e.g., task salience, social pressure). The study also identified "9 constructs (e.g., refinedness, explainability), 5 heuristics (e.g., inter- and intrasystem consistency heuristics), and 10 cues (e.g., version and tone) used by students to assess the credibility of GenAI-generated information."

Reports effect sizes.

Research paradigm

Qualitative constructivism

Author conclusions

The authors conclude: "Our study emphasizes the importance of ongoing efforts to understand people's information evaluation behaviors and the contexts that shape these behaviors as new technologies are incorporated into our information environment. Our findings also highlight a need for literacy programs for college students in higher education. Such programs have the potential to increase both motivation and ability by improving students' understanding of the fundamental working mechanisms of GenAI and relevant strategies for evaluating information credibility for different tasks and remain updated with reliable tools that can facilitate their credibility assessments of GenAI-generated information for academic tasks."

Risk of bias

Selection bias: Recruitment through university listservs, flyers, and snowball sampling may skew toward more engaged or technology-savvy students; Sample composition bias: 76% of participants aged 18-30, 68% from information science majors, predominantly from R1 institutions limits generalizability; Recall bias: Interview data relies on participants' self-reporting of past behaviors, which can be affected by memory errors; Interviewer bias: Four researchers conducted interviews independently with 5-8 participants each; potential inconsistency in interview administration; Temporal bias: Data collected April-June 2023, reflects only ChatGPT 3.5 experience, not current GenAI tools; Selection bias: Participants self-selected through recruitment methods (university listservs, flyers, snowball technique); Sampling bias: Predominantly from information technology-related majors (68%) at R1 institutions; Recall bias: Findings derived from self-reported interview data susceptible to memory biases; Temporal bias: Data collected April-June 2023 with ChatGPT 3.5, may not reflect current user behaviors with newer versions; Academic context bias: Limited to academic tasks, does not capture non-academic usage patterns; Selection bias: Participants were from R1 institutions and predominantly information technology majors, limiting generalizability; Recall bias: Study relied on participants' self-reporting of past behaviors and experiences with GenAI tools; Temporal limitation: Data collection occurred April-June 2023, capturing only ChatGPT 3.5 experiences, not current versions with source-linking features; Task scope bias: Limited to academic tasks only, may not capture behaviors in broader or non-academic settings; Researcher bias: Four independent researchers conducted interviews; potential for interviewer effects despite standardized guide

Limitations

The authors identified four key limitations: "First, students in the sample were from R1 institutions and predominantly from information technology-related majors, which limits the generalizability of findings to broader populations
Second, the tasks that students mentioned during the interviews were limited to academic tasks
Therefore, our findings regarding motivational factors may not fully capture students' behaviors in assessing GenAI credibility in broader or nonacademic settings
Third, the findings were derived from interview data that relied on participants' self-reporting, which can introduce inaccuracies or gaps in information due to memory biases or misunderstandings
Fourth, data were collected from April to June 2023 and mostly reflected students' experiences with a basic version of ChatGPT (3.5)."

Open questions raised

Limited understanding of how students evaluate credibility across diverse academic disciplines and non-R1 institutions
Need for research on broader, non-academic contexts for GenAI credibility assessment
Lack of understanding of students' credibility evaluation with newer GenAI versions (ChatGPT-4, Copilot) that offer source connections
Need for controlled experimental designs beyond interview methodology
Limited exploration across diverse demographics, majors, and institution types
Understanding of users' information behaviors when using GenAI tools, including when and how they evaluate system outputs, is limited. Future research should diversify samples across demographics, majors, and institution types to enhance generalizability. Exploring a wider range of GenAI tools, particularly newer versions with source-linking features, and employing varied research methodologies, including controlled experiments, will provide more comprehensive understanding. The opaque nature of GenAI algorithms and absence of traditional cues signifying information credibility need further investigation.

Extracted from: pdfAgreement 71%

Explore related topics

AI Manuscript Writing Academic Writing Qualitative