AI evidence extraction

AI literacy and its implications for prompt engineering strategies

Nils Knoth, Antonia Tolzin, Andreas Janson, Jan Marco Leimeister · Computers and Education Artificial Intelligence · 2024

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

7/10

Relevance

0/4

Quality (LMQS)

Evidence

277

Citations

82.83

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1016/j.caeai.2024.100225

Methodology & findings

Study design

Mixed methods study combining quantitative assessment of prompt engineering skills and LLM output quality with qualitative analysis of students' intuitive behaviors towards LLM-based AI systems

Main result

The study found that "higher-quality prompt engineering skills predict the quality of LLM output, suggesting that prompt engineering is indeed a required skill for the goal-directed use of generative AI tools." Additionally, "certain aspects of AI literacy can play a role in higher quality prompt engineering and targeted adaptation of LLMs within education."

Reports effect sizes.

Research paradigm

Mixed methods (quantitative and qualitative)

Author conclusions

The authors conclude: "We, therefore, argue for the integration of AI educational content into current curricula to enable a hybrid intelligent society in which students can effectively use generative AI tools such as ChatGPT." This conclusion is based on findings that "prompt engineering is indeed a required skill for the goal-directed use of generative AI tools."

Risk of bias

Not explicitly stated in abstract; Selection bias: Study participants were students from higher education context; Potential confounders: Prior experience with AI systems, technical background, language proficiency; Selection bias: Sample appears to be students from higher education, which may not represent general population; Potential confounders: Prior technology experience, domain knowledge, and individual differences in problem-solving approaches not fully characterized in abstract; Measurement bias: Quality assessment of prompts and LLM outputs depends on evaluator judgment criteria

Limitations

The paper states that "research on the perspectives of non-experts using LLM-based AI systems through prompt engineering and on how AI literacy affects prompting behavior is lacking," indicating a significant research gap that this study addresses but may not fully resolve given the novelty of the domain.

Open questions raised

The paper identifies the lack of research on perspectives of non-experts using LLM-based AI systems through prompt engineering and how AI literacy affects prompting behavior, particularly in higher education contexts. Future research directions focus on AI literacy curriculum integration.
The authors identify that "research on the perspectives of non-experts using LLM-based AI systems through prompt engineering and on how AI literacy affects prompting behavior is lacking," particularly "when considering the implications of LLMs in the context of higher education."
The authors identify that "research on the perspectives of non-experts using LLM-based AI systems through prompt engineering and on how AI literacy affects prompting behavior is lacking," particularly in the context of higher education.

Data: not_statedCode: not_statedExtracted from: pdfAgreement 66%

Explore related topics

AI Literacy Training Doctoral Training Empirical Study