AI evidence extraction

SLRMentor: An LLM-Based Tool Supporting Learning of SLR in Software Engineering

Rodolfo Gil-Pereira, Ronnie de Souza Santos, Cleyton Magalahes, Italo Santos · arXiv (Cornell University) · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Design Science research with pilot validation.

Sample

N = 4, 3 groups

Primary method

Descriptive summaries and qualitative interpretation rather than statistical inference. Reliability considered indirectly through participants' comparisons between tool-generated artifacts and manually produced counterparts, focusing on perceived methodological alignment, transparency, and support for critical reasoning.

Main result

The study found that "participants' interactions with SLRMentor indicate that the tool was mainly used to clarify concepts, revisit methodological choices, and reflect on planning decisions they had already made manually, rather than to simply follow procedural steps or adopt generated artifacts." Participants experienced SLRMentor "primarily as a learning-oriented support that assists with understanding the SLR process while still requiring active methodological judgment."

Reports effect sizes.

Research paradigm

pragmatist/design science

Author conclusions

"A conversational assistant can support learning by making methodological reasoning more explicit during planning activities and by enabling comparison between tool-supported guidance and researchers' own decisions." The authors conclude that "conversational assistants may complement, but do not replace, established approaches to teaching systematic literature reviews in software engineering" and that "SLRMentor can function as a complementary scaffold that supports understanding and orientation during SLR activities while still requiring active student judgment and refinement."

Risk of bias

Small sample size (n=4) limits generalizability; Self-selection bias: participation was voluntary; Lack of control group or comparison condition; Potential social desirability bias in post-hoc feedback; All participants were novice reviewers from a single graduate course; No random assignment; Descriptive analysis without statistical inference limits validity claims; Voluntary participation may introduce selection bias; Post-hoc evaluation after course completion and grading may affect responses; Only half of enrolled students chose to participate; Potential response bias due to participants' prior experience with manual SLR conduct; Small sample size (n=4); Self-selection bias (voluntary participation); Voluntary nature of participation after course completion may select for motivated students; Single educational setting (one graduate course); Retrospective evaluation after manual completion of SLR

Open questions raised

Need for further investigation of conversational assistants as supports for reflection, comparison, and sense-making
How such tools influence development of independent methodological judgment over time and across levels of research experience
Larger-scale validation studies with broader and more diverse student populations
Expansion of support to later stages of the SLR process (study selection, data extraction, quality assessment, synthesis)
Extension of the assistant to full review lifecycle to enable methodological reasoning beyond planning
Further investigation of conversational tools as supports for reflection, comparison, and sense making

Data: Validation data; SLRMentor validation dataCode: SLRMentor live versionExtracted from: pdfAgreement 64%

Explore related topics

Systematic Review Automation Literature Discovery Case Study