AI evidence extraction

DESIGNING A HUMAN-AI COLLABORATIVE MODULAR APPROACH TO AUTOMATING SYSTEMATIC LITERATURE REVIEWS: FROM OBJECTIVES TO REPORTING

Md Aidul Islam · Trepo - Institutional Repository of Tampere University · 2025

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

10/10

Relevance

0/4

Quality (LMQS)

Evidence

Citations

Methodology & findings

Study design

Mixed-methods study combining artifact design (AI-assisted framework development) with user evaluation survey.

Sample

unknown

Primary method

The abstract mentions user surveys with Likert-scale ratings for usability metrics (ease of interaction, intuitiveness, responsiveness). Descriptive statistics (percentages for time savings, mean scores for usability) are reported, but detailed statistical analysis methods are not described in the abstract.

Main result

The framework substantially reduces the effort required for screening and data extraction. "83% of participants reported saving at least 25% of their time, and 37% reported saving over 50%, indicating a strong reduction in manual workload." Additionally, "Participants also rated the system highly in terms of usability and reliability, with average scores of 3.9/5 for ease of interaction, 3.8/5 for intuitiveness, and 3.6/5 for responsiveness."

Reports effect sizes.

Research paradigm

pragmatist/design science

Author conclusions

"This study contributes to advancing AI-driven research methodologies by providing a practical framework for automated and transparent evidence synthesis." The framework demonstrates that through "natural language understanding and context-driven reasoning, the framework ensures improved accuracy, efficiency, and reproducibility in literature synthesis."

Risk of bias

Not explicitly stated in the abstract; Selection bias in user survey sample (not specified how participants were recruited); Self-reported time savings may be subject to recall bias and social desirability bias; Survey design bias possible depending on how questions were framed; Limited information on participant expertise with SLR methodology; Selection bias: Participants were self-selected survey respondents, likely biased toward users willing to engage with the system; Response bias: Self-reported time savings and usability ratings are subjective and prone to social desirability bias; Lack of control group: No comparison to traditional SLR methods or baseline condition; Limited sample documentation: Abstract does not specify sample size or participant characteristics; Selection bias in survey participants; Self-reported time savings (no objective measurement); Small sample size for survey (not specified in abstract); Lack of control group comparison; Selection bias in user survey participants; Potential for positive response bias in subjective usability ratings; Selection bias: User survey participants may be self-selected (likely early adopters or motivated users); Self-report bias: Survey responses on time savings and usability are subjective and unverified; Demand characteristics: Participants may inflate positive ratings knowing they are evaluating a research system; Lack of control group: No comparison to traditional SLR methods or alternative systems; Small/unknown sample size: Number of survey participants not specified in abstract; Potential funding bias: Institutional affiliation may create incentive to report positive results

Open questions raised

Not explicitly stated in the abstract
The paper identifies the need for advancing AI-driven research methodologies and addresses the gap that "the conventional SLR process is time-consuming, labor-intensive, and susceptible to human bias."

Data: not_statedCode: not_statedExtracted from: pdfAgreement 57%

Explore related topics

Systematic Review Automation Literature Discovery Design Science