AI evidence extraction

Harnessing generative AI to drive responsible business research and accelerate social impact

David S. Steingard, Dave Reibstein, Mark Normandin · Journal of Social Impact in Business Research · 2025

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1108/jsibr-11-2024-0051

Methodology & findings

Study design

Case study analysis of AI-assisted peer review implementation.

Sample

N = 322, 9 groups

Main result

The study found that when ChatSDG+RR7 recommended a "Pass," the editor-in-chief agreed in 84.52% of cases (131 of 155 submissions). When the AI recommended a "Cut," the editor concurred in 100% of cases (13 of 13). The overall editor-in-chief acceptance rate was 61.18% (197 out of 322 submissions). Additionally, "nearly one in eight papers endorsed by human reviewers would not have met the more rigorous, evidence-based thresholds of ChatSDG+RR7," indicating that ChatSDG+RR7 "does not merely replicate human judgment but provides a stricter and more consistent evaluation standard."

Reports effect sizes.

Research paradigm

Pragmatist/mixed-methods; combines positivist evaluation (quantitative metrics) with interpretive judgment (human expertise)

Author conclusions

The authors conclude that "Our findings indicate that GenAI, when thoughtfully integrated through a HIL model, enhances the quality of peer review by increasing consistency, reducing bias, speeding up the review process and aligning editorial decisions more closely with normative responsible research standards. These results suggest that AI-assisted peer review is not simply a technical augmentation but a transformative catalyst that advances both the rigor and societal relevance of business scholarship." They further state that their study "demonstrates that AI-assisted peer review enhances evaluation quality by increasing consistency, reducing bias and on top of that improving efficiency."

Risk of bias

Training data bias: AI model trained on previously accepted Honor Roll papers, which may themselves reflect historical biases in human editorial judgment; Input bias: Authors note that GenAI training datasets are 'often flawed because they are based on human judgment that is inherently inconsistent and biased'; Algorithmic bias: Potential disadvantage to underrepresented authors and research domains from biased AI models; Hallucination risk: GenAI system is 'prone to errors, biases and hallucinations' due to pattern-learning nature versus fixed algorithmic rules; Limited sample context: Analysis limited to RRBM Honor Roll submissions only; generalizability to other peer review contexts unclear; Selection bias in benchmark dataset: 141 previously accepted papers used for calibration may not represent rejected papers equally; Training data bias: ChatSDG+RR7 trained on human editorial decisions that may contain inconsistencies and biases; GenAI hallucination risk: prone to errors, biases and hallucinations unlike algorithmic AI; Underrepresentation bias: risk that bias within AI models can disadvantage underrepresented authors and research domains; Selection bias: evaluation limited to already-published articles presumed to meet minimum academic quality standards; Human reviewer bias: historical Honor Roll decisions used for benchmarking may reflect prior reviewer biases; Training data bias: ChatSDG+RR7 trained on human editorial decisions which may contain inconsistency and bias; AI hallucination risk: GenAI prone to errors, biases and hallucinations as acknowledged by authors; Selection bias: Only 322 Honor Roll submissions analyzed; limited generalizability; Journal bias: Analysis limited to journals represented in Honor Roll (99 unique journals, none predatory, but potentially non-representative); Potential algorithmic bias: May disadvantage underrepresented authors and research domains; Human rater bias: Editor-in-chief decisions subject to inconsistency and bias despite AI calibration

Limitations

The authors state: "Without sufficient governance, unsanctioned or poorly regulated applications can amplify bias, validate low-quality research or legitimize flawed methodologies—particularly in predatory journals that lack robust editorial standards." Additionally, they note that "Editors who depend too heavily on algorithmic outputs may overlook scholarly depth and complexity, while bias within AI models can disadvantage underrepresented authors and research domains." Furthermore, they acknowledge: "Although AI can achieve high levels of accuracy with large data sets, those data sets are often flawed because they are based on human judgment that is inherently inconsistent and biased
When bias is present, AI is only as reliable as its inputs."

Open questions raised

Further investigation into how AI-human collaboration transforms editorial roles, decision-making and accountability across academic publishing
Need for stronger ethical and regulatory frameworks to govern AI applications in peer review
Limited prior attention to how AI tools address challenges of assessing social impact and producing higher-impact scholarship
Few studies explicitly measuring societal impact; AI can play vital role in closing this gap
Prior sustainability forecasting studies rarely incorporate deep SDG contextual knowledge
Further research on maintaining epistemic legitimacy and ground truth in AI-assisted evaluation

Data: The paper references the RRBM Honor Roll dataset (364 awardees across 99 unique journals as of publication). The paper states the Honor Roll is an "open access global platform" but does not explicitly provide a URL or statement about data availability.Code: ChatSDG+RR7 is mentioned as a custom-built generative AI tool developed from prior work (Shinn, 2024) and integrated into the RRBM Honor Roll peer review process, but no code repository URL is provided.Extracted from: pdfAgreement 57%

Explore related topics

AI Peer Review Peer Review Experimental