AI evidence extraction

Self-reflection enhances large language models towards substantial academic response

Baoxue Li · npj Artificial Intelligence · 2025

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

2.33

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1038/s44387-025-00045-3

Methodology & findings

Study design

Empirical validation study combining computational experiments with case study analysis.

Sample

N = 4259, 10 groups

Primary method

Quantitative evaluation methods: (1) LLM-as-a-Judge scoring (1-100 scale) for systematic response quality assessment; (2) Descriptive statistics: mean scores with standard deviation (SD) reported across article types and comment types; (3) Ranking analysis: average ranking comparisons across different LLM evaluators; (4) Good/Same/Bad (GSB) classification using DeepSeek-R1; (5) Error/failure mode frequency analysis (percentage distribution); (6) Variance comparison across methods. Software/tools: Chroma vector database, all-MiniLM-L6-v2 (BERT) embedding model, pypdf2 for PDF text conversion. No formal inferential statistical tests (t-tests, ANOVA, chi-square) reported; analysis is primarily descriptive and comparative.

Main result

The study found that the proposed RBB-LLM framework demonstrates superior performance for academic response letter writing. Specifically, "the proposed method has the highest score with the lowest variance on different article types and different comment types, which demonstrates the adaptability, effectiveness, and stability of the proposed method." The framework "improves by 9 to 10 points compared to DP and by more than 4 points compared to the second-best SR." Additionally, "the average performance of the proposed method across all article types exceeds the direct prompting by 10.05 points and outperforms the second-highest performing self-refine by 4.97 points."

Reports effect sizes.

Research paradigm

Empirical-computational (mixed methods with LLM-based evaluation and human validation)

Author conclusions

The authors conclude that "the proposed dual-loop reflection method successfully mitigates the shallow reasoning problem and improves the quality of response writing. Validation results on more than 3700 comments of 200 papers demonstrate the effectiveness and superiority of the proposed method on different article types and comment types." They further state that "LLM-assisted response letter writing is not a trivial task, but faces the shallow reasoning problem, which means that the LLM sometimes outputs responses that are polished in structure and style but fail to address the core of the comment." The framework is positioned as achieving AI-assisted response letter writing while "the vast amount of publicly available resources has the potential to be harnessed by LLMs, allowing the knowledge and experience contained within these resources to reach even peripheral researchers."

Risk of bias

Potential confirmation bias: both initial response and reflection generated by same LLM model; Evaluator bias: LLM evaluator may have preferences for polished but superficial content despite instructions to the contrary; Selection bias in validation set: 200 papers randomly selected from 4,259 articles; representativeness across domains acknowledged but potential imbalance; Training data bias: reflection bank constructed from Nature Communications only (2024 publications), may not generalize to other venues or time periods; Funding/development bias: framework developers also evaluated their own method using same LLM model; Self-bias risk in LLM evaluation: same model (GLM-4-Flash) used for reflection generation, response generation, and initial evaluation; Selection bias: only Nature Communications papers with publicly available peer reviews included; non-selectable text articles excluded; Confirmation bias: potential homogenization when same LLM generates both response and reflection; Evaluation bias: LLM-as-a-Judge may prefer polished but superficial content; memory reset between comments may be insufficient; Temporal bias: data limited to January-September 2024 from single publisher; Domain bias: validation set proportionally representative but limited to Nature Communications taxonomy; Language bias: study focused on English-language peer reviews; Model self-assessment bias: Same LLM used for generating initial responses and reflections may exhibit confirmation bias; Evaluator bias: Primary evaluation conducted using GLM-4-Flash, the same model used to construct the reflection bank, though mitigated by testing with additional evaluators; Selection bias in dataset: Only Nature Communications papers with publicly available peer review documents included; articles without accessible reviews or non-selectable text excluded; Domain representation bias: Training data comes only from Nature Communications (physical sciences, earth and environmental, biological sciences, health sciences, scientific community and society); may not generalize to other disciplines or venues; Single-round review bias: Only first-round reviewer comments included; later review rounds potentially more challenging; Matching algorithm artifacts: Text matching algorithm relies on specific patterns in peer review documents that may vary across journals

Limitations

The authors acknowledge several limitations: "First, in the construction of the reflection bank, we only applied the first round of review information, i.e., the first round of review comments followed by the authors' responses
In fact, some papers have multiple rounds of review, and the reviewers' satisfaction with the authors' responses in the first round is actually implied in the questions in the second round, which is a part of the information we did not utilize." Second, "searching for relevant quadruples from the reflection bank uses only reviewers' comments and the standard RAG technique, which can make the retrieved texts potentially not optimal." Third, "One promising direction for future work is using different LLMs for the extrospection process
In the current framework, both the initial response and the reflection are generated by the same model
This approach carries a potential risk of confirmation bias or homogenization, as the LLM might have its own inherent biases or tendencies in reasoning." Additionally, the authors note: "Nevertheless, we admit that even this method cannot fully match the detailed and context-aware judgment of human experts, which is a limitation of the present work."

Open questions raised

Extension to multi-round reviews: "In future work, we would extend our framework to model these dialogue-based reviewer-author chains...structuring the data as conversational threads (e.g., Comment 1 → Response 1 → Follow-up Comment 2 → Final Response 2)."
Cross-venue generalization: "it is promising to extend our collection to include peer-reviewed documents from different academic venues. For example, incorporating materials from computer science conference proceedings on OpenReview (e.g., ICLR, NeurIPS) would expose our model to different review styles and formats."
Cross-lingual expansion: "expanding the dataset to include cross-lingual peer reviews from other major publishers represents a significant next step."
Improved retrieval: "One direction that could be improved is to introduce LLM to rewrite or split the reviewer questions for parallel retrieval, which would improve the relevance of the retrieved information."
Advanced reasoning techniques: "Incorporating these more advanced CoT techniques or LLMs with reasoning capability, e.g., DeepSeek-R1, can further improve the quality of response writing."
Heterogeneous reflection: "using different LLMs for the extrospection process...a future enhancement could use a heterogeneous setup where a separate 'critic' LLM generates the reflection."

Data: Peer-reviewed documents from Nature Communications (https://www.nature.com/ncomms/), published January 1, 2024 to September 25, 2024. Total 4,259 articles collected; 4,059 used for reflection bank construction; 200 used for validation. Approximately 79,000 comments from peer review documents.; Peer review documents sourced from Nature Communications website (https://www.nature.com/ncomms/), covering 4,259 articles published January 1, 2024 to September 25, 2024; Reflection bank based on 4,059 articles (~79,000 comments) constructed by authors; Test set: 200 articles (3,700+ comments) for validation; Note: Original Nature Communications peer review documents are publicly available through the journal's website; authors' processed dataset availability not specified; Peer review documents from Nature Communications (January 1, 2024 to September 25, 2024): https://www.nature.com/ncomms/; 4,059 articles with peer review files used for reflection bank construction; 200 articles (3,700 comments) used for validation testingCode: Not explicitly mentioned in the paperExtracted from: pdfAgreement 56%

Explore related topics

AI Manuscript Writing Academic Writing Experimental