AI evidence extraction

KID-Review: Knowledge-Guided Scientific Review Generation with Oracle Pre-training

Weizhe Yuan, Pengfei Liu · Proceedings of the AAAI Conference on Artificial Intelligence · 2022

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.83

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1609/aaai.v36i10.21418

Methodology & findings

Study design

This is a design science study proposing KID-REVIEW, a knowledge-guided neural framework for scientific review generation.

Primary method

Design science research with iterative development. The framework incorporates existing architectural components (BART, GNNs, GAT) with novel integration strategies for knowledge graphs and oracle pre-training.

Main result

The study found that oracle pre-training significantly improves scientific review generation, with "the largest improvement 8.1 for Aspect Coverage and 4.69 for Aspect Recall respectively." Additionally, "using citation graph and concept graph can both achieve consistently higher Recommendation Accuracy than vanilla system without knowledge enhancement. The observed largest improvements are 7.66 and 4.46 for adding citation knowledge and concept knowledge, respectively. Besides, the combination of both knowledge can get an even higher Recommendation Accuracy boost, at most 11.88." The paper also demonstrates that "by adding a concept graph, the generated reviews are more likely to give specific examples and are more prone to ask questions," contributing to better constructiveness.

Research paradigm

Design science / Engineering

Author conclusions

The authors conclude that "This is the first work that neuralizes scientific review generation task with different types of knowledge" and that "Our work not only shows the complementarity between pre-trained knowledge (e.g., BART) and diverse types of knowledge graphs (e.g., citation graph) for scientific review generation, which could provide a reference for other generation tasks, but also presents how different types of knowledge play different roles." They emphasize that their oracle pre-training "ensures a smoothing training process" and provides better results on evaluation metrics.

Risk of bias

Human evaluation bias: Only one co-author annotated the 40 test papers (foot_2 acknowledges eight annotators total but specific human evaluation used single annotator); Selection bias: Test papers limited to CV/NLP domain from ICLR/NeurIPS conferences; Annotation bias: Pairwise ranking for constructiveness uses subjective judgment; Native language bias analyzed but potential confounding with paper quality; Human annotator bias (only co-authors used for 40-paper human evaluation); Domain bias (papers limited to CV/NLP domain for human evaluation); Language bias (native English speaker bias analyzed in Appendix); Training data bias (limited to ICLR 2017-2020 and NeurIPS 2016-2019); Human annotation bias: Only one co-author annotated generated reviews for the 40-paper human evaluation set; Native language bias: Analysis only examined bias regarding native English speaker authorship, not comprehensive demographic factors; Limited test set diversity: Human evaluation conducted only on CV/NLP papers, limiting generalizability; Information extraction bias: Concept graph construction limited to abstract section, potentially missing salient entities from full paper content

Limitations

The authors acknowledge that "our systems still suffer from constructiveness due to factuality errors." They further state that "While our system may be helpful in some cases, it is not a replacement for a skilled human reviewer
Completely relying on it will result in unfair reviews since, based on our observations, there are still many factually incorrect comments being generated." Additionally, the oracle pre-training approach is limited because "in practice, we found it challenging to find a satisfying local optimum when training the newly initialized GAT and pre-trained BART together when feeding non-oracle texts."

Open questions raised

More nuanced general world knowledge: "Our systems still suffer from constructiveness due to factuality errors. If a system can understand the more fine-grained relationships between papers (e.g., paper A is a combination of existing work B and C), then it can better judge the novelty of submission and give more constructive comments."
Text editing research integration: "We claim that editing text towards grammatically correct descriptions is crucial for a high-quality scientific review generation system"
Better content selection strategies: "it is still valuable to explore more effective content selection strategies when dealing with lengthy source input"
More nuanced general world knowledge - currently uses single citation embedding; need fine-grained relationships between papers (e.g., paper A combines existing work B and C) to better judge novelty
Connecting text editing research with scientific review generation - grammar error correction and clarity assessment needed for high-quality reviews
More effective content selection strategies when dealing with lengthy source input

Data: ASAP-Review dataset (Yuan, Liu, and Neubig 2021); S2ORC dataset (Lo et al. 2020) - used for citation graph construction; ASAP-Review dataset (introduced by Yuan, Liu, and Neubig 2021) - consists of ICLR papers from 2017-2020 and NeurIPS papers from 2016-2019 with aligned reviews; S2ORC dataset (Lo et al. 2020) - large corpus of 81.1M English-language academic papers; CNNDM dataset (Hermann et al. 2015) - used for BART pre-training; ASAP-Review dataset (Yuan, Liu, and Neubig 2021) - ICLR papers from 2017-2020 and NeurIPS papers from 2016-2019 with aligned reviews; S2ORC dataset (Lo et al. 2020) - 81.1M English-language academic papers used for citation graph constructionCode: The authors state "We release our systems and provide a demo service" but no specific GitHub/GitLab URLs provided in the paper; Authors state "We release our systems and provide a demo service" but no specific GitHub URL provided in paper; Authors state they "release our systems and provide a demo service" but specific GitHub/repository URLs are not provided in the paper textExtracted from: pdfAgreement 53%

Explore related topics

AI Peer Review Peer Review Design Science