AI evidence extraction

SciSciGPT: advancing human–AI collaboration in the science of science

Erzhuo Shao, Yifang Wang, Yifan Qian, Zhenyu Pan, Han Liu, Dashun Wang · Nature Computational Science · 2025

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

17.43

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1038/s43588-025-00906-6

Methodology & findings

Study design

Mixed methods: (1) Design science research with artifact development and iterative refinement; (2) Exploratory pilot study comparing SciSciGPT performance with human researchers (n=3 domain experts at different career stages) on identical research tasks; (3) Expert review through semistructured interviews with SciSci experts; (4) Postdoctoral evaluation of outputs using five-point rating scale across five dimensions (effectiveness, technical soundness, analytical depth, visualization quality, documentation clarity); (5) Two detailed case studies demonstrating system functionality..

Primary method

Design Science Research (DSR) with iterative artifact development guided by capability maturity model; case study-based evaluation; expert review and pilot study

Main result

SciSciGPT successfully demonstrates that "SciSciGPT accelerated the research process, completing the same tasks in about 10% of the average time required by experienced researchers in the field" and that "when evaluating the quality of work, the three expert evaluators found that SciSciGPT's output was stronger than the human researchers' work across various dimensions we examined." The system automates complex workflows, supports diverse analytical approaches, accelerates research prototyping and iteration, and facilitates reproducibility through case studies demonstrating its ability to streamline empirical and analytical research tasks.

Research paradigm

Design science research with human-centered AI collaboration

Author conclusions

The authors conclude that "SciSciGPT offers a chat interface for public use that functions similarly to ChatGPT alongside a fully open-source implementation, ensuring transparency and enabling other researchers to reproduce and build on the work." They further state that "By combining these capabilities into a seamless, AI-powered research workflow, SciSciGPT has the potential to lower technical barriers and enhance efficiency, enabling a new mode of human–AI collaboration in SciSci." Importantly, they emphasize that "As AI capabilities continue to evolve, frameworks such as SciSciGPT may play increasingly pivotal roles in scientific research and discovery. At the same time, these new advances also raise critical challenges, from ensuring transparency and ethical use to balancing human and AI contributions."

Risk of bias

Small sample size (n=3) in pilot study limits generalizability; Participants in pilot study may have operated under time pressure, potentially not reflecting full capabilities; Selection bias: postdoctoral reviewers may not be representative of all SciSci researcher perspectives; Task completion constraints may have disadvantaged human participants relative to SciSciGPT; Funding source (Northwestern University affiliations) could introduce institutional bias; Comparison with human researchers using general-purpose LLM tools rather than pure human performance; Single domain focus (SciSci) limits generalizability claims about the framework's applicability across domains; Limited sample size (n=3 human researchers); Task completion constraints may not reflect human full capabilities; Participants allowed to use general-purpose AI tools (Claude 3.5, GPT-4o, ChatGPT-o1), making direct comparison unclear; Self-evaluation mechanism in EvaluationSpecialist may introduce systematic biases in quality assessment; Small sample size (3 human researchers) limits generalizability; Participant task completion constraints may not reflect actual research capabilities; Task completion time pressure may have disadvantaged human researchers; Evaluators were postdoctoral researchers who may have domain bias; Limited diversity in expertise levels among evaluated participants

Limitations

The authors acknowledge that "given the limited sample size, these results should be interpreted as exploratory rather than offering generalizable insights." Additionally, they note that "participants may have been operating under task completion constraints and that their results may not reflect their full capabilities, especially in unconstrained research settings with unlimited time for refinement." Furthermore, "SciSciGPT produced excessively detailed documentation" which "extended the evaluators' reading time and increased their cognitive load, potentially leading to a suboptimal user experience." The paper also acknowledges that while SciSciGPT is intended as a prototype, "its performance and value are expected to grow with the advancement of LLMs."

Open questions raised

Need for improvement in LLM reasoning and complex reasoning abilities
Requirement to address transparency and ethical use of AI in scientific research
Need to balance human and AI contributions in collaborative research
Gap in training next generation of scientists for AI-integrated research ecosystem
Extension of SciSciGPT to data-intensive domains beyond SciSci and disciplines traditionally less reliant on computational methods
Development toward higher maturity levels in the proposed LLM agent capability maturity model (particularly memory architecture and advanced human-AI collaborative paradigms)

Data: SciSciNet (original): https://doi.org/10.6084/m9.figshare.c.6076908.v1; SciSciNet (variant used in SciSciGPT): https://doi.org/10.57967/hf/6649; SciSciCorpus: https://doi.org/10.57967/hf/6650; SciSciNet: https://doi.org/10.6084/m9.figshare.c.6076908.v1; SciSciNet variant used in SciSciGPT: https://doi.org/10.57967/hf/6649; SciSciNet - available via Figshare at https://doi.org/10.6084/m9.figshare.c.6076908.v1; SciSciNet variant used in SciSciGPT - available via Hugging Face at https://doi.org/10.57967/hf/6649; SciSciCorpus - available via Hugging Face at https://doi.org/10.57967/hf/6650Code: Chat interface for public use: https://sciscigpt.com; GitHub open-source implementation: https://github.com/Northwestern-CSSI/SciSciGPT; Zenodo: https://doi.org/10.5281/zenodo.17271393; GitHub: https://github.com/Northwestern-CSSI/SciSciGPT; Web interface: https://sciscigpt.comExtracted from: pdfAgreement 42%

Explore related topics

AI-Assisted Discovery Human-AI Collaboration Case Study