AI evidence extraction

OpenBioRQ: Unsolved Biomedical Research Questions for Agents

Minbyul Jeong · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Benchmark construction via multi-stage pipeline (crawl→extract→refine→dedup→export) followed by systematic agentic evaluation.

Main result

The study reveals that "biomedical agents almost never invent identifiers: Qwen3.6 reaches 99.6% existence and DeepSeek-V4 99.8%" at the existence level (L1), yet "roughly 15.9% link to the wrong paper" at the content-support level (L2). The paper shows "on this hardest subset, held-out models from the same lineage as the difficulty anchors solve only ∼17%, while three independent frontier agents (Gemini-3Pro, Opus-4.7, GPT-5.5) span a wide 29–60% range." Additionally, "on the hardest questions, agents abandon their tools, and for the most collapse-prone model, blocking tool access entirely barely changes its score—so tools stop paying off exactly where they are needed most."

Research paradigm

Empirical evaluation of AI systems (benchmark development and assessment)

Author conclusions

The author concludes: "OpenBioRQ targets: 12,553 unsolved biomedical research questions across 12 domains" and "it is the first benchmark in the agentic × unsolved regime." The paper demonstrates that "when the answer is fixed in advance, a model can pass by echoing the source it was handed, and these failures have no room to appear," thus "Reframing the task this way turns evaluation into a faithfulness-and-abstention probe." On the central finding: "Because a wrong-paper citation resolves, a reader is more inclined to trust it; it may therefore be more hazardous than an obvious fabrication—a severity gap I pose as a hypothesis." Finally: "The benchmark is thus hard, non-saturating (the best agent still leaves ∼33–40% unsolved), and discriminating across capability tiers."

Risk of bias

Roster selection bias: The frozen-core set is constructed by definition on failures of three specific open-weight models, creating circularity in difficulty assessment despite independent frontier-model validation; Judge-dependent L2 audit: The wrong-paper verdict relies on LLM judges (GLM-5.1 primary, Opus-4.7 secondary); disagreement exists (κ=0.755), and preliminary human spot-check shows low agreement (κ=0.29/0.51); Abstract-only evaluation: L2 citation support assessed on title+abstract only, not full-text, potentially underestimating true support rates (full-text re-judge yielded 16.1% vs abstract-only 15.9%); Track-dependent openness: Self-containment and resolution rates differ by source track (retrieval-verified 95% vs expert-consensus 75%), indicating potential source-specific biases; Temperature sensitivity: Empirical difficulty is decoding-sensitive; the core set changes substantially between T=0.3 (defining temperatures) and T=0 (final evaluation); Tool-use confound: No-tool ablation on GLM-5.1 shows overlapping CIs with tool condition (30.8% vs 26.6%), yet the most-collapse-prone model's results conflate agentic collapse with tool inefficacy; Selection bias: Core set defined on roster-model failures, potentially non-representative of true unsolved questions; Source bias: Four specialized tracks (retrieval verified, expert consensus, priority setting, expand) may oversample consensus-backed questions vs. genuinely disputed research frontiers; Judge bias: LLM-based citation assessment (L2 content support) shows inter-judge agreement of κ=0.755 (GLM-5.1 vs Opus-4.7), with preliminary human spot-check showing much lower κ=0.29/0.51; Temperature sensitivity: Empirical difficulty is decoding-sensitive; core bucket composition changed when measuring at T=0 vs T=0.3; Confirmation bias risk: Source-framing-only openness judgment collapsed to zero answered/unknown labels; only retrieval-grounded re-judgment detected genuine openness; Single-model pathology risk: Agentic collapse behavior is model-dependent and may not generalize across architectures; Membership/contamination bias: LLM-refined questions share median 3-word spans with sources (86.5% ≤5 verbatim words), but paraphrased exposure in pretraining cannot be excluded; Selection bias: Questions are derived from authoritative open-problem sources (JLA, NICE, WHO documents), not random sampling across biomedical literature; Roster-dependent difficulty: Difficulty buckets defined by performance of three specific open-weight models; may not generalize across model families; Judge bias: LLM-judge used for L2 content support (15.9% wrong-paper rate under primary GLM judge vs 10.6% under Opus judge); human-expert agreement lower (κ=0.29-0.51); Confirmation bias risk: Source-framing-only refinement initially produced zero answered labels; retrieval-grounded judgment required to mitigate; Contamination risk partially addressed: Verbatim-overlap check shows median 3 words shared with source; perplexity analysis single-model and may conflate memorization with complexity; Temperature sensitivity: Empirical difficulty shown to be decoding-sensitive; frozen-core definition required T=0 adjustment

Limitations

The authors state that "Narrative or guideline resolution is undetectable by this audit, and its recall is unquantified." They also note "The wrong-paper verdict the central result rests on is an LLM-cross-family proxy
agreement with human medical experts is deferred to future work." Additionally, "both checks target surface memorization (exact strings, perplexity) and would not detect paraphrased exposure of a question's underlying problem in pretraining
since my items are LLM-rephrasings of public sources this cannot be excluded." Finally, "human-expert agreement deferred" on checklist scoring and "preliminary non-expert human spot-check flags fewer, agreeing on the clear cases but only at binary κ=0.29/0.51 (vs the LLM-vs-LLM 0.755)."

Open questions raised

Human expert agreement on L2 (content-support) citations: "agreement with human medical experts is deferred to future work, and I release a human-annotation set to enable it"
Full-text L2 auditing: The paper uses title+abstract only; full-text evaluation is identified as future work
Mechanistic understanding of agentic collapse: The paper observes the phenomenon but does not provide causal analysis of why tool use stops on hard questions
Narrative and guideline resolution detection: "Narrative or guideline resolution is undetectable by this audit"—a recall limitation acknowledged as unquantified
Paraphrased memorization detection: The paper cannot exclude "paraphrased exposure of a question's underlying problem in pretraining" given that items are LLM rephrasings of public sources
Human-expert agreement on checklist-based grading (deferred to future work)

Data: OpenBioRQ: 12,553 unsolved biomedical research questions (mentioned as dataset; release status and URL not explicitly provided in excerpt but referenced as 'Dataset' in abstract header); OpenBioRQ corpus: 12,553 unsolved biomedical research questions across 12 domains (retrieval_verified: 6,648; expert_consensus: 5,905; priority_setting: 525; expand: 483); Gold-answer-bearing slice: 1,969 questions (mcp benchmark with gold.jsonl); Human-annotation set for future validation released; OpenBioRQ corpus (12,553 unsolved biomedical research questions across 12 domains); Human-annotation set for L2 content-support validation (released to enable future expert adjudication); 1,969-question gold-answer slice (mcp_benchmark_with_gold.jsonl)Code: Code repository mentioned in abstract header as 'Code' but specific URL not provided in excerpt; Code availability mentioned: "Dataset\nCode" listed in abstract header; Specific repository URLs not provided in paper text; Dataset and code repositories mentioned in paper header but specific URLs not provided in extracted textExtracted from: pdfAgreement 51%

Explore related topics

Citation Integrity Research Integrity Benchmarking