AI evidence extraction

Structure Liberates: How Constrained Sensemaking Produces More Novel Research Output

James Mooney, Zae Myung Kim, Young-Jun Lee, Dongyeop Kang · arXiv (Cornell University) · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Mixed methods: (1) Dataset construction using LLM-based trajectory generation from citation neighborhoods grounded in sensemaking framework; (2) Model distillation across multiple LLM backbones (3B-70B parameters); (3) Upstream evaluation using manual annotation (n=3 graduate-level annotators on 100 trajectories) with LLM-as-Judge validation; (4) Diversity measurement using multiple metrics (Self-BLEU, embedding similarity, BertScore, Sentence Mover's Distance); (5) Downstream evaluation with coding agents generating research artifacts on 5 manually curated citation sets..

Primary method

Design science research with sensemaking-grounded framework design

Main result

The study found that "Target-trained models achieve the strongest diversity under multiple automatic metrics and a 2.0% gain in aggregate plan quality over Infer-with no quality-diversity tradeoff: Target dominates Infer on both axes." Additionally, "Target trajectories improve the quality of generated scientific artifacts when conditioning coding agents, with particular gains in scientific grounding and improvements in executability and downstream utility."

Research paradigm

Positivist/empiricist with design science orientation

Author conclusions

The authors conclude that "models trained on constrained, targeted-reconstruction trajectories produce outputs that are more diverse and more novel than those trained explicitly for open-ended exploration-an advantage that propagates downstream into stronger execution success, tighter plan-artifact alignment, and higher judged quality." They further state: "Our results suggest that reconstructive trajectories achieve higher quality while enabling more diverse ideation, and that such structured, strong supervision enhances the creativity of downstream coding agents-highlighting the importance of principled, structured ideation and planning."

Risk of bias

Self-preference bias in downstream evaluation (partially mitigated by using Claude-Sonnet-4 judge from different model family); Selection bias in manually curated benchmark (only 5 citation sets that met computational feasibility criteria); Judge model bias (LLM-as-Judge may have systematic preferences that differ from human expert judgment); Limited human annotation sample (100 trajectories with 3 annotators); Filtering criteria for downstream evaluation may systematically exclude certain research directions; Small downstream benchmark (n=5 citation sets) may not generalize; Manual filtering introduces curator bias in artifact selection; LLM judge (Qwen3.5-35B) used for evaluation may have systematic biases; Teacher model (Qwen3-235B) used for trajectory generation may bias distilled outputs; Computational constraints in downstream execution (local artifacts only, no web search) may limit generalizability; Selection bias in downstream artifact evaluation: only 5 manually curated citation sets from computational domains that were feasible to execute; Potential self-preference bias in coding agent and paper writing model evaluation (both GPT family), though mitigated by using Claude Sonnet 4 as judge; LLM-as-Judge validation: inter-annotator agreement measured on only 100 samples across 4 conditions; Teacher model dependency: all trajectory generation used single LLM (Qwen3-235B-A22B-Instruct-2507); Confounding factor: Target trajectories have access to target paper summary during generation, while Infer do not

Limitations

The authors acknowledge that "such tools could be misused to generate plausible-sounding but unverified research proposals at scale
We encourage users to treat generated trajectories as starting points for human-guided research rather than as finished scientific contributions." Additionally, the downstream artifact evaluation was "intentionally small and manually curated, prioritizing feasibility and controlled comparison over scale" with only "5 unique citation sets" tested.

Open questions raised

The authors identify that 'a gap exists between the simplified design in current systems and the structured processes followed by human researchers.' They note that 'the actual upstream process used by human researchers is considerably more complex, consisting of multiple detailed steps that are more structurally organized.' Future work should treat 'ideation-not just execution-as a first-class target for evaluation and improvement.'
Gap between simplified design in current research agents and structured processes followed by human researchers
Need for principled, structured ideation and planning in LLM-driven research agents
Lack of controlled primitives to isolate and study ideation capabilities
Missing understanding of how upstream planning shapes scientific discovery
Gap between simplified design in current LLM research agents and structured processes followed by human researchers in ideation phase

Data: SCISENSE-Traj: 100K-scale dataset of sensemaking-based research trajectories (promised as released); Semantic Scholar Open Research Corpus (S2ORC) - used for corpus construction; SCISENSE-Traj: 100K-scale dataset of sensemaking-based research trajectories (to be released as open-source); S2ORC (Semantic Scholar Open Research Corpus) - publicly available source corpus; SCISENSE-Traj: 100K-scale dataset of sensemaking-based research trajectories (released by authors); Semantic Scholar Open Research Corpus (S2ORC) (Lo et al., 2020) - source corpus for citation neighborhoods; CelebA dataset (mentioned in case studies)Code: SCISENSE-LM weights and prompt templates promised as released (specific URL not provided in text); Anonymous repository mentioned: https://anonymous.4open.science/r/sciphi-prod-F6B6; SCISENSE-LM weights and all prompt templates (released as fully open-source, supplementary code repository referenced but URL not explicitly provided in text); SCISENSE-Traj, SCISENSE-LM weights, and prompt templates released as open-source (supplementary code repository mentioned but specific URL not provided in text)Extracted from: pdfAgreement 56%

Explore related topics

RAG for Research Literature Discovery Design Science