AI evidence extraction

TiAb Review Plugin: A Browser-Based Tool for AI-Assisted Title and Abstract Screening

Yuki Kataoka, Masahiro Banno, Michihito Kyo, Shuri Nakao, Tomoo Sato, Shunsuke Taito et al. · arXiv (Cornell University) · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Mixed-method evaluation comprising: (1) ML equivalence verification using 10-fold cross-validation on six public datasets comparing TypeScript implementation against original Python ASReview implementation; (2) systematic LLM parameter tuning on depression benchmark dataset testing 16 parameter configurations across two model families; (3) cross-dataset LLM validation on five publicly available datasets with retrospective ground-truth labels..

Primary method

Design science research with iterative implementation and empirical validation

Main result

The study found that "The TypeScript classifier produced top-100 rankings 100 percent identical to the original ASReview across all six datasets" and for LLM screening, "recall was 94 to 100 percent with precision of 2 to 15 percent, and Work Saved over Sampling at 95 percent recall (WSS@95) ranged from 48.7 to 87.3 percent." The tool successfully integrates LLM screening and ML active learning into a browser-based, no-code environment.

Research paradigm

Design science / artifact-driven

Author conclusions

"We developed a functional browser extension that integrates LLM screening and ML active learning into a no-code, serverless environment, ready for practical use in systematic review screening." The authors also state that the tool "provides LLM-assisted T&A screening directly within the researcher's existing web workflow" and "operates entirely within the browser, requiring no software installation, server infrastructure, or programming skills."

Risk of bias

Retrospective evaluation only with known ground-truth labels; Single LLM model family tested (Google Gemini); Benchmark datasets concentrated in critical care medicine; Implementation equivalence verified only for top-100 records; Retrospective evaluation only with known ground-truth labels; no prospective validation; Single LLM vendor (Google Gemini) tested; generalizability to other models unknown; Evaluation limited to critical care medicine datasets with well-defined eligibility criteria; ML equivalence tested only on top-100 ranked records, not full ranking lists; No formal ablation study on sensitivity-prioritization prompt instruction; Funding from JSPS and pharmaceutical company speaker honoraria (potential bias on LLM recommendations); Retrospective evaluation only—no prospective validation in live systematic reviews; Limited to single LLM provider (Google Gemini); generalizability to other LLMs unknown; ML evaluation limited to top-100 ranked records; lower-ranked divergence not assessed; Benchmark datasets concentrated in critical care medicine; generalizability to other domains uncertain; Fixed inclusion threshold (0.5) applied across all datasets; threshold optimization per dataset not explored

Limitations

"Regarding ML evaluation, we verified implementation equivalence only for the top-100 ranked records
divergence in lower-ranked records, while unlikely given identical algorithms, was not assessed
Regarding LLM evaluation, we tested only one model family (Google Gemini)
performance may differ with other LLMs." Additionally, "All evaluations in this study are retrospective, using datasets with known ground-truth labels
No prospective study has yet been conducted to measure the tool's impact on screening efficiency, reviewer time, or error rates in a live systematic review project."

Open questions raised

Prospective validation study needed to measure tool's impact on screening efficiency, time savings, and error rates in live systematic review projects
Support for additional LLM providers (e.g., OpenAI, Anthropic, Alibaba) to increase flexibility and reduce dependence on single API
Formal usability testing using standardized instruments such as the System Usability Scale (SUS) needed to provide evidence on tool's ease of use across different user populations
Prospective validation study needed to measure tool impact on screening efficiency, time savings, and error rates in live systematic review projects
Support for additional LLM providers (OpenAI, Anthropic, Alibaba) to increase flexibility and reduce API vendor lock-in
Formal usability testing using standardized instruments such as the System Usability Scale (SUS) to assess ease of use across different user populations

Data: ASReview benchmark datasets (6 datasets for ML equivalence testing); Depression benchmark dataset from Oami et al. (1,993 records; 280 relevant; prevalence 14.1%); Five public datasets (CQ1-CQ5) derived from Oami et al. sepsis-related clinical practice guideline systematic reviews (1,038 to 5,628 records, 0.5 to 2.0 percent prevalence); Depression benchmark dataset (1,993 records; 280 relevant; prevalence 14.1%) - source: Bannach-Brown et al., Zenodo; Five sepsis-related clinical practice guideline datasets (CQ1-CQ5) - source: Oami et al. 2024, JAMA Network Open; Six ASReview benchmark datasets used for ML equivalence verification - available via ASReview project; Depression benchmark dataset (1,993 records; 280 relevant; 14.1% prevalence); Five datasets (CQ1-CQ5) derived from Oami et al.: sepsis-related clinical practice guideline systematic reviews (1,038–5,628 records; 0.5–2.0% prevalence); All datasets publicly available; fold generation script available at experiments/asreview/make_folds.pyCode: https://github.com/youkiti/tiab-review-plugin; https://github.com/youkiti/tiabreview-plugin; https://github.com/youkiti/tiab-review-plugin (source code, documentation, experimental data under MIT License); Fold generation script: experiments/asreview/make_folds.py; Prompt template: src/lib/prompt-templates.ts; https://github.com/youkiti/tiab-review-plugin (source code, documentation, experimental data); Chrome Web Store: https://chromewebstore.google.com/detail/tiab-reviewplugin/alejlnlfflogpnabpbplmnojgoeeabijExtracted from: pdfAgreement 68%

Explore related topics

Systematic Review Automation Literature Discovery Design Science