AI evidence extraction

TiAb Review Plugin: A Browser-Based Tool for AI-Assisted Title and Abstract Screening

Yuki Kataoka, Masahiro Banno, Michihito Kyo, Shuri Nakao, Tomoo Sato, Shunsuke Taito et al. · arXiv (Cornell University) · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Mixed-method evaluation comprising: (1) ML equivalence verification using 10-fold cross-validation on six public datasets comparing TypeScript implementation against original Python ASReview implementation; (2) systematic LLM parameter tuning on depression benchmark dataset testing 16 parameter configurations across two model families; (3) cross-dataset LLM validation on five publicly available datasets with retrospective ground-truth labels..

Primary method

Design science research with iterative implementation and empirical validation

Main result

The study found that "The TypeScript classifier produced top-100 rankings 100 percent identical to the original ASReview across all six datasets" and for LLM screening, "recall was 94 to 100 percent with precision of 2 to 15 percent, and Work Saved over Sampling at 95 percent recall (WSS@95) ranged from 48.7 to 87.3 percent." The tool successfully integrates LLM screening and ML active learning into a browser-based, no-code environment.

Research paradigm

Design science / artifact-driven

Author conclusions

"We developed a functional browser extension that integrates LLM screening and ML active learning into a no-code, serverless environment, ready for practical use in systematic review screening." The authors also state that the tool "provides LLM-assisted T&A screening directly within the researcher's existing web workflow" and "operates entirely within the browser, requiring no software installation, server infrastructure, or programming skills."

Risk of bias

Retrospective evaluation only with known ground-truth labels; Single LLM model family tested (Google Gemini); Benchmark datasets concentrated in critical care medicine; Implementation equivalence verified only for top-100 records; no prospective validation; generalizability to other models unknown; No formal ablation study on sensitivity-prioritization prompt instruction; Funding from JSPS and pharmaceutical company speaker honoraria (potential bias on LLM recommendations); ML evaluation limited to top-100 ranked records; lower-ranked divergence not assessed; generalizability to other domains uncertain; Fixed inclusion threshold (0.5) applied across all datasets; threshold optimization per dataset not explored

Limitations

"Regarding ML evaluation, we verified implementation equivalence only for the top-100 ranked records
divergence in lower-ranked records, while unlikely given identical algorithms, was not assessed
Regarding LLM evaluation, we tested only one model family (Google Gemini)
performance may differ with other LLMs." Additionally, "All evaluations in this study are retrospective, using datasets with known ground-truth labels
No prospective study has yet been conducted to measure the tool's impact on screening efficiency, reviewer time, or error rates in a live systematic review project."

Open questions raised

Prospective validation study needed to measure tool's impact on screening efficiency, time savings, and error rates in live systematic review projects
Support for additional LLM providers (e.g., OpenAI, Anthropic, Alibaba) to increase flexibility and reduce dependence on single API
Formal usability testing using standardized instruments such as the System Usability Scale (SUS) needed to provide evidence on tool's ease of use across different user populations
Evaluation of the tool with multiple LLM models beyond Google Gemini

Data: ASReview benchmark datasets (6 datasets for ML equivalence testing); Depression benchmark dataset from Oami et al. (1,993 records; 280 relevant; prevalence 14.1%); Five public datasets (CQ1-CQ5) derived from Oami et al. sepsis-related clinical practice guideline systematic reviews (1,038 to 5,628 records, 0.5 to 2.0 percent prevalence); All datasets publicly available; fold generation script available at experiments/asreview/make_folds.pyCode: https://github.com/youkiti/tiab-review-plugin; Fold generation script: experiments/asreview/make_folds.py; Prompt template: src/lib/prompt-templates.ts; Chrome Web Store: https://chromewebstore.google.com/detail/tiab-reviewplugin/alejlnlfflogpnabpbplmnojgoeeabijExtracted from: pdf

Explore related topics

Systematic Review Automation Literature Discovery Design Science