Towards LLM Accelerated Rapid Reviews for Software Tool Discovery -- Case for Log Anomaly Detection
Jesse Nyyssölä, Hamza Bin Mazhar, Alexander Bakhtin, Matteo Esposito, Nana Reinikainen, Yuqing Wang et al. · ArXiv.org · 2026
AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.
Methodology & findings
Study design
Multi-stage systematic pipeline combining: (1) broad keyword search on Scopus yielding 3233 papers, (2) LLM-based title-abstract screening using Gemini-2.5-Flash and GPT-4.1-Mini with probability thresholding at 0.90, (3) manual validation of stratified random sample (n=386) using Cochran's formula at 95% confidence level, (4) regex-based link extraction from downloaded PDFs, (5) manual assessment of extracted links for tool suitability, and (6) automated repository execution using Claude Code (Opus 4.6) agent with predefined action plan framework..
Primary method
Design Science Research with pragmatic engineering approach; iterative pipeline design with LLM component selection based on published benchmarks
Main result
The study demonstrates that LLM-accelerated rapid reviews can effectively identify software tools from academic literature. "From the 569 papers that passed the LLM screening we could download 470. They included 315 links that matched our regexes. There were 206 unique links and 109 duplicates." The abstract screening reached "precision of 0.933, recall of 0.909 and F1-score of 0.921," placing results "in the top 5% when compared against SESR-eval results." Ultimately, "we got 24 successfully running tools" out of 83 identified tools, with the pipeline requiring "only 4 hours of human effort, of which 3 hours were manual PDF downloading, and 12 hours of LLM running time."
Research paradigm
pragmatism
Author conclusions
The authors conclude: "This paper presents an LLM accelerated pipeline for conducting rapid reviews for software tool discovery. We tested our approach on software log anomaly detection tools but we could have chosen any other software engineering area where tools are a typical outcome." They further state: "To address RQ1, we argue that the use of LLMs improved both the quality and efficiency of the rapid review process for software tool discovery. For RQ2, we presented a list of 24 repositories that received an installation grade of 2 or 3 by the LLM. Although any qualitative assessment of the tools was outside the scope of this paper, the running tools offer a good starting point for anyone who wants to further assess them or integrate them into agentic frameworks such as MAPE-K."
Risk of bias
Selection bias: Search limited to Scopus academic database, excluding practitioner-built tools without academic papers; Internal validity threat: Inability to distinguish repository failures from LLM agent failures; Environmental bias: CPU-only execution environment excludes GPU-dependent tools; Single-author assessment: Manual tool relevance evaluation conducted by single author without inter-rater agreement; Timeout bias: 20-minute execution timeout may exclude legitimate tools requiring longer setup; Evidence loss: Tools distributed via Docker containers or cloud infrastructure not captured; Selection bias: Search string may not capture all relevant tools; tools absent from academic literature not identified; Hardware limitations: CPU-only environment excludes GPU-dependent tools; Timeout bias: 20-minute execution limit may exclude legitimately slower tools; Manual evaluation bias: Single author assessment of 206 unique links without inter-rater reliability checking; Dataset availability bias: Tools with unavailable or proprietary datasets excluded; Selection bias: Manual PDF downloading (3 hours) was infeasible for some papers (99 of 569), potentially introducing systematic bias against papers with restricted access; Operator bias: Single author conducted manual link-to-tool assessment, inconsistent with dual-reviewer approach used in abstract screening; Detection bias: LLM agent may systematically fail on repositories with poor documentation regardless of functional correctness, conflating repository quality with artifact quality; Hardware bias: CPU-only environment explicitly excludes GPU-dependent tools, creating systematic exclusion of deep learning-intensive approaches; Timeout bias: 20-minute execution limit caused 15 repositories to timeout; authors found 4 of 15 would have succeeded without time limit, suggesting systematic undercount; Language/framework bias: Pipeline designed primarily for Python with local dependencies, systematically excluding containerized, cloud-based, or non-Python tools; Publication bias: Restricting to academic literature misses practitioner-built tools (e.g., Angle Grinder, Logfile Navigator explicitly acknowledged as missing)
Limitations
- The authors acknowledge several limitations: "One notable threat to Internal validity is that it is not always clear whether a failure by the LLM runner is caused by the repository or the agent/prompt
- Hence, we risk conflating two different things: How good is the repository and how good is the LLM-agent." For external validity: "The most obvious threat for External validity is that the pipeline was only tested on log anomaly detection tools, and generalization to other SE tool areas is unconfirmed
- Another limiting factor is the CPU-only environment where we ran the experiments, and as such we know for certain that the results do not generalize for GPU environments." Additionally, "Another limiting factor is that our pipeline was designed around Python repositories with local dependencies
- This does not account for tools which may require Docker containers or cloud infrastructure."
Open questions raised
- Tools absent from academic literature (e.g., Angle Grinder, Logfile Navigator) not captured by literature search
- Generalizability beyond log anomaly detection tools to other software monitoring areas (metrics, tracing tools)
- More robust execution environments with broader OS support, GPU support, and dynamic timeout mechanisms
- Better methods to determine whether failures stem from repositories or LLM agent limitations
- Expansion beyond academic literature to include GitHub, PyPI, and other tool-hosting platforms
- Formalization of workflow as LLM Agent Skills for easier adoption
Explore related topics
Related papers
- PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and ExplanationAndrea C. Tricco · 2018 · 40,391 citations
- Rayyan—a web and mobile app for systematic reviewsMourad Ouzzani · 2016 · 24,664 citations
- Cochrane Handbook for Systematic Reviews of Interventions2019 · 14,420 citations
- Guidance for conducting systematic scoping reviewsMicah D.J. Peters · 2015 · 7,472 citations
- Updated methodological guidance for the conduct of scoping reviewsMicah D.J. Peters · 2020 · 6,688 citations
- Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resourcesMichael Gusenbauer · 2019 · 2,116 citations