Skip to content

Systematic-Literature-Reviews-with-Artificial-Intelligence

Resources and tools for conducting Systematic Literature Reviews using Artificial Intelligence

View on GitHub

Publications

Publication 1

Property Value
DOI 10.1101/2025.06.13.25329541v1))

Publication 2

Property Value
DOI 10.1038/s42256-020-00287-7))

Publication 3

Property Value
DOI 10.1186/s13643-019-1074-9))

Publication 4

Property Value
DOI 10.1002/jrsm.1715))

Can large language models replace humans in systematic reviews? Evaluating GPT‐4's efficacy in screening and extracting data from peer‐reviewed and grey literature in multiple languages

AbstractSystematic reviews are vital for guiding practice, research and policy, although they are often slow and labour‐intensive. Large language models (LLMs) could speed up and automate systematic reviews, but their performance in such tasks has yet to be comprehen...

Property Value
DOI 10.1002/jrsm.1715

| Year | 2024 | | Venue | Research Synthesis Methods | | Citations | 170 |

Authors: Qusai Khraisha, Sophie Put, Johanna Kappenberg, Azza Warraitch, Kristin Hadfield

Rayyan—a web and mobile app for systematic reviews

Property Value
DOI 10.1186/s13643-016-0384-4

| Year | 2016 | | Venue | Systematic Reviews | | Citations | 18782 |

Authors: Mourad Ouzzani, Hossam Hammady, Zbys Fedorowicz, Ahmed Elmagarmid

Publication 7

Property Value
DOI 10.1186/s13643-016-0384-4))

Publication 8

Property Value
DOI 10.1038/s41591-022-02139-w))

An open source machine learning framework for efficient and transparent systematic reviews

AbstractTo help researchers conduct a systematic review or meta-analysis as efficiently and transparently as possible, we designed a tool to accelerate the step of screening titles and abstracts. For many tasks—including but not limited to systematic reviews and meta...

Property Value
DOI 10.1038/s42256-020-00287-7

| Year | 2021 | | Venue | Nature Machine Intelligence | | Citations | 836 |

Authors: Rens van de Schoot, Jonathan de Bruin, Raoul Schram, Parisa Zahedi, Jan de Boer, Felix Weijdema, Bianca Kramer, Martijn Huijts, Maarten Hoogerwerf, Gerbrich Ferdinands, Albert Harkema, Joukje Willemsen, Yongchao Ma, Qixiang Fang, Sybren Hindriks, Lars Tummers, Daniel L. Oberski

PRISMA AI reporting guidelines for systematic reviews and meta-analyses on AI in healthcare

Property Value
DOI 10.1038/s41591-022-02139-w

| Year | 2023 | | Venue | Nature Medicine | | Citations | 111 |

Authors: Giovanni E. Cacciamani, Timothy N. Chu, Daniel I. Sanford, Andre Abreu, Vinay Duddalwar, Assad Oberai, C.-C. Jay Kuo, Xiaoxuan Liu, Alastair K. Denniston, Baptiste Vasey, Peter McCulloch, Robert F. Wolff, Sue Mallett, John Mongan, Charles E. Kahn, Viknesh Sounderajah, Ara Darzi, Philipp Dahm, Karel G. M. Moons, Eric Topol, Gary S. Collins, David Moher, Inderbir S. Gill, Andrew J. Hung

Toward systematic review automation: a practical guide to using machine learning tools in research synthesis

Property Value
DOI 10.1186/s13643-019-1074-9

| Year | 2019 | | Venue | Systematic Reviews | | Citations | 433 |

Authors: Iain J. Marshall, Byron C. Wallace

Information

Property Value
Language Python
Stars 0
Forks 0
Watchers 0
Open Issues 0
License MIT License
Created 2025-12-02
Last Updated 2026-02-19
Last Push 2025-12-19
Contributors 1
Default Branch main
Visibility private

Datasets

This repository includes 4 dataset(s):

Dataset Format Size

| data | | 0.0 KB |

| benchmarks.json | .json | 1.43 KB |

| papers.json | .json | 2.78 KB |

| tools.json | .json | 2.47 KB |

Reproducibility

No specific reproducibility files found.

Status

  • Issues: Enabled
  • Wiki: Disabled
  • Pages: Enabled

README

Systematic Literature Reviews with Artificial Intelligence

A comprehensive collection of resources, tools, and research for conducting Systematic Literature Reviews (SLRs) using Artificial Intelligence and Large Language Models.

Table of Contents

  1. Overview
  2. AI Tools for Systematic Reviews
  3. Key Research Papers
  4. Methodological Guidelines
  5. Performance Benchmarks
  6. Getting Started
  7. Resources in This Repository

Overview

Systematic reviews constitute a critical foundation for evidence-based decision-making across disciplines. However, the labor-intensive nature of traditional SLRs - requiring weeks to months of manual work - has driven significant interest in AI-assisted automation.

Key Statistics: - AI screening tools can reduce workload by 40-95% - otto-SR reproduced 12 Cochrane reviews in 2 days (equivalent to ~12 work-years manually) - GPT-4 achieves median accuracy >85% for PICO element extraction

AI Tools for Systematic Reviews

Open Source Tools

Tool Description Key Features Link
ASReview Active learning for systematic reviews Open-source, 95% workload reduction, Python-based asreview.nl
RobotReviewer ML system for RCT assessment Free, web-based, bias assessment robotreviewer.net
Colandr Open-source screening tool Free, collaborative colandrapp.com
FAST2 Active learning screening Open source GitHub

Commercial/Freemium Tools

Tool Description Pricing Link
Rayyan AI-powered review management Free tier available rayyan.ai
Elicit AI research assistant Free: basic / Pro: $42/mo elicit.com
Covidence Cochrane-recommended tool Free for Cochrane reviews covidence.org
DistillerSR Enterprise review software Subscription-based distillersr.com
Laser AI Living systematic reviews Commercial laser.ai
otto-SR End-to-end LLM workflow Web platform ottosr.com
EPPI-Reviewer Comprehensive review tool Subscription eppi.ioe.ac.uk

Specialized LLM Applications

Tool/Method Application Model
Systematic Review Extractor Pro Data extraction Custom GPT
otto-SR Screening Agent Abstract/full-text screening GPT-4.1
otto-SR Extraction Agent Data extraction o3-mini-high

Key Research Papers

Foundational Papers

  1. ASReview Framework (2021)
  2. van de Schoot, R. et al. "An open source machine learning framework for efficient and transparent systematic reviews"
  3. Nature Machine Intelligence 3, 125-133

  4. DOI: 10.1038/s42256-020-00287-7

  5. Rayyan Original Paper (2016)

  6. Ouzzani, M. et al. "Rayyan - a web and mobile app for systematic reviews"
  7. Systematic Reviews 5, 210
  8. DOI: 10.1186/s13643-016-0384-4

Recent LLM Research (2024-2025)

  1. otto-SR: Automation of Systematic Reviews with LLMs (2025)
  2. Demonstrated 96.7% sensitivity, 97.9% specificity in screening

  3. Manuscript PDF | medRxiv

  4. LLMs for Systematic Reviews: Scoping Review (2025)

  5. "Large language models for conducting systematic reviews: on the rise, but not yet ready for use"
  6. Journal of Clinical Epidemiology

  7. ScienceDirect

  8. GPT-4 Evaluation for SLR (2024)

  9. Khraisha, Q. et al. "Can large language models replace humans in systematic reviews?"
  10. Research Synthesis Methods

  11. DOI: 10.1002/jrsm.1715

  12. LLM-Assisted SLR System (2025)

  13. "Enhancing systematic literature reviews with generative AI"
  14. JAMIA 32(4), 616
  15. Oxford Academic

Methodology & Guidelines

  1. PRISMA-AI Guidelines (2023)
  2. "PRISMA AI reporting guidelines for systematic reviews and meta-analyses on AI in healthcare"
  3. Nature Medicine

  4. DOI: 10.1038/s41591-022-02139-w

  5. Practical Guide to ML in Research Synthesis (2019)

  6. "Toward systematic review automation: a practical guide"
  7. Systematic Reviews
  8. DOI: 10.1186/s13643-019-1074-9

Methodological Guidelines

PRISMA-AI Framework

The PRISMA-AI extension provides standardized reporting for AI-related systematic reviews: - Search strategy documentation - Quality assessment with AI-specific criteria - Transparent result reporting - Technical reproducibility requirements

LLM Integration Guidelines

When integrating LLMs into systematic reviews:

  1. Screening Phase
  2. Use zero-shot or few-shot classification
  3. Define clear inclusion/exclusion criteria in prompts

  4. Maintain human oversight for borderline cases

  5. Data Extraction

  6. Use structured prompts (RISEN framework)
  7. Validate extracted data against source documents

  8. Document prompt versions for reproducibility

  9. Quality Assurance

  10. Dual verification (AI + human) recommended
  11. Report sensitivity and specificity metrics
  12. Document AI model versions and parameters

Performance Benchmarks

Screening Accuracy

Tool/Method Sensitivity Specificity Notes
otto-SR 96.7% 97.9% GPT-4.1 based
Human dual review 81.7% 98.1% Traditional approach
Rayyan AI 97-99% 19-58% At <2.5 threshold
ASReview Variable Variable Depends on dataset

Data Extraction

Model Precision Recall Notes
GPT-based (pooled) 83.0% 86.0% Mean across studies
BERT-based Lower Lower Compared to GPT
otto-SR extraction 93.1% accuracy - o3-mini-high

Time Savings

Stage Traditional AI-Assisted Reduction
Screening 8-12 weeks 2-3 weeks ~75%
Data extraction 10-16 weeks 3-5 weeks ~70%
Per-paper extraction 36 min 27 sec + 13 min review ~60%

Getting Started

For Beginners

  1. Start with Rayyan - Free tier, user-friendly interface
  2. Try ASReview - Open source, well-documented
  3. Read the PRISMA guidelines - Understand methodological requirements

For Advanced Users

  1. Explore otto-SR - State-of-the-art LLM automation
  2. Build custom GPT extractors - Use RISEN framework
  3. Combine tools - ASReview for screening + ChatGPT for extraction

Python Implementation

# Install ASReview
pip install asreview

# Basic usage
from asreview import ASReviewProject

# See ASReview documentation: https://asreview.readthedocs.io/

Resources in This Repository

Papers Directory (/papers)

File Description
otto-SR_manuscript.pdf Full otto-SR methodology paper
otto-SR_full_paper.pdf medRxiv preprint
LLM_systematic_reviews_scoping.pdf Scoping review of LLMs in SLRs
ASReview_paper_info.txt Citation info for ASReview paper

Scripts

File Description
create_repo.py GitHub repository creation script
download_resources.py Resource download automation

Key GitHub Repositories

Additional Resources

Library Guides

Tutorials & Blogs

Citation

If you use resources from this repository, please cite the original sources appropriately.

Contributing

Contributions are welcome! Please submit issues or pull requests for: - New tools or resources - Updated benchmarks - Bug fixes or corrections

License

This repository is for educational and research purposes. Individual papers and tools may have their own licenses.

Last updated: December 2025

Description

Resources and tools for conducting Systematic Literature Reviews using Artificial Intelligence

Installation

git clone https://github.com/Digital-AI-Finance/Systematic-Literature-Reviews-with-Artificial-Intelligence.git
cd Systematic-Literature-Reviews-with-Artificial-Intelligence
pip install -r requirements.txt

Usage

See the repository contents for usage examples.

(c) Joerg Osterrieder 2025