Cierco Jimenez et al. BMC Medical Research Methodology (2022) 22:322 https://doi.org/10.1186/s12874-022-01805-4 RESEARCH Open Access Machine learning computational tools to assist the performance of systematic reviews: A mapping review Ramon Cierco Jimenez1,2*, Teresa Lee3, Nicolás Rosillo4, Reynalda Cordova5,6, Ian A Cree1, Angel Gonzalez2 and Blanca Iciar Indave Ruiz1 Abstract Background: Within evidence-based practice (EBP), systematic reviews (SR) are considered the highest level of evidence in that they summarize the best available research and describe the progress in a determined field. Due its methodology, SR require significant time and resources to be performed; they also require repetitive steps that may introduce biases and human errors. Machine learning (ML) algorithms therefore present a promising alternative and a potential game changer to speed up and automate the SR process. This review aims to map the current availability of computational tools that use ML techniques to assist in the performance of SR, and to support authors in the selec- tion of the right software for the performance of evidence synthesis. Methods: The mapping review was based on comprehensive searches in electronic databases and software reposi- tories to obtain relevant literature and records, followed by screening for eligibility based on titles, abstracts, and full text by two reviewers. The data extraction consisted of listing and extracting the name and basic characteristics of the included tools, for example a tool’s applicability to the various SR stages, pricing options, open-source availability, and type of software. These tools were classified and graphically represented to facilitate the description of our findings. Results: A total of 9653 studies and 585 records were obtained from the structured searches performed on selected bibliometric databases and software repositories respectively. After screening, a total of 119 descriptions from publi- cations and records allowed us to identify 63 tools that assist the SR process using ML techniques. Conclusions: This review provides a high-quality map of currently available ML software to assist the performance of SR. ML algorithms are arguably one of the best techniques at present for the automation of SR. The most promising tools were easily accessible and included a high number of user-friendly features permitting the automation of SR and other kinds of evidence synthesis reviews. Keywords: Systematic reviews, Mapping review, Evidence-based practice, Software development, Machine learning, Automatization Background Evidence-based practice (EBP) establishes a rigorous approach to gathering and summarising the best avail- able evidence within a specific field or research purpose *Correspondence: ciercor@iarc.who.int [1–3]. This paradigm has significantly changed the dis- 1 International Agency for Research on Cancer (IARC/WHO), Evidence courses and practices in various fields such as biomedi- Synthesis and Classification Branch, Lyon, France cal sciences, education, medicine, psychology, and public Full list of author information is available at the end of the article © The Author(s) 2022. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://​creat​iveco​mmons.​org/​licen​ses/​by/4.​0/. The Creative Commons Public Domain Dedication waiver (http://​creat​iveco​ mmons.​org/​publi​cdoma​in/​zero/1.​0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. Cierco Jimenez et al. BMC Medical Research Methodology (2022) 22:322 Page 2 of 14 policy [3–7]. Evidence-based medicine (EBM) developed is a need for computational assistance. The WCT and these principles to identify and evaluate medical infor- EBP in general would benefit considerably from compu- mation and provide structured summaries of the avail- tational assistance to perform SR [3, 36] but tools that able evidence to inform decision in health care and adapt well to the particularities of the fields of pathology improve the diagnosis and treatment of patients [1, 3, 8]. and cancer diagnosis are not available. Systematic reviews (SR) are evidence synthesis studies The number of software tools and workflows to sup- that follow a structured method and are considered the port the performance of systematic reviews, systematic most reliable source of evidence in the hierarchy of lev- maps, and meta-analyses is growing rapidly [29, 37–39]. els of evidence [9–11]. A SR aims to select, identify, criti- Use of natural language processing (NLP) and machine cally appraise, and synthesise the best available evidence learning (ML) algorithms to reduce time and workload within pre-specified eligibility criteria to answer a clearly in the SR process is becoming increasingly popular [29, defined research question [9, 12-14]. This practice allows 40, 41]. However, despite significant progress, integra- the consolidation of large amounts of findings from pub- tion of high-quality methodological approaches with lications and the identification of potential evidence gaps user-friendly applications is rare. Well adapted open- in a specific field. Without SR, decision-making processes source software is also rare, and integration among the are vulnerable to bias and would be often only based on different software tools is poor. A vast number of free a subset of studies that may not be representative of the and fee-based tools exist, but there is a lack of validation knowledge base of the field. In addition, information and consensus when it comes to identifying which tool overload due to increasing number of scientific publica- best fits specific needs. This limits the utility of computa- tions, publication bias and heterogeneity of reporting, are tional tools, being especially difficult to find solutions to challenges faced during decision-making process. These assist specific steps of the SR. At present day some of the raise the risk of obtaining biased results and flawed con- computational tools (e,g., web-applications, algorithms, clusions, and accurate evidence synthesis are key in many executables, etc.) assisting the SR process are shown in fields to informing the decision-making process. Promot- Fig. 1, they can be found in online catalogues/reposito- ing, enhancing, and facilitating the production of SR is ries like the SR toolbox [42]. Around 160 tools assist the therefore vital in the use of the best available evidence to reviewer in either one specific step (during record search inform healthcare decision-making processes [15–17]. [43], screening [39, 44, 45], data extraction [45], risk of The number of published systematic reviews has bias assessment/ critical appraisal, etc.), or guide the user increased exponentially in recent years [18, 19]. How- through several steps or the whole SR process [46–48]. ever, conducting a SR is still a complex, challenging and Especially promising are ML techniques [49–51] for time-consuming process [20, 21], and it requires a multi- the automation of systematic reviews steps [41, 46, 52]. disciplinary team with at least one experienced reviewer Within the artificial intelligence (AI) discipline, ML [22]. The use of computational tools to assist and facili- methods are considered the most promising techniques tate various stages of conducting a SR has always been for working with unstructured data. These methods, relevant and the development of new tools has also seen usually combined with NLP technologies are used for a progressive increase [23–25]. Currently many tools are text classification and data extraction, result in effec- available [23–26], some of them providing support dur- tively assisting the article screening process during the ing some stages of the SR process and others supporting performance of SR [29, 53, 54]. Machine learning is a the entire workflow [27–33]. The types of software used multidisciplinary field that consists of the development can vary; including algorithms, packages (collections of of computer algorithms that can “learn” how to perform functions or algorithms), libraries (collections of pack- a specific task [51]. By using mathematics and statistics, ages), desktop apps (programs that are executable from the algorithms are trained to make classifications or pre- the desktop), and they may range from being locally run dictions based on a provided set of training data, driv- from a device to web-based applications (software acces- ing decision-making within specific applications. Unlike sible and executable through a web browser) that are conventional algorithms, ML systems pretend to imitate hosted on a webserver. human learning behaviour and can improve their per- As the WHO Classification on Tumours Programme formance without being directly re-programmed [54, (WCT) [34, 35] we wish to promote evidence-based 55]. So far ML algorithms are primarily employed to practice in pathology. We need to review a very large assist the article screening during the systematic review amount of scientific literature to classify each of the 3,000 process. This process of screening publication records tumour types in the classification, ideally applying struc- implies categorising them into groups (i.e., included or tured evidence synthesis methods by conducting SR. To excluded), considering the research question and pre- produce so many SR with limited human resources there defined eligibility criteria. Article selection is usually Cierco Jimenez et al. BMC Medical Research Methodology (2022) 22:322 Page 3 of 14 Planning Work network • PredicTER Identify the issue and determine • TaskExchange the question Whole process • Parsifal • SLR-Tool Deduplication • CADIMA • ReLiS • SRDB.PRO • PICO Portal • SRA • Synthesisr • JBI-SUMARI • SASR • StArt • Open-meta.app • SESRA • Thoth Screening • EPPI-Reviewer • Screen-IT Write a protocol • Abstrackr • Metagear • SRA Search • Inciteful.xyz • Publish or • ArticleNet • Papers-review • Subscreen • 2DSearch • Jane Perish • ASReview • PICO Portal • SWIFT-Active • A2A • JSTOR Text • PubReMiner • BioReader • RAx Screener • Ananse anlyser • PubVenn • Colandr • Rayyan • SWIFT-Review Search for studies • APSE • LDS Shiny • RAx • Covidence • Revtools • SyRF • BADERI • Leaf • RCT tagger • DistillerSR • RobotAnalyst • Sysrev • BEST • Lingo3G • Researchr • DoCTER • RobotSearch • Bioreader • Litmaps • RobotAnalyst • Carrot2 • Litsearchr • Sci2 Tool Sift and select studies • Citationchaser • Mapping • Searchrefiner Data extraction • EvidenceSET • RAx • CitNetExplorer MEDLINE • SensPrecOpti- • CERC • ExaCT • Refchaser Discuss and conclude • CloudSERA • Medline mizer • Cochrane_ • Graph2Data • Scholarcy • Colandr (Pubmed) • SLRqub scraper • Import.io • SyRF overall findings • Coremine trend • SRA • Colandr • MetaDigitise • Sysrev Systematic Extract review data from studies Medical • Medline • Textpresso • ContentMine • Metagear • Table Builder • CTD Transpose • Trialstreamer • Covidence • Numbat • Topictagger • DistillerSR • MeSHSIM • VOSViewer • DAA • PICOtron • WebPlotDigi- Dissemination • DOC Search • NAILS • Voyant Tools • Distiller SR • PlotDigitizer tizer • Findpapers • OmixLitMiner • Wordstat 9.0 • EPPI-Reviewer • PubMed2XL • Wordstat 9.0 Assess the quality of the • GScraper • Paperscraper • Yale MeSH studies • Import.io • PDQ-Evidence Analyser Critical apprisal • EPPI-Reviewer • Robvis • Pex • AMSTAR • GRADEpro • Sysrev Checklist • ORBIT Matrix • Systematic Reference • EndNote • Qiqqa • Covidence Generator Review Combine the data management • JabRef • Refbase • CrowdCARE • RAx Assistant • BibDesk • KBibTex • Wikindx • DistillerSR • RobotReviewer • Citavi • Mendeley • Zotero • Colwiz • Paperpile Discuss and conclude • Docear • Papersapp Meta-analysis • Metafor • OpenMEE • Comprehensive • Metagear • OpenMetaAn- overall findings Publication Meta-Analysis • MetaGenyo alyst • Manuscript Matcher • Dmetar • MetaInsight • Pymeta • EPPI-Reviewer • MetaLight • PythonMeta • MAVIS • Metamisc • RevMan Figures • Prismadia- • ROSES Systematic review • ALBATROSS gramR flowchart • Meta DTA • MetaXL • Xmeta • Meta-CART • MIX 2.0 • Gephi • PRISMAstate- • yEd Graph • Meta-Essentials • NetMetaXL • PRISMA 2020 ment Editor SR database • Cochrane • MAGICapp Summary of • EvidenceSET • RAx Dissemination • BEME library • PEDro Findings/Report • Gephi • RevMan • Campbell • Epistemonikos • EviAtlas • GRADEpro • SyRF Collaboration • JBI EBP Fig. 1 Software tools assisting the Systematic Review process. The central figure represents from the early to the late stages of the systematic review process described by the Cochrane Foundation (Higgins JPT et al., Cochrane Handbook for Systematic Reviews of Interventions version 6.3, Cochrane, 2022). The obtained tools were grouped into blocks depending on the covered SR step (Planning: Process of planning and writing the protocol for a SR; Deduplication: Process of removing duplicated records and articles retrieved by the search in a SR; Screening: Process of screening records and articles retrieved by the search in a SR; Data Extraction: Process of data extraction in the SR included studies; Critical appraisal and/or Bias assessment: Evaluation of the methodological quality and/or risk of bias in a SR included studies; Meta-analysis: Process of pooling findings of included studies, using statistical methods; Summary of findings/Report: Process of summarizing and reporting of findings; Work network: Process of networking; Whole process: whole systematic review process including all steps; Search: process of elaboration of search strategy, running the search and/or obtaining records retrieved by the search; Reference Management: process of screening and selection of records, as well reporting and scientific writing allowing management of large numbers of reference records and in text citations; Figures: Visualisation of data; SR Databases: Registration, collection and dissemination of SRs; Publication: Process of publishing the results) performed by two independent human reviewers, revis- review can be used to train the algorithm for the screen- ing first title and abstract of the retrieved records, and ing of recently published records. These algorithms are later full text of the article. The first step compromising trained according to the computer–human interaction the revision of title and abstract of a bibliometric record, where the availability of a training data set and the pur- is a task for which ML algorithms can be employed. pose of the classification strategy are the key points [41, These algorithms can be trained to develop the abil- 49, 51, 55]. They can be broadly classified as supervised ity to categorise, using so called “training data sets” of learning (trained on labelled data), unsupervised learn- records screened by human reviewers. This applica- ing (trained without labelled data) and semi-supervised tion of ML could be used to facilitate updating system- learning (trained by a small, labelled dataset and a large atic reviews, since the categorisation from the original unlabelled data set) algorithms [53]. Cierco Jimenez et al. BMC Medical Research Methodology (2022) 22:322 Page 4 of 14 As mentioned above, these algorithms/software are relevant publications in the electronic databases MED- more and more used for article screening in systematic LINE (through PubMed), EMBASE and Web of Sci- grams Abstrack® [39], ASReview® [56], Colandr® [33], reviews [53, 55, 56], with some examples being the pro- ence. Database specific terms (MeSH and Emtree) and EPPI-Reviewer® [57] and Rayyan® [33], all easily accessible keywords for the concepts of “Systematic review” and “Machine learning (ML)” were combined with Boolean online. Other steps of the SR process such as data extrac- operators to produce tailored search strings for each tion [54, 58] or risk of bias assessment [59] have been also database. Multiple variations of search terms were com- As an example, the software RobotReviewer® is able to been exploring whether ML tools can facilitate the work. bined to produce different sets of results. Final search strategies are available as supplementary material (See assign low, high or unclear risk of bias to randomized con- Additional file 1). trol trials (RCTs) [59]. Additionally, the development of In addition, a structured search was conducted to instance DistillerSR® [41, 46]), and important efforts are automatic data extraction tools is being investigated (for identify ML tools in the following repositories of soft- ware: The Comprehensive Perl Archive Network (CPAN) underway to explore whether ML tools can be used effi- [63], The Comprehensive R Archive Network (CRAN) ciently in combination with each other. [64], GitHub [65], The National Centre for Text Min- Most steps in the SR process can potentially ben- ing (NaCTeM) [66], The PHP Extension Community efit from automation [60], but they often require more Library (PECL) [67], The Python Package Index (PyPI) sophisticated computational methods than those pro- [68], SourceForge [69] and the Systematic Review Tool- vided by ML [44]. However, developing automatic box (SR toolbox) [70]. Due to the limitations of the search screening tools based on combined ML techniques engines of these websites no search strings could be used seems feasible; plenty of research and developments to retrieve records and multiple, iterative searches using have been done in this field in recent years [26, 39, 41, single keywords were performed. Searches were con- 46, 52]. Considering these advances, the rapid evolu- ducted in the mentioned electronic databases and soft- tion of the area, and the difficulties in identifying the ware repositories from ­01st of April to the ­31st of May best suited tool for each task and field, we aimed with 2021, with no language restrictions. this project to systematically map available ML tools that assist the SR process [61]. No other mapping Eligibility criteria review on this topic has been published, and our find- Any publication or repository record describing a ML ings will identify existing tools, detect potential devel- software to assist the SR process in any field was consid- opment gaps, and help to guide future research towards ered eligible. All publications or records with sufficient the most promising areas. technical description (reporting at least the software name, a short tool description and its purpose) were included when reporting in English, Spanish, German, Methods or French, and when developed or updated within the We conducted a Mapping Review to identify existing ML last 10 years. Reference management software and tools tools to assist during the SR process. A protocol was reg- that were not accessible to the reviewers (i.e., download- istered in the Open Science Forum (OSF) platform [62] able, importable, or executable from their source) were (Available at https://​osf.​io/​wmy7n/?​view_​only=​c501b​ excluded. tronic databases were imported into EndNote® and dupli- 501ed​e84b9​6b3c3​353e3​e81de​b0), since the International Publication records retrieved by the search in elec- Prospective Register of Systematic Reviews (PROSPERO) does currently not accept registrations for mapping cates were removed. Records were screened for eligibility reviews. based on their titles and abstracts by two reviewers (RCJ To identify all relevant ML tools comprehensive and BII) independently. Full text PDFs were obtained searches in several bibliographic databases and software for all abstracts deemed relevant for inclusion and fur- repositories were performed. Additionally searching ther assessed against the inclusion criteria by the same in software repositories allowed us to identify tools not two reviewers independently to obtain a final number of mentioned in journal articles, conference abstracts, or included publications. Discrepancies were resolved by similar technical literature. consensus. Records retrieved by the search in repositories were soft Excel® and duplicates were removed. Records were registered in an ad hoc developed data base using Micro- Search and selection A tailored search strategy was developed in collabora- screened for eligibility based on their titles and sum- tion with an information specialist (TL) to search for maries provided by the first reviewer (RCJ). A second Cierco Jimenez et al. BMC Medical Research Methodology (2022) 22:322 Page 5 of 14 reviewer (BII) revised the selection, resolving disagree- Results ment by consensus. A total of 9653 studies were retrieved from the structured searches in bibliometric databases (PubMed, Embase and Web of Science), 1491 of which were duplicates. Of Data extraction and synthesis the remaining 8162 items that were screened, 7970 were A data extraction form was developed, piloted, and excluded during a review of title and abstract (see Fig. 2). refined to capture basic information for each identified The remaining 192 publications were assessed by full tool. Extracted data included: computer science method text, resulting in 105 studies being excluded for reasons applied for the development of the tool (e.g., natural lan- as not reporting on a ML tool [15], not providing a link guage processing, supervised learning), stage of system- to the tool [47], not using ML methods [23], using ML atic review process the tool assists with, tool release date but not being adapted for SR [16], or not being published or publication date of the description, date of last update, in previously defined languages [4]. 86 publications were licensing and pricing details (free access or requir- identified describing ML tools and included. In addition, ing payment for the complete version), as well as being a total of 585 records were obtained from the selected open-source (the source code is available and repurposed) software repositories (CPAN, CRAN, GitHub, NaCTeM, or not, and the source to access/download the tool (e.g., PyPI, SourceForge and SR toolbox), 17 of which were software repository, hyperlink). In addition, the retrieved duplicates. The remaining 568 records were screened, tools were categorised according to how much back- excluding 536 and selecting 32 that described ML tools. ground in programming is needed to use the software, After the screening process, a total of 119 descriptions considering mainly whether the tool implementation of ML tools from publications and repository records needs modifications that require advanced programming were included for the data extraction process (See Fig. 2). skills (e.g., creation or adaptation of algorithms to use From those descriptions 63 tools that assist the SR pro- functions within a package). cess using ML techniques were identified (Table 1). We Extracted data were compiled in a summary of findings described basic information for each of the identified (SoF) table and identified tools graphically summarised ML tools, including the step of the SR where it operates, in a mapping infographic. pricing options, open-source availability, computational Identification of studies via databases and registers Identification of studies via other methods Records identified from Software repositories: 585 Identification Studies identified from Records removed before Databases: 9653 Studies removed before CPAN: 58 screening: screening: CRAN: 100 PubMed (MEDLINE) = 1919 GitHub = 329 Duplicate records NaCTeM = 21 Duplicate records Web of Science = 2429 removed = 17 Removed = 1491 PyPI = 33 Embase = 5305 SourceForge = 21 SR toolbox = 23 Studies excluded by title and Studies screened: 8162 abstract: 7970 Reports excluded: 536 Reports sought for retrieval: 568 No ML tool mentioned = 398 Screening Studies sought for retrieval: 192 Not ML tool for SR = 8 Studies excluded: 105 Not specified if ML or link to tool not provided (or not No ML tool mentioned = 15 Reports assessed for eligibility: 568 working) = 47 Studies assessed for eligibility: 192 Not specified if ML or link to tool Other SR tools not using ML not provided (or not working) = 47 = 71 Other SR tools not using ML = 23 Not in English, Spanish, Not in English, Spanish, German German or French = 5 or French = 4 Not updated in the last 10 Not ML tool for SR = 16 years = 7 Studies included: 87 Reports included: 32 Included Studies and reports included: 119 Fig. 2 PRISMA flow diagram for the mapping review. Adapted From: Page MJ et al. 2021. SR: Systematic Review; ML: Machine Learning; CPAN: The Comprehensive Perl Archive Network; CRAN: The Comprehensive R Archive Network; NaCTeM: National Centre for Text Mining; PyPI: The Python Package Index Cierco Jimenez et al. BMC Medical Research Methodology (2022) 22:322 Page 6 of 14 Table 1 Identified machine learning tools that assist the systematic review process by type of computational tool Index ML tool name Step in the systematic Computational Last update Number Hyperlink to access to the review process the tool is methods of tool used involved citations Machine learning algorithms   1 A Keyword-Based Literature Data extraction NLP 2019 1 http://​www.​mdpi.​com/​2073-​ Review Data Generating 8994/​12/6/​903/​s1 Algorithm (*)   2 Active_learning_document_ Screening NLP, SL 2020 1 https://​github.​com/​afcar​vallo/​ screening (*) active_​learn​ing_​docum​ent_​ scree​ning   3 Active-learning-for-system- Screening NLP, SL 2019 1 https://​github.​com/​sxzha​ atic-review (*) ng1201/​active-​learn​ing-​for-​ syste​matic-​review   4 DAE-FF (*) Screening SL 2019 1 https://​github.​com/​gkont​onats​ ios/​DAE-​FF   5 InclusionCriteria (*) Screening NLP 2019 1 https://​github.​com/​infoq​ualit​ ylab/​Inclu​sionC​riter​ia   6 Machine Learning Func- Screening SL 2020 1 https://​syste​matic​revie​wsjou​ tions (*) rnal.​biome​dcent​ral.​com/​artic​ les/​10.​1186/​s13643-​020-​01520- 5#​Sec14   7 PubmedClassifier (*) Screening NLP, SL 2018 1 https://​github.​com/​Yujia​Bao/​ Pubme​dClas​sifier   8 RAPTOR (*) Data extraction NLP 2021 1 https://​github.​com/​Cochr​ aneSc​hizop​hrenia/​RAPTOR   9 Rules_cochranereviews (*) Screening, Data extraction NLP 2020 1 https://​github.​com/​dsuri​an/​ rules_​cochr​anere​views   10 SLR_SearchStrings (*) Search, Screening NLP 2018 1 http://​bit.​ly/​2PwL3​7v   11 TextRank (*) Search NLP, USL 2020 2 https://​cran.r-​proje​ct.​org/​web/​ packa​ges/​textr​ank/​index.​html Desktop-applications   12 ASReview Screening NLP, SL 2021 2 https://​asrev​iew.​nl   13 ASReview-covid Screening NLP, SL 2021 1 https://​github.​com/​asrev​iew/​ asrev​iew-​covid​19   14 FASTREAD (*) Screening SL 2020 1 https://​github.​com/​fastr​ead/​ src/​tree/​v1.4.0   15 GAPscreener Screening NLP, SL 2017 2 http://​hugen​aviga​tor.​net/​ HuGEN​aviga​tor/​HNDes​cript​ ion/​opens​ource_​GAP.​html   16 Pvtopic (*) Screening, Data extraction SL 2016 1 http://​nactem.​ac.​uk/​pvtop​ic   17 RapidMiner Screening NLP, SL, SSL, USL 2021 6 http://​rapid-i.​com   18 Rax Search, Screening, Data NLP, SL 2021 1 https://​raxter.​io extraction   19 SWIFT-Review Search, Screening SL 2019 8 https://​www.​sciome.​com/​ swift-​review Software packages/libraries   20 BIBOT (*) Search NLP 2018 1 https://​github.​com/​Nurtal/​ BIBOT-​light-​versi​on   21 Costumer (*) Search SL 2017 2 https://​github.​com/​UBESP-​ DCTV/​costu​mer   22 Litsearchr (*) Search NLP 2019 2 https://​github.​com/​eliza​ grames/​litse​archr   23 Revtools (*) Deduplication, Screening NLP 2019 2 https://​cran.​rproj​ect.​org/​packa​ ge=​revto​ols   24 Rnatlp (*) Data extraction NLP 2020 1 https://​github.​com/​Senso​rNet-​ UFAL/​rnatlp Web-applications   25 2DSearch Search NLP 2021 2 https://​www.​2dsea​rch.​com Cierco Jimenez et al. BMC Medical Research Methodology (2022) 22:322 Page 7 of 14 Table 1 (continued) Index ML tool name Step in the systematic Computational Last update Number Hyperlink to access to the review process the tool is methods of tool used involved citations   26 Abstrackr Screening SL 2019 23 http://​abstr​ackr.​cebm.​brown.​ edu   27 Aggregator Search SL 2017 1 http://​arrow​smith.​psych.​uic.​ edu/​cgi-​bin/​arrow​smith_​uic/​ RCT_​Tagger.​cgi?​ID=​22379   28 Carrot2 Search NLP 2021 1 https://​search.​carro​t2.​org   29 Chilibot Search NLP 2017 1 http://​www.​chili​bot.​net   30 Cochrane RCT Classifier Screening SL 2021 1 https://​crsweb.​cochr​ane.​org/​ login.​html   31 Cochrane Register of Studies Search NLP 2021 1 https://​commu​nity.​cochr​ane.​ org/​help/​tools-​and-​softw​are/​ crs-​cochr​ane-​regis​ter-​studi​es   32 Colandr Screening, Data extraction NLP, SL 2018 6 www.​colan​drapp.​com   33 Concept Encoder Screening NLP, SL 2021 1 https://​www.​front​eo.​com/​en/​ produ​cts/​conce​ptenc​oder   34 COREMINE medical Search NLP 2021 1 http://​www.​corem​ine.​com/​ medic​al   35 DistillerSR Search, Screening, Data NLP, SL 2021 10 https://​www.​evide​ncepa​rtners.​ extraction com   36 DoCTER Screening NLP, SSL 2017 1 https://​www.​icf-​docter.​com   37 Doctor Evidence Search NLP 2021 3 https://​www.​drevi​dence.​com   38 Epistemonikos Search SL 2021 3 https://​www.​epist​emoni​kos.​ org   39 EPPI-Reviewer Screening NLP 2021 17 https://​eppi.​ioe.​ac.​uk/​eppir​ eview​er-​web   40 ExaCT Data extraction SL 2010 7 https://​bio-​nlp.​org/​EXACT   41 FACTA + Search NLP 2019 1 http://​www.​nactem.​ac.​uk/​facta   42 Heoro Search NLP 2021 1 https://​www.​heoro.​com   43 IRIS.AI Search, Screening, Data NLP, SL 2019 1 https://​the.​iris.​ai extraction   44 Leximancer Search NLP, USL 2021 2 https://​www.​lexim​ancer.​com   45 Linguamatics Search NLP 2021 1 https://​www.​lingu​amati​cs.​com   46 PICO Portal Screening, Deduplication NLP, SSL 2021 1 https://​picop​ortal.​org   47 Rayyan Screening NLP, SL 2021 17 https://​www.​rayyan.​ai   48 RCT tagger Search NLP, SL 2017 6 http://​arrow​smith.​psych.​uic.​ edu/​cgi-​bin/​arrow​smith_​uic/​ RCT_​Tagger.​cgi   49 Research Screener Screening NLP, SL 2021 1 https://​resea​rchsc​reener.​com   50 RobotAnalyst Search, Screening SL 2018 11 http://​www.​nactem.​ac.​uk/​ robot​analy​st   51 RobotReviewer (*) Data extraction, Risk of Bias NLP, SL 2020 14 https://​www.​robot​revie​wer.​ net/   52 RobotSearch Screening NLP, SL 2019 5 https://​github.​com/​ijmar​shall/​ robot​search   53 Sample Size Search (SSS) Tool Search NLP 2020 1 https://​iheal​th.​uemc.​es for PubMed   54 Screen4Me Screening NLP 2020 1 https://​commu​nity.​cochr​ane.​ org/​sites/​defau​lt/​files/​uploa​ ds/​S4M_​webin​ar_​slides_​Feb_​ 2019.​pdf   55 SRA Search, Deduplication, NLP 2021 5 https://​sr-​accel​erator.​com Screening   56 SWIFT-Active Screener Screening SL 2021 4 https://​www.​sciome.​com/​ swift-​activ​escre​ener Cierco Jimenez et al. BMC Medical Research Methodology (2022) 22:322 Page 8 of 14 Table 1 (continued) Index ML tool name Step in the systematic Computational Last update Number Hyperlink to access to the review process the tool is methods of tool used involved citations   57 SyRF Screening NLP 2021 2 http://​syrf.​org.​uk   58 Sysrev Screening SL 2021 1 https://​sysrev.​com   59 Thalia (*) Search NLP 2019 2 http://​nactem-​copio​us.​man.​ac.​ uk/​Thalia   60 Trial2rev (*) Search, SR updates SSL 2021 1 https://​github.​com/​evide​nce-​ surve​illan​ce/​trial​2rev   61 Trialstreamer Search NLP, SL 2021 1 https://​trial​strea​mer.​robot​revie​ wer.​net   62 Voyant Tools Search NLP 2021 1 https://​voyant-​tools.​org   63 Wordstat Search NLP 2021 1 https://​prova​lisre​search.​com/​ produ​cts/​conte​nt-​analy​sis-​ softw​are NLP: Natural language processing; SL: Supervised learning; SSL: Semi-supervised learning; USL: Unsupervised learning. (*) = requirement of a programming background for the correct deployment of the tool methods involved, number of citations/times mentioned The most applied AI methodology by the tools was in this review, date of the last update, hyperlink to the NLP 48 (76%), mostly to improve, accelerate, or auto- tool, and the necessity of a programming background to mate the underlying text analytics. Supervised learning use the software in Table 1. To facilitate the tools descrip- was employed in more than half of the tools 37 (58%) tion in the table they were grouped by category of appli- whereas, unsupervised and semi-supervised algorithms cation (e.g., algorithms, web-applications). In addition, were less frequently utilized. we graphically represented the identified ML tools into In terms of open-source status, 31 (49%) of the blocks depending on the covered SR step (see Fig. 3). retrieved tools were open-source, 15 (24%) were not Among the 63 tools identified, the majority were cited accessible at all and for 17 (27%) the accessibility was not between 1 and 5 times (52, 83%) within the screen- described. Most of the identified tools were freely availa- ing of the 119 descriptions obtained from publications ble (44, 70%), but 14 (22%) required payment for a license obtained between 6 and 23 citations, being Abstrackr® and repository records. The remaining tools (11, 18%) and in 5 (8%) cases this was not clearly described. [23], EPPI-Reviewer® [57], Rayyan® [33], RobotReviewer® When comparing licencing requirement with the [14], and RobotAnalyst® [11] the top 5 most referenced updates over time we observed that within the 14 fee- based tools, only 1 (13%) were outdated, while 13 (92%) tools in this review. have been updated over the last five years and 13 (86%) All the identified software tools assist early stages of in the last year. In contrast, from the 44 freely avail- the SR process, with the 63 (100%) tools addressing auto- able tools, 1 (15%) remained out to date, 43 (97%) were mation of the steps of literature search, screening, and updated over the last five years and 13 (30%) were data extraction; 30 (47%), 35 (55%) and 11 (17%) tools updated in the last year. respectively. Whereas late SR stages have been less well- We also detected that of the 44 free tools, 30 (68%) exploited, with 1 tool count. were open-source, 5 (12%) were not, and 9 (20%) were In terms of updates over time most identified tools not described. In contrast, from the 14 fee-based tools, have been updated or released over the last 5 years (62, 10 (72%) were not open-source, one (7%) was open- 98%), whereas only 1 (2%) case was considered nearly source and three (7%) were not described. out of date due to older updates (see Table 1). It is worth Variations in tools’ software considering open-source mentioning that around half of the identified tools have availability and licencing were also noticeable. On one been released or updated over the last year (29, 46%). hand, from the 39 identified web applications, only 10 Thirty-nine (62%) of the included tools were web appli- (26%) were open-source, while 14 (36%) were not open- cations, 11 (17%) were algorithms, 5 (8%) were libraries source, and 15 (38%) were not described. In contrast, or packages and 8 (13%) were desktop applications. Also, of the 11 algorithms all were open-source. The 5 librar- 22 (35%) tools required a programming background to be ies and packages had similar an equal profile, where all used whereas 41 (65%) could be used by a less skilled cus- were open-source. Amongst the 8 desktop apps, 5 (63%) tomer (see Table 1). were open-source, 1 (12%) was not open-source, and in Cierco Jimenez et al. BMC Medical Research Methodology (2022) 22:322 Page 9 of 14 Search Identify the issue and determine Deduplication • SRA the question • PICO Portal • Revtools • 2DSearch • Linguamatics • Aggregator • Litsearchr • BIBOT • Rax Discuss Write and conclude a protocol • Carrot2 • RCT tagger Screening overall findings • Chilibot • RobotAnalyst • Cochrane Register of • Sample Size Search Tool Systematic review • AbstrackR • PICO Portal Studies for PubMed Search for studies • Active_learning_docu- • PubmedClassifier • COREMINE medical • SLR_SearchStrings ment_screening • Pvtopic • Customer • SRA Dissemination • Active-learning-for-sys- • RapidMiner • DistillerSR • Swift-Review Sift and select studies tematic-review • Rax • Doctor Evidence • TextRank • ASReview • Rayyan • Epistemonikos • Thalia • ASReview-covid • Research Screener • FACTA+ • Trial2rev • Cochrane RCT Classifier • Revtools • Heoro • Trialstreamer Extract data from studies • Colandr • RobotAnalyst • IRIS.AI • Voyant Tools • Concept Encoder • RobotSearch • Leximancer • Wordstat • DAE-FF • rules_cochranereviews Assess the quality • DistillerSR • Screen4Me of the studies • DoCTER • SLR_SearchStrings Data extraction • Doctor • SRA Combine the data • EPPI Reviewer • Swift-Active Screener • A Keyword-Based • METAGEAR • FASTREAD • Swift-Review Literature Review Data • Pvtopic • GAPscreener • SyRF Generating Algorithm • RAPTOR • Sys_review_ml • Colandr • Rax Discuss and conclude • InclusionCriteria • IRIS.AI • Sysrev • DistillerSR • Rnatlp overall findings • Machine Learning • ExaCT • RobotReviewer Functions • IRIS.AI • rules_cochranreviews Systematic review Critical appraisal Systematic review updates Dissemination • RobotReviewer • Trial2rev Requirement of a payment licensing to access the full version Open-source availability Fig. 3 Machine learning tools that assist the Systematic Review process. The central figure represents from the early to the late stages of the systematic review process described by the Cochrane Foundation (Higgins JPT et al., Cochrane Handbook for Systematic Reviews of Interventions version 6.3, Cochrane, 2022). The obtained ML tools were grouped into blocks depending on the covered SR step. Next to the tool names, the yellow symbol means the requirement of a payment licencing to access to the full version of the tool and the green symbol open-source availability. SR = Systematic Review, ML = Machine learning 2 (25%) cases this was not described. Algorithms and algorithms require evaluation and training using a pre- packages/libraries had the highest proportion of freely selected and labelled set of records, and their perfor- available tools with 5 (100%) and 11 (100%) respectively, mance varies greatly depending on this previous step of compared to web and desktop applications, with 23 (59%) training [41, 71, 72]. These requirements have hampered and 5 (63%) tools respectively. extended use of such tools by systematic reviewers for a long time and constitute an important barrier for their Discussion use in in topics with no representative training data sets SR and meta-analyses are recognized as the highest level to train the algorithms, as is often the case in the field of of evidence [3, 9, 11, 13, 17], and the growing number pathology. Also, when reviewing the retrieved tools more of available tools to assist during the performance of SR closely, it is obvious that most are not adapted to be used probably reflects an increasing recognition of the utility by users with no background in informatics or program- of this type of studies (see Fig. 1). It probably also reflects ming skills, who would require a tool with a user-friendly an increasing appreciation of the potential value of com- interface. Despite recent advances, increasing numbers putational methods to simplify the performance of such of developments and new computational solutions to highly structured reviews of the scientific literature, assist during the SR process, it remains a challenge for and also to improve their reliability and reproducibility. a reviewer to select the best suited software for each SR Recent years have seen the development of many ML project. The results of this mapping review provide an tools that aim to reduce the immense human resources overview of the currently available ML tools to assist dur- and time effort required by a multi-disciplinary team to ing the performance of a SR and will help future review- develop such a review [29, 38, 52, 56]. One of the most ers and researchers to identify the right tool for each assisted steps in the whole review process is the article project and facilitate the development of new evidence screening, where these tools assist the reviewer by sug- synthesis methodologies (see Fig. 3). gesting, classifying, or selecting records, and can either It is evident that the ML tools identified by this review help or even replace the reviewer during certain parts have been created with different aims by a great variety of of the process [28, 52, 71]. However, the current ML developers, ranging from individuals and small research Cierco Jimenez et al. BMC Medical Research Methodology (2022) 22:322 Page 10 of 14 groups to large organisations dedicated to evidence- freely available, which may in part be compensating for based medicine and systematic reviewing at a large scale the funding disadvantage [26, 75]. [31, 37, 44, 56, 57]. These efforts have shown widely vari- Our findings show a major interest of developers in able success so far and our final tool map shows that only computational methods to assist the early stages of the a few tools are suitable for use by reviewers without pro- SR process specifically during the screening of articles. gramming backgrounds. Nevertheless the high propor- It is in this step where automation seems to have greater tion of free (70%) and open-source (49%) tools we have potential for success, with 55% of the tools assisting in detected in this mapping review may indicate efforts by this step and being the most promising of ML solutions. the SR community to overcome these limitations and The other two SR steps with promising developments produce tools to facilitate systematic review production are literature searching and data extraction, with 47% for all types of users. We believe that there are signs of and 17% of the tools respectively. This seems to point a growing movement of developers in the field that will towards an existing interest in the improvement of those probably continue to promote the progress of the auto- stages using ML approaches, but probably less success in mation of SR steps. It is crucial that such emerging col- the development. Within the other SR stages, a reduced laborations continue to be facilitated in the future and number of tools was identified, describing a minor inter- open sharing of data as well as methods being promoted est or lack of potential for ML solutions for these SR [23, 25, 26, 29]. Interestingly, amongst the free tools stages. about 68% are open-source, showing once more that the It is understandable that web-applications are the big- two concepts of open-source and free-of-charge are simi- gest group of retrieved ML tools (62% of all tools, being lar, but describe views based on fundamentally different 59% of them freely available and 26% open-source). This values: open-source is a methodology used to facilitate type of software generally permits easy access directly the development of software in a given field/task, while from a web browser and a user-friendly interface that free software is a social movement aiming to provide doesn’t require any advanced knowledge of the software equity in access. We identified also 14 (22%) ML tools or programming background for a successful use. Util- that require the payment for a licence, and as expected, ity and acceptability are likely to be high for these types 10 (72%) are not openly accessible. This may point to an of tools due to intuitive interfaces and potential to adapt interest of private developers in these types of tools and to different reviewer profiles, but we have not been able their potential commercial value [57, 73, 74]. The access to assess this in our review due to a lack of reporting of to such privately developed ML tools will be limited to such features. However, web-based applications require the organisations or individuals that can afford their extensive resources and a host (be it an institution, fees and will therefore not be an option for all review- group, or enterprise) to provide web server maintenance, ers. However, some of the payment-based tools might technical support, and to warrant a proper implemen- provide free access or fee reduction purchase depending tation of the tool. These resources are not always avail- on the review purpose and/or the team conditions (e,g., able for developers and it this is likely the reason for the collaborating memberships, shared interests), as applying large number of libraries/packages and algorithms we substantial discounts to lower middle-income country retrieved. These constitute the second largest block com- (LMIC) users and other similar situations. prising 25% of all retrieved ML tools and are fully devel- Most of the identified tools (98%) were updated or oped, but inactive software tools that require further released during the last 5 years, and 48% of all retrieved steps for their implementation. These tools comprise one ML tools during the last year, which proves an increas- or more algorithms and require knowledge in informat- ing interest in ML software development in the field and ics and programming skills for their application, which probably indicates a marked demand for such tools. Not significantly reduces the tools’ usability. Hence not a surprisingly, ML tools requiring payment for a license solution for all reviewers, but still a valuable tool that were far more frequently updated during the last year permits customization and can be tailored to the spe- (86%) than freely accessible tools (30%), showing the cific needs of a project with the necessary skills. Interest- advantage of the licenced approach for the rapid develop- ingly, ML packages/libraries and algorithms showed the ment of ML tools. However, most free tools (97%) have highest proportion of free (100% both) and open-source also been released or updated during the last 5 years, (100% both) tools compared to the other types of soft- suggesting that this approach can also be efficient and ware described. Desktop applications were rather rare produce updated products without relying on commer- with 8 mapped ML tools, even though this type of tool cial strategies. Some not-for-profit research institutions allows the user to locally execute the software from a appear to be highly interested in the development and computer after installation. This process and laborious promotion of SR automatization tools that are made implementation steps may limit their usability, but no Cierco Jimenez et al. BMC Medical Research Methodology (2022) 22:322 Page 11 of 14 programming skills are required, and this may turn them and plain language explanations on how to adapt tools into one of the most promising solutions. This may be to specific needs, may also speed up the development even more so the case if an easy installation and compat- of better ML tools, or even promote the creation of ibility with common operating software can be assured. new ones. However, our findings suggest that further developments Our systematic mapping review holds potential for bias are needed and that at this stage it results still difficult inherent to the limitations of its methodology. However to assess which type of software is best suited for single being a mapping exercise, risk of bias as that of selec- review projects and reviewer profiles. Factors such as tive reporting [77], could be minimized by applying few research topic, composition and expertise of the review exclusion criteria and reporting on all identified tools team, available resources, and technical skills, still for which we could retrieve sufficient information. This need to be considered and are the challenges for future also avoided a potential selection bias, and by following development. a previously defined and registered protocol we assured Only a few tools obtained more than 5 citations either the reliability and reproducibility of our work. The lack in scientific publications or software repositories. Not of advanced search functionalities in the search engines and most used tools as Abstrackr® [39], EPPI-Reviewer® surprisingly, among those were some of the best-known of software repositories did not permit sophisticated [57], Rayyan® [33], and RobotReviewer® [31] (see search strategies and ML tool registries might not have been detected. Nevertheless, the iterative search process Table 1).However, the majority of identified tools (83%) in these repositories combined with the sensitive search were cited less than 5 times, suggesting that despite the strategy applied in the bibliographic databases strengthen increasing development of software to assist the SR pro- the completeness of our findings, and the high number cess, new tools will have to compete with a few dominant of records screened makes this mapping review highly well-known tools. Efforts to improve the diffusion of the reliable. However, due to fast evolution of the targeted newly developed tools is therefore key, together with an field, new potential tools have been developed since the improvement of currently applied methodology. performance of this project, being the Elicit tool [78] We believe that collaboration to improve already an example of a tool that the developed search strat- available ML tools may yield well adapted software that egy haven’t considered. Despite the fact that the applied can provide a wide range of functionalities needed for methodology does not provide a synthesis of the find- systematic reviews, as shown by the already existing ings or a critical appraisal of the methodological quality variety of ML tools and the recent acceleration in the of the retrieved publications, our mapping exercise has launch of new and updated version. There are projects value and pertinence. Our description of available tools, such as Metaverse [76], where developers collaborate visually summarized in two comprehensive infographics to collect, integrate, and expand available functions, provide a decision support tool for reviewers, research- following open-source principles and making the tools ers and other decision-makers conducting and funding freely accessible to the evidence synthesis commu- evidence synthesis projects. This mapping review covers nity. Other projects as SR-Accelerator [28], integrate the breadth of science in ML tools and is needed to assist several tools in a suite to assist in more than one step related questions. The unique overview that it provides of the SR procedure, aiming to produce software that will inform future reviewers, developers, and research in guides and assists the reviewer during the whole pro- the field. cess. Additionally software repositories or toolboxes such as the SR toolbox [42, 70] exist to promote and share already available tools that assist the SR process. Conclusion Databases or repositories with specific training sets Systematic reviews (SR) are considered the most reli- are also a resource that helps the community of devel- able source in the hierarchy of the evidence levels, they opers perform collaborative work, providing the nec- permit the combination of large amounts of findings essary platforms for the sharing of data, information, from scientific publications and the identification of and expertise. potential evidence gaps in a field. Without SR, decision- Following this successful development, more efforts making processes are exposed to bias and flawed con- should be undertaken to facilitate communication and clusions. The development of computational tools to knowledge exchange among developers and users, so assist the systematic review process is rapidly expand- that usability and functionality of already existing tools ing, this reflects an increasing interest on the produc- can be improved and adapted to the needs of different tion of this type of studies. Our review provides an systematic review projects. Training in SR automation overview of available software to assist the performance for reviewers, provision of basic programming skills, of SR according to SR steps, and a complete map of ML Cierco Jimenez et al. BMC Medical Research Methodology (2022) 22:322 Page 12 of 14 tools, showing that ML algorithms represent one of the Author details 1 International Agency for Research on Cancer (IARC/WHO), Evidence Synthesis most investigated methods for the assistance of SR. The and Classification Branch, Lyon, France. 2 Laboratori de Medicina Computa- most promising approaches focus on the automation cional, Unitat de Bioestadística, Facultat de Medicina, Universitat Autònoma or semi-automation of parts of the process and include de Barcelona, Bellaterra, Spain. 3 International Agency for Research on Cancer (IARC/WHO), Services to Science and Research Branch, Lyon, France. 4 Servicio a high number of easy to use and easy to access web- de Medicina Preventiva, Hospital Universitario 12 de Octubre, Madrid, based applications that permit the use of ML software Spain. 5 International Agency for Research on Cancer (IARC/WHO), Nutrition for SR and other kind of evidence synthesis reviews. Our and Metabolism Branch, Lyon, France. 6 Department of Nutritional Sciences, University of Vienna, Vienna, Austria. results have uncovered the current state of open-source development and how it could support a call for the Accepted: 26 November 2022 formation of collaborative working groups in this field. Promoting and facilitating the production of SR by using computational assistance is therefore crucial in the use of the best available evidence to inform healthcare or References any decision-making processes. 1. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. BMJ (Clinical Research Ed). 1996;312(7023):71–2. Supplementary Information 2. Leach MJ. Evidence-based practice: a framework for clinical practice The online version contains supplementary material available at https://​doi.​ and research design. Int J Nurs Pract. 2006;12(5):248–51. org/​10.​1186/​s12874-​022-​01805-4. 3. Marchevsky AM, Wick MR. Evidence-based pathology: systematic litera- ture reviews as the basis for guidelines and best practices. Arch Pathol Lab Med. 2015;139(3):394–9. Additional file 1: Supplementary file1. Search strategies used in the 4. Satterfield JM, Spring B, Brownson RC, Mullen EJ, Newhouse RP, Walker mapping review. The file contains three tables with the developed search BB, et al. Toward a transdisciplinary model of evidence-based practice. strategies in the electronic databases MEDLINE, EMBASE and Web of Milbank Q. 2009;87(2):368–90. Science. 5. Spring B. Evidence-based practice in clinical psychology: what it is, why it matters; what you need to know. J Clin Psychol. 2007;63(7):611–31. Acknowledgements 6. Brownson RC, Fielding JE, Maylahn CM. Evidence-based public health: a The authors want to thank the editor and the journal for considering this fundamental concept for public health practice. Annu Rev Public Health. paper and the reviewers for their valuable feedback that helped us improve 2009;30(1):175–201. the quality of this project. 7. Hill EK, Alpi KM, Auerbach M. Evidence-based practice in health educa- tion and promotion: a review and introduction to resources. Health Authors’ contributions Promot Pract. 2009;11(3):358–66. RCJ: Conceptualization, methodology, writing—original draft. TL: Methodol- 8. Szajewska H. Evidence-based medicine and clinical research: both are ogy, writing—review, and editing. NR: Writing—visualization, review, and needed neither is perfect. Ann Nutr Metab. 2018;72(Suppl 3):13–23. editing. RC: Conceptualization, writing—review, and editing. IAC: Supervision, 9. Mulrow CD. Rationale for systematic reviews. BMJ. 1994;309(6954):597–9. funding acquisition, writing—review and editing. AGW: Supervision, writing— 10. Gupta S, Rajiah P, Middlebrooks EH, Baruah D, Carter BW, Burton KR, review and editing. BIIR: Conceptualization, methodology, supervision, writ- et al. Systematic review of the literature: best practices. Acad Radiol. ing— review and editing. All authors read and approved the final manuscript. 2018;25(11):1481–90. 11. Gopalakrishnan S, Ganeshkumar P. systematic reviews and meta-analysis: Funding understanding the best evidence in primary healthcare. J Family Med The WHO Classification of Tumours series is published by IARC on a not-for- Prim Care. 2013;2(1):9–14. profit basis, funded solely by sales of its books and subscription website. The 12. Armstrong R, Hall BJ, Doyle J, Waters E. ‘Scoping the scope’ of a cochrane author(s) received no financial support for the research, authorship, and/or review. J Public Health. 2011;33(1):147–50. publication of this article. 13. Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997;126(5):376–80. Availability of data and materials 14. Rudnicka AR, Owen CG. An introduction to systematic reviews and meta- All data generated or analysed during this study are included in this published analyses in health care. Ophthalmic Physiol Opt. 2012;32(3):174–83. article and its additional files. 15. Weed DL. the need for systematic reviews in Oncology. J Natl Cancer Inst. 2018;110(8):812–4. Declarations 16. Damen JAAG, Hooft L. The increasing need for systematic reviews of prognosis studies: strategies to facilitate review production and improve Ethics approval and consent to participate quality of primary research. Diagn Prognostic Res. 2019;3(1):2. Not applicable. 17. Sofaer N, Strech D. The need for systematic reviews of reasons. Bioethics. 2012;26(6):315–28. Consent for publication 18. Ioannidis JPA. The mass production of redundant, misleading, Not applicable. and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94(3):485–514. Competing interests 19. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic The content of this article represents the personal views of the authors and reviews a day: how will we ever keep up? PLoS Med. 2010;7(9): e1000326. does not represent the views of the authors’ employers and associated institu- 20. Borah R, Brown AW, Capers PL, Kaiser KA. Analysis of the time and workers tions. Where authors are identified as personnel of the International Agency needed to conduct systematic reviews of medical interventions using for Research on Cancer / World Health Organization, the authors alone are data from the PROSPERO registry. BMJ Open. 2017;7(2): e012545. responsible for the views expressed in this article and they do not necessar- 21. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and ily represent the decisions, policy or views of the International Agency for reporting characteristics of systematic reviews. PLoS Med. 2007;4(3): e78. Research on Cancer / World Health Organization. 22. Uttley L, Montgomery P. The influence of the team in conducting a systematic review. Syst Rev. 2017;6(1):149. Cierco Jimenez et al. BMC Medical Research Methodology (2022) 22:322 Page 13 of 14 23. Tsafnat G, Dunn A, Glasziou P, Coiera E. The automation of systematic 45. Cheng SH, Augustin C, Bethel A, Gill D, Anzaroot S, Brun J, et al. Using reviews. BMJ: British Medical Journal. 2013;346: f139. machine learning to advance synthesis and use of conservation and 24. Marshall C, Brereton P, Kitchenham B. Tools to support systematic reviews environmental evidence. Conserv Biol. 2018;32(4):762–4. in software engineering: a cross-domain survey using semi-structured 46. Hamel C, Kelly SE, Thavorn K, Rice DB, Wells GA, Hutton B. An evaluation interviews. Proceedings of the 19th International Conference on Evalua- of DistillerSR’s machine learning-based prioritization tool for title/abstract tion and Assessment in Software Engineering; Nanjing, China: Association screening – impact on reviewer-relevant outcomes. BMC Med Res Meth- for Computing Machinery; 2015. p. Article 26. odol. 2020;20(1):256. 25. van Altena AJ, Spijker R, Olabarriaga SD. Usage of automation tools in 47. Adams CE, Polzmacher S, Wolff A. Systematic reviews: work that needs to systematic reviews. Res Synth Methods. 2019;10(1):72–82. be done and not to be done. J Evid Based Med. 2013;6(4):232–5. 26. Beller E, Clark J, Tsafnat G, Adams C, Diehl H, Lund H, et al. Making 48. Kohl C, McIntosh EJ, Unger S, Haddaway NR, Kecke S, Schiemann J, et al. progress with the automation of systematic reviews: principles of the Online tools supporting the conduct and reporting of systematic reviews International Collaboration for the Automation of Systematic Reviews and systematic maps: a case study on CADIMA and review of existing (ICASR). Syst Rev. 2018;7(1):77. tools. Environmental Evidence. 2018;7(1):8. 27. Rathbone J, Carter M, Hoffmann T, Glasziou P. Better duplicate detection 49. Deo RC. Machine learning in medicine. Circulation. 2015;132(20):1920–30. for systematic reviewers: evaluation of systematic review assistant-dedu- 50. Patel L, Shukla T, Huang X, Ussery DW, Wang S. Machine learning meth- plication module. Syst Rev. 2015;4(1):6. ods in drug discovery. Molecules. 2020;25(22):5277. 28. Clark J, Glasziou P, Del Mar C, Bannach-Brown A, Stehlik P, Scott AM. A full 51. Bi Q, Goodman KE, Kaminsky J, Lessler J. What is machine learning? A systematic review was completed in 2 weeks using automation tools: a primer for the epidemiologist. Am J Epidemiol. 2019;188(12):2222–39. case study. J Clin Epidemiol. 2020;121:81–90. 52. Bannach-Brown A, Przybyła P, Thomas J, Rice ASC, Ananiadou S, Liao 29. Marshall IJ, Wallace BC. Toward systematic review automation: a practical J, et al. Machine learning algorithms for systematic review: reducing guide to using machine learning tools in research synthesis. Syst Rev. workload in a preclinical review of animal studies and reducing human 2019;8(1):163. screening error. Syst Rev. 2019;8(1):23. 30. Rout BK, Sikdar BK. hazard identification, risk assessment, and control 53. O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text measures as an effective tool of occupational health assessment of haz- mining for study identification in systematic reviews: a systematic review ardous process in an iron ore pelletizing industry. Indian J Occup Environ of current approaches. Syst Rev. 2015;4(1):5. Med. 2017;21(2):56–76. 54. Boudin F, Nie J-Y, Bartlett JC, Grad R, Pluye P, Dawes M. Combining clas- 31. Marshall IJ, Kuiper J, Banner E, Wallace BC. Automating biomedical evi- sifiers for robust PICO element detection. BMC Med Inform Decis Mak. dence synthesis: RobotReviewer. Proc Conf Assoc Comput Linguist Meet. 2010;10(1):29. 2017;2017:7–12. 55. Olorisade BK, Quincey Ed, Brereton P, Andras P. A critical analysis of stud- 32. Marshall C, Brereton P, Kitchenham B. Tools to support systematic reviews ies that address the use of text mining for citation screening in systematic in software engineering: a feature analysis. Proceedings of the 18th reviews. Proceedings of the 20th International Conference on Evaluation International Conference on Evaluation and Assessment in Software and Assessment in Software Engineering; Limerick, Ireland: Association Engineering; London, England, United Kingdom: Association for Comput- for Computing Machinery; 2016. p. Article 14. ing Machinery; 2014. p. Article 13. 56. van de Schoot R, de Bruin J, Schram R, Zahedi P, de Boer J, Weijdema 33. Harrison H, Griffin SJ, Kuhn I, Usher-Smith JA. Software tools to support F, et al. An open source machine learning framework for efficient and title and abstract screening for systematic reviews in healthcare: an evalu- transparent systematic reviews. Nat Mach Intell. 2021;3(2):125–33. ation. BMC Med Res Methodol. 2020;20(1):7. 57. Thomas J, Brunton J. EPPI-Reviewer 4: Software for Research Synthesis. 34. Nagtegaal ID, Odze RD, Klimstra D, Paradis V, Rugge M, Schirmacher P, 2010. et al. The 2019 WHO classification of tumours of the digestive system. 58. Boudin F, Nie J-Y, Dawes M. Clinical information retrieval using document Histopathology. 2020;76(2):182–8. and PICO structure. Human Language Technologies: The 2010 Annual 35. Tan PH, Ellis I, Allison K, Brogi E, Fox SB, Lakhani S, et al. The 2019 World Conference of the North American Chapter of the Association for Com- Health Organization classification of tumours of the breast. Histopathol- putational Linguistics; Los Angeles, California: Association for Computa- ogy. 2020;77(2):181–5. tional Linguistics; 2010. p. 822–30. 36. Wick MR, Marchevsky AM. Evidence-based principles in pathology: exist- 59. Marshall IJ, Kuiper J, Wallace BC. RobotReviewer: evaluation of a system ing problem areas and the development of “quality” practice patterns. for automatically assessing bias in clinical trials. J Am Med Inform Assoc. Arch Pathol Lab Med. 2011;135(11):1398–404. 2016;23(1):193–201. https://​doi.​org/​10.​1093/​jamia/​ocv044. 37. Cleo G, Scott AM, Islam F, Julien B, Beller E. Usability and acceptability of 60. Automating the Systematic Review Process: A Bibliometric Analysis. four systematic review automation software packages: a mixed method 61. Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types design. Syst Rev. 2019;8(1):145. and associated methodologies. Health Info Libr J. 2009;26(2):91–108. 38. Gates A, Guitard S, Pillay J, Elliott SA, Dyson MP, Newton AS, et al. Per- https://​doi.​org/​10.​1111/j.​1471-​1842.​2009.​00848.x. formance and usability of machine learning for screening in systematic 62. Foster MED, Deardorff MA. Open Science Framework (OSF). Journal of the reviews: a comparative evaluation of three tools. Syst Rev. 2019;8(1):278. Medical Library Association. 2017;105(2). 39. Gates A, Johnson C, Hartling L. Technology-assisted title and abstract 63. Comprehensive Perl Archive Network (CPAN). https://​www.​cpan.​org/. screening for systematic reviews: a retrospective evaluation of the Accessed 13 May 2021. Abstrackr machine learning tool. Syst Rev. 2018;7(1):45. 64. The Comprehensive R Archive Network (CRAN). https://​cran.r-​proje​ct.​ 40. Kreimeyer K, Foster M, Pandey A, Arya N, Halford G, Jones SF, et al. org/. Accessed 11 May 2021. Natural language processing systems for capturing and standardizing 65. GitHub, Inc. https://​github.​com/. Accessed 10 May 2021. unstructured clinical information: a systematic review. J Biomed Inform. 66. The National Centre for Text Mining (NaCTeM). http://​www.​nactem.​ac.​ 2017;73:14–29. uk/. Accessed 7 May 2021. 41. Jaspers S, De Troyer E, Aerts M. Machine learning techniques for the 67. The PHP Extension Community Library (PECL). https://​pecl.​php.​net/. automation of literature reviews and systematic reviews in EFSA. EFSA Accessed 5 May 2021. Supporting Publications. 2018;15(6):1427E. 68. The Python Package Index (PyPI). https://​pypi.​org/. Accessed 4 may 2021. 42. Marshall C, Brereton P. Systematic review toolbox: a catalogue of tools to 69. SourceForge. https://​sourc​eforge.​net/. Accessed 3 May 2021. support systematic reviews. Proceedings of the 19th International Con- 70. The Systematic Review Toolbox (SR toolbox). http://​syste​matic​revie​ ference on Evaluation and Assessment in Software Engineering. 2015. wtools.​com/. Accessed 3 May 2021. 43. Russell-Rose T, Shokraneh F. 63 2Dsearch: facilitating reproducible and 71. Mujtaba G, Shuib L, Idris N, Hoo WL, Raj RG, Khowaja K, et al. Clinical valid searching in evidence synthesis. BMJ Evidence-Based Medicine. text classification research trends: systematic literature review and open 2019;24(Suppl 1):A36. issues. Expert Syst Appl. 2019;116:494–520. 44. Westgate MJ. revtools: an R package to support article screening for 72. Riccio V, Jahangirova G, Stocco A, Humbatova N, Weiss M, Tonella P. Test- evidence synthesis. Res Syn Meth. 2019;10(4):606–14. ing machine learning based systems: a systematic mapping. Empir Softw Eng. 2020;25(6):5193–254. https://​doi.​org/​10.​1007/​s10664-​020-​09881-0. Cierco Jimenez et al. BMC Medical Research Methodology (2022) 22:322 Page 14 of 14 73. Covidence. https://​www.​covid​ence.​org/​terms/. Accessed 25 Nov 2021. 74. Dr.Evidence. https://​www.​drevi​dence.​com/?​hsLang=​en. Accessed 25 Nov 2021. 75. Wallace BC, Noel-Storr A, Marshall IJ, Cohen AM, Smalheiser NR, Thomas J. Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach. J Am Med Inform Assoc. 2017;24(6):1165–8. 76. Metaverse. https://​rmeta​verse.​github.​io/​updat​es/​2019/​09/​05/​backg​ round-​to-​metav​erse.​html. Accessed 25 Nov 2021. 77. Higgins JP AD. Assessing risk of bias in included studies. Cochrane Hand- book for Systematic Reviews of Interventions. Chichester, UK: John Wiley & Sons, Ltd; 2019. p. 187–241. 78. Elicit: The AI Research Assistant. https://​elicit.​org/. Accessed 22 Sept 2022. Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in pub- lished maps and institutional affiliations. Ready to submit your research ? Choose BMC and benefit from: • fast, convenient online submission • thorough peer review by experienced researchers in your field • rapid publication on acceptance • support for research data, including large and complex data types • gold Open Access which fosters wider collaboration and increased citations • maximum visibility for your research: over 100M website views per year At BMC, research is always in progress. Learn more biomedcentral.com/submissions