Published online: 2020-11-09 Original article Establishing key research questions for the implementation of artificial intelligence in colonoscopy: a modified Delphi method Authors Omer F. Ahmad1 , Yuichi Mori 2, 3 , Masashi Misawa2, Shin-ei Kudo2 , John T. Anderson 4 , Jorge Bernal 5, Tyler M. Berzin 6, Raf Bisschops7 , Michael F. Byrne 8, Peng-Jen Chen9 , James E. East10, 11 , Tom Eelbode12, Daniel S. Elson13, 14 , Suryakanth R. Gurudu15, Aymeric Histace 16 , William E. Karnes17, Alessandro Repici18, 19, Rajvinder Singh 20, Pietro Valdastri 21, Michael B. Wallace 22 , Pu Wang 23 , Danail Stoyanov1, Laurence B. Lovat 1, 24 Institutions 17 H. H. Chao Comprehensive Digestive Disease Center, 1 Wellcome/EPSRC Centre for Interventional & Surgical Division of Gastroenterology & Hepatology, Sciences, University College London, London, UK Department of Medicine, University of California, 2 Digestive Disease Center, Showa University Northern Irvine, California, USA Yokohama Hospital, Yokohama, Japan 18 Department of Gastroenterology, Humanitas Clinical 3 Clinical Effectiveness Research Group, Institute of and Research Center, IRCCS, Rozzano, Milan, Italy Health and Society, University of Oslo, Oslo, Norway 19 Humanitas University, Department of Biomedical 4 Department of Gastroenterology, Gloucestershire Sciences, Pieve Emanuele, Milan, Italy Hospitals NHS Foundation Trust, Gloucester, UK 20 Department of Gastroenterology and Hepatology, Lyell 5 Computer Science Department, Universitat Autonoma McEwan Hospital, Adelaide, South Australia, Australia de Barcelona and Computer Vision Center, Barcelona, 21 School of Electronics and Electrical Engineering, Spain University of Leeds, Leeds, UK 6 Center for Advanced Endoscopy, Beth Israel Deaconess 22 Division of Gastroenterology & Hepatology, Mayo Medical Center, Harvard Medical School, Boston, Clinic, Jacksonville, Florida, USA Massachusetts, USA 23 Department of Gastroenterology, Sichuan Academy 7 Department of Gastroenterology and Hepatology, of Medical Sciences and Sichuan Provincial People’s University Hospitals Leuven, TARGID KU Leuven, Hospital, Chengdu, China Leuven, Belgium 24 Gastrointestinal Services, University College London 8 Division of Gastroenterology, Vancouver General Hospital, London, UK Hospital, University of British Columbia, Vancouver, British Columbia, Canada submitted 24.6.2020 9 Division of Gastroenterology, Tri-Service General accepted after revision 9.11.2020 Hospital, National Defense Medical Center, Taipei, published online 9.11.2020 Taiwan 10 Translational Gastroenterology Unit, John Radcliffe Bibliography Hospital, Oxford, UK Endoscopy 2021; 53: 893–901 11 Oxford NIHR Biomedical Research Centre, University DOI 10.1055/a-1306-7590 of Oxford, Oxford, UK ISSN 0013-726X 12 Medical Imaging Research Center, ESAT/PSI, KU © 2020. The Author(s). Leuven, Leuven, Belgium This is an open access article published by Thieme under the terms of the Creative Commons Attribution License, permitting unrestricted use, distribution, 13 Hamlyn Centre for Robotic Surgery, Institute of Global and reproduction so long as the original work is properly cited. (https:// Health Innovation, Imperial College London, London, creativecommons.org/licenses/by/4.0/) UK Georg Thieme Verlag KG, Rüdigerstraße 14, 14 Department of Surgery and Cancer, Imperial College 70469 Stuttgart, Germany London, London, UK 15 Division of Gastroenterology and Hepatology, Mayo Clinic, Scottsdale, Arizona, USA Appendix 1s, 2s, Table 1s 16 ETIS, Universite de Cergy-Pointoise, ENSEA, CNRS, Supplementary material is available under Cergy-Pointoise Cedex, France https://doi.org/10.1055/a-1306-7590 Ahmad Omer F et al. Establishing key research … Endoscopy 2021; 53: 893–901 | © 2020. The Author(s). 893 Original article Corresponding author Results The top 10 ranked questions were categorized into Omer F. Ahmad, MB BS, Wellcome/EPSRC Centre for five themes. Theme 1: clinical trial design/end points (4 Interventional & Surgical Sciences, Charles Bell House, 43–45 questions), related to optimum trial designs for polyp de- Foley Street, London, W1W 7TS, United Kingdom tection and characterization, determining the optimal end ofahmad123@gmail.com points for evaluation of AI, and demonstrating impact on interval cancer rates. Theme 2: technological develop- ABSTR AC T ments (3 questions), including improving detection of Background Artificial intelligence (AI) research in colo- more challenging and advanced lesions, reduction of false- noscopy is progressing rapidly but widespread clinical im- positive rates, and minimizing latency. Theme 3: clinical plementation is not yet a reality. We aimed to identify the adoption/integration (1 question), concerning the effective top implementation research priorities. combination of detection and characterization into one Methods An established modified Delphi approach for re- workflow. Theme 4: data access/annotation (1 question), search priority setting was used. Fifteen international ex- concerning more efficient or automated data annotation perts, including endoscopists and translational computer methods to reduce the burden on human experts. Theme scientists/engineers, from nine countries participated in an 5: regulatory approval (1 question), related to making reg- online survey over 9 months. Questions related to AI imple- ulatory approval processes more efficient. mentation in colonoscopy were generated as a long-list in Conclusions This is the first reported international re- the first round, and then scored in two subsequent rounds search priority setting exercise for AI in colonoscopy. The to identify the top 10 research questions. study findings should be used as a framework to guide fu- ture research with key stakeholders to accelerate the clini- cal implementation of AI in endoscopy. Introduction dominate the research agenda for the coming decade. In the Artificial intelligence (AI)-based technologies are being devel- field of colonoscopy, a recent review proposed some key princi- oped at a rapid pace for gastrointestinal endoscopy, in particu- ples for AI system development and testing [11]. However, to lar for colonoscopy. Some AI-based systems have now achieved our knowledge, there has been no prior publication of a formal regulatory approval to assist colorectal polyp detection and systematic process to identify research priorities for AI in characterization [1]. However, widespread clinical implementa- endoscopy. tion in routine colonoscopy practice is not yet a reality. The aim of this study was to identify research priorities relat- The vast majority of AI research in endoscopy to date, and ed to the implementation of AI in colonoscopy. Specific atten- more broadly within healthcare, has focused on preclinical or tion was given to colonoscopy, as AI systems are most transla- retrospective studies. These studies have been crucial in the tionally mature in this field of endoscopy. It was however ex- early phase of development [2]. However, following a number pected that many of these AI implementation research priori- of a recent prospective randomized controlled trials (RCTs) for ties would be broadly applicable to general endoscopy. AI in colonoscopy, the emphasis is now quickly shifting along the translational roadmap to real-world implementation and Methods clinical deployment [3 – 8]. Broad challenges related to the im- Study design plementation of AI in medicine, including topics such as inte- gration into existing clinical workflows, data sharing, transpar- A modified Delphi process is an established method for deter- ency, and patient safety, have been highlighted by opinion and mining consensus opinion. This involves collating individual perspective articles [9]. Workshops have been conducted, for anonymized opinions from an expert group and establishing a example by the National Institutes of Health and Radiological consensus using an iterative process via a number of rounds. Society of North America, to identify key research priorities for The research methodology used in this study was based on AI in medical imaging, although this focused predominantly on those previously published by the European Society of Gastro- foundational AI research topics, such as the development of intestinal Endoscopy, European Association for Endoscopic Sur- new image reconstruction methods and novel machine-learn- geons, and American Society of Colon and Rectal Surgeons to ing algorithms tailored to clinical imaging data [10]. identify research priorities [12 – 14]. The Delphi methodology It is now widely recognized that the most translationally ad- for research priority setting studies differs from the process ty- vanced AI applications in medicine, with the largest number of pically used to create consensus statements, where a prede- reported RCTs, belong to gastrointestinal endoscopy. The spe- fined threshold is determined for consensual agreement. In- cialty is now in a prime position to become a leader for imple- stead, for a research priority setting Delphi study, generated mentation research. Despite this, there has been minimal pub- questions are scored through a number of rounds to finally es- lished literature exploring the opportunities and challenges re- tablish a predefined number of top ranked questions. lating to this critical next stage in endoscopy, which is likely to 894 Ahmad Omer F et al. Establishing key research … Endoscopy 2021; 53: 893–901 | © 2020. The Author(s). For this study, the aim was to identify the top 10 ranked Round 2: Prioritization rating questions, in keeping with previously published major research priority setting studies [15]. A web-based, research electronic Participants were asked to rank the questions generated from data capture (REDCap) survey was designed for the purposes round 1 following the steering committee review, on a scale of of this Delphi process and was used in each round of the study. 1 (low priority) to 5 (high priority). The questions were grouped The study was conducted over a 9-month period between into nine themes and presented in a randomized order. The sur- March 2019 and November 2019. vey software mandated that every question was assigned a score by each participant. The questions were ranked according Steering committee and expert participants to their total score. The steering committee reviewed the re- A steering committee consisting of translational endoscopists sults and used the top 10 ranked questions including tied and computer scientists (O.F.A., L.B.L., J.T.A., P.V., D.S.E.) scores, which consisted of 28 questions in total, for re-ranking formed a key advisory group on the format and execution of in round 3. this study. The steering committee identified and invited participants Round 3: Final consensus rating by personal communication to create an international body of The top 10 ranked questions (including tied scores) generated experts with experience in translational AI in colonoscopy. The from round 2 were redistributed to all participants for re-scor- following inclusion criteria were used: current involvement in ing. The questions were presented in rank order and with an clinician and engineer/computer scientist collaborative re- associated mean score from round 2. On this occasion, partici- search in the field of AI or computer-aided diagnosis/detection pants were asked to re-score using a wider scale of 1 (very low (CAD) in colonoscopy with a specific focus on those with experi- priority) to 10 (very high priority) in order to gain greater discri- ence across the translation pipeline (i. e. case identification, mination between questions. The survey software mandated data acquisition/curation, algorithm development, clinical that every question was assigned a score by each participant. evaluation, and deployment considerations). Publication his- The final results were once again analyzed by the steering com- tory was also considered, with a requirement of at least one mittee, using the total score, and in addition the percentage of peer-reviewed publication in the field of AI/CAD in colonoscopy very high priority responses (9 or 10 scores) for questions with listed on PubMed. In addition, geographic diversity was consid- tied rank, to identify the final top 10 research questions. ered to ensure representation from the major regions involved in AI research and development in endoscopy. A total of 15 participants (12 endoscopists and three transla- Results tional computer scientists/engineers) from nine countries were There was a complete (100 %) participant response rate for all invited to form the expert group (see Appendix 1s, available in three rounds of the study. The steering committee review and online-only Supplementary material); none declined to partici- analysis of round 1 responses generated 59 individual research pate. The group included participants from North America (n = questions which are listed in Appendix 2s. These were categor- 5), Europe (n = 5), and the Asia–Pacific region (n = 5). The study ized into nine themes as shown in ▶ Table 1. aims and methodology were described from the outset of the In round 2, the mean scores for the 59 questions, scored on a study, with a clear primary objective to identify the top 10 re- scale of 1 to 5, ranged from 2.69 to 4.63 (Table 1s). The top 10 search questions related to the implementation of AI in colo- noscopy. ▶ Table 1 The nine themes and numbers of questions generated for each. Round 1: Question generation All 15 participants were invited to list an unlimited number of Research theme Number of questions research questions related to the implementation of AI and CAD in colonoscopy. The raw, verbatim responses were then Data (access, sharing/privacy, curation) 8 collated to generate an anonymous long-list for review by the Technological developments 11 steering committee. Multiple responses addressing the same fundamental issue were consolidated into a single question, Clinical adoption and integration into the endoscopy 10 suite noting the number of times each issue was submitted. Respon- ses were reviewed to ensure they were clearly understood. Care Performance metrics, clinical trial design, and end 10 was taken to preserve the original meaning and avoid any points amendment to the underlying theme where re-drafting was Clinical applications 5 necessary. Responses that did not allow for the generation of Training and education of workforce 3 clear research questions were excluded. The remaining respon- ses were then categorized into nine broad themes for the pur- Regulatory approval 3 poses of round 2. Questions that could have been allocated to Ethical and legal issues 6 several categories were assigned to one by consensus amongst Health economics 3 the steering committee. Ahmad Omer F et al. Establishing key research … Endoscopy 2021; 53: 893–901 | © 2020. The Author(s). 895 Original article ranked questions, including tied scores, that were redistributed The final top 10 questions were from five themes: clinical for round 3 included a total of 28 questions from eight themes. trial design/end points (4 questions), technological develop- In round 3, the mean scores for the 28 questions, scored on ments (3 questions), clinical adoption/integration (1 question), a wider scale of 1 to 10, ranged from 6.13 to 8.80 (▶ Table 2). data access/annotation (1 question), and regulatory approval The percentage of responses scored as a very high priority (9 or (1 question). ( ▶ Table 3). 10) ranged from 7 % to 60 %. ▶ Table 2 Questions in rank order following the final round 3 process. Question Rank Total Mean Percentage score score of responses scored as very high priority (9 or 10) What is the optimum clinical trial design to demonstrate efficacy for polyp detection AI/CAD 1 132 8.80 53 software? How do we improve the performance of AI/CAD to detect more challenging and advanced 2 126 8.40 47 lesions (e. g. subtle flat lesions and sessile serrated lesions)? How do we reduce false-positive rates for detection systems to avoid the user developing 3 118 7.87 47 “alert fatigue”? What are the optimal clinical end points for evaluation of AI/CAD? 4 118 7.87 27 Can we effectively combine polyp detection and characterization into one workflow? 5 116 7.73 60 Can we produce more efficient or automated annotation methods for data to reduce the bur- 6 115 7.67 40 den on human experts? How do we make the regulatory approval process more efficient and overcome hurdles? 7 113 7.53 33 How do we demonstrate that AI/CAD detection systems have an impact on interval colorectal 8 112 7.47 40 cancer rates? What is the optimum clinical trial design to demonstrate efficacy for polyp characterization 9 112 7.47 27 (optical diagnosis) AI/CAD software? How do we optimize CAD/AI so that it can be used in real-time with minimal latency? 10 111 7.40 53 What impact might AI/CAD detection and characterization systems have on colonoscopy sur- 11 111 7.40 20 veillance intervals and what are the associated costs? Can AI/CAD make endoscopy workflow more efficient (e. g. automated report writing)? 12 109 7.27 20 Can AI/CAD be used effectively to measure the quality of colonoscopy? 13 107 7.13 27 How should regulatory agencies deal with the iterative nature of software improvements in 14 107 7.13 13 AI/CAD? How do we develop quality assurance for annotation/labelling of data? 15 106 7.07 27 What impact will AI/CAD have on endoscopy training and performance? 16 105 7.00 33 How do we address data privacy, consent, and ownership issues to effectively share data 17 105 7.00 27 across different countries and centers for AI/CAD development? What effect will AI/CAD have on colonoscopy outcomes in relation to health economics (e. g. 18 105 7.00 20 faster workflow, fewer colonoscopies, reduction in colorectal cancer rates) and how do we measure this? How do we define standardized metrics for directly comparing the performance characteris- 19 104 6.93 13 tics of different AI software? How do we obtain enough data for categories that might be important for clinical application 20 102 6.80 7 but are under-represented (e. g. dysplasia detection in inflammatory bowel disease)? What performance thresholds (e. g. ASGE PIVI) are necessary to consider a resect & discard 21 100 6.67 27 strategy when employing computer-aided diagnosis tools during colonoscopy? Who owns the intellectual property in AI/CAD model development and can this be protected? 22 100 6.67 13 How do we audit AI/CAD systems once they are deployed in the clinical environment? 23 100 6.67 7 896 Ahmad Omer F et al. Establishing key research … Endoscopy 2021; 53: 893–901 | © 2020. The Author(s). ▶ Table 2 (Continuation) Question Rank Total Mean Percentage score score of responses scored as very high priority (9 or 10) How do we train AI/CAD systems once they are deployed in order for them to improve and 24 99 6.60 13 learn continuously in a clinical environment? How do we develop large collaborative, standardized datasets for external validation of AI/ 25 98 6.53 20 CAD systems? Could AI/CAD polyp detection and characterization systems distract endoscopists and impair 26 97 6.47 13 performance? How do we best train users/clinicians to critically evaluate the AI/CAD system including aware- 27 92 6.13 13 ness of limitations to safeguard against incorrect AI/CAD decisions? What is the best type of training data (videos, static images, or both) that should be used for 28 92 6.13 7 developing polyp detection systems? AI, artificial intelligence; CAD, computer-aided diagnosis/detection; ASGE, American Society for Gastrointestinal Endoscopy; PIVI, preservation and incorporation of valuable endoscopic innovations. ▶ Table 3 Final top 10 questions grouped by themes. Theme Questions Performance metrics, clinical What is the optimum clinical trial design to demonstrate efficacy for polyp detection AI/CAD software? trial design, and end points What are the optimal clinical end points for evaluation of AI/CAD? How do we demonstrate that AI/CAD detection systems have an impact on interval colorectal cancer rates? What is the optimum clinical trial design to demonstrate efficacy for polyp characterization (optical diagnosis) AI/ CAD software? Technological developments How do we improve the performance of AI/CAD to detect more challenging and advanced lesions (e. g. subtle flat lesions and sessile serrated lesions)? How do we reduce false-positive rates for detection systems to avoid the user developing “alert fatigue”? How do we optimize CAD/AI so that it can be used in real-time with minimal latency? Clinical adoption and inte- Can we effectively combine polyp detection and characterization into one workflow? gration into endoscopy Data (access, sharing/priva- Can we produce more efficient or automated annotation methods for data to reduce the burden on human experts? cy, curation, and annotation) Regulatory approval How do we make the regulatory approval process more efficient and overcome hurdles? AI, artificial intelligence; CAD, computer-aided diagnosis/detection. that are challenging for AI or omitting low quality images. Discussion Moreover, these studies do not account for the real-world This is the first international collaborative effort to systemati- endoscopist – AI interaction. Ideally, AI technologies should be cally identify the research questions and priorities related to AI evaluated within the intended clinical pathway, reporting pa- in colonoscopy with a particular focus on clinical implementa- tient outcomes as end points [16]. For this reason, questions tion. In this study, an established modified Delphi method was related to prospective evaluation and trial design rank highly used to determine the top 10 ranked research priorities, which in this study. However, retrospective in-silico studies, using were grouped into five broad themes. carefully curated benchmark datasets, may be important for The first theme, clinical trial design and related end points, comparisons of different algorithms and for external validation predominates the list, containing four questions. The majority purposes, particularly as they may allow for a more objective of published studies evaluating AI in colonoscopy are retro- measure of standalone technical performance. spective, evaluating algorithms outside the clinical environ- The top ranked questions include those related to optimum ment, using datasets labelled by endoscopists. These studies trial designs for polyp detection (CADe) and characterization often suffer from selection bias, for example by excluding cases (CADx). To date, among the published trials, there are only five Ahmad Omer F et al. Establishing key research … Endoscopy 2021; 53: 893–901 | © 2020. The Author(s). 897 Original article RCTs for standalone CADe software, four parallel and one tan- The second theme, technological developments, includes dem in design [4, 5, 7, 8, 17], and one prospective CADx trial three questions. The first relates to how we can improve CADe [18]. systems to detect more challenging and advanced lesions. To Many considerations regarding AI trial design are similar to date, the published CADe RCTs have demonstrated a significant the general evaluation of novel endoscopic technologies and increase only in the detection of non-advanced adenomas, as have been discussed in detail elsewhere [19]. There are how- summarized by a recently published meta-analysis [22]. It has ever unique challenges for AI trials. It can be difficult to account long been debated whether the additional detection of non-ad- for the genuine contribution of AI assistance owing to potential vanced lesions actually translates into any reduction in interval operator bias and modification of endoscopist behavior. Some CRC. It is not unreasonable therefore to focus development of studies have used an independent observer, allowing for un- CADe systems to detect advanced lesions, particularly challen- blinding of AI outputs in missed-lesion scenarios, and one dou- ging lesions that may otherwise be overlooked. ble-blind RCT deployed a sham AI system [5]. Such approaches There have been very limited preclinical studies assessing can provide mechanistic insights; however, the definitions in- the ability of CADe to detect sessile serrated lesions (SSLs) and volved in these studies can be highly subjective. Another signif- advanced flat lesions, such as laterally spreading tumors (LSTs). icant issue is that algorithm performance is also dependent on A recent review evaluated the training and test datasets for the quality of the procedure, which can be highly variable. CADe studies with at least 100 lesions, demonstrating that the Therefore, the selection of operators, for example low level or majority of studies did not differentiate across the type of flat high level detectors, should be considered. Furthermore, per- lesions, especially for non-granular LSTs, most likely due to their formance errors can occur for AI models. low population prevalence [23]. Furthermore, retrospective False-positive CADe outputs can be variably defined, often studies and endoscopic datasets may suffer from a selection on the basis of duration or deemed clinical relevance, making bias, containing optimally captured images. Future research direct comparisons between trials difficult. In one trial, the should focus on creating enriched datasets with images of sub- false-positive rate was not reported and instead the resection tle advanced flat lesions and SSLs, particularly in scenarios rate of non-neoplastic lesions was considered, which may be where human perceptual errors can occur. Moreover, prospec- particularly relevant to device safety [8]. tive trials in higher risk patient populations may actually allow CADx models can produce incorrect classifications or be de- us to determine if the use of AI translates to increased detec- signed to provide no output in cases of insufficient confidence. tion of these subtle lesions. For CADx trials, evaluating the impact of AI on clinical workflow The second question within this theme asks how we could will depend upon its position within the clinical decision-mak- potentially reduce the false-positive rates associated with ing process: a second read, concurrent read, or independent di- CADe. False-positive outputs could be problematic by leading agnosis [20, 21]. Special protocols for image acquisition, hand- to “user fatigue.” To date, prospective trials have not suggested ling of poor-quality images, and additional time taken for ana- that false-positive outputs have significantly impacted on lysis are important CADx considerations. workflow. Nevertheless, it would be advantageous to reduce Another challenge for AI deployment is ensuring its general- false-positive outputs. Retraining of algorithms with scenarios izability to new clinical settings and populations. Ideally, exter- that currently lead to false-positives could be a simple mecha- nal validation should occur, with models being evaluated in in- nism, whilst other approaches may include the use of recurrent stitutions where the training data were not collected. neural networks, which have memory and can process tempor- Determining whether CADe systems have an impact on in- al sequences of frames, mimicking the behavior of human terval colorectal cancer (CRC) would likely require long-term endoscopists. Further research on the acceptable false-positive longitudinal follow-up and reliable linkage to cancer registries. rate for endoscopists may be useful but also it should be recog- Given that post-colonoscopy cancer is a relatively rare out- nized that CADe systems are currently designed as “red flag” come, long-term studies would need to be large and well techniques. Dedicated “challenges” or competitions, co-devel- designed to account for the potential confounders. Ideally oped between computer scientists and endoscopists, aimed at long-term outcomes for patients randomized to AI assistance tackling a specific problem have proved beneficial in the past or standard colonoscopy would provide some insight; however, [24]. Such a challenge aimed at addressing false-positives could the associated financial costs of designing an adequately be invaluable in helping to identify state of the art approaches. powered and robust study may be a barrier. The third question within the technical development theme To advance the first theme, dedicated AI endoscopy working highlights the challenge of latency, which refers to the delay groups, ideally created by professional societies, should aim to between the display of an endoscopic image frame and the out- consolidate trial designs and produce robustly defined out- put from the AI system. Minimal latency is crucial, particularly come measures. Recently, an international working group pro- for CADe systems, where real-time highlighting of lesions is re- duced the CONSORT-AI and SPIRIT-AI extensions, aimed speci- quired. The degree of latency could also be a limitation when AI fically at promoting standardized and transparent reporting of is deployed using cloud- or server-based computing. There are AI interventional trials [16]. Our study could be used to address published studies evaluating acceptable levels of latency for the additional challenges specific to AI in colonoscopy and de- telesurgery [25]. Further similar research is required to specifi- velop recommendations for the design and reporting of AI trials cally identify acceptable latency levels for endoscopy. in endoscopy. 898 Ahmad Omer F et al. Establishing key research … Endoscopy 2021; 53: 893–901 | © 2020. The Author(s). The theme of clinical adoption and integration into the It is noteworthy that the 11th ranked question, with an equal endoscopy includes one question, which concerns the effective mean score to the 10th ranked question but a lower proportion combination of polyp detection and characterization into one of very high priority scores, belongs to the healthcare econom- workflow. This approach could mitigate the effect of increased ics theme, which is likely to be crucial for widespread imple- detection by CADe systems of diminutive hyperplastic polyps, mentation. The impact of CADe and CADx systems on colonos- particularly in the rectosigmoid, potentially avoiding unneces- copy surveillance intervals and associated financial costs war- sary polypectomies [26]. However, the design of such a system, rants further investigation, as it will likely underpin reimburse- particularly with seamless transition from detection to charac- ment policies. One study has just been published, as an add-on terization of the same lesion, may be challenging. To date, no to a previous CADx clinical trial, which demonstrated that AI prospective study has been published that evaluates a system assistance specifically for a diagnose-and-leave strategy resul- combining both CADe and CADx into one workflow, although ted in significant cost reductions for colonoscopy when consid- demonstrations have been published as a video case report ering public health insurance systems in four countries [33]. and abstract [27, 28]. Future research should specifically ad- There are several limitations to our study. Although we used dress workflow challenges, such as the ability to reliably detect methods based on previously published research priority set- and characterize the same unique polyp when switching from ting exercises, bias can be introduced at different stages of the white light to virtual chromoendoscopy, dealing with instances Delphi process. Questions were consolidated and reformatted when multiple polyps are in view, and preferably avoiding the by the steering committee, which could lead to inadvertent need for manual selection of a region of interest. Additional changes to the underlying theme; however, such changes CADx studies that use only white light to predict histopatholo- were only made where absolutely necessary and efforts were gy would also be valuable as highlighted by a recent preliminary made to preserve the original meaning. study [29]. Another limitation relates to the sample size of experts: al- There is one question in the data theme that relates to devel- though it was acceptable for a Delphi study, the group was rela- oping more efficient or automated annotation methods to re- tively small owing to the specific selection of translational re- duce the burden on human experts. Currently, most AI algo- searchers currently involved in AI implementation. This was mi- rithms are developed using a fully supervised learning ap- tigated to some extent by allowing an unlimited number of proach. This requires manual annotation of large numbers of questions to be generated, leading to a comprehensive and endoscopic image frames. This can be incredibly time-consum- thematically diverse long-list. Furthermore, whilst there was ing and expensive, particularly in medical applications where clear discrimination of the very top priorities, the remaining domain expertise is required. Automated or semi-automated question scores were narrowly distributed, possibly because of annotation strategies based on machine learning approaches the small sample size. Owing to rapid growth in the field, our that mimic human annotators are promising areas for future re- findings could now be validated with a larger group of transla- search [30]. Moreover, research aimed at active learning ap- tional AI researchers. The creation of a database of translational proaches, where algorithms iteratively determine which unla- AI researchers, perhaps by dedicated working groups within beled data samples should be annotated by the human could professional societies, would assist validation, reduce potential dramatically improve efficiency. Dedicated computer vision selection bias, and also benefit future collaborative research in competitions or “challenges” for endoscopic video labelling the field. could help accelerate progress in this area further. The creation It is also important to emphasize that the top 10 priorities of datasets for this purpose requires careful co-development were identified by an expert group who are involved in transla- between endoscopists and computer scientists. A recent publi- tional research and focused on advancing clinical implementa- cation provided an overview of existing endoscopic datasets tion at this point in time. The priorities include likely short-term available for AI research, highlighting that few exist and the barriers, largely related to AI evaluation and technical issues, majority are relatively small [31]. that could soon be addressed. Therefore, repeating the exercise The final theme and question in our top 10 priorities concerns in 5 years’ time would be valuable. Furthermore, inclusion of a improving the efficiency of the regulatory approval process. It is wider range of stakeholders, including endoscopists not in- generally accepted that AI-based technologies can differ from volved in AI development, patients, public health researchers, traditional software as a medical device (SaMD). A recent review and ethicists, could have resulted in a different ranking of top article provided an overview of regulatory pathways in relation priorities. to gastrointestinal endoscopy [32]. Current regulatory approval In conclusion, this is the first reported international research pathways for AI are evolving and the associated uncertainty priority setting exercise for AI in colonoscopy. Although specific could delay clinical translation. Regulatory pathways differ glob- attention was given to colonoscopy, the majority of the themes ally, although the International Medical Device Regulators For- and key research questions will apply to the use of AI in general um is a voluntary group that develops harmonized principles for endoscopic practice. The results from this study provide a com- SaMD. Clearly a balance must be achieved between promoting prehensive framework to stimulate further discussions and col- innovation and ensuring patient safety. It is possible that greater laborative research amongst the key stakeholders involved in AI collaboration between regulators and other stakeholders, in- implementation, with a view to accelerating the translation of cluding AI developers and clinicians, may lead to more stream- effective AI systems in endoscopy. lined pathways for clinical translation. Ahmad Omer F et al. Establishing key research … Endoscopy 2021; 53: 893–901 | © 2020. The Author(s). 899 Original article Acknowledgments References J.E. East was funded by the National Institute for Health Re- [1] Kudo S, Mori Y, Misawa M et al. Artificial intelligence and colonosco- search (NIHR) Oxford Biomedical Research Centre. The views py: Current status and future perspectives. Dig Endosc 2019; 31: expressed are those of the author(s) and not necessarily those 363–371 of the National Health Service, the NIHR, or the Department of [2] Ahmad OF, Soares AS, Mazomenos E et al. Artificial intelligence and Health. R. Bisschops and T. Eeldbode are supported by a grant computer-aided diagnosis in colonoscopy: current evidence and fu- from Research Foundation Flanders (FWO). P. Valdastri was ture directions. Lancet Gastroenterol Hepatol 2019; 4: 71–80 supported in part by the Royal Society, by the Engineering and [3] Mori Y, Kudo S, Berzin TM et al. Computer-aided diagnosis for colo- Physical Sciences Research Council under Grant EP/P027938/1. noscopy. Endoscopy 2017; 49: 813–819 [4] Wang P, Berzin TM, Glissen Brown JR et al. Real-time automatic de- Funding tection system increases colonoscopic polyp and adenoma detection rates: a prospective randomised controlled study. Gut 2019; 68: 1813–1819 Wellcome Trust 203145Z/16/Z [5] Wang P, Liu X, Berzin TM et al. Effect of a deep-learning computer-ai- ded detection system on adenoma detection during colonoscopy Engineering and Physical Sciences Research Council 203145Z/16/Z (CADe-DB trial): a double-blind randomised study. Lancet Gastroen- EP/P027938/1 terol Hepatol 2020; 5: 343–351 [6] Su J-R, Li Z, Shao X-J et al. Impact of a real-time automatic quality control system on colorectal polyp and adenoma detection: a pro- spective randomized controlled study (with videos). Gastrointest En- dosc 2020; 91: 415–424.e4 Competing interests [7] Liu W-N, Zhang Y-Y, Bian X-Q et al. Study on detection rate of polyps and adenomas in artificial-intelligence-aided colonoscopy. Saudi J Y. Mori has received consultancy and lecture fees from Olympus Cor- Gastroenterol 2020; 26: 13–19 poration. M. Misawa has received consultancy and lecture fees from [8] Repici A, Badalamenti M, Maselli R et al. Efficacy of real-time com- Olympus Corporation. T. Berzin has received consultancy fees from puter-aided detection of colorectal neoplasia in a randomized trial. Wision AI, Medtronic, and Boston Scientific. M.F. Byrne is CEO and Gastroenterology 2020; 159: 512–520.e7 shareholder of Satisfai Health; founder of AI4GI joint venture, a co-de- velopment agreement between Olympus America and AI4GI in artifi- [9] He J, Baxter SL, Xu J et al. The practical implementation of artificial cial intelligence and colorectal polyps. J.E. East is on the clinical advi- intelligence technologies in medicine. Nat Med 2019; 25: 30–36 sory boards of Boston Scientific, Lumendi (2017 – 2019), and Satisfai [10] Langlotz CP, Allen B, Erickson BJ et al. A roadmap for foundational re- Health (2020), and has received speaker’s fees from Falk. W.E. Karnes search on artificial intelligence in medical imaging: From the 2018 is co-founder and chief medical officer of DocBot. A. Repici has re- NIH/RSNA/ACR/The Academy Workshop. Radiology 2019; 291: 781– ceived consultancy fees and is on the advisory board for Medtronic; 791 he has received consultancy fees, research support, and is on the ad- [11] Vinsard DG, Mori Y, Misawa M et al. Quality assurance of computer- visory board for FujiFilm. M.B. Wallace has received consultancy fees aided detection and diagnosis in colonoscopy. Gastrointest Endosc from Virgo Inc., Cosmo/Aries Pharmaceuticals, Anx Robotica (2019), 2019; 90: 55–63 Covidien, and GI Supply; he has received research grants from Fuji- [12] Rees CJ, Ngu WS, Regula J et al. European Society of Gastrointestinal film, Boston Scientific, Olympus, Medtronic, Ninepoint Medical, and Endoscopy – Establishing the key unanswered research questions Cosmo/Aries Pharmaceuticals; he also has stock/stock options in Vir- within gastrointestinal endoscopy. Endoscopy 2016; 48: 884–891 go Inc.; he has provided consulting on behalf of Mayo Clinic to GI Sup- ply (2018), Endokey, Endostart, Boston Scientific, and Microtek; he [13] Francis N, Kazaryan AM, Pietrabissa A et al. A research agenda for the has also received general payments/minor food and beverage from European Association for Endoscopic Surgeons (EAES). Surg Endosc Synergy Pharmaceuticals, Boston Scientific, and Cook Medical. D. 2017; 31: 2042–2049 Stoyanov is a shareholder in Odin Vision and Digital Surgery. L.B. Lo- [14] Burt CG, Cima RR, Koltun WA et al. Developing a research agenda for vat is a minor shareholder and is on the advisory board of Odin Vision; the American Society of Colon and Rectal Surgeons: Results of a Del- he has received research grants from Medtronic, Pentax Medical, and phi approach. Dis Colon Rectum 2009; 52: 898–905 DynamX; he is also on the advisory board for DynamX and Ninepoint [15] Hart AL, Lomer M, Verjee A et al. What are the top 10 research ques- Medical. O.F. Ahmad, S. Kudo, J.T. Anderson, J. Bernal, R. Bisschops, P. tions in the treatment of inflammatory bowel disease? A priority set- Chen, T. Eelbode, D.S. Elson, S.R. Gurudu, A. Histace, R. Singh, P.Val- ting partnership with the James Lind Alliance J Crohns Colitis 2017; dastri and P.Wang declare that they have no conflict of interest. 11: 204–211 [16] Liu X, Cruz Rivera S, Moher D et al. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Nat Med 2020; 26: 1364–1374 [17] Wang P, Liu P, Glissen Brown JR et al. Lower adenoma miss rate of computer-aided detection-assisted colonoscopy vs routine white- light colonoscopy in a prospective tandem study. Gastroenterology 2020; 159: 1252–1261.e5 [18] Mori Y, Kudo S, Misawa M et al. Real-time use of artificial intelligence in identification of diminutive polyps during colonoscopy: A prospec- tive study. Ann Intern Med 2018; 169: 357–366 [19] Hassan C, Repici A. The FUSE enigma: Wide-angle or wide-minded? Gastrointest Endosc 2018; 88: 865–867 900 Ahmad Omer F et al. Establishing key research … Endoscopy 2021; 53: 893–901 | © 2020. The Author(s). [20] East JE, Rees CJ. Making optical biopsy a clinical reality in colonoscopy. [27] Mori Y, Kudo S-E, Misawa M et al. Simultaneous detection and char- Lancet Gastroenterol Hepatol 2018; 3: 10–12 acterization of diminutive polyps with the use of artificial intelligence [21] Bisschops R, East JE, Hassan C et al. Advanced imaging for detection during colonoscopy. VideoGIE 2019; 4: 7–10 and differentiation of colorectal neoplasia: European Society of Gas- [28] Guizard N, Ghalehjegh SH, Henkel M et al. Artificial intelligence for trointestinal Endoscopy (ESGE) Guideline – Update 2019. Endoscopy real-time multiple polyp detection with identification, tracking, and 2019; 51: 1155–1179 optical biopsy during colonoscopy. Gastroenterology 2019; 156: [22] Barua I, Vinsard D, Jodal H et al. Artificial intelligence for polyp de- S48–S49 tection during colonoscopy: a systematic review and meta-analysis. [29] Sánchez-Montes C, Sánchez FJ, Bernal J et al. Computer-aided predic- Endoscopy 2020: doi:10.1055/a-1201-7165 tion of polyp histology on white light colonoscopy using surface pat- [23] Hassan C, Bhandari P, Antonelli G et al. Artificial intelligence for non- tern analysis. Endoscopy 2019; 51: 261–265 polypoid colorectal neoplasms. Dig Endosc 2020: doi:10.1111/ [30] Lutnick B, Ginley B, Govind D et al. An integrated iterative annotation den.13807 technique for easing neural network training in medical image analy- [24] Bernal J, Tajkbaksh N, Sanchez FJ et al. Comparative validation of sis. Nat Mach Intell 2019; 1: 112–119 polyp detection methods in video colonoscopy: Results from the [31] Borgli H, Thambawita V, Smedsrud PH et al. HyperKvasir, a compre- MICCAI 2015 Endoscopic Vision Challenge. IEEE Trans Med Imaging hensive multi-class image and video dataset for gastrointestinal 2017; 36: 1231–1249 endoscopy. Sci Data 2020; 7: 283 [25] Xu S, Perez M, Yang K et al. Determination of the latency effects on [32] Ahmad OF, Stoyanov D, Lovat LB. Barriers and pitfalls for artificial in- surgical performance and the acceptable latency levels in telesurgery telligence in gastroenterology: Ethical and regulatory issues. Tech In- using the dV-Trainer simulator. Surg Endosc 2014; 28: 2569–2576 nov Gastrointest Endosc 2020; 22: 80–84 [26] Byrne MF, Chapados N, Soudan F et al. Real-time differentiation of [33] Mori Y, Kudo S, East JE et al. Cost savings in colonoscopy with artificial adenomatous and hyperplastic diminutive colorectal polyps during intelligence-aided polyp diagnosis: an add-on analysis of a clinical analysis of unaltered videos of standard colonoscopy using a deep trial (with video). Gastrointest Endosc 2020; 92: 905–911.e1 learning model. Gut 2019; 68: 94–100 Ahmad Omer F et al. Establishing key research … Endoscopy 2021; 53: 893–901 | © 2020. The Author(s). 901