Graefe's Archive for Clinical and Experimental Ophthalmology https://doi.org/10.1007/s00417-022-05653-2 BASIC SCIENCE Validation of an autonomous artificial intelligence–based diagnostic system for holistic maculopathy screening in a routine occupational health checkup context Octavi Font1 · Jordina Torrents‑Barrena2 · Dídac Royo1 · Sandra Banderas García3,4 · Javier Zarranz‑Ventura5,6 · Anniken Bures1,7 · Cecilia Salinas1,7 · Miguel Ángel Zapata1,4 Received: 4 June 2021 / Revised: 15 March 2022 / Accepted: 31 March 2022 © The Author(s) 2022 Abstract Purpose This study aims to evaluate the ability of an autonomous artificial intelligence (AI) system for detection of the most com- mon central retinal pathologies in fundus photography. Methods Retrospective diagnostic test evaluation on a raw dataset of 5918 images (2839 individuals) evaluated with non-mydriatic cameras during routine occupational health checkups. Three camera models were employed: Optomed Aurora (field of view — FOV 50º, 88% of the dataset), ZEISS VISUSCOUT 100 (FOV 40º, 9%), and Optomed SmartScope M5 (FOV 40º, 3%). Image acquisition took 2 min per patient. Ground truth for each image of the dataset was determined by 2 masked retina specialists, and disagreements were resolved by a 3rd retina specialist. The specific pathologies considered for evaluation were “diabetic retinopathy” (DR), “Age- related macular degeneration” (AMD), “glaucomatous optic neuropathy” (GON), and “Nevus.” Images with maculopathy signs that did not match the described taxonomy were classified as “Other.” Results The combination of algorithms to detect any abnormalities had an area under the curve (AUC) of 0.963 with a sensitivity of 92.9% and a specificity of 86.8%. The algorithms individually obtained are as follows: AMD AUC 0.980 (sensitivity 93.8%; specificity 95.7%), DR AUC 0.950 (sensitivity 81.1%; specificity 94.8%), GON AUC 0.889 (sensitivity 53.6% specificity 95.7%), Nevus AUC 0.931 (sensitivity 86.7%; specificity 90.7%). Conclusion Our holistic AI approach reaches high diagnostic accuracy at simultaneous detection of DR, AMD, and Nevus. The integration of pathology-specific algorithms permits higher sensitivities with minimal impact on its specificity. It also reduces the risk of missing incidental findings. Deep learning may facilitate wider screenings of eye diseases. Keywords Artificial intelligence · Screening · Retinography · Diabetic retinopathy · Age-related macular degeneration * Sandra Banderas García sandrabanderasgarcia@gmail.com 4 Ophthalmology Department Hospital Vall d’Hebron, 1 Barcelona, Spain Optretina Image Reading Team, Barcelona, Spain 5 2 Institut Clinic of Ophthalmology (ICOF), Hospital Clinic, BCN MedTech, Department of Information Barcelona, Spain and Communication Technologies, Universitat Pompeu 6 Fabra, Barcelona, Spain Institut d’Investigacions Biomediques August Pi I Sunyer 3 (IDIBAPS), Barcelona, Spain Facultat de Cirurgia i Ciències Morfològiques, Universitat 7 Autònoma de Barcelona (UAB), Barcelona, Spain Instituto de Microcirugía Ocular (IMO), Barcelona, Spain 13 Vol.:(0123456789) Graefe's Archive for Clinical and Experimental Ophthalmology Key messages High diagnostic accuracy in artificial intelligence (AI) system for one pathology in ophthalmology has been achieved but still to be acquired for screening purposes. A combination algorithm made of several specific-pathology ones reached high sensitivity and specificity for diabetic retinopathy, age macular degeneration, glaucomatous optic neuropathy and nevus. This holistic system could successfully be used in large-population screening programs at working-age. deep learning (DL) [25] represents a breakthrough that has Introduction dramatically improved the state-of-the-art in many tasks such as speech recognition, image processing, and text In 2010, 65% of those afflicted by blindness worldwide generation, among others [26]. In the field of medicine, (32.4 million total) and 76% of those with moderate or DL has been most successful in medical imaging analysis, severe vision impairment (191 million) had a preventable by enabling the creation of computer-aided diagnosis sys- or treatable cause [1]. The causes vary among regions, but tems (CADx) with expert-level accuracies. There are many the trend since 1990 shows a decreased incidence due to examples in dermatology [27], radiology [28], gastroenter- cataract or refractive errors while age macular degenera- ology [29], and ophthalmology [30, 31]. In fact, the first tion (AMD), glaucoma, and diabetic retinopathy (DR) are ever US Food and Drug Administration (FDA)-approved on the rise [1]. In developed countries, this trend is even autonomous AI is a screening tool for DR [32]. more pronounced: AMD is the leading cause of blindness in While the results of the AI performance presented in many people aged 75 years and older [2], whereas DR is the most publications are encouraging, there are still questions to be frequent cause of preventable blindness in the working-age answered regarding their real-world application. The vast population (adults between 20 and 74 years old) [3]. Future majority of publications are limited to retrospective studies projections do not show any sign of these diseases slowing taken on datasets captured in populations with prior condi- down either. AMD will affect 288 million people in 2040 tions in a hospital setting [33–35]. Also, most publications [4], glaucoma will impact 111.8 million in 2040 [5], and 191 limit the scope of their algorithm to just one pathology which, million people will suffer DR by 2030 [6]. These diseases while interesting, is not ideal for screening purposes [36]. are treatable with good outcomes if detected early in the This work aims to overcome both limitations, by pre- course of the disease [7–10], but they often are not sympto- senting a retrospective study performed on 2839 patients matic until late stages of their development. Thus, it is essen- evaluated by digital fundus images taken with handheld tial to have good screening systems for a timely diagnosis. non-mydriatic cameras, on a routine checkup performed Running screening programs at large scale is costly, being onsite at work centers. The algorithm evaluated in this the relatively high fixed costs per equipment the main driver study has previously been successful in detecting signs of the charge [11]. This is even more noticeable in low- of DR, AMD, glaucoma, and nevus, the most common density areas, which are underserved by traditional screen- eye pathologies [24]. The novelty of our proposal is the ing approaches performed in primary-care settings [12]. combination of multiple pathology-specific algorithms to Additional issues of such programs stem from their limited achieve holistic maculopathy detection. Each algorithm is scope. Primary care physicians, with limited ophthalmologi- trained to identify individual diseases, and, in conjunction, cal expertise, might often miss abnormalities outside the the final output increases the diagnostic accuracy of the AI original screening program or have lower sensitivity than system for ocular pathology detection. retinal experts [13, 14]. Ophthalmology is also a leading specialty in alternative forms of healthcare delivery. For instance, mobile digital non-mydriatic cameras are getting more affordable and have good specificity and sensitivity for DR [15, 16], which has enabled many screening plans in Materials and methods underserved areas [17–20]. There are also many examples of successful telemedicine screening plans in countries such as Study population Australia, USA, India, Singapore, and Spain [21–24]. We present a retrospective diagnostic test evaluation. In parallel to the improvements in imaging and digitali- The dataset consists of 5918 images from a population zation of healthcare, artificial intelligence (AI) based on 13 Table 1  Breakdown of the study validation datasets, as well as the training datasets for each of the AI algorithms that compose the screening system. (M/F male/female, LE/RE left eye/right eye) Dataset n participants (M/F) Age median (std) n eyes (LE/RE) n images Abnormal (%) AMD (%) DR (%) GON (%) Nevus (%) Other (%) (range) Study validation Original 2839 (1786 M/1053 43 (11.52) [5–87] 5483 (2747 5918 321 (5.42%) 14 (0.23%) 0 (0%) 107 (1.81%) 110 (1.86%) 90 (1.5%) F) LE/2736 RE) Origi- 3337 (1999 M/1338 46 (15.22) [5–96] 6009 (3013 6452 855 (13.25%) 398 (6.17%) 150 (2.32%) 107 (1.66%) 110 (1.7%) 90 (1.4%) Graefe's Archive for Clinical and Experimental Ophthalmology nal + enriched F) LE/2996 RE) Algorithm training AMD – 75 (11.52) [5–98] – 7218 – 4859 (67.31%) – – – – (982 M/1526 F) (1761 LE/1945 RE) DR 71,455 66 (11.42) [11–99] 116,501 (60,947 139,813 – – 14,376 (10.28%) – – – (26,691 M/19,296 LE/55,554 RE) F) Glaucoma 1206 (619 M/587 F) 54 (15.58) [5–96] 1738 (794 LE/944 2366 – – – 1168 (49.37%) – – RE) GON 18,750 42 (11.64) [5–96] 29,352 (15,633 30,054 – – – – 4470 (14.87%) – (8.215 M/10,535 LE/13,719 RE) F) Abnormality 31,877 49 (17.33) [5–99] 52,791 (27,846 53,194 17,433 (32.77%) – – – – – (13,843 M/18,034 LE/24,945 RE) F) a The total number of participants as well as the total number of eyes for the AMD training dataset could not be obtained, since the non-pathological images used for training were not assigned to a patient. The reported numbers (M/F, LE/RE, age) are from the pathological cases, which were referenced to a patient b The sex split and age median only accounts for 65% of the dataset. One of the image sources for this dataset did not include sex nor age information per patient 13 Graefe's Archive for Clinical and Experimental Ophthalmology of 2839 individuals, taken between the 9th of January Digital Fundus Image evaluation and the 13th of March of 2020. The median age was 43 years old with a standard deviation of 11.52. From The ground truth of the data was evaluated per eye. For the study population, 1786 (63%) were male and 1053 patients with multiple captures, an automated quality filter- female (37%) (see Table 1 for a detailed breakdown). ing was employed to select the highest-quality image. After- Participants of this study were enrolled in a consecu- ward, each image was graded by 2 specialists (intragrader tive series during routine occupational health checkups variability kappa of 0.86 and 0.79 respectively) in a 2-tiered offered by their employer as medical benefits [37]. All approach (Fig. 1). In case of discrepancies, a 3rd retinal spe- ophthalmologic check-ups were performed by a single cialist reviewed the image (intragrader variability kappa of provider (Optretina, Sant Cugat, Spain). The images 0.83). The first step of the labeling process was to classify were obtained by a trained technician using handheld the image as “normal” or “abnormal,” considering the lat- non-mydriatic cameras on the participating center office ter as any digital fundus image showing pathological signs. premises, in a room which had been setup with adequate Abnormal images were further subclassified per pathology. lighting conditions. The camera models employed were The specific pathologies considered for evaluation were DR Optomed Aurora (field of view — FOV 50º, 88% of the (defined as more than mild DR, as per the 2019 revision dataset), ZEISS VISUSCOUT 100 (FOV 40º, 9% of the of the American Academy of Ophthalmology’s Preferred dataset), and Optomed SmartScope M5 (FOV 40º, 3% Practice Pattern) [38, 39], AMD (defined as mild or worse), of the dataset). Image acquisition took around 2 min per GON (suspicious glaucomatous optic neuropathy was patient. The raw image dataset was included in the study, defined by a cup-to-disc ratio of 0.7 or more in the vertical and no images were discarded due to low resolution or axis and/or other typical changes caused by glaucoma, such were modified prior to the analysis. as localized notches or retinal nerve fiber layer defects or Fig. 1  Labeling flowchart. The flowchart depicts the 2-tiered approach followed by all spe- cialists to label the dataset. The ground truth was agreed by at least 2 graders 13 Graefe's Archive for Clinical and Experimental Ophthalmology peripapilar hemorrhages), and Nevus (defined with clinical For the sample size calculation, we estimated a prevalence parameters as an hyperpigmented lesion beneath the retina). of retinal abnormalities in an occupational health checkup Images classified as abnormal (with possible signs of macu- context of 7.8% with a 95% confidence interval, as per our lopathy) not matching the described taxonomy were classi- previous study [42]. With these figures, the total number fied as other in tier 2. of participants needed was 2784. Additionally, we also confirmed that the sample size of our enriched dataset was large enough to ensure 80% statistical power (β = 0.2) on our Dataset enrichment sensitivity and specificity metrics, given the reported null hypothesis and the levels of pathological prevalence [43]. We aim to assess the effectiveness of our automated screening algorithm on a wide-range general population. Because of the Training dataset sampling bias of the initial population (working age partici- pants, mostly without known prior pathologies), the prevalence For algorithm development, macula-centered digital fun- for AMD and DR was far below that reported in the literature dus images were retrospectively obtained from Optreti- for the general population [4, 40]. To balance the data, we na’s own image bank (AMD, GON, Nevus, Abnormal- enriched the dataset with 384 AMD and 150 DR pathological ity) and Institut Català de la Salut and EyePacs (Kaggle). images to match the prevalence in the general population of For AMD, Glaucoma, and DR, images were taken from a our environment [18, 41]. AMD images were obtained from clinical setting, while Nevus and abnormality images were Optretina’s image bank (the sample was randomly selected sourced from screenings, mostly performed with portable from a cohort of 2212 AMD cases screened from January cameras. All images were evaluated by at least 1 expert 2013 to May 2020) [24]. DR images were randomly selected retinologist, following the previously described criteria. from a series of positive cases detected in the Institut Català The exact breakdown of the training dataset can be found de la Salut (ICS) screening program for diabetics (Barcelona, in Table 1. Spain). In both cases, the enriched images were labeled by two expert retinal specialists, following the procedure detailed in Individual algorithms Fig. 1. The dataset details are shown in Table 1. For each dataset, we trained binary classificators (disease/ Statistical analysis no disease) using convolutional neural networks (CNNs). This process, with the right training data, allows the CNN The primary outcome of the analysis is the diagnostic to automatically learn features from the images that can be accuracy of the AI system, defined by its sensitivity extrapolated successfully outside of the training data. The and specificity, versus the ground truth. Since the AI “AMD” algorithm uses a custom neural network architec- system performs a holistic screening, as well as pathol- ture [42] using RGB images of 512 × 512 pixels [36]. The ogy-specific diagnostic, we calculated the sensitivity “DR” algorithm uses an InceptionV3 architecture [44] with and specificity for both. The operating threshold was inputs of 512 × 512.44, “Glaucoma” uses a ResNet50 [45] fixed before the analysis and was not adjusted during the with inputs of 224 × 224.45, “Nevus” detection employs an tests. The secondary outcomes are the receiver operating InceptionV3 at 299 × 299, and the abnormal images detector characteristic (ROC) curve and the area under the curve another InceptionV3 at 299 × 299. The optimization algo- (AUC) index. All reported 95% CIs were obtained by rithm to train the network was ADAM. We also used batch performing a non-parametric bootstrap (1000 samples, normalization, as well as using the weights of pretrained with replacement). Study success was defined as reach- ImageNet networks where possible (InceptionV3, ResNet50) ing a predefined threshold of sensitivity and specificity to speed up the training. on our holistic general screening algorithm. The hypoth- The performance of the algorithm was measured by eses of interest were the area under the receiver operating curve (AUC). The reported sensitivity and specificity points have been taken H0 ∶ p < p0vsHA ∶ p ≥ p0 without adjusting the decision threshold (threshold = 0.5). where p is the sensitivity or specificity of the AI system. The development datasets were split in an 80/10/10 fash- The predefined sensitivity and specificity thresholds were ion, where 80% of the data was used for training, 10% for p0 = 0.75 and p0 = 0.775, respectively, benchmark defined validation (adjusting hyperparameters), and 10% to test by the FDA in their first-approved AI diagnostic system [32]. the results. This data was split by patient (not image) and A one-sided 2.5% type I error binomial test was performed is completely independent from the dataset presented for for both null hypotheses. the study validation. 13 Graefe's Archive for Clinical and Experimental Ophthalmology Fig. 2  Algorithm execution flowchart. The predictions are performed at the image level. 5 neural networks process independently each image and in case any algorithm is positive, the screened image is classified as “abnormal” 13 Graefe's Archive for Clinical and Experimental Ophthalmology Screening algorithm of Nevus images (54/110), and 49% of other maculopathies images (44/90). The screening algorithm is a combination of five indepen- The AMD algorithm correctly detected 90% of AMD dently trained neural networks. Four of these neural net- images (358/398). False positive rates were 62.6% of DR works target specific pathologies (AMD, DR, GON, and images (94/150), 6.7% of GON images (7/107), 0% of Nevus Nevus), while the fifth one has been trained as an outlier images (0/110), and 22.5% of other maculopathies images detector, with a training dataset containing images from the (20/90). aforementioned pathologies as well as other undetermined The DR algorithm correctly detected 68.6% DR images maculopathies (Fig. 2). Each image evaluated by the sys- (103/150). False positive rates were 4.1% of AMD images tem is processed independently by each of the five neural (16/398), 0% of GON images (0/107), 0.9% of Nevus images networks and, at a second step, their response is combined (1/110), and 11.7% of other maculopathies images (10/90). in a single output. If an algorithm detects signs of any of the The GON algorithm correctly detected 58.2% of GON individual pathologies, the screened image is classified as images (62/107). False positive rates were 10.6% of AMD Abnormal. A complete diagram of the AI system architec- images (42/398), 12.6% of DR images (19/150), 2.9% of ture is presented in Fig. 2 102 Nevus images (3/110), and 8.8% of other maculopathies images (7/90). The Nevus algorithm correctly detected 88.2% of the Results Nevus-specific images (90/110). False positive rates were 88% of AMD images (350/398), 100% of DR images The combined abnormality algorithm correctly identi- (150/150), 2.9% of GON images (3/107), and 63.7% of other fied 92% of the analyzable images annotated as Abnormal maculopathies images (57/90). (776/843). The performance of individual disease algorithms Sensitivity and specificity obtained from all algorithms is herein described: 99% of AMD images were correctly are summarized in Table 2. The threshold for each individual identified (394/398), 100% of DR images were correctly CNN was not adjusted to boost the sensitivity or specificity to classified (150/150), 71% of GON images were correctly a certain operating point. Sensitivity and specificity were cal- identified (71%, 75/107), 90% of Nevus images were cor- culated per eye, using the best quality image if multiple were rectly identified (99/110), and 73% of undetermined mac- available. Discarding eye duplicates had little effect in the ulopathies were correctly classified (65/90). Insufficient metrics in our study cohort (sensitivity: 92.1% duplicates vs quality of the images was observed in 0.23% of the cases 92.8% non-duplicates; specificity: 87.6% duplicates vs 86.8% (15/6452), which could not be graded. Of these, 80% were no duplicates). Enforcing high-quality standards in the pre- labeled as abnormal by our graders (12/15). processing pipeline, the effect is more noticeable (sensitivity: The single NN (neural network) abnormality algorithm 92.8% any quality vs 92.6% high quality; specificity: 86.8% correctly detected 82% (691/843) of the analyzable images. any quality vs 89.1% high quality). The dataset, as classi- The percentage of images correctly classified disclosed by fied by the automatic quality algorithm, consisted of 53.4% pathologies was 98.8% of AMD images (393/398), 94.6% of high-quality images (n = 3444), 43.6% of acceptable-quality DR images (142/150), 45.6% of GON images (48/107), 49% images (n = 2809), and 3.0% of low-quality images (n = 196). Table 2  Summary of sensitivity, Algorithm Sensitivity (95% CI) [p-value] Specificity (95% CI) [p-value] specificity, and AUC aggregated and per individual algorithm Combined algorithm: Abnormality 92.9% (91.0, 94.6) [< 0.001]a 86.8% (85.8, 87.7) [< 0.001]b Single NN algorithms: Abnormality 83.4% (80.6, 85.9) 93.4% (92.7, 94.0) AMD 93.8% (91.6, 96.3) 95.7% (95.2, 96.2) DR 81.1% (75.3, 88.1) 94.8% (94.1, 95.4) Glaucoma 53.6% (43.1, 63.1) 95.7% (95.2, 96.2) Nevus 86.7% (80.7, 94.0) 90.7% (90.1, 91.5) a p-value for sensitivity on the combined abnormality algorithm was computed using a one-sided tailed binomial test using a sensitivity of p = 0.75 as the null hypothesis b p-value for specificity on the combined abnormality algorithm was computed using a one-sided tailed binomial test using a sensitivity of p = 0.775 as the null hypothesis 13 Graefe's Archive for Clinical and Experimental Ophthalmology Fig. 3  Receiver operating curve for the combined and individual algorithms To represent the best operation points, we plotted the treatment, if necessary, can prevent visual loss in a very ROC curve for the combined algorithm and the individual high percentage of patients [47–49]. Simultaneous screening models in Fig. 3. Additionally, we also computed the AUC for multiple pathologies of the retina has previously been of the combined algorithm (0.963) and the single NN ver- contemplated in some publications, both associated with sion (0.948). human reading and artificial intelligence, mainly combining the detection of AMD and DR [14, 50], and also glaucoma [51]. To date, all artificial intelligence studies using fundus Discussion images for other diseases than diabetes have been carried out only in existing databases, with no clinical validation studies The proposed study has exceeded the expectations of the performed prospectively. holistic solution proposed for screening the central retina’s Currently, screening programs in most countries focus diseases. The reported diagnostic accuracy levels are simi- on DR, probably for cost-effectiveness reasons. Our study lar to other algorithms already available on the market and population is relatively young, and a priori, healthy, despite higher than those required by the FDA for approval [32]. previous studies that report alterations in fundus images in However, the proposed system presents the additional ben- almost 8% of cases in this type of population [37]. Although efit that several pathologies are simultaneously screened these alterations do not usually represent serious or urgent together with those that cause greater visual loss among cases, any pathology in this population, young and work- industrialized countries. These are also the main causes of ing, can have significant socio-economic repercussions. The preventable, non-reversible blindness, which are experienc- incorporation of artificial intelligence and the simultaneous ing more growth in the world [46]. screening of several diseases can make these early detection It is widely demonstrated that early detection of these systems more cost-effective. Despite the fact that the objec- diseases (mainly AMD, DR, and GON) and their early tive of this study is not an economic evaluation, the use of 13 Graefe's Archive for Clinical and Experimental Ophthalmology automatic detection software can reduce previously reported screening, but a better screening workflow with hybrid sys- costs, lower than 10 euros per patient [37]. tems. We believe that by integrating the image acquisition One of the most important causes of loss of effective- process with an online platform for automated data collec- ness of AI is related to the quality of the images [52]. In our tion, it is possible to instrument the whole process and guide series, we have had 3% of fundus photographs of low qual- the technician through, with the additional benefit that the ity, a figure significantly lower than other published series images are automatically assigned to the right patient and such as Abramoff et al. (above 8%) [32] or Liu et al. (16%) checked for adequate quality prior to running any subsequent [53]. We would like to note that our numbers are obtained on diagnostic analysis. real conditions, with portable cameras and, generally, under In conclusion, the use of an autonomous AI-based diag- certainly strict timeframes. This difference can be due to dif- nostic system based on fundus images for holistic macu- ferent reasons. While portable cameras traditionally tended lopathy screening in a routine occupational health checkup to be of lower quality than desktop cameras, the technical context seems effective, with high levels of sensitivity and advances in recent years have improved the quality of the specificity that improves further those achieved by specific images, and current cameras like Aurora are of equivalent algorithms. The application of these systems could allow quality. Moreover, screening has been carried out in rela- more extensive screening programs with greater detection tively young people, in which ocular media opacities are of pathology in working-age patients. much less common and tend to have more dilated pupils in scotopic conditions. One of the most interesting points is the study of comorbidities; to date, it is also one of the limits set Author contribution All the listed authors meet the criteria for the authorship according to the ICMJE guidelines. by AI. With this type of approach, lesions can be detected by All persons who meet the authorship criteria are listed as authors, independent algorithms that combine together in a holistic and all authors certify that they have participated sufficiently in the diagnosis. This results in a more robust system, less likely work to take public responsibility for the content, including participa- to miss incidental findings, with higher overall sensitivity, tion in the concept, design, analysis, writing, or revision of the manu- script. Furthermore, each author certifies that this material or similar while only penalizing slightly on specificity. The use of AI material has not been and will not be submitted to or published in any for DR screening is already being implemented success- other publication before its appearance in the Graefe’s Archive for Clin- fully in some countries. However, this pathology-specific ical and Experimental Ophthalmology. O. Font, MA. Zapata, and D. approach carries the risk of ignoring other possible findings, Royo made a substantial contribution to the present work in the design of the study, analysis and interpretation of data, manuscript drafting, since the neural networks employed are not designed for it. and approval of the version of the manuscript to be published. MA. We do believe that it would be beneficial if those IA systems Zapata, A. Bures, and C. Salinas had contributed to acquisition of data, are combined and set up with a more holistic approach, to conception and design of the study, critical revision for important intel- minimize the risk of ignoring these incidental findings. The lectual content, and approval of the version to be published. O. Font, J. Zarranz-Ventura, and S. Banderas had an important participation in combination of multiple algorithms also makes it easier to the conception of the study, drafting of the paper, and approval of the deploy improvements on the system. We can tackle algo- present version of the manuscript. Edition of the manuscript was taken rithms per pathology, and any improvements in the individ- by MA. Zapata and S. Banderas. J. Torrents and J. Zarranz-Ventura’s ual models will benefit in the final output. We have already contribution to the present work included furthermore critical revision for important intellectual content and approval of the present version. achieved gains in multiple retrainings of the DR algorithm, All the authors agree to be accountable for all aspects of the work in and we believe that the rest of algorithms could be similarly ensuring its accuracy and integrity. improved in the near future. GON algorithm alone shows weaker performance than the rest of single algorithms, Funding Open Access Funding provided by Universitat Autonoma de making the single and combined algorithms not reliable for Barcelona. screening of this pathology. Data availability Any information not included in the present manu- This study’s limits are those determined by carrying out script is available upon request by contacting the corresponding author. a retrospective study, those related to the population studied (in this case, younger and with a lower rate of pathology Code availability The AI system described in this study is copyrighted than the general population), and the limits derived from by Optretina S.L. and consequently the underlying source code is not the pathology studied. To compensate for possible biases, available. the database has been enriched with a presence similar to the population of AMD and DR. It would be convenient, Declarations in the future, to introduce other highly prevalent patholo- Ethics approval Optretina is a company supervised by the ethics com- gies in the population, such as the presence of epiretinal mittee of the Vall d’Hebron Hospital in Barcelona, who approved this membranes or macular signs associated with high myopia. clinical trial (Study code PR(OPT)370/2019). Another area that we want to study further is in the image capture workflow, to offer not only an automated way of 13 Graefe's Archive for Clinical and Experimental Ophthalmology Consent to participate Patient’s informed consent was exempted 12. Lee SJ, McCarty SC, Taylor HR, Keeffe JE (2008) Costs of mobile because of the retrospective nature of the study, using fully anonymized screening for diabetic retinopathy: a practical framework for rural retinal images. populations. Aust J rural Health 9:186–192. https://​doi.​org/​10.​ 1046/j.​1038-​5282.​2001.​00356.x 13. Farley TF, Mandava N, Prall FR, Carsky C (2008) Accuracy Conflict of interest MA. Zapata is the founding member and medical of primary care clinicians in screening for diabetic retinopathy director of Optretina. D. Royo is founding partner and technical direc- using single-image retinal photography. Ann Fam Med 6:34–428. tor of the company. O. Font is employed in Optretina and A. Bures and https://​doi.​org/​10.​1370/​afm.​857 C. Salinas work as fundoscopy reader in the company. The authors 14. Chan CKW, Gangwani RA, McGhee SM et al (2015) Cost- declare no other direct financial conflicts of interest as individuals. effectiveness of screening for intermediate age-related macular degeneration during diabetic retinopathy screening. Ophthalmol- Open Access This article is licensed under a Creative Commons Attri- ogy 122:85–2278. https://​doi.​org/​10.​1016/j.​ophtha.​2015.​06.​050 bution 4.0 International License, which permits use, sharing, adapta- 15. Ahmed J, Ward TP, Bursell SE et al (2006) The sensitivity and tion, distribution and reproduction in any medium or format, as long specificity of nonmydriatic digital stereoscopic retinal imaging in as you give appropriate credit to the original author(s) and the source, detecting diabetic retinopathy. Diabetes Care 29:9–2205. https://​ provide a link to the Creative Commons licence, and indicate if changes doi.​org/​10.​2337/​dc06-​0295 were made. The images or other third party material in this article are 16. Massin P, Erginay A, Mehidi AB et al (2003) Evaluation of a new included in the article's Creative Commons licence, unless indicated non-mydriatic digital camera for detection of diabetic retinopa- otherwise in a credit line to the material. If material is not included in thy. Diabet Med 20:635–641. https://d​ oi.o​ rg/1​ 0.1​ 046/j.1​ 464-5​ 491.​ the article's Creative Commons licence and your intended use is not 2003.​01002.x permitted by statutory regulation or exceeds the permitted use, you will 17. Levy J, Lifshitz T, Goldfarb D, Knyazer B, Belfair N (2011) need to obtain permission directly from the copyright holder. To view a Screening for diabetic retinopathy with a mobile non-mydri- copy of this licence, visit http://​creat​iveco​mmons.​org/​licen​ses/​by/4.​0/. atic digital fundus camera in Southern Israel. Isr Med Assoc J13:40–137 18. Romero-Aroca P, Sagarra-Alamo R, Traveset-Maeso A et al (2010) The non-mydriatic camera as a screening method in dia- betics in Tarragona. Arch Soc Esp Oftalmol 85:232–238. https://​ References doi.​org/​10.​1016/j.​oftal.​2010.​09.​005 19. Spurling GKP, Deborah AA, Hayman NE, Hansar N, Cooney AM, 1. Bourne RAA, Stevens GA, White RA et al (2013) Causes of Jackson CL (2010) Retinal photography for diabetic retinopathy vision loss worldwide, 1990–2010: a systematic analysis. Lancet screening in Indigenous primary health care: the Inala experience. Glob Health 1:339–349. https://d​ oi.o​ rg/1​ 0.1​ 016/S ​ 2214-1​ 09X(13)​ Aust N Z J Public Health 34(SUPPL):30–33. https://​doi.​org/​10.​ 70113-X 1111/j.​1753-​6405.​2010.​00549.x 2. Klaver CCW, Wolfs RC, Vinberling JR et al (1998) Age-specific 20. Beynat JA, Charles A, Astruc K et al (2009) Screening for dia- prevalence and causes of blindness and visual impairment in an betic retinopathy in a rural French population with a mobile non- older population: the Rotterdam study. Arch Ophthalmol 116:623– mydriatic camera. Diabetes Metab 35:49–56. https://​doi.​org/​10.​ 658. https://​doi.​org/​10.​1001/​archo​pht.​116.5.​653 1016/j.​diabet.​2008.​07.​002 3. Mohamed Q, Gillies MC, Wong TY (2017) Management of dia- 21. Kumar S, Tay-Kearney ML, Chaves F, Constable IJ, Yogesan betic retinopathy. JAMA 298:902–916. https://​doi.​org/​10.​1001/​ K (2006) Remote ophthalmology services: Cost comparison of jama.​298.8.​902 telemedicine and alternative service delivery options. J Telemed 4. Wong WL, Su X, Li X et al (2014) Global prevalence of age- Telecare 12:19–22. https://d​ oi.o​ rg/1​ 0.1​ 258/1​ 35763​ 30677​ 53213​ 99 related macular degeneration and disease burden projection for 22. Sharma M, Chakrabarty AS, Rathi P, Sharma RC, Pratibha G 2020 and 2040: a systematic review and metanalysis. Lancet Glob (2011) An integrated, mobile service for diabetic retinopathy in Health 2:e106–e116. https://​doi.​org/​10.​1016/​S2214-​109X(13)​ rural India. Community Eye Health 24:17–18 70145-1 23. Nguyen HV, Tan GSW, Tapp RJ et al (2016) Cost-effectiveness 5. Tham YC, Li X, Wong TY et al (2014) Global prevalence of glau- of a national telemedicine diabetic retinopathy screening program coma and projections of glaucoma burden through 2040: a system- in Singapore. Ophthalmology 123:2571–2580. https://​doi.​org/​10.​ atic review and meta-analysis. Ophthalmology 121:2081–2090. 1016/j.​ophtha.​2016.​08.​021 https://​doi.​org/​10.​1016/j.​ophtha.​2014.​05.​013 24. Zapata MA, Arcos G, Fonossola A et al (2017) Telemedicine for 6. Ting DSW, Ming Cheung GCM, Wong TY (2016) Diabetic retin- a general screening of retinal disease using nonmydriatic fundus opathy: global prevalence, major risk factors, screening practices cameras in optometry centers: three-year results. Telemed J E and public health challenges: a review. Clin Exp Ophthalmol Health 23:30–36. https://​doi.​org/​10.​1089/​tmj.​2016.​0020 44:260–277. https://​doi.​org/​10.​1111/​ceo.​12696 25. Lecun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 7. Ferris FL (1993) How effective are treatments for diabetic retin- 521:436–444. https://​doi.​org/​10.​1038/​natur​e14539 opathy? JAMA-J Am Med Assoc 269:1–129 26. Krizhevsky A, Sutskever I, Hinton GE (2021) ImageNet classifica- 8. Lim LS, Mitchell P, Seddon JM et al (2012) Age-related macular tion with deep convolutional neural networks. Adv Neural Inform degeneration. Lancet 379:1728–1738. https://​doi.​org/​10.​1016/​ Process Syst 25:1–9 S0140-​6736(12)​60282-7 27. Esteva A, Kuprel B, Novoa RA et al (2017) Dermatologist-level 9. Hamilton WS (2007) New treatments for age-related macular classification of skin cancer with deep neural networks. Nature degeneration. Lancet 370:1–1480. https://d​ oi.o​ rg/1​ 0.1​ 016/S​ 0140-​ 542:18–115. https://​doi.​org/​10.​1038/​natur​e21056 6736(07)​61627-4 28. Lakhani P, Sundaram B (2017) Deep Learning at chest radiogra- 10. Jonas JB, Aung T, Bourne RR et al (2017) Glaucoma. Lancet phy: automated classification of pulmonary tuberculosis by using 390:2183–2193. https://​doi.​org/​10.​1016/​S0140-​6736(17)​31469-1 convolutional neural networks. Radiology 284:574–582. https://​ 11. Lairson DR, Pugh JA, Kapadia AS et al (1992) Cost-effectiveness doi.​org/​10.​1148/​radiol.​20171​62326 of alternative methods for diabetic retinopathy screening. Diabetes 29. Urban G, Tripathi P, Alkayali T et al (2018) Deep learning Care 15:77–1369. https://​doi.​org/​10.​2337/​diaca​re.​15.​10.​1369 localizes and identifies polyps in real time with 96% accuracy 13 Graefe's Archive for Clinical and Experimental Ophthalmology in screening colonoscopy. Gastroenterology 155:1069-1078.e8. evaluation, macular degeneration, and suspected glaucoma. Clin https://​doi.​org/​10.​1053/j.​gastro.​2018.​06.​037 Ophthalmol 4:419–429. https://​doi.​org/​10.​2147/​OPTH.​S2357​51 30. Gulshan V, Peng L, Coram M et al (2016) Development and 43. Bujang MA, Adnan TH (2016) Requirements for minimum sam- validation of a deep learning algorithm for detection of diabetic ple size for sensitivity and specificity analysis. J Clin Diagn Res retinopathy in retinal fundus photographs. JAMA 316:2402–2410. 10:YE01–YE06. https://​doi.o​ rg/​10.​7860/​JCDR/​2016/1​ 8129.8​ 744 https://​doi.​org/​10.​1001/​jama.​2016.​17216 44. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking 31. De-Fauw J, Ledsam JR, Romera-Paredes B et al (2018) Clini- the inception architecture for computer vision. https://​doi.​org/​10.​ cally applicable deep learning for diagnosis and referral in reti- 1109/​CVPR.​2016.​308 nal disease. Nat Med 24:1342–1350. https://​doi.​org/​10.​1038/​ 45. Wu S, Zhong S, Liu Y (2018) Deep residual learning for image s41591-​018-​0107-6 steganalysis. Multimed Tools Appl 77:10437–10453. https://​doi.​ 32. Abràmoff MD, Lavin PT, Birch M, Shah N, Folk CJ (2018) Pivotal org/​10.​1007/​s11042-​017-​4440-4 trial of an autonomous AI-based diagnostic system for detection of 46. GBD (2019) Blindness and Vision Impairment Collaborators, diabetic retinopathy in primary care offices. Npj Digital Medicine Vision Loss Expert Group of the Global Burden of Disease Study 1:39. https://​doi.​org/​10.​1038/​s41746-​018-​0040-6 (2021) Causes of blindness and vision impairment in 2020 and 33. Burlina PM, Joshi N, Pekala M et al (2017) Automated grading trends over 30 years, and prevalence of avoidable blindness in of age-related macular degeneration from color fundus images relation to VISION 2020: the right to sight: an analysis for the using deep convolutional neural networks”. JAMA Ophthalmol Global Burden of Disease Study. Lancet Glob Health 9:e144– 135:6–1170. https://​doi.​org/​10.​1001/​jamao​phtha​lmol.​2017.​3782 e160. https://​doi.​org/​10.​1016/​S2214-​109X(20)​30489-7 34. Brown JM, Campbell JP, Beers A et al (2018) Automated diag- 47. AREDS2-HOME Study Research Group, Chew EY, Clemons TE nosis of plus disease in retinopathy of prematurity using deep et al (2014) Randomized trial of a Home Monitoring System for convolutional neural networks. JAMA Ophthalmol 136:10–803. Early Detection of Choroidal Neovascularization Home Moni- https://​doi.​org/​10.​1001/​jamao​phtha​lmol.​2018.​1934 toring of the Eye (HOME) study. Ophthalmology 21:535–544. 35. Ahn JM, Kim S, Ahn KS et al (2018) A deep learning model for https://​doi.​org/​10.​1016/j.​ophtha.​2013.​10.​027 the detection of both advanced and early glaucoma using fundus 48. Ferris FL (1994) Results of 20 years of research on the treatment photography”. PLoS ONE 13:1–8. https://​doi.​org/​10.​1371/​journ​ of diabetic retinopathy. Prev Med 23:740–742. https://​doi.​org/​10.​ al.​pone.​02079​82 1006/​pmed.​1994.​1127 36. Chew EY, Schachat AP (2015) Should we add screening of age- 49. Phu J, Agar A, Wang H, Masselos K, Kalloniatis M (2020) Man- related macular degeneration to current screening programs for agement of open-angle glaucoma by primary eye-care practition- diabetic retinopathy? Ophthalmology 122:6–2155. https://d​ oi.o​ rg/​ ers: toward a personalised medicine approach. Clin Exp Optom. 10.​1016/j.​ophtha.​2015.​08.​007 https://​doi.​org/​10.​1111/​cxo.​13114 37. Zapata MA, Martín R, Garcia-Arumí C, etal (2020) Remote 50. González-Gonzalo C, Sánchez-Gutiérrez V, Hernández-Martínez screening of retinal and optic disc diseases using handheld non- P et al (2019) Evaluation of a deep learning system for the joint mydriatic cameras in programmed routine occupational health automated detection of diabetic retinopathy and age-related macu- checkups onsite at work centers. Graefes Arch Clin Exp Ophthal- lar degeneration. Acta Ophthalmol 98:368–377. https://d​ oi.o​ rg/1​ 0.​ mol 1-9. https://​doi.​org/​10.​1007/​s00417-​020-​04860-z 1111/​aos.​14306 38. Wilkinson CP, Ferris FL 3rd, Klein RE et al (2003) Proposed 51. Ting DSW, Cheung CYL, Lim G et al (2017) Development and international clinical diabetic retinopathy and diabetic macular validation of a deep learning system for diabetic retinopathy and edema disease severity scales. Ophthalmology 110:1677–1682. related eye diseases using retinal images from multiethnic popu- https://​doi.​org/​10.​1016/​S0161-​6420(03)​00475-5 lations with diabetes. JAMA 318:2211–2223. https://​doi.​org/​10.​ 39. Flaxel CJ, Adelman RA, Bailey ST et al (2020) Diabetic retin- 1001/​jama.​2017.​18152 opathy preferred practice pattern. Ophthalmology 127:66–145. 52. Schmidt-Erfurth U, Sadeghipour A, Gerendas BS, Waldstein SM, https://​doi.​org/​10.​1016/j.​ophtha.​2019.​09.​025 Bogunović H (2018) Artificial intelligence in retina. Prog Retin 40. Yau JWY, Rogers SL, Kawasaki R et al (2012) Global preva- Eye Res 67:1–29. https://d​ oi.o​ rg/1​ 0.1​ 016/j.p​ retey​ eres.2​ 018.0​ 7.0​ 04 lence and major risk factors of diabetic retinopathy. Diabetes Care 53. Liu J, Gibson E, Ramchal S et al (2021) Diabetic retinopathy 35:556–564. https://​doi.​org/​10.​2337/​dc11-​1909 screening with automated retinal image analysis in a primary 41. Spanish Eyes Epidemiological (SEE) Study Group (2011) Preva- care setting improves adherence to ophthalmic care. Ophthalmol lence of age-related macular degeneration in Spain. Br J Ophthal- Retina 5:71–77. https://​doi.​org/​10.​1016/j.​oret.​2020.​06.​016 mol 95:931–936. https://​doi.​org/​10.​1136/​bjo.​2010.​187773 42. Zapata MA, Royo-Fibla D, Font O et al (2020) Artificial intelli- Publisher's note Springer Nature remains neutral with regard to gence to identify retinal fundus images, quality validation, laterality jurisdictional claims in published maps and institutional affiliations. 13