Bioethical Inquiry (2022) 19:407–419 https://doi.org/10.1007/s11673-022-10200-z SYMPOSIUM: EMERGING TECHNOLOGIES Bias in algorithms of AI systems developed for COVID-19: A scoping review Janet Delgado & Alicia de Manuel & Iris Parra & Cristian Moyano & Jon Rueda & Ariel Guersenzvaig & Txetxu Ausin & Maite Cruz & David Casacuberta & Angel Puyol Received: 2 February 2022 / Accepted: 6 May 2022 / Published online: 20 July 2022 # The Author(s) 2022 Abstract To analyze which ethically relevant biases were finally included. We propose a typology of biases have been identified by academic literature in artificial identified in the literature based on bias, limitations and intelligence (AI) algorithms developed either for patient other ethical issues in both areas of analysis. Results on risk prediction and triage, or for contact tracing to deal health disparities and SDOH were classified into five with the COVID-19 pandemic. Additionally, to specif- categories: racial disparities, biased data, socio- ically investigate whether the role of social determinants economic disparities, unequal accessibility and work- of health (SDOH) have been considered in these AI force, and information communication. SDOH needs to developments or not. We conducted a scoping review be considered in the clinical context, where they still of the literature, which covered publications from seem underestimated. Epidemiological conditions de- March 2020 to April 2021. Studies mentioning biases pend on geographic location, so the use of local data on AI algorithms developed for contact tracing and in studies to develop international solutions may in- medical triage or risk prediction regarding COVID-19 crease some biases. Gender bias was not specifically were included. From 1054 identified articles, 20 studies addressed in the articles included. The main biases are J. Delgado J. Rueda Department of Philosophy 1, Faculty of Philosophy, University of FiloLab Scientific Unit of Excellence of the University of Granada, Granada, Spain Granada, Granada, Spain e-mail: jdelgadr@ull.edu.es e-mail: ruetxe@gmail.com A. Guersenzvaig A. de Manuel : I. Parra : C. Moyano : Elisava School of Design and Engineering, UVIC-UCC, D. Casacuberta (*) : A. Puyol Barcelona, Spain Department of Philosophy, Universitat Autònoma de Barcelona, e-mail: a.guersenzvaig@gmail.com Barcelona, Spain e-mail: david.casacuberta@gmail.com T. Ausin Institute for Philosophy of the Spanish National Research Council A. de Manuel (CSIC), Madrid, Spain e-mail: aliciademanuel@hotmail.com e-mail: txetxu.ausin@cchs.csic.es I. Parra e-mail: airismind@gmail.com M. Cruz C. Moyano Andalusian School of Public Health (EASP), Granada, Spain e-mail: cristian.moyano@uab.cat e-mail: maiteazul.cruz@gmail.com A. Puyol e-mail: Angel.Puyol@uab.cat 408 Bioethical Inquiry (2022) 19:407–419 related to data collection and management. Ethical prob- algorithm —through sampled data or data in which lems related to privacy, consent, and lack of regulation societal biases already existed (Tsamados et al. 2022). have been identified in contact tracing while some bias- Previous databases can generate unfair results if there is related health inequalities have been highlighted. There no representativity of the population diversity or if some is a need for further research focusing on SDOH and segment is over-represented while others are under- these specific AI apps. represented (Tsamados et al. 2022). Design biases are those associated with previous conceptual decisions Keywords artificial intelligence . bias . digital contact made by the providers to create the machine learning tracing . COVID-19 . patient risk prediction (ML) system which may generate results that systemi- cally affect a segment of the population. While other reviews analyse the application of AI Background designed for COVID-19 (Guo et al. 2021), our intention is to focus this one in two specific topics: biases exclu- During the COVID-19 pandemic, one of the most wide- sively in AI systems developed for 1) DCT and 2) spread measures adopted to control, minimize, and mit- medical triage regarding COVID-19, as they have been igate the impact of COVID-19 was the development of two of the most widely developed automatized systems mobile apps that use a variety of technologies to log during the early phases of the pandemic and they need to information used to identify the spread of the disease, be evaluated. In addition, although certain social health physical symptoms of individuals, and possible close conditions have been discussed in general terms, social contacts. Digital contact tracing (DCT) via smartphone determinants of health (SDOH) are explicitly addressed apps was established as a new public-health intervention in neither clinical research nor technical development of in many countries in 2020 to reduce the levels of apps. Consequently, we hypothesize that there may be a COVID-19 transmission (Colizza et al. 2021). Never- lack of qualitative data analysis in their application, theless, DCT systems may perpetuate some biases that which causes an overlook of health disparities and influence the results obtained, while raising security and SDOH and that have effects on people’s health. Clinical privacy concerns (Bengio et al. 2020; Sun et al. 2021). research and app development research are mostly fo- Another big effort has been put into developing cused on biological-only data, which not only may decision-making support devices to help clinicians at retain biases and exacerbate health inequalities (Röösli, the bedside. There is an increasing multiplicity of artifi- Rice, and Hernandez-Boussard 2021) but also underes- cial intelligence (AI) systems and algorithms focused on timate social-related biases in their analysis. COVID-19 early detection in risk patients and their Thus, in this scoping review we aim to summarize prognosis (Jamshidi et al. 2020). Studies prove that some of the ethically relevant types of bias that have these novel technologies support medical triage in those already been identified in literature in AI systems devel- circumstances where healthcare resources are scarce. oped for DCT, and for patient risk prediction (PRP) or Still, their results show some limitations due to technical medical triage to deal with COVID-19 pandemic. In issues or regarding social, cultural, and economical as- addition, a secondary goal is to analyse if there is any pects that have been overlooked. relationship pointed out by previous literature between A general definition of bias would be a “strong the biases and social determinants of health in AI sys- inclination either in favor or against something” tems and algorithms developed for COVID-19. (Moseley 2021). By algorithmic biases in AI, we refer to systematic errors in a computer system, with a con- sequent deviation from the expected prediction behavior Materials and Methods of an AI tool (Amann et al. 2020). These deviations can come either from the design of the algorithm or from A scoping review can be a useful approach when the previous data collection, coding, and selection. Both information on a topic has not been comprehensively options have to be taken into account while analysing reviewed, which is the case of research themes. This possible limitations and poor performances of AI out- scoping review followed the recommended five-step comes in the areas stated above. Regarding the data, framework for scoping reviews (Arksey and O’Malley biases emerge mainly from the data used to train the 2005; Munn et al. 2018; Pham et al. 2014; Tricco et al. Bioethical Inquiry (2022) 19:407–419 409 2016). The research questions were: What are the biases Data Extraction and Data Synthesis identified in the literature in AI systems or algorithms developed for COVID-19 triage or PRP and DCT? Do Data extraction was undertaken by the four reviewers researchers and engineers consider the SDOH? Do they using an agreed template designed ad hoc. We created a relate these biases to possible health disparities? table gathering the main characteristics and results of the studies to collect information from the data extraction. Finally, we developed a narrative synthesis of the main Search Strategy and Selection Process findings. We carried out the search strategy in Pubmed, Medline, CINAHL, Scopus, Wiley Online Library, WOS, and Results Arxiv.org, between the March 1 and the April 7, 2021. The search strategy was initially developed in Pubmed After screening by title and abstract, out of the 1054 and then adapted to the other databases (Table 1). identified articles after eliminating duplicates, 134 were A manual search was performed to retrieve additional included for full-text reading, and sixteen met the relevant documents not located by the electronic search. criteria for the extraction of relevant information. A Studies fulfilling the criteria shown in Table 2 were manual search provided four additional studies; thus, eligible. Following the removal of duplicates, two re- twenty were finally eligible for inclusion (Fig. 1). viewers screened all studies by title and abstract. Full- Of those selected, we found ten narrative reviews text articles were obtained for full-text screen, and each (Grantz et al. 2020; Klingwort and Schnell 2020; Mali one was read by two reviewers, based on the inclusion and Pratap 2020; Malik et al. 2021; Marabelli, Vaast, and exclusion criteria. Discrepancies were discussed and Li 2021; Mbunge 2020; Park, Choi, and Ko 2020; and a consensus was reached. When consensus was Roche 2020; Röösli, Rice, and Hernandez-Boussard not possible, all the four reviewers made a decision 2021; Scott and Coiera 2020; Shachar, Gerke, and together. Adashi 2020), five original articles (Casiraghi et al. 2020; Hisada et al. 2020; Marabelli, Vaast, and Li 2021; Moss and Metcalf 2020 ; Ravizza et al. 2021), Table 1 Search strategy two case studies (Gulliver, Fahmi, and Abramson 2020; #1 Bias Sáez et al. 2021), two systematic reviews (Mbunge et al. #2 Bias OR Social determinants of health 2020; Wynants et al. 2020), and one rapid review #3 Algorithmic injustice OR prejudice OR discrimination (Anglemyer et al. 2020). For countries, we found five #4 Artificial Intelligence OR Machine Learning OR AI OR articles from the United States (Grantz et al. 2020; Algorithms Marabelli, Vaast, and Li 2021; Moss and Metcalf #5 Artificial Intelligence OR Machine Learning OR AI OR 2020; Röösli, Rice, and Hernandez-Boussard 2021; Algorithms OR Deep Learning Shachar, Gerke, and Adashi 2020), one from the United #6 Smartphone OR Geolocation OR App OR Mobile Phone States and South-Korea (Park, Choi, and Ko 2020), two #7 Contact tracing app from Australia (Scott and Coiera 2020; Gulliver, Fahmi, #8 Social determinants of health OR Income OR social protection and Abramson 2020), Italy (Casiraghi et al. 2020; OR Education OR Unemployment OR job insecurity OR Ravizza et al. 2021), India (Malik et al. 2021; Mali Working life conditions OR Food insecurity OR Housing OR basic amenities OR environment OR Early childhood and Pratap 2020) and Swaziland (Mbunge 2020; development OR Social inclusion OR non-discrimination OR Mbunge et al. 2020), and one from The Netherlands Structural conflict OR health services (Wynants et al. 2020), Spain (Sáez et al. 2021), Germa- #9 Covid-19 OR Coronavirus OR SARS-CoV-2 ny (Klingwort and Schnell 2020), Japan (Hisada et al. #10. #2 AND #4 AND #9 2020), New Zealand (Anglemyer et al. 2020), and Can- #11. #1 AND #4 AND # 8 AND #9 ada (Roche 2020). While four studies addressed both #12. #3 AND #5 AND #9 triage or PRP and DCT, three studies addressed only #13. #1 AND #6 AND #9 triage or PRP, and thirteen only DCT. #14. #1 AND #7 AND #9 Table 3 summarizes the different apps identified in this study. We use the term “bias” to refer to the 410 Bioethical Inquiry (2022) 19:407–419 Table 2 Inclusion and exclusion criteria Inclusion criteria Type of publication Studies with quantitative methodology, mixed methodology, qualitative, interventions, narrative reviews, scoping reviews, systematic reviews/meta-analysis, randomized clinical trials, editorials, and letters to editor. Subject or domain being studied Articles focused on or with mention of biases, algorithms, or AI systems developed for COVID-19, used in triage, early detection of risk patients, and contact tracing. Language English or Spanish. Participant/population AI systems or algorithms developed for any population group. Intervention Any intervention related to our condition. Date Articles published or accepted between March 2020 and April 2021. Exclusion criteria Type of publication Books and theses, not accepted for publication preprints, conferences, and abstracts. Subject or domain being studied Articles about COVID-19 which do not include any algorithm or AI system or which do not mention biases. Articles focused on bias, but not on AI systems for COVID-19. Articles about other topics than triage, risk prediction, or contact tracing (for instance, vaccines, clinical trials). Language Different from English or Spanish. Participants/population No exclusion criteria. Intervention No exclusion criteria. Date Articles prior to 2020. systematic errors in a computer system causing an incli- people that can be considered to be unfair and cause a nation or prejudice for or against a person or a group of deviation from the expected prediction behavior of an Fig. 1 Scoping Review Flow Diagram following the PRISMA 2020 statement proposed by Page et al. (2021) Bioethical Inquiry (2022) 19:407–419 411 Table 3 DCTApps identified in the scoping review App Type Technology Reference TraceTogether Contact Tracing Bluetooth (BlueTrace protocol) (Roche 2020) (Singapour) The-Corona-Warn-App Contact Tracing Control smartphone (Mbunge 2020) (Germany) COVIDSafe Outbreak Response Bluetooth (BlueTrace protocol) (Gulliver, Fahmi, and Abramson 2020) (Australia) Stopp Corona Contact Tracing Bluetooth (Mbunge 2020) (Austria) BeAware App Contact Tracing Bluetooth and Global System for (Mbunge et al. 2020) (Bahrain) Mobile Communications technology HaMagen Contact Tracing Bluetooth and GPS (Mbunge et al. 2020) (Israel) bStayHomeSafe (China) Contact Tracing Bluetooth, GPS, and WiFi (Mbunge 2020) CoronaApp Contact Tracing Global Positioning System (Mbunge et al. 2020) (Colombia) Aarogya Setu Contact Tracing Global Positioning System (Mbunge 2020) (India) GH COVID-19 (Ghana) Outbreak Response Global Positioning System and GIS (Mbunge et al. 2020) CoronaMadrid Symptom tracking Global Positioning System (Mbunge 2020) (Spain) Social Monitoring (Russia) Quarantine compliance Global Positioning System (Mbunge et al. 2020) Yahoo! JAPAN App (Japan) Contact Tracing WSSCI (Hisada et al. 2020) StopCovid Contact Tracing Bluetooth (Roche 2020) (France) STOPV Contact Tracing Global Positioning System, (Roche 2020) (France) Semantic Data, Epidemiological Data and Test Results Private Kit: Safe Paths Contact Tracing Global Positioning System (Roche 2020) (USA) Covid Alert Contact Tracing Bluetooth (Roche 2020) (Canada) AI tool, and “limitations” to refer to facts or situations needed to develop more rigorous prediction models that allow only some actions and make others and validate the existing ones through collaborative impossible. efforts. Our results identified different types of data-related AI Systems Developed for Triage and PRP bias: Bias 1. Data source variability contributes to bias in distrib- uted research networks of COVID-19 data sharing One of the most relevant aspects addressed in the liter- (Sáez et al. 2021), and they play an important role in ature is data-related biases. According to a systematic data quality. The case study reported by Sáez et al. review (SR) of COVID-19 prognostic and risk predic- (2021) shows the limitations that multisource vari- tion methods (Wynants et al. 2020), there is a high risk ability may have for COVID-19 machine learning of bias in the studies included due to a poor description (ML) research on international distributed research of the population, which raises concerns about the reli- networks. They used the nCov2019 dataset, includ- ability of their predictions when applied to clinical prac- ing patient-level data from several countries, to tice. An immediate exchange of well-documented indi- discover and classify severity subgroups, dividing vidual participant data from COVID-19 studies is them into six types: 1) mild disease with no 412 Bioethical Inquiry (2022) 19:407–419 comorbidity, 2) elderly + severe pulmonary disease 4. The quick development of AI systems carries great + comorbidity, 3) middle-aged + severe pulmonary risk due to skewed training data, lack of reproduc- disease + no comorbidity, 4) elderly + mild disease ibility, and lack of a regulated COVID-19 data + no comorbidity, 5) elderly + severe systemic resource (Röösli, Rice, and Hernandez-Boussard disease + comorbidity, 6) elderly + severe pulmo- 2021). Without comprehensive bias mitigation nary disease + heart failure. The problem appears strategies, this can exacerbate existing health dis- when this division is conditioned by the data’s parities. “The source code of any AI model should country of origin. Groups 1 and 4 data were collect- be shared publicly to ensure that the models can be ed in China and groups 2, 3, 5, and 6 in the Philip- widely applied, generalized, and transparently com- pines. In the last case, data came from the COVID- pared” (Röösli, Rice, and Hernandez-Boussard 19 tracker, owned by the Department of Health of 2021, 191). the Republic of Philippines; in the case of China, data came mostly from patient reports. Due to these variations in the sources, results show some incon- sistencies, limiting the model. Potential biases of Limitations multisource variability for ML can be generalized in large cross-border distributed research networks. A sufficient amount of high-quality data is crucial for How can we prevent such biases? Sáez et al. (2021) the successful implementation of AI in COVID-19 propose 1) a routine assessment of the variability management. Designing practical AI-based algo- among data sources in ML and statistical method- rithms is challenging because of the huge and com- ologies to potentially reduce biases or extra costs, 2) plex data that emerge as a consequence of the varied a complete data quality assessment, incorporating manifestations of the COVID-19 infection, ranging source and temporal variability, 3) reporting data from asymptomatic to severe clinical disease (Malik quality and its impacts as a routine practice in pub- et al. 2021). Moreover, the principal obstacle to lications, and 4) building consciousness about data implement these systems in the clinical context is quality and variability. the regulation of the data exchange obtained by the 2. Casiraghi et al. (2020) developed an explainable AI application. Additionally, AI-based algorithms PRP model for COVID-19 risk assessment aimed can offer a binary answer to a specific question to avoid data bias. Their model was designed to be about the disease in context but cannot offer alter- used in emergency departments for an early assess- native predictions (Malik et al. 2021). Finally, it is ment of PRP in COVID-19 patients, integrating necessary to consider how meaningful and in-depth clinical, laboratory, and radiological data. The study data can be generated at every point of healthcare carried out a comparative evaluation of different activity (Malik et al. 2021). imputation techniques to manage the problem of missing data in the prediction for COVID-19 pa- tients. However, the lack of a shared dataset hin- Other ethical issues dered an objective comparative evaluation with the best models (Casiraghi et al. 2020). Transparency in AI algorithms is essential to understand 3. There are biases in COVID-19 prediction models predictions and target populations, unrecognized due to unrepresentative data samples, high proba- biases, class imbalance problems, and their capacity bility of model overfitting, imprecise information to generalize emerging technologies across hospital on the study populations, and the use of a model settings and populations (Röösli, Rice, and unsuited to the task (Röösli, Rice, and Hernandez- Hernandez-Boussard 2021). To ensure that models Boussard 2021). Models developed in elite and can be broadly applied, generalized, and compared, affluent academic health systems that are not repre- the source code of an AI system should be shared sentative of the general population lack external publicly, and regulatory frameworks should be creat- validity (Röösli, Rice, and Hernandez-Boussard ed to facilitate data sharing (Röösli, Rice, and 2021). Hernandez-Boussard 2021). Bioethical Inquiry (2022) 19:407–419 413 AI Systems Developed for DCT measurement errors (Klingwort and Schnell 2020). These errors are due to the devices’ different signal Bias strengths and the fact that signal is not transmitted in all directions. Characteristics of the physical en- 1. Uncontrolled application development could gener- vironment (windows, walls, or doors) can affect the ate inadequate data collection and biases due to the range of discoverable devices. In addition to the loss of some data or an insufficient frequency of four efficiency conditions (mass adoption, well- monitoring, which can lead to an inability to com- equipped population, numerous diagnostic tests, pare data collected from different regions (Ravizza and fair and transparent uses), these monitoring et al. 2021). Although it does not affect the core applications have many reliability limitations, espe- functionality of the app, it can influence further use cially in the Bluetooth reading forecast and the of the collected data: most ML models have relied calibration (Roche 2020). This can add noise and on Chinese data, which can limit scalability to other produce many false positives. populations (Scott and Coiera 2020). 2. Data-related problems: Most applications have not 2. The media alter the nature of searches, producing reached operational maturity (Scott and Coiera biases in areas of potential clusters. Whenever the 2020), and their effectiveness has not been proved media report a location of a positive COVID-19 (Anglemyer et al. 2020). Even modelling studies patient, many people who are close to the informed provide low-certainty evidence of a reduction in location ask for additional information related to secondary cases if CT is used together with other COVID-19 (Hisada et al. 2020). public health measures such as self-isolation. Co- 3. DCT Applications (DCTApps) pose a high risk of hort studies provide very low-certainty evidence discrimination, especially to affected people that digital DCT may produce more reliable counts (Mbunge 2020). Specifically, internet-of-things of contacts and reduce time to complete DCT (IoT) based DCTApps collect data from the entire (Anglemyer et al. 2020). The performance of population in real time, which is later analysed to emerging technologies is not yet stable in account map COVID-19 hotspots. Such data include ethnic of the lack of availability of a sufficient COVID-19 information, demographic details, and socioeco- dataset, the inconsistency of some of the available nomic status, which can influence the allocation datasets, the non-aggregation of the dataset, and and distribution of COVID-19 resources, potential- missing data and noise (Mbunge et al. 2020). ly leading to discrimination. 4. False negatives are an obstacle and may be deliber- DCTApps may use the IoT to transfer data to national ately generated because infected people do not health systems. However, they are not globally stan- want to reveal their true status (Klingwort and dardized, and they face a lot of problems based on Schnell 2020). To overcome this problem, the interoperability (heterogeneity of connection standards detection of relevant contacts should be refined as and communication protocols, data semantics, formats, the issue is fundamentally a problem of microscale different operating systems, and programming lan- spatial analysis. Applications must develop the guages). Consequently, each country has developed its microgeographic analytical capability to specify own app. Data formats and structures should be stan- what kind of proximity constitutes a sufficient con- dardized to avoid noise, prevent incomplete data, and tagion risk to trigger a notification (Roche 2020). improve quality (Mbunge 2020; Mbunge et al. 2020). Determining a standardized list of data, symptoms, clin- ical signs, risk factors, and comorbidities associated Limitations and Technical Problems with coronavirus can contribute to ensuring compatibil- ity of databases between regions and countries and to 1. Accuracy: The most widely proposed type of improving interoperability (Ravizza et al., 2021). COVID-19 application uses Bluetooth signals to DCT becomes less effective when dealing with track encounters with people diagnosed as infected asymptomatic individuals, since symptom checkers after the encounter; the accuracy of automatic and apps rely on pulse, temperature, and sleeping pat- DCTApps suffers from B luetooth-based terns (Hellewell et al. 2020, cited in Mbunge 2020). Due 414 Bioethical Inquiry (2022) 19:407–419 to built-in privacy mechanisms, the resulting data for losses. Concerns were raised regarding the uneven scientific research based on these applications are limit- scope and granularity of disclosures by municipal and ed to counts of positive or negative encounters from local governments. Korea’s National Human Rights selective populations, where the odds of encounters Commission issued a recommendation to ameliorate cannot be calculated (Klingwort and Schnell 2020). privacy concerns, suggesting that the revelation of ex- ceedingly detailed information was unwarranted. In- Other Ethical Issues stead of disclosing data to the public, information could be used to sanitize establishments, potentially avoiding 1. Privacy concerns: The use of DCTApps raises eth- stigma and business decline. That is, instead of publicly ical, legal, security, and privacy concerns (Roche revealing the precise locations of an infected individual, 2020). To be acceptable, this interference with fun- less granular data could be revealed, with the same damental rights must be justified, reasonable, pro- effect on tracking and quarantine (Park, Choi, and Ko portionate, and politically consensual. DCTApps 2020). provide little or no privacy to infected people and The correlation of data, the exchange of information, require them to disclose their data, raising difficult and the ability to extract information from different issues of consent, privacy, ethics, and trade-offs entry points contribute to the increasing fragility of the between public and private goods (Scott and anonymization of data. This anonymization is even Coiera 2020). more fragile when information is collected over time and through data cross-referencing (Roche 2020). The DCTApps violate the security, confidentiality, integ- deactivation of DCTApps must be programmed so that rity, and data availability of COVID-19 patients and monitoring does not continue beyond the health emer- contact persons, which can sometimes cause mental gency and is not tacitly established as standard practice. health issues like stress, anxiety, or depression Otherwise, risks of mass surveillance could arise. (Mbunge, 2020). Apps like TraceTogether, COVIDSafe, or BeAware support access to multiple 2. Lack of regulation: There are no specific regulations data access points and the monitoring and surveillance for DCTApps. However, their use of data, access, or of infected or isolated people, which threatens the secu- privacy has been adapted to international, national, rity of public health data, and may be a violation of and state laws such as the European Union’s Gen- privacy (Mbunge 2020). eral Data Protection Regulation (GDPR) and the The study of Park, Choi, and Ko (2020) in South California Consumer Privacy Act (CCPA). These Korea recreates privacy-related problems. In the face of legal frameworks can be adapted to help address the outbreak of COVID-19, the South Korean Ministry concerns about privacy, human rights, due process, of Health and Welfare (MOHW) made the following and equality (Shachar, Gerke, and Adashi 2020). In information available to the public on the Internet or certain countries, like the United States, the lack of through a press release: 1) the route and means of state regulation makes it more difficult to guarantee transportation of infected people, 2) the medical institu- that these applications follow ethical standards tions that treated the infected people, 3) and the health (Shachar, Gerke, and Adashi 2020), and there are status of those in contact with an infected person. In no global WHO guidelines on health data shared addition to these items, sex, nationality, and age of those and transmitted via 5G technology (Mbunge et al. infected were available, although their names were not 2020). Though some countries’ regulations protect disclosed. Some municipal and local governments went citizens better, potential “digital scars” are left in further and provided very detailed routes and the names society as long as the governments and private of restaurants, shops, and other commercial premises institutions continue having long-term and unlimit- that infected people visited. The locations of the infected ed access to citizens’ data for surveillance purposes. people attracted extensive news coverage. Some of these people were affected by an unwanted invasion of 3. Consent: The efficacy of DCTApp depends on the privacy and were even the subject of public disdain. level of population uptake, its ability to accurately Restaurants, shops, and other commercial venues that detect infectious contacts, and the extent of adher- infected people visited experienced dramatic economic ence to self-isolation by notified contacts (Scott and Bioethical Inquiry (2022) 19:407–419 415 Coiera 2020). DCT must be handled with care: Biased Data these technological solutions are proposed as the only tool available to ensure a process of deconfine- The reliance on AI may create a false sense of objectiv- ment, a requirement that would make it a sine qua ity and fairness (Röösli, Rice, and Hernandez-Boussard non condition accessible to police control. The risks 2021). The pervasiveness of biases is a failure to devel- of seeing such an established form of socio-spatial op mitigation strategies and has exacerbated the risk of “triage” and patients and certain categories of the existing health disparities, hindering the adoption of ostracized population are huge (Roche 2020). other tools that could actually improve patient out- comes. As an example, the Medical Information Mart Although DCTApps use WiFi, GPS, or Bluetooth for Intensive Care (MIMIC) is a publicly available, de- protocols to monitor people’s movement, users have identified, and broadly studied dataset for critical-care the right to opt-out and configure their devices, jeopar- patients. A MIMIC-equivalent for COVID-19 from di- dizing the monitorization of positive cases (Mbunge verse data sources could incentivize urgently needed 2020). DCTApp should allow people to practice with- data sharing and interoperability to enable diverse, drawal of consent (Mbunge 2020), as problematic uses population-based tailored therapy—a step that could of technologies may well remain once the pandemic is decisively reduce biases and disparities in healthcare over. This can potentially advantage powerful groups while bolstering clinical judgement and decision-mak- that can obtain financial and political benefits from ing. One of the main methodological problems is the perpetuating the use of IT surveillance while having selection process (Klingwort and Schnell 2020). The questionable effects on society (Marabelli, Vaast, and sample of the population using the application will not Li 2021). be random, and subpopulations with a higher prevalence Privacy issues related to forcing a population to use of undetected infections will likely have lower coverage. an app can lead to much lower coverage rates In addition, models that include comorbidities associat- (Klingwort and Schnell 2020). However, we find ed with worse outcomes in COVID-19 may perpetuate opposite scenarios in countries that have not devel- structural biases that have led to historically disadvan- oped any specific app. Brazil, for example, has taged groups disproportionately suffering those comor- increased its technological surveillance in order to bidities. To avoid further harm to minority groups al- minimize the COVID-19 transmission chain ready most affected by COVID-19, resource allocation (Mbunge et al. 2020). This enforcement of massive models must go beyond a utilitarian foundation and surveillance can raise issues about power, abuse, must be able to identify needs amongst these patients and data exploitation. (Moss and Metcalf 2020). Socio-Economic Disparities Health Disparities and Social Determinants of Health in AI Systems Developed for Triage In DCT, the ability to make use of notifications to and DCT for COVID-19 minimize one’s own risk by self-quarantining is far too dependent on one’s personal wealth and capacity to Racial Disparities afford to stay at home (Moss and Metcalf 2020). DCTApps’ designers must be attuned to the context of Health disparities are related to the emergence of biases social life in which such systems can produce harmful, in ML systems in the U.S. context where Black and difficult-to-foresee effects that replicate or amplify pre- Latinx communities have been the most severely affect- existing inequalities. Attending the contextual use of ed by COVID-19 (Moss and Metcalf 2020). This is due such a system could collectivize risk by identifying to long-standing disparities in health outcomes for these and emphasizing the necessary forms of social support communities, the impact of environmental determinants for self-quarantine and medical care: adequate sick leave of health, and the disproportionate number of workers and quarantine leave policies, robust testing, and the whose jobs do not allow them to stay at home (Moss and economic relief that targets individual workers over Metcalf 2020). large companies. 416 Bioethical Inquiry (2022) 19:407–419 During the pandemic, ML has been involved in the a high incidence (e.g., Chad or the Central African production and distribution of risk through society Republic) where ICT infrastructures are very poor. (Moss and Metcalf 2020), generating risks and its un- These factors can hinder the development of technolog- even distribution in society. Many of the predictive ical innovation policies as part of their response to surveillance algorithms used in DCT control focus at- COVID-19 (Mbunge 2020). tention on populations where bias is very present, espe- cially in highly racialized or lower-income populations (Moss and Metcalf 2020). In this sense, ML can be Discussion epidemiologically effective, while unethical. This is the first study to offer a categorization by typol- Unequal Accessibility ogy of the main biases, limitations, and related ethical issues identified in current scholarship on AI systems AI-based global health initiative is recommended, since developed for triage or PRP and DCTApps during AI-based approaches may not be accessible in countries COVID-19. Many papers have already stressed the mat- with limited resources (Malik et al. 2021). Regarding ter of bias in artificial intelligence in different areas, such socio-economic disparities and the digital gap, the lack as the content and use of websites (Baeza-Yates, 2018), of population coverage can leave certain populations at the process of recruitment (Dastin 2018), the racial bias risk (Ravizza et al. 2021). Digital solutions can exacer- in assessment in criminal sentences (Angwin et al. bate existing disparities between those who do not have 2016), or the racial bias in facial recognition access to smartphones or who live in areas without (Buolamwini and Gebru 2018), among others. Howev- connectivity, because of ethnicity, socio-economic sta- er, the specific focus on the role of SDOH in the emer- tus, or age (Anglemyer et al. 2020), with equity impli- gence of bias in these systems and its analysis is a novel cations for at-risk populations with poor access to the contribution that can enrich the perspective of techno- Internet and digital technology. Digital deserts or data logical development for future crises. poverty in certain geographical areas are concerning, One of the main findings from this review is that especially because the effectiveness of DCTApps de- while references to “health disparities” are relatively pends on their massive voluntary adoption and a sys- more frequent in the study of AI systems, references to tematic screening (Roche 2020). Across country bor- “SDOH” are rather uncommon. We may also point out ders, the health gap and inequalities in healthcare pose a that definitions and terminologies vary from one author problem for the integration of emerging technologies. to another, which has made it more difficult to identify Even in developed countries, risk groups may not have and systematize them. Based on the results, we argue access to broadband, smartphones, or wearable technol- that SDOH and health disparities are rarely taken into ogy. For a community to benefit from this technology, account in triage and PRP studies and are mostly related most people need to be equipped with mobile devices. to DCTApps. These findings suggest that SDOH are This applies to only eighty per cent of the US popula- undervalued in a clinical context and need to be given tion, sixty-five per cent of Russians, and forty-five per more consideration. cent of Brazilians (Marabelli, Vaast, and Li 2021). Chil- Our review shows that data are geographically de- dren, elderly, or individuals with fewer resources are pendent and that their use for training across regions and excluded from the stored information (Grantz et al. countries results in biases in the outcomes. The use of 2020). local data to develop international solutions can increase biases in other local populations due to epidemiological Workforce and Information and Communication differences. Previous studies have suggested that in- Technologies Infrastructure creasing the diversity of data sets from different popu- lations and geographical regions and demonstrating the Developing an app and maintaining the system requires reproducibility of AI-based algorithms in different set- a specific workforce and a consistent information and tings would be required if the usage of AI tools is going communication technologies (ICT) infrastructure that to be generalized (Chen and See 2020; Nagendran et al. may be lacking. Some countries may struggle with the 2020; Zou and Schiebinger 2021). As some authors technological infrastructure, especially in countries with have pointed out, the shared use of public databases Bioethical Inquiry (2022) 19:407–419 417 might facilitate future international analysis aimed at increasing marginalization, social disadvantage, and bad noticing the interspatial diversity and multiple SDOH healthcare outcomes (Kapilashrami and Hankivsky factors affecting under-represented groups (Hendl and 2018; Figueroa et al. 2021). Biases in AI should be Roxanne 2022). But such a proposal should dialogue viewed through an intersectional lens because several with crucial aspects for bioethics, like privacy, volun- of them occur together and in an overlapping way. This tariness, and informed consent (Ausín and Andreu approach, vital to understanding SDOH and their Martínez 2020). relation to AI outcomes, has some difficulties. As As SDOH are highly contextual and based on cultur- Foulds et al. (2020) explain, data sparsity becomes a al, social, and economic aspects that differ between bigger problem when the number of dimensions or places, it might not be enough to maximize representa- distinct values increase, leading to uncertainty in fair- tivity in global databases. The homogenization of com- ness measurements. munity characteristics will always lead to possible As datasets are imbalanced and can create unfair biases if we use SDOH as a pattern of analysis. From decisions to minority groups, there must be a this perspective, and to be fair with different realities all counterbalance in which different inputs can be given over the world, scalability is definitely in tension with different weights in the final prediction. For instance, disparities in local areas. If this be true, AI systems Figueroa et al. (2021) explore how non-binary persons based on global datasets would need other procedures are under-represented in medical research, concluding to adapt the results because they would fail to consider that very few digital health interventions for non-binary SDOH at a more than superficial level. There is a need individuals have been developed. As the authors states, to continue doing research to see if massive AI systems “the absence of research on electronic health and mobile are the best option instead of local and smaller databases health in sexual and gender minority populations can adapted to different cultural communities. lead to digital interventions being ineffective or even We have also found that ML solutions can be epide- harmful for these groups, through biases in the data.” miologically effective and, at the same time, ethically Although there are some papers and proposals on the fraught because of design biases. The rapid develop- theme in a general level (Foulds et al. 2020; Roy, ment of DCT solutions has proven to accelerate the Iosifidis, and Ntoutsi 2021), further research is required identification of infected people despite its ethical cost, to develop an intersectional perspective in AI systems raising worrying issues of lack of privacy, biased data, —regarding the conceptualization and the computation- or socio-economic disparities. Our findings reiterate the al design—both in a general level and also specifically lack of comparative studies and literature about the as developed for DCT and for PRP or medical triage to effectiveness and convenience of DCTApps during the deal with any future COVID-19 pandemics that may pandemic outbreak, as previous research has concluded. come. This makes it more difficult to evaluate the success of Our findings are in agreement with Ausín and the solutions and their cost in opportunity as well as to Andreu Martínez (2020), who have found some ethical assess changes and improvements of this technology. elements to take into consideration for DCT: Further research on this issue is required. Finally, gender is never questioned as a possible bias 1. The security and safety of technological systems on the results in published studies about biases in AI responds to the duty not to cause harm and to systems for COVID-19. Race, age, or socioeconomic minimize harm, protecting individuals and groups. status, on the contrary, appear more often as indicators 2. The intervention must be proportional and benefi- in the data used in AI systems. Usually, biases are not cial given the severity of the situation. due to a single attribute (e.g., gender) but to the combi- 3. The installation of the app must be voluntary and nation of some of them (e.g., race, gender, poverty). require people’s consent and not carry a penalty for That is why, instead of the classical single attribute non-acceptance. fairness, there is a need for a multi-attribute solution, 4. 4.The data must be pseudonymized to protect which follows the so-called intersectional approach. The privacy. intersectionality approach is useful to understand how 5. Applications and other technologies must be avail- gender intersects with other social identities (e.g., race, able and accessible to everyone, regardless of eco- ethnicity, socioeconomic status, sexuality, or disability), nomic or technological level. 418 Bioethical Inquiry (2022) 19:407–419 There are some limitations to our study. First, we had Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, some problems identifying AI from other algorithms or adaptation, distribution and reproduction in any medium or format, statistical and mathematical methods without AI be- as long as you give appropriate credit to the original author(s) and cause of a lack of clarity in the literature. Second, our the source, provide a link to the Creative Commons licence, and search strategy may have overlooked some concepts indicate if changes were made. The images or other third party material in this article are included in the article's Creative Com- related to SDOH or included them in the category. mons licence, unless indicated otherwise in a credit line to the Third, the majority (thirteen out of twenty) of the in- material. If material is not included in the article's Creative Com- cluded articles are narrative reviews (n = 10), rapid mons licence and your intended use is not permitted by statutory review (n = 1), or case studies (n = 2). Narrative reviews regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of can be associated with selection bias (Pae 2015). How- this licence, visit http://creativecommons.org/licenses/by/4.0/. ever, these articles were selected because they addressed bias and other ethical issues more explicitly than other types of articles or empirical studies and because our References aim was to identify current known biases in literature so future research can look for other kind of possible biases Amann J., A. Blasimme, E. Vayena, et al. 2020. Explainability for during the development and application of the AI sup- artificial intelligence in healthcare: A multidisciplinary perspec- port systems. Finally, since we restricted our review to tive. BMC Medical Informatics and Decision Making 20(1): 310. studies published in English and Spanish, we might Anglemyer A., T.H.M. Moore, L. Parker, et al. 2020. Digital contact tracing technologies in epidemics: A rapid review. have missed relevant work published in other Cochrane Database of Systematic Reviews 8(8): CD013699. languages. Angwin, J., J. Larson, S. Mattu, and L. Kirchner. 2016. Machine Recognizing these limitations, we hope that our scop- Bias. ProPublica, May 23. https://www.propublica. ing review can help to document the types and extent of org/article/machine-bias-risk-assessments-in-criminal- biases actually present in specific AI algorithms for sentencing. Accessed April 27, 2022. Arksey, H., and L. O’Malley. 2005. Scoping studies: Towards a triage or PRP and DCT in the context of the COVID- methodological framework. International Journal of Social 19 pandemic. Research Methodology: Theory and Practice 8(1): 19–32. Ausín, T., and M.B. Andreu Martínez. 2020. Ética y protección de datos de salud en contexto de pandemia: Una referencia especial al caso de las aplicaciones de rastreo de contactos. Conclusions Enrahonar An International Journal of Theoretical and Practical Reason 65: 47–56. Baeza-Yates, R. 2018. Bias on the Web. Communications of the The analysis of previous literature shows that the main ACM 61(6): 54–61. sources of biases identified in both triage or PRP and Bengio, Y., R. Janda, Y. Yu, et al. 2020. The need for privacy with DCT AI systems for COVID-19 are mainly related to public digital contact tracing during the COVID-19 pandem- data source variability and inadequate data collection. In ic. The Lancet Digital Health 2(7): 342–344. addition, ethical problems related to privacy, consent, Buolamwini, J., and T. Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. and the lack of regulation have been identified in DCT Proceedings of Machine Learning Research 81:1–15. Apps. Biases related to health disparities and SDOH are Casiraghi, E., D. Malchiodi, and G. Trucco, et al. 2020. Explainable not the main topics of the studies but are in some machine learning for early assessment of COVID-19 risk pre- way included, especially in narrative reviews of diction in emergency departments. IEEE Access 8: 196299– DCT Apps. Although there is some concern on the 196325. https://doi.org/10.1109/ACCESS.2020.3034032. Chen, J., and K.C. See. 2020. Artificial intelligence for COVID-19: topic, a theoretical framework addressed to re- Rapid review. Journal of Medical Internet Research 22(10): searchers and engineers would facilitate the compre- e21476. hension and identification of potential biases in fu- Colizza, V., E. Grill, and R. Mikolajczyk, et al. 2021. Time to ture technologies and their uses. evaluate COVID-19 contact-tracing apps. Nature Medicine 27(3): 361–362. Dastin, J. 2018. Amazon scraps secret AI recruiting tool that Funding Open Access Funding provided by Universitat showed bias against women. Reuters, October 11. Autonoma de Barcelona. This work has been funded by the https://www.reuters.com/article/us-amazon-comjobs- BBVA Foundation for SARS-CoV-2 and COVID-19 Research automation-insight/amazon-scraps-secret-airecruiting-tool- in Humanities (Detección y eliminación de sesgos en algoritmos that-showed-bias-against-womenidUSKCN1MK08G. de triaje y localización para la COVID-19). Accessed April 27, 2022. Bioethical Inquiry (2022) 19:407–419 419 Figueroa, C.A., T. Luo, A. Aguilera, and C.R. Lyles. 2021. The Guidance for authors when choosing between a systematic or need for feminist intersectionality in digital health. The scoping review approach. BMC Medical Research Lancet Digital Health 3(8): e526–e533. Methodology 18(1): 1–7. Foulds, J.R., R. Islam, K.N. Keya, and S. Pan. 2020. Bayesian Nagendran, M., Y. Chen, C.A. Lovejoy et al. 2020. Artificial intelli- modeling of intersectional fairness: The variance of bias. gence versus clinicians: Systematic review of design, reporting Proceedings of the 2020 SIAM International Conference on standards, and claims of deep learning studies. BMJ 368: m689. Data Mining: 424–432. Pae, C.U. 2015. Why systematic review rather than narrative Grantz, K.H., H.R. Meredith, D.A.T. Cummings et al. 2020. The review? Psychiatry Investigation 12(3): 417–419. use of mobile phone data to inform analysis of COVID-19 Page, M.J., J.E. McKenzie, P.M. Bossuyt, et al. 2021. The pandemic epidemiology. Nature Communications 11(1): 1–8. PRISMA 2020 statement: An updated guideline for reporting Gulliver, R., M. Fahmi, and D. Abramson. 2020. Technical consid- systematic reviews. BMJ 372: n71. erations when implementing digital infrastructure for social Park, S., G.J. Choi, and H. Ko. 2020. Information technology- policy. Australian Journal of Social Issues 56(2): 269–287. based tracing strategy in response to COVID-19 in South Guo, Y., Y. Zhang, T. Lyu, et al. 2021. The application of artificial Korea—privacy controversies. JAMA 323(21): 2129–2130. intelligence and data integration in COVID-19 studies: A Pham, M.T., A. Rajić, J.D. Greig, J.M. Sargeant, A. scoping review. Journal of the American Medical Papadopoulos, and S.A. McEwan. 2014. A scoping review Informatics Association 28(9): 2050–2067. of scoping reviews: Advancing the approach and enhancing Hellewell, J., S. Abbott, A. Gimma, N.I. Bosse, C.I Jarvis,T.W. the consistency. Research Synthesis Methods 5(4): 371–385. Russell, ... and R.M. Eggo. 2020. Feasibility of controlling COVID-19 outbreaks by isolation of cases and contacts. The Ravizza, A., F. Sternini, F. Molinari, E. Santoro, and F. Cabitza. 2021. Lancet Global Health 8(4), e488–e496. A proposal for COVID-19 applications enabling extensive epi- Hendl, T., and T. Roxanne, 2022. Digital surveillance in a pan- demiological studies. Procedia Computer Science 181: 589–596. demic response: What bioethics ought to learn from Roche, S. 2020. Smile, you’re being traced! Some thoughts about Indigenous perspectives. Bioethics 36(3): 305–312. the ethical issues of digital contact tracing applications. Hisada, S., T. Murayama, K. Tsubouchi, et al. 2020. Surveillance Journal of Location Based Services 14(2): 71–91. of early stage COVID-19 clusters using search query logs Röösli, E., B. Rice, and T. Hernandez-Boussard. 2021. Bias at and mobile device-based location information. Scientific warp speed: How AI may contribute to the disparities gap in Reports 10(1): 18680. the time of COVID-19. Journal of the American Medical Jamshidi, M., A. Lalbakhsh, J. Talla et al. 2020. Artificial intelli- Informatics Association 28(1): 190–192. gence and COVID-19: Deep learning approaches for diag- Roy, A., V. Iosifidis, and E. Ntoutsi. 2021. Multi-Fair Pareto nosis and treatment. IEEE Access 8: 109581–109595. Boosting. arXiv preprint. arXiv:2104.13312. Kapilashrami A., and O. Hankivsky. 2018. Intersectionality and why Sáez, C., N. Romero, J.A. Conejero, and J.M. García-Gómez. it matters to global health. Lancet 391(10140): 2589–2591. 2021. Potential limitations in COVID-19 machine learning Klingwort, J., and R. Schnell, 2020. Critical limitations of digital due to data source variability: A case study in the nCov2019 epidemiology: Why COVID-19 apps are useless. Survey dataset. Journal of the American Medical Informatics Research Methods 14(2): 95–101. Association 28(2): 360–364. Mali, S.N., and A.P. Pratap. 2020. Targeting infectious coronavi- Scott, I.A., and E.W. Coiera. 2020. Can AI help in the fight against rus disease 2019 (COVID-19) with artificial intelligence (AI) COVID-19? Medical Journal of Australia 213(10): 439–441. applications: Evidence based opinion. Infectious Disorders– Shachar, C., S. Gerke, and E.Y. Adashi. 2020. AI surveillance Drug Targets 21(4): 475–477. during pandemics: Ethical implementation imperatives. Malik, Y.S., S. Sircar, S. Bhat, et al. 2021. How artificial intelli- Hastings Center Report 50(3): 18–21. gence may help the Covid-19 pandemic: Pitfalls and lessons Sun, R., W. Wangm, M. Xue, G. Tyson, S. Camtepe, and D.C. for the future. Reviews in Medical Virology 31(5):1–11. Ranasinghe. 2021. An empirical assessment of global Marabelli, M., E. Vaast, and J.L. Li. 2021. Preventing the digital COVID-19 contact tracing applications. 43rd International scars of COVID-19. European Journal of Information Conference on Software Engineering (ICSE): 1085–1097. Systems 30(2): 176–192. Tricco, A.C., E. Lillie, W. Zarin et al. 2016. A scoping review on Mbunge, E. 2020. Integrating emerging technologies into the conduct and reporting of scoping reviews. BMC Medical COVID-19 contact tracing: Opportunities, challenges and Research Methodology 16(1): 1–10. pitfalls. Diabetes Metabolic Syndrome: Clinical Research Tsamados A., N. Aggarwal, J. Cowls et al. 2022. The ethics of and Reviews 14(6): 1631–1636. algorithms: Key problems and solutions. AI & Society 37: Mbunge, E., B. Akinnuwesi, S.G. Fashoto, A.S. Metfula, and P. 215–230. Mashhwama. 2020. A critical review of emerging technolo- Wynants, L., B. Van Calster, G.S. Collins. et al. 2020. Prediction gies for tackling COVID-19 pandemic. Human Behavior and models for diagnosis and prognosis of covid-19: Systematic Emerging Technologies 3(1): 25–39. review and critical appraisal. BMJ 369: m1328. Moseley, D. 2021. Bias. In The international encyclopedia of ethics, edited by H. LaFollette, 1–6. John Wiley & Sons. Zou, J., and L. Schiebinger. 2021. Ensuring that biomedical AI Moss, E., and J. Metcalf. 2020. High tech, high risk: Tech ethics benefits diverse populations. The Lancet 67: 103358. lessons for the COVID-19 pandemic response. Patterns 1(7): 100102 Publisher’s note Springer Nature remains neutral with regard to Munn, Z., M.D.J. Peters, C. Stern, C. Tufanaru, A. McArthur, and jurisdictional claims in published maps and institutional E. Aromataris. 2018. Systematic review or scoping review? affiliations.