AI & SOCIETY https://doi.org/10.1007/s00146-022-01614-9 OPEN FORUM “Just” accuracy? Procedural fairness demands explainability in AI‑based medical resource allocations Jon Rueda1,6 · Janet Delgado Rodríguez2 · Iris Parra Jounou3 · Joaquín Hortal‑Carmona4 · Txetxu Ausín5 · David Rodríguez‑Arias1,6 Received: 7 June 2022 / Accepted: 5 December 2022 © The Author(s) 2022 Abstract The increasing application of artificial intelligence (AI) to healthcare raises both hope and ethical concerns. Some advanced machine learning methods provide accurate clinical predictions at the expense of a significant lack of explainability. Alex John London has defended that accuracy is a more important value than explainability in AI medicine. In this article, we locate the trade-off between accurate performance and explainable algorithms in the context of distributive justice. We acknowledge that accuracy is cardinal from outcome-oriented justice because it helps to maximize patients’ benefits and optimizes limited resources. However, we claim that the opaqueness of the algorithmic black box and its absence of explainability threatens core commitments of procedural fairness such as accountability, avoidance of bias, and transparency. To illustrate this, we discuss liver transplantation as a case of critical medical resources in which the lack of explainability in AI-based allocation algorithms is procedurally unfair. Finally, we provide a number of ethical recommendations for when considering the use of unexplainable algorithms in the distribution of health-related resources. Keywords Artificial intelligence · Distributive justice · Explainability · Medical AI · Procedural fairness 1 Introduction geographical) inputs that could contribute to describe, explain and foresee health and disease (Buch et al. 2018; Traditionally, progress in clinical practice has been made Ellahham et al. 2020). All of this can save time, energy, through the systematic study of accumulated experience and money, especially in the diagnosis and management and statistical analysis of reduced amounts of data. Arti- of patients. Machine Learning (ML) has also been applied ficial intelligence (AI) provides new opportunities for the to reinforce safety by minimizing risks and reducing the development of evidence-based medicine by simultaneously uncertainty about harmful events (Ellahham et al. 2020). observing and rapidly processing an almost unlimited num- The application of AI has achieved a revolutionary change in ber of medical and nonmedical (sociodemographic, ethnic, radiological diagnosis, for instance, by improving the accu- racy of image analysis in the early detection of breast pathol- ogies (Robertson et al. 2018). Patient triage is another area * Jon Rueda in which AI has been introduced, through wearable devices ruetxe@ugr.es designed to monitor remotely and analyse vital signs—e.g., 1 Department of Philosophy 1, University of Granada, consciousness. In these AI systems, algorithms are trained Granada, Spain to classify disease conditions based on severity, which helps 2 University of Granada, Granada, Spain predict survival in the pre-hospital environment (Kim et al. 3 2018; Ellahham et al. 2020), as well as in emergency depart- Universitat Autònoma de Barcelona, Barcelona, Spain ments through electronic triage (e-triage) (Levin et al. 2018; 4 Andalusian Health System, University of Granada, Granada, Ellahham et al. 2020), and in Intensive Care Units (ICUs) Spain (Che et al. 2016; Nanayakkara et al. 2018). 5 Institute of Philosophy, Spanish National Research Council, Much of the excitement around medical AI is due to the Madrid, Spain high accuracy of these models. Advanced AI systems enable 6 FiloLab Scientific Unit of Excellence, University of Granada, diagnoses, prognoses, and clinical recommendations based Granada, Spain 13 Vol.:(0123456789) AI & SOCIETY on numerous variables, which allows tailoring medical deci- London’s main argument in which he advocates for the pri- sions to the specificities of each patient. Unfortunately, the macy of accuracy over explainability. This will be followed increased precision of these systems often occurs at the by a clarification of the role of distributive justice in the con- expense of a decrease in their explainability. Although we text of medical AI, to show how accurate performance and will clarify later that the relationship between both proper- explainable algorithms may conflict. While we recognize ties does not necessarily always involve an inverse corre- the relevance of accuracy from an outcome-oriented justice lation, the ethical debate has mainly focused on the view perspective, we argue that explainability ensures procedural that the more accurate AI is, the less we are able to under- fairness, which is required for accountability, avoidance of stand the mechanism by which the algorithmic outcome bias, and transparency. After that, we will analyse AI-based (and the clinical conclusion) is reached. This trade-off has liver allocation for transplantation as an example to show the sparked an interesting ethical debate on medical AI. Alex advantages of introducing explainability as a key require- John London has claimed that accuracy should be a more ment for AI-based distribution of critical medical resources. important value than explainability in the applications of AI Finally, we develop five practical suggestions for those con- to medicine (London 2019). He argues that the absence of sidering the adoption of unexplainable algorithms for sup- causal knowledge is already a common phenomenon in some porting allocation decisions in healthcare. implemented medical practices—such as the use of aspirin or lithium. Similarly, London points out that empirically validating the accuracy of AI in healthcare is significantly 2 Clarifying the concepts and the debate more important than knowing the underlying factors that lead a system to produce an output. Thus, explainability is AI-based medicine sparked an increasing number of bioethi- an unnecessary expectation that could misdirect the develop- cal controversies and philosophical debates, including the ment of AI in medicine. challenges raised by accurate AI applications in healthcare In this article, we argue that London’s position is prob- that lack explainability. We shall start by clarifying the terms lematic. We focus our critique on one area where the inex- ‘accuracy’ and ‘explainability’. plicability of AI is of particular concern: the allocation of The future of high-performance medicine will prob- scarce medical resources. AI has great potential to support ably rely on the synergy between human and AI interaction decisions about the allocation of scarce resources (such as (Topol 2019). In high-income countries, AI has the potential organs or intensive medical services). Problems of inexpli- to become an integral part of daily medical practice in a cability in this area have been underexplored in the AI ethics wide range of domains—such as radiology, pathology, der- literature. In this respect, the contribution of our article is matology, critical care, ophthalmology, cardiology, gastroen- twofold. On the one hand, we show that, while accuracy is terology, mental health, and so on. One of the main appeals a fundamental value from an outcome-oriented justice per- of advanced machine learning algorithms is that they offer spective, explainability is an indispensable requirement from substantial predictive power, diagnostic accuracy, and more a procedural justice perspective. Some core aspirations in AI tailored drug prescriptions. The clinical applicability and ethics that align with procedural fairness—(i.e. transparency, efficacy of AI-tools certainly require further validation, but avoidance of bias, and accountability) suggest that being some algorithms have already proved to outperform human able to explain the algorithms used for allocating scarce specialists in disease detection, speed of interpretation, risk medical resources is a requirement. Consequently, unlike estimation for readmission and mortality, screening of scans, London’s famous assertion that accuracy is the main value and X-ray image classification. As Eric Topol explains, in that should lead AI-based decisions in medicine, we contend the case of AI, the neural net interpretation is contrasted that explainability also fulfils important ethical functions, with physicians’ evaluation using a plot of true-positive ver- especially in choices that entail distributive justice claims, sus false-positive rates, for which the area under the curve is and should therefore be an aspiration to be pursued in future used to express the level of accuracy (Topol 2019). technical developments. On the other hand, beyond this the- In this context, accuracy commonly refers to the achieved oretical reframing, we also offer some practical guidance. preciseness in the performance of medical diagnostic or As highly accurate predictive algorithms whose internal prognostic tasks. We have, on the one side, precision, which processes are inexplicable may continue to emerge in the is the ratio of true positives to the number of true and false near future, we provide a set of ethical and fairness-based positives. On the other side, recall (also called sensibil- suggestions for evaluating the adoption of these systems in ity) is the relation between true positives and the number distributive healthcare decisions. of true positives and false negatives. The greater accuracy The structure of the article is as follows. We will start of AI applications in healthcare is forged from the training with a conceptual clarification of the terms ‘accuracy’ and of algorithms based on huge amounts of data. Algorithmic ‘explainability’ in medical AI. We will then expand on training in a vast trove of inputs enables the development of 13 AI & SOCIETY a remarkable capability for pattern recognition. Importantly, “the ability to understand and evaluate the internal mech- AI algorithms can not only provide accurate outputs but also anism of a machine, algorithm, or computational process lead to clinical recommendations that are based on masses in human terms” (Cutillo et al. 2020) or, more exactly, the of data that are unmanageable for humans. Overcoming the property of an AI system that enables the ability of an agent human limitation in processing a great number of variables, to understand its mechanisms. Similarly, explaining entails and the remarkable precision of clinical outputs offered by providing “causes of observed phenomena in a comprehen- these algorithms are two of the greatest benefits of the accu- sible manner through a linguistic description of its logical racy of AI in medicine. and causal relationships.” (Holzinger et al. 2019) London, But, as the saying goes, all that glitters is not gold. Some- in a similar vein, recalls an Aristotelian definition: “For times the price to be paid for greater accuracy is the loss Aristotle, explanations are logical arguments in which the of explainability. Accuracy involves a degree of complexity particular to be explained is subsumed under a more general that turns the means by which the output was reached intrac- set of claims that clarify the causal factors responsible for table. This is a particular challenge for various deep-learning generating the particular.” (London 2019, p. 16) In other models in medicine. While the designers may understand words, explainability would exist when the causal relation- the architecture of the AI system that processes and classi- ships found in the model and their reasons or dynamics are fies the data, it is often impossible to explain how and why known. However, explainability, as well as accuracy, is not a particular outcome is given (London 2019). Hence, the a fixed or binary property, but a gradual one—both being operation of the evolving and self-trained algorithms may sometimes measured in percentages. In the clinical setting, derive into a “black box” that makes the decision process broadly speaking, health professionals can explain in more inscrutable to humans. In short, accurate AI-based clinical detail the output if they are able to understand and logi- estimates may often lack substantial explainability. In many cally communicate how and why the AI support system has cases, moreover, the degree (or absence) of explainability reached a particular result. can be proportional to the accuracy of the predictions. Con- Explanations can be subject-centric or model-centric sider the following insightful fragment: (Watson et al. 2019a; Tsamados et al. 2020). The former refers primarily to the audience of the explanations, that is, There is an inherent tension between machine learning the ultimate recipients of AI-based decisions and the targets performance (predictive accuracy) and explainability. for which explanations can be adjusted. The latter refers to Often the best-performing methods such as deep learn- explanations of the model independently of the target of ing are the least transparent, and the ones providing a explanations. These two approaches are not entirely inde- clear explanation (e.g., decision trees) are less accurate pendent of one another. For instance, the black box problem (Holzinger et al. 2019). arises when it is not possible to know, not even for the devel- There is, however, an important caveat here. Greater opers of the algorithm, the inner logic that has led from the accuracy does not necessarily imply a decrease in the processing of the data to the corresponding result (Vayena explainability of the system. While this is something that et al. 2018). Certainly, it can be of great scientific interest happens in many developments, it does not mean that it is to the developers to know how and why the deep learning not technically possible to obtain very precise and at the model has reached a particular output, as explanations have same time highly explainable systems. In fact, there are epistemic value (Robbins 2019; Durán 2021). recent examples where satisfactory levels of both properties Although explanations can be thus epistemically signifi- have achieved in medical applications, for example, using cant to persons, in this article we are concerned about the capsule networks (LaLonde et al. 2020; Gulum et al. 2021). ethical implications of (un)explainability. The approach to The tension between accuracy and explainability may thus explainability that prevails in the clinical setting is the sub- not be so inherent in the future. As we shall see below, the ject-centric one. Explanations are due to patients. Suppose fact that both characteristics are not always mutually exclu- that, as some have suggested, those affected by the AI-based sive in the rapidly evolving technical domain is an important clinical decision support systems are entitled to request an nuance for the ethical debate. explanation (Robbins 2019). Such explanations are not due What does explainability mean, anyway? Explainability to patients because of their epistemic value, but because of is an epistemic concept that refers to the possibility—and, their ethical one: clinical decisions are morally relevant and more often, the difficulty—to know how and why an AI require justifications—they must not be arbitrary—because algorithm has yielded a particular output. It answers the they affect patients’ health and well-being. question of how does it work? in terms of intelligibility Consequently, the relevance of explainability can be (Floridi et al. 2018). The know-how and know-why notions advocated (and challenged) on ethical grounds. Explain- relate to the development of a causal understanding of output ability has been considered, for instance, one of the core production. In that sense, explainability can be defined as principles of AI Ethics. According to Floridi et al. (2022), 13 AI & SOCIETY the justification of the principle of explainability is not only of AI in healthcare settings. According to London, what defended by the significance of its epistemological facet (i.e., is of utmost importance in decisions based on AI systems to interpret and understand how the system works), but also is accuracy—defined as the knowledge of particular facts from the ethical need for holding accountable the system that are proven and supported by empirical evidence. If the for the way it works. Hence, explainability is not described results are accurate,1 and if it is possible to empirically ver- in terms of causal knowledge but intertwined with transpar- ify that accuracy, explainability (which would derive from ency of the internal processes of the AI models and their general principles that explain these particularities) is then underlying assumptions that can reproduce societal biases dispensable. Standard clinical models show regularities and and systemic inequalities. Moreover, Amann et al. stated patterns in the data introduced through associations, but do that unexplainable AI-based clinical decision support sys- not directly track causal relationships. Therefore, requesting tems threaten core ethical values in medicine (including the explainability from machine learning techniques would be traditional bioethical principles of autonomy, beneficence, too demanding and inconsistent. The requirement of explain- non-maleficence, and justice) and may even lead to deleteri- ability may in some cases be even deleterious, as the demand ous consequences for individuals as well as for public health for more explainability from some deep-learning algorithms (Amann et al. 2020). For instance, if erroneous clinical deci- may reduce their accuracy (London 2019, p. 15). sions based on unexplainable algorithms have a detrimental Likewise, Scott Robbins argues that not all medical deci- impact on patients’ well-being or if they lead to widening sions need explanations. We routinely rely on pharmaco- existing socially rooted health injustices, ethical qualms will logical treatments that we do not know how they work, but inevitably arise. In addition, as Char et al. show, the safety we know they work (e.g., lithium, the same example given of any healthcare application depends upon the ability to by London). Robbins acknowledges that explainability may inspect it to see the mechanisms at work and understand how sometimes be useful, but he denies that the principle of the application might fail (Char et al. 2020). Non-inspectable explainability should be required across the board (Robbins AI systems may sometimes pose a risk of harming patients 2022, p. 497). The requirement of explainability is therefore and so questions may arise about the responsibility of the not always ethically necessary. algorithms in these cases, which may create, in turn, a signif- After these preliminary clarifications, it should be clearer icant backlash against AI. (Needless to say, an adverse soci- that the debate we are engaging with relates to the ethical etal reaction may also occur if the algorithmic predictions tensions involved in the trade-off between accuracy and are not accurate enough). This backlash may consist, among explainability in AI-based medicine. In the next section, we other things, in discouraging public approval and a lack of will focus on the prospective impact of AI in the allocation trust in unexplainable systems that makes patients, health of scarce medical services and resources. We will suggest professionals, and society reluctant to these AI applications. that some concerns related to distributive justice, which have Still, some authors have compellingly argued that explain- remained mainly underexplored in XAI debates, can make ability is not always a necessary aspiration in medical AI us rethink the ethical requirement of explainability in AI (London 2019; Robbins 2019). They question whether the medicine. requirement of explainability is too demanding and whether it is consistent with how other medical practices are valued. In some way, the rise of the explainable artificial intelli- 3 Distributive justice and medical AI gence (XAI) debate in health practices makes us reflect on the broader epistemological foundations and ethical goals Distributive justice theories analyse how to fairly allocate of medicine. Among these authors, the view presented by socially valuable goods and resources. Resource constraints London is of particular interest (London 2019). London in medicine limit the capacity to meet existing demands and rightly reminds us that uncertainty and incompleteness are needs, resulting in a wide variety of ethical, political, and pervasive characteristics of medical knowledge. The absence economic controversies about how to allocate scarce health- of a thorough understanding of the underlying causal mecha- related resources. In this section, we shall address how the nisms in the pathophysiology of some diseases or specific tug-of-war between accuracy and explainability involves drug recommendations pervades medical decision-making. a discussion of distributive justice. We will engage in this Physicians cannot always explain how and why a particu- debate from the perspectives of outcome-oriented justice and lar intervention works, without this being consequential for procedural justice. We argue that whereas accuracy is mainly their practice or their patients’ entitlements. Consequently, as London remarks: “decisions that are atheoretic, associa- tionist, and opaque are commonplace in medicine.” (London 1 It may also be possible to interpret that accuracy works in Lon- 2019, p. 17) These reflections are very relevant because they don’s paper as a stand-in for beneficence, the core bioethics principle. make us question what we should value in the application We will return to this idea in the next section. 13 AI & SOCIETY relevant from outcome-oriented stances, explainability is a explainability and utility are not incompatible. Being capa- requirement for procedural fairness accounts. We finally ble of explaining the AI mechanism that leads to the out- claim that distributive decisions supported by AI should be come may be useful to acquire epistemic knowledge and based on ‘comprehensive outcomes’, to try to reconcile the to check if inappropriate considerations have been taken ethical values of explainability and accuracy in the upcom- into account in the process (Robbins 2022). Such epistemic ing development of AI. knowledge can subsequently improve clinical decisions, Outcome-oriented justice highlights that achieving fair making them more accurate and thereby increasing effec- effective results is the main criterion by which distribution tiveness. Regarding inappropriate considerations, explain- justice can be assessed. Looking for effective outcomes is ability can be necessary to verify whether the process has measured by the ability to bring about a beneficial state of considered factors that are clinically and ethically irrelevant. affairs: justice is promoted when valuable results are fairly An essential point is, therefore, to verify whether the results materialized in the real world. Various theories adopt an are unethically biased, that is, if they unfairly favour or dis- outcome-oriented account when discussing distributive jus- advantage particular collectives based on morally irrelevant tice issues. Consequentialism and its utilitarian variants are characteristics. probably the most prominent examples. Utilitarianism aims At this point, outcome-oriented justice encounters at least to maximize the best consequences, yielding the greatest two challenges. First, by maximizing clinical utility through amount of good for the greatest number of people (Bentham the most accurate and individualized prospective outcome 1789; Mill 1863). In medical debates, utilitarian positions analysis, this can systematically favour or disadvantage spe- commonly underscore the need for making good and fair cific groups (Starke et al. 2021). This is a recurrent critique use of scarce resources through maximizing benefits and of utilitarianism in debates on rationing, namely when the attending to cost-effectiveness. Benefits can be measured, for scarcity forces us to choose whom the resource will be allo- instance, in terms of the maximum number of lives saved, cated to among the patients who can benefit from it. Some of the number of years of life produced, and of quality- groups benefit less from scarce resources due to age or worse adjusted-life-years (so-called QALYs) (Williams 1985). health status, implying that the elderly or the poor (who have However, if we consider non-utilitarian outcome-oriented greater comorbidities due to social determinants of health) theories of justice such as Martha Nussbaum’s version of typically fare worse in rationing decisions (Marmot 2017; the capability approach, benefits can then be measured in Rueda 2021). While addressing in more detail this contro- terms of the fair distribution of central human capabilities versy is beyond the scope of this article, it is worth men- worldwide (Nussbaum 2007). tioning that looking for maximized accuracy in the benefits From typical outcome-oriented justice perspectives, accu- is not morally neutral. Rather, that conception implicitly racy is generally more important than explainability when displays what is most desirable in the use of AI to base dis- AI is used to support distributive decisions. When it comes tributive decisions. to optimizing outcomes, accuracy is of primary importance. Second, if explainability is required to avoid unethical Starke et al. have claimed—on pragmatist grounds—that biases, it is because the processes (and not only the results) “the outcome-based clinical utility of any medical machine need to be addressed. Procedural justice accounts become learning program should be put to the forefront.” (Starke important as well. In what follows, we pay attention to that et al. 2021) From an outcome-oriented justice, the loss of dimension of distributive justice as applied to the discus- explainability can be a reasonable price to pay when benefits sions on medical AI. outweigh the costs. Procedural justice emphasizes that the process on In this regard, London (2019) points out that medical which the decisions are based is a fundamental aspect of practice frequently considers the experience of benefit with- the judgments about justice. In other words, something is out enough knowledge of the underlying causal system to fair depending on the decision-making procedure that has describe how the benefits are brought about. In this sense, led to its outcome. There are different possible applica- the opacity and lack of causal knowledge of some machine tions of procedural fairness approaches depending on the learning approaches are not so different from daily aspects of domain. A famous example of this conception of justice is medical decision-making. The causal understanding is often John Rawls’ contractarian approach to political philosophy incomplete, and in these cases, the empirical validation of an (Rawls 1971). In the legal context, conversely, procedural intervention’s practical benefits is the most important task. If justice is sometimes related to having due process. Moreo- the demand for explanations of how some interventions work ver, there has been an increase of the philosophical literature exceeds careful empirical validation, patients, resources and on procedural fairness in the development of ML, and it progress is negatively affected, London states. is generally concerned with how to operationalize fairness That being said, the perspective of outcome-oriented statistically in algorithms, how to problematize the back- justice is not necessarily opposed to explainability, as ground of underlying structural injustice, or how to suggest 13 AI & SOCIETY what kinds of processes would lead to fairer final decisions when the model has operated with unrepresentative samples, (Hedden 2021; Zimmermann and Lee-Stronach 2022; Grote inaccurate or limited training data sets, skewed inputs given and Keeling 2022). Furthermore, in public health ethics and by human operators in the labelling of samples, proxies that health politics, a prominent version of procedural justice is hide variables connected to social identities such as race, the “accountability for reasonableness” defended by Normal gender or social class. The problem of bias, nevertheless, Daniels. This proposal was developed to address healthcare is not solved by simply trying to assess algorithmic perfor- priority setting in contexts of resource constraints (Daniels mance across diverse demographics. Technology-centred and Sabin 1998). Accountability for reasonableness—which solutions are limited when they neglect that biases are also remarks that fair processes need transparency, publicity on a sociopolitical issue related to underlying health inequi- rationales, and open mechanisms to revise the decisions— ties in society (Pot et al. 2021). Biases can surreptitiously can be applied to XAI and distributive justice in medicine. lead to favouring or disadvantaging particular social groups We shall focus next on the connection of explainability with in contexts of historical discrimination (Char et al. 2020; transparency, avoidance of bias, and accountability. Moss and Metcalf 2020), which can lead AI to reproduce Transparency in this context has been defined as the societal prejudices and systemic inequalities, or even rein- procedure that makes the inner workings of AI algorithms force discriminatory practices. An opaque or unexplainable observable, informing about the underlying data, variables, procedure prevents the verification of whether the decision is and relations within the system that lead to the outcome free from inappropriate considerations and unethical biases (Durán and Jongsma 2021). Transparency is commonly (Robbins 2019, pp. 497–8). Physicians and patients are thus opposed to opacity, which characterizes hidden processes not only concerned about the mere outputs but also about the that are inscrutable for humans (e.g., when it is impossible characteristics and features on which such results are based to survey the algorithm training data). Sometimes, the term (Amann et al. 2020). Explainability plays a crucial role in ‘transparency’ is paired with explainability, as when used to facilitating output accountability, that is, following Floridi refer to “the transparency of the reasons for the AI-generated et al., the sense in which someone is responsible for the way decision” (Robbins 2022, p. 500). Both terms have different the AI system works (Floridi et al. 2018). conceptual nuances, though. Transparency is best under- Explainability can also be fundamental for the ascription stood as a sort of possibility condition of interpretability of moral responsibility, which is also related to account- and explainability. In that sense, transparency of the internal ability. The ascription of moral responsibility is a daunting mechanisms of the model is what enables explainability in challenge in scenarios with multiple factors and actors that ethical terms (Tsamados et al. 2020). Importantly, transpar- influence the decisions. Suppose that AI-based clinical deci- ency of the AI processes is not always intrinsically valu- sion support systems gain a key role in tragic distributive able and, in some cases, it can be problematic—e.g., when dilemmas—e.g., when rationing implies denying to a patient transparency unjustifiably endangers the privacy of patients a life-saving medical resource. We think that inexplicabil- (Watson et al. 2019b). However, transparency can be instru- ity—and the resulting lack of accountability—would be mentally valuable beyond enabling explainability. Transpar- especially problematic in those cases because the denial of ency facilitates uncovering sources of biases (Starke et al. the resource may result in the death of the non-selected per- 2021), accountability (Coeckelbergh 2020), which is highly son. Of course, patients could also complain about the use of related to safety in the clinical setting (Char et al. 2020; more explainable but less accurate algorithms, as far as low Felder 2021; Tsamados et al. 2020, Yoon et al. 2021), and precision may lead to misallocating resources. But the point the trustworthiness of the system (Vayena et al. 2018; Char here is that the lack of transparency and explainability may et al. 2020; Tsamados et al. 2020; Durán and Jongsma 2021). undermine scrutability, making it difficult to trace the moral Explainability and transparency are important to avoid responsibility for the decisions that have been made (Tsa- biases in AI. Biases are one-sided tendencies or systematic mados et al. 2022). Insofar as explainability involves under- deviations or errors (Moseley 2019; Amann et al. 2020; Char standing the interconnected causal steps behind an output, it et al. 2020; Starke et al. 2021; Tsamados et al. 2020). The is related to the re-traceability of the chains of events that is term ‘bias’ is commonly employed pejoratively as it usu- fundamental to accountability judgments. Therefore, having ally hinders our epistemological and ethical goals. At the access to the steps an algorithm followed and being able to epistemological level, biases lead us away from truth; in the explain them is useful to assign responsibilities in the event normative domain, they lead us away from justice. Biased of bad consequences happening. This is of particular inter- data can be harmful, as they lead to unjustified bad clini- est to the stakeholders. From an ethical checking perspec- cal consequences for some patients. There are various types tive, it is a good governance mechanism to inform affected and sources of biases (and possible solutions for them) in individuals how an algorithm-based decision was reached medical AI (Vayena et al. 2018; Chorás et al. 2020; Cutillo (Mökander and Floridi 2021). Moreover, the quest for et al. 2020). For instance, AI can lead to biased outcomes accountability is not just a matter of settling responsibilities 13 AI & SOCIETY when bad outcomes occur. As long as algorithms begin to AI-based liver allocation among candidates for a transplant. silently structure the allocation of medical goods, the firms Our purpose is to turn our previous theoretical framework that develop them must be accountable because otherwise into a concrete case and show the practical relevance of the they will be incentivized to create progressively more com- trade-off between accuracy and explainability. plex and inscrutable programs (Martin 2019). For the last two decades, the distribution of transplantable Finally, we shall warn that the two approaches to jus- livers has been governed in most countries by a criterion of tice that have discussed display legitimate interests that are need/urgency based on short-term mortality prediction. In not always opposed to each other. Presenting the choice other words, priority is given to those who are at the greatest between maximization of benefits or fair procedures as a risk of dying while awaiting transplantation. This prediction false dichotomy would be unsatisfactory, since both are rel- is commonly made by a scoring system, the Model for End- evant aspirations of justice for many, including us. We also stage of Liver Disease (MELD), which uses linear regression believe that there are theoretical examples that can help us on three recipient variables: creatine, bilirubin, and Interna- to build bridges between outcome-oriented and procedural tional Normalized Ratio (Freeman 2007). The more recent perspectives. introduction of suboptimal organs (from donors with greater Consider Amartya Sen’s distinction between “culmina- deterioration) underscores the need of introducing donor tion outcomes” and “comprehensive outcomes” (Sen 1997, variables into allocation decisions. The goal is to specify the 2009). The former are mere outcomes that do not attend risk of transplant failure and, by doing so, to maximize organ to the processes that generate them, while the latter do. longevity. If suboptimal organs were transplanted to patients Although this distinction has gone unnoticed in the realm of with a good prognosis, re-transplantation would probably be AI ethics, this categorization has immediate consequences. necessary. Conversely, if optimal organs were transplanted Giving value to comprehensive outcomes requires us to seek in patients with a poor prognosis, the patient would likely not only precise results but also to pay attention to the pro- die with a functional organ that could have provided more cesses that produce them, where we believe that explain- life to another recipient. These inefficiencies in the context ability should occupy an important role. Thus, we think that, of chronic and increasing organ shortages are the entry point ideally, a relevant normative aspiration for the fair distribu- for AI tools, which are aimed at achieving more accurate tion of scarce resources through AI is the pursuit of compre- post-transplant predictions (Brown et al. 2012; Ershoff et al. hensive outcomes. In other words, the ethical ideal guiding 2020, Wingfield et al. 2020). developers of distributive AI algorithms in healthcare should The first evolution towards a more complex predictive seek to maximize both the accuracy of the predictions and algorithm that takes into account both donor and recipi- the explainability of their processes. ent characteristics is the Transplant Benefit System (TBS), The problem lies, as we have shown in the previous sec- implemented in the United Kingdom. Although not properly tion, in that in the practical current and near-state develop- generated with AI models, this system uses 28 variables—7 ments of AI it is often difficult to reconcile accuracy with from the donor and 21 from the recipient—to establish a explainability. So, this ideal of seeking comprehensive better prediction of donor-recipient matching outcomes. This outcomes may seem difficult to realize in the short term, system aims at directing transplantable livers to the recipient although it is not technically impossible. Precisely, the fact where the organ can last the longest (Wingfield et al. 2020). that there are already some successful examples in terms of The greater complexity of the TBS model already poses an accuracy and explainability (as we mentioned in the previous explainability problem with respect to the previous MELD section), reinforces our thesis that future efforts in AI design model, as the increase of variables blurs their connection should try to combine both aspirations for the allocation of with the result. However, they all have a linear relationship scarce health-related resources. This, of course, does not with the result, which makes it possible to assess the relative exempt us from continuing to consider real-world examples weight they bring to the allocation decision. where the ethical trade-off is present. Therefore, in the next Neural networks have also been used to generate models section, we will approach a practical case in which the analy- for predicting 90-day mortality after hepatic transplanta- sis of the trade-off between accuracy and explainability can tion that are expected to be more accurate than linear mod- illuminate this kind of distributive challenge on medical AI. els (Briceño et al. 2014). These models detect nonlinear relationships that optimize organ-recipient matching and allow the liver to be directed to the pair with the best prog- 4 The case of AI‑based liver allocation nosis for organ survival within the recipient (Brown et al. 2012; Dorado-Moreno et al. 2017; Briceño 2020; Ershoff AI can influence the allocation of scarce healthcare resources et al. 2020). Apart from its direct application for alloca- in a variety of domains, such as ICU prognosis or organ tion purposes, these models have a wide range of applica- distribution. In this section, we briefly address the case of tions as a tool to improve physician confidence in marginal 13 AI & SOCIETY organs’ usefulness and to personalize informed consent for 5 Recommendations for the distributive use transplantation (Wingfield et al. 2020). of unexplainable algorithms In comparison with linear models, AI-based liver dis- tribution algorithms have increased accuracy (Dorado- Ideally, future developments of AI to support allocation Moreno et al. 2017; Briceño 2020). However, these decisions should reconcile the internal explainability of models do not allow assessing or reporting the relative the system with a highly accurate predictive capability weight of each of the input variables in the final prediction as far as technically possible. However, in the short term, (Briceño et al. 2014; Wingfield et al. 2020; Briceño 2020). we will probably continue to have inexplicable algorithms Although variables that should be considered irrelevant in that may help predict benefits in distributional dilemmas. liver distribution—e.g., patient’s place of residence, occu- It follows from our previous arguments that using opaque pation, gender or ethnicity—can be excluded from the AI algorithms to allocate scarce health resources is problem- programming, its content may be indirectly captured by atic, but not completely dismissable. The use of highly the combination of other clinical variables, thus having a accurate but inexplicable AI systems may be ethically certain, although not measurable, impact on the allocation justifiable in some circumstances. What considerations decision. This involves a risk of unfairness resulting from should we make to render the use of unexplainable algo- a deficit of explainability: data that is a priori innocuous rithms ethically acceptable? In this final section, we offer and has no understandable or justifiable relationship with some recommendations to assess the ethical (in)adequacy transplant survival may become inadvertently decisive in of using unexplainable AI in the future allocation of lim- the allocation judgment. And still, disregarding these vari- ited medical resources. In what follows, we shall provide ables would decrease the accuracy of the matching and five key considerations: may result in allocation errors. This is an example of the tension between accuracy 1. Ensuring trust through monitoring public preferences. and explainability in the distribution of a scarce medical To avoid social disapproval and backlash against AI resource. Higher accuracy in graft survival prediction can adoption, it is important to draw on available empiri- only be obtained at the expense of losing explainability to cal studies on when the loss of explainability is widely patients and clinicians (Wingfield et al. 2020). On the one disfavoured. hand, AI-based liver allocation has the potential of giving accurate predictions of graft survival and helps to optimize As in any public ethical debate, generating solid evi- the benefits that each organ can provide. Tailoring the allo- dence on population attitudes is critical. To our knowl- cation based on donor-recipient matching is valuable from edge, there is a significant lacuna in empirical data on an outcome-oriented perspective, as it helps to comply with people’s preferences on the issue we have been discussing, the aspiration of efficiency. with a few notable exceptions. In a recent article, Nuss- On the other hand, the loss of explainability is worri- berger and colleagues have shown how people have robust some from a procedural fairness perspective. Renouncing and positive attitudes toward both interpretability and the explainability of these processes entails giving up the accuracy in the application of AI. However, interestingly, pretension of knowing the fundamental factors that account when there is a direct trade-off between these two proper- for and justify the output. Inexplicability makes it difficult to ties in high-stake contexts as in the allocation of scarce know whether and to what extent the model has considered medical treatments, people tend to sacrifice interpretability variables that are deemed ethically irrelevant. The reduc- in favour of accuracy (Nussberger et al. 2022). Similarly, tion of explainability menaces the aspiration of holding the in another recent article that used citizens’ juries, most model accountable in case of unethical biases or bad conse- jurors preferred to preserve a higher level of precision at quences for the recipients. Patients who are put behind on the cost of reducing explainability in the case of kidney the waiting list or ruled out from organ allocation may find transplantation, but they value explainability more than themselves entitled to request on what basis the allocation accuracy in other contexts of AI application such as crimi- is grounded—especially when the decision may result in nal justice (Veer et al. 2021). the death of the rejected recipient. This may reduce the trust The previous line of research is relevant for at least two in the AI-based distributive decision, which could in turn reasons. On the one hand, it demonstrates that both accu- threaten public support for the whole transplantation system. racy and explainability are important for building public To summarize, the use of emerging AI models for hepatic confidence in medical AI. But, on the other hand, it shows transplantation offers the promise of remarkable predictive that the value of explainability should not be overestimated accuracy, but the inherent byproduct of this gain is an ethi- when there is a trade-off with predictive accuracy. Need- cally expensive—in terms of procedural fairness—loss of less to say, ethical disputes cannot be solved in referendum explainability. 13 AI & SOCIETY mode (Savulescu et al. 2021). Yet, having empirical evi- the algorithm, but the entire distributive process that goes dence on the majority public preferences is important in beyond algorithmic functioning. Thus, although AI predic- terms of assessing which technological innovations may tion may be gestated in inexplicable algorithmic processes, or may not have societal backlash. In this respect, we call explainability may play a role at other points in the proce- for more empirical research on the public attitudes about dure of deciding and justifying the concrete resource allo- using unexplainable but accurate algorithms to distribute cation. When explanations of the predictive process cannot health resources in real-world cases. be given, explanations of distributional decisions must be provided to avoid a sense of arbitrariness. 2. Attending context-sensitivity and resource dependency. In forthcoming AI-based distributive judgments, the 4. Fulfilling the institutional duty to provide explanations. trade-off between accuracy and explainability should be Patients affected by the use of unexplainable algorithms context-dependent and sensitive to the resources avail- in distributional contexts should be able to receive, if able. they wish, explanations—not necessarily about the con- crete algorithmic inner-working—but of the healthcare This recommendation is best understood with a couple of institution’s general rationale for adopting AI for sup- examples, in which we distinguish prioritisation from ration- porting these assignments, of the ethical considerations ing cases. Imagine using AI to prioritise (i.e. establish the in the design and training of the algorithms, and of the order of access to) ICU beds. Using opaque algorithms to main ethical criteria that have been used to guide the suggest which patients should be allocated beds first (e.g., distribution. based on the prediction of worsening health status) is not so problematic if all candidates eventually access the resource. The fact that explanations cannot be given for the internal Similarly, and returning to the case of the previous section, mechanisms of predictive algorithms does not preclude other if we have three unassigned transplantable livers and we types of explanations. Indeed, some have argued that tech- have three or fewer compatible candidates on the waiting nologically-focused approaches to explainability are limited. list, accuracy may be preferable to explainability if we seek It is not simply a matter of providing technical explanations to optimize the benefits of these organs, and because no one of the medical algorithm, but of providing institutional will be excluded. This will be different, however, in a ration- explanations of the reasons for its adoption (Theunissen ing situation, where some potential beneficiaries would be and Browning 2022). Healthcare institutions should convey denied the resource. In those situations, we should weigh meaningful information to end-users and to medical profes- more carefully the implications of using unexplainable AI. sionals about the rationale of using highly-accurate algo- After all, context-dependency in a non-ideal world means rithms despite their inexplicability. These explanations may that allocation decisions should be case-by-case, depend- be aimed, among others, at presenting the main reason for ing on the resources available and the number of potential this technological adoption, justifying when waiving higher candidates. levels of explainability is acceptable in the interest of main- taining high predictive accuracy, ensuring that the company 3. Acknowledging that an algorithmic prediction does not developing the algorithm took precautions to avoid unfair equate to a distributive decision. The use of predictive biases or demographic incompatibilities, and reporting on self-learning algorithms does not entail automated deci- the ethical principles that have guided the assignment. Thus, sions; namely, AI intends to support human and institu- health systems adopting AI can partially satisfy the value tional agency but should not replace it, especially when of explainability in the allocation of resources influenced the prediction is based on inscrutable algorithmic pro- by AI. cesses. 5. Considering alternatives and measuring costs. In assess- The distinction between prediction and decision is rel- ing the use of unexplainable algorithms, the opportunity evant from a fairness perspective (Hedden 2021). The unfair- cost of not using them, the economic and environmen- ness of the algorithmic process (relative to statistical criteria, tal cost of adopting them, and the available alternatives its opaqueness, or the presence of biases) is not identical must be taken into account. to the unfairness of the final decisions (Grote and Keeling 2022). Among other reasons, because the final decision When considering the introduction of AI for the alloca- does not have to follow the algorithmic recommendation. tion of scarce health resources, the following three issues In addition, decoupling prediction from decision allows need to be addressed. First, weighing the need and justifica- us to broaden the fairness-related idea of ‘process’. Then, tion for AI; it must be assessed whether and how AI is a sig- by ‘process’ we do not simply mean the inner working of nificant improvement over previous predictive systems, even 13 AI & SOCIETY if it relies on inexplicable algorithmic processes. Second, the call Ayudas Fundación BBVA a Equipos de Investigación Científica opportunity costs (that is, the value or disvalue of forgoing SARS-CoV-2 y COVID-19, en el área de Humanidades. JR also thanks a La Caixa Foundation INPhINIT Retaining Fellowship (LCF/BQ/ the use of AI to support distributional decisions) must be DR20/11790005). DR-A thanks the funding of the Spanish Research considered. Third, the economic and environmental costs of Agency (codes FFI2017-88913-P and PID2020-118729RB-I00). such systems, if they are to be adopted, must be measured IPJ also thanks the funding of the Spanish Research Agency (code and taken into account. PID2019-105422GB-I00). Data availability statement Not applicable. 6 Concluding remarks Declarations Conflict of interest All authors declare that they have no conflict of We have shown that AI may increasingly generate diffi- interest. cult ethical challenges in the distribution of scarce medi- cal resources, such as organ allocation, and that not all Open Access This article is licensed under a Creative Commons Attri- theories of justice would place the same value on accuracy bution 4.0 International License, which permits use, sharing, adapta- tion, distribution and reproduction in any medium or format, as long and explainability. While accuracy is a fundamental value as you give appropriate credit to the original author(s) and the source, of outcome-oriented justice, explainability is an indispen- provide a link to the Creative Commons licence, and indicate if changes sable requirement of procedural fairness. As far as techni- were made. The images or other third party material in this article are cally possible, we have argued that AI developments should included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in ideally pursue ‘comprehensive outcomes’, accommodating the article's Creative Commons licence and your intended use is not the importance of outcomes and the processes that produce permitted by statutory regulation or exceeds the permitted use, you will them. We hope that this article’s contribution helps to use- need to obtain permission directly from the copyright holder. To view a fully reframe the debate on distributive justice in medical copy of this licence, visit http://​creat​iveco​mmons.​org/​licen​ses/​by/4.​0/. AI. Furthermore, in cases where the use of highly accurate but inexplicable algorithms may be beneficial in supporting References distribution decisions, we have offered five recommenda- tions for ethically assessing the adoption of AI. These sug- Amann J, Blasimme A, Vayena E, Frey D, Madai VI (2020) Explain- gestions are not exhaustive, and future contributions will ability for artificial intelligence in healthcare: a multidisciplinary need to expand and refine them, but they may offer practical perspective. BMC Med Inform Decis Mak 20(1):1–9. https://​doi.​ org/​10.​1186/​s12911-​020-​01332-6 guidance for those considering the introduction of these ML Bentham J (1789) An introduction to the principles of morals and leg- systems in healthcare settings. islation. Clarendon Press, Oxford (1907) Finally, the challenges presented here require further Briceño J (2020) Artificial intelligence and organ transplantation: chal- societal deliberation. We believe that one of the key axes lenges and expectations. Curr Opin Organ Transplant 25(4):393– 398. https://​doi.​org/​10.​1097/​MOT.​00000​00000​000775 of the future debate should revolve around the factors of Briceño J, Cruz-Ramírez M, Prieto M, Navasa M, Ortiz J, de Urbina R, public acceptance of these AI applications. According to Orti M-ÁG-B et al (2014) Use of artificial intelligence as an inno- our argument, maximizing comprehensive outcomes could vative donor-recipient matching model for liver transplantation: mean greater transparency, accountability, and avoidance results from a multicenter Spanish study. J Hepatol 61(5):1020– 1028. https://​doi.​org/​10.​1016/j.​jhep.​2014.​05.​039 of bias. Linking explainability to accuracy in the outcomes Brown TS, Elster EA, Stevens K, Graybill JC, Gillern S, Phinney S, can contribute to generating a “virtuous circle” in AI-based Salifu MO, Jindal RM (2012) Bayesian modeling of pretransplant medical resource allocations. Both properties are valuable variables accurately predicts kidney graft survival. Am J Nephrol since public confidence in AI is therefore at stake. And also 36(6):561–569. https://​doi.​org/​10.​1159/​00034​5552 Buch VH, Ahmed I, Maruthappu M (2018) Artificial intelligence in not holding back its beneficial developments that can assist medicine: current trends and future possibilities. Br J Gen Pract us, if aligned with prominent ethical ideals, in a fairer alloca- 68(668):143–144. https://​doi.​org/​10.​3399/​bjgp1​8X695​213 tion of our healthcare resources. Char DS, Abràmoff MD, Feudtner C (2020) Identifying ethical con- siderations for machine learning healthcare applications. Am J Acknowledgements We thank the audience of the Workshop Bias and Bioethics 20(11):7–17. https://​doi.​org/​10.​1080/​15265​161.​2020.​ Discrimination in Algorithmic Decision Making (Leibniz University 18194​69 of Hannover), the Ethics and Technology Workshop of the Ethics and Che Z, Purushotham S, Khemani R, Liu Y (2016) Interpretable deep Technology Graduate Group, and the 16th World Congress of Bioethics models for ICU outcome prediction. In: AMIA ... annual sympo- (University of Basel). We also thank the comments of Pedro Garrido sium Proceedings. AMIA symposium 2016, pp 371–380 that helped us to improve the writing of the manuscript. Chorás M, Pawlicki M, Puchalski D, Kozik R (2020) Machine learn- ing—the results are not the only thing that matters! What about Funding Funding for open access publishing: Universidad de Granada/ security, explainability and fairness? In: Krzhizhanovskaya VV CBUA. This research is funded by the project “Detección y eliminación et al (eds) Computational science—ICCS 2020. 20th interna- de sesgos en algoritmos de triaje y localización para la COVID-19” of tional conference Amsterdam, The Netherlands, June 3–5, 2020 13 AI & SOCIETY Proceedings, Part IV. Springer, Cham, pp 615–628. https://​doi.​ respect to clinical outcomes compared with the emergency sever- org/​10.​1007/​978-3-​030-​50423-6 ity index. Ann Emerg Med 71(5):565-574.e2. https://​doi.​org/​10.​ Coeckelbergh M (2020) AI ethics. MIT Press, Cambridge 1016/j.​annem​ergmed.​2017.​08.​005 Cutillo CM, Sharma KR, Foschini L, Kundu S, Mackintosh M, Mandl London AJ (2019) Artificial intelligence and black-box medical deci- KD (2020) Machine intelligence in healthcare-perspectives on sions: accuracy versus explainability. Hast Cent Rep 49(1):15–21. trustworthiness, explainability, usability, and transparency. NPJ https://​doi.​org/​10.​1002/​hast.​973 Dig Med 3:47. https://​doi.​org/​10.​1038/​s41746-​020-​0254-​2,p.1 Marmot M (2017) Social justice, epidemiology and health inequali- Daniels N, Sabin J (1998) The ethics of accountability in managed care ties. Eur J Epidemiol 32(7):537–546. https://​doi.​org/​10.​1007/​ reform. Health Aff (project Hope) 17(5):50–64. https://d​ oi.o​ rg/1​ 0.​ s10654-​017-​0286-3 1377/​hltha​ff.​17.5.​50 Martin K (2019) Ethical implications and accountability of algo- Dorado-Moreno M, Pérez-Ortiz M, Gutiérrez PA, Ciria R, Briceño J, rithms. J Bus Ethics 160(4):835–850. https://​doi.​org/​10.​1007/​ Hervás-Martínez C (2017) Dynamically weighted evolutionary s10551-​018-​3921-3 ordinal neural network for solving an imbalanced liver transplan- Mill JS (2014) Utilitarianism. Cambridge University Press, tation problem. Artif Intell Med 77(March):1–11. https://​doi.​org/​ Cambridge 10.​1016/j.​artmed.​2017.​02.​004 Mökander J, Floridi L (2021) Ethics-based auditing to develop trust- Durán JM (2021) Dissecting scientific explanation in AI (SXAI): a case worthy AI. Mind Mach 0123456789:2–6. https://d​ oi.o​ rg/1​ 0.1​ 007/​ for medicine and healthcare. Artif Intell 297:103498. https://​doi.​ s11023-​021-​09557-8 org/​10.​1016/j.​artint.​2021.​103498 Moseley D (2019) Bias. International encyclopedia of ethics. Major ref- Durán JM, Jongsma KR (2021) Who is afraid of black box algorithms? erence works. https://​doi.​org/​10.​1002/​97814​44367​072.​wbiee​861 On the epistemological and ethical basis of trust in medical AI. Moss E, Metcalf J (2020b) High tech, high risk: tech ethics lessons for J Med Ethics medethics-2020-106820. https://​doi.​org/​10.​1136/​ the COVID-19 pandemic response. Patterns (New York, N.Y.) medet​hics-​2020-​106820 1(7):100102. https://​doi.​org/​10.​1016/j.​patter.​2020.​100102 Ellahham S, Ellahham N, Simsekler MCE (2020) Application of arti- Nanayakkara S, Fogarty S, Tremeer M, Ross K, Richards B, Bergmeir ficial intelligence in the health care safety context: opportunities C, Xu S et al (2018) Characterising risk of in-hospital mortality and challenges. Am J Med Qual 35(4):341–348. https://d​ oi.​org/​ following cardiac arrest using machine learning: a retrospective 10.​1177/​10628​60619​878515 international registry study. PLoS Med 15(11):e1002709. https://​ Ershoff BD, Lee CK, Wray CL, Agopian VG, Urban G, Baldi P, Can- doi.​org/​10.​1371/​journ​al.​pmed.​10027​09 nesson M (2020) Training and validation of deep neural networks Nussbaum M (2007) Frontiers of justice: disability, nationality, spe- for the prediction of 90-day post-liver transplant mortality using cies membership. The Belknap Press of Harvard University Press, UNOS registry data. Transpl Proc 52(1):246–258. https://​doi.​org/​ Cambridge 10.​1016/j.​trans​proce​ed.​2019.​10.​019 Nussberger AM, Lan Luo L, Celis E, Crockett MJ (2022) Public atti- Felder RM (2021) Coming to terms with the black box problem: how tudes value interpretability but prioritize accuracy in artificial to justify AI systems in health care. Hast Cent Rep 51(4):38–45. intelligence. Nat Commun 13(1):5821. https://​doi.​org/​10.​1038/​ https://​doi.​org/​10.​1002/​hast.​1248 s41467-​022-​33417-3 Floridi L, Cowls J, Beltrametti M, Chatila R, Chazerand P, Dignum Pot M, Kieusseyan N, Prainsack B (2021) Not all biases are bad: V, Luetge C et al (2018) AI4People—an ethical framework for equitable and inequitable biases in machine learning and radi- a good AI society: opportunities, risks, principles, and recom- ology. Insights Imaging 12(1):10–11. https://​doi.​org/​10.​1186/​ mendations. Mind Mach 28(4):689–707. https://​doi.​org/​10.​1007/​ s13244-​020-​00955-7 s11023-​018-​9482-5 Rawls J (1971) A theory of justice. Harvard University Press, Floridi L et al (2022) AI4People, pp 699–700 Cambridge Freeman RB Jr (2007) The model for end-stage liver disease comes Robbins S (2019) A misdirected principle with a catch: explicabil- of age. Clin Liver Dis 11(2):249–263. https://​doi.​org/​10.​1016/j.​ ity for AI. Minds Mach 29(4):495–514. https://​doi.​org/​10.​1007/​ cld.​2007.​04.​005 s11023-​019-​09509-3 Grote T, Keeling G (2022) Enabling fairness in healthcare through Robertson S, Azizpour H, Smith K, Hartman J (2018) Digital image machine learning. Ethics Inf Technol 24(3):1–13. https://​doi.​org/​ analysis in breast pathology-from image processing techniques to 10.​1007/​s10676-​022-​09658-7 artificial intelligence. Transl Res 194(April):19–35. https://​doi.​ Gulum MA, Trombley CM, Kantardzic M (2021) A review of explain- org/​10.​1016/j.​trsl.​2017.​10.​010 able deep learning cancer detection models in medical imaging. Rueda J (2021) Ageism in the COVID-19 pandemic: age-based dis- Appl Sci (switzerland). https://​doi.​org/​10.​3390/​app11​104573 crimination in triage decisions and beyond. Hist Philos Life Sci Hedden B (2021b) On statistical criteria of algorithmic fairness. Philos 43(3):1–7. https://​doi.​org/​10.​1007/​s40656-​021-​00441-3 Public Aff 49(2):209–231. https://​doi.​org/​10.​1111/​papa.​12189 Savulescu J, Gyngell C, Kahane G (2021) Collective reflective equi- Holzinger A, Langs G, Denk H, Zatloukal K, Müller H (2019) Causa- librium in practice (CREP) and controversial novel technologies. bility and explainability of artificial intelligence in medicine. Bioethics January 2020:1–12. https://​doi.​org/​10.​1111/​bioe.​12869 Wiley Interdiscip Rev Data Min Knowl Discov 9(4):e1312. (first author (2022) [anonymized for blind review]) https://​doi.​org/​10.​1002/​widm.​1312. (p. 2) Sen A (1997) Maximization and the act of choice. Econometrica Kim D, You S, So S, Lee J, Yook S, Jang DP, Kim IY et al (2018) A 65:745–779 data-driven artificial intelligence model for remote triage in the Sen A (2009) The idea of justice. Harvard University Press, Cambridge prehospital environment. PLoS ONE 13(10):e0206006. https://​ Starke G, De Clercq E, Elger BS (2021) Towards a pragmatist deal- doi.​org/​10.​1371/​journ​al.​pone.​02060​06 ing with algorithmic bias in medical machine learning. Med LaLonde R, Torigian D, Bagci U (2020) Encoding visual attributes Health Care Philos 24(3):341–349. https://​d oi.​o rg/​1 0.​1 007/​ in capsules for explainable medical diagnoses. In: International s11019-​021-​10008-​5,p.2 conference on medical image computing and computer-assisted Theunissen M, Browning J (2022) Putting explainable AI in context: intervention. Springer, Cham, pp 294–304 institutional explanations for medical AI. Ethics Inf Technol Levin S, Toerper M, Hamrock E, Hinson JS, Barnes S, Gardner H, 24(2):1–10. https://​doi.​org/​10.​1007/​s10676-​022-​09649-8 Dugas A, Linton B, Kirsch T, Kelen G (2018) Machine-learning- based electronic triage more accurately differentiates patients with 13 AI & SOCIETY Topol EJ (2019) High-performance medicine: the convergence of algorithms: beyond the black box. BMJ (online) 364(March):10– human and artificial intelligence. Nat Med 25(1):44–56. https://​ 13. https://​doi.​org/​10.​1136/​bmj.​l886 doi.​org/​10.​1038/​s41591-​018-​0300-7 Williams A (1985) Economics of coronary artery bypass grafting. Br Tsamados A, Aggarwal N, Cowls J, Morley J, Roberts H, Taddeo M, Med J (Clin Res Ed). https://​doi.​org/​10.​1136/​bmj.​291.​6497.​736-a Floridi L (2020) The ethics of algorithms: key problems and solu- Wingfield LR, Ceresa C, Thorogood S, Fleuriot J, Knight S (2020) tions. SSRN Electron J. https://​doi.​org/​10.​2139/​ssrn.​36623​02 Using artificial intelligence for predicting survival of individual Vayena E, Blasimme A, Glenn Cohen I (2018) Machine learn- grafts in liver transplantation: a systematic review. Liver Transpl ing in medicine: addressing ethical challenges. PLoS Med 26(7):922–934. https://​doi.​org/​10.​1002/​lt.​25772 15(11):e1002689. https://​doi.​org/​10.​1371/​journ​al.​pmed.​10026​89 Yoon CH, Torrance R, Scheinerman N (2021) Machine learning in Veer SN van der, Riste L, Cheraghi-Sohi S, Phipps DL, Tully MP, medicine: should the pursuit of enhanced interpretability be aban- Bozentko K, Atwood S et al (2021) Trading off accuracy and doned? J Med Ethics medethics-2020-107102. https://​doi.​org/​10.​ explainability in AI decision-making: findings from 2 citizens’ 1136/​medet​hics-​2020-​107102 juries. J Am Med Inform Assoc 28(10):2128–2138. https://​doi.​ Zimmermann A, Lee-Stronach C (2022) Proceed with caution. Can J org/​10.​1093/​jamia/​ocab1​27 Philos 52(1):6–25. https://​doi.​org/​10.​1017/​can.​2021.​17 Watson DS, Krutzinna J, Bruce IN, Griffiths CE, McInnes IB, Barnes MR, Floridi L (2019a) Clinical applications of machine Publisher's Note Springer Nature remains neutral with regard to learning algorithms: beyond the black box. BMJ (clin Res Ed) jurisdictional claims in published maps and institutional affiliations. 364(March):l886. https://​doi.​org/​10.​1136/​bmj.​l886 Watson DS, Krutzinna J, Bruce IN, Griffiths CEM, McInnes IB, Barnes MR, Floridi L (2019b) Clinical applications of machine learning 13