AI & SOCIETY https://doi.org/10.1007/s00146-022-01389-z OPEN FORUM Justificatory explanations in machine learning: for increased transparency through documenting how key concepts drive and underpin design and engineering decisions David Casacuberta1 · Ariel Guersenzvaig2 · Cristian Moyano‑Fernández1,3 Received: 2 August 2021 / Accepted: 17 January 2022 © The Author(s) 2022 Abstract Given the pervasiveness of AI systems and their potential negative effects on people’s lives (especially among already mar- ginalised groups), it becomes imperative to comprehend what goes on when an AI system generates a result, and based on what reasons, it is achieved. There are consistent technical efforts for making systems more “explainable” by reducing their opaqueness and increasing their interpretability and explainability. In this paper, we explore an alternative non-technical approach towards explainability that complement existing ones. Leaving aside technical, statistical, or data-related issues, we focus on the very conceptual underpinnings of the design decisions made by developers and other stakeholders during the lifecycle of a machine learning project. For instance, the design and development of an app to track snoring to detect possible health risks presuppose some picture or another of “health”, which is a key notion that conceptually underpins the project. We take it as a premise that these key concepts are necessarily present during design and development, albeit perhaps tacitly. We argue that by providing “justificatory explanations” about how the team understands the relevant key concepts behind its design decisions, interested parties could gain valuable insights and make better sense of the workings and outcomes of systems. Using the concept of “health”, we illustrate how a particular understanding of it might influence decisions during the design and development stages of a machine learning project, and how making this explicit by incorporating it into ex-post explanations might increase the explanatory and justificatory power of these explanations. We posit that a greater conceptual awareness of the key concepts that underpin design and development decisions may be beneficial to any attempt to develop explainability methods. We recommend that “justificatory explanations” are provided as technical documentation. These are declarative statements that contain at its simplest: (1) a high-level account of the understanding of the relevant key concepts a team possess related to a project’s main domain, (2) how these understandings drive decision-making during the life-cycle stages, and (3) it gives reasons (which could be implicit in the account) that the person or persons doing the explanation consider to have plausible justificatory power for the decisions that were made during the project. Keywords Explainability in machine learning · Justifying reasons · Decision-making · Justificatory explanations · Health 1 Introduction As AI systems become all-pervasive, they increasingly have effects on the world, society, and individuals, espe- David Casacuberta and Ariel Guersenzvaig have contributed cially affecting already marginalised people and commu- equally to this work and wish to be acknowledged as co-first nities (Benjamin 2019). To mitigate harms and prevent authors. 1 * David Casacuberta Philosophy Department, Universitat Autònoma de david.casacuberta@uab.cat Barcelona, Barcelona, Spain 2 * Ariel Guersenzvaig ELISAVA Barcelona School of Design and Engineering aguersenzvaig@elisava.net UVIC-UCC​, Barcelona, Spain 3 Cristian Moyano‑Fernández Institute of Philosophy, CSIC, Madrid, Spain cristian.Moyano@uab.cat 13 Vol.:(0123456789) AI & SOCIETY risks as they intervene in key aspects of our lives from worth avoiding, which can be monitored thanks to the use education to law enforcement and from health to the jus- of an app. tice system, it becomes crucial to assess how the outcomes We take it as a premise that key concepts are thus—at of these systems are generated and based on what reasons. least tacitly—necessarily present during design and devel- Unlike earlier AI systems based, e.g., on decision trees, opment. Any person coming up with the idea of connecting modern systems, and especially those using neural net- snoring to health has some picture or another of how they are works, are said to operate as a “black box” (Holm 2019), related. Any health app that tracks snoring and connects it which makes it difficult to comprehend what the algorithm to health presupposes this picture. Concurrently, our analy- exactly did and why it generated the intermediate and final sis also assumes that developers, data scientists, and others outcomes it produced. There is a growing subfield within involved have a set of skills and knowledge primarily related AI and AI Ethics concerned with making AI systems, to coding languages, technical aspects, or mathematical and especially machine learning (ML), more “explainable” statistical procedures, but are not, as a collective, well versed by reducing their opaqueness and increasing their inter- in the philosophy of medicine, or theories of health and well- pretability and explainability. This is often referred to as being. (We could extend this to theories of learning, justice, explainable artificial intelligence (XAI) (Barredo Arrieta happiness, and many other strategic concepts with which et al. 2020). developers work.) Naturally, nothing prevents an individual In this paper, we take an alternative, non-technical yet programmer from acquiring this knowledge, but we assume complementary perspective towards explainability by laying that declarative knowledge about these key philosophical our focus on the very conceptual underpinnings of the design concepts is not a standard feature. decisions made by developers and other stakeholders during We will argue that a more explicit understanding of the the life-cycle of a machine learning project. We focus on relevant key concepts behind an AI system can be opera- the role particular understandings of key, overarching con- tionalized by the developers and others involved to provide cepts in a given domain play during the different stages of rich explanations of how they are understood and how they the machine learning process related to that domain. These guide design and development. These explanations might understandings, in turn, and this is the core of our argu- complement those provided by the system itself (which is ment, dramatically influence other aspects of a system such an XAI endeavour), as well as other technical system docu- as decisions about training datasets, metrics, model develop- mentation and analyses (relating to, e.g., datasets, metrics, or ment, etc. These subjects frequently appear in the literature code). We contend that a developing team that has insights and we do not engage with them directly; we take no issue in how they understand the relevant key concepts behind with discussions around a system’s actions or outputs nei- their decision would be capable of offering rich explanations ther with the math and code behind it nor with what would of their own design intentions related to the system. In turn, constitute fairness in a dataset. Rather, we concentrate on these explanations can enable interested parties (from team these key concepts that drive and influence decision-making. members to developers to scholars, but also from policymak- Let us tackle an example. Consider a team working on a ers to external auditors to professionals operating a system system (a smartphone app, for instance) that logs and tracks that is already deployed) to gain valuable insights and make a person’s snoring as well as their lifestyle and other infor- proper sense of the workings and outcomes of systems, pos- mation such as heartbeat rate or weight, so the person can sibly in connection with other data. Hayashi (2020) argues monitor their health and avoid complications. Many apps that “knowing where a [neural] network is looking within like these exist, and claim to use some sort of machine learn- the image does not tell the user what it is doing with that part ing (Klaus et al. 2021). of the image”. This paper makes the case that, by fostering We contend that any effort to design, develop, and deploy greater awareness of the normative key concepts behind a systems like these necessarily employs some conception or system, its designers could tell a lot, to use Hayashi’s phras- another of critical concepts related to the relevant domain as ing, about what the system is doing or is supposed to be a conceptual underpinning. In the case of the app for track- doing and what for, even if the exact details about what a ing snoring, a key concept would be “health”. Furthermore, system is actually doing remains partially unbeknownst to we posit that these key concepts have normative power in them. that they articulate and structure a myriad of design and In the next sections, we will first delve deeper into the engineering decisions all along the ML lifecycle. These topics of reasons and explanations (Sect. 2). Then, we will concepts also prescribe standards for the domain at hand, move to present different conceptual understandings of the either explicitly or implicitly. The very notion of “snoring” concept of “health” (Sect. 3). In the section thereafter, we being a “health problem” to be monitored to prevent greater will illustrate our case and show how the different under- risks such as heart attacks and strokes involves a normative standings of the concept of “health” might lead to different understanding of “snoring” as something undesirable and design and development decisions as well as to results that 13 AI & SOCIETY Fig. 1  Generic example of a justificatory explanation Given there is no consensus as to how the key concept X (e.g. health, justice, happiness, fun, etc) should be understood, we, as a development team, aim to clarify our understanding of X. We understand X as X1, and not as X2 nor Xn, which are alternative ways of understanding X that can be found in the literature about X. We are aware that choosing an alternative understanding of the key concept might have resulted in different designs for the system, and, in turn, possibly in different outcomes. Our specific understanding of X as X1 conditioned the life-cycle of our machine learning project in such and such manner. Given X, the problem definition looked like such and such. In line with our understanding of X1, such and such were our intentions and goals for the system. Our understanding of X conditioned what counted as relevant data in such and such way, and it affected the collection and preparation of data in such and such way. (The stages of model development and deployment would follow a similar structure.) might be altogether different depending on what notion of model, and act. When designers design artifacts, they poise health conceptually underpins the system (Sect. 4). In the them to behave in certain ways.” final sections, we discuss the issues that we have raised and This is not to say that this embedding of intentionality provide recommendations (Sect. 5). We end with a conclu- actually ensures that the system will behave in the intended sion (Sect. 6). manner nor that it will be used in the way planned by the designers. Science and technology (STS) scholarship con- sistently shows that the adoption and use of any technology 2 Reasons, explanations, and justifications is a dynamic process, which is neither fully determined by the designer nor by the artefacts’ properties, but by users, Specifically considering machine learning, explainability is and socially and historically situated practises (see e.g. Mac- roughly about making the outcomes of an AI system under- Kenzie and Wajcman 1999). standable by humans, so that it can be assessed. It seeks to Be that as it may, we will not discuss intentions and break with the “black box” model in which an outcome is values either. While it is a hugely relevant topic to discuss not accompanied by sufficient elements of analysis to explain accountability and what happens when things go South, the how a system arrived at it. Being able to produce explana- point to our discussion, is how goals, intentions, and even tions has also been linked to transparency and responsibility the general bird’s eye view of the functioning of a system in a moral sense (Coeckelbergh 2019: pp. 116–123). Refer- can be traced down to how key notions related to a par- ring to robotic systems, Wortham and Theodorou (2017) ticular problem domain conceptually underpin the design argue that AI systems should be “transparent” in the sense and development process. We posit that explicitly explain- of enabling humans (users or other stakeholders) a general ing how a team understands these notions can facilitate an understanding of the systems’ goals and functioning. As analysis of the system, the decision-making that led to it, as advanced above, we do not take issue with this discussion well as the results by providing “justificatory explanations”. around the computational techniques being developed in The term “explanation” alludes to two different but XAI to generate explanations, neither are we concerned with often complementary meanings. First, an explanation can the discussion on whether those explanations would count be understood as an account that clarifies or details some- as valid ones (see e.g. Coeckelbergh 2019: p. 122; Barredo thing; and second as providing reasons for an action. To Arrieta et al. 2020; Zednik 2021). integrate both meanings and avoid vagueness, we refer to Rather, we seek to render the black box less opaque by “justificatory explanations”1 to emphasise that the explana- arguing in favour of explaining the decisions made during tions a design and development team generates should not the design and development stages of these systems. The only be focused on providing a descriptive account but also first layer of analysis has to do with the intentionality, values, on providing reasons which could be considered to justify purposes, and goals embedded in them by their designers. an outcome. Figure 1 offers a generic example of what such To illustrate, when an app aims at detecting heavy snoring an explanation might look like. and recommending ways to prevent stroke or avoid marital Along this line, a justificatory explanation must provide disputes caused by being snappy due to poor sleeping pat- not only a plausible high-level account of what has hap- terns, it is the app’s creators who incorporate into the system pened during the process that led to the outcome, but first those intentions and values about what is worth tracking, predicting, and recommending. Deborah Johnson (2006: p. 201) writes that the “act of designing a computer sys- 1 Searle (2001: pp. 110–111) uses this same term, but our under- tem always requires intentionality—the ability to represent, standing of “justificatory reasons” is somewhat different and more lenient than his. 13 AI & SOCIETY and foremost, it must contain sufficient justifying reasons the explaining person, as it occurs with decisions that are for the actions or decisions. Justifying reasons are “reasons based on intuition as is often the case with designerly activi- which, very roughly, favour or justify an action, as judged ties (Cross 2007; Kolko 2011). Yet, that the justificatory by a well-informed, impartial observer” (Alvarez 2017). In explanation is generated ex-post or is not necessarily the real other words, a justificatory explanation is adequate when it reason behind an act, need not be a critical problem for our provides reasons according to which an impartial observer argument. Flyvbjerg (2001) writes on the issue of “made-up could accept an account as plausible and its outcome as rea- reasons” that “[s]uch justification need not be illegitimate sonably justified. rationalisation since it can be the ex-post test of whether To illustrate again, imagine we push someone out of the individual intuitive reasons are also generally valid and col- way at a pedestrian crossing. When asked for an explana- lectively acceptable.” tion of what we did and why, we can say that “we pushed There is another caveat, that a team can produce a justifi- the person because we had to”. However, while “pushing catory explanation does not mean that the decisions or acts the person” could count as a plausible account, the second, that are accounted for are actually “justified” (i.e., containing justificatory part could hardly count as an adequate justifi- sufficient justifying reasons, as judged by a well-informed, cation as it fails to provide persuasive reasons to plausibly impartial observer). The justificatory explanation only pro- justify the action. Conversely, if we explain that we felt com- vides some information that can enable an observer to come pelled to push someone out of the way to prevent the person to an evaluative judgement. The threshold we stipulate for from being hit by an out-of-control car, then our explanation justificatory explanations is not that they need to be accepted becomes an adequate explanation as it provides both a plau- as containing sufficient justifying reasons by this impartial sible account of what went on as well as well as justifying observer, which would be desirable, but too high a threshold. reasons for having acted the way we did. In short, a justifica- After reading justificatory explanations, an observer need tory explanation clarifies what went on and offers reasons not agree with the development team about the rightness for having acted in a particular way. of the understandings of the key notions nor about the pro- To summarise preliminarily, a justificatory explanation cedures nor about the quality or goodness of the decisions. is, for our purposes and at its simplest, a short statement The threshold to meet is that the observer can say: “I see the (i.e., a document) that contains: (1) a high-level account reasons that guided the team’s decision-making. I under- of the understanding of the relevant key concepts a team stand how the team conceptualised the key terms X Y Z, and possess related to a project, (2) how these understandings I see how these concepts further influenced decisions along drive decision-making during the life-cycle stages, and (3) the line and why they did what they did.” After this, armed it gives reasons (which could be implicit in the account) with this new knowledge, further evaluative assessments of that the person or persons doing the explanation consider the system and the system’s outcomes may be undertaken. to have plausible justificatory power for the decisions that To exemplify and flesh out this discussion, we will con- were made. The document serves to link the decisions made sider how the design and development stages of a machine during design and development and their outcomes to a con- learning project are influenced by particular understandings ceptual why that underpins the whole pursuit from the very of the concept of “health”. We will also discuss how making beginning (i.e., it explains the decisions that were made in this understanding explicit by incorporating it into ex-post light of how key concepts were understood). explanations might increase the explanatory and justifica- Therefore, when we talk about explainability, we are not tory power of these explanations by revealing something just expecting an explanation in terms of a factual account that might be hidden, not in the opacity of the black box but of what a system did (i.e., what parameter values were set by on the designers’ and developers’ minds, which provides the learning algorithm), but insights for instance on reasons another layer of analysis for rendering the black box less why a system assigned those values. The question remains opaque. whether XAI can provide such explanations (Coeckelbergh For that, let us go back to the team working on an app 2019: pp. 121, 122). Humans, on the other hand, can and that logs and tracks a person’s snoring to prevent the health do constantly provide justificatory explanations for their complications associated with it. We can safely assume that behaviour. the designers believe that through tracking snoring people However, any justificatory explanation may be rejected by can become aware of potential risks and undertake actions others on the grounds of failing to provide sufficient justify- to become more “healthy”. This belief is grounded on the ing reasons (“I pushed the person out of the way because knowledge that snoring can be more than just a nuisance and the devil made me do it”). There is a more critical caveat: an is related to serious health risks such as high blood pressure, actual act may be motivated by reasons other than those put heart conditions, and stroke (Yunus et al. 2018). forward by the person doing the explanation. Furthermore, However, what does “healthy” mean in this context? these reasons might even be fully or partially unknown to Saying that something is “healthy” or “unhealthy” might 13 AI & SOCIETY count as making an objective description of a state. Yet, a risk factor, and that based on the symptoms, it would be at the same time, it can also count as making an inherently advisable for them to see a doctor for further evaluation. We normative evaluative assertion. Along this line, “healthy” see how these overarching concepts become normative and is an example of a so-called thick concept, i.e., concepts guide action. that are both descriptive and evaluative. Thick concepts are Along these lines, the attempt to solve any problem in opposed to thin concepts, which are either evaluative or nor- machine learning comes along with some notion of optimal mative. Given that thick concepts are both “action-guiding” performance in relation to a goal or solution, according to and “guided by the world” (Williams 2006: pp. 140, 141), which the machine learning model is evaluated. What counts to say that something is “healthy” serves to guide action as optimal performance is thus inextricably connected to the according to socially embedded guidelines concerning what way key concepts are understood. Consequently, the perfor- is to be preferred and what is to be avoided, as well as what mance of, say, a convolutional neural network that is applied is desirable or undesirable. This is why, understanding what to the analysis of sounds of snoring and sleep apneas is “healthy” means for a team is so important. If snoring is assessed, for instance, in relation to the quality of the (pre- unhealthy, snoring must be avoided. dictive) detection of high blood pressure, heart conditions, Seen from this perspective, tracking snoring makes per- stroke, daytime sleepiness, or motor vehicle accidents due to fect sense. Since loud snoring may be seen as indicative of lack of sleep (assuming detecting these medical issues was a health condition, tracking snoring, and sleep apneas, i.e., the goal that was stipulated). the periods of silence when breathing stops or nearly stops, All of this is fraught with ethical issues that are related to serves as a proxy for detecting many mild or severe health the particular conception of health a team (medical experts, issues. Comparing a person’s snoring patterns to the average engineers, developers, data scientists, and so on) adopts, snoring found in healthy people might make perfect sense from the stage of data preparation to system deployment, for detecting unhealthy patterns. We see here how consider- either tacitly or explicitly. We contend that being aware of ing loud snoring as a health issue is a defining feature dur- the particular understanding of the relevant key concepts ing the problem-definition stage. Any claim about “loud” that guide the design and development of an ML system snoring or sleep apneas that are “too frequent” necessitates and being able to produce declarative statements about it a measure of what type of snoring is within the normal (i.e., may greatly benefit the explainability and interpretability healthy) range. (By the way, more than five sleep apneas per of a system by offering a perspective about the conceptual hour of sleep is usually taken to be indicative of risk, while grounding of the system. below that number it is perfectly fine.) The thrust of medical deliberations is about when to attribute a particular (thick) concept to a biological state. Appealing to a particular conception of health, a medical 3 A case in point: conceptions of health professional is able to offer justifying reasons for calling a state or condition “healthy” or “diseased”. It is clear that AI We advanced above that to operate a team working on an systems do not have a sense of purpose and value integration AI system related to health, the team members necessarily in the way most people do (as when we push someone out must have at least some picture or another of the concept of of the way to prevent them from being hit by a car). Yet, in “health”. But what is “health”? the architecture of any AI system, the datasets, the neces- At first sight, “health” appears to be, at least for those of sary mathematical modelling, the optimization algorithms, us in the Western world, an unambiguous notion having to and other features that conform the system are put in place do with not being “sick” or “ill”, and perhaps with doing for something, i.e., to perform some tasks and achieve some and feeling well. For the purposes of this paper, we call this goals. At its most general, this goal might be to reduce messy informal approximation to the concept an “everyday under- problems (such as determining a person’s health status) to standing”, which is the primary layer to structure ideas and mathematical ones that can be solved using software and evaluations. In the case of health, an everyday understanding numerical methods (making health-related predictions based articulates ideas about how the body functions and should on the normality of a person’s snoring patterns together with function, and what functionings are especially relevant and other data). The makers of the snoring tracking app might worthy of further consideration. Furthermore, the way we not go as far as to claim that a person is “diseased” based on understand health is interrelated to how we conceptualise their snoring, but if it detects risk indicators, the app might the human body (for example, as a deterministic machine present the user with information that makes use of many that breaks down with disease or as an organism that is part thick notions related to health. For example, it will inform of a dynamic system involving the physical but also the them that the snoring loudness is “above average” (which social). These everyday understandings of health and other is certainly not good!), that 10 sleep apneas per hour are concepts have theoretical underpinnings arising from the 13 AI & SOCIETY wider scientific and non-scientific theories, cultural tradi- this as a reference class, as there is much statistical variation tions, and folk knowledge (Trollope-Kumar and Last 2002: regarding testosterone levels among males and females and pp. 685, 686).2 across the age range. Since species design seems to be con- Without aiming to exhaust the topic, in this section, we tingent on sex, age, and, in some cases, race, the statistical will review three influential positions about health that in abstractions should be made from reference classes smaller one way or another might underpin our everyday under- than species (Boorse 1977: p. 558), e.g., “a 35 years old standing of health. The discussion will allow us to observe white woman”. that “health” is far from an unambiguous and self-evident For Boorse (1977: p. 554), “the normal is the natural”, notion. We will consider one “naturalist” theory of health in an organism is healthy when its functioning conforms to its Sect. 3.1, whereby “health” is a notion that is free of norma- natural design. What is normal functioning, then? Again, tive values and determined by empirical facts. In Sect. 3.2, normality must be understood in a statistical sense and func- we will also consider two “normativist” theories, whereby tion in a biological sense. “A normal function of a part or the notion of “health” necessarily reflects value judgments. process within members of the reference class is a statisti- cally typical contribution by it to their individual survival 3.1 A naturalistic theory of health: Boorse’s and reproduction” (Boorse 1977: p. 555). Typical contribu- “biostatistical” theory tions are those “within or above some chosen central region of their population distribution” (Boorse 1977: p. 559). This Possibly, the most vigorously debated naturalist theory of entails that abnormal functioning is not a sufficient condi- health is the “biostatistical theory” of health proposed by tion by itself to be regarded as a disease. To count as such, Christopher Boorse (1977, 2014). The theory aims to pro- the functioning has to be subnormal, i.e., “below” average vide a completely objective account of health and disease. functioning and thus detrimental in some way to the highest In Boorse’s empirical view, health and disease are nothing level goals of survival and reproduction. else than biological states. To the naturalists, to say some- According to biostatistical theory, health “in a member thing is healthy is merely to give a value-neutral descrip- of the reference class is normal functional ability: the readi- tion of an empirical fact. In this way, the naturalist account ness of each internal part to perform all its normal func- aims to give a value-free, objective definition of what health tions on typical occasions with at least typical efficiency” and disease are. In the words of Boorse (1977: p. 543): “if (Boorse 2014: p. 684). The theory concedes that there are diseases are deviations from the species biological design, dysfunctional states that are nonetheless statistically normal, their recognition is a matter of natural science, not evalua- such as tooth decay. However, since these conditions are tive decision”. due to environmental agents and not are in the design of the For Boorse (1977: p. 542), health is “normal functioning” species, they do not contradict the important role statistical and diseases (or “pathological conditions”) are “internal normality plays in the theory; they cannot be considered states that depress a functional ability below species-typical functional designs that are shown to be empirically typical levels” relative to sex and age. (Boorse 1977: p. 542, 2014: (Boorse 1977: pp. 555, 556, 2014: p. 705). p. 684). Health is to be understood as the “total absence of pathological conditions” (pp. 683–684). 3.1.1 The problem with reference classes In this theory, typical levels for a species are those close to the statistical mean (Boorse 1977: pp. 558, 559). To It is clear that to assess the normality of a biological state, determine whether an organism is healthy in relation to the we need some sort of benchmark of normality, a reference species-typical level Boorse introduces the notion of “refer- against which things are compared. “Normal” snoring can ence class”, “a natural class of organisms of uniform func- be determined statistically, but how can we determine levels tional design; specifically, an age group of a sex of a species” above which snoring becomes “unhealthy” or risky? While (Boorse 1977: p. 555). For example, according to this view, Boorse convincingly shows that comparisons at a species to determine if a person has normal levels of testosterone, level can be inoperative and that a smaller reference class we must first determine this person’s age and sex and use is needed. It is not clear why it is appropriate to factor sex, age, and race instead of other criteria. Can we possibly find other suitable classes? Building on Kingma (2010), it could 2 We make no claims about the truth value of “everyday” under- be argued that there are no empirical facts that determine standings. Our goal is simply to emphasise its often tacit and intui- that “young adults” is an appropriate reference class, but tive nature, which may or may not be correctly aligned with scholarly “people with obesity” (who are more prone to snoring) or theories and scientific evidence. Because we take no issue with this “people with tooth decay” are not. discussion, we prefer not to use the terms “lay theories” (or “folk the- ories”), which are often contrasted with scientific theories and have As Moore (1993) famously insisted, we cannot base a connotations we prefer to avoid. normative assertion on a natural property, and as the case of 13 AI & SOCIETY tooth decay being statistically normal indicates, more seems In contrast to Boorse’s position, normativists argue that to be required for something to be “healthy” than to say that medical diagnosis does not consist of descriptions of empiri- it is “normal” in a statistical sense. Furthermore, if Boorse’s cal data, but that such descriptions always contain subjec- account is to be value-free, what constitutes an appropri- tive, cultural, or ethical evaluations. According to the nor- ate reference class and what does not should be justified mativists, quantitative or statistical data are not sufficient to without normative judgments. However, we would reject determine the health status of a person, but rather a holistic “smokers over 65 with obesity” or “young adult males with assessment of the general state of the person is required tooth decay” as suitable reference classes, because they con- (Nordenfelt 2007: p. 6). Lennart Nordenfelt, one of the main flict with our most basic intuitions about being healthy. We representatives of normativism, argues that health is not do not see them as appropriate reference classes, because identified with the absence of illness, but refers to the abil- smoking, tooth decay, and obesity are clearly something to ity to achieve (non-arbitrary) vital goals (Nordenfelt et al. be avoided from a medical perspective. However, this rejec- 2001; Nordenfelt 2007). This conception of health implies tion is a normative choice, it is not value-free, it reflects accepting the intertwining of values conditioned by a sub- our cultural, political, social, aesthetical, and even, perhaps, jective agent, its goals, and the circumstances in which the religious values. agent acts. Similarly, an illness is not an anatomical defect The biostatistical theory does have many virtues, “once of the “normal body”, as in Boorse’s theory, but a situation the reference classes are fixed [it] gives an accurate and perceived as problematic. The subjective recognition of a value-free analysis of health and disease” (Kingma 2010: p. situation or state as problematic would thus constitute the 132). However, it necessitates auxiliary normative elements fundamental prerequisite for seeking medical help. (for instance, for the fixation of the necessary reference classes), which strongly underscore the need to declare them 3.2.1 Health according to the World Health Organization if one wants to make a system explainable, as these values (WHO) that drive analysis cannot be inferred from the dataset. The definition proposed by the WHO has been character- ised as “idealistic” and to “give no help” (Bircher 2005). By 3.2 Two normativist characterizations of health defining health as “a state of complete physical, mental and social well-being and not merely the absence of disease or Normativists, contrary to naturalists, argue that defining infirmity” (WHO 1948), health seems to become an unrea- health inevitably involves all sorts of norms and values. sonably high standard. Such conception of health is incom- Therefore, to say that something is healthy is to irremediably plete and ambiguous, and has been criticised as postponing give an evaluation. While normativism does not necessarily the difficult work of understanding what it is. Understand- imply a full-blown socially constructed view of all biological ing health as a maximum state of well-being that cannot be processes or states, normativists generally argue that human reduced to biological aspects, the WHO’s approach is also values, interests, and beliefs are determinants in the identifi- far from Boorse’s naturalism. However, its ambitious defini- cation of the relevant processes. This value-ladenness is thus tion hardly helps to guide medical applicability. inextricably linked to the notion of health. Analogously, the Furthermore, the term “state” used by the WHO defini- related notion of “disease” is not objective in the same way tion of health refers to an absolute, static, and decontextual- the mineral composition of a rock can be said to be objec- ized moment, which does not take into account the changing tive—there is no “natural, objectively definable set of human and variable conditions of people. Presenting health in such malfunctions that cause disease” (Murphy 2021). an ambitious way makes it difficult for it to be durable over The general assumption of the normativist approach is time and compatible with minor discomfort. Such a defi- that any conception of health, illness, and medical needs nition obscures how to prioritise health care or treatment, should not be reduced to the biological factors. On the con- because the number of unhealthy people can be constant trary, it is necessary to adhere to a broader, holistic, and and endless. For this reason, other normativist approaches relational concept of health. Phenomena such as chronic have chosen to understand the “health-illness process” as a stress, a product of the enormous changes in the environ- dynamic experience involving both dimensions, shaped by ment in which we live, show the inefficiency of a biological personal and group history. Thus, instead of being presented medical model focused on bodily symptoms, which neglects antagonistically to illness, health would reflect a continu- to address the links of diseases with the environment and ous process of changes between complex moments in life, psychosocial and productive factors (Marmot and Wilkinson during which favourable and unfavourable conditions may 1999; Wilkinson and Pickett 2006). Thereby, the emergence coexist. In line with this dynamic and holistic definition of perspectives based on consideration of social determi- of health, the capabilities approach provides a meaningful nants of health has become widespread in the last years. contribution. 13 AI & SOCIETY 3.2.2 A capabilitarian approach to health divergence will lead to different decision-making regarding the promotion of health than a developer who holds a bio- The theory of health advocated from the capabilities statistical, naturalist conception of health. A team working approach is not value-free and anchored in biology, neither from a normativist approach might focus more on working is it one that is wholly centred on the concept of disease. with individual and collective goals, and subjective states Instead, the conception of health is understood as the per- than purely on parsing data from medical statistics. Yet, son’s true, actual ability to achieve or exercise a cluster of this divergence cannot easily be inferred from a dataset or basic human activities (“functionings”). These activities are a model. in turn specified through reasoning about what constitutes a minimal conception of a life with human dignity in the modern world (Venkatapuram 2011: pp. 42, 43). 4 The role of our conception of health The capabilitarian perspective to health may be reminis- when building algorithms cent of Nordenfelt’s postulates, which defines health as the ability to achieve vital goals. Both are committed to focusing In this section, we explore how fairness, explainability, and on the abilities and not the actual achievements or function- other ethical concerns may shift depending on the implicit ings, but Nordenfelt’s arguments have a significant draw- or explicit concept of health guiding our model. The defini- back: a lack of substantive content to describe what these tion of health that the system developers and designers are putative vital goals might look like. However, a concept of going to use may have an impact in different parts of the health should be much more explicit about these vital goals lifecycle of a machine learning algorithm. Because we take to help us determine an adequate environment to flourish and a non-technical approach to this paper, we propose a simpli- be healthy. A capabilitarian approach to health can be taken fied view of the different stages conforming the lifecycle: to overcome the challenges encountered by Nordenfelt’s (1) problem definition, (2) data collection and preparation, theory in regards to the lack of substantive content. Martha (3) model development, and (4) model deployment (see e.g. Nussbaum has provided a list of central capabilities, which, Ashmore, et al. 2021; Google 2017). We will explore some though hotly contested by other capabilitarian scholars, enu- of the effects that different conceptions of key notions might merates the minimal capabilities (e.g., life, bodily integrity, have during these stages and how every stage may be con- affiliation, etc.) that are indispensable for a life worthy of ditioned by these alternative understandings. As a starting human dignity (Nussbaum 2006: pp. 76, 77). point, Fig. 2 illustrates the justificatory explanations around A meaningful advantage of the capabilities approach is the concept of health that could be produced by a team. In the idea of a sufficient threshold (Nussbaum 2006). Unlike this case, their understanding is grounded on a naturalist some other normativist perspectives as Nordenfelt’s, the perspective. inclusion of a minimum level of capabilities or vital goals helps to rethink the individual bonds between the pluralistic and subjective experience of health and a common basis for 4.1 Problem definition stage being healthy. As Thomas Schramme (2007) has noted, the definition of vital goals proposed from normativism may When defining which variables will be relevant for our be too broad. It is not completely reasonable that a person’s model, the concept of health that guides our reasoning may subjective preferences can determine their health status. have implications. For example, from a perspective of pri- Schramme (2007: p. 14) uses the example of an ambitious vacy, a naturalistic approach is more suitable to keep sensi- athlete to illustrate this objection. For this athlete, it could tive data safe. A machine learning model that is based on be a great disgrace not to have professional success, but it a Boorsean approach may need less sensitive information would seem odd to argue that they have become unhealthy. of users than a capabilitarian one, which implies collect- Although they may feel unhappy, there is still a gap until ing information about lifestyles, emotional well-being, vital they can be diagnosed as unhealthy. Certainly, there should goals, and life plans which are highly sensitive. Therefore, a be a spectrum or a sliding scale from complete health to model that uses a naturalistic understanding of what health is maximal illness, but here the use of thresholds becomes becomes more privacy-friendly (Cavoukian 2009). crucial. A list of basic capabilities whether participatively However, when considering explanations, normativity defined by a community or established by a scholar (such as should be added to the equation. As we stated above, an Nussbaum’s list) constrains the scope and adds a degree of adequate explanation needs to justify the outcome. Explana- intersubjectivity by adding sufficiency levels (i.e., a minimal tions provide us with reasons for having acted or acting in threshold). a certain way, offering us more than just some correlations The upshot of all this is that if a developer or designer between numbers. Causes and conditions do play a role in has a normativist conception of health (however, tacit), this the explanation, but the reason for a certain course of action 13 AI & SOCIETY Fig. 2  Example of a justifica- tory explanation for an app to The development team understands "health" in a biostatistical sense. We are aware that health can be detect and evaluate snoring conceptualised differently but for our predictive purposes, an approach based on statistical and mathematical analysis of medical factors is more adequate for generating a prediction than one that is based on social determinants of health such as education or disposable income. The global goal of our app is to offer a numerical value representative of sleep quality (a score) taking as a benchmark the normal (i.e., healthy) sleeping noise and snoring patterns of a similar individual within the same reference class (considering sex, age, and ethnicity). Besides a global score, the system offers a probability value, indicating the likelihood of increased risk for having medical conditions such as hypertension, diabetes, coronary heart disease, stroke, chronic obstructive pulmonary disease, etc. The predictions are the result of the analysis of sleep noise and snoring patterns detected during sleep in combination with a machine learning model (a logistic regression model, or a random forest model, among many others) trained with sociodemographic, behavioural, and lifestyle quantitative variables coming from publicly available medical datasets such as the Global Health Data Exchange (GHDx) from the University of Washington and others. is always needed if we are looking for an explanation (Von public spaces by, for instance, focusing on the prevention Wright 1981). and reduction of crime through real-time video analysis. A Open approaches like the one developed by the WHO justificatory explanation would require the team to clarify turn unnecessarily difficult to present a coherent explanation these concepts (safety, security, crime, prevention, etc.). It of what the algorithm ultimately does. However, because would also mandate to explain how the captured video data it is the official characterisation by the reference world will conceptually map to the stipulated goals and to the key organisation about health, it seems likely that developers terms (for example by saying that the task will be to generate and researchers would quote this definition, only to find out a prediction of criminal intention through analysing location, that it is useless on its own for developing a machine learn- gait, and facial data of individuals). When the justificatory ing model. We imagine that researchers faced with these explanation is reviewed by a third party, questions could pos- difficulties would decide to use data proxies instead that may sibly arise about what kind of model of criminal behaviour capture in theory what health is according to the WHO. connects gait to criminal intentions. These questions might It is especially at this stage (in relation to the deployment lead to further probing the concepts and the very soundness stage) where we believe that our proposal can be particu- of the theories (or lack thereof) behind the project. This, larly beneficial as it aims to shed light on the fundamental in turn, might serve to gain insights on potential risks and assumptions that are made by a team. For instance, as we harms and to challenge or reject the project as a whole. will see below, these assumptions can drive the use of sim- plistic proxies to represent complex realities in other project 4.2 Data collection and preparation stage stages, which can lead to unacceptable outcomes. Moreo- ver, conceptual clarity during this first stage affords a broad We should stop considering data collection a safe, objective perspective to the prevention of risks and harms that goes procedure. As shown by Gebru et al. (2021), a vast array of beyond technical fairness metrics that could be applied at scholars’ data plays a critical role when developing an ML later stages, such as comparing and evaluating error rates per application. A model needs to match training and evalua- group to detect inadmissibly discriminatory outcomes (see, tion datasets with the reality it tries to model. Inaccuracies e.g., the case of COMPAS famously analysed by Angwin and misrepresentations or underrepresentation of segments et al. 2016). of the population may lead to mistakes, biases, and unjust By exploring key concepts and their entanglements with impacts, i.e., discriminations. design decisions through justificatory explanations all pro- The conception of health that underpins an ML project ject stakeholders (including civil society, administrators, will have relevant effects on the data collection and prepara- and policy makers) can reflect on and evaluate the very tion process. As explained in Sect. 4.1 , choosing a Boorsean framing of the problem to detect problematic aspects that approach may imply the need for fewer variables in the prob- might otherwise fail to be captured by analysing the data- lem definition section. Similarly, the Boorsean approach can sets or the code from a mathematical or statistical perspec- be developed by accessing fewer data sources than a capa- tive. An explanatory justification (or the lack thereof!) may, bilitarian one, which could be assumed to be more difficult for instance, serve to examine the scientific validity of the to obtain. As a matter of fact, an algorithm created using a assumptions that underpin a system. naturalistic view of health needs just biological data, which To illustrate, imagine a team working on a video surveil- are generally included in the patient’s medical records and lance system that aims to “ensure safety and security” in medical databases. Therefore, one or a few sources should 13 AI & SOCIETY normally suffice. Instead, if we are considering a capabili- 4.3 Model development stage tarian approach, many other data sources will have to be included, possibly something coming from digital social net- As we have argued in Sect. 3.1, a naturalistic approach works in which users share lifestyles and feelings or other cannot capture all the nuances we associate with our con- forms of user-generated data. As stated in seminal studies ception of health. Calling someone healthy is not value- like Sweeney (2000) and more recent ones like Cecaj et al. free, and it is something more than just being free of dis- (2016), or Yin et al. (2015) in supposedly anonymized data- ease. In an actual ML development, we possibly need to bases, individuals can be re-identified when crossing refer- include some normativist elements in our definition of ences between different databases. health. But that is easier said than done. Despite trying When considering health disparities and biases, the most to present a more detailed and specific definition of the common relevant variables are race, sex, gender, age, and different dimensions of a healthy human being, the capa- socio-economic status (Underwood et al. 2012; Kawachi bilitarian approach is not fully articulated yet to serve as a et al. 2005; D'Ignazio and Klein 2020; Criado Perez 2019). proper normative framework for health. As we have men- The first variables are quite manageable from a data science tioned in Sect. 3.2.2, Nussbaum (2006: pp. 76, 77) gives us point of view, but socio-economic status is more difficult to a list of central capabilities that minimally ensure human calculate and obtain. A common proxy, i.e., a value that can flourishing. Some of those characteristics, like “length be seen as representative of another value, for socio-eco- of lifespan”, can be easily converted into numbers and nomic status is insurance type and has been used in several included in a machine learning algorithm, but some oth- studies such as Seyyed-Kalantari et al. (2020) or Chen et al. ers like “possess practical reason to form a conception (2019). Geographic data (Soobader et al. 2001), property of the good” or “be able to play” are difficult to define in value (Ware 2019), or even online activity in social networks non-ambiguous terms, and once defined, it is not easy to (Levy Abitbol et al. 2019) have been used as proxies, as turn them into a set of numbers that make sense nor to find well. In many countries, those types of data can be retrieved representative data sources or adequate proxies. with ease from public databases and turned into numbers Nussbaum’s presentation of health is good enough for for the model. However, neither insurance type, location, philosophical and ethical purposes, but is not specific property, nor social networking activity are equivalent to enough to give a coherent background that can be trans- socio-economic status, and socio-economic status does not lated into a self-explanatory machine learning model. cover by itself all the nuances of a normativist approach to Therefore, a new type of effort is needed, interdisciplinary health. Moreover, depending on the proxy we use to capture research in which ethicists, philosophers, and data scien- “socio-economic status”, we will consider some minority tists work together to generate normativist understandings subgroups in our metrics but not others. Using social net- of health with much more detail than current ones. In the working data—a trending fad right now—will not take into case of the capabilities approach, for instance, this would consideration people within the digital divide that does not be about clearly defining the relationship between a capa- have access to digital media. Other proxies like property bility and a legitimate data proxy. value or geographic data will not be able to capture all the No matter how well we define our capabilities, some richness included in Nussbaum’s list, for example, not con- biological data will need to be included, as well. And not sidering people that may have basic needs covered, but do merely because of the obvious biomedical reasons. Those not fare well in a mental well-being score, nor the relative data have to be included if we want a system that is both value of material resources (a person might own a bike, accurate and just. As we mentioned before, the main biases which might be worthless without cycling paths around). found in algorithms for health are related to age, sex, From a naturalist perspective, weight and age might be gender, race, and socio-economic status. When a group relevant and decisive factors in the snoring app to keep track becomes a majority, underrepresented subgroups may of and process, as a person that is overweight or has obesity become vulnerable to being mistreated when deploying is statistically more prone to snoring. Conversely, from a the model. Such vulnerability is not easily solved just by normativist perspective, the issue of where the person was increasing the number of patients that are members of such born, lives, and works might be more relevant. According to subgroups. In Seyyed-Kalantari et al. (2020), large pub- the “Social Determinants of Health” perspective, it is a tru- lic datasets currently used to generate machine learning ism to say that having poor access to healthy dietary options, algorithms to diagnose chest X-rays are analysed. Authors good housing, or opportunities to exercise can also be linked observe that female patients were the least favoured sub- to heart disease or stroke. Also from this perspective, if one group, even though the proportion of female patients was wants to contribute to improving a person’s health, it might only slightly less than male patients in the different data- make much more sense to track these issues than to be con- sets analysed (Seyyed-Kalantari et al. 2020: p. 239). cerned with snoring and noise produced during sleep. 13 AI & SOCIETY 4.4 Model deployment stage Instead, if using a capabilitarian approach, a model based on quantitative predictions will not be helpful, and we would When and why will our ML model be used? And who are the need a more participatory approach instead, in which citi- subjects to whom the model will be helpful? To decide when zens can democratically decide which health issues are more it is helpful to deploy the model and use it on the real popu- relevant and which sections of the population are more in lation, we need medical and health criteria to do so. That need of study, analysis and deployment, possibly in relation means that our conception of what health is will play a role to capabilities that are worth promoting. It is a truism to when deciding whether we need to use a specific machine state that the postal code is at the group level a better pre- learning model with a given population or not. dictor of health and well-being than many medical variables Deploying a model targeted to a specific population that combined. we think have special issues may become a self-fulfilling In this stage, ML models can and are constantly opti- prophecy. Depending on the results of the algorithm, a spe- mised. Further model iteration, interpretation, and evalu- cific medical procedure may be used in such a population. ation of results are critical. For example, if not developed Then, the data will be used to improve the algorithm in with inclusivity in mind, ML could exacerbate health care another lifecycle, producing a feedback loop that builds and disparities in dermatological care (Adamson and Smith amplifies those specific medical needs of such a segment of 2018). A purely naturalist approach might overlook this need the population. by laying the focus on statistically normality and by assum- The phenomenon we are postulating here is akin to the ing that health is an objective matter. well-known self-fulfilling prophecy of turning neighbour- hoods less safe by monitoring them with video cameras. Let us suppose that the city police considers that one neighbour- 5 Discussion hood does have more criminal activities than others. There- fore, they deploy surveillance cameras there to detect crimi- When we develop an algorithm to track and predict health nal activities. Because there are more cameras, it is easier to issues in humans, disparities will inevitably show up. To spot criminal activities, so more crime is detected and there mitigate potential harms, the first step we need to consider is are more arrests in this neighbourhood than in others. Data where such disparities come from, and whether they should on arrests and convictions are included in future models, so also be considered discriminatory and conducive to unjust there is more “evidence” that the surveilled neighbourhood impacts. is less safe. Naturally, there may be other neighbourhoods To do so, we need to consider all the phases in a machine as dangerous as this one, or even more, but because they are learning model development, but especially the first ones, less monitored, such data are below the radar. The decision problem definition and the data collection and preparation to deploy cameras is never value-free, but the values that stages. Here, problems are framed in a particular way and drive the deployment may be perversely hidden in the data people and their relative health are reduced to data put into that is generated and further used (see e.g. Alexander 2019). a set of rows and columns on a CSV file. We posit that prob- Likewise, if a model is deployed with a Boorsean per- lems are “framed” often in subjective and dynamic ways, spective of health in mind, it will give a false impression of which depend on particular conceptions of the concepts more precision as it gathers more data (e.g., an app for track- associated with a domain, as well as on the pursued out- ing snoring aimed at people with obesity, will likely work in comes, power dynamics, and many other historical contin- predicting potential strokes or diabetes). However as it limits gencies. We should infuse meaning beyond the numerical the relevance of social indicators of health, it may leave out values that make up the datasets, so we know what those sections of the population with health issues that can be numbers stand for and what are the aims behind them. We understood through a capabilitarian approach to health but will not be able to characterise such aims unless we under- are not well represented in a purely statistical approach like stand our ultimate goal: in the case of the app for tracking the Boorsean (if we focus primarily on evaluating snoring snoring, it was the promotion of health. But still, as we have we pay less attention to other social factors that might be seen, health is an elusive notion that can be conceptual- much more relevant such as education or the availability of ised in multiple ways. Therefore, we need to have a proper a health insurance plan). Moreover, under a naturalist con- understanding of what actual concept of health has led to ception of health, a “healthy” baby may be one within sta- a given system. For that, we need to clarify what we actu- tistical normality, whereas a baby with Spina bifida, Down ally talk about when we talk about a system that contrib- syndrome, or any disability for that matter, may be counted utes to improving health or assists users with health-related as “unhealthy” due to being statistically rare. However, these decisions. classifications are dramatically fraught and linked to the bal- At the same time, we cannot just import a list of basic ances of power and tensions of the present and the past. human capabilities as those defined by Nussbaum (2006: 13 AI & SOCIETY pp. 76, 77) into a system and expect our algorithm to gen- intelligence mainly through generalist media and social net- eralise and predict anything useful from that. Proxies might works, and tend to believe that machine learning algorithms be unavoidable. Yet, it is crucial that selecting a proxy is do really “understand” faces, diseases or human behaviour guided by the developers’ mind set. In other words, to gen- in a way akin to how a person does. erate explanations that are both accurate and relevant for When developing a machine learning algorithm for health humans, we need to understand clearly how each element purposes, we will measure some target variable that is going in our understanding of health (or any other relevant key to be a construct. We should see the predictions of the model concept) relates to specific proxies, and what are the limita- as an indicator of some complex combination of properties tions of such links. Even non-medical data might turn out to in the individual and not as real property in itself. When be relevant for health-related endeavours, depending on our that app that tracks snoring tells us that someone’s sleep understanding of the issue. Data about a person’s engage- score last night was, say, 75 due to heavy snoring, neither ment in community activities or their early childhood edu- the development team nor the person snoring nor the medi- cation might be a better indicator of health risks than their cal staff should take such a number at face value. The score heartbeat rate or the noises produced during sleeping. But value calls for a hermeneutical effort, as it needs to be inter- again, to come to this realisation, a developer needs to be preted to discover the meaning behind the number and how guided by a particular conception of health. it links to a general understanding of what health is. We should also take into consideration cultural differ- A simile may be useful here. Consider crime risk assess- ences. As we stated in Sect. 3.2.2, capabilitarian models are ments generated by machine learning algorithms. Govern- open to cultural differences. That is good for a general, phil- ments, the press, and software vendors and manufacturers osophical model, but the explanatory justifications for our present such models as predicting how crime will rise or models should take into account the rapidly changing nature drop depending on the neighbourhood. However, that is not of cultural identification, and update the model’s premises really what the algorithm does, as the database does not to the specific cultural background we want to reflect. We include the “number of crimes detected” but the “number of now label it as neurodiversity or as mere sexual orientation crimes previously solved by the police and judged in a trial”. what a couple of decades ago was commonly considered a And those numbers may be very different. If the police tend disease (or even a crime). Today, many more people iden- to make arrests in some neighbourhoods but less in oth- tify themselves as “Black” or “Latino” than 3 decades ago. ers, we will not be measuring crime, but police biases. If Furthermore, some people are considered “White” in one some judges are more strict than others, we will not be really continent and “Non-white” in another. Even our envisioning measuring crime again, but judicial biases (Yapo and Weiss of sex as something purely biological is rapidly changing as 2018; Lee 2018). Again, just like with “health”, we see here new understandings of gender arise. We should be extremely how a clear understanding of the concept of “crime” is cru- careful with older datasets, which may no longer reflect the cial for the discussion. Consider how these biases are actu- values and identities of the society we are trying to model. ally happening, despite the link between crime and arrests A map is not the territory, and the model is not the thing being quite close, and how things can possibly go wrong we are modelling. Furthermore, artificial intelligence appli- when we use an individual’s “net worth” or “activity on cations are sometimes presented as having a real understand- Twitter” as a proxy measure for health. ing of the things mapped, when it is not so. If we remove any Another important lesson here is how we should do away sort of social determinant of health in our model, we will with having accuracy as the only factor in the creation of not be able to consider how socio-economic status, race, a model and to consider the inclusion of other metrics in or gender leads to differences in development and on the search for fairness, such as equal accuracy across groups way people are treated when entering the health system. A (Srivastava et al. 2019). A normativist conception of health machine learning model primarily based on a naturalistic would sway a developing team in that direction. In the same approach to health will that way become a very simplis- way, adding mathematical elements to protect the patient’s tic map with no chance of giving us an accurate picture of privacy, such as differential privacy (Dwork and Roth 2014), how health and sickness are not equally distributed among will make the model somewhat less accurate. This should a population. It might also occlude the discovery of the rea- not be a problem. We are not considering abstract math- sons for such inequality. Without an explicit declaration of ematical functions but models that will be used to classify what is understood behind a system’s key concepts (in this humans and a little less accuracy is welcomed if we get more case health), many issues may remain hidden and negatively equity and a better understanding of the real relationship impact the lives of those affected by the system, both in the between those scores and the real lives of people and their present and in the future. health. Taking the map as the territory is also problematic for the Moreover, a naturalistic model of health is inherently general public, which has access to the world of artificial more prone than a normativist to suggest that accuracy is 13 AI & SOCIETY possible at all. We can measure heart rate accurately and ours underscores the need to state the way the key overarch- derive a benchmark from these measurements, but from a ing concepts are conceptualised. We have argued that it is normativist perspective, it makes no sense to talk about an not enough to talk about “flourishing”, “health”, or “safety”, accurate prediction of affiliation, the capacity to feel love because these notions are intrinsically fraught concepts that and be loved, or bodily integrity. Yet, all of these themes can need further clarification to truly offer the transparency and be linked to health issues. the explanatory justifications everyone expects. The same So far, we have focused on the design and development can be said about purposes or tasks: saying that the purpose of a whole system (a smartphone app that tracks snoring) was to improve health will not cut it. to exemplify our discussion, but our arguments in favour of transparency about concepts are also valid for parts of the process. The need to declare how a team understands health 6 Conclusion becomes especially relevant for example when pretrained third-party models are integrated into other systems via As Gebru et al. (2021) show, the way datasets are defined is APIs, as it often occurs. Conceivably, the third-party model critical to assure a useful and ethical behaviour when using could have been trained by people with different conceptual a model in the wild. In this paper, we have shown that, underpinnings. Without conceptual transparency, disaster is when creating AI systems, our conception of health may served. have implications for all the stages along the lifecycle. Using As it becomes clear from Figs. 1 and 2, our proposal is a Boorsean approach to health facilitates the generation of rather modest in technical terms. One possibility would sim- simpler and more accurate models, but we may generate sim- ply be to add metadata to the code. This could be done in plifications and ethically inadmissible discriminations in the the way the problem type is declared in a model’s metadata process by excluding people outside the normal distribution (e.g., “Problem type: summarization”) or like the environ- or by choosing biased reference classes. A system based on mental impact of model training expressed in ­CO2 emissions that view might require reference classes that appear to be is shared in the metadata by the Hugging Face community objective but are not only highly contested but also depend- (Hugginface 2021). It could also be operationalized in terms ent on norms and values (e.g., race and sex). By being as simple as adding a text section in the system’s documenta- grounded on a broader understanding of what health is, a tion where the key concepts for the system are considered. capabilitarian approach will possibly lead to more complex Another possibility would be to integrate these justifica- and less -accurate algorithms, but ones that are prima facie tory explanations into the “model cards” framework pro- less prone to being biased and causing negative impacts. posed by Margaret Mitchell and her associates (Mitchell In short, if we want to create machine learning models in et al. 2019). This framework encourages transparent model a health domain that are not only accurate but also just and reporting and documentation accompanying trained machine developed for full human interaction, we need to consider learning models. Model cards include information about and define in detail our understanding of what health is. model details, intended use, metrics, evaluation, training Mistakes, biases, and inequalities will inevitably arise, but data, etc. While there is certainly much common ground they might be easier to spot and resolve thanks to the con- between our proposal and the model cards framework, the ceptual transparency afforded by justificatory explanations. focus of our paper is much narrower and goes one step fur- A non-technical intervention like the one we are proposing ther specifically in what the authors refer to as the area of here might aid the critical work that is required to properly “ethical considerations” where they recommend to consider evaluate intrinsically fraught systems going beyond examin- questions such as “Is the model intended to inform deci- ing datasets and code. sions about matters central to human life or flourishing— We should avoid the oracular approach to artificial intel- e.g., health or safety? Or could it be used in such a way?” ligence, in which an algorithm generates “objective values” (Mitchell et al. 2019: p. 6). Likewise, another possibility based on “pure data”, and humans are no longer necessary. would be to include the explanations into the “datasheets Humans (from policymakers to civil society to individuals) for datasets” proposed by Gebru and her associates (2021), need to have access to an explanation that offers an account where the motivation of a dataset, its composition, collec- of how the algorithm works in terms they can understand. tion process, recommended uses, and so on are documented. We need to consider how any computer-based intervention Gebru’s framework suggests including information about the in medicine starts with a specific understanding of health, “reasons for creating the dataset”, “for what purpose the which is linked to values that are very difficult to turn into dataset was created”, and “if there were specific tasks in raw data. Therefore, we need to change our paradigm and mind?”. create systems based on other models where humans and Despite the many similarities between our proposal and algorithms interact, instead of just the algorithm stating like Mitchell’s and Gebru’s more comprehensive frameworks, an oracle what the human must do. In this view, for instance, 13 AI & SOCIETY a diagnosis is made by a human-assisted by the machine, in Humanities (Detección y eliminación de sesgos en algoritmos de and a key element in such collaboration is how the com- triaje y localización para la COVID-19). David Casacuberta and Cris- tian Moyano-Fernández are part of the consolidated research network puter system enhances human decision-making, but does “Grup d’Estudis Humanístics de Ciència I Tecnologia” (GEHUCT) not replace it with something whose justification cannot be (“Humanistic Studies of Science and Technology Research Group”), fully comprehended or is outright unavailable. Such systems recognised and funded by the Generalitat de Catalunya, reference 2017 need to include in their documentation the key concepts (in SGR 568. Ariel Guersenzvaig is part of the emergent research network “Human, Information, Material, Technology and Society Research” our examples, the justificatory explanations of “health”) that (HIMTS), recognised and funded by the Generalitat de Catalunya, ref- underpinned its design and development, so the human users erence 2017 SGR 1773. and stakeholders can have a proper understanding of what the engineering team aimed at and what the implications of Open Access This article is licensed under a Creative Commons Attri- its use might be. bution 4.0 International License, which permits use, sharing, adapta- tion, distribution and reproduction in any medium or format, as long We want to end by considering one objection to our as you give appropriate credit to the original author(s) and the source, proposal, which would be to reiterate that most model and provide a link to the Creative Commons licence, and indicate if changes software developers are technically oriented people who are were made. The images or other third party material in this article are usually not trained in philosophy or the social sciences. The included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in thrust of the objection is that they would have difficulty with the article's Creative Commons licence and your intended use is not or would be unable to produce fully blown explanatory jus- permitted by statutory regulation or exceeds the permitted use, you will tifications or precise descriptions of the key concepts that need to obtain permission directly from the copyright holder. To view a underpin the system they are designing or have designed. copy of this licence, visit http://​creat​iveco​mmons.​org/​licen​ses/​by/4.​0/. We would reply that all we are asking is that they externalise concepts they must already possess and are already guiding their decision-making either explicitly or implicitly. We find References it reasonable to expect that the creators of high-impact sys- Adamson AS, Smith A (2018) Machine learning and health care dis- tems in areas such as health, security, or education, to name parities in dermatology. JAMA Dermatol 154(11):1247–1248 a few, must be able to explain to others what they understand Alexander M (2019) The new jim crow: mass incarceration in the age by the thick, value-laden and fraught concepts on which their of colourblindness. Penguin, London work is ultimately based. We posit that it is not unreasonable Alvarez M (2017) Reasons for action: justification, motivation, expla- nation. The Stanford Encyclopedia of Philosophy (Winter 2017 to expect from them and from other relevant decision-mak- Edition). https://p​ lato.s​ tanfo​ rd.e​ du/a​ rchiv​ es/w ​ in201​ 7/e​ ntrie​ s/r​ easo​ ers involved in the design of AI systems to provide explana- ns-​just-​vs-​expl. Accessed 15 Apr 2021 tions in terms that can be understood and evaluated by the Angwin J, Larson J, Mattu S, Kirchner L (2016) Machine bias. Pro- public. We would also contend that the higher the potential Publica. Retrieved 1/12/2021 https://​www.​propu​blica.​org/​artic​le/​ machi​ne-​bias-​risk-​asses​sments-​in-​crimi​nal-​sente​ncing. Accessed impact of a system in our lives, the more explicit and sophis- 1 Dec 2021 ticated the understanding of these key concepts needs to be Ashmore R, Calinescu R, Paterson C (2021) Assuring the machine and the more developers and decision-makers need to be learning lifecycle: desiderata, methods, and challenges. ACM aware of alternative conceptual understandings. Now, we Comput Surv 54(5):1–39. https://​doi.​org/​10.​1145/​34534​44 Barredo Arrieta A, Díaz-Rodríguez N, Del Ser J et al (2020) Explaina- have nothing like the space we would need to discuss how ble Artificial Intelligence (XAI): concepts, taxonomies, opportuni- this could be done, but it is clear that it might be necessary ties and challenges toward responsible AI. Inf Fusion 58:82–115. for AI teams to work with experts from other domains such https://​doi.​org/​10.​1016/j.​inffus.​2019.​12.​012 as philosophy, anthropology, sociology, and the humanities Benjamin R (2019) Race after technology. Polity, London Bircher J (2005) Towards a Dynamic definition of health and dis- (for integrating ethicists in the process, see Van Wynsberghe ease. Med Healthc Philos 8(3):335–341. https://​doi.​org/​10.​1007/​ and Robbins 2014). Our argument is that it needs to be done. s11019-​005-​0538-y If we really want transparency, accountability, and explain- Boorse C (1977) Health as a theoretical concept. Philos Sci 44(4):542– ability, we need to go all the way beyond code and data to 573. http://​www.​jstor.​org/​stable/​186939 Boorse C (2014) A second rebuttal on health. J Med Philos 39:683– illuminate the other black box that resides in the minds of 724. https://​doi.​org/​10.​1093/​jmp/​jhu035 all of those who design and develop AI systems. Providing Cavoukian A (2009) Privacy by design: the 7 foundational principles. justificatory explanations might be a good start. Information and privacy commissioner of Ontario, Canada, vol 5, p 12 Cecaj A, Mamei M, Zambonelli F (2016) Re-identification and infor- mation fusion between anonymized CDR and social network data. Funding Open Access Funding provided by Universitat Autonoma J Ambient Intell Humaniz Comput 7(1):83–96. https://d​ oi.o​ rg/1​ 0.​ de Barcelona. This work has been partially funded by the Ministry 1007/​s12652-​015-​0303-x of Science, Innovation and Universities within the State Subprogram Chen IY, Szolovits P, Ghassemi M (2019) Can AI help reduce dis- of Knowledge Generation through the research project FFI2017- parities in general medical and mental health care? AMA J Ethics 85711-P Epistemic innovation: the case of biomedical sciences; and 21(2):167–179. https://​doi.​org/​10.​1001/​amaje​thics.​2019.​167 by the BBVA Foundation for SARS-CoV-2 and COVID-19 Research Coeckelbergh M (2019) AI ethics. MIT Press, Cambridge 13 AI & SOCIETY CriadoPerez C (2019) Invisible women: exposing data bias in a world Schramme T (2007) A qualified defence of a naturalist theory of designed for men. Random House, New York health. Med Healthc Philos 10:11–17. https://​doi.​org/​10.​1007/​ Cross N (2007) Designerly ways of knowing. Birkhäuser, Basel s11019-​006-​9020-8 D’Ignazio C, Klein LF (2020) Data feminism. MIT Press, Cambridge Searle J (2001) Rationality in action. MIT Press, Cambridge Dwork C, Roth A (2014) The algorithmic foundations of differential Seyyed-Kalantari L, Liu G, McDermott M et al (2020) CheXclusion: privacy. Found Trends Theor Comput Sci 9(3–4):211–407. https://​ fairness gaps in deep chest X-ray classifiers. In: BIOCOMPUT- doi.​org/​10.​1561/​04000​0004 ING 2021: proceedings of the Pacific symposium, pp 232–243 Flyvbjerg B (2001) Making social science matter: why social inquiry Soobader MJ, LeClere FB, Hadden W, Maury B (2001) Using aggre- fails and how it can succeed again. Cambridge University Press, gate geographic data to proxy individual socioeconomic status: Cambridge does size matter? Am J Public Health 91(4):632–636. https://​doi.​ Gebru T, Morgenstern J, Vecchione B et al (2021) Datasheets for datasets. org/​10.​2105/​ajph.​91.4.​632 Commun ACM 64(12):86–92. https://​doi.​org/​10.​1145/​34587​23 Srivastava M, Heidari H, Krause A (2019) Mathematical notions vs. Google (2017) The 4 stages of machine learning: from BI to ML. human perception of fairness: a descriptive approach to fairness Build with Google Cloud. https://​stora​ge.​googl​eapis.​com/​build​ for machine learning. In: Proceedings of the 25th ACM SIGKDD cms.g​ oogle.c​ om.a.a​ ppspo​ t.c​ om/d​ ocume​ nts/t​ he-4-s​ tages-o​ f-m ​ achi​ international conference on knowledge discovery & data mining, ne-​learn​ing_​11-6.​pdf. Accessed 15 Nov 2021 pp 2459–2468 Hayashi Y (2020) New unified insights on deep learning in radiologi- Sweeney L (2000) Uniqueness of simple demographics in the U.S. cal and pathological images: beyond quantitative performances population. Data Privacy Lab White Paper Series LIDAP-WP4. to qualitative interpretation. Inform Med Unlocked 19:100329. Carnegie Mellon University, School of Computer Science, Pitts- https://​doi.​org/​10.​1016/j.​imu.​2020.​100329 burgh, PA, USA Holm EA (2019) In defense of the black box. Science 364(6435):26– Trollope-Kumar K, Last J (2002) Lay concepts of health and illness. In: 27. https://​doi.​org/​10.​1126/​scien​ce.​aax01​62 Breslow L (ed) Encyclopedia of public health, vol L-R. Macmillan Huggingface (2021) Hugging face. https://​huggi​ngface.​co/​models?​ Reference USA, New York, pp 685–686 other=c​ o2_e​ q_e​ missi​ ons&s​ ort=d​ ownlo​ ads. Accessed 1 Dec 2021 Underwood JM, Townsend JS, Tai E et al (2012) Racial and regional Johnson DG (2006) Computer systems: moral entities but not moral disparities in lung cancer incidence. Cancer 118(7):1910–1918. agents. Ethics Inf Technol 8:195–204. https://​doi.​org/​10.​1007/​ https://​doi.​org/​10.​1002/​cncr.​26479 s10676-​006-​9111-5 Van Wynsberghe A, Robbins S (2014) Ethicist as designer: a prag- Kawachi I, Daniels N, Robinson DE (2005) Health disparities by race matic approach to ethics in the lab. Sci Eng Ethics 20(4):947–961. and class: why both matter. Health Aff 24(2):343–352. https://d​ oi.​ https://​doi.​org/​10.​1007/​s11948-​013-​9498-4 org/​10.​1377/​hltha​ff.​24.2.​343 Venkatapuram S (2011) Health justice: an argument from the capabili- Kingma E (2010) What is it to be healthy? Br J Philos Sci 61(2):241– ties approach. Polity Press, Cambridge 264. https://​doi.​org/​10.​1093/​analys/​67.2.​128 Von Wright GH (1981) Explanation and understanding of action. Rev Klaus K, Stummer AL, Ruf S (2021) Accuracy of a smartphone appli- Int Philos 135:127–142. https://​www.​jstor.​org/​stable/​23945​379 cation measuring snoring in adults-how smart is it actually? Int J Ware JK (2019) Property value as a proxy of socioeconomic status in Environ Res Public Health 18(14):7326. https://​doi.​org/​10.​3390/​ education. Educ Urban Soc 51(1):99–119. https://d​ oi.o​ rg/1​ 0.1​ 177/​ ijerp​h1814​7326 00131​24517​714850 Kolko J (2011) Exposing the magic of design: a practitioner’s guide WHO (1948) Preamble of the constitution of the World Health Organi- to the methods and theory of synthesis. Oxford University Press, sation as adopted by the International Health Conference (Official Oxford Records of the WHO, 2:100). Geneva Lee NT (2018) Detecting racial bias in algorithms and machine Wilkinson RG, Pickett K (2006) Income inequality and population learning. J Inf Commun Ethics Soc. https://​doi.​org/​10.​1108/​ health: a review and explanation of the evidence. Soc Sci Med JICES-​06-​2018-​0056 62(7):1768–1784. https://​doi.​org/​10.​1016/j.​socsc​imed.​2005.​08.​036 Levy Abitbol J, Fleury E, Karsai M (2019) Optimal proxy selection for Williams B (2006) Ethics and the limits of philosophy. Routledge, socioeconomic status inference on twitter. Complexity 5915:1–15. Abingdon https://​doi.​org/​10.​1155/​2019/​60596​73 Wortham RH, Theodorou A (2017) Robot transparency, trust and util- MacKenzie D, Wajcman J (1999) The social shaping of technology. ity. Connect Sci 29(3):242–248. https://​doi.​org/​10.​1080/​09540​ Open University Press, Buchingham 091.​2017.​13138​16 Marmot MG, Wilkinson RG (1999) Social determinants of health. Yapo A, Weiss J (2018) Ethical implications of bias in machine learn- Oxford University Press, Oxford ing. In: Proceedings of the 51st Hawaii international conference Mitchell M, Wu S, Zaldivar A et al (2019) Model cards for model on system sciences reporting. In: Proceedings of the conference on fairness, account- Yin L, Wang Q, Shaw SL et al (2015) Re-identification risk versus data ability, and transparency, Atlanta, GA, USA. https://​doi.​org/​10.​ utility for aggregated mobility research using mobile phone loca- 1145/​32875​60.​32875​96 tion data. PLoS One 10(10):e0140589. https://​doi.​org/​10.​1371/​ Moore GE (1993) Principia ethica. Cambridge University Press, New journ​al.​pone.​01405​89 York Yunus F, Khan S, Mitra DK et al (2018) Relationship of sleep pat- Murphy D (2021) Concepts of disease and health. The Stanford Ency- tern and snoring with chronic disease: findings from a nationwide clopedia of Philosophy (Spring 2015 edition). https://​plato.​stanf​ population-based survey. Sleep Health 4(1):40–48. https://d​ oi.o​ rg/​ ord.​edu/​archi​ves/​spr20​21/​entri​es/​health-​disea​se/. Accessed 15 10.​1016/j.​sleh.​2017.​10.​003 Apr 2021 Zednik C (2021) Solving the black box problem: a normative frame- Nordenfelt L (2007) Establishing a middle-range position in the theory work for explainable artificial intelligence. Philos Technol of health: a reply to my critics. Med Healthc Philos 10:29–32. 34(2):265–288. https://​doi.​org/​10.​1007/​s13347-​019-​00382-7 https://​doi.​org/​10.​1007/​s11019-​006-​9018-2 Nordenfelt L, Khushf G, Fulford KWM (2001) Health, science, and Publisher's Note Springer Nature remains neutral with regard to ordinary language. Rodopi, Amsterdam jurisdictional claims in published maps and institutional affiliations. Nussbaum M (2006) Frontiers of justice: disability, nationality, species membership. Cambridge University Press, New York 13