Environmental Modelling and Software 158 (2022) 105552 Contents lists available at ScienceDirect Environmental Modelling and Software journal homepage: www.elsevier.com/locate/envsoft Which predictive uncertainty to resolve? Value of information sensitivity analysis for environmental decision models Fridolin Haag a ,∗, Sara Miñarro b , Arjun Chennu a a Leibniz Center for Tropical Marine Research (ZMT), Fahrenheitstrasse 6, 28359 Bremen, Germany b Institute of Environmental Science and Technology (ICTA), Universitat Autònoma de Barcelona, 08193 Bellaterra, Spain ARTICLE INFO ABSTRACT Dataset link: https://doi.org/10.5281/zenodo.7 Uncertainties in environmental decisions are large, but resolving them is costly. We provide a framework 156065, https://doi.org/10.5281/zenodo.7155 for value of information (VoI) analysis to identify key predictive uncertainties in a decision model. The 783, https://doi.org/10.5281/zenodo.7155948 approach addresses characteristics that complicate this analysis in environmental management: dependencies , https://doi.org/10.5281/zenodo.7156015 in the probability distributions of predictions, trade-offs between multiple objectives, and divergent stakeholder Keywords: perspectives. For a coral reef fisheries case, we predict ecosystem and fisheries trajectories given different man- Sensitivity analysis agement alternatives with an agent-based model. We evaluate the uncertain predictions with preference models Multi-criteria decision analysis based on utility theory to find optimal alternatives for stakeholders. Using the expected value of partially Uncertainty perfect information (EVPPI), we measure how relevant resolving uncertainty for various decision attributes is. Agent-based modeling The VoI depends on the stakeholder preferences, but not directly on the width of an attribute’s probability Coral reef management distribution. Our approach helps reduce costs in structured decision-making processes by prioritizing data collection efforts. 1. Introduction than the magnitude of variation in the outcomes (Borgonovo et al., 2016; Felli and Hazen, 1998; Razavi et al., 2021). The analysis sup- Environmental management decisions need to be made in the face ports us in answering the questions: would our decision for a specific of large uncertainties. Confronted with these uncertainties, an intuitive management alternative change if we had more information? And if response is to collect additional information on the reasonable assump- so, which information would most influence our choice? Or conversely, tion that more information will reduce uncertainty. This in turn may resolving which of the uncertainties would bring more utility? Based on improve our understanding and, ultimately, lead to better management the answers, we can rationally prioritize between research, monitoring, decisions. and implementation activities for a concrete decision. However, collecting more information requires time and effort, Tackling complex decisions in a rational way and conducting VoI which could otherwise be allocated to management actions, particu- larly in countries with limited resources for environmental manage- analysis is facilitated by a quantitative representation of the decision ment. When we know enough about a system to make a sensible and its uncertainties (see Section 2.1). This is achieved by a decision decision, requesting ‘‘more science’’ rather becomes a technique to model, which consists of a predictive system model and a preference delay implementation (Gregory et al., 2006). Faced with the trade- model (Reichert et al., 2015; Haag et al., 2019b). The predictive system off between researching and managing an ecosystem, it is important model (also called prediction model, system model, assessment model, to consider: to what extent is the use of limited resources to improve or simulation model in the literature) provides measures of various our understanding of a system justified by the resulting potential for system attributes (e.g., coral cover, crop yield, breeding pairs, etc.) improved management? given an input state. This allows us to assess the potential consequences This question can be approached with the concept of value of in- if we implemented different management alternatives (also called op- formation (Howard, 1966; Feltham, 1968). Value of information (VoI) tions, variants, actions, strategies, or scenarios in the literature). The analysis allows us to determine the value (in the sense of benefit or preference model (also called evaluation or utility model) represents utility) of additional information for deciding between different alter- how decision-makers, stakeholders, or society perceive and evaluate natives. We can think of VoI analysis as a form of sensitivity analysis the predicted consequences of the system state. In this study, we with a focus on the sensitivity of the decision to new information rather ∗ Corresponding author. E-mail address: fridolin.haag@leibniz-zmt.de (F. Haag). https://doi.org/10.1016/j.envsoft.2022.105552 Received 9 March 2022; Received in revised form 5 August 2022; Accepted 3 October 2022 Available online 16 October 2022 1364-8152/© 2022 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). F. Haag et al. Environmental Modelling and Software 158 (2022) 105552 focus on quantitative preference models based on multi-attribute utility Decisions about their local management are not only difficult because theory (Keeney and Raiffa, 1993). coral reefs and their fisheries are intricately linked, but they also Value of information analysis is an established approach in health fulfill a variety of needs in often resource-constrained socio-economic care economics and medical decision making (e.g., Felli and Hazen, contexts (Ferse et al., 2014). Therefore, the divergent objectives and 1998; Fumie and Thompson, 2004; Jackson et al., 2022). There is also perspectives of various stakeholders need to be considered in decision growing interest in conservation biology (reviewed by Bolam et al., making. Typically, the available data and our ability to collect data is 2019) and environmental management in general (Keisler et al., 2014; limited, requiring iterative and adaptive management approaches that Eidsvik et al., 2015). However, many environmental decision problems include targeted efforts for the collection of additional information. share characteristics that complicate VoI analysis and at the same time After discussing our framework for VoI analysis (Section 2), we make it especially relevant for these problems. introduce the structure of the reef management decision case (Sec- In this paper we address two research gaps for VoI analysis that tion 3.1). We describe how to analyze the EVPPI for predictions of emerge from the complexity of environmental decisions. First, we fisheries management alternatives and discuss design choices for such lack an integrative framework for jointly tackling common characteris- efforts (Section 3). Based on the analysis results (Section 4), we discuss tics of such decisions, including (1) multiple, conflicting management the usefulness of VoI analysis and suggest desirable extensions to our objectives (management criteria), (2) uncertain consequences of man- implementation (Section 5). agement alternatives with continuous distributions rather than discrete states or hypotheses, and (3) probability distributions of consequences 2. Framework for analyzing value of information for decision that are not independent from each other. Multi-criteria decision ap- models proaches (MCDA/MCDM) have been used with VoI analysis before (e.g., Bates et al., 2014; Runge et al., 2011; Eidsvik et al., 2015), 2.1. Decision modeling but mostly have addressed these aspects separately (see Bolam et al., 2019). VoI analysis for continuous probability distribution remains Making a rational decision between several alternatives is facilitated conceptually straightforward, but can become challenging for (condi- by a quantitative representation of the consequences of implementing tionally) dependent probability distributions (e.g., Eidsvik et al., 2015; an alternative and of the stakeholder preferences regarding these conse- Myklebust et al., 2020). Second, multiple, conflicting perspectives on quences. The combination of predictive system models with preference an environmental decision problem commonly exist. However, the models to evaluate decision alternatives we call a decision model consideration of multiple stakeholder perspectives in VoI analysis has (see Haag et al., 2019b). In the following, we provide the concep- not been discussed widely in the literature before. We address this issue tual background to VoI analysis for such models (Fig. 1). A concrete and investigate the influence of diverging stakeholder perspectives on implementation is detailed in Section 3. the results and conclusions from VoI analysis. After a comprehensive problem structuring phase (e.g., Marttunen The aim of this paper is to develop an integrated approach to et al., 2017), the first step of decision modeling is to predict the conducting VoI analysis given the characteristics of environmental deci- consequences 𝒚 𝑎 = (𝑦𝑎,1 , … , 𝑦𝑎,𝑚 ) for each alternative 𝑎 on the system at- sions, and highlight relevant considerations when applying the method. tributes (1, … , 𝑚) that are relevant for decision making according to the The following questions guide our approach: stakeholders. These predictions can come from a mathematical model, but also from experts (e.g., Nicol et al., 2019). Given that environmen- 1. How can we estimate the VoI based on the correlated output of tal or socio-ecological systems can neither be completely understood, complex system models? nor fully observed, nor perfectly represented in a model, it makes 2. How to analyze VoI in decision contexts with multiple, conflict- sense to conceptualize these predictive system models as probabilistic ing objectives? models with uncertain parameters or inputs (Reichert et al., 2015; 3. What is the influence of diverging stakeholder perspectives on Reichert, 2020). Therefore, the consequences for an alternative 𝑎 are the conclusions of VoI analysis? better represented as a random vector Y𝑎 , as we do not obtain a point To address these questions, we create a framework to conduct VoI estimate for the predicted variables, but distributions of predictions 𝑝Y𝑎 . analysis for complex decisions, bringing together elements that have The distribution of predictions is a more informative description of our been developed previously. Our VoI analysis rests upon a decision knowledge than a point estimate or aggregated value. The predicted model that combines system predictions and stakeholder preferences distributions will often not be independent. For instance, the biomass (see above and Section 2.1). The predictions can come from one or of carnivorous fish may be negatively correlated with biomass of prey more arbitrarily complex models and can exhibit dependencies in their species. distributions. To efficiently conduct VoI analysis for dependent proba- The second step is to identify a utility function, 𝑢, that quantifies a bility distributions, we adapt an algorithm by Strong and Oakley (2013) stakeholder’s preferences with regard to the predicted attribute levels as a replacement for traditional double-loop Monte Carlo procedures. and their uncertainties. The utility function represents the trade-offs While different measures of the VoI have been proposed (see Eidsvik the stakeholder is willing to make between uncertain consequences. et al., 2015 and Section 2.2), we use the expected value of partially Let Y denote the set of all possible consequences for all attributes perfect information (EVPPI; also called EVPXI or EVXI). This measures considered relevant in the decision. A multi-attribute utility function the expected gain in utility if we would decide based on perfect 𝑢 ∶ Y → [0, 1] maps from a space of predicted consequence regarding information about one (or few) variables of interest instead of being system attributes to a utility space. It returns the utility of potential uncertain about them. consequences, with larger utilities representing more preferred system To implement our framework, we investigate a case study on coral states (Keeney and Raiffa, 1993). Standard utility theory does not reef fisheries management for islands in the Indo-Pacific. It is a complex consider uncertainty of preferences; in the following we keep to this decision problem in a region that requires sound management of pre- limitation. A more extensive and rigorous treatment of multi-attribute cious and fragile ecosystems (Eddy et al., 2021). Coral reefs are under utility theory than outlined here can be found in textbooks (e.g., Keeney intense pressure from various local and global stressors, such as climate and Raiffa, 1993; French, 1986). change, pollution, fisheries, and other impacts (Burke et al., 2012). A Once we specified parameters of a utility function based on a reef’s ecological complexity poses substantial challenges for predicting stakeholder’s preferences, we can calculate the resulting multi-attribute the state of the reef under a management alternative. While different utility 𝑢(𝒚 𝑎 ) for an individual prediction of consequences 𝒚 𝑎 = (𝑦𝑎,1 , … , modeling approaches can be helpful (Kelly et al., 2013), we use a 𝑦𝑎,𝑚 ). For instance, 𝑦1 could be the prediction for the coral cover in % spatially explicit agent-based model in our study (Miñarro et al., 2018). and 𝑦𝑚 the prediction for herbivorous fish biomass per 𝑚2 three years 2 F. Haag et al. Environmental Modelling and Software 158 (2022) 105552 Fig. 1. Scheme of decision modeling and value of information (VoI) analysis followed in this study. The predicted uncertain consequences of alternatives (𝑌𝑖 (𝑎)) are evaluated according to stakeholder preferences to determine an optimal baseline choice by maximizing expected utility (EU). The sensitivity of this choice to new information is then investigated with VoI analysis as a form of sensitivity analysis. For a practical case, more alternatives, several different predictive models, more attributes, and more stakeholders may exist than depicted here. from now. However, as the predictions are uncertain, we need to For illustration, let us consider a decision about a system in which evaluate their entire distribution. Given uncertain consequences Y(𝑎) only one among all the variables is uncertain. Following ideas intro- for an alternative 𝑎, we receive corresponding utilities U𝑎 = 𝑢(Y(𝑎)). duced by Felli and Hazen (1998), we consider three perspectives of a For a given stakeholder preference profile, we obtain a ranking of the decision’s sensitivity to that uncertain variable: alternatives based on their expected utility (EU): • Threshold perspective: If we varied the variable from its lowest to EU(𝑎) = E[U𝑎 ] = 𝑢(𝒚)𝑝𝒀 𝑎 (𝒚)d𝑦 (1) highest value, is there any threshold level at which the optimal ∫𝑦 choice changes? If not, the decision is insensitive to that variable. with 𝑝𝒀 𝑎 the probability distribution of the consequences of imple- • Probabilistic perspective: Given the probability distribution of the menting alternative 𝑎 as measured by all of the system attributes. The variable, how likely is it that the threshold is crossed? If this alternative with the highest EU is then determined by maximizing: is very unlikely, the optimal choice remains insensitive to the variable, even though a threshold exists. max E[𝑢(Y(𝑎))] (2) • Utility foregone perspective: How large is the expected difference 𝑎=1…𝐴 in utility between our baseline choice and the optimal choice if Utility theory is prescriptive because a rational decision maker should the threshold would be crossed? If the difference in utility is small choose the alternative with the highest EU, 𝑎∗ , given their prefer- enough, the penalty for having chosen a suboptimal alternative ences (encoded in the function 𝑢) and the probability distributions of will be small enough that we can consider the decision to be predicted consequences 𝑝𝒀 𝑎 . insensitive. Utility theory allows determining the optimal choice for individuals or groups with homogeneous preferences. For environmental manage- Value of information can be thought of as a sensitivity measure ment decisions, the societal utility of an alternative must be considered that takes both the probabilistic and utility foregone perspectives into from conflicting stakeholder perspectives. Which decision-making cri- account (Felli and Hazen, 1998; Borgonovo et al., 2016). Converse terion should be considered rational or fair for a group or society is to utility forgone, it measures the gain in utility we can expect from an unresolved question (de Jonge, 2012). However, making a number knowing a decision aspect, such as a system attribute or parameter, of contested assumptions, such as the possibility for interpersonal with (more) certainty. With VoI we thus measure how sensitive the comparisons of utility, we can construct a group utility function from optimal choice is to potential new information, before we acquire it. the individuals’ utility functions (Keeney, 2013). To obtain a group If the VoI is small, it is unlikely that the baseline optimal choice expected utility EU𝐺 (𝑎) of an alternative 𝑎 across 𝐾 stakeholders per- changes with that new information or the difference in utility that this spectives, we aggregate individual expected utilities EU𝑘 using the change would bring is small. Thus, VoI supports us in prioritizing which weighted arithmetic mean. The weights 𝑤𝑘 with 0 < 𝑤𝑘 ≤ 1 are scaling uncertainties should be reduced by further information collection. factors for the utility between stakeholders that sum to one (Eq. (3)). Several variants of VoI measures have been proposed. They share the same concept, but differ with regard to the additional information ∑ 𝐾 EU𝐺 (𝑎) = 𝑤𝑘 ⋅ EU𝑘 (𝑎) (3) that is considered. The expected value of perfect information (EVPI), 𝑘=1 measures the sensitivity of obtaining perfect information on all modeled aspects of the decision (Eidsvik et al., 2015). The expected value of 2.2. Value of information sensitivity analysis partially perfect information (EVPPI, also abbreviated EVXPI or EVXI), measures the sensitivity of obtaining perfect information on parts of the The optimal baseline choice in a decision is the alternative 𝑎∗ uncertainties, for example, one or several parameters (Eidsvik et al., obtained by maximizing EU, given the current state of information and 2015). This is the measure we focus on in the following. the current preferences. However, in practice, we are usually concerned Given that aleatory uncertainty cannot be reduced by more studies about the sensitivity or robustness of such a conclusion: How bad or data collection, EVPI and EVPPI can be thought of as upper bounds would it be if we picked this seemingly optimal alternative, but some for the actual VoI we can expect. In practice, we will always only ob- information turned out to be different in reality? tain imperfect information. The expected value of sample information 3 F. Haag et al. Environmental Modelling and Software 158 (2022) 105552 (EVSI) is a measure for the impact of obtaining additional data, but not a stakeholders has an exponential utility function (Keeney and Raiffa, perfect information. Estimating EVSI is still difficult in many contexts, 1993), these are the respective certainty equivalents of the situation but it is an active area of research (e.g., Williams and Johnson, 2018; where information is available for free and where no further informa- Kunst et al., 2020). tion is available (Eidsvik et al., 2015). If a stakeholder is risk-neutral, the certainty equivalents are the expected values, as in Eq. (4). 2.3. The expected value of partially perfect information If we expand our analysis to several variables of interest (e.g., growth rates, herbivore fish biomass, revenue for fishers, etc.), we The idea of VoI is to calculate the added value of a piece of infor- can determine a ranking of the variables in terms of their EVPPI. mation before we actually expend the effort and cost to obtain it. In the This provides a quantitative and transparent prioritization of the most case of EVPPI, we estimate the expected gain if we knew one or a group relevant uncertainties. This can serve as the basis for directing study of variables of interest (parameters, inputs, or attributes) with certainty. design and deciding about upcoming expenditure of costs and efforts A formal and extensive treatment as well as algorithms to estimate the towards collecting data and information. EVPPI is available in the literature (Brennan et al., 2007; Eidsvik et al., 2015; Borgonovo et al., 2016; Heath et al., 2017). Here, we sketch out 2.4. Given-data approach to estimating EVPPI and threshold sensitivity the main ideas with the aim to provide an intuitive understanding of the approach. In this study we focus on uncertainties of the predictions Since analytical solutions can rarely be found, various simulation- of the attributes. However, VoI analysis can be conducted for other based approaches to estimate the EVPPI have been developed. The clas- variables within the models by following the same approach. sical procedure is a two-level nested Monte Carlo simulation (e.g., Felli Let us assume we have determined the optimal baseline choice (𝑎∗ ) and Hazen, 1998; Brennan et al., 2007). However, this requires many for a coral reef management decision, given the current state of data simulations. In addition, to account for dependencies in the distri- and knowledge. We are now interested to find out how sensitive that butions of predictions, Markov-Chain-Monte-Carlo (MCMC) or other choice is to one aspect of the decision: our prediction of the resulting conditional resampling procedures with high computational burden are coral cover (cc) under a marine protected area (MPA) alternative. We needed with a nested approach (Strong and Oakley, 2013). denote this variable of interest Y𝑖 = Ycc,MPA . If we had perfect informa- The alternative we describe here, is a given-data approach that tion about the resulting coral cover if an MPA were implemented, what only requires a single-loop Monte Carlo sample, as one obtains from would be the expected added value to the decision? a probabilistic sensitivity analysis. This procedure has been developed The idea of the analysis is that we pretend we could – with a by Strong and Oakley (2013) and is more generally discussed by Bor- certain cost and effort – gain clairvoyance on this variable and found, gonovo et al. (2016). The method was originally conceived for VoI for instance, that Y𝑐𝑐,𝑀𝑃 𝐴 = 𝑦𝑐𝑐,𝑀𝑃 𝐴 = 58%. This gives us a better calculations based on the net benefit of alternatives, here we adapt it estimate for the variable and can have three effects: (a) the conditional to usage with EU for risk-neutral preferences. We focus on this approach because it can handle the conditional distributions of the other variables, Y−𝑖 |Y𝑖 = 𝑦𝑖 , change if they are dependencies in the distributions of variables while being computa- not independent, (b) the distribution of all utilities U𝑎 change if they tionally more feasible in connection with complex system models, as a are not independent from the variable of interest, and (c) the optimal single-loop Monte Carlo sample is sufficient. A drawback of the method alternative might change. is that it is only efficient for calculating the EVPPI of single variables. We now calculate the maximum utility given this new information, To calculate EVPPI for sets of variables other approaches have been max𝑎=1…𝐴 {E[U𝑎 |Y𝑐𝑐,𝑀𝑃 𝐴 = 58]}, and determine the best alternative. If suggested (Strong et al., 2014; Heath et al., 2017). the best alternative is still the baseline optimal choice 𝑎∗ , the additional While we refer the reader to the original publication by Strong and information that coral cover under the MPA alternative will be 58% did Oakley (2013) for the details, the basic algorithm can be described as not help us make a better choice than we would have made anyway. follows. Rather, there is an opportunity cost, if effort was expended towards collecting this information. Consequently, the value of this information • From a Monte Carlo simulation we receive 𝑆 samples of the would be zero as the EU of the optimal alternative given the informa- uncertain variables and of the 𝑆 corresponding utilities for each tion, 𝑎∗ , and the EU of the baseline choice given the information is alternative. identical: max𝑎=1…𝐴 {E[U𝑎 |Y𝑐𝑐,𝑀𝑃 𝐴 = 58]} − E[U𝑎∗ |Y𝑐𝑐,𝑀𝑃 𝐴 = 58] = 0. • The vector of samples 𝒚 𝑖 of the variable of interest Y𝑖 for which we However, if the new information changed our optimal choice, this want to calculate the EVPPI, as well as the corresponding utilities difference will be positive and indicates the value of collecting this 𝒖(𝑎) for all management alternatives are now both reordered such information. that 𝑦(1) (2) (𝑆) 𝑖 ≤ 𝑦𝑖 ≤ ⋯ ≤ 𝑦𝑖 . The superscript denotes the reordered While this illustrates the VoI for one particular value 𝑦𝑐𝑐,𝑀𝑃 𝐴 , we position in the vector of variable samples. are interested in the expected VoI for Y𝑐𝑐,𝑀𝑃 𝐴 over its entire probability • The reordered samples and the corresponding reordered utilities distribution of the coral cover within the predicted range. We can are partitioned into 𝐾 bins of equal size 𝐽 , with 𝐽 ⋅ 𝐾 = 𝑆. estimate this by sampling from the predicted probability distribution of • For each bin 𝑘, we calculate the EU for each alternative. This is an coral cover, calculating the corresponding EU given that we ‘know’ this approximation of the EU conditional on the value of the variable sampled value, and tracking the corresponding change in the optimal of interest 𝑦𝑖 being in this bin. We then take the maximum EU choice. If the optimal choice does not change for any of these values, across the alternatives. it implies that no matter how much we improve our predictions our • The arithmetic mean of these maxima across all 𝐾 bins [ is taken conclusions would never change. as { an approximation }] to the first term in Eq. (4): E max𝑎=1...𝐴 More formally, for a risk-neutral stakeholder we calculate the EVPPI E[U𝑎 |Y∗ ] . The second term { of Eq.} (4), the EU of the baseline about the variable of interest Y𝑖 with: optimal choice – max𝑎=1…𝐴 E[U𝑎 ] – can be directly calculated [ ] as in Eq. (2). { } EVPPI(Y𝑖 ) = E max E[U𝑎 |Y𝑖 ] − E[U𝑎∗ |Y𝑖 ] 𝑎=1...𝐴 The corresponding estimator for EVVPI can be written as: [ ] (4) { } { } { } 1 ∑ ∑ 𝐾 𝐽 ⋅𝑘 = E max E[U𝑎 |Y𝑖 ] − max E[U𝑎 ] 1 𝑎=1...𝐴 𝑎=1…𝐴 EVPPI(Y𝑖 ) = max 𝑢(𝑠) (𝑎) 𝐾 𝑘=1 𝑎=1...𝐴 𝐽 (𝑠)=1+𝐽 ⋅(𝑘−1) The EVPPI can be viewed as the difference between a ‘‘posterior’’ { } 1 ∑ 𝑠 𝑆 value of the decision with new information and a ‘‘prior’’ value without − max 𝑢 (𝑎) (5) the information, as in the second part of Eq. (4) (Eidsvik et al., 2015). If 𝑎=1...𝐴 𝑆 𝑠=1 4 F. Haag et al. Environmental Modelling and Software 158 (2022) 105552 We propose that the same algorithm is efficient for threshold sen- 3.1.2. Societal perspectives and objectives sitivity analysis that takes into account conditional distributions. The Different users and interest groups hold varied perspectives on a reef aim here is to estimate the EU of alternatives given that a variable of site, its value, and its relevant services. Societal evaluation ultimately interest Y∗ takes on different values and then identify threshold values determines which form of management is optimal. Therefore, we need of Y∗ where the optimal alternative changes. This could be achieved to consider different stakeholder perspectives on the issue. For the reef by varying Y∗ in a one-factor-at-a-time sensitivity analysis. However, of the inhabited island we investigate, we seek to represent a diversity to take a regional view on sensitivity and account for dependencies of views by exploring four archetypal perspectives: across attributes and alternatives, we need to consider the conditional distributions of all variables given the specific values of the variable of • local livelihoods perspective, focused on ensuring food security interest. and economic benefit to local fishers • reef conservation perspective, focused on ecosystem health and To understand the relationship between specific values of Y∗ and the resilience EU of the alternatives, we can employ the algorithm described above. • extraction perspective, focused on maximizing fishing yield and The relationship can be approximated by calculating the arithmetic economic benefits mean of the variable of interest in each bin and the corresponding • balanced perspective, focused on balancing the different aspects EU for each alternative in this bin. Thresholds can then be determined and interests visually from a scatterplot or by calculating a threshold criterion. As it is sample-based, we may not be able to identify one threshold, but The values that are implicit in these perspectives can be expressed in rather a range of Y∗ as a threshold region. the form of an objectives hierarchy (Keeney, 1992) that we developed based on literature (Maynard et al., 2017; Brown et al., 2018; McField and Kramer, 2007) as well as our knowledge of the case and context 3. Implementation for a reef management problem (Fig. 2). We chose these four archetypal perspectives to illustrate a wide range of views on the management issue, but they are not meant 3.1. Case study description to represent actual individual stakeholders or groups. For real-world decision support, these perspectives and their objectives need to be elicited and co-developed locally. We use these perspectives to guide 3.1.1. Problem background an analysis for multiple stakeholder perspectives, which can be adapted In this case study, we investigated the local reef management for an to a particular decision context. island that can be considered as typical in the Spermonde archipelago in Indonesia, using the framework described in Section 2. The Sper- 3.1.3. Management alternatives, attributes, and time scale monde archipelago is a complex of about 70 islands located off South- At the investigated reef site, fishing occurs for subsistence and west Sulawesi, most of them inhabited and surrounded by coral reefs. commercial reasons and destructive fishing techniques are employed The region lies in the center of the Coral Triangle, which is the most occasionally (see description of the no restrictions alternative in Ta- biodiverse marine region worldwide (Burke et al., 2012). As for many ble SI-1). To define management alternatives, we first developed a small islands and island nations in the Indo-Pacific, it is necessary strategy generation table (Gregory et al., 2012). We identified three to find a balance between the exploitation and conservation of their management factors: (1) gear and technique restrictions, (2) access natural resources in the face of local and global changes: from liveli- restrictions, and (3) fishing quotas. Based on the strategy generation hoods and ecosystem degradation to climate change and globalized table, we developed four management alternatives expected to result economics. in decreasing degrees of fishing pressure (see Table 1): Major local stressors on the reefs in the area are overfishing and the use of destructive fishing techniques. Fish are consumed for nu- • no restrictions: no restrictions with continued intense fishing pres- trition locally, but mostly sold, partly as live fish (Radjawali, 2012). sure including destructive fishing Target species are diverse and shift with global demands and local • no destructive fishing : enforcing a ban of bomb and cyanide fishing supply (Ferse et al., 2014). A wide range of fishing techniques are • MPA subsistence: implementing a marine protected area, which employed, among them destructive techniques such as bomb fishing or allows only subsistence fishing for locals cyanide fishing. For fisherfolk of the islands, alternative livelihoods are • MPA no-take: implementing a strict marine protected area, mak- often neither attainable nor desired (Ferse et al., 2014). Additionally, ing it a no-take zone fisherfolk are often embedded in elaborate systems of patron–client relationships that provide benefits such as social protection (Glaser The management alternatives, if implemented, will have conse- quences for very different aspects of the socio-ecological system of the et al., 2015). These relationships can influence fishing behavior and island reef. The consequences that are relevant for deciding between may encourage overfishing and destructive fishing (Glaser et al., 2015; the management alternatives are captured by attributes of the system Miñarro et al., 2016). (Fig. 2 and Table SI-2). The attributes should describe the consequences The effects of fishing pressure are exacerbated by pollution from in a way that is understandable and useful for decision-makers and point and diffuse sources as well as sedimentary run-off stemming from stakeholders (Keeney and Gregory, 2005). The degree of fulfillment of the islands and the Makassar urban area (Teichberg et al., 2018). Larger the decision objectives can then be quantified based on the attribute scale pressures such as global climate change are expected to cause levels. For instance, the attribute ‘‘total biomass of browsers, scrapers, ocean warming and sea-level rise, resulting in the decline of reef health. and grazers in g∕m2 ’’ can be used to determine the achievement of the For this study, we investigated an exemplary reef site of a typical objective of having a high biomass of herbivorous fish in the reef. island in the region. The site is 200 × 160 m and part of a larger reef The considered time scale of consequences can make a large differ- area. The majority of the seafloor is less than 5 m deep, but descends ence in decision-making. Short-term and long-term consequences can to 15 m depth (Fig. SI-1). To improve the local situation, we focus on diverge. For instance, an over-exploitation of fishing resources can be fisheries management as a way to mitigate a primary local stressor for advantageous in the short term, but detrimental in the long term. In the reef. this study, we consider consequences three to six years in the future. 5 F. Haag et al. Environmental Modelling and Software 158 (2022) 105552 Fig. 2. Scheme of predicted model outputs (boxes on the left), their transformation/aggregation to attributes that measure decision-relevant consequences of management alternatives (middle, also see Table SI-2), and the hierarchy of objectives (right side) used in evaluating these alternatives based on the attribute predictions. The objective hierarchy is also the structure of the preference model based on utility theory that is used for evaluation. Consequences that are measured by the attributes are mapped to an aggregated utility. 3.2. Prediction of management consequences (e.g., number of boats). Four types of fisheries were considered: (1) bomb fishing, (2) cyanide fishing, (3) non-destructive commercial fish- Predicting the consequences of management alternatives on com- ing, and (4) non-destructive subsistence fishing (see Table SI-2 for their plex systems is a difficult task. To predict the ecological consequences specification in SEAMANCORE). The different management alternatives of management alternatives on reef fishes, reef benthos, and fisheries allowed specific combinations of these types of fishing to occur in the yield, we adapted a previously developed model of the system (SEA- reef, from all types in the no restrictions alternative to none in the MANCORE; Miñarro et al., 2018). This model combines an agent-based no-take alternative (Table 1). model of fish stocks and fishing behavior with a corresponding model For each simulation run, we specified the initial conditions in a for benthic community dynamics. The implementation is detailed in the spatially explicit manner: the water depth (Fig. SI-1), the benthic cover, next sections. To account for the often neglected cost of implementing biomass of fish functional groups, and the fishery modes to apply and enforcing management alternatives (McCrea-Strub et al., 2011), (Table 1 and Table SI-3). Additionally, we set over 100 other core we used the ‘‘required patrol days’’ as a proxy attribute for these parameters that govern the model behavior (Miñarro et al., 2018). For costs (Brown et al., 2018). We used our contextual knowledge to each simulated time step, the model outputs a 10 × 10 cm resolution estimate this attribute’s distribution for each management alternative. map of benthic cover, a 20 × 20 m resolution map of biomass of fish These were assumed to be Poisson distributions with different rate functional groups, and the total fishery yield differentiated by depth, parameters. target species, and fishery type. We simulated at a temporal resolution of 1 day for both the benthic and the fish grids. 3.2.1. Model of reef and fishery dynamics under management alternatives The management alternatives were assessed against the background SEAMANCORE is a spatially-explicit 2D model that simulates the of an uncertain environment in which most aspects are beyond man- dynamics of the benthic and fish populations as well as fisheries in a agement control. For instance, biological processes, such as death coral reef through an agent-based simulation (Miñarro et al., 2018). rates or feeding rates, were assumed to be independent of manage- We used SEAMANCORE to predict the temporal dynamics of the reef ment alternatives at the considered scale. However, these processes benthos and fish community in a 200 × 160 m area. The predicted enti- are usually variable and our knowledge about them is incomplete. We ties consisted of four benthic functional groups (hard coral, macroalgae represented this with two approaches. Firstly, the rules governing the and turf, hard substrate and cropped algae, non-stabilized substrate), stocks of three functional fish groups (carnivores, browsers and grazers, cellular automaton and model agents have a stochastic element. For scrapers), and different fisheries (Table SI-3). example, rules to change a hard substrate cell to an algae cell are The aim for prediction is to capture the inherent stochasticity of triggered only with a certain probability at each time step. Secondly, the system as well as the directed effects of the management alter- we used probability distributions instead of point estimates for the core natives. We achieve this by differentiating between core parameters parameters. and processes of the SEAMANCORE model that are unaffected by We used the calibration of Miñarro et al. (2018) as a basis for the the alternatives and alternative-specific parameters and processes. If we model parameterization. We then defined probability distributions for intend to understand the effect of the alternatives, it is only mean- 37 of the models parameters (e.g., reproduction rates, feeding rates, ingful to compare the set of predictions where the core parameters time for colonization; Fig. SI-2) and 15 input variables (initial benthic are shared. One model parameterization of the natural processes can cover and initial fish biomass; Fig. SI-2). Other parameters, such as be viewed as one potential configuration of the world. The four man- the probability of transition between benthic substrata, remained the agement alternatives then lead to four different futures. Therefore, same across all simulations. Parameter distributions were identified for one core parameterization, we created four simulations with the based on literature values and judgment by the authors, guided by alternative-specific parameters set in addition. natural constraints (e.g., a death rate cannot be negative), plausibility The difference between the four management alternatives was rep- of the simulation results, and using maximum entropy distributions. resented by modified inputs and parameters in the fishing module Distributions of model parameters were assumed to be independent, 6 F. Haag et al. Environmental Modelling and Software 158 (2022) 105552 Table 1 Fishing types allowed under the management alternatives. Name of alternative Subsistence fishing Commercial fishing Dynamite fishing Cyanide fishing (1) No restrictions Yes Yes Yes Yes (2) No destructive fishing Yes Yes No No (3) MPA, subsistence Yes No No No (4) MPA, no take zone No No No No except for initial benthic coverages which were assumed to come coral cover will also inform us about herbivore biomass. For an esti- from a multivariate normal distribution. We assumed all parameters mation approach of VoI that is based on sampling, we therefore need to be constant across the modeled temporal and spatial dimensions. By to create a sample that retains the relevant (conditional) dependencies propagating this prior information through the model with Monte-Carlo in the predictions. Given the way we set up our simulations, we simulation, we received a corresponding distribution of model outputs. obtain these dependencies from the SEAMANCORE modeling. For other An external stressor we modeled explicitly were coral bleaching modeling approaches this can be less straightforward. events (Eddy et al., 2021). We included strong bleaching that turn coral Such dependencies exist within the predictions from a simulation cells into hard substrate and mild bleaching that resets the age of coral run for one alternative. Firstly, the parts of the ecological system are cells. Bleaching affects coral cells with a certain probability depending connected. For example, high carnivore biomass will often coincide on the depth of their location. The effects of other external stressors with lower biomass of herbivorous fish. Secondly, predictions are cor- – such as temperature change, eutrophication, or loss of connectivity related in time, as the future system state depends on previous time to other reefs – are captured indirectly by the distributions of the steps. In the resampling of predictions for a management alternative, parameters — such as colonization rates of coral and algae, growth we retained relations between samples for different attributes regarding rates of functional groups, or external recruitment. points in time and simulation parameterization. This means for one alternative a particular sample of all attributes comes from the same 3.2.2. Obtaining of decision-relevant predictions simulation run and time point. Dependencies also exist between the consequences of different alter- To predict the uncertain consequences of management alternatives, natives because they are predicted based on shared core parameters, we used Monte-Carlo simulations to obtain a population of predictions as described above. Across the alternatives, we retained relations re- from independent runs of the SEAMANCORE model. We drew 1100 garding the core parameter samples. This represents system properties samples from the probability distributions of the model core parameters or shared external influences that are the same for all alternatives. (Fig. SI-2). These were combined with the definitions of each alter- However, we randomized the resulting predictions regarding time. In native. The SEAMANCORE model was thus run for 1100 samples×4 this way, we represent different time points at which the effects of the alternatives = 4800 times in total. For five of the 1100 parameter alternatives are assessed in the future. This means a particular attribute combinations the model produced nonsensical results: either the model sample across the alternatives comes from a model run with the same returned an undefined value for biomass of a fish functional group or core parameters, but potentially different time points. We illustrate the scraper biomass was continuously above a threshold of 95 g∕m2 for effect of retaining different correlations in Fig. SI-7. more than 60 days. These runs were excluded. As we focus on the situation three to six years after implementing a 3.3. Evaluation of management alternatives management alternative, we ran the model for 2280 time steps (days) and then took the values for years four to six into the future as the 3.3.1. Preference model structure and parameters basis for the attribute predictions. To smooth out short-term, noisy To understand the differences in utility that the management al- fluctuations due to asynchronous updating of fish and benthos grids, ternatives would bring to different societal actors, we specified a hi- we used a 42-day (6 weeks) rolling mean of the time series for the erarchical utility model for each of the four archetypal stakeholder benthos cover and a 14-day (2 weeks) rolling mean for the time series perspectives. Each model encodes an assumed preference profile for a of fish biomass. stakeholder perspective and is meant to represent specific interests for The system model outputs are not directly of interest as attributes in that stakeholder. Based on the evaluations of management alternatives the decision. We therefore transformed and aggregated model outputs with these models, we can then identify areas of conflicts and consen- to obtain predictions for attributes that would be understandable to sus. As outlined in Section 2.1, we can also calculate aggregate results interested stakeholders (Fig. 2 and Table SI-2). For instance, the daily across stakeholder perspectives. time series of fish biomass for the three functional groups was aggre- In the following, we outline our approach to hierarchical utility gated to arrive at weekly average total fish biomass. The variability models, a detailed treatment can be found in Haag et al. (2019a) in time thus became part of the prediction uncertainty. From the and Reichert et al. (2015). The structure of the preference models perspective of strategic management, the spatially explicit output of is given by a hierarchy of objectives (Fig. 2). We assume that all the model was also not relevant. Therefore, we aggregated the model stakeholder perspectives share the same set of objectives but differ in outputs for the entire area. their preferences; for instance, the trade-offs they are willing to make The empirical joint distribution of the aggregated and transformed between the objectives. model outputs was the basis for the decision-relevant attribute predic- To build the model, we first specify a marginal value function for tions. We aimed for a sample of size 𝑆 = 120 000 for each attribute of each of the seven objectives on the lowest level of the hierarchy, 𝑣𝑜𝑝 (𝑦𝑝 ) each alternative. To ensure equal sample size, attributes measured on a (Fig. SI-3). These map from the attribute space to a relative degree weekly scale were downsampled (from 170 820 samples) and attributes of achievement for each of these objectives. Then, we aggregate these with monthly scale were upsampled with replacement (from 42 705 valuations along the hierarchy with nested aggregation functions, 𝐹𝑘 , samples). For the attribute ‘‘required patrol days’’, we directly drew 𝑆 that we specify for each aggregation step. We arrive at a multi-attribute samples from the specified Poisson distributions. value function over all attributes, 𝑣(𝑦1 , … , 𝑦𝑚 ). With this function we A crucial consideration when creating this sample of attribute pre- receive an overall evaluation of each decision alternative. Lastly, as dis- dictions is that dependencies exist across attributes and alternatives. cussed by Dyer and Sarin (1982), this multi-attribute value function is This should be considered in VoI analysis. For instance, if coral cover converted to a utility function 𝑢(𝑣(𝑦1 , … , 𝑦𝑚 ), 𝑟) at the highest objective correlates with herbivore biomass, having better information about in the hierarchy, given the risk attitude 𝑟. As we assume stakeholders 7 F. Haag et al. Environmental Modelling and Software 158 (2022) 105552 to be risk neutral, the EU of an alternative is its expected value based The aim was to understand the impact of better knowledge of the on the evaluation with the value function. The preference model used predictions of specific attributes. To identify whether the uncertainty to evaluate one set of consequences of alternative 𝑎 can therefore be of an attribute prediction was more relevant for some alternatives than written as (for the indices refer to Fig. 2): others, we focused on individual variables for each of the alternatives 𝑢(𝒚 𝑎 ) =𝐹0 ( and calculated the EVPPI of 4 alternatives for 7 attributes, resulting in 28 variables of interest. That is, if we had perfect knowledge of some 𝐹1 (𝑣𝑜1,1 (𝑦1,𝑎 , 𝜽𝑜1,1 ), 𝑣𝑜1,2 (𝑦2,𝑎 , 𝜽𝑜1,2 ), 𝒘𝐹1 , 𝛾𝐹1 ), attribute’s predictions Y𝑖 (𝑎) for alternative 𝑎, how much additional 𝑣𝑜2 (𝑦3,𝑎 , 𝜽𝑜2 ), utility would this be expected to provide to a stakeholder? 𝑣𝑜3 (𝑦4,𝑎 , 𝜽𝑜3 ), (6) To estimate EVPPI, we implemented the algorithm as described in 𝐹4 (𝑣𝑜4,1 (𝑦5,𝑎 , 𝜽𝑜4,1 ), 𝑣𝑜4,2 (𝑦6,𝑎 , 𝜽𝑜4,2 ), 𝒘𝐹4 , 𝛾𝐹4 ), Section 2.4 and ran the analysis separately for each of the 28 variables of interest and each stakeholder perspective. We also calculated the 𝑣𝑜5 (𝑦7,𝑎 , 𝜽𝑜5 ), EVPPI across perspectives based on averaging the EUs of the perspec- 𝒘𝐹0 , 𝛾𝐹0 ) tives in each bin and on the EU of the baseline optimal alternative As aggregation functions 𝐹𝑘 for the values 𝑣 on each hierarchical across the perspectives. For the main analysis, we chose values of 𝐾 = level we chose functions of the family of weighted generalized means 300 and 𝐽 = 400 to mitigate the chance of bias. We also investigated (also called power means) with the form: the dependency of EVPPI on these choices (Fig. 7, Fig. SI-9). ( 𝑛 )1∕𝛾 To understand the added value of EVPPI over a simpler threshold ∑ 𝛾 view on sensitivity (see Section 2.2), we conducted a threshold sensi- 𝐹𝑘 (𝑣1 , … , 𝑣𝑛 , 𝒘, 𝛾) = 𝑤 𝑖 ⋅ 𝑣𝑖 , 𝛾 ∈ R∗ (7) 𝑖=1 tivity analysis. This means, we estimated the EU of alternatives given ∑ that a variable of interest Y∗ takes on different values as described in with weight parameters (scaling factors) 0 < 𝑤 < 1 and 𝑛𝑖=1 𝑤𝑖 = 1. Section 2.4. We repeated this analysis for all 28 variables of interest The parameters of the preference model (Eq. (6)) were changed for and each stakeholder. To identify thresholds visually, we normalized each of the stakeholder perspectives based on assumed preferences in line with their concerns (see Section 3.1.2). These parameters are: the EU values to the baseline optimal alternative for a stakeholder and estimated a smoothing spline model for this relationship. • how they evaluate changes on the attribute scales, such as di- minishing returns regarding fish catch or coral cover (shapes of marginal value functions, 𝜽, Fig. SI-3). 4. Results • how they trade off changes in one objective relative to the other objectives (weight parameters, 𝒘, Table SI-3) 4.1. Predicted consequences of reef fishery management • to what degree a poor achievement of objectives can be compen- sated. For example, can a high enough fished biomass compen- The inputs and parameters of the predictive system model were sate for a very low coral cover, or is a ‘‘one out all out’’ view described by probability distributions. Together with the stochastic appropriate (degree of non-additivity, 𝜸, Table SI-3). processes in the model, this led to distributions for the obtained outputs As the aim in this study was to explore the space of potential and, consequently, for the derived predictions of the relevant attributes perspectives, the preference profiles were designed by the authors ac- (Fig. 3). If we would only consider point predictions in our decision cording to the archetypes (see 3.1.2). For a practical decision problem, making (e.g., median lines in Fig. 3), we would disregard a lot of the parameters should be inferred from stakeholder data. These data relevant information. can be collected by choice experiments (Hensher et al., 2015) or other The distributions of attributes that describe the state of the reef – forms of preference elicitation (Haag et al., 2019a). coral cover, herbivore biomass, and total fish biomass – are wide. This means, the predictive uncertainty about their future is high. The differ- 3.3.2. Estimation of expected utility ences between alternatives appear less pronounced when considering Based on the empirical distributions of attribute predictions, we first the marginal distributions. This suggests the alternative had relatively calculated the optimal baseline choice. Each stakeholder preference less effect on the outcome compared to the stochasticity of the system. profile was treated separately. For each sample 𝑦 = 𝑦1 , … , 𝑦7 of the As expected, decreased fishing pressure generally leads to increased 7 attributes, we calculated the utility of each alternative using the fish biomass, especially of carnivores. Increases in herbivore biomass respective preference model. Since we have 𝑆 = 120 000 samples, we are smaller due to the increasing predation pressure from carnivores. obtained 120 000 utilities, the set of which we denote 𝑈𝑎 for alternative Coral cover increases when destructive fishing is stopped, but decreases 𝑎. By taking the arithmetic mean of U𝑎 we received the EU for each slightly when fishing is stopped completely as feeding pressure by alternative assuming risk neutrality: E[U𝑎 ]. The rational imperative is { } scrapers increases. to pick the alternative with the highest EU: max𝑎=1…4 E[U𝑎 ] . This is The distributions of attributes connected with fishery yield – car- the optimal baseline choice for a stakeholder preference profiles, given nivorous and herbivorous biomass to sell, fish for local consumption our current information about the system attributes. Based on the evaluation of management alternatives with the four – exhibit long tails (Fig. 3). With the high fishing pressure in the no preference profiles associated with the stakeholder perspectives, we restrictions alternative, few carnivorous fish can be sold due to over can identify areas of conflicts and consensus. For also providing an exploitation and hence stock depletion in the considered 3–6 year time aggregated view across the stakeholder perspectives, we followed the frame. The carnivorous fishing yield is higher in total with less intense approach outlined in Section 2.1. Using Eq. (3), we calculated the EU of fishing. On the other hand, when only subsistence fishing occurs there an alternative across perspectives giving all perspectives equal weight. is little surplus of (especially herbivorous) fish to be sold. With the high fishing pressure of the no restrictions alternative, it is more likely that 3.4. Sensitivity analysis using value of information not enough fish can be caught for local consumption in comparison to the alternatives with no destructive fishing or with a protected area Value of information analysis can be conducted for any uncertain in- that allows subsistence fishing. With a strict no-take zone, no fishing is puts or parameters of a decision model. For this case study, we focused assumed to occur. Therefore the attributes related to fishery yield are on exploring the EVPPI with regard to uncertain attribute predictions. always zero. 8 F. Haag et al. Environmental Modelling and Software 158 (2022) 105552 Fig. 3. Marginal distributions and boxplots of predictions (y-axis) of the attributes (panels) of a coral reef area under four management alternatives (x-axis). Distributions per alternative are based on 120 000 samples derived from 1100 independent simulations of the SEAMANCORE reef model, which were transformed, aggregated, and resampled. The predictions cover the time 3 to 6 years after the management alternatives were activated in the model. Required patrol days were directly sampled from Poisson distributions. Boxplots show the 0.25, 0.5, and 0.75 quartiles of these data, and whiskers extend to the maximum and minimum points within 1.5 times the interquartile range. Only a proportion of outliers is visualized. 4.2. Optimal baseline choice under uncertainty subsistence fishing, and for the extraction profile the alternative with no restrictions would be most desirable (Fig. 4). Except for the con- Given the uncertain predictions of the attributes and our preference servation profile, a strict MPA with a no-take zone receives the lowest EU in all profiles; for the conservation profile the no restrictions al- models for the different perspectives (Section 3.3), we calculated the ternative results in sightly lower EU. This can be explained by the utility for each predicted sample of the alternatives (distributions in missing fulfillment of any socio-economic objectives by an alternative Fig. 4). The expectation over these utilities, the expected utility (EU), that enforces a no-take zone. The alternative with no restrictions is is the criterion that a rational decision should be based on. This EU not optimal for most preference profiles. Even a moderate restriction integrates over the predictive uncertainties and is therefore a single of fisheries can lead to higher fish biomass and also fished biomass, number (solid markers in Fig. 4). especially of carnivorous fish, even in the short time frame studied. For the balance and local livelihoods preference profile, a ban Based on these results, no clear consensus for a best management of destructive fishing practices would be the optimal alternative, for alternative emerges between the different perspectives. However, we the conservation profile a marine protected area (MPA) with only can identify two aspects that might help come to such a consensus in 9 F. Haag et al. Environmental Modelling and Software 158 (2022) 105552 Fig. 4. Distribution of utilities (y-axis) and the EU (solid markers) for the management alternatives (x-axis). Each stakeholder preference profile (panel) is represented by a different utility function which maps the consequences of the alternatives into a utility between 0 and 1. Utilities are relative and allow ranking of alternatives within a perspective. an iterative process: (1) some reduction in fishing seems beneficial for consequences on its other attributes or it coincides with even more fishery yield even in a short time horizon and (2) the lack of fulfillment preferred consequences for the no-take MPA alternative. of socio-economic objectives due to a complete ban of fisheries can The analysis falls short in two regards. First, we do not take into hardly be counterbalanced by better conservation outcomes. account how probable a crossing of a threshold would be: how probable would it be that we actually see coral cover greater than 62% under 4.3. Threshold view on decision sensitivity the MPA with subsistence fishing alternative? Second, once we crossed a threshold, we disregard how large the potential gain in utility would There are large overlaps in the distributions of the alternatives’ be from taking the optimal instead of the now sub-optimal alternative: utilities (Fig. 4). This means, we might see future system states in which if coral cover was above 62%, how much higher would the utility of different conclusions about the optimal alternative would be drawn. deciding for the no-take MPA alternative be in comparison to sticking Therefore, we need to investigate the sensitivity of the decision. Before with the MPA with subsistence fishing? Both aspects are crucial for considering the VoI view of sensitivity analysis, we identify thresholds understanding the sensitivity of a decision. This is the point of the for individual variables of interest – here the attribute predictions for analysis of the VoI. one alternative – that lead to changes in the optimal alternative. In Fig. 5 we see the resulting EU of the management alternatives 4.4. Results of the value of information analysis relative to the baseline optimal alternative as we vary a variable of interest along its range. This relative EU is given as a function of To have a more comprehensive measure of decision sensitivity than a variable of interest, while retaining a probabilistic view and the the threshold view, we calculated the EVPPI of the variables of interest. correlation structure in all other variables. The lower the EVPPI of a variable is, the lower the sensitivity of the Based on this we can identify thresholds at which the ordering decision to it and vice versa. If we had perfect information about that of the decision alternatives changes. Since our results are based on variable, this either would seldom change the optimal alternative, the simulations, the thresholds are small regions rather than exact points. gain in utility due to choosing the new optimal alternative would be As an example, the baseline optimal alternative for the conversation small, or both. A ranking of the variables of interest based on their perspective is the MPA with subsistence fishing. We can now investigate expected VoI can then support us in identifying the key uncertainties the decision’s sensitivity to the predicted coral cover of the MPA with and prioritizing their resolution. subsistence fishing alternative (lower left panel of Fig. 5; results for all The EVPPI varies by variable of interest and stakeholder preference variables and perspectives are given in Figs. SI-5–8). profile (Fig. 6, Table SI-5). Comparing all stakeholder perspectives, There are two thresholds. If we knew the coral cover of the MPA two commonalities exist. The attribute of required patrol days had with subsistence fishing alternative would turn out to be below 23%, relatively low EVPPI, while the attribute regarding fish available for the no restrictions or no destructive fishing alternatives would now local consumption had high EVPPI. Otherwise the results are more nu- provide higher utility. If we could be certain that the coral cover of the anced. Across variables, the livelihoods perspective often receives lower MPA with subsistence fishing alternative would be between 23% and EVPPI than the other perspectives; the decision is less sensitive for this 62%, it would be the optimal choice. In this region, the VoI is zero, perspective. This demonstrates how the VoI depends on the stakeholder as the best alternative is not sensitive to the precise value of the coral preference models and how far away (in terms of probability of change) cover prediction. the baseline optimal alternative is from the others (see Fig. 4). If the coral cover of the MPA with subsistence fishing alternative The EVPPI is not directly linked to the width of the probability would be higher than 62%, the no-take MPA would be optimal. This distributions of the variables (Fig. 3). The herbivore biomass of the may seem counter intuitive as, all else being equal, the utility of no restrictions alternative has a markedly narrower distribution than the MPA with subsistence fishing alternative should increase with the total fish biomass of this alternative. Yet, for all profiles as well as increasing coral cover as higher cover is preferred. It is, however, across profiles both have a similar EVPPI. As expected, for variables a consequence of the correlation structure in the predictions. Either that are known with certainty, for instance, fisheries variables in the the high coral cover for that alternative coincides with less preferred no-take MPA alternative, the EVPPI is zero. 10 F. Haag et al. Environmental Modelling and Software 158 (2022) 105552 Fig. 5. Visualization of a threshold view on sensitivity for the conservation perspective. Each panel shows the variation of expected utility (EU) of alternatives (y-axis) when one variable of interest, meaning one attribute of one alternative, takes on different values (x-axis) while keeping the conditional variability in all other attributes. For clearer presentation, the EU of alternatives is divided by the EU of the baseline optimal alternative; thus, the y-axis shows ratios relative to this alternative. The lines are a loess smoothing of these data points (dots). The alternative with highest relative EU is the optimal alternative given that the variable of interest takes on a specific value on the x-axis. At the threshold values of this variable the optimal alternative changes (dotted vertical lines). In regions between thresholds where the EU of the baseline optimal alternative is higher, the value of information (VoI) is zero, as the best alternative is not sensitive to the precise value of the variable of interest in that range. A subset of the results for the conservation stakeholder preference profile is depicted. Figs. SI-5–8 show the other results. Fig. 6. Expected value of partially perfect information (EVPPI; x-axis) of uncertain attribute predictions of interest (y-axis) differentiated by alternative for which they were predicted (colors). Results for different stakeholder perspectives are shown in panels, with the leftmost shaded panel showing the result of an aggregated view across the perspectives. A single bar represents the expected gain in units of utility for a particular preference profile if we had perfect information about the prediction of the attribute of interest of a specific alternative. For variables that are known with certainty, such as fish catch under a no-take zone, the VoI is zero by definition. 11 F. Haag et al. Environmental Modelling and Software 158 (2022) 105552 For the conservation perspective, we can summarize our analysis as follows (for brevity, we do not discuss the other perspectives here, but they can be similarly analyzed). The optimal baseline choice given our current state of knowledge would be implementing an MPA with subsistence fishing (Fig. 4). However, this choice is sensitive to the actual realizations of the predicted attributes. Thresholds exist that would make a different choice optimal (Fig. 5 and Fig. SI-6). We expect the optimal choice to be sensitive to the days with food for local consumption, herbivore biomass in the reef, and total fish biomass (Fig. 6). We expect it to be insensitive to the number of patrol days and the actual fished biomass for selling. For any variable, except the coral cover, better knowledge about the true consequences when implementing the no-take MPA alternative is hardly relevant for the decision (green bars in Fig. 6). This alternative is unlikely to become the best one for the conservation perspective. Based on the EVPPI analysis, the conclusion for the conservation preference profile would be that understanding the trajectory of the reef and its organisms better should be a priority. However, the uncertainty about the fish for local consumption is also relevant. Further efforts di- rected at improved understanding of these aspects will be most critical for decision-making as the determined baseline choice may not be the best if our knowledge about the respective attributes was improved. On the other hand, further investigation of the patrolling effort or the sold fish biomass is unlikely to change the conclusions regarding the optimal management alternative. For the other preference profiles, the Fig. 7. Dependence of the EVPPI (y-axis) on the bin size parameter 𝐽 (x-axis) of the list of priorities differs, with some commonalities as described above. estimation algorithm. The variables of interest are the attribute predictions (colors) for the no destructive fishing alternative. Only results for the conservation perspective are The estimation algorithm for EVPPI that we propose in Section 2.4 shown. The upper dotted line indicates the expected value of perfect information, the has a hyper-parameter, the size of the bins, 𝐽 . The choice of 𝐽 can lower dotted line is at zero, the minimum possible EVPPI. Results for other perspectives have a significant effect on the resulting estimate (Fig. 7 and Fig. SI- and variables of interest are given in Figure SI 9. For our case study we used a bin 9). For small bin sizes the estimator is upwardly biased due to the size of 400. maximization step. As 𝐽 → 1 the estimates converge to the expected value of perfect information across all variables (EVPI). If each sample is placed in its own bin, i.e., 𝐽 = 1, the estimated EVPPI is equal to the a wide distribution for each management alternatives, having perfect EVPI. The estimated EVPPI converges to zero as 𝐽 → 𝑆, as both terms information on it did not have high informational value. Consequently, of Eq. (5) become equal. Thus, for large sizes of the bins, the estimator to improve decision robustness, it is not always the largest uncertain- is downwardly biased. ties that require addressing. Rather, it is the most decision-relevant In our case, most EVPPI estimates are relatively stable, using from uncertainties, which can be identified through the analysis of VoI. 100 bins with 1200 samples each to 12 000 bins with 10 samples each This is in line with several studies that have investigated the factors (Fig. 7). This confirms our choice of a bin size of 400 for the analyses that influence the VoI in a decision, but found it can vary in unexpected above. However, in specific cases, estimates can also be sensitive to the and sometimes counter intuitive ways (e.g., Eeckhoudt and Godfroid, bin size (e.g., days with fish for local consumption in Fig. 7 or Fig. SI-9). 2000; Gould, 1974). Dependencies between alternatives can be one influencing factor: while we may be very uncertain about the future 5. Discussion coral cover, we may be quite certain that one alternative results in higher coral cover than another (cf. Reichert and Borsuk, 2005). 5.1. Relevance of value of information analysis for the case study Delquié (2008) has shown that under quite general assumptions the VoI is highest when a stakeholder’s baseline choice is indifferent In the case study on coral reef fisheries management, we found that between two alternatives. The VoI decreases with increasing utility the attributes measuring consequences related to fish biomass, food difference between the alternatives in the baseline case. Our results security, and sales of fish groups had the highest EVPPI for at least show the same pattern, as for the livelihoods and extraction preference one stakeholder preference profile. On the other hand, better estimates profiles, which have a larger spread among the utility of alternatives for the required patrolling effort had low EVPPI for any single and (Fig. 4), the EVPPI is generally lower than for the other two preference across stakeholder perspectives. Likewise, hard coral cover, which is profiles (Fig. 6). routinely monitored and prominently reported, was not among the top Value of Information is specific to the investigated decision. If the three variables in terms of EVPPI for any perspective. informational value of a variable, e.g., coral cover, is low in a partic- The analysis provided a reasoned and prescriptive focus for the ular decision this does not imply we should stop regular monitoring. design of future investigations for the decision case. The effect of the Historical baselines and operating protocols remain important and can management alternatives on the reef fish, their catch, and livelihood be of great value in another decision and for improved understanding impacts should be a focus of future data collection efforts to reduce of the complex system dynamics. decision uncertainty. However, this conclusion depends on the stake- holder perspective considered. For the conservation perspective, better 5.2. Evaluation and outlook of VoI framework knowledge about the amount of fish sold had low expected informa- tional value, whereas for the livelihoods perspective better knowledge This study showed how VoI analysis, and specifically the EVPPI, is about the biomass of fish functional groups in the reef was not very a useful form of sensitivity analysis for decision models. Based on the relevant (Fig. 6). approach in this study, we highlight three key directions for further Importantly, the EVPPI is not directly linked to the extent of un- development that we consider relevant in the context of environmental certainty in the predictions (Fig. 3). Even though the coral cover had management decisions. 12 F. Haag et al. Environmental Modelling and Software 158 (2022) 105552 The first direction is extending the approach to uncertainty of pref- to (a) move forward with implementation of the baseline optimal erence model parameters. Standard utility theory (e.g., French, 1986; alternative, (b) gather more information if we deem the decision too Keeney and Raiffa, 1993) does not consider uncertainty of preferences sensitive to potential new information, or (c) conduct implementation nor do any VoI applications we are aware of. However, in practice and information gathering in parallel. the stakeholder preferences are also uncertain and we have shown in In the environmental domain, adaptive management is a common this study that the preference model can make a substantial difference iterative approach and VoI analysis has been used to improve adap- for the VoI analysis results. The uncertainty of stakeholder preferences tive management decisions (e.g., Moore and Runge, 2012; Williams could be included in VoI analysis by using the expected expected utility et al., 2011; Runge et al., 2011). In adaptive management, we plan concept (Chajewska et al., 2000; Haag et al., 2019b). Considering the monitoring and data collection activities in parallel to implementing a uncertainty about the consequences of management alternatives and management alternative. There will be feedback processes in the system the uncertainty about the societal evaluation of these consequences after implementing a management alternative that we need to take into on equal footing in VoI analysis will allow us to differentiate better account in long-term management. Therefore, the idea of revisiting the where further studies are actually needed. Depending on the case, the same decision context and updating our state of knowledge is key. uncertainty about the social evaluation could be the primary cause for This can include (1) updating changed stakeholder perspectives, (2) decision uncertainty (Gregory et al., 2006). monitoring how the predicted trajectories have played out, (3) updating The second direction for development is improving uncertainty the future model predictions, and (4) coming up with the plan in quantification of the variables of interest (e.g., the predictions). Quanti- terms of what to focus on for the next monitoring phase. In modeling tative VoI analysis is only meaningful to the degree that we can specify approaches that optimize over iterative management problems, such or infer probability distributions for these variables, ideally derived as Markov decision processes, VoI analysis can also play an important from empirical sources. The question of a variable’s VoI is only ever role (Chadès et al., 2017; Williams et al., 2011; Williams and Johnson, addressed in the ‘‘small world’’ (Savage, 1954) of our specified model. 2018). However, the assessment of uncertainties in predictive system models Using VoI in environmental management and conservation practice is usually limited, especially regarding structure and dependencies. We is still at an early stage, though its value is increasingly recognized (see also disregarded crucial structural uncertainties in the predictive reef studies in Bolam et al., 2019; Keisler et al., 2014). Along structured model. Missing processes, such as changes in fish population structure decision-making approaches (Gregory et al., 2012), many opportunities might entail larger uncertainties than all the included parametric uncer- for broader application of VoI analysis beyond local management or tainty. More comprehensive uncertainty assessments are a large task for conservation decisions exist. For example, VoI analysis can be of inter- the environmental modeling community, but there are many advances est when designing large-scale research programs (Rushing et al., 2020) or monitoring programs (Bal et al., 2018). in this direction (e.g., Uusitalo et al., 2015; Reichert, 2020). The long-term benefits of investing in VoI analysis include a well- A way to address large uncertainties that are difficult to quantify reasoned allocation of resources as it provides a ranked list of the – examples for reef systems are crown-of-thorns starfish outbreaks or expected benefit of addressing uncertainties. For VoI measures that powerful storms – are scenarios (e.g., Walker et al., 2003; Wright are based on utility, this relative benefit can be difficult to interpret. et al., 2019). Scenarios can be modeled by repeating the analysis for If costs and benefits of additional information are not measured in different possible futures – in the form of constraints or modified the same units, we cannot directly determine which uncertainties we ecosystem processes – and qualitatively evaluating the differences in should resolve. Any practical decision about information collection the conclusions. This would be possible without fundamental changes requires making trade-offs with the (opportunity) cost of acquiring this to the presented approach. information (e.g., Maxwell et al., 2015). A straightforward approach The third direction for development is advancing methods to es- to this issue is a cost–benefit analysis to find the Pareto optimal set of timate EVPPI more efficiently. Especially with environmental models cost-efficient information seeking activities (Marchese et al., 2018). that often entail significant computational effort, simulation-based ap- proaches that rely on many model runs and resampling from condi- 6. Conclusions tional distributions, such a nested Markov-Chain-Monte-Carlo (Brennan et al., 2007; Felli and Hazen, 1998), are infeasible. The algorithm we Difficult environmental decisions can benefit from structured ap- implemented based on Strong and Oakley (2013) and Borgonovo et al. proaches (Gregory et al., 2012) and the conceptual foundation of (2016) is fast, only requires a given probabilistic sensitivity analysis rational decision making under uncertainty is well established (Keeney sample, and can handle conditional dependencies. On the other hand, it and Raiffa, 1993; Reichert et al., 2015). Understanding the sensitivity can only be sensibly used for investigating the EVPPI of single variables, of a decision to uncertainties remains a key challenge to support better and it still requires a large sample size. decision making. Our intuitions about the benefit of more information An important consideration when using the proposed algorithm is may not be correct, but the costs of additional data acquisition are often the choice of the bin size 𝐽 as this can lead to bias, as investigated in high. We propose that sensitivity analysis known under the umbrella Section 4.4. As Strong and Oakley (2013) have also shown before, the term value of information (VoI) analysis is useful to estimate the upward and downward biases appeared for extreme values of 𝐽 with a robustness of current conclusions and indicate where to focus future large region of stability in between these. However, in our case there data collection. were few estimates where such a stable region was small (Fig. SI-9). The complexities of environmental issues make the practical appli- The conditions under which the algorithm can reliably be used thus cation of VoI analysis challenging. In this study, we tackled VoI analysis require further investigation. in the framework of multi-attribute value/utility theory (MAVT/MAUT) with: (1) a continuous uncertain prediction space, (2) dependencies 5.3. Value of information analysis to support iterative environmental man- in the distribution of these predictions, (3) multidimensional objective agement functions that include trade-offs between objectives, and (4) divergent stakeholder perspectives. This included adapting a fast algorithm for VoI analysis is most useful if we can take actions to address the key estimation based on a probabilistic sensitivity analysis sample. We uncertainties and improve the information state before or after decision analyzed expected value of partially perfect information (EVPPI) for making. Thus, it fits well with decision contexts that have an iterative a decision model of local coral reef fishery management. This led to a aspect, such as the regular strategic considerations of environmental ranking of the sensitivity of predictive uncertainty of management al- monitoring programs. For the case study, we presented the first steps of ternatives. Our framework can be used as a template for other decision such an iterative approach. Based on the analysis results, we can decide cases. 13 F. Haag et al. Environmental Modelling and Software 158 (2022) 105552 Two simple, but practically relevant, conclusions were corroborated References in our case study. First, the EVPPI of a variable cannot directly be mapped to the extent of uncertainty in that variable. Large uncer- Bal, P., Tulloch, A.I.T., Chadès, I., Carwardine, J., McDonald-Madden, E., Rhodes, J.R., 2018. Quantifying the value of monitoring species in multi-species, multi-threat tainties in predictions do not prevent robust decision making per se systems. Methods Ecol. Evol. 9 (7), 1706–1717. http://dx.doi.org/10.1111/2041- (cf. Reichert and Borsuk, 2005). More data collection is not always the 210X.13037. answer. Second, the results of VoI analysis depend on the preference Bates, M., Sparrevik, M., de Lichy, N., Linkov, I., 2014. The value of information for managing contaminated sediments. Environ. Sci. Technol. 48 (16), 9478–9485. models used to evaluate the predicted consequences (cf. Delquié, 2008). http://dx.doi.org/10.1021/es500717t. The variables that stakeholders require more information on can differ. Bolam, F.C., Grainger, M., Mengersen, K., Stewart, G., Sutherland, W., Runge, M., If we disregard the variety in stakeholder perspectives, any VoI analysis McGowan, P., 2019. Using the value of information to improve conservation will give an incomplete picture of the actual value of a piece of decision making. Biol. Rev. 94 (2), 629–647. http://dx.doi.org/10.1111/brv.12471. Borgonovo, E., Hazen, G., Plischke, E., 2016. A common rationale for global sensitivity information in a specific context. measures and their estimation. Risk Anal. 36 (10), 1871–1895. http://dx.doi.org/ Value of information analysis fits into many structured and iterative 10.1111/risa.12555. approaches to decision-making and assessment, such as adaptive man- Brennan, A., Kharroubi, S., O’Hagan, A., Chilcott, J., 2007. Calculating partial expected value of perfect information via Monte Carlo sampling algorithms. Med. Decis. Mak. agement. It facilitates identifying and ranking (Fig. 6) key uncertainties 27 (4), 448–470. http://dx.doi.org/10.1177/0272989X07302555. and thus key aspects for further investigation and data collection. Brown, C., Parker, B., Ahmadia, G., Ardiwijaya, R., Purwanto, Game, E., 2018. The While the extent and intricacy of a quantitative modeling approach, as cost of enforcing a marine protected area to achieve ecological targets for the employed in this study, will need to be aligned with the concrete needs recovery of fish biomass. Biol. Cons. 227, 259–265. http://dx.doi.org/10.1016/j. biocon.2018.09.021. and resources available, any practical decision case can benefit from a Burke, L., Reytar, K., Spalding, M., Perry, A., 2012. Reefs at Risk Revisited in the Coral deliberation about the value of new information. Which information – Triangle. World Resources Institute, URL: https://www.wri.org/publication/reefs- if any – would likely change our conclusions? risk-revisited-coral-triangle. Chadès, I., Nicol, S., Rout, T., Péron, M., Dujardin, Y., Pichancourt, J., Hastings, A., Hauser, C., 2017. Optimization methods to solve adaptive management problems. Theor. Ecol. 10 (1), 1–20. http://dx.doi.org/10.1007/s12080-016-0313-0. CRediT authorship contribution statement Chajewska, U., Koller, D., Parr, R., 2000. Making rational decisions using adaptive utility elicitation. In: Proceedings of the Seventeenth National Conference on Arti- ficial Intelligence and Twelfth Conference on Innovative Applications of Artificial Fridolin Haag: Conceptualization, Methodology, Software, Formal Intelligence. AAAI Press, pp. 363–369. analysis, Writing – original draft, Review & editing, Visualization. Sara de Jonge, J., 2012. Rethinking Rational Choice Theory. a Companion on Rational and Miñarro: Methodology, Resources, Writing – review & editing. Arjun Moral Action. Palgrave Macmillan, URL: https://doi.org/10.1057/9780230355545. Chennu: Conceptualization, Methodology, Resources, Writing – review Delquié, P., 2008. The value of information and intensity of preference. Decis. Anal. 5 (3), 129–139. http://dx.doi.org/10.1287/deca.1080.0116. & editing, Visualization, Supervision, Funding acquisition. Dyer, J., Sarin, R., 1982. Relative risk-aversion. Manage. Sci. 28 (8), 875–886. http: //dx.doi.org/10.1287/mnsc.28.8.875. Eddy, T., Lam, V., Reygondeau, G., Cisneros-Montemayor, A., Greer, K., Palomares, M., Declaration of competing interest Bruno, J., Ota, Y., Cheung, W., 2021. Global decline in capacity of coral reefs to provide ecosystem services. One Earth 4 (9), 1278–1285. http://dx.doi.org/10. 1016/j.oneear.2021.08.016. The authors declare that they have no known competing finan- Eeckhoudt, L., Godfroid, P., 2000. Risk aversion and the value of information. J. Econ. cial interests or personal relationships that could have appeared to Educ. 31 (4), 382–388. http://dx.doi.org/10.1080/00220480009596456. Eidsvik, J., Mukerji, T., Bhattacharjya, D., 2015. Value of Information in the Earth influence the work reported in this paper. Sciences: Integrating Spatial Modeling and Decision Analysis. Cambridge University Press, Cambridge, http://dx.doi.org/10.1017/CBO9781139628785. Felli, J., Hazen, G., 1998. Sensitivity analysis and the expected value of per- Data and software availability fect information. Med. Decis. Mak. 18 (1), 95–109. http://dx.doi.org/10.1177/ 0272989x9801800117. Feltham, G.A., 1968. The value of information. Account. Rev. 43 (4), 684–696. The workflow to reproduce the analysis is available at https://d Ferse, S., Glaser, M., Neil, M., Schwerdtner Máñez, K., 2014. To cope or to sustain? oi.org/10.5281/zenodo.7156065. It depends on two submodules: the Eroding long-term sustainability in an Indonesian coral reef fishery. Reg. Environ. Change 14 (6), 2053–2065. http://dx.doi.org/10.1007/s10113-012-0342-1. SEAMANCORE simulation model (https://doi.org/10.5281/zenodo.71 French, S., 1986. Decision Theory: An Introduction To the Mathematics of Rationality. 55783) and an R package with functions for VoI analysis (https://doi. In: Ellis Horwood series in mathematics and its applications, Ellis Horwood; Halsted org/10.5281/zenodo.7155948). The dataset of simulation results from Press, Chichester, West Sussex, UK. SEAMANCORE used in the analysis is available at https://doi.org/10.5 Fumie, Y., Thompson, K., 2004. Value of information analysis in environmental health risk management decisions: Past, present, and future. Risk Anal. 24 (3), 635–650. 281/zenodo.7156015. http://dx.doi.org/10.1111/j.0272-4332.2004.00464.x. Glaser, M., Breckwoldt, A., Deswandi, R., Radjawali, I., Baitoningsih, W., Ferse, S., 2015. Of exploited reefs and fishers – A holistic view on participatory coastal Acknowledgments and marine management in an Indonesian archipelago. Ocean Coast. Manage. 116, 193–213. http://dx.doi.org/10.1016/j.ocecoaman.2015.07.022. Gould, J., 1974. Risk, stochastic preference, and the value of information. J. Econom. We thank Daniel Schürholz for coding advice. This project benefited Theory 8 (1), 64–84. http://dx.doi.org/10.1016/0022-0531(74)90006-4. greatly from the resources of the ZMT Datalab; we are thankful to Gregory, R., Failing, L., Harstone, M., Long, G., McDaniels, T., Ohlson, D., 2012. Structured Decision Making: a Practical Guide to Environmental Management Miroslav Shaltev and Joscha Schmiedt for support. We also thank Choices. Wiley-Blackwell, Chichester, West Sussex, UK. the three reviewers and the editor for their excellent comments and Gregory, R., Failing, L., Ohlson, D., Mcdaniels, T., 2006. Some pitfalls of an overem- suggestions. phasis on science in environmental risk management decisions. J. Risk Res. 9 (7), 717–735. http://dx.doi.org/10.1080/13669870600799895. Haag, F., Lienert, J., Schuwirth, N., Reichert, P., 2019a. Identifying non-additive multi- attribute value functions based on uncertain indifference statements. Omega 85, Appendix A. Supplementary data 49–67. http://dx.doi.org/10.1016/j.omega.2018.05.011. Haag, F., Reichert, P., Maurer, M., Lienert, J., 2019b. Integrating uncertainty of pref- erences and predictions in decision models: An application to regional wastewater Supplementary material related to this article can be found online planning. J. Environ. Manage. 252, 109652. http://dx.doi.org/10.1016/j.jenvman. at https://doi.org/10.1016/j.envsoft.2022.105552. 2019.109652. 14 F. Haag et al. Environmental Modelling and Software 158 (2022) 105552 Heath, A., Manolopoulou, I., Baio, G., 2017. A review of methods for analysis of the Nicol, S., Brazill-Boast, J., Gorrod, E., McSorley, A., Peyrard, N., Chadès, I., 2019. expected value of information. Med. Decis. Mak. 37 (7), 747–758. http://dx.doi. Quantifying the impact of uncertainty on threat management for biodiversity. org/10.1177/0272989x17697692. Nature Commun. 10 (1), 3570. http://dx.doi.org/10.1038/s41467-019-11404-5. Hensher, D., Rose, J., Greene, W., 2015. Applied Choice Analysis, second ed. Cambridge Radjawali, I., 2012. Examining local conservation and development: Live reef food University Press, Cambridge. fishing in spermonde archipelago, Indonesia. Rev. Gest. Cost. Integr. 12 (4), Howard, R., 1966. Decision Analysis: Applied Decision Theory. Stanford Research 545–557. http://dx.doi.org/10.5894/rgci337. Institute. Razavi, S., Jakeman, A., Saltelli, A., Prieur, C., Iooss, B., Borgonovo, E., Plischke, E., Jackson, C., Baio, G., Heath, A., Strong, M., Welton, N., Wilson, E., 2022. Value of Lo Piano, S., Iwanaga, T., Becker, W., Tarantola, S., Guillaume, J., Jakeman, J., information analysis in models to inform health policy. Annu. Rev. Stat. Appl. 9 Gupta, H., Melillo, N., Rabitti, G., Chabridon, V., Duan, Q., Sun, X., Smith, S., (1), http://dx.doi.org/10.1146/annurev-statistics-040120-010730. Sheikholeslami, R., Hosseini, N., Asadzadeh, M., Puy, A., Kucherenko, S., Maier, H., Keeney, R., 1992. Value-Focused Thinking: A Path to Creative Decisionmaking. Harvard 2021. The future of sensitivity analysis: An essential discipline for systems modeling University Press, Cambridge, Mass.. and policy support. Environ. Model. Softw. 137, 104954. http://dx.doi.org/10. Keeney, R., 2013. Foundations for group decision analysis. Decis. Anal. 10 (2), 103–120. 1016/j.envsoft.2020.104954. http://dx.doi.org/10.1287/deca.2013.0265. Reichert, P., 2020. Towards a comprehensive uncertainty assessment in environmental Keeney, R., Gregory, R., 2005. Selecting attributes to measure the achievement of research and decision support. Water Sci. Technol. http://dx.doi.org/10.2166/wst. objectives. Oper. Res. 53 (1), 1–11. http://dx.doi.org/10.1287/opre.1040.0158. 2020.032. Keeney, R., Raiffa, H., 1993. Decisions with Multiple Objectives: Preferences and Value Reichert, P., Borsuk, M., 2005. Does high forecast uncertainty preclude effective Trade-Offs, revised ed. Cambridge University Press. decision support? Environ. Model. Softw. 20 (8), 991–1001. http://dx.doi.org/10. Keisler, J., Collier, Z., Chu, E., Sinatra, N., Linkov, I., 2014. Value of information 1016/j.envsoft.2004.10.005. analysis: the state of application. Environ. Syst. Decis. 34 (1), 3–23. http://dx. Reichert, P., Langhans, S., Lienert, J., Schuwirth, N., 2015. The conceptual foundation doi.org/10.1007/s10669-013-9439-4. of environmental decision support. J. Environ. Manage. 154, 316–332. http://dx. Kelly, R.A., Jakeman, A.J., Barreteau, O., Borsuk, M.E., ElSawah, S., Hamilton, S.H., doi.org/10.1016/j.jenvman.2015.01.053. Henriksen, H.J., Kuikka, S., Maier, H.R., Rizzoli, A.E., van Delden, H., Voinov, A.A., Runge, M., Converse, S., Lyons, J., 2011. Which uncertainty? Using expert elicitation 2013. Selecting among five common modelling approaches for integrated en- and expected value of information to design an adaptive program. Biol. Cons. 144 vironmental assessment and management. Environ. Model. Softw. 47, 159–181. (4), 1214–1223. http://dx.doi.org/10.1016/j.biocon.2010.12.020. http://dx.doi.org/10.1016/j.envsoft.2013.05.005. Rushing, C., Rubenstein, M., Lyons, J., Runge, M., 2020. Using value of information Kunst, N., Wilson, E., Glynn, D., Alarid-Escudero, F., Baio, G., Brennan, A., Fairley, M., to prioritize research needs for migratory bird management under climate change: Goldhaber-Fiebert, J., Jackson, C., Jalal, H., Menzies, N., Strong, M., Thom, H., a case study using federal land acquisition in the United States. Biol. Rev. 95 (4), Heath, A., 2020. Computing the expected value of sample information efficiently: 1109–1130. http://dx.doi.org/10.1111/brv.12602. Practical guidance and recommendations for four model-based methods. Value Savage, L., 1954. Foundations of Statistics. In: Foundations of Statistics, Wiley, Oxford, Health 23 (6), 734–742. http://dx.doi.org/10.1016/j.jval.2020.02.010. England. Marchese, D., Bates, M., Keisler, J., Alcaraz, M., Linkov, I., Olivetti, E., 2018. Value Strong, M., Oakley, J., 2013. An efficient method for computing single-parameter of information analysis for life cycle assessment: Uncertain emissions in the green partial expected value of perfect information. Med. Decis. Mak. 33 (6), 755–766. manufacturing of electronic tablets. J. Cleaner Prod. 197, 1540–1545. http://dx. http://dx.doi.org/10.1177/0272989X12465123. doi.org/10.1016/j.jclepro.2018.06.113. Strong, M., Oakley, J., Brennan, A., 2014. Estimating multiparameter partial expected Marttunen, M., Lienert, J., Belton, V., 2017. Structuring problems for multi-criteria value of perfect information from a probabilistic sensitivity analysis sample: a decision analysis in practice: A literature review of method combinations. European nonparametric regression approach. Med. Decis. Mak. 34 (3), 311–326. http://dx. J. Oper. Res. 263 (1), 1–17. http://dx.doi.org/10.1016/j.ejor.2017.04.041. doi.org/10.1177/0272989X13505910. Maxwell, S., Rhodes, J., Runge, M., Possingham, H., Ng, C., McDonald-Madden, E., Teichberg, M., Wild, C., Bednarz, V., Kegler, H., Lukman, M., Gärdes, A., Heiden, J., 2015. How much is new information worth? Evaluating the financial benefit of Weiand, L., Abu, N., Nasir, A., Miñarro, S., Ferse, S., Reuter, H., Plass-Johnson, J., resolving management uncertainty. J. Appl. Ecol. 52 (1), 12–20. http://dx.doi.org/ 2018. Spatio-temporal patterns in coral reef communities of the spermonde 10.1111/1365-2664.12373. archipelago, 2012–2014, I: Comprehensive reef monitoring of water and benthic Maynard, J., Marshall, P., Parker, B., Mcleod, E., Ahmadia, G., van Hooidonk, R., indicators reflect changes in reef health. Front. Mar. Sci. 5, http://dx.doi.org/10. Planes, S., Williams, G., Raymundo, L., Beeden, R., 2017. A Guide to Assessing 3389/fmars.2018.00033. Coral Reef Resilience for Decision Support. UN Environment, Nairobi, Kenya. Uusitalo, L., Lehikoinen, A., Helle, I., Myrberg, K., 2015. An overview of methods to McCrea-Strub, A., Zeller, D., Rashid Sumaila, U., Nelson, J., Balmford, A., Pauly, D., evaluate uncertainty of deterministic models in decision support. Environ. Model. 2011. Understanding the cost of establishing marine protected areas. Mar. Policy Softw. 63, 24–31. http://dx.doi.org/10.1016/j.envsoft.2014.09.017. 35 (1), 1–9. http://dx.doi.org/10.1016/j.marpol.2010.07.001. Walker, W., Harremoës, P., Rotmans, J., van der Sluijs, J., van Asselt, M., Janssen, P., McField, M., Kramer, P., 2007. Healthy reefs for healthy people: a guide to indicators Krayer von Krauss, M., 2003. Defining uncertainty: A conceptual basis for uncer- of reef health and social well-being in the Mesoamerican Reef Region. Healthy tainty management in model-based decision support. Integr. Assess. 4 (1), 5–17. Reefs for Healthy People Initiative, Miami, FL. http://dx.doi.org/10.1076/iaij.4.1.5.16466. Miñarro, S., Leins, J., Acevedo-Trejos, E., Fulton, E., Reuter, H., 2018. SEAMANCORE: A Williams, B., Eaton, M., Breininger, D., 2011. Adaptive resource management and spatially explicit simulation model for assisting the local management of coral reefs. the value of information. Ecol. Model. 222 (18), 3429–3436. http://dx.doi.org/ Ecol. Model. 384, 296–307. http://dx.doi.org/10.1016/j.ecolmodel.2018.05.026. 10.1016/j.ecolmodel.2011.07.003. Miñarro, S., Navarrete Forero, G., Reuter, H., van Putten, I., 2016. The role of Williams, B., Johnson, F., 2018. Value of sample information in dynamic, structurally patron-client relations on the fishing behaviour of artisanal fishermen in the uncertain resource systems. PLOS ONE 13 (6), e0199326. http://dx.doi.org/10. spermonde archipelago (Indonesia). Mar. Policy 69, 73–83. http://dx.doi.org/10. 1371/journal.pone.0199326. 1016/j.marpol.2016.04.006. Wright, G., Cairns, G., O’Brien, F., Goodwin, P., 2019. Scenario analysis to support Moore, J., Runge, M., 2012. Combining structured decision making and value-of- decision making in addressing wicked problems: Pitfalls and potential. European J. information analyses to identify robust management strategies. Conserv. Biol. 26 Oper. Res. 278 (1), 3–19. http://dx.doi.org/10.1016/j.ejor.2018.08.035. (5), 810–820. http://dx.doi.org/10.1111/j.1523-1739.2012.01907.x. Myklebust, H., Eidsvik, J., Sperstad, I., Bhattacharjya, D., 2020. Value of information analysis for complex simulator models: Application to wind farm maintenance. Decis. Anal. 17 (2), 134–153. http://dx.doi.org/10.1287/deca.2019.0405. 15