ARTICLE https://doi.org/10.1038/s41467-019-13472-z OPEN Human confidence judgments reflect reliability- based hierarchical integration of contextual information Philipp Schustek 1,3*, Alexandre Hyafil 1,2,3* & Rubén Moreno-Bote1 1234567890():,; Our immediate observations must be supplemented with contextual information to resolve ambiguities. However, the context is often ambiguous too, and thus it should be inferred itself to guide behavior. Here, we introduce a novel hierarchical task (airplane task) in which participants should infer a higher-level, contextual variable to inform probabilistic inference about a hidden dependent variable at a lower level. By controlling the reliability of past sensory evidence through varying the sample size of the observations, we find that humans estimate the reliability of the context and combine it with current sensory uncertainty to inform their confidence reports. Behavior closely follows inference by probabilistic message passing between latent variables across hierarchical state representations. Commonly reported inferential fallacies, such as sample size insensitivity, are not present, and neither did participants appear to rely on simple heuristics. Our results reveal uncertainty-sensitive integration of information at different hierarchical levels and temporal scales. 1 Center for Brain and Cognition & Dept. of Information and Communication Technologies, Pompeu Fabra University, 08005 Barcelona, Spain. 2 Center for Research in Mathematics, Campus UAB Edifici C, 08193 Bellaterra, Barcelona, Spain. 3These authors contributed equally: Philipp Schustek, Alexandre Hyafil. *email: philipp.schustek@gmail.com; alexandre.hyafil@gmail.com NATURE COMMUNICATIONS | (2019)10:5430 | https://doi.org/10.1038/s41467-019-13472-z | www.nature.com/naturecommunications 1 ARTICLE NATURE COMMUNICATIONS | https://doi.org/10.1038/s41467-019-13472-z A s sensory evidence is inherently ambiguous, it needs to be both the tendency and the sample size, we can control the integrated with contextual information to minimize the reliability of previous observations upon which inference about uncertainty of our perception of the world and thus allow the context should be based. Overall, this task structure creates for successful behavior. Suppose that we observe just a few pas- hierarchical dependencies among latent variables that should be sengers exiting an airplane at an airport whose city hosts a soccer resolved by bottom-up (inferring the context from previous final. If we find that four of them are supporters of the red team observations) and top-down message passing (inferring the cur- and two support the blue team, we might conclude that there rent state by combining current observations with the inferred were more supporters of the red team in the airplane. This context)6. inference, based on incomplete sensory evidence, can be We find that participants can track and use the inferred improved by contextual information. For instance, there might be reliability of previous observations suggesting that they build a many more blue than red supporters in the world. Then, despite probabilistic representation of the context. The inferred context is our initial observation, we might want to revise our inference and integrated with the current observations to guide decisions and rather conclude, based on the context, that the airplane carried confidence judgments about the value of a latent variable at a more blue than red supporters. lower hierarchical level. Decision confidence is found to closely While in the previous example context was certain and by itself correspond to the actual accuracy of making correct decisions. As able to resolve observational ambiguity, contextual information is a clear signature of probabilistic inference over the context, we very often ambiguous. For instance, we might just know that find that the sample size of previous observations is used by our there is an event in the city that attracts more of a certain type of participants to infer the reliability of the context. This in turn has people, but we do not know which type. Extending our example, a strong effect on decision confidence of a lower-level variable we would first need to infer the context (whether the event that depends on the context. The observed behavior in our par- attracts more people of the red or blue type) by observing samples ticipants eludes previously reported biases in judgments and of passengers leaving several airplanes. By using the inferred decision making16, such as sample size insensitivity17–19, and also context, we can better estimate whether the next plane carries resists explanations based on simpler heuristics20,21. Overall, all more of one type of people given only on a small sample of its the reported effects in both tasks are consistent, quantitatively passengers. Thus, in real-life, both observations and context and qualitatively, with the optimal inference model. Thus, our commonly provide incomplete information about a behaviorally results support the view that humans may form mental repre- relevant latent variable. In these cases, inference should be based sentations akin to hierarchical graphs22 that support reliability- on probabilistic representations of both observational and con- based inference to guide confidence estimates of our decisions. textual information1–6. Indeed, recent work has shown that humans can track a con- Results textual binary variable embedded in noise that partially informs The airplane task probes inference of latent variable. We about what specific actions need to be performed to obtain designed two experiments to test whether humans can use the reward7. Additionally, humans can infer the transition probability reliability of contextual information to guide decisions and con- between two stimuli where the transition probability itself fidence judgments about a latent variable at a lower hierarchical undergoes unexpected changes, defining a partially observable level. While in some previous studies, instructions to the parti- context8. These results and other studies suggest that a refined cipants were quite abstract and often appealed to mathematical form of uncertainty representation is held at several hierarchical terms21, here we attempted to facilitate understanding of the levels by the brain9–14. However, in this previous research, the complex relationships of the task variables by instructing parti- reliability of the context has rarely been manipulated directly and cipants in intuitive and naturalistic terms. Thus, we described the independently15 from the reliability of the current observation1,7,8. task to our participants by using the analogy of airplanes arriving Therefore, it is unclear up to what degree contextual inference at an airport whose unknown passenger proportions were to be reflects its uncertainty and interacts with the inferred reliability of estimated. In the first experiment (Experiment 1), the context is the current observation as it would be expected from representing neutral and stable across all the trials encompassing the session, it with a joint probability distribution over both observations and while in the second experiment (Experiment 2) context varies context. While some effects predicted by hierarchical probabilistic across blocks of a few trials but remains constant within each inference have been previously reported in isolation, no study has block. We instructed our participants that the context consists of —to our knowledge—thoroughly assessed a body of behavioral a tendency of the encountered airplanes to carry more passengers predictions of hierarchical probabilistic inference and tested them of either of the two types. Formally, Experiment 1 corresponds to against alternative heuristics model. the classical urn problem with unknown fractions of red and blue To address the above question, we developed a reliability-based balls, and Experiment 2 corresponds to a hierarchical extension hierarchical integration task that allows us to directly control where the urns are themselves correlated and partially observable reliability in order to evidence characteristic patterns of prob- (see Methods). As no feedback was given that instructed our abilistic inference. Our task was intuitively framed to our parti- participants how they ought to make their confidence reports, cipants using the analogy of flight arrivals to an airport whose city the experiments probe their internal capacity to estimate hosts an event, rather than relying on an abstract or mathematical uncertainties. description of the dependencies between the latent variables. The goal was to decide whether the flight just landed carried more passengers of the red or blue type based on the observation of The effects of sample size on confidence reports. In Experiment only a small sample of passengers leaving the airplane, and to 1, participants were told that the airplanes arriving to an airport report the confidence in that decision. However, as the event is carry both blue- and red-type passengers, in an unknown pro- known to tend to attract more of either of the two types of portion, and that these proportions would be uncorrelated from passengers, knowledge of this context, if inferred correctly, would one plane to the next. Thus, in this case, no context was assumed be useful to solve the task. The crucial ingredient of our task is that would make our participants believe that the passenger that inference of the context is based on the observation of small proportions across consecutives planes would be interdependent. samples of passengers exiting previously arrived planes, making After observing a small sample of passengers randomly exiting the context partially, but not fully, observable. By manipulating the plane, displayed as red and blue filled circles on the screen 2 NATURE COMMUNICATIONS | (2019)10:5430 | https://doi.org/10.1038/s41467-019-13472-z | www.nature.com/naturecommunications NATURE COMMUNICATIONS | https://doi.org/10.1038/s41467-019-13472-z ARTICLE a b Sample Inferred representation Confidence Response Sample size Sample Probability pdf R B Passenger proportion c d 1 Sample size Confidence blue majority 5 Slope confidence curve 0.8 Large 0.6 Small 2.5 0.4 0.2 0 0 0 20 40 60 60 100 3 5 7 9 11 13 Proportion of blue samples (%) Sample size Fig. 1 Posterior-based confidence features sample size effects a Task: The colored dots (sample) represent two kinds of passengers (blue and red) that disembarked a very large airplane. The participants are subsequently asked to report the confidence in their decision that the airplane carried more blue or red passengers (blue majority) by horizontally moving the cursor line (orange). In this case, because the sample suggests a blue majority, the response cursor should be on the right. b Sample size increases posterior-based confidence in a blue majority suggested by the blue majority of the sample. Confidence (right) is computed as expected accuracy from the area under the curve for the inferred proportion (middle) from the observed sample (left). Although the proportion of blue passengers (green line, middle) is the same for all three samples (rows), the inferred distribution depends on sample size. The larger the confidence, the closer the response line on the previous panel should be to the rightmost border. c Confidence in blue majority should increase with the proportion (%) of blue samples for all sample sizes, but it does so with a higher slope for larger sample sizes (color coded). d Consequently, the slope parameter of fitted sigmoidal functions increases with sample size. (Fig. 1a, first frame), participants were asked to report both possible blue passenger proportions that are larger than one whether the airplane carried more blue or red passengers, i.e., its half23,24 (Fig. 1c, d). The result is that confidence in a blue passenger majority, and their confidence in this decision by majority increases with sample size because the distribution is moving a line along a horizontal bar (second frame). Importantly, more concentrated around the observed proportion of blue there was no direct feedback about normative confidence reports: passengers. More generally, a central feature of probabilistic participants received a binary feedback after each response, i.e., inference is sample size dependence, which here magnifies the whether they correctly identified the latent passenger majority. In confidence in the airplane majority that is suggested by the addition, indirect feedback was provided at regular pauses every sample proportion. five trials through some aggregated performance score based on We tested whether human participants (n = 24) obeyed this the ideal observer which was solely intended to maintain our critical pattern or whether they neglected size17,19. Confidence in participants engaged in the task. While such feedback could in a blue majority was found to increase with the proportion of blue principle be marginally used to adapt one’s responses, partici- samples. As predicted, this increase was larger the larger the pants did not seem to modify their responses accordingly: first, sample size is (Pearson correlation, pooled across participants, feedback was hardly indicative of the optimal policy (see Meth- ρ = 0.31, p = 4.08 × 10−6) (Fig. 2a, b). These results were found ods); second, participants performed the task well from trial one for most of our participants individually (21 out of 24; and did not improve over time (see Supplementary Fig. 6a). permutation test, p < 0.05; Supplementary Fig. 2). Consistently, An ideal observer (Fig. 1b) should infer a distribution over an confidence judgments were highly predictive of the probability airplane’s proportion of blue (or, equivalently, red) passengers that the chosen majority was correct (Pearson correlation, ρ = based on the observed proportion of blue passengers and the 0.81, p = 1.27 × 10−45, see Supplementary Fig. 1 for details), sample size. The proportion of blue samples (passengers), called suggesting that participants performed the task well and gave “sample proportion”, is computed as NB/N, where NB (NR) is the confidence reports that follow from an internal measure of number of observed blue (red) passengers, respectively, and N = uncertainty. NB + NR is the sample size. The inferred distribution over To further confirm that sample size was an important feature passenger proportions concentrates around passenger propor- of our participants’ confidence reports, we performed a model tions suggested by the sample (Fig. 1b, green vertical line)17, and comparison in which we contrasted the optimal inference model its width reduces the larger the sample size is. The decision with two heuristic models, the ‘ratio’ and the ‘difference’ model. whether the majority is blue or red is uniquely based on the The ratio model assumes that confidence is a function of the proportion of blue samples, but the confidence report should be sample proportion alone. This could be the result of a simpler based on both the sample proportion and the sample size. approach in which the population estimate is a point estimate Specifically, in this example, decision confidence of the ideal corresponding to the sample proportion which is a more suitable observer is the belief that the majority is blue, which equals the approach in the limit of large samples that are representative of area under the distribution summing up the probability of all their population16,17. The difference model estimates confidence NATURE COMMUNICATIONS | (2019)10:5430 | https://doi.org/10.1038/s41467-019-13472-z | www.nature.com/naturecommunications 3 ARTICLE NATURE COMMUNICATIONS | https://doi.org/10.1038/s41467-019-13472-z a 1 b 10 Sample size Confidence blue majority Slope confidence curve 0.8 Large 8 0.6 Small 6 0.4 4 0.2 2 0 0 0 20 40 60 80 100 3 5 7 9 11 13 Proportion of blue samples (%) Sample size Fig. 2 Human confidence estimates vary with sample size as predicted by probabilistic inference. a Confidence in a blue majority increases with the proportion of blue samples (solid lines), and it does so more steeply the larger the sample size is (color coded). Optimal model is represented in light colors. b The slope of the confidence curve in a increases with sample size. Participants feature a quantitatively similar increase as the optimal model (solid line). Error bars indicate SEM across participants. based on the difference of blue and red samples, NB–NR. As the should start with inference of the current context (whether there ratio heuristic, this statistic is informative of decision correctness is a tendency to observe passengers from airplanes with blue or but additionally covaries with sample size, as the ideal observer red majorities). Next, this contextual information should be model, but not directly through sample size. It would correspond integrated with the current sample to report confidence and to the optimal model if the true proportion would only take two decide whether the current airplane it is more likely to hold a red possible values (e.g., 60% blue passengers or 40% blue or blue majority (Fig. 3). passengers), i.e. if subjects would discard the variability of the Thus, the generative structure of the observations that were true proportion within each of the two categories (see ‘Analytical shown to the participants is hierarchical, with a higher-level approximation for Experiment 1’ in Supplementary Methods). To variable that determines the context for a block of always five account for possible distortions on the response and/or calibra- trials, which either favors red or blue airplane majorities, and tions of heuristics estimators, all model estimates (either from the which in turn generates airplane majorities at the lower optimal or heuristics models) were passed through a logistic hierarchical level across the sequence of trials in the block function that mapped the estimate onto the unity interval. The (Fig. 3a). Both hierarchical levels feature hidden variables that are logistic response mapping was fitted for each model and not observable by the participants. From the generated airplane participant individually (Methods). proportions, samples are drawn, which correspond to the actual The comparison between the optimal model and the ratio observations of the participants (Fig. 3b). Note that the generative model shows that the latter is clearly rejected because of its process is purely top-down, from the context (high-level hidden incapacity to take sample size into account (Supplementary variable) to airplane passenger proportions (low-level hidden Fig. 4). Even though the confidence estimates of the difference variables) and then to the samples (observables). However, model are sensitive to sample size, they typically do not inference by the ideal observer should first run bottom-up from correspond to the notion of uncertainty that our participants previously observed samples to infer the value of the contextual report: the difference model predicts a linear relationship between variable (Fig. 3c; open nodes) and then top-down from this the sample size and the slope of the confidence curve, while inferred context (bottom open node) to the variable representing subjects displayed a clear sublinear relationship (see Supplemen- the passenger proportion of the current airplane (orange node). tary Fig. 3). We can thus dissociate the experimental reports from For the ideal observer, this can be formulated as message passing these simple but covariant heuristics and conclude that the between the hidden variables (Methods). It is worth emphasizing response patterns of our participants suggest a probabilistic that the task is about inferring the passenger majority of the inference approach. Moreover, as the difference model (corre- current airplane, at the lower hierarchical level, rather than asking sponding to the optimal response when the variability of true for the context. Note also that, in contrast to change-point proportion is discarded) can be ruled out, our results suggest that detection paradigms25 subjects were explicitly told that a new our participants’ inference process incorporated not only context had to be inferred at the beginning of each block. uncertainty about the passenger majority on the plane (blue or As with Experiment 1, we studied how an ideal observer would red) but also about its magnitude (the proportion). behave under specific manipulations of the reliability of the currently observed sample through its sample size and the Reliability-based hierarchical integration of ideal observer. In reliability of the context as controlled by the sample size of Experiment 2, participants were told that several airplanes with previously observed airplanes. As with the previous experiment, unknown passenger proportions would arrive at an airport, as we first point to patterns of behavior that should be indicative of before, but that consecutive airplanes would feature correlated reliability-based probabilistic inference in our hierarchical task. passenger proportions because of an event in the city that attracts First, we expect that confidence in blue majority of the current more travelers of one type. Thus, if the sample of a previous airplane grows with the proportion of blue samples (Fig. 4a), as in airplane is highly suggestive of a blue airplane proportion, then the previous task. However, in addition, we also expect that the participant could not only infer that this previous airplane confidence in a blue majority should be higher in blocks whose carries a blue majority, but also that the next airplane is more actual tendency favors blue airplane majorities, which is indeed likely to carry a blue majority, even before observing a sample of the pattern that an ideal observer would show (Fig. 4a). This is passengers leaving it. Importantly, in Experiment 2, there was no because, averaged across trials, the ideal observer can infer what feedback about decision correctness of each trial’s airplane the block tendency is, which on average should be aligned to the majority, only an overall score after each block (see Methods and true block tendency, resulting in a higher confidence in blue Supplementary Fig. 6b). Inference of an ideal observer in our task majorities. 4 NATURE COMMUNICATIONS | (2019)10:5430 | https://doi.org/10.1038/s41467-019-13472-z | www.nature.com/naturecommunications NATURE COMMUNICATIONS | https://doi.org/10.1038/s41467-019-13472-z ARTICLE a Context Environment Unobservable––hidden Block tendency Trial 1 Trial 2 Current trial T Airplane 1 Airplane 2 Airplane T b Observations R B c Response m2(b) T 1 2 Top–down Bottom–up m1(b) M (b) Agent Internal representations M (b) Fig. 3 Schematic of the hierarchical structure for learning empirical priors Participants are told that across a block of five trials (1, 2,…T ≤ 5) they will see passengers from five different airplanes arriving to the same airport. As before, they are asked to report their decision confidence whether the current airplane carried more red or blue passengers. The schematic illustrates the hypothetical examples of an ideal observer that estimates confidence based on the proportion of blue samples of the current airplane T and on the samples observed in previous trials. The generative model of the observations is as follows. a Within a block of five trials, the context, called block tendency, is first selected, which corresponds to choosing either a positively (magenta) or negatively (cyan) skewed distribution over airplane proportions. This context (distribution) is maintained throughout the block of five trials, but on each trial a new blue majority (blue-red horizontal bars indicating the passenger proportion in each airplane) is randomly sampled from that distribution. In the example, the context favors airplanes of red majorities. b Sample generation given the airplane majority is the same as for the previous task. c The internal representation of the agent (orange background) mirrors the dependence structure in the environment (green background). Probabilistic inference is performed by message passing between the nodes which internally represent the inferred block tendency and the airplane’s passenger proportion of each trial (see Methods). Previous trials (t < T) provide evidence about the block tendency through the messages mt(b). They are probabilistically integrated into an overall belief about the block tendency M(b) which provides top-down constraints on the inference of a new airplane’s blue proportion (orange node). The confidence in a blue majority of the current airplane T held by the ideal observer (response bar, right) should follow from both the current sample proportion and the inferred block tendency from previous samples. Second, averaged across sample proportions and samples sizes, confidence in the current trial should be constant (Fig. 4e), confidence in a blue (red) majority in the current airplane should because an earlier trial provides the same evidence for the context increase the higher the inferred tendency of blue (red) passengers as a recent one, on average across blocks. Finally, the more trials is. Because of the symmetry across these two cases, we defined a have been observed in the block, the better the inference about (block-) aligned confidence to indicate the confidence in the the current context ought to be. Thus, on average across blocks, direction (passenger type) that is aligned to the actual block aligned confidence should increase with the number of previous tendency and pooled the results across these two cases. For the observations which indicates accumulation of evidence for the ideal observer, aligned confidence increases with the aligned contextual variable (Fig. 4f). inferred tendency (Fig. 4b). In other words, the inferred context It is important to emphasize that these patterns correspond to informs inference of the current airplane’s proportion to the predictions of the ideal observer model. They will be used as a degree that the context is reliable itself. benchmark for a direct comparison to behavioral data without Sample size of the current observation should play a very fitting any parameters. Consequently, we do not expect a perfect important role in modulating decision confidence as it indicates match in terms of absolute values, but we would expect similar increased reliability of the sample relative to the prior. Indeed, patterns of variation if participants follow a probabilistic aligned confidence increases with the aligned sample proportion, inference strategy. and it does so with a higher slope when sample size is large (Fig. 4c). Similarly, if the context is inferred probabilistically, the Human behavior follows patterns of probabilistic inference. reliability of previous trials should be taken into account. As a We first tested whether human participants can infer and use consequence, the sample size of the previous observation should contextual reliability by studying whether they followed the pat- modulate aligned confidence (Fig. 4d). For instance, if the terns described above. We found that our participants’ confidence previous sample was large and suggested a red majority, then in a blue majority increased with the proportion of blue samples, confidence in a red majority in the current trial should be larger. but that confidence in a blue majority was larger when the block Another pattern that is expected from the ideal observer is that favored airplanes with blue majorities as opposed to red majo- the weights (see Methods) of all previous trials in a block onto the rities (Fig. 5a). This result indicates that participants not only NATURE COMMUNICATIONS | (2019)10:5430 | https://doi.org/10.1038/s41467-019-13472-z | www.nature.com/naturecommunications 5 ARTICLE NATURE COMMUNICATIONS | https://doi.org/10.1038/s41467-019-13472-z a 1 Block tendency b 0.9 c 1 Confidence blue majority Blue Aligned confidence Aligned confidence 0.75 Red 0.8 0.75 0.7 0.5 0.5 0.6 Sample size 0.25 Large 0.5 Small 0.25 0 0 25 50 75 100 0.25 0.5 0.75 1 25 50 75 100 Blue samples (%) Aligned inferred tendency Aligned sample proportion (%) d e 1.4 f 0.75 Influence of previous trials 0.8 Aligned confidence Aligned confidence T3 1.3 0.7 0.7 T4 Sample size 1.2 0.65 0.6 Large T5 Small 0.5 1.1 0.6 25 50 75 100 1 2 3 4 1 2 3 4 5 Aligned proportion of previous sample (%) Trial in block Trial in block Fig. 4 Characteristic behavioral patterns of probabilistic inference in the hierarchical inference task a Confidence in a blue majority of the current airplane (current trial) should increase with the proportion of blue samples, as in the previous task, but in addition confidence should be larger in a block that favors blue majorities (cyan) than in a block favoring red majorities (magenta). b Information of the block tendency should gradually increase the confidence in the corresponding trial majority. Thus, responses can be pooled with respect to the real block tendency. We refer to it as ‘aligned confidence’ and use the same concept for other relative quantities below. c Confidence in the aligned airplane majority increases with the aligned sample proportion. This modulation is stronger for larger sample sizes (green) compared to smaller ones (orange) while it has no effect for an indifferent sample (50% sample proportion, crossing point between the two lines). d Likewise, aligned confidence increases with the aligned sample proportion of the preceding trial and is modulated by its respective sample size. e The influence of all previous trials, determined by the weights of a regression analysis, should be equal on average (e.g., trials 1–2 on trial 3, T3). However, it decreases with the number of previous trials due to normalization. f Aligned confidence increases across trials within a block because evidence for the block tendency accumulates across trials in the block. All patterns are derived from the ideal observer model (see Methods). a 1 b Block tendency Confidence blue majority 0.8 0.8 Blue Aligned confidence Red 0.6 0.7 0.4 0.6 0.2 Exp 0.5 Opt 0 0 20 40 60 80 100 0.2 0.4 0.6 0.8 1 Proportion of blue samples (%) Aligned inferred tendency M (b) Fig. 5 Inferred block tendency affects confidence reports. a Confidence in blue majority is higher when the block tendency favors blue majorities (cyan) than when it favors red majorities (magenta). Experimental results (data points) are shown along with optimal behavior (solid lines), indicating an integration of sample information with a learned belief about the block tendency. b Aligned confidence (black) increases with the optimally inferred belief about the block tendency and is a close correlate of the optimal response (red), suggesting that participants internally track a graded belief based on previously available evidence. Error bars indicate SEM across participants. relied on the current sample to infer the current airplane no thresholding nor any sign of categorical representation. This majority, but that they also inferred the context and used it to shows that the contextual variable—for which we never explicitly modulate their confidence judgments. asked—is represented in a graded manner, as it would be Further evidence for this result comes from the observation expected from a probabilistic agent. Our participants not only that aligned confidence increases with the strength of the inferred followed this pattern qualitatively, but they also seemed to adhere tendency aligned to the block as computed by the ideal observer, quite closely to the quantitative, parameter-free, predictions made indicating that the more evidence was collected for a given block’s by an ideal observer (Fig. 5b; Pearson correlation on binned tendency, the larger the modulation on the confidence reported in values, pooled across participants, ρ = 0.77, p = 5.13 × 10−33), the current trial was. The gradual increase (which was also except for the fact that contextual information did not affect present at an individual level, Supplementary Fig. 8a) shows how predicted confidence as much as when contextual information nuanced the representation of the contextual variable is as there is was high (Fig. 5b, rightmost part), which was also observed on a 6 NATURE COMMUNICATIONS | (2019)10:5430 | https://doi.org/10.1038/s41467-019-13472-z | www.nature.com/naturecommunications NATURE COMMUNICATIONS | https://doi.org/10.1038/s41467-019-13472-z ARTICLE a 5 b 1 c Slope confidence curve Sample size 0.8 Sample size Sample size Aligned confidence Aligned confidence Large Large 4 0.8 Large 0.75 Small Small Small 0.7 3 0.6 0.65 2 0.4 0.6 1 0.55 2 4 6 8 10 12 20 40 60 80 20 40 60 80 Sample size Aligned sample proportion (%) Aligned proportion previous sample (%) Fig. 6 Sample size effects reveal reliability-based information integration. a As in the basic task (Fig. 2b), the slope (data points) of the confidence curves over the sample proportion increases with sample size and tightly follows the optimal pattern (solid line). b The modulation of aligned confidence with the aligned sample proportion of the current trial is larger when the sample size is high (green) than when it is low (orange). Significant signed differences of a bin-wise one-sided signed rank test are indicated, *0.01 < p ≤ 0.05, **p ≤ 0.01. c The modulation of aligned confidence with the aligned sample proportion of the previous trial is larger when the sample size of the previous trial is high (green) than when it is low (orange), similar to the previous panel. Error bars indicate SEM across participants in a–c. participant by participant basis (one-sided signed rank on fitted be evidenced through another linear regression analysis in which slopes, p = 0.004). Thus, even though the inferred tendency is the previous trial index is used to predict the average weight of subjective to the participant, the correlation with the inferred the previous trial on the aligned confidence (regression on the tendency of the ideal observer shows that participants must be means across participants, separately for current trials position 3, tracking a similar quantity. 4, and 5: p-values 0.41–0.89 for trials with 2–4 previous trials Next, we studied how reliability governs hierarchical informa- respectively). Apparently, there are no signatures of temporally tion integration (see Fig. 4c, d). Both the current sample and selective evidence integration for the contextual variable such as a previous samples should be relied upon more strongly when their confirmatory bias, which is characterized by an insufficient belief reliabilities, controlled by sample size, are high. We first revision once a belief has been established. If it were present, later confirmed that the slope of the confidence curve increases with trials would be expected to have a lower influence here. sample size of the current observation (Fig. 6a; Pearson Probabilistic inference on the other hand, never fully collapses correlation of slope with sample size, pooled across participants, onto one specific interpretation and hence never excludes ρ = 0.49, p = 8.67 × 10−14), indicating that participants used the evidence for competing hypotheses. Similarly, this rather reliability of the current observation to form confidence estimates, balanced weighting is also inconsistent with some sort of leaky as in the previous task without hierarchical dependencies (see prior integration scheme in which evidence presented long ago is Fig. 2b). fading from memory. In agreement with these findings, evidence Beyond the finding above that participants learn the block for the block tendency, and thus also aligned confidence, tendency (Fig. 5a), they should use it selectively and rely more increases over the trials within a block (Fig. 7b). A linear strongly on the sample compared to prior information when regression analysis of aligned confidence as a function of the sample evidence is reliable (Fig. 6b, pattern: Fig. 4c). Indeed, the aligned trial index clearly shows the expected increase (regression modulation with the aligned sample proportion is stronger for on means across participants, p = 8.68 × 10−9). Overall, hier- larger sample sizes and leads to the crossover of the two archical integration offers a parsimonious explanation for context conditional curves (signed difference of conditional slopes from integration which does not require explicit memorization of linear regression, signed rank test across participants, p = 1.44 × previous samples after they are integrated into the context-level 10−5). On average across trials, prior information increases variable. aligned confidence (Fig. 6b). Relative to this offset, behavior is less Interestingly, the most obvious quantitative departure from the strongly driven by smaller samples because they provide less expected patterns was that human participants appear to rely less information so that the agent resorts more closely to the on contextual information as the observed effects of previous top–down expectations gained from previous trials. trials were smaller than the predictions from the ideal observer. Direct control of the reliability through sample size allows us to For instance, the effect of previous trials on aligned confidence is study whether the inferred reliability of the context interacts with weaker (see e.g., Fig. 7a) but does not depend on how long ago the the reliability of the current observation to inform confidence information was acquired. Further support for such an insensi- judgments. Using this degree of freedom, we tested whether tivity to prior information is provided by trials in which an ideal participants used the reliability of the previously observed sample. observer would e.g., estimate a red majority despite more blue We found that, consistent with the pattern predicted by the ideal samples because of a high prior belief in a red tendency. We observer, aligned confidence increased with the aligned sample found that most participants make these evidence-opposing proportion of the previous sample and that this increase was choices (see Methods, one-sided signed rank test with respect to larger the larger its corresponding sample size was (Fig. 6c; non-hierarchical ratio model with realistic response noise, p = signed-rank test for positive difference of linear regression slopes 0.007; Supplementary Fig. 5b). There is however a tendency to across participants, p = 0.002; see also dependence on previous stay on the side of the category boundary that is suggested by the message mt-1(b) Supplementary Fig. 8b). momentary evidence, as they make significantly fewer opposing A central prediction of the probabilistic model is that all choices than the optimal model (one-sided signed rank test, p = previous trials should have equal influence on behavior on 0.008). average across blocks (see Fig. 4e, f). We determined their Finally, we tested whether this relative insensitivity to prior influence from a regression analysis on the confidence judgments information could be explained by mismatching assumptions (see Methods) and found a rather balanced influence of all about the magnitude of the block tendency which we modeled previous trials (Fig. 7a). Accordingly, no significant trend could with specific skewed distributions of passenger proportions under NATURE COMMUNICATIONS | (2019)10:5430 | https://doi.org/10.1038/s41467-019-13472-z | www.nature.com/naturecommunications 7 ARTICLE NATURE COMMUNICATIONS | https://doi.org/10.1038/s41467-019-13472-z a 1.5 b 0.75 Opt Weight of sample proportion 1 Exp Exp 0.5 Opt Aligned confidence 1.5 0.7 1 0.5 0.65 1.5 1 0.5 0.6 1 2 3 4 1 2 3 4 5 Trial in block Trial in block Fig. 7 Behavior reflects hierarchical evidence integration across trials. a On average across blocks, all previous trials provide the same information about the block tendency irrespective of their temporal distance to the current trial. From top to bottom, trials number 3–5 of each block are predicted from the indicated previous trials (sample proportion). Participants show a balanced weighting despite smaller weights compared to the ideal observer model (red). b Participants accumulate evidence about the block tendency in a gradual fashion. Aligned confidence increases over trials within a block despite a smaller effect compared to the optimal model (red). Error bars indicate SEM across participants. a 1 b 0.8 c 1 Block tendency Sample size Confidence blue majority Aligned confidence Aligned confidence 0.8 Blue Large 0.8 Red 0.7 Small 0.6 0.6 0.4 0.6 Exp 0.4 0.2 Fit 0.5 0 0 20 40 60 80 100 0.2 0.4 0.6 0.8 1 20 40 60 80 Proportion of blue samples (%) Aligned inferred tendency Aligned sample proportion (%) d e 1.5 f Weight of sample proportion 0.75 Sample size 0.7 1 Aligned confidence Aligned confidence Large 0.5 0.7 Small 1.5 1 0.65 0.65 0.5 0.6 1.5 1 0.55 0.5 0.6 20 40 60 80 1 2 3 4 1 2 3 4 5 Aligned proportion previous sample (%) Trial in block Trial in block Fig. 8 Patterns for fitted block tendency. Behavioral patterns in the hierarchical inference task (Experiment 2) compared to a fitted model assuming a mismatched block tendency and a sigmoidal response mapping accounting for distortions on the response. The fits of this model closely reproduce the patterns produced by participants. Error bars indicate SEM across participants. the red and blue contexts (see Fig. 3a). In fact, some behavioral between the experimental and the assumed skewed distributions biases, such as confidence under- and overestimation26, can be by the participants. partly explained by choosing (structurally) mismatched prob- ability distributions for the task at hand27,28. To test this Model comparison favors probabilistic inference of context. possibility, we used a model that allowed for a differently skewed The previous analysis has shown that behavior adheres to the distribution implementing this block tendency (see Methods) and main features of probabilistic inference in a reliability-based compared it to the ideal observer model. To correct for other hierarchical task. We have seen that these patterns were quali- distortions, both models used an additional mapping onto the tatively reproduced by the optimal model without the fitting of final response. We found that the model with the mismatched free parameters, and that a simple extension of the ideal observer block tendency almost perfectly described the patterns of model largely improved the qualitative fits of the patterns. probabilistic inference (Fig. 8; exceedance probability p ≈ 1, for To go beyond qualitative patterns of behavior and provide a patterns see Supplementary Fig. 7) and that participants appear to more quantitative account of the results and the adherence of subjectively assume a weaker block tendency as evidenced by the behavior to reliability-based hierarchical inference, we fitted the expectation value of the skewed Beta-distribution used to model a ideal observer estimate of the contextual variable to behavior and blue block tendency (optimal 0.61, median across participants compared it to simpler heuristic estimates that do not rely on 0.55, one-sided signed rank test for difference, p = 1.68 × 10−4). probabilistic inference. These simpler models assumed specific This suggests that qualitative differences arise from a mismatch forms for the accumulated contextual information that depart 8 NATURE COMMUNICATIONS | (2019)10:5430 | https://doi.org/10.1038/s41467-019-13472-z | www.nature.com/naturecommunications NATURE COMMUNICATIONS | https://doi.org/10.1038/s41467-019-13472-z ARTICLE from the optimal computations, as follows (contextual variable models that humans use while they perform a task8,23,28,31,32. M = M(b) in Fig. 3; see Methods). However, most of this work has focused on simple inference In the ‘averaging’ model we assume that the estimate of the problems in which the value of a hidden variable has to be esti- contextual variable M equals the presented percentages of mated based on noisy evidence24,33,34, without any hierarchical previous trials in a block and thus neglects sample size. In the structure. In contrast, even visual processing in normal condi- ‘tally’ model we assume that the estimate of the contextual tions should rely on hierarchical schemes where hidden variables variable M equals the ratio between the total number of blue at higher levels constrain the values of partially observed variables samples observed so far in all previous trials within a block over at lower levels35. Hierarchical representations allow to exploit the number of all red and blue samples observed within a block inferential constraints by learning them from experience with so far. This is similar to pooling the samples of all trials, as if they related situations by exploiting abstract similarities through were drawn from a common population. Thus, as larger samples contextual variables. Such joint inference over structured prob- contribute more points, this model is sensitive to sample size, but ability distributions is a crucial ingredient for theories such as in a different way than the ideal observer model. Finally, in the predictive coding3,6,36. However, whether human inferences rely ‘difference’ model, contextual information is a sigmoidal on ubiquitous probabilistic representations across a hierarchy of function of the running average of the differences between the variables is largely unknown. number of blue and red samples in all previous trials. All these Addressing this important question requires the ability to models only differ in how they estimate the contextual variable independently control the reliability of higher-level and lower- M. To introduce as few constraints as possible on the integration level variables to test, for instance, whether and how behaviorally of M with the current sample (NB/N,N) and to compute the final reported confidence is modulated by them. If reliability cues response, we used a flexible generalization Eq. (14) of the produce modulations of confidence reports in accordance with sigmoidal response mapping Eq. (13), attempting to reduce noise theoretically predicted patterns, then such observations would for model comparison. Even though all three heuristic constitute evidence in favor of mental representations similar to approaches are close correlates of the optimally estimated probabilistic graphical models. Previous work has studied per- contextual variable, we found that the three models were inferior ception and decision making in similar hierarchical schemes like to the probabilistic strategy of the ideal observer model ours1,7,8,15, but it has been difficult to independently modulate (Supplementary Fig. 3). the reliability at both higher and lower hierarchical levels. For Beyond quantitative comparisons, heuristic models also failed example, when using stimulus duration and stimulus strength as to quantitatively reproduce the defining features of subject an indirect proxy to control reliability15, the way these manip- behavior. Specifically, participants’ responses were influenced by ulations affect reliability depends on the specifics of the sensory the proportion of blue passengers in previous trials of the block, system and sensory noise. In our task, uncertainty emerged not and that influence increased with trial position in the block as from sensory noise but from a hidden cause for stochastically subjects accumulated evidence about current context across generated stimuli, and the reliability of both levels could be trials (Supplementary Fig. 10). Such feature was seen in the controlled directly and independently through sample size, thus optimal model. By contrast, in all three heuristics models, the providing an objective measure of trial-to-trial reliability inde- influence of the proportion of blue passengers in previous trials pendent of the sensory system. Our task revealed that humans remained constant across the block, as in these heuristics models modulate their confidence not only based on the reliability of the evidence about the current context is averaged and not currently observed sample, but also on the inferred reliability of accumulated across trials (see Eqs. 9–11; Supplementary the context which is itself a function of previous samples. Spe- Fig. 10c-e). Moreover, the ‘averaging’ model was, by construc- cifically, we have found strong dependencies of confidence on the tion, insensitive to the sample size of previous trials, unlike our sample size of current and previous observations, and these participants (Supplementary Fig. 10b). dependencies adhered to the predicted trends and patterns of hierarchical probabilistic inference. Dependencies on previous observations emerged only based on the distribution of previous Discussion stimuli, without any trial-to-trial external feedback that could be One important question is whether humans can hold probabil- used to modulate the priors. In summary, while previous studies istic representations of contextual variables and use them to had already shown in isolation sample size sensitivity37 and some improve inference of lower-level variables by providing suitable form of hierarchical probabilistic reasoning1,7,8,15, here the con- constraints on their possible values. Here, we report that humans junction of both phenomena and the very detailed correspon- can perform reliability-based hierarchical inference in a task in dence between human and optimal behaviors builds strong which they have to report their decision confidence about the evidence for ubiquitous reliability-based integration of hier- value of a lower-level variable that is constrained by a higher- archical information, even without extensive prior training on level, partially observable variable. We controlled evidence by the task. using reliability cues in the form of sample size, giving us enough It is possible that our participants did not truly hold prob- leverage to test the identified patterns of hierarchical probabilistic abilistic uncertainty representations over a mental graphical inference. The similarity between observed and probabilistic representation across multiple levels, but that they rather used inference patterns of behavior, the strong dependence of con- very sophisticated heuristics that we were not able to characterize. fidence on currently and previously observed samples sizes, and a However, estimating uncertainty about latent variables is a par- model comparison between optimal and heuristic models, sup- ticularly difficult problem for heuristic approaches just based on ports the notion that humans can mentally hold and update point estimates that disregard the distributional format that the ubiquitous representations of uncertainty over structured estimate should take5, e.g., that several airplane proportions are knowledge representations such as graphs22. consistent with a given sample. In our task, for instance, learning A large body of research has addressed the question whether, calibrated confidence reports would require repeated exposure to and under what conditions, humans can perform probabilistic the same sample together with supervising feedback about the inference, typically, by using perceptual tasks10,29,30. More actual latent variable (airplane majority). Even for very simple recently, the usage of confidence reports has opened a window to problems, the scarcity of such data makes this frequentist more directly examine how uncertainty is handled in internal approach to uncertainty estimation practically difficult and thus NATURE COMMUNICATIONS | (2019)10:5430 | https://doi.org/10.1038/s41467-019-13472-z | www.nature.com/naturecommunications 9 ARTICLE NATURE COMMUNICATIONS | https://doi.org/10.1038/s41467-019-13472-z un-ecological. As we did not provide supervising feedback, our (Supplementary Fig. 1). By contrast, our participants behavior participants presumably held accurate internal trial-by-trial was found to be little affected by numerosity or other forms of representations of uncertainty38,39. Although we cannot com- sensory noise (see ‘Sensory noise’ in Supplementary Methods). pletely rule out the use of non-probabilistic or heuristic shortcuts, The easiness with which our participants seemed to perform the main patterns of probabilistic inference have been fulfilled by probabilistic inference over the mental representation of a our participants. Their generalizations are hard to conceive graphical model at several levels of a hierarchy should not distract without relying on an internal generative model of the observa- us from the computational difficulty of the inference tions. This is in line with previous studies (e.g.,40,41) which process. Typically, probabilistic inference even in simpler tasks conclude that human inferences are model-based or use internal involves complex operations such as normalization and simulations42. marginalization5,49,50. Interestingly, inference in our task can be One clear limitation of our study is that it shows that humans considerably facilitated if the conditional independence properties can use reliability-based hierarchical integration of evidence, but between variables are exploited. In this case, the distribution it does not speak to the circumstances when this occurs. In factorizes so that only local computations (marginalization) need particular, our results contrast with a vast literature that has to be performed whose results can be passed on as messages. reported deviations from the norms of rational inference in Hence, the graphical structure of the model facilitates inference human judgments such as sample size insensitivity17,19,43. One which may even be implemented with recurrent neural important methodological difference between this previous work populations51. and ours is that behavioral economics has typically dealt with Apart from the tractability of the computations, we must bear situations that have been conveyed using mathematical terms21. in mind that the goal of the participant is not necessarily pure We speculate that the success of our participants in ‘under- inference, but the maximization of some subjective cost-benefit standing’ the hierarchical structure of the task is the result of the measure52. Further research is needed to test what constitutes the way the task has been framed and communicated. We put par- main challenges to probabilistic inference for humans such as ticipants in an imaginary yet intuitive setting of arrivals to an imposing adequate structural constraints that leverage contextual airport whose city hosts an event and refrained from using terms knowledge or the use of tractable approximations due to limited such as “urns” or “correlations”, which mathematically define our cognitive resources. task on an abstract level. Evidently, this approach was successful In sum, we have developed a novel reliability-based hier- in at least two respects. First, the task structure is clearly com- archical task based on which we found that humans are sensitive municated so that participants make roughly correct assumptions to the reliability of both high- and low-level variables. Our results for inference. Second, our participants managed to interrogate reveal uncertainty-sensitive integration of information in hier- cognitive systems that are capable of probabilistic inference14. archical state representations and suggest that humans can hold Interestingly, a recent proposal has suggested that intuitive tasks mental representations similar to probabilistic graphical models that sidestep high demands on working memory and natural where top-down and bottom-up messages can inform behavio- language may improve performance44. The existence of such rally relevant variables. framing effects45 onto the algorithmic nature of perceptual inference mechanisms should be tested in a future experiment. A Methods related but slightly different hypothesis is that probabilistic Participants. All participants were invited to complete three sessions on different inference would be shaped during lifetime experience by repeated days within three consecutive weeks. The sessions were targeted to take about 35 exposure to choices between options that require integration min (Session 1) and 45 min (Sessions 2,3). In total 25 participants (15 female, 10 between sources of varying reliability. In the lab, such probabil- male) were recruited mainly among students from the Pompeu Fabra University in Barcelona. The study was approved by the Ethics Committee of the Department istic inference process would only be applied if the task bears (CIREP approval #0031). We excluded data from one participant that did not some similarity with the problems already encountered by the complete the experiment. The median age was 25 (minimum 20, maximum 43). subjects in their life. In support of such hypothesis, a recent study We accepted all healthy adults with normal or corrected to normal vision. We did find sample size sensitivity in how subjects updated product obtained written confirmation of informed consent to the conditions and the payment modalities of the task. Irrespective of their performance, they were paid 5 evaluation by learning about previous consumers’ ratings37, € for session 1 and 7 € for sessions 2 and 3. which is a highly familiar task routinely performed in everyone’s Additionally, they had the chance to obtain a bonus payment which was everyday life. determined by the mean of their final score after removing the worst trials (2.3%). However, our work has also revealed some differences between The score S = 1 − |y − yopt| of a response y was computed based on the proximity to the optimal confidence report yopt (see below for details of the optimal model in optimal and observed behavior. Most strikingly, we have found both experiments). As such, the overall score reflected the ability of the subject to evidence that top-down information is relied upon less strongly correctly infer the probability that the observed stimuli would be sampled from one relative to information from the specific instances of the sam- category or the other. The payment was determined by comparison to an array of ple28. Such a tendency to discount prior information is indeed five thresholds that were set according to the {0.5, 0.6, 0.7, 0.8, 0.9} cumulative quantiles of the empirical score distribution across prior participants. A higher reminiscent of the biases that emerge when the representativeness score S corresponds to a better performance so that participants were payed an heuristic is used16,17. However, as we have shown with a model additional bonus of {1, 2, 3, 4, 5} € if their final score was higher or equal to the that assumes a different block tendency (Fig. 8), all these differ- quantile thresholds. This is a way of rewarding their efforts to optimize their ences could be attributed to mismatched assumptions about the responses. prior distribution of the context. In general, when comparing Written task instruction explained that we would score their responses with respect to the chances that their decision would be correct and that bonus behavior against normative approaches, the interpretation of payments would be based on that score. Additionally, they were informed that their deviations should consider as much as possible the internal score was to be compared to the other participants and that the experimenter could assumptions, constraints and motivations that the participant monitor their behavior on-line via a second screen from outside. obeys46,47. Accounting for such differences might be crucial to interpret and possibly account for many cognitive biases27,48. Stimuli & responses. The task was presented on an LCD screen with a computer Beyond these differences in the central inference stage, there running Matlab Psychtoolbox 3.0.12. Immediately after trial onset, our participants could be alternative sources of distortions in the conversion from were shown the sample consisting of red and blue solid circles arranged on a two- dimensional grid about the screen center (Fig. 1a). The only feature that dis- the estimate into a motor report. Taking into account such tinguished the sampled passengers was the dot color that we chose to be either blue distortions allowed to capture some other part of the departure or red. Because the positions of the dots are communicated not to be informative, of our participants’ behavior to the optimal observer the sample is completely summarized by the sufficient statistics. We tried to make 10 NATURE COMMUNICATIONS | (2019)10:5430 | https://doi.org/10.1038/s41467-019-13472-z | www.nature.com/naturecommunications NATURE COMMUNICATIONS | https://doi.org/10.1038/s41467-019-13472-z ARTICLE the number of dots (sufficient statistics in our task) easily perceptible while making each of the 5 trials within a block is their locations appear as random as possible. Adequate grid spacing was introduced to prevent the circles from overlapping. Furthermore, we kept red and blue samples pðNB ; N; μjν 1 ; ν 2 ; bÞ / BinðNB jN; μÞ  Cat ðNj1=n; ¼ ; 1=nÞ  pðμjν 1 ; ν 2 ; bÞ : ð2Þ separate along the horizontal direction (details in SI). The geometric placement on the screen is not considered to be part of The display is static until the participant makes a response by clicking the USB- the generative model as we assume that only the sufficient statistics matter. mouse which clears the display of the sample. After a short delay of 300 ms, the The expression in Eq. (2) defines the probability distribution for the sufficient program shows a centered horizontally elongated response bar of random statistics of the observations of trial t to which we refer more concisely by horizontal extent with a vertical line marking its center. In addition, the response pðqt ; Nt ; μt jb; ν 1 ; ν 2 Þ, thus equivalently expressing it in terms of each trial’s sample cursor (Fig. 1a, orange vertical line) is shown at a random and uniformly proportion q ¼ NB =ðNB þ NR Þ of the number of blue (NB) and red (NR) distributed initial horizontal position along the response bar. Participants can passengers, and the sample size N = NB + NR. We drop the conditioning on the adjust the horizontal position of the response cursor by moving the mouse parameters of the categorical distribution over sample sizes to keep the notation horizontally and confirm the input with a click to report their choice about the uncluttered. Using this expression, the entire sampling distribution over all airplane’s passenger majority and their subjective confidence in its correctness. The variables of all trials within a block is: movement range of the response cursor was bounded to the horizontal extent of the response bar. The raw response is linearly mapped onto an interval between Q 5 pðq1 ; ¼ ; q5 ; N1 ; ¼ ; N5 ; μ1 ; ¼ ; μ5 ; bjν 1 ; ν 2 Þ ¼ pðbÞ pðqt ; Nt ; μt jb; ν 1 ; ν 2 Þ : [0,1] and interpreted as the confidence in a blue trial majority y. Consequently, the t¼1 corresponding quantity for the confidence in a red majority is 1 − y. ð3Þ Note that given the block tendency b, the per-trial quantities, such as μt, are Experiment 1: Procedure & instructions. First, participants read detailed written conditionally independent. instructions of the task. We introduced the task metaphor that relates to judging the (hidden) majority of passengers on a flight and used it to explain the mathe- matical assumptions in more intuitive terms (see Supplementary Methods). Experiment 2: Procedure & Instructions. Experiment 2 comprises the sessions 2 Additionally, our participants were given 30 trials to familiarize with the and 3 and was carried out with the same 25 participants as in Experiment 1 handling of the task. The subsequent experimental session (session 1) consisted of (session 1). Despite the hierarchical extension across blocks of five trials, the 280 trials with pauses together with feedback after every 5 trials. The sample sizes N handling of the task and the presentation of the sample is virtually the same. The were independent and identically distributed (i.i.d.) samples from {3, 5, 7,…, 13} changes to the latent structure should lead to a different interpretation of the while the hidden airplanes’ passenger proportions μ were i.i.d. samples from a Beta information which we attempted to convey by an extension of the task metaphor (4,4) distribution. Then, the number of blue passengers of the sample is determined (see Supplementary Methods) and written task instructions. by a draw from a Binomial distribution NB ~ Bin(N,μ). After each trial, the As for Experiment 1 and prior to starting session 2, participants completed two participant receives feedback about the correctness of his decision (whether the very short training sessions. First, they were given 20 trials (4 blocks) with a strong cursor was placed on the side corresponding to the underlying passenger majority) and visually obvious block tendency (sample sizes f8; ¼ ; 11g, block tendency but no supervising feedback regarding his confidence estimate. In addition, a two Beta(15,7)). Then another 30 trials under slightly harder conditions (sample sizes second time-out was presented for incorrect decisions which is signaled by a f3; ¼ ; 11g, block tendency Beta(15,7)). Importantly, this only permits them to horizontal ‘progress bar’ which linearly diminishes over time indicating the fraction understand the structure of the reasoning task, such as the dependence between the of the waiting time left. During time-out, there is nothing a participant can do to variables, and get familiarized with the task environment in increasingly difficult proceed but wait. In principle, the correctness feedback could be used by conditions. We intentionally provided as little information as possible as to how participants to learn the mapping from stimuli to the probability of selecting the they should respond. The important point was to make clear what the structure of correct category. In practice however, subject behavior was found to be very stable the process was that generated the samples. Thus, we did not monitor their from the first test trial and throughout the session (Supplementary Fig. 6). performance, nor give them any feedback about how specifically they should place Every five trials, a pause screen was shown which provided information about the response cursor. They could, however, ask the experimenter to clarify the how many out of all trials had already been completed. To motivate engagement in assumptions behind the task. We proceeded to the actual experimental session the task, we gave motivational feedback as an average hSi of the score S (distance to when our participants reported that they had ‘understood’ the task. The above- optimal observer, see above) over the last 5 trials since the last pause. Such feedback mentioned procedure was clear enough to achieve that, and yet sparse enough not was uninformative as to how subjects should change their behavior to improve to reveal the normative response strategy against which we wanted to compare their score: Because it averaged performance over 5 trials, it was very unlikely they their behavior. could use to learn current mappings and shape future responses (see stability of After familiarizing, our participants completed 270 trials of the experimental participants behavior Supplementary Fig. 6). Additionally, they also received a session 2 with an even more difficult setting of the parameters (sample sizes time-out of a few seconds proportional to 1  hSi. The overall rationale behind the f3; ¼ ; 11g, block tendency Beta(14,9)). On the third session, on a different time-out was to more strongly incentivize task engagement and prevent click- appointment, the participants just continued the instructed task of session 2 for 300 through. trials with identical settings to obtain more data. In Experiment 2 and different from Experiment 1, no feedback nor time-out was provided after each trial. However, as in Experiment 1, every five trials, i.e., Experiment 2: Generative model for the stimuli. In Experiment 2, trials of one after each block in Experiment 2, participants were presented with a pause screen block are tied together because they depend on a common unobserved variable with a score based on the results of the last block and a time-out of a few seconds selecting the context. There were two possible contexts: one biased towards red proportional to 1  hSi. As described before, the purpose was mainly to engage passengers, the other towards blue passengers. To keep the notation simple below, participants with the task. That they may have used this extremely sparse and we use the same variable names for the generative process (Fig. 3a) as for the ideal indirect information to somehow guide future responses in Experiment 2, seems observer (Fig. 3c), although in general, an agent’s representation is not necessarily even more unlikely than in Experiment 1 as participants already showed no signs of the same as the generative process in the environment. First and once for every converging to the normative strategy over time there (Supplementary Fig. 6b) block, the binary variable b governing the prevalence for either red (b = 0) or blue where the task was less complex than in Experiment 2 with several hidden (b = 1) passenger majorities in the airplanes, called block tendency, is drawn from variables. a Bernoulli distribution b~ Bernoulli(0.5). Then for every trial, the unobserved proportion of blue passengers of the airplane μ is drawn from a mixture of two Beta distributions depending on the block tendency b. Ideal observer for Experiment 1. The ideal observer model is assumed to know the actual generative process of the observations. Based on the observed passengers, it infers the most likely airplane proportion. Due to the choice of a conjugate prior distribution p(μ) for the Binomial probabilistic model NB ~ Bin(N, μ) above, pos- pðμjν 1 ; ν 2 ; bÞ ¼ b  Betaðμjv1 ; v2 Þ þ ð1  bÞ  Betaðμjv2 ; v1 Þ : ð1Þ terior inference yields a Beta-distribution over the latent airplane proportion μ. Specifically, to give calibrated responses, i.e., confidence estimates that correspond to the actual odds of making correct decisions, the prior distribution used for The Beta distribution is parameterized by two parameters (v1 = 14, v2 = 9), inference must correspond to the actual base rates specified by Betaðμj4; 4Þ. The chosen such that the resulting  distribution  over the passenger proportion μ is confidence in e.g., a blue trial majority c(B) of an ideal observer can be expressed as skewed. By convention, Beta μt jv1 ; v2 is negatively skewed (v1 ≥ v2) and models a the belief that choosing a blue majority is correct by integrating over the corre- blue block tendency. The greater the expectation v1/(v1 + v2) ≈ 0.609 the more sponding subspace23 of inferred blue majorities. extreme this effect because more airplanes with a majority of blue passengers (μ > 0.5) as opposed to red passengers (μ < 0.5) will be encountered. R1 Once the block tendency b has been selected in a block, sampling of the cðBÞ ¼ 1  cðRÞ ¼ pðμ > 0:5jNB ; NR Þ ¼ BetaðμjNB þ 4; NR þ 4Þdμ ð4Þ 0:5 observed passengers in the following 5 trials within a block proceeded as in Experiment 1. First, the sample size N is determined by an i.i.d. drawn from a uniform categorical distribution CatðNj1=n; ¼ ; 1=nÞ over all n sample sizes Heuristic models for experiment 1. Here we describe two heuristic models that N 2 f3; ¼ ; 11g. Then, the number of blue passengers of the sample is determined humans could use to estimate the probability of blue passenger majority on the by a draw from a Binomial distribution NB ~ Bin(N, μ). Hence, the distribution for airplane. NATURE COMMUNICATIONS | (2019)10:5430 | https://doi.org/10.1038/s41467-019-13472-z | www.nature.com/naturecommunications 11 ARTICLE NATURE COMMUNICATIONS | https://doi.org/10.1038/s41467-019-13472-z 1. Ratio model trial t through point-wise multiplication and proper renormalization ψM. In the ratio model, the response is simply mapped from the proportion of Q T1 blue passengers in the sample NB/(NB + NR) MT ðbÞ ¼ ψ1 pðbÞ mt ðbÞ ð9Þ M t¼1 cðBÞ ¼ σð2NB =ðNB þ NR Þ  1Þ As more evidence is gathered (trials), more factors can be absorbed into the belief about b without having to store data from all previous trials independently as where σ is a sigmoid function with possible distortions that provides output it is efficiently encoded in MT(b). Subsequently, this knowledge serves as top-down in the [0 1] range (see below). constraint on future inferences on the trial level. Consequently, to derive the 2. Difference model probability of a blue trial majority on the current trial, the integration of In the ratio model, the response is mapped from the difference between the momentary evidence (Eq. (6)) can be expressed as number of blue and red passengers in the sample NB − NR. Again the difference is mapped onto the [0 1] range using a sigmoid with possible distortions:   1 X Z1     p μT  0:5jD1 ; ¼ ; DT ¼ M ðbÞ p DT jμT p μT jb dμT ð10Þ ψ b¼f0;1g T cðBÞ ¼ σ ððNB  NR Þ 0:5 Proper normalization for the constants ψ, ψM and ψmt can be obtained analytically (see Supplementary Methods). Distorted reports of internal confidence estimates. Apart from inference, behavior may be influenced by extraneous factors, e.g., due to motor control Heuristic models to estimate the block tendency. Here we describe three constraints. We accounted for those by a nonlinear transformation of the con- heuristic models that humans could use to estimate the block tendency. fidence estimate c 2 ½0; 1 onto our model’s prediction of the response ^y. First, we standardize the output c0 ¼ 2ðc  0:5Þ which then enters the argument 1. Averaging model of a logistic sigmoid function through the polynomial Z ¼ ω0 þ ω1 c0 þ ω2 c03 . The computation of the optimal estimate of a blue block tendency from previous trials, MT in Eq. (9), requires marginalization over hidden variables ^y ¼ 1 þ expðZÞ 1 ð5Þ and normalization, which could be computationally difficult. Instead, 0 As we assume symmetry, only odd powers of c are used. In other words, the participants could resort to approximations or heuristics. For the first distorted confidence estimate ^y should lead to the same decision confidence model, the heuristic averaging model, we assume that the estimate of a blue regardless of whether the estimated majority is blue or red. block tendency (b = 1) is approximated by computing the average of the This function is flexible and able to approximate a wide range of distorted presented fractions of blue samples qt ¼ NBt =ðNBt þ NRt Þ in the trials t reports including the identity mapping and various forms of probability prior to the current trial T (T ≥ 2). distortion53,54. It only accounts jointly for all effects which affect the final avg P T1 judgment. Other systematic deviations during confidence estimation which are MT ðb ¼ 1Þ ¼ T1 1 qt ð11Þ t¼1 conditional on a subset of the input space can only be partially accounted for, e.g., deviations for extreme values of the sample proportion. This estimate neglects sample size and corresponds to the implicit assumption that the inferred airplane’s passenger proportion of each trial is well captured Ideal observer for Experiment 2. The ideal observer model (see Fig. 3c) we by a point estimate, i.e., by its respective sample proportion17. The model gives describe here makes use of the generative process described in the main text and the same weight to each trial and thus ignores the fact that some trials provide more information than others due to different sample sizes. As for the other Fig. 3a, b. It updates a probability distribution over the observations of all in-block avg trials and their respective latent variables ðμ1 ; ¼ ; μT Þ up to the current trial T. The models below, indifference is assumed on the first trial MT¼1 ðb ¼ 1Þ ¼ 0:5. avg The way the heuristics top-down message MT is integrated into the parameters (v1,v2) defining the block tendency are part of the generative structure confidence estimation process is described below (see Flexible mapping and assumed to be known. Consequently, inference amounts to an updating of the capturing current and prior information integration). distribution over the latent variables through a calculation of the posterior dis- 2. Tally model tribution conditional on the observations (We identify the distributions by their Similarly, this model computes a tally of all blue samples observed prior to the respective arguments and e.g., write p(D|μ) for the distribution over the sufficient current trial T versus the number of all samples observed in a block so far. statistics of the sample. We often use the abbreviation D = (q,N) for the observa- PT1 tions, omitting parameters and index according to in-block trials t) as tly N MT ðb ¼ 1Þ ¼ PT1t¼1 Bt ð11Þ ðNBt þNRt Þ   Q T t¼1 p μ1 ; ¼ ; μT ; bjD1 ; ¼ ; DT / pðbÞ pðDt jμt Þpðμt jbÞ ð6Þ t¼1 This corresponds to pooling the samples of all trials, as if they were drawn from a common population of unknown population proportion. The current trial is labeled T, and p(b) is the prior probability for block type b 3. Difference model (p(b = 0) = p(b = 1) = 0.5). Note that the probability distributions related to one block are independent to observations from previous blocks: in contrast to change- The heuristic difference model considers the difference between the number of detection task paradigms, the model has explicit knowledge of when a new context blue and red samples dt ¼ NBt  NRt in every observed trial t within a block as start, and does not have to infer it. We checked in a control analysis that responses informative to establish a belief about the block tendency. Across trials, it is were only influenced by response in the same block but were not contaminated by accumulated by computing (T ≥ 2): responses in the previous block (Supplementary Fig. 9). This showed that subjects MTd ðb ¼ 1; ωÞ ¼ P1T1 ð13Þ indeed incorporated the block structure into their inference process. The same 1þexpðω t¼1 dt =ðT1ÞÞ knowledge was incorporated into heuristics models (see below) as well. The logistic sigmoidal function ensures that the result always takes a value We would like to compute the probability of a blue latent trial majority, namely between zero and one and that it can be interpreted as a proper belief, as in the that μT is larger than 0.5. For this purpose, all variables relating to previous trials previous two approximations. The parameter ω adjusts the sensitivity to the which are not of interest must be integrated out. sample-difference statistics dt and can be determined by a fit to behavioral data.   P R1     p μT  0:5jD1 ; ¼ ; DT ¼ ψ1 p DT jμT p μT jb dμT Flexible mapping capturing hierarchical integration. This is a more flexible b¼f0;1g 0:5 ð7Þ extension of the response mapping described before that can be used for the Q R1 T1     hierarchical learning task (Experiment 2). More concretely, we want to integrate  pðbÞ p Dt jμt p μt jb dμt any given prior belief M, not necessarily derived from a probabilistic model, with t¼1 0 the momentary sample D = (q, N) and map it onto the modeled response The constant ψ ensures normalization and can be recovered analytically as ðq; N; M Þ7!^y. As a mere function approximator, it is agnostic to the mechanisms shown below. Because of conditional independence given the block tendency b, the that participants may use to combine information. Correspondingly, its parameters high-dimensional distribution factorizes so that only one-dimensional integrals ω must be determined by a fit to the experimental data. Here, this process is over the latent variables of previous trials must be performed. Examining the graph approximated by a polynomial function Z of the input (q,N,M) that is fed into a structure (see Fig. 3), we see that they may be considered messages mt(b) which are logistic sigmoid as in Eq. (5). passed upwards to update the block-level variable b. Z ¼ ω1 þ ω2 q0 þ ω3 q0 N þ ω4 M þ ω5 q03 þ ω6 q03 N þ ω7 NM 0 þ ω8 M 03 þ ω9 NM 03 1 R 1     mt ðbÞ ¼ ψ mt p Dt jμt p μt jb dμt : ð8Þ ð14Þ 0 The argument Z contains only odd powers of q and M because we assume For proper normalization ψmt, they are themselves probability distributions that symmetry and no preference for estimating either red/blue majorities. convey bottom-up evidence for the block tendency variable b = {0, 1} based on the Correspondingly, both quantities are standardized beforehand by the mapping observations Dt = (qt, Nt) . These bottom-up messages from previous trials within a x0 ¼ 2ðx  0:5Þ. As they are also independent from one another, no corresponding block are integrated to update the belief MT(b) about the block tendency b prior to product terms are included. 12 NATURE COMMUNICATIONS | (2019)10:5430 | https://doi.org/10.1038/s41467-019-13472-z | www.nature.com/naturecommunications NATURE COMMUNICATIONS | https://doi.org/10.1038/s41467-019-13472-z ARTICLE Preliminary testing revealed that the inclusion of nonlinear terms is indicate large differences on the group level. We consider values of pexc ≥ important to capture finer-grained patterns of behavior. The sample size N is 0.95 significant (marked with *) and values of pexc ≥ 0.99 very significant (marked introduced into some terms to model its magnifying effect for the signed with **). quantities (q, M). We performed a weight normalization by the SD of each polynomial (for the input data) which was absorbed into the indicated weights ω. Regression for sample size dependence. Separate regression analyses conditional The particular choice of the terms in Eq. (14) balances flexibility with model on sample size N are used to determine the slope of the psychometric curves of the complexity (and optimization for scarce behavioral data). We manually tested confidence judgments in a blue trial majority over the sample proportion of blue different parameterizations but did not find crucial differences for other samples q (Figs. 1, 2, 6). For a given sample size N, we use a logistic sigmoid with a reasonable choices of the mapping. linear weight ωN to relate the standardized sample proportion q0N ¼ 2ðqN  0:5Þ to the modeled response ^y. Response distribution. We assume that the probability of obtaining the behavioral ^y ¼ 1þexp½ω 1 ð16Þ confidence report yt on trial t conditional on the data dt and the model parameters N q0N  is a Gaussian distribution truncated to the interval from zero to one N½0;1 ðyt j^yt ; θÞ. We note that with this parameterization unbiased judgments are assumed. The mean parameter of the normal distribution is set to the model prediction ^yt . Conditioning reduces the number of data points available for fitting. To avoid The latter is denoted by ^y to distinguish it from the response y of the participant numerical singularities (sigmoid collapses to step function) due to finite data, we which is formally represented by a draw from the response distribution to account use the likelihood function (Eq. (15)) but with the truncated Gaussian replaced by a for task-intrinsic behavioral variability beyond the variations captured by the Gaussian. This choice effectively leads to weighted regression assigning less model. The standard deviation (SD) parameter θ of the Gaussian is assumed to be probability density to responses close to the extremes (e.g., a response of 1 is constant and robustly estimated from the data (see Supplementary Methods). assigned ½ of the density due to spill-over of the Gaussian into [1, ∞)). In this When analyzing the patterns of behavior produced by a fitted model (either a (heuristic) scheme, outlying responses are given less importance which translates heuristic model or the optimal model with distortions), we computed the expected into higher stability of the weight estimate. value of the mode under the truncated gaussian noise. Because of such truncated noise, the expected value is more centered than the noiseless model prediction: Regression for previous trial weights. To estimate the weight on the N ðαÞ  N ðβÞ sample proportion of previously presented in-block trials on the current con- hyt j^yt ; θi ¼ ^yt þ θ fidence estimate we perform a regression analysis (see Figs. 4e and 7a). Prob- ϕðβÞ  ϕðαÞ abilistic integration of evidence for the block tendency M (Eq. (9)) results where α ¼ ^yt =θ and β ¼ ð1  ^yt Þ=θ in a nonlinear increase of aligned confidence with the number of previously As our data might be contaminated by other processes such as lapses, we take observed trials which saturates due to normalization. Hence, as the relative precaution against far outlying responses. The response likelihood is calculated for contribution of each trial decreases as more trials are observed, we perform the all responses as regression analysis separately for different numbers ð2; ¼ ; T  1Þ of predictors Q T (previous trials). pðyjd1 ; ¼ ; dT Þ ¼ ð1  ϵÞN½0;1 ðyt j^yt ; θÞ þ ϵ: ð15Þ P t¼1 ^y ¼ 1 T1 ð17Þ 1 þ exp½ t¼1 ωt q0t  Additionally, to prevent isolated points from being assigned virtually zero As before, we use a logistic sigmoid with a linear combination of standardized probability we generally add a small probability of ϵ ¼ 1:34 ´ 104 to all. This sample proportion q0t ¼ 2ðqt  0:5Þ of each previous trial t to the modeled corresponds to the probability of a point at four standard deviations from the response ^y. Again, this conditioning reduces the number of data points available for standard normal distribution. For non-outlying points this alteration is considered fitting (570/5 = 114 trials) from which up to four weights have to be determined. negligible. To avoid singularity problems common to fitting mixture models, we To avoid numerical singularities due to finite data, we use the likelihood function constrained the SD parameter θ to be larger than 0.01 during fitting. (Eq. (15)) but with the truncated Gaussian replaced by a Gaussian (see above). Inferential patterns for fitted block tendency. The probabilistic model assumes Evidence-opposing choices due to contradictory prior. Evidence-opposing that the block tendency from which the trial-by-trial (airplane) proportions μ are choices are a crucial prediction of the ideal observer model which occur when the drawn is given by one of two skewed Beta-distributions (see Methods). By con- prior belief overrides contradictory evidence from the current sample. If we e.g., vention a ‘blue’ (b = 1) context is characterized by the block tendency record a response that reports a blue majority while the sample majority is red, we Betaðμjν 1 ¼ 14; ν 2 ¼ 9Þ while the ‘red’ context (b = 0) is correspondingly denoted call this an evidence opposing choice (confidence judgment). This can be attributed by Betaðμjν 2 ; ν 1 Þ. The two distributions are symmetric with respect to the block to an influence of an opposing prior belief or task-intrinsic response noise (input- aligned trial majorities, ~μb ¼ b  μ þ ð1  bÞ  ð1  μÞ, which immediately follows independent). To avoid biased estimates because of the latter, the analysis is from the property of the Beta distribution: Betað~ μb¼1 jν 1 ; ν 2 Þ ¼ Betað~ μb¼0 jν 2 ; ν 1 Þ. A conditional on trials that on average provide opposing evidence to the sample. We variation of the optimal inference routine (Eqs. (7–9)) is used that allows for only used trials whose aligned sample proportion is smaller than 0.5 as it opposes different values of the parameters v1,v2 governing the block tendency with the the tracked prior belief (on average). restriction that v1 ≥ v2. In addition, the sigmoidal response mapping (Eq. (13)) is Crucially, in Experiment 1, we found that noise basically does not lead to used to allow for nonlinear distortions of the output. evidence opposing choices (see Supplementary Methods). Nevertheless, we make a conservative estimate by comparing behavior to a model whose evidence opposing Estimating model evidence. The evidence that each participant’s data lends to choices just result from noisy responses in the absence of any prior belief tracking. each model is derived from predictive performance in terms of the cross-validation This reference model ^y ¼ ~q þ ϵ just reports the aligned sample proportion ~q plus log likelihood (CVLL). For training, we maximized the logarithm of the response independent noise ε drawn from a truncated Gaussian distribution of standard likelihood (Eq. (15)). To maximize the chances of finding the global maximum deviation SD = 0.1. even for non-convex problems or shallow gradients, every training run first uses a genetic algorithm and then refines its estimate with gradient based search Binning for visualization and analyses. To impose minimal constraints on data (MATLAB ga, fmincon). The CVLL for each participant and model is summarized for visualization (see Figs. 5–7), we plotted the responses by grouping them into by the median of the logarithm of the response likelihood (Eq. (15)) on the test set approximately equally filled bins across participants. The number of bins was across all cross validation (CV) folds (SI). manually chosen to achieve an appropriate trade-off between resolution and noise of the estimated bins values. Importantly, this only affects visualization. Unless Group level comparison. Instead of making the assumption that all participants stated otherwise, the underlying ungrouped data is used for testing. The condi- can be described by the same model, we use a hierarchical Bayesian model selection tional curves in Figs. 6b, c were determined by the cumulative quantiles Q of the method (BMS)55 that assigns probabilities to the models themselves. This way, we sample size distribution (many ≥ Q(0.6), few < Q(0.4)) and (many > Q(0.5), few ≤ Q assume that different participants may be described by different models. That is a (0.5)) respectively. more suitable approach for group heterogeneity and outliers which are certainly present in the data. The algorithm operates on the CVLL for each participant Reporting summary. Further information on research design is available in ðp ¼ f1; ¼ ; PgÞ and each model ðm ¼ f1; ¼ ; MgÞ under consideration and the Nature Research Reporting Summary linked to this article. estimates a Dirichlet distribution Dirðrjα1 ; :::; αM Þ that acts as a prior for the multinomial model switches upm. The latter are represented individually for each subject by a draw from a multinomial distribution upm  Multð1; rÞ whose para- Data availability meters are rm ¼ αm =ðα1 þ ::: þ αM Þ. We use the CVLL and assume an unin- The data that support the findings of this study are available as Supplementary Data. formative Dirichlet prior a0 = 1 on the model probabilities. P Later, for model R comparison, exceedance probabilities, pexc ¼ 10:5 Betaðαi ; αj Þ, are calculated j≠i Code availability corresponding to the belief that a given model is more likely to have generated the The code for data analysis is available publicly at https://github.com/pschustek/ data than any other model under consideration. High exceedance probabilities empirical_priors. NATURE COMMUNICATIONS | (2019)10:5430 | https://doi.org/10.1038/s41467-019-13472-z | www.nature.com/naturecommunications 13 ARTICLE NATURE COMMUNICATIONS | https://doi.org/10.1038/s41467-019-13472-z Received: 15 October 2018; Accepted: 7 November 2019; 32. Pouget, A., Drugowitsch, J. & Kepecs, A. Confidence and certainty: distinct probabilistic quantities for different goals. Nat. Neurosci. 19, 366–374 (2016). 33. Kiani, R., Corthell, L. & Shadlen, M. N. Choice certainty is informed by both evidence and decision time. Neuron 84, 1329–1342 (2014). 34. Sanders, J. I., Hangya, B. & Kepecs, A. Signatures of a statistical computation in the human sense of confidence. Neuron 90, 499–506 (2016). References 35. Lee, T. S. & Mumford, D. Hierarchical Bayesian inference in the visual cortex. 1. Körding, K. P. et al. Causal inference in multisensory perception. PLoS ONE 2, J. Opt. Soc. Am. A 20, 1434–1448 (2003). e943–e943 (2007). 36. Rao, R. P. N. & Ballard, D. H. Predictive coding in the visual cortex: a 2. Kemp, C. & Tenenbaum, J. B. The discovery of structural form. Proc. Natl functional interpretation of some extra-classical receptive-field effects. Nat. Acad. Sci. USA 105, 10687–10692 (2008). Neurosci. 2, 79 (1999). 3. Friston, K. The free-energy principle: a rough guide to the brain? Trends Cogn. 37. Martino, B. D., Bobadilla-Suarez, S., Nouguchi, T., Sharot, T. & Love, B. C. Sci. 13, 293–301 (2009). Social information is integrated into value and confidence judgments 4. Tenenbaum, J. B., Kemp, C., Griffiths, T. L. & Goodman, N. D. How to grow a according to its reliability. J. Neurosci. 37, 6066–6074 (2017). mind: statistics, structure, and abstraction. Science 331, 1279–1285 (2011). 38. Ma, W. J. Organizing probabilistic models of perception. Trends Cogn. Sci. 16, 5. Pouget, A., Beck, J. M., Ma, W. J. & Latham, P. E. Probabilistic brains: knowns 511–518 (2012). and unknowns. Nat. Neurosci. 16, 1170–1178 (2013). 39. Ma, W. J. & Jazayeri, M. Neural coding of uncertainty and probability. Annu 6. Clark, A. Whatever next? Predictive brains, situated agents, and the future of Rev. Neurosci. 37, 205–220 (2014). cognitive science. Behav. Brain Sci. 36, 181–204 (2013). 40. Green, C. S., Benson, C., Kersten, D. & Schrater, P. Alterations in choice 7. Purcell, B. A. & Kiani, R. Hierarchical decision processes that operate over behavior by manipulations of world model. Proc. Natl Acad. Sci. USA 107, distinct timescales underlie choice and changes in strategy. Proc. Natl Acad. 16401–16406 (2010). Sci. USA. https://doi.org/10.1073/pnas.1524685113 (2016). 41. Baker, C. L., Jara-Ettinger, J., Saxe, R. & Tenenbaum, J. B. Rational 8. Meyniel, F., Schlunegger, D. & Dehaene, S. The sense of confidence during quantitative attribution of beliefs, desires and percepts in human mentalizing. probabilistic learning: a normative account. PLoS Comput. Biol. 11, Nat. Hum. Behav. 1, 0064 (2017). e1004305–e1004305 (2015). 42. Battaglia, P. W., Hamrick, J. B. & Tenenbaum, J. B. Simulation as an engine of 9. Friston, K. The free-energy principle: a unified brain theory? Nat. Rev. physical scene understanding. Proc. Natl Acad. Sci. USA 110, 18327–18332 Neurosci. 11, 127–138 (2010). (2013). 10. Doya, K., Ishii, S. & Pouget, A. Bayesian Brain—Probabilistic Approaches to 43. Tversky, A. & Kahneman, D. Belief in the law of small numbers. Psychol. Bull. Neural Coding. (MIT Press, 2007). 76, 105–110 (1971). 11. Nassar, M. R., Wilson, R. C., Heasly, B. & Gold, J. I. An approximately 44. Oaksford, M. & Chater, N. Précis of Bayesian rationality: the probabilistic Bayesian delta-rule model explains the dynamics of belief updating in a approach to human reasoning. Behav. Brain Sci. 32, 69–120 (2009). changing environment. J. Neurosci. 30, 12366–12378 (2010). 45. Tversky, A. & Kahneman, D. The framing of decisions and the psychology of 12. Orbán, G., Fiser, J., Aslin, R. N. & Lengyel, M. Bayesian learning of visual choice. Science 211, 453–458 (1981). chunks by human observers. Proc. Natl Acad. Sci. USA 105, 2745–2750 (2008). 46. Schwartenbeck, P. et al. Optimal inference with suboptimal models: addiction 13. Payzan-LeNestour, E. & Bossaerts, P. Risk, unexpected uncertainty, and and active Bayesian inference. Med Hypotheses 84, 109–117 (2015). estimation uncertainty: Bayesian learning in unstable settings. PLoS Comput. 47. Rahnev, D. & Denison, R. N. Suboptimality in perceptual decision making. Biol. 7, e1001048–e1001048 (2011). bioRxiv. https://doi.org/10.1101/060194 (2018). 14. Meyniel, F. & Dehaene, S. Brain networks for confidence weighting and 48. Jarvstad, A., Hahn, U., Rushton, S. K. & Warren, P. A. Perceptuo-motor, hierarchical inference during probabilistic learning. Proc. Natl Acad. Sci. USA cognitive, and description-based decision-making seem equally good. Proc. 114, E3859–E3868 (2017). Natl Acad. Sci. USA 110, 16271–16276 (2013). 15. Zylberberg, A., Wolpert, D. M. & Shadlen, M. N. Counterfactual Reasoning 49. Beck, J. M., Latham, P. E. & Pouget, A. Marginalization in neural circuits with Underlies the Learning of Priors in Decision Making. Neuron 1–15 (2018) divisive normalization. J. Neurosci. 31, 15310–15319 (2011). https://doi.org/10.1016/j.neuron.2018.07.035. 50. Moreno-Bote, R., Knill, D. C. & Pouget, A. Bayesian sampling in visual 16. Tversky, A. & Kahneman, D. Judgment under uncertainty: heuristics and perception. Proc. Natl Acad. Sci. USA 108, 12491–12496 (2011). biases. Science 185, 1124–1131 (1974). 51. Pitkow, X. & Angelaki, D. E. Inference in the brain: statistics flowing in 17. Kahneman, D. & Tversky, A. Subjective probability: a judgment of redundant population codes. Neuron 94, 943–953 (2017). representativeness. Cogn. Psychol. 3, 430–454 (1972). 52. Gershman, S. J., Horvitz, E. J. & Tenenbaum, J. B. Computational rationality: a 18. Griffin, D. & Tversky, A. The weighing of evidence and the determinants of converging paradigm for intelligence in brains, minds, and machines. Science confidence. Cogn. Psychol. 24, 411–435 (1992). 349, 273–278 (2015). 19. Kareev, Y., Arnon, S. & Horwitz-Zeliger, R. On the misperception of 53. Gonzalez, R. & Wu, G. On the shape of the probability weighting function. variability. J. Exp. Psychol. Gen. 131, 287–297 (2002). Cogn. Psychol. 38, 129–166 (1999). 20. Gigerenzer, G. & Gaissmaier, W. Heuristic decision making. Annu Rev. 54. Prelec, D. The probability weighting function. Econometrica 66, 497–527 Psychol. 62, 451–482 (2011). (1998). 21. Kahneman, D. Thinking, fast and slow. (Farrar, Straus and Giroux, 2011). 55. Stephan, K. E., Penny, W. D., Daunizeau, J., Moran, R. J. & Friston, K. J. 22. Behrens, T. E. J. et al. What is a cognitive map? Organising knowledge for Bayesian model selection for group studies. NeuroImage 46, 1004–1017 flexible behaviour. bioRxiv. https://doi.org/10.1101/365593 (2018). (2009). 23. Moreno-Bote, R. Decision confidence and uncertainty in diffusion models with partially correlated neuronal integrators. Neural Comput. 22, 1786–1811 (2010). Acknowledgements 24. Drugowitsch, J., Moreno-Bote, R., Churchland, A. K., Shadlen, M. N. & P.S. was supported by a FI-AGAUR scholarship of the Secretariat for Universities and Pouget, A. The cost of accumulating evidence in perceptual decision making. Research of the Ministry of Business and Knowledge of the Government of Catalonia and J. Neurosci. 32, 3612–3628 (2012). the European Social Fund. R.M.-B. is supported by BFU2017-85936-P and FLAGERA- 25. Glaze, C. M., Kable, J. W. & Gold, J. I. Normative evidence accumulation in PCIN-2015-162-C02-02 from MINECO (Spain) and Howard Hughes Medical Institute unpredictable environments. eLife 4, e08825 (2015). (HHMI, ref 55008742). A.H. is supported by the Jovenes Investigadores Grant from 26. Drugowitsch, J., Moreno-Bote, R. & Pouget, A. Relation between belief and Spanish Mineco (PSI2015-74644-JIN). This work was supported by CERCA Programme/ performance in perceptual decision making. PLoS ONE 9, e96511–e96511 Generalitat de Catalunya. The funders had no role in study design, data collection and (2014). analysis, decision to publish or preparation of the manuscript. We are highly thankful to 27. Beck, J. M., Ma, W. J., Pitkow, X., Latham, P. E. & Pouget, A. Not noisy, just Alessia Cavallo for supporting experimental data acquisition. wrong: the role of suboptimal inference in behavioral variability. Neuron 74, 30–39 (2012). 28. Schustek, P. & Moreno-Bote, R. Instance-based generalization for human Author contributions judgments about uncertainty. PLOS Comput. Biol. 14, 1–27 (2018). P.S. and R.M conceived the study. P.S. designed the task, ran the experiments, devised 29. Ernst, M. O. & Banks, M. S. Humans integrate visual and haptic information and implemented the models. P.S. and A.H. performed the data analyses. P.S., A.H. and in a statistically optimal fashion. Nature 415, 429–433 (2002). R.M. discussed the results and wrote the manuscript. 30. Jacobs, R. A. & Kruschke, J. K. Bayesian learning theory applied to human cognition. Wiley Interdiscip. Rev. Cogn. Sci. 2, 8–21 (2011). 31. Fleming, S. M. & Lau, H. C. How to measure metacognition. Front. Hum. Competing interests The authors declare no competing interests. Neurosci. 8, 443 (2014). 14 NATURE COMMUNICATIONS | (2019)10:5430 | https://doi.org/10.1038/s41467-019-13472-z | www.nature.com/naturecommunications NATURE COMMUNICATIONS | https://doi.org/10.1038/s41467-019-13472-z ARTICLE Additional information Open Access This article is licensed under a Creative Commons Supplementary information is available for this paper at https://doi.org/10.1038/s41467- Attribution 4.0 International License, which permits use, sharing, 019-13472-z. adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Correspondence and requests for materials should be addressed to P.S. or A.H. Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless Peer review information Nature Communications thanks Bahador Bahrami and the indicated otherwise in a credit line to the material. If material is not included in the other, anonymous, reviewers for their contribution to the peer review of this work. Peer article’s Creative Commons license and your intended use is not permitted by statutory reviewer reports are available. regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/ Reprints and permission information is available at http://www.nature.com/reprints licenses/by/4.0/. Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. © The Author(s) 2019 NATURE COMMUNICATIONS | (2019)10:5430 | https://doi.org/10.1038/s41467-019-13472-z | www.nature.com/naturecommunications 15