SERIEs https://doi.org/10.1007/s13209-019-0194-8 ORIGINAL ARTICLE Which estimator to measure local governments’ cost efficiency? The case of Spanish municipalities Isabel Narbón-Perpiñá1 · Maria Teresa Balaguer-Coll2 · Marko Petrović3 · Emili Tortosa-Ausina3 Received: 7 August 2018 / Accepted: 4 June 2019 © The Author(s) 2019 Abstract We analyse overall cost efficiency in Spanish local governments during the crisis period (2008–2015). To this end, we first consider some of the most popular nonparametric methods to evaluate local government efficiency, data envelopment analysis and free disposal hull, as well as recent proposals, namely the order-m partial frontier and the nonparametric estimator proposed by Kneip et al. (Econom Theory 24(6):1663–1697, 2008). Second, to compare the four methods and choose the most appropriate one for our particular context and dataset (local government cost efficiency in Spain), we carry out an experiment via Monte Carlo simulations and discuss the relative performance of the efficiency scores under various scenarios. Our results suggest that there is no one approach suitable for all efficiency analysis. We find that for our sample of 1846 Spanish local governments, the average cost efficiency would have been between 0.5417 and 0.7543 during the period 2008–2015, suggesting that Spanish local governments could have achieved the same level of local outputs with about 25% and 46% fewer resources. Keywords Efficiency · Local government · Monte Carlo simulations · Nonparametric frontiers JEL Classification C14 · C15 · H70 · R15 We thank Víctor Giménez, Diego Prior, and José Luis Zafra for helpful suggestions. We are particularly grateful to two anonymous reviewers whose comments have contributed to improve the overall quality of the paper. All authors acknowledge the financial support of the Ministerio de Economía y Competitividad (ECO2017-88241-R and ECO2017-85746-P), Generalitat Valenciana (PROMETEO/2018/102) and Universitat Jaume I (UJI-B2017-33 and UJI-B2017-14). The usual disclaimer applies. B Emili Tortosa-Ausina tortosa@uji.es Extended author information available on the last page of the article 123 SERIEs 1 Introduction Managing available resources efficiently at all levels of government (central, regional, and municipal) is essential, particularly in times of crisis, such as the one that until recently had serious effects on several European countries. Given that increasing taxes and deficit is politically costly (Doumpos and Cohen 2014), a reasonable way to operate in difficult circumstances (not only during crises) is to improve economic efficiency (De Witte and Geys 2011). Since local regulators must provide the best possible services at the lowest cost, developing a system for evaluating local government per- formance that allows benchmarks to be set over time could have relevant implications (Ferreira Da Cruz and Cunha Marques 2014). As a consequence, local government efficiency has attracted much scholarly interest in the field of public administration and the literature now is extensive (see, for comprehensive reviews, Narbón-Perpiñá and De Witte 2018a, b). However, despite the high number of empirical contributions, a major challenge is the lack of clear, standard methodology to evaluate municipalities’ efficiency. Although this problem is well known in the efficiency measurement literature, in local government most studies focus on one approach only, few have attempted to use two or more methods for comparative purposes. For instance, De Borger and Kerstens (1996a) analysed local governments in Belgium using five different reference tech- nologies, two nonparametric (data envelopment analysis or DEA, and free disposal hull or FDH) and three parametric frontiers (one deterministic and two stochastic). They found large differences in the efficiency scores for identical samples and, as a consequence, suggested using different methods to control for the robustness of results whenever the problem of choosing the “best” reference technology is unsolved. Other studies drew similar conclusions after comparing the efficiency estimates of DEA and the stochastic frontier approach (SFA),1 or DEA and FDH or other nonparametric variants.2 Some contributions have also attempted to compare parametric and non- parametric efficiency methods (mainly SFA with DEA or other DEA variants) by using Montecarlo simulations.3,4 However, none of these studies concluded that there is a superior method (Andor and Hesse 2014). Since there is no obvious way to choose an efficiency estimator, the method selected may affect the efficiency analysis (Geys and Moesen 2009b) and could lead to biased results. Therefore, if local government decision makers set a benchmark based on an “incorrect” efficiency score, the economic impact might not be neutral, mislabelling some inefficient municipalities as efficient and vice versa. Hence, although we note that the measurement technique might be not entirely neutral in the case of local 1 Athanassopoulos and Triantis (1998), Worthington (2000), Geys and Moesen (2009b), Boetti et al. (2012), Nikolov and Hrovatin (2013), Pevcin (2014). 2 Balaguer-Coll et al. (2007), Fogarty and Mugera (2013); El Mehdi and Hafner (2014). 3 Some important studies in this area are those of Banker et al. (1993), Banker et al. (1996), Ruggiero (1999), Resti (2000), Ruggiero (2007), Krüger (2012), Badunenko et al. (2012) and Andor and Hesse (2014). 4 Essentially, these methods use generated artificial data, to investigate situations that influence the perfor- mance of the efficiency estimators under different assumptions by comparing the “true” efficiency levels to those estimated through one or more of these techniques (Resti 2000). 123 SERIEs governments’ efficiency, one should ideally report efficiency scores that are more reliable, or closer to the “truth” (Badunenko et al. 2012).5 The present investigation addresses these issues by comparing four nonparametric methodologies and uncovering which measures might be more appropriate to assess local government cost efficiency in Spain. In other words, we attempt to ascertain which method leads to the most reliable results when evaluating cost efficiency in our particular dataset. The study contributes to the literature in three specific aspects. First, we seek to compare four nonparametric methodologies that cover traditional and more recently developed nonparametric proposals, namely DEA, FDH, the order-m partial frontier (Cazals et al. 2002) and the bias-corrected DEA estimator proposed by Kneip et al. (2008). These techniques have been widely used in the previous literature, but little is known about their performance in comparison with each other. Second, we attempt to determine which of these methods should be applied to measure cost efficiency in a given situation. In contrast to previous local government efficiency literature, which has regularly compared techniques and made alternative proposals, we carry out an experiment via Monte Carlo simulations, identifying those methods that perform better in different settings. In doing so, we adapt the simulated technology in order to adequately describe the characteristics of the local government sector by employing a cost function setting with multiple outputs. Finally, based on the simulation results, we discuss the relative performance of the efficiency estimators under various scenarios and seek to determine which method should be used in each one. Our final contribution is to identify which methodologies perform better with our particular dataset. From the simulation results, we determine in which scenario our data lies in, and follow the suggestions related to the performance of the estimators for this scenario. Therefore, we use a consistent method to choose an efficiency estimator, which provides a significant contribution to previous literature in local government efficiency. We use a sample of 1846 Spanish local governments of municipalities between 1000 and 50,000 inhabitants for the period 2008–2015. While other studies based on Spanish data (as well as data from other countries) focus on a specific region or year, our study examines a much larger sample of Spanish municipalities comprising various regions for several years. The sample is also relevant in terms of the period analysed. The economic and financial crisis that started in 2007 had a huge impact on most Spanish local government revenues and finances in general. In addition, the budget constraints became stricter with the law on budgetary stability,6 which introduced greater control over public debt and public spending. Under these circumstances, issues related to Spanish local government efficiency have gained relevance and momentum. Evaluation techniques give the opportunity to identify policy programmes that work, to analyse aspects of a programme that can be improved, and to identify other public programmes that do not meet the stated objectives. In fact, gaining more insights into the amount of local government inefficiency might help to further support effective policy measures 5 We will elaborate further on this a priori ambitious expression. 6 Ley General Estabilidad Presupuestaria (2007, 2012), or Law on Budgetary Stability. 123 SERIEs to correct and/or control it. Therefore, it is obvious that obtaining here a reliable efficiency score would have relevant economic and political implications.7 Our results suggest that there is no one approach suitable for all efficiency analysis. When using these results for policy decisions, local regulators must be aware of which part of the distribution is of particular interest and if the interest lies in the efficiency scores or the rankings estimates. We find that for our sample of Spanish local gov- ernments, all methods showed some room for improvement in terms of possible cost efficiency gains, although they present large differences in inefficiency levels. Both DEA and FDH methodologies showed the most reliable efficiency results, according to the findings of our simulations. Therefore, our results indicate that the average cost efficiency would have been between 0.5417 and 0.7543 during the period 2008–2015, suggesting that Spanish local governments could have achieved the same level of local outputs with about 25% and 46% fewer resources. From a technical point of view, the analytical tools introduced in this study make an interesting contribution that exam- ines the possibility of using a consistent method to choose an efficiency estimator, and the results provide evidence on how efficiency could certainly be assessed to provide some additional guidance for practitioners, scholars and policy makers. The paper is organised as follows: Section 2 gives an overview of the methodologies applied to determine the cost efficiency. Section 4 describes the data used. Section 3 shows the methodological comparison experiment and the results for the different scenarios. Section 5 suggests which methodology performs better with our dataset and presents and comments on the most relevant efficiency results. Finally, Sect. 6 summarises the main conclusions. 2 Methodologies Frontier or best practice methods aim to model the frontier of the technology, rather than modelling the average use of the technological possibilities. As indicated by Bogetoft and Otto (2010), this has certain advantages (from a practical point of view), since it is better to “learn from the best” than to “imitate mediocre performance”. In this benchmarking literature, however, and as indicated in Introduction, there is no consensus as to the most appropriate method to measure efficiency.8 We distinguish between nonparametric (such as DEA) and parametric methods (such as SFA), and several subcategories are found within each family of methods. The main difference between the two measurement techniques is that whereas the former 7 In this respect, Law 27/2013, of Rationalisation and Sustainability of the Local Administration (LRSAL, Ley de Racionalización y Sostenibilidad de la Administración Local ) is the most significant reform since Law 7/1985, on Local Government (LBRL, Ley Reguladora de las Bases del Régimen Local ). The three objectives of the Law are: (i) to guarantee the financial sustainability of all public administrations; (ii) to strengthen confidence in the stability of the Spanish economy; and (iii) to strengthen Spain’s commitment to the European Union in terms of budget stability. Thus, the four principles of previous legislation are maintained: budgetary stability, multi-annuality, transparency and effectiveness, and efficiency in the allo- cation of public resources. This contributes to ultimately reinforcing some of the elements of the previous law, introducing three new principles: financial sustainability, responsibility, and institutional loyalty. 8 Some excellent monographs in the field are those by Fried et al. (1993), Färe et al. (1994), Färe et al. (1985), Charnes et al. (1994) and, more recently, Daraio and Simar (2007a), among many others. 123 SERIEs is quite flexible and not subject to the “parametric straitjacket”, it cannot differentiate between noise and inefficiency. In contrast, parametric methods can make such a distinction, but must select functional forms—also for the distribution of efficiency. Therefore, selecting a particular methodology used to boil down to choosing between “the lesser of two evils” (Berger 1993). However, both parametric and nonparametric methods have evolved, and new proposals are now available (some of which have been reviewed in Fried et al. 2008). In this section, we present our four different nonparametric techniques to measure cost efficiency,9 namely, DEA, FDH, order-m and Kneip et al.’s (2008) bias-corrected DEA estimator, which we will refer to as KSW. 2.1 Data envelopment analysis (DEA) and free disposal hull (FDH) Data envelopment analysis (DEA, Charnes et al. 1978; Banker et al. 1984) and free disposal hull (FDH, Deprins et al. 1984) are efficiency measurement techniques which share the underpinnings of adopting a frontier approach, in which efficient units (municipalities) lie on the empirical frontier, whereas the rest are defined as inef- ficient. Both DEA and FDH share their nonparametric nature, as well as the fact of being based on linear programming techniques. We consider an input-oriented DEA model because public sector outputs are established externally (the minimum services that local governments must provide), and it is therefore more appropriate to evaluate efficiency in terms of the minimisation of inputs (Balaguer-Coll and Prior 2009).10 The mathematical formulation for the cost efficiency measurement (Färe et al. 1994) corresponding to DEA aims to minimise costs, for given levels of outputs, by solving the following programme for each local government and each sample year11 : 9 Different types of efficiency can be distinguished, depending on the data available for inputs and outputs: technical efficiency (T E) requires data on quantities of inputs and outputs, while allocative efficiency (AE) requires additional information on input prices. When these two measures are combined, we obtain the economic efficiency, also called cost efficiency (C E = T E · AE). In the public sector, there are often no prices for public goods due to the sector non-market nature (Kalb et al. 2012). In this paper, we measure local government cost efficiency since we have information relative to specific costs, although it is not possible to decompose it into physical inputs and input prices. 10 Due to the multi-input and/or multi-output nature of local governments, considering frontier methods (particularly using nonparametric methods such as DEA) has been a very popular choice. However, as suggested by an anonymous referee, estimating the impact of a chapter of local expenditure on the production of good or service using a panel of municipalities that are homogeneous after controlling for observable and unobservable characteristics is an option that, although not usual in this literature, might also be pondered. Unfortunately, due to the way local governments report their accounting information, it is not possible to allocate the resources (as shares of the budget) to provide a particular service or infrastructure. Even if this information were currently available, it would only be possible to evaluate how efficiently the specific service analysed were provided. 11 Alternatively to this input (or cost minimisation) orientation, an output orientation might also be adopted. However, in public sector studies, in general, and local government, in particular, it is generally adopted an input orientation for a variety of reasons such as their (in several cases) non-controllable nature. 123 SERIEs min(θk ,λ1 ...λn ) θk  n s.t. θk xk − λi xi ≥0 i=1 n −yk, p + λi yi, p ≥0 (1) i=1  n λi =1 p = 1, . . . , P i=1 λi ≥0 i = 1, . . . , n where xk and xi represent the observed inputs (i.e., the total costs) corresponding to municipalities k and i, respectively. Similarly, yk, p and yi, p denote the observed out- puts for unit k and i with respect to output p; λi are the relative weights which describe the relative importance of the unit considered to determine the virtual reference used as a comparison in order to evaluate unit k; n are the total number of observations; and θk represents the cost efficiency coefficient for each k municipality. The constraint n i=1 λi = 1 implies variable returns to scale (VRS), which assures that each DMU is compared only with others of a similar size. The free disposal hull (FDH) estimator proposed by Deprins et al. (1984) is an exten- sion of DEA whose main difference is that the former drops the convexity assumption. Therefore, FDH cost efficiency is defined as follows: min(θk ,λ1 ...λn ) θk  n s.t. θk xk − λi xi ≥0 i=1 n −yk, p + λi yi, p ≥0 (2) i=1  n λi = 1 p = 1, . . . , P i=1 λi ∈ {0, 1} i = 1, . . . , n Finally, the solution to the mathematical linear programming problems (1) and (2) yields optimal values for θk . Local governments with efficiency scores of θ < 1 are inefficient, while efficient units receive efficiency scores of θ = 1. 2.2 Robust variants of DEA and FDH The traditional nonparametric techniques DEA and FDH have been widely applied in efficiency analysis; however, it is well known that they present several drawbacks, such as the influence of extreme values and outliers, the “curse of dimensionality”12 or the difficulty of drawing classical statistical inference. Hence, we also consider two alternatives to DEA and FDH estimators that are able to overcome most of these drawbacks. The first is order-m (Cazals et al. 2002), a partial frontier approach that mitigates the influence of outliers and the curse of dimensionality, and the second is 12 An increase in the number of inputs or outputs, or a decrease in the number of units for comparison, implies higher efficiencies Daraio and Simar (2007a). 123 SERIEs Kneip et al.’s (2008) bias-corrected DEA estimator (KSW), which allows for consistent statistical inference by applying bootstrap techniques. 2.2.1 Order-m Order-m frontier (Cazals et al. 2002) is a robust alternative to DEA and FDH estima- tors that involves the concept of partial frontier. The order-m estimator, for finite m units, does not envelope all data points and is consequently less extreme. In the input orientation case, this method uses as a benchmark the expected minimum level of input achieved among a fixed number of m local governments producing at least out- put level y (Daraio and Simar 2007a). The value m represents the number of potential units against which we benchmark the analysed unit. Following (Daraio and Simar 2007a, p. 72), the order-m input efficiency scores are estimated as follows: 1. First, for a given output level y, draw a sample m with replacement among those yi,m , such that yi,m ≥ y. 2. The FDH estimator is applied by using the sub-sample drawn in step 1, estimating the efficiency coefficient θ̂i . 3. Steps 1 and 2 are repeated B times, obtaining B efficiency coefficients, θ̂ib (b = 1, . . . , B). 4. Finally, we obtain θ̂i,m which is the mean of the estimated B efficiency coefficients:  b θi,m = B1 θ̂i If m goes to infinity, the order-m estimator converges to FDH. The most reason- able value of m is determined as the value for which the super-efficient observations becomes constant (Daraio and Simar 2005). Note that order-m scores are not bounded by 1 as DEA or FDH. A value greater than 1 indicates super-efficiency, showing that the unit operating at the level (x, y) is more efficient than the average of m peers randomly drawn from the population of units producing more output than y (Daraio and Simar 2007a). 2.2.2 Kneip et al.’s (2008) bias-corrected DEA estimator (KSW) The KSW (Kneip et al. 2008) is a bias-corrected DEA estimator which derives the asymptotic distribution of DEA via bootstrapping techniques. Simar and Wilson (2008) noted that DEA and FDH estimators are biased by construction, implying that the true frontier would lie under the DEA estimated frontier. Badunenko et al. (2012) explained that, the bootstrap procedure to correct this bias, based on sub-sampling, “uses the idea that the known distribution of the difference between estimated and boot- strapped efficiency scores mimics the unknown distribution of the difference between the true and the estimated efficiency scores”. This procedure provides consistent sta- tistical inference of efficiency estimates (i.e., bias and confidence intervals for the estimated efficiency scores). In order to implement the bootstrap procedure (based on sub-sampling), first let s = n d for some d ∈ (0, 1), where n is the sample size and s is the sub-sample size. Then, the bootstrap is outlined as follows: 123 SERIEs 1. First, a bootstrap sub-sample Ss∗ = (X i∗ , Yi∗ )i=1 s is generated by randomly drawing (independently, uniformly, and with replacement) s observations from the original sample, Sn . 2. Apply the DEA estimator, where the technology set is constructed with the sub- sample drawn in step (1), to construct the bootstrap estimates θ̂ ∗ (x, y). 3. Steps 1 and 2 are repeated B times, θ̂b∗ (b = 1, . . . , B), using the resulting bootstrap ∗ values to approximate the conditional distribution of s 2/( p+q+1) ( θθ̂ ∗ (x,y) (x,y) −1), which ∗ allows us to approximate the unknown distribution of n 2/( p+q+1) ( θθ(x,y) (x,y) −1). The ∗ values p and q are the output and input, respectively. Moreover, θ and θ represents the true score coming from a known and an unknown data-generating process. The bias-corrected DEA efficiency score, which is adjusted by the s sub-sample size, is given by: θbc = θ ∗ − Bias ∗ (3) where the bias is adjusted by employing the s sub-sample size.    s 2/( p+q+1) 1  ∗ B ∗ Bias = θ̂b − θ ∗ (4) n B b=1 4. Finally, for a given α ∈ (0, 1), the bootstrap values are used to find the quantiles δα/2,s , δ1−α/2,s in order to compute a symmetric 1 − α confidence interval for θ (x, y)   θ̂(x, y) θ̂ (x, y) , (5) 1 + n −2/( p+q+1) δ1−α/2,s 1 + n −2/( p+q+1) δα/2,s 3 Methodological comparison In contrast to the previous literature, in this section, we compare DEA, FDH, order-m, and KSW approaches following the method proposed by Badunenko et al. (2012).13 Our aim is to uncover which measures perform best with our particular dataset, that is, which ones are the most appropriate to measure local government efficiency in Spain in order to provide useful information for local governments’ performance decisions. To this end, we carry out the experiment via Monte Carlo simulations. We first define the data-generating process, the parameters and the distributional assumptions on data, adapted to the local government framework in order to make our simulation more realistic. Second, we consider the different methodologies and take several stan- dard measures to compare their behaviour. Next, after running the simulations, we 13 Badunenko et al. (2012) compared parametric methodologies, represented by the nonparametric kernel SFA estimator of Fan et al. (1996), with nonparametric, represented by the bias-corrected DEA estimator of Kneip et al. (2008). They assess the performance of these estimators via Monte Carlo simulations and discuss which estimator should be employed in various scenarios. Finally, they consider how these estimators work in practice by determining which scenario corresponds to three different data sets. 123 SERIEs discuss the relative performance of the efficiency estimators under the various sce- narios. Finally, we decide which methods are the most appropriate to measure local government efficiency in Spain. 3.1 Simulations Several previous studies analysing local government cost efficiency with parametric techniques used the SFA estimator developed by Aigner et al. (1977) and Meeusen and Van den Broeck (1977) as a model to estimate cost frontiers.14 These studies considered the input-oriented efficiency where the dependent variable is the level of spending or cost, and the independent variables are output levels. As a parametric approach, SFA establishes the best practice frontier on the basis of a specific functional form, most commonly Cobb-Douglas or Translog. Moreover, it allows researchers to distinguish between measurement error and inefficiency term. Following this scheme, we conduct simulations for a production process with one input or cost (c) and two outputs (y1 and y2 ).15 We consider a Cobb-Douglas cost function (CD). For the baseline case, we assume constant returns to scale (CRS) (γ = 1).16 We establish α = 1/3 and β = γ − α.17 We simulate observations for outputs y1 and y2 , which are distributed uniformly on the [1, 2] interval. Moreover, we assume that the true error term (υ) is normally distributed N (0, συ2 ), and the true cost efficiency is T C E = ex p(−u), where u is half-normally distributed N + (0, σu2 ) and independent from υ. We introduce the true error and inefficiency terms in the frontier formulation, which takes the following expression: β c = y1α · y2 · ex p(υ + u), (6) where c is total costs and y1 and y2 are output indicators. For reasons explained in Sect. 2, there is no observable variation in input prices, so input prices are ignored [see, for instance, the studies of Kalb (2012), and Pacheco et al. (2014)]. We simulate six different combinations for the error and inefficiency terms, in order to model various real scenarios. Table 1 contains the matrix of the different scenarios. It shows the combinations when συ takes values 0.01 and 0.05 and σu takes values 0.01, 0.05, and 0.1. The rows in the table represent the variation of the error term (συ ), while the columns represent the variation of the inefficiency term (σu ). The first 14 See, for instance, the studies of Worthington (2000), De Borger and Kerstens (1996a), Geys (2006), Ibrahim and Salleh (2006), Geys and Moesen (2009a, b), Kalb (2010), Geys et al. (2010), Kalb et al. (2012) or Štastná and Gregor (2015), Lampe et al. (2015), among others. 15 As we will see in Sect. 4, local governments are considered as multiproduct organisations in which the joint use of their resources (input or costs) give rise to several services and facilities (outputs). In the experiment, for simplicity, we use a multi-output model with two outputs. 16 In Sect. 3.4, we consider robustness checks with increasing and decreasing returns to scale to make sure that our simulations accurately represent the performance of our methods. 17 We use α = 1/3 given that it is a common value for calibration of the Cobb-Douglas function in the related literature (see for instance Badunenko et al. 2012). Robustness checks concerning different values of α show similar qualitative results to the baseline experiment and are available upon request. 123 SERIEs Table 1 Combinations of error and inefficiency terms in Monte Carlo simulations to model scenarios σu = 0.01 σu = 0.05 σu = 0.1 συ = 0.01 s1: = 1.0 s3: = 5.0 s5: = 10.0 συ = 0.05 s2: = 0.2 s4: = 1.0 s6: = 2.0 We simulate six different combinations for the error (συ ) and inefficiency (σu ) terms, in order to model various real scenarios. The rows represent the variation of the error term, while the columns represent the variation of the inefficiency term. The parameter is the ratio between σu and συ , which sets each scenario row is the case where the variation of the error term is relatively small, while the second row shows a large variation. The first column is the case where the inefficiency term is relatively small, while the second and third columns represent the cases where variation in inefficiency is relatively larger. The parameter, which sets each scenario, is the ratio between of σu and συ . Within this context, scenario 1 is the case when the error and the inefficiency terms are relatively small (σu = 0.01, συ = 0.01, = 1.0), which means that the data has been measured with little noise and the units are relatively efficient, while scenario 6 is the case when the error and the inefficiency terms are relatively large (σu = 0.1, συ = 0.05, = 2.0), which means that the data is relatively noisy and the units are relatively inefficient. For all simulations we consider 2000 Monte Carlo trials, and we analyse two dif- ferent sample sizes, n = 100 and 200.18,19 We note that nonparametric estimators do not take into account the presence of noise; however, we want to check how it affects the performance of our estimators since all data tend to have noise.20 3.2 Measures to compare the estimators’ performance In order to compare the relative performance of our four nonparametric methodologies, we consider the following median measures over the 2000 simulations. We use median values instead of the average, since it is more robust to skewed distributions. n • Bias(T C E) = n1 i=1 (T C Ei − T C Ei ) 1 n • R M S E(T C E) = [ n i=1 (T C E i − T C E i )2 ]1/2  • U pwar d Bias(T C E) = n1 i=1 n 1 · (T C Ei > T C Ei ) n c −n d • Kendall’s τ (TCE)= 0.5n(n−1) where T C E i is the estimated cost efficiency of municipality i in a given Monte Carlo replication (by a given method) and T C E i is the true efficiency score. The bias reports 18 Krüger (2012) notes that the low number of replications, ranging from 5 to 100, is a weakness in most previous studies on Monte Carlo investigation of efficiency measurement methods. 19 To ease the computational process, we use samples of n = 100 and 200 to conduct simulations. In Sect. 3.4, we consider a robustness check with a bigger sample size (n = 500) to ensure that our simulations accurately represent the performance of our data. 20 In Sect. 3.4, we consider a robustness check with no noise to ensure that our simulations accurately represent the performance of our data. 123 SERIEs the difference between the estimated and true efficiency scores. When it is negative (positive), the estimators are underestimating (overestimating) the true efficiency. The R M S E (root-mean-squared error) measures the standard deviation or error from the true efficiency. The upward bias is the proportion of T C E larger than the true efficien- cies. It measures the percentage of overestimated or underestimated cost efficiencies. Finally, the Kendall’s τ test represents the correlation between the predicted and true cost efficiencies, where n c and n d are the number of concordant and discordant pairs in the data set, respectively. This test identifies the differences in the ranking distributions of the true and the estimated ranks. We also compare the densities of cost efficiency across all Monte Carlo simulations (i.e., densities for the TCE and the estimated efficiencies of the four estimators) in order to report a more comprehensive description of the results, not only restrict them to a single summary statistic—the median. We use violin plots to compare the true distribution for different percentiles of our sample.21 For each draw, we sort the data by the relative value of true efficiency and we plot densities at the 5%, 50% and 95% percentiles. Accordingly, we can analyse the performance of each estimator for a specific part of our sample. So, for example, if we were interested in estimating the poorer performers, we would focus on which estimator perform best only at the 5% percentile the efficiency distribution. 3.3 Relative performance of the estimators Table 2 provides baseline results for the performance measures of the cost efficiency with the CD cost function. First we observe that the median bias of the cost efficiency scores is negative in DEA and KSW in all cases. This implies that the DEA and KSW estimators tend to underestimate the true cost efficiency in all scenarios. FDH and order-m present positive median bias except for scenario 2 in FDH, implying a tendency to overestimate the true efficiency. Bias for all methodologies tends to increase with the sample size when the bias is negative, and decrease when the bias is positive, except for order-m in scenarios 1, 3 and 5. The RMSE is smaller when συ is small, except for FDH in scenario 5 and order-m in scenarios 3 and 5. Moreover, the RMSE of the cost efficiency estimates increases with the sample size for all cases except for FDH in scenarios 1, 3, 5 and 6 and order-m in scenarios 5 and 6. We also consider the upward bias. This shows the percentage of observations for which cost efficiency is larger than the true value (returning a value of 1). The desired value is 0.5. The values less (greater) than 0.5 indicate underestimation (overestima- tion) of cost efficiencies. In this setting, DEA and KSW systematically underestimate the true efficiency. Moreover, as the sample size increases, so does the percentage of underestimated results. In contrast, FDH and order-m tend to overestimate the true efficiency, but as the sample size increases overestimated results decrease. Finally, we analyse Kendall’s τ for the efficiency ranks between true and estimated efficiency 21 The violin plot combines the density trace (or smoothed histogram) and the box plot (initially conceived by Tukey) into a single figure that reveals the structure found within the data. The name violin plot originated due to the early studies using these procedures, which resulted in graphics with the appearance of a violin. For details, see Hintze and Nelson (1998). 123 Table 2 Baseline results with Cobb-Douglas cost function Biasa RMSEb Upward biasc Kendall’s τ d DEA FDH Order-m KSW DEA FDH Order-m KSW DEA FDH Order-m KSW DEA FDH Order-m KSW 123 s1: συ = 0.01, σu = 0.01 n = 100 − 0.0298 0.0074 0.0231 − 0.0350 0.0330 0.0096 0.0307 0.0375 0.0500 0.9800 0.9900 0.0100 0.2491 0.1227 0.0615 0.2549 n = 200 − 0.0348 0.0070 0.0287 − 0.0391 0.0376 0.0094 0.0370 0.0415 0.0250 0.9600 0.9850 0.0050 0.2573 0.1590 0.0790 0.2588 s2: συ = 0.05, σu = 0.01 n = 100 − 0.0892 − 0.0111 0.0049 − 0.1006 0.1013 0.0338 0.0399 0.1111 0.0500 0.6400 0.6900 0.0100 0.0681 0.0551 0.0511 0.0687 n = 200 − 0.1028 − 0.0205 0.0043 − 0.1130 0.1134 0.0428 0.0466 0.1225 0.0250 0.5000 0.6200 0.0050 0.0707 0.0597 0.0542 0.0705 s3: συ = 0.01, σu = 0.05 n = 100 − 0.0182 0.0322 0.0477 − 0.0246 0.0238 0.0392 0.0548 0.0285 0.1200 1.0000 1.0000 0.0600 0.6753 0.4443 0.3169 0.6877 n = 200 − 0.0239 0.0289 0.0512 − 0.0293 0.0281 0.0351 0.0581 0.0325 0.0700 0.9900 1.0000 0.0350 0.6843 0.5192 0.3812 0.6911 s4: συ = 0.05, σu = 0.05 n = 100 − 0.0707 0.0133 0.0303 − 0.0832 0.0857 0.0410 0.0528 0.0960 0.0900 0.7500 0.8100 0.0400 0.3060 0.2547 0.2415 0.3059 n = 200 − 0.0849 0.0024 0.0279 − 0.0963 0.0972 0.0421 0.0565 0.1072 0.0500 0.6300 0.7550 0.0250 0.3132 0.2710 0.2564 0.3123 s5: συ = 0.01, σu = 0.1 n = 100 − 0.0101 0.0525 0.0684 − 0.0177 0.0203 0.0624 0.0768 0.0238 0.2000 1.0000 1.0000 0.1000 0.8057 0.5928 0.5146 0.8182 n = 200 − 0.0170 0.0453 0.0689 − 0.0232 0.0230 0.0537 0.0763 0.0275 0.1150 0.9950 1.0000 0.0600 0.8174 0.6586 0.5738 0.8254 SERIEs Table 2 continued SERIEs Biasa RMSEb Upward biasc Kendall’s τ d DEA FDH Order-m KSW DEA FDH Order-m KSW DEA FDH Order-m KSW DEA FDH Order-m KSW s6: συ = 0.05, σu = 0.1 n = 100 − 0.0580 0.0347 0.0519 − 0.0724 0.0755 0.0591 0.0722 0.0869 0.1300 0.8200 0.8800 0.0700 0.4996 0.4170 0.4030 0.4990 n = 200 − 0.0726 0.0207 0.0485 − 0.0854 0.0867 0.0530 0.0717 0.0974 0.0750 0.7200 0.8400 0.0400 0.5065 0.4403 0.4332 0.5065 This table provides the baseline results for the performance of the methodologies in the Monte Carlo experiment. We simulate six scenarios, which represent different combinations for the error (συ ) and inefficiency (σu ) terms a The bias reports the difference between the estimated and true efficiency scores. When it is negative (positive), the estimators are underestimating (overestimating) the true efficiency b The R M S E (root-mean-squared error) measures the standard deviation or error from the true efficiency c The upward bias is the proportion of estimated efficiencies larger than the true efficiencies (returning a value of 1). The desired value is 0.5. The values less (greater) than 0.5 indicate underestimation (overestimation) of cost efficiencies d Kendall’s τ shows the correlation coefficient for the efficiency ranks between true and estimated efficiency scores 123 SERIEs scores. In each scenario and sample size, DEA and KSW have a larger Kendall’s τ ; they therefore perform best at identifying the ranks of the efficiency scores. We also analyse other percentiles of the efficiency distribution, since it is difficult to conclude from the table which methods perform better. Figures 1, 2 and 3 show results for the 5th, 50th and 95th percentiles of true and estimated cost efficiencies. We compare the distribution of each method with the TCE.22 For visual simplicity, we show only the case when n = 100. Figures with sample size n = 200 do not vary greatly and are available upon request. The figures show that results depend on the value of the parameter. As expected, when the variance of the error term increases our results are less accurate. Note that nonparametric methodologies assume the absence of noise. In contrast, when the variance of the inefficiency term relative to the variance of the error term increases, our results are more precise. Under scenario 1 (see Fig. 1a, c, e), when both error and inefficiency terms are relatively small, DEA and KSW methodologies consistently underestimate efficiency (their distributions are below the true efficiency in all percentiles). If we consider median values and density modes, order-m tends to overestimate efficiency in all percentiles, while FDH also tends to overestimate efficiency at the 5th and 50th per- centiles. Moreover, we observe that FDH performs well in estimating the efficiency units in the 95th percentile. Although scenario 4 (see Fig. 2b, d, f) is the opposite case to scenario 1, when both error and inefficiency terms are relatively large, they have the same value of . As in scenario 1, DEA and KSW methodologies consistently underestimate efficiency. On the other hand, we see from the 5th percentile that both FDH and order-m tend to overestimate efficiency. However, at the 50th and 95th percentiles, both methods perform better at estimating the efficiency units since their median values and density modes are closer to the TCE distribution. Similarly, in scenario 2 (see Fig. 1b, d, f), when the error term is relatively large but the inefficiency term is relatively small, DEA and KSW tend to underestimate the true efficiency scores, while FDH and order-m appear to be close to the TCE distribution (in terms of median values and mode). This scenario yields the poorest results as the disper- sion of TCE is much more squeezed than the estimators’ distributions. Therefore, when is small, all four methodologies perform less well in predicting efficiency scores. Scenario 3 (see Fig. 2a, c, e), the error term is relatively small but the inefficiency term is relatively large. Because the value has increased, all methodologies do better at predicting the efficiency scores. At the 5th and 50th percentiles, we observe that DEA and KSW underestimate efficiency, while order-m and FDH tend to overestimate it. However, if we consider the median and density modes, DEA (followed by KSW) is closer to the TCE distribution in both percentiles. At the 95th percentile FDH does better at estimating the efficient units, while DEA and KSW slightly underestimate efficiency and order-m slightly overestimates it. In scenario 5 (see Fig. 2a, c, e), the error variation is relatively small, but the inefficiency variation is very large. This scenario shows the most favourable results 22 We consider that a particular methodology has a better or worse performance depending on the similarities found between its efficiency distribution and the true efficiency distribution. 123 SERIEs Fig. 1 Violin plots in scenario 1 and 2 for 5th, 50th, and 95th percentiles of cost efficiency estimates under a Cobb-Douglas cost function because the TCE distribution is highly dispersed and therefore better represents the estimators’ performance. At the 5th and 50th percentiles, DEA and KSW densities are very close to the true distribution of efficiency, while FDH and order-m overestimate it. In contrast, at the 95th percentile FDH seems to be closer to the TCE although it slightly overestimates it. 123 SERIEs Fig. 2 Violin plots in scenario 3 and 4 for 5th, 50th and 95th percentiles of cost efficiency estimates under a Cobb-Douglas cost function Finally, in scenario 6 (see Fig. 3b, d, f) the error term is relatively large and the inefficiency term is even larger. Again, we observe that when the variation of the inef- ficiency term increases (compared with scenarios 2 and 4), all the estimators perform better. At the 5th and 50th percentiles, DEA and KSW slightly underestimate efficiency and FDH and order-m slightly overestimate it (in terms of median values and density 123 SERIEs Fig. 3 Violin plots in scenario 5 and 6 for 5th, 50th and 95th percentiles of cost efficiency estimates under a Cobb-Douglas cost function mode). However, despite all methods being quite close to the TCE distribution, DEA underestimates less than KSW, and FDH overestimates less than order-m. Finally, at the 95th percentile FDH (followed by order-m) is the best method to determine a higher number of efficient units because its mode and median values are closer to the true efficiency. 123 SERIEs To sum up, in this subsection we have provided the baseline results for the rel- ative performance of our four nonparametric methodologies. We have considered four median measures as well as other percentiles of the efficiency distribution. We found that the performance of the estimators vary greatly according to each particular scenario. However, we observe that both DEA and KSW consistently underestimate efficiency in nearly all cases, while FDH and order-m tend to overestimate it. Moreover, we note that DEA and KSW perform best at identifying the ranks of the efficiency scores. In Sect. 3.5, we will explain in greater detail which estimator to use in the various scenarios. 3.4 Robustness checks We consider a number of robustness checks to verify that our baseline experiment represents the performance of our estimators. Results for each robustness test are given in the extra Annex. • No noise: All our nonparametric estimators assume the absence of noise. However, in the baseline experiment we include noise in each scenario. In this situation, we consider the case where there is no noise in the data-generating process. Results show that DEA and KSW perform better at predicting the efficiency scores, while FDH and order-m are slightly worse than the baseline experiment. All methods perform better at estimating the true ranks, except order-m in scenario 1. In short, we find that when noise is absent, DEA and KSW have a greater performance. • Greater variation in the inefficiency term: In the baseline experiment we set dif- ferent values for the inefficiency term (σu = 0.01, 0.05 and 0.1). We also consider the case in which bigger efficiency shocks exist (σu = 0.2 and σu = 0.3). As expected, we observe some small improvement in the performance of the median measures for DEA and KSW, while the performance for FDH and order-m slightly decrease. All methods perform better when estimating the true efficiency ranks. In general, despite the small quantitative variations, the results of the baseline experiment seem to hold. • Changes in sample size: The baseline experiment analyses two different sample sizes, n = 100 and 200. We also consider the case where the sample size is very large, that is, n = 500. There is a slight deterioration in the performance of DEA and KSW, while FDH and order-m vary depending on the scenario. However, the results only differed slightly. We find no qualitative changes from the baseline results. • Returns to scale: The baseline experiment assumes CRS technology. We also con- sider the case where the technology assumes decreasing and increasing returns to scale (γ = 0.8 and γ = 1.2). We find a slight deterioration in the performance of DEA and KSW estimators. Performance for order-m improves with decreasing returns to scale and deteriorates with increasing returns to scale, while FDH varies depending on the scenario. However, despite these minor quantitative differences, the qualitative results do not change. • Different m values for order-m: Following Daraio and Simar’s (2007a) suggestion, in order to choose the most reasonable value of m we considered different m sizes 123 SERIEs (m = 20, 30 and 40). In our application, the baseline experiment sets m = 30. In general, compared with the other m values, there are some quantitative changes (i.e., performance with m = 20 worsens, while with m = 40 it improves slightly); however, the qualitative results from the baseline case seem to hold. In sort, we find that after considering several robustness checks, we do not see any major differences from the baseline experiment. Therefore, despite the initial assumptions done, our simulations accurately depict the performance of our estimators. 3.5 Which estimator in each scenario Based on the above comparative analysis of the four methodologies’ performance, inspired by our results as well as Badunenko et al.’s (2012) proposal, we summarise which ones should be used in the various scenarios, assuming that the simulations remain true for different data-generating processes. Table 3 suggests which estimators to use for each scenario when taking into account the efficiency scores. The first row in each scenario shows the relative magnitudes of the estimators compared with the True Cost Efficiency (TCE), while the rest of the rows suggest which estimators to use for each percentile (5th, 50th or 95th). In some cases, the methodologies vary little in terms of identifying the efficiency scores. Badunenko et al. (2012) conclude that if the value is small, as in scenario 2 ( = 0.2), the efficiency scores and ranks will be poorly estimated.23 This scenario yields the worst results, since the estimators are far from the “truth”. Although Table 3 suggests scenario 2, we do not recommend efficiency analysis for this particular scenario, since it would be inaccurate. Although scenarios 1 and 4 present better results than scenario 2 (when = 1), estimators also perform poorly at predicting the true efficiency scores. In scenario 1, FDH seems to be the best method to estimate efficiency in all percentiles; however, DEA should also be considered at the 5th percentile (the TCE remains between DEA and FDH at this percentile). Similarly, in scenario 4 FDH predominates at the 5th percentile, although DEA should also be considered. On the other hand, both FDH and order-m perform better at the 50th and 95th percentiles. For efficiency rankings, DEA and KSW methodologies show a fairly good performance when ranking the observations in both scenarios. Similarly, scenario 6 performs better than scenarios 1 and 4, since the variation of the inefficiency term increases and, as a consequence, the value of also increases ( = 2). In this scenario the best methodologies for estimating the true efficiency scores seem to be DEA and FDH at the 5th and 50th percentiles, and FDH (followed by order-m) at the 95th percentile. In contrast, DEA and KSW methodologies are better at ranking the observations. In scenario 3, the value increases again ( = 5), and all the methodologies predict the efficiency scores more accurately. For the 5th and 50th percentiles, the closest estimator to the true efficiency seems to be DEA (followed by KSW). At the 95th percentile FDH is the best method. For the rankings, however, DEA and KSW provide more accurate estimations of the efficiency rankings. 23 It is difficult to obtain the inefficiency from a relatively large noise component. 123 123 Table 3 Relative performance of the efficiency scores at the 5th, 50th and 95th percentiles σu = 0.01 σu = 0.05 σu = 0.1 συ = 0.01 Scenario 1 Scenario 3 Scenario 5 KSW