Measuring Achievement and Shortfall Improvements in a Consistent Way

In measuring improvements over time of bounded variables, one can focus on achievements or shortfalls. However, rankings of alternative social states in terms of achievements and shortfalls do not necessarily mirror one another. We characterize axiomatically different families of achievement and shortfall improvement indices, and present the necessary and sufficient conditions under which they rank social states in a consistent way. Empirical illustrations using child mortality data from South Africa suggest that consistency between achievement and shortfall improvements in standards of living is not only a matter of theoretical import but is also a problem that can be encountered in practice to a large extent.


Introduction
When assessing the standard of living of our societies, it is important to look not only at the prevailing levels of different key socio-economic and development indicators but also to their changes over time. While several studies have explored the dynamics of well-being (e.g., Dasgupta and Weale, 1992;Dasgupta, 1993;Easterly, 1999;Mazumdar, 1999), only a few of them have attempted to define "improvement" or "progress" measures in a rigorous and satisfactory way-an important issue that, as will be argued below, has received insufficient attention from scholars. The only standard of living improvement measures proposed so far that we are aware of are those of Kakwani (1993), which were axiomatically characterized a few years later by Majumder and Chakravarty (1996) and extended to the multidimensional framework by Tsui (1996) and Chakravarty and Mukherjee (1999). Unfortunately, these measures do not address a relevant matter that has been ignored so far in the literature and which will be the main concern of this paper: the problem of "consistently measuring achievement and shortfall improvement." Given the bounded nature of virtually all indices of standard of living (typical examples include health or education variables like life expectancy, child or adult mortality rates, literacy or school attendance rates, educational attainment and so forth), it is a priori possible to focus on the distribution of achievements or on the corresponding distribution of shortfalls with respect to the upper bound when measuring improvements over time. To illustrate: improvements in the coverage of public health plans could be assessed via the percentage of vaccinated children (an achievement indicator) or through the percentage of unvaccinated children (a shortfall indicator). While both approaches seem attractive in their own right, the few formal attempts to measure improvements in standards of living we are aware of have-somewhat strangely-only focused on the changes in the shortfall distributions (Kakwani, 1993;Majumder and Chakravarty, 1996;Tsui, 1996;Chakravarty and Mukherjee, 1999). However, we see a priori no reason to focus exclusively on those distributions and disregard their achievement counterparts. Apparently, both perspectives offer complementary views of the same problem, so it seems important to assess them in relation to each other. In this context, a natural question that might arise is: will improvements in shortfalls and improvements in achievements mirror each other or not? In this paper we will consider these complementary approaches simultaneously and present the conditions under which both classes of measures rank alternative states of affairs in a consistent way.
In the last few years, there has been a bourgeoning debate on the consistent measurement of achievement and shortfall inequality for bounded variables. The potential (and actual) mismatch between certain achievement and shortfall inequality measures was signaled by Clarke et al. (2002), and several authors have attempted to overcome that problem (Erreygers, 2009;Lambert and Zheng, 2011;Lasso de la Vega and Aristondo, 2012). Even if some of the results presented in this paper are inspired in the aforementioned works, there are important differences that are worth pointing out. On the one hand, we are not dealing with inequality but with improvements over time, so the functional forms of the indices we will be working with are completely different. On the other hand, while the notion of "inequality" is the same regardless of whether we are considering achievements or shortfalls (i.e., in both cases we are measuring the spread of a set of numbers), the notion of "improvement" can be considered as being, so to say, directional (as it depends on the end from which one stares at it), so it is important to distinguish between the two perspectives. Hence, while the same inequality index is used to measure attainment and shortfall inequality (the only thing that changes is the domain of the inequality index), we will need to define a specific improvement function for achievements and a specific improvement function for shortfalls. As a consequence, the conditions that are needed to satisfy the attainment-shortfall consistency test presented in this paper will differ with respect to the conditions used in the aforementioned papers.
In order to illustrate the usefulness of our proposal, we present the evolution over time of child health outcomes for the different municipalities in South Africa in the period 2001-07 using census microdata from the Integrated Public Use Microdata Series (IPUMS) project. The rest of the paper is organized as follows.
In Section 2, we present the main axioms and characterize the improvement measures used in the paper. In Section 3, we explore the problem of consistently measuring achievement and shortfall improvements. Section 4 shows the empirical illustration and Section 5 concludes. The proofs are relegated to the Supporting Information.

Axioms and Improvement Measures
In this paper, the different units of analysis {1, . . . , n} ( ) n ∈N will be referred to as "individuals," even if in practice one might actually work with households, neighborhoods, municipalities, countries or any other group. The achievement level of a given individual will be measured with a certain standard of living indicator that will be tracked in two different moments in time (say, T1 and T2, with T1 < T2). We assume that such an indicator is measured in a continuous positive scale-an almost universal assumption in standards of living or well-being measurement-and that its values are naturally bounded from above and below. This last assumption is very common for most indicators that are typically incorporated in standard of living or well-being assessments. For instance: health or education variables (like life expectancy, mortality rates, educational attainment) can not increase or decrease indefinitely, so it is highly plausible to place a lower and upper bound on them. In this paper, the lower and upper bounds will simply be denoted by L and U respectively, with 0 ≤ L < U. It might be worth emphasizing that we are assuming that our achievement indicators can actually attain the values of the lower and upper bounds L, U. This is in contrast with the approach followed in other conceptually related studies, where the upper bound U is assumed to be unattainable (Kakwani, 1993;Majumder and Chakravarty, 1996;Tsui, 1996). Our choice has been motivated by the fact that many variables typically included in the assessment of the standard of living (e.g., literacy rates, enrolment ratios, gender gaps) do very often reach their upper bounds. 1 This apparently minor technicality will have important consequences in the derivation of the functional form of our improvement indices.
In order to measure improvements over time for a given individual we need to introduce some notation. We will denote by x ∈ [L, U] the achievement level of a given unit of analysis in time T1. Analogously, we will denote by y ∈ [L, U] the achievement level of the same unit of analysis in time T2. In this context, we can naturally define the shortfalls associated to achievements x and y as p: = U − x and q: = U − y. Clearly, p, q ∈ [0, U − L]. When it comes to measuring the notion of "improvement," we should first decide whether the latter will be assessed through changes in achievements or in shortfalls. In this respect, the few formal attempts to measure improvements in standards of living we are aware of have only focused on the changes in shortfalls (Kakwani, 1993;Majumder and Chakravarty, 1996;Tsui, 1996;Chakravarty and Mukherjee, 1999). The fact that the corresponding achievements are disregarded in those papers is somewhat surprising, particularly because the existence of both approaches is acknowledged from the start. Since we consider that changes in both achievements and shortfalls are essentially measuring two sides of the same coin, in this paper we will incorporate them simultaneously and show the conditions under which they rank alternative states of affairs in a consistent way. When improvements in standards of living are assessed through changes in achievements, we will introduce a so-called "achievement improvement index." Formally: an achievement improvement index ι a is defined as a non-trivial real-valued function ι a : [L, U] 2 → ℝ. The values of ι a (x, y) should be interpreted as the improvement in standard of living of a given unit of analysis when the corresponding achievement changes from x to y. Analogously, a shortfall improvement index ι s is defined as a non-trivial real-valued function ι s : [0, U − L] 2 → ℝ and its values ι s (p, q) should also be interpreted as the improvement in standard of living observed when the shortfall changes from p to q.
Interestingly, the fact of introducing different indices to measure achievement and shortfall improvements (i.e., ι a and ι s ) is in contrast with the approach followed in the measurement of achievement and shortfall inequality. In the latter case, the same inequality index D is used to measure both concepts; the only thing that changes is the domain (i.e., one compares inequality of a distribution of achievements D(x1, . . . , xn) vis-à-vis inequality of the corresponding distribution of shortfalls D(U − x1, . . . , U − xn)). As mentioned in the introduction, the notion of "inequality" is the same regardless of whether we are considering one distribution or the other, but the notion of "improvement" depends on whether we use achievements or shortfalls.
Given the fact that the achievement and shortfall improvement measures are based on the same underlying ideas, our axioms will be presented in the following way. We will first present the general intuition behind the corresponding axiom and then formally show how this idea translates into certain restrictions for the ι a and ι s functions separately. Our first axiom reads as follows.
Continuity (CN): The improvement indices are continuous functions in their domains, that is: ι a and ι s are continuous. This is an extremely common assumption in the literature of socio-economic indices. It requires that small changes in the achievements or shortfalls of individuals produce small changes in the corresponding improvement function. Stated otherwise: the change in standard of living does not abruptly change as individuals' achievements or shortfalls are slightly altered. Among other things, this property ensures that our measures will not be dramatically affected by measurement errors.
Monotonicity (MN): When a given unit of analysis sees its standard of living increasing from T1 and T2, then the corresponding improvement index should increase. In other words, ι a (x, y) is increasing in y and ι s (p, q) is increasing in p.
This assumption is quite unexceptionable for any index attempting to measure improvements in standard of living between any two moments in time.
Homotheticity (HM): When comparing and ranking two alternative states of affairs in terms of improvements, the results should remain unaltered when the corresponding achievements or shortfalls are changed by some multiple λ. Formally: for all x1, y1, x2, y2 ∈ [L, U] and all λ > 0 such that λx1, λy1, λx2, λy2 ∈ [L, U], one has that Analogously for ι s : for all p1, q1, p2, q2 ∈ [0, U − L] and all λ > 0 such that λ p 1 , λq 1 , ], one has that It is important to note that Homotheticity should be interpreted as an ethical judgment with respect to improvement and not as a change in the measurement scale. Therefore, it should not be confused with other similar properties suggested in the literature, like "Scale Invariance" or "Unit Consistency." 2 These important differences have already been highlighted in Tsui (1996, p. 296).
Upward Sensitivity (US): Other things being equal, an improvement index should reward those improvements occuring at higher achievement levels. Formally: consider a hypothetical scenario with L ≤ z < w ≤ U. Take now any δ > 0 such that z + δ, w + δ ∈ [L, U]. Then one has that Downward Sensitivity (DS): Other things being equal, an improvement index should reward those improvements occuring at lower achievement levels. Formally: consider a hypothetical scenario with L ≤ z < w ≤ U. Take now any δ > 0 such that z + δ, w + δ ∈ [L, U]. Then one has that . Then one has that Even if this property has never been used in the improvement indices suggested so far, it places more value on improvements leading to more egalitarian distributions. To illustrate: if a donor agency successfully invests limited resources in increasing the life expectancy of a country from 40 to 45 rather than in increasing life expectancy for another country from 50 to 55, the resulting distribution will be less unequal and might therefore be more desirable in order to enhace overall welfare.
Additivity (AD): For any three periods, the improvement from period T1 to T3 may be expressed as the sum of the improvement from T1 to T2 and that from period T2 to T3. Formally: consider x, y, z ∈ [L, U]. Then According to Additivity, the evaluation of the changes in living standards between two moments in time depends exclusively on these two moments in time and is not affected by the intermediate changes that might have occurred in between. This is a quite standard assumption that has already been introduced in the works of Kakwani (1993), Majumder and Chakravarty (1996), Tsui (1996) and Chakravarty and Mukherjee (1999). 3 One of the consequences of AD is that when no changes occur between T1 and T2, then our improvement measures should take the value of 0 (i.e., when Multiplicativity (MU): For any x, y, z ∈ [L, U] one has that According to Multiplicativity, if there is an α-fold improvement in standards of living from T1 to T2 and a β-fold improvement from T2 to T3, then there has been a αβ-fold improvement from T1 to T3. In contrast to Additivity, this axiom captures a complementary non-additive intuition that is also commonly used when assessing changes over time.
In order to render results easily interpretable, in some cases one normalizes the values of our improvement indicators between well-known bounds. Normalization stipulates that the improvement functions for a society with (n =)1 individual take their maximal value-equal to A-whenever the achievement indicator starts at its lowest level in T1 and ends up at its highest level in T2. While somewhat arbitrary and without universal application, the practice of bounding the values of socio-economic indicators is quite extensive in the literature. Standard and simple choices for such upper bounds could be 1, 10 or 100.
Combining these different axioms, we are able to characterize our achievement and shortfall improvement indices. We first present a quite general characterization result using a short list of "core axioms" that is further refined with the incorporation of the Upward and Downward Sensitivity and Normalization axioms.

Theorem 1. An achievement improvement index ι a satisfies the "achievement version" of the axioms CN, MN, HM and AD if and only if it can be written either as
for all x, y ∈ [L, U] (x, y > 0 for ι +2 a ) and for some real parameters α > 0, C1 > 0, C2 > 0. Analogously, a shortfall improvement index ι s satisfies the "shortfall version" of those axioms if and only if it can be written either as Remark 1. Interpretation of the indices. Theorems 1 and 2 characterize axiomatically additive and multiplicative improvement indices respectively. The constants Ci from equations (11)-(14) are normalization parameters that can be chosen so that the range of values of our improvement indices fall within a pre-specified interval (see Remark 3 below for more details). When no change at all is observed between times T1 and T2, ι +1 a , ι +2 a , ι +1 s and ι +2 s take the value of 0. Therefore, values of the additive indices above (resp. below) 0 should be interpreted as an improvement (resp. worsening)  As can be seen in Tables 2a and 2b, parameters α, β, ζ and ξ regulate the extent to which improvements at higher or lower achievement levels are given more importance or not for some of the indices shown in Table 1 (i.e., they regulate whether the indices comply with the US and DS axioms or not). The farther away these parameters are from the value of 1, the more our measures will reward those improvements occuring in different parts of the distribution. At the other extreme,   whenever α = β = 1 or when ζ = ξ = 1, our measures will not be sensitive to the places where improvements take place. Interestingly, when α = β = 1 and C1 = C3 it turns out that ι ι

Remark 4. Relationship with other measures. To our knowledge, the improvement indices characterized in Theorems 1 and 2 are the first measures of their kind that explicitly incorporate the achievement and shortfall perspectives in a common framework. When choosing A = 1, our shortfall improvement indices ι +1
s , ι +2 s partly coincide with the improvement indices suggested by Kakwani (1993, p. 314), which can be written as follows: Majumder and Chakravarty (1996). In addition, our characterization results widen the class of available improvement indices, including the preceding ones as particular cases.

Consistency between Achievement and Shortfall Improvement
Having defined four achievement and the corresponding four shortfall improvement indicators, it seems natural to ask whether these couples of measures As we see the latter approach as quite natural when imposing consistency requirements, it is the one we have implemented in this paper in the context of improvements in standard of living. However, the fact of having achievement-specific and shortfall-specific improvement functions forces us to introduce some changes to our consistency condition, which reads as follows.

Achievement and Shortfall Consistency (AS): Let x, y, z, w ∈ [L, U] be any achievements and let
In words, AS imposes that if a country A is considered to have experienced less overall improvement in standard of living than another country B when measured with achievement indicators, then country A should also be considered to have experienced less overall improvement in standard of living than country B when measured with the corresponding shortfall indicators. When AS is satisfied, we will say that the couple of achievement and shortfall improvement indices (ι a , ι s ) is consistent.
Our main result in this section is as follows: Given the regulating role of α, β, ζ and ξ regarding the compliance with the Upward or Downward Sensitivity axioms (see Tables 2a and 2b), we can conclude that giving more importance to the changes occurring in different parts of the distribution is at odds with the Achievement and Shortfall Consistency axiom. For the additive case, it is easy to check that whenever α = β = 1, ι +1 a and ι +1 s are the same up to a multiplicative constant. Similarly, in the multiplicative case one has that whenever ζ = ξ = 1, ι ×2 a and ι ×2 s can be obtained from one another via simple normalization transformations. Therefore, consistency between ι +1 a and ι +1 s and between ι ×2 a and ι ×2 s is obtained only when these couples of functions are very similar in nature. Interestingly, this result is reminiscent of the findings reported by Lambert and Zheng (2011), who find that the class of achievement and shortfall inequality indices that are consistent is quite restricted.

Empirical Illustration
In September 2000, the United Nations presented the Millennium Declaration, a milestone in international cooperation inspiring development efforts in order to improve the living conditions of millions of people around the world. As a result of the Millennium Declaration, all 193 United Nations member states agreed to achieve a series of time-bound targets-with a deadline of 2015-widely known as the "Millennium Development Goals" (MDGs, see www.un.org/ millenniumgoals). One of those goals-MDG #4-prompts countries all over the world to reduce child mortality. Clearly, this is a health outcome that can a priori be approached from two angles: the shortfall perspective (i.e., reduce child mortality) or the attainment one (i.e., increase child survivorship). While the official MDG #4 is stated in terms of shortfalls, 6 one might legitimately wonder whether changes in children's health will be consistent when assessed via the shortfall and attainment perspectives, respectively. For this purpose, in this section we use census microdata from South Africa to assess the levels of child health improvement over time at the municipal level using the attainment and shortfall indices introduced in this paper.

Data and Indicators
In order to construct child health indicators at the municipal level we use census microdata samples from the Integrated Public Use Microdata Series database (IPUMS, see https://international.ipums.org/international) for South Africa. We have used the censuses samples from years 2001 and 2007. In both years, there were 225 municipalities in South Africa (these will be the units of analysis in this empirical illustration). In order to measure child health outcomes for municipality i, we simply compute the percentage of surviving children born to women in that municipality between ages 20 and 39, which will be denoted by Pi. This indicator is particularly suitable for small size populations and has been used among other things to describe the socio-demographic characteristics of indigenous populations in Latin America (ECLAC, 2010) and to explore the distribution of human development levels with high geographical detail (Permanyer, 2013). Clearly, Pi is an attainment indicator; its corresponding shortfall version is defined as Qi := 100 − Pi (i.e., the percentage of non-surviving children born to women between ages 20 and 39). In this context, one has that L = 0 and U = 100.  Table 1 are quite similar, so they are not shown here to save space). For Figure 2, values of ιi above (resp. below) 0 indicate that an actual improvement (resp. worsening) in child survivorship 7 In this context, when we speak about overall "improvement" or "deterioration" we just refer to the general shape and position of the respective density functions, not to the specific changes observed for each municipality (which can not be inferred from that information only). 8 To illustrate: assume, without loss of generality, that a distribution of achievements is ordered (x1 ≤ . . . ≤ xn). The hypothetical distributions of paired data {(xi, yi = xi)}1≤i≤n and {(xi, yi = xn−i+1)}1≤i≤n have exactly the same marginal distributions but the individual-level improvements are completely different in the two cases (there are no changes whatsoever in the first one and extreme changes are observed in the second one). Source: Author's calculations using IPUMS data.  Source: Author's calculations using IPUMS data.

Empirical Results
percentages has taken place in municipality i. Regarding Figure 3, the improvement threshold equals 1. As can be seen in Figures 2 and 3, most municipalities in South Africa have experienced deteriorations in child survivorship percentages, no matter whether we use additive or multiplicative improvement indices.
We will now explore empirically the extent to which the assessment of improvement levels is consistent when using the shortfall and achievement perspectives. For that purpose, we compare the municipalities ranking that is obtained using achivement improvement indicators with the ranking derived from the values of the corresponding shortfall improvement indicators. Table 3 shows the values of Kendall's tau coefficient 9 (henceforth τ) associated to the values of those achievement and shortfall indicators for alternative choices of the corresponding parameters. The definition of that statistic fits perfectly in our framework, since our consistency axiom (AS) precisely demands that the same set of municipalities is coherently ranked by alternative measures. The first columns of Table 3 show the values of τ for different parameter specifications of ι +1 a , ι +1 s , ι ×2 a and ι ×2 s . When α = ζ and β = ξ, ι +1 a can be obtained from ι ×2 a via a monotonic transformation (and vice versa) and the same can be said about ι +1 s and ι ×2 s . Since Kendall's tau is invariant under monotonic transformations, the corresponding coefficients are the same for ( , ) ι ι , so they are shown together in the table. By the same token, since ι ×1 a and ι ×1 s can be obtained from ι +2 a and ι +2 s respectively via monotonic transformations, their corresponding τ is also the same, so they are shown together in the last column of Table 3. It turns out that in virtually all comparisons considered in this paper, the municipality rankings that are obtained from the values of achievement and shortfall improvement indices are not completely consistent, that is, there exist couples of municipalities whose relative ranking is reversed when using one approach or the other. When this happens, the corresponding τ is strictly smaller than 1. For the case of South Africa there are certain choices of improvement measures for which τ ≃ 0.7 or even lower, that is, many couples of municipalities (in some cases 30 percent or even more) are not ranked consistently according to ι a and ι s . Therefore, our assess- ments of the child health improvements experienced in South African municipalities can differ to a great extent when using achievement or shortfall indicators.

Summary and Concluding Remarks
In this paper we have presented new indices of improvements in standards of living that address an important issue which has been ignored so far in the literature: the problem of "consistently measuring achievement and shortfall improvement." Given the bounded nature of virtually all standard of living indicators, it is possible to measure their improvements over time on the basis of the levels of achievement or on the basis of the corresponding shortfalls with respect to the upper bound. Integrating both approaches into a common framework, we have proposed the corresponding achievement and shortfall improvement indices and characterized them axiomatically. In a way, our improvement indices are reminiscent of the directional income mobility indices suggested by Fields and Ok (1999), adapted to the case where the variable of interest is bounded.
We argue that achievement and shortfall improvements are two sides of the same coin and that it is important to check whether both sides are measured in a consistent way. Such consistency can be imposed in different ways. A strong requirement to fulfill the consistency condition is to impose that both achievement and shortfall improvement indices take exactly the same values. A weaker requirement simply states that the orderings derived from the values of the indices have to be the same. It turns out that even when starting from the weaker requirement, the only achievement and shortfall improvement indices that rank alternative states of affairs in a consistent way are those that can be easily obtained from one another via straightforward monotonic transformations. Or the other way around: it is only when achievement and shortfall improvement indices are essentially the same functions (up to some normalization constants) that they are able to rank alternative states of affairs consistently. As shown in our analysis, the only way in which achievement and shortfall improvement indices can be consistent is to get rid of the Upward and Downward Sensitivity axioms-a couple of requirements that reward those improvements occurring at higher and lower achievement levels, respectively. This somewhat discouraging result is in line with the findings reported by Lambert and Zheng (2011) in the context of consistent achievement-shortfall inequality measurement. Given the fact that the achievement and shortfall approaches are not perfectly complementary (i.e., the orderings derived from one approach can not be deduced from the orderings of the other), it becomes necessary to give them separate and careful attention.
We have empirically illustrated our methodology exploring the evolution of child survivorship percentages in South African municipalities between the censuses of 2001 and 2007. Among other things, our results indicate that the corresponding municipality rankings derived from the values of our achievement and shortfall indicators are not always completely consistent (that is, there always exist some couples of municipalities which are inconsistently ranked with both kind of indicators). Even if both rankings tend to be highly correlated, in the case of South Africa there are as many as 30 percent of couples of municipalities that are inconsistently ranked by certain specifications of our achievement and shortfall