International Journal of Educational Development 96 (2023) 102684 Contents lists available at ScienceDirect International Journal of Educational Development journal homepage: www.elsevier.com/locate/ijedudev The politics of good enough data. Developments, dilemmas and deadlocks in the production of global learning metrics Clara Fontdevila a, b a Department of Sociology - Universitat Autònoma de Barcelona, Avinguda Eix Central. Edifici B, 08193 Bellaterra (Cerdanyola del Vallès), Spain b School of Education - University of Glasgow, St Andrew’s Building, 11 Eldon Street,G3 6NHGlasgow, United Kingdom A R T I C L E I N F O A B S T R A C T Keywords: The indicator framework established for the monitoring of SDG4 is unambiguous on the need to advance in the Sustainable Development Goal 4 production of global learning data. However, the production of SDG4 learning metrics has been riddled with Learning metrics technical and political difficulties. Drawing on a combination of documentary analysis, interviews and non- UNESCO Institute for Statistics participant observation, this paper reconstructs the process of negotiation of the data suppliers, statistical rou­ Data production International large-scale assessments tines and reporting standards necessary for the production of SDG4 learning metrics. The paper thus offers Global indicators insight into the mechanics of global quantification, and on the transformative impact of such processes on the agendas and relationships of partaking organizations. 1. Introduction comparable way. Up to five targets in SDG4 include one or more learning-related indicator, and 4 out of 12 global indicators require Goal- and target-setting have a long history in the field of education, reporting on student learning, skills or knowledge. constituting today a key instrument of global governance and being both Thus, with the advent of SDG4, the production of globally- a source and a manifestation of increasing transnational interdepen­ comparable learning data has become an institutional priority for the dence (King, 2016; Mundy, 2010). The Sustainable Development Goal UNESCO Institute for Statistics (UIS), designated as the custodian 4/Education 2030 agenda, adopted in 2015, adds to a long list of efforts agency for most SDG4 global indicators. However, and despite broad in global coordination to support the universal right to education. Along agreement on the centrality of learning measurement, the production of with 16 other goals, SDG4 is integrated into the UN Sustainable Devel­ SDG4 learning metrics has been far from straightforward and conflict- opment Agenda adopted by the United Nations General Assembly and is free. Difficulties encountered in the process have not been exclusively largely conceived as the natural successor of both the Education for All technical in nature, but also stem from the political obstacles inherent to agenda and the education-related Millennium Development Goals the collective nature of indicator-making. Thus, the production of global (Sachs-Israel, 2017). learning metrics relies necessarily on data suppliers other than statistical One of the most significant shifts entailed by the SDG4 agenda is the offices and national governments – most notably, producers of cross- focus on learning outcomes – a transformation sometimes referred to as national assessments, whose relationship with the UIS is compara­ the quality turn or learning turn and defined as an effort to transcend a tively less institutionalized. Expectedly, reconciling the multiple ex­ focus on schooling and enrolment figures as key indicators of progress pectations of these different data suppliers and international (Fontdevila, 2021; Sayed, Ahmed and Mogliacci, 2018). Accordingly, organizations has proven a considerable challenge for the UIS. the indicator framework established for the monitoring of SDG4 is un­ However, the process of indicator-production behind the making of ambiguous on the need to advance in the production of global learning SDG4 learning data remains an empirically under-researched process. data – so that student achievement can be reported in an internationally To be sure, there has been some comment on the process by which E-mail address: Clara.Fontdevila@uab.cat. https://doi.org/10.1016/j.ijedudev.2022.102684 Received 1 November 2021; Received in revised form 12 September 2022; Accepted 4 October 2022 Available online 4 November 2022 0738-0593/© 2022 The Author. Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). C. Fontdevila International Journal of Educational Development 96 (2023) 102684 learning-related targets were translated into quantitative indicators, and IIEP-UNESCO, and tasked with monitoring student learning through the the effects that such transformation had on our understanding of edu­ production of comparable reports (Cussó and d’Amico, 2005). In the cation quality (Unterhalter, 2019; Wulff, 2020). Nevertheless, such ac­ case of UIS’, one of its first inroads into the area of learning assessment counts have tended to focus on the moment of indicator selection, while was the creation of the Literacy Assessment and Monitoring Programme leaving unaddressed the very production of such data – that is, the (LAMP), initiated in 2003 and oriented at measuring adult literacy. negotiation of the data sources and methodological considerations However, the project suffered from a number of political, technical and driving data-collection and data-harmonization efforts. Indeed, the institutional setbacks, and ‘never gained sufficient global prestige’ quantification labor performed by international organizations, and the (Addey, 2018, p. 400). mundane operations through which global datasets are constructed, In the area of data harmonization, the UIS appears to be something of remain an empirical gap. a late-comer.2 Initiatives oriented at the harmonization (rather than the In this light, this paper explores the process of production of the collection) of learning data include the Assessment of Learning Out­ global indicator corresponding to Target 4.1, a commitment to ‘ensure comes set in place in 2008 and the Observatory of Learning Outcomes that all girls and boys complete free, equitable and quality primary and established in 2011. However, such attempts at the production of global secondary education leading to relevant and effective learning outcomes’ learning data found very limited success. This was very much a conse­ (UNGA, 2017, p. 8). Known as Indicator 4.1.1, this metric refers to the quence of the reluctance to share certain data (e.g. test items) exhibited ‘Proportion of children and young people (a) in grades 2/3, (b) at the end of by the main global and regional organizations involved in learning as­ primary, and (c) at the end of lower secondary achieving at least a minimum sessments, along with the competition between the IEA and the OECD3 – proficiency level in (i) reading and (ii) mathematics, by sex’ (UNGA, 2017, but also of the skepticism exhibited by some countries. p. 8). This is in fact perceived by a range of education stakeholders as a The establishment of the Learning Metrics Task Force (LMTF) in crucial metric that, combined with completion rates, constitutes the 2012 is probably the most visible and better-known example of UIS’s basis for the SDG4 unofficial ‘leading indicator’ – that is, the main early efforts to the harmonization of learning data. Envisaged as a multi- benchmark against which the overall progress of the SDG4 agenda will stakeholder partnership, it was co-convened by the Center for Universal be measured. Education (CUE) at the Brookings Institution and the UIS. The group Drawing on a combination of documentary analysis, interviews and organized three consultative processes and brought together up to 30 non-participant observation, the paper reconstructs the process through organizations that met on different occasions from July 2012 to which this indicator has been attached to specific data suppliers, sta­ September 2013, benefiting also from the contributions of nearly 200 tistical routines and reporting standards. In so doing, the paper aims to experts. Importantly, the socialization effect brought about by LMTF gain insight into the mechanics of global quantification, and to under­ meetings played an crucial role in constructing new and shared mean­ stand the transformative impact of an inherently collaborative project ings and the legitimation of the assessment program, and in setting the on the agendas and relationships of the UIS and other partaking orga­ parameters of the learning measurement debate (Fontdevila, 2020). nizations. Overall, the paper finds that if the learning turn was a story of However, and despite the inclusive and pluralistic character of the consensus-building, the quantification of learning targets is one of LMTF, some have observed that the engagement of certain constitu­ fragmentation – a process in which many of the ambiguities1 strategi­ encies was frequently superficial or simply nominal and that, despite cally exploited in the negotiation of SDG4 have become readily being presented as a collaboration between the UIS and CUE-Brookings, apparent. the latter was always fare more in control of the agenda than the former. Hence, and while the LMTF was central in revitalizing the debate around 2. Trying and failing and trying again. Historical attempts at the the harmonization of learning data and in creating a sense of common harmonization of learning data purpose among different agents involved in the measurement of learning, its legacy was somewhat tainted by the prominence of While cross-national learning data enjoys an unprecedented promi­ Brookings – which, to some extent, came at the expense of UIS’s cen­ nence in the SDG4/Education 2030 agenda, attempts at the production trality in such debate. of globally comparable learning data have a long history within UN Thus, the UIS transitioned into the post-2015 era without the circles – in fact, it is possible to document a rich history of unsuccessful necessary infrastructure or resources to produce globally-comparable attempts to harmonize learning data at a global level. The magnitude of learning data. At the same time, and given the unprecedented the challenge faced by UIS in relation to the SDG4 learning indicators emphasis put on education quality and learning outcomes, with the needs to be understood in relation to UNESCO’s erratic historical record adoption of the SDG4 agenda the production of such data became an in the collection and the harmonization of learning data. inescapable responsibility. In particular, the production of the global It is important to bear in mind that, when it comes to the production indicator corresponding to SDG4 Target 4.1 became a de facto of learning data, UNESCO’s trajectory is a rather irregular one, with a number of one-off attempts with limited continuity over time. In fact, UNESCO’s efforts in the production of learning-data date back to a 2 This cannot be disconnected from the fact that the production of globally twelve-country cross-national study conducted between 1959 and 1969 comparable learning data only came to perceived as part of UIS’ core moni­ under the sponsorship of UNESCO Institute for Education in Hamburg toring mandate with the advent of SDG4. Hence, while one of the Education for (UIE, 1962). Later on, in 1992, UNESCO launched a project to monitor All monitoring indicators approved in 2000 focused on the share of grade 4 student achievement in selected countries – the Monitoring Learning students having mastered a set of learning competencies, such competencies Achievement initiative (MLA), implemented jointly with UNICEF. With were to be defined nationally (World Education Forum, 2000). Consequently, a similar objective, the Southern African Consortium for Monitoring this EFA indicator did not entail a clear harmonization mandate for UIS. 3 Education Quality (SACMEQ) was launched in 1995 with the support of While IEA is widely recognized as a pioneer in the area of cross-national learning assessment, since the turn of the century its work has been over­ shadowed by the visibility of PISA – and, more in general, by the expansion of the OECD in the area of educational measurement. In consequence, and even if 1 As I have argued elsewhere (Fontdevila, 2021), the process of their testing programs differ in terms of target population and assessment consensus-building behind the negotiation of SDG4 often relied on approach, the OECD and IEA are perceived as rival organizations (cf. Morgan, word-tweaking or ’word-smithing’ practices oriented at preventing the alien­ 2011). It should be however noted that, according to different informants, such ation of any constituency or country. This was the case of heavily contested animosity or inter-organizational infighting has tended to fluctuate over time – issues such as the role of the private sector, but also of the use of learning eventually decreasing with the retirement of some key personalities (Fontde­ outcomes as a measure of education quality. vila, 2021). 2 C. Fontdevila International Journal of Educational Development 96 (2023) 102684 organizational priority for the Institute, and the production of Indicator instrumentalizing quantification practices as part of self-serving agenda. 4.1.1. ended up acquiring a high-stakes quality for the UIS. This totalizing understanding of the global quantification practices has, Importantly, what was at stake was not only the credibility of UIS however, been called into question. Thus, Erkkilä and Piironen (2018) within education circles, but also within the broader UN environment have recently noted that the rationales guiding the production of in­ and the international statistical community. Thus, the UIS is accountable dicators are more complex than most accounts suggest – noting critically to the education agencies and Ministries of Education brought together that the production of indicators is unlikely to be driven by a single, by the SDG4 initiative, but also to the UN Statistical Commission and the rational and well-defined motivation. Similarly, Dahler-Larsen (2014) Inter-Agency and Expert Group on SDG Indicators (IAEG-SDGs), has taken issue with the notion that indicators are guided by a consistent responsible for the development of the global framework of indicators and coherent vision regarding their ultimate objectives. Such observa­ for the monitoring of SDGs (ECOSOC, 2017). Agencies responsible for tions suggest that, far from being a neat process, the making of global SDG data, as is the case for the UIS, are expected to report periodically indicators is a messy endeavor, driven by different and even contra­ on the development of methodologies and data-collection tools. This dictory rationalities. process represents an important source of reputational pressure for the It follows from this that it is necessary to unpack the making of global UIS and is perceived by the global education community and donor indicators. One strategy for doing so is to focus on the very production of circles as a test of UIS’ competence. global indicators. This notion was introduced by Davis, Kingsbury and Taking into consideration these developments, this paper analyzes Merry (2012) in their operationalization of the trajectory of global in­ how the UIS has dealt with the challenges posed by SDG4 in terms of dicators and refers to the phase in which the conceptualization of the learning measurement – an area in which the UIS was not well posi­ indicator is coupled to an already existing or newly created dataset. tioned to affirm its leadership and expert authority. Specifically, the Importantly, the authors caution against placing excessive emphasis on paper inquires first into the political and technical strategies mobilized the promulgators of a given indicator (that is, the organization in charge by the UIS in order to produce the learning metrics required by the SDG4 of its packaging and dissemination), since the production of an indicator monitoring framework, and then analyzes how this labor has impacted involves a much wider and dispersed range of actors in charge of on the authority and credibility of the UIS within the global education data-collection efforts (including international organizations, national field. statistical offices and NGOs etc.). The authors note that the collective character of indicator production is likely to translate into challenges 3. Opening the black box of indicator production: analytical associated with the difficulty of securing the cooperation and diligence considerations of data suppliers, and with tensions and friction between different data sources. In order to make sense of these dynamics, it is necessary to theo­ A key point to bear in mind when examining processes of indicator retically articulate the mechanics of global quantification as an object of production is the social labor behind the transformation of raw data into study. This section introduces some conceptual remarks necessary to authoritative representations. This point has been raised by Espeland examine an oft-neglected aspect of the quantification labor performed and Stevens (2008), who draw attention to the fact that indicators by international organizations – namely, the production of indicators (i. become authoritative as they ‘move upwards’. Such views have more e., the process through which specific data suppliers, statistical routines recently been echoed by Rocha de Siqueira (2017), who argues that and reporting standards are selected and developed. The section also producers of indicators are aware of the imperfection of their own develops a methodological framework appropriate to the empirical datasets but do not see this approximate character as inherently prob­ challenges posed by the study of such processes, integrating the ad­ lematic. The author has put forward the notion of good enough data in vances made by the so-called practice turn (Neumann, 2002) with the order to shed light on these dynamics. The concept captures the speci­ principles of thick description (Merriam and Tisdell, 2016). ficity of data recognized by both producers and consumers as imperfect (or even prone to error) but accepted as an authoritative source by virtue of their practicality or convenience. 3.1. The making of global indicators The production of global metrics has significantly accelerated over 3.3. Making sense of organizational effects the last two decades – accordingly, a profuse scholarship on global in­ dicators has emerged. However, research examining global quantifica­ As noted above, much of the literature on global indicators has tion practices has tended to focus on a particular subset of metrics – focused on the monitored populations or a loosely defined audience. specifically, those reporting systems that rate, rank or categorize coun­ Conversely, the effects of indicators on the organizations producing tries (see for instance Kelley and Simmons’s (2014) or Cooley’s (2015) them have been less systematically examined. Freistein (2016) has work) – and has tended to focus on the impact of indicators over the recently put forward the notion of organizational effects – a concept that monitored populations. As we will see in this section, such thematic captures the impact of indicators on the operational logic of interna­ focus has come at the expenses of an understanding of (1) the collective tional organizations in charge of their creation or dissemination. The nature and the micro-social foundations of the processes of author thus notes that the creation of indicators is at least partially indicator-making; and (2) the effects of global indicators over the very motivated by the need to signal and ascertain authority over certain organizations producing them. This section advances a series of concepts issues. oriented at bringing to the fore such themes in a theoretically-informed Echoing Freistein’s (2016) arguments, there is growing recognition way. that international organizations engage in the production of indicators as a strategy to position and (re)brand themselves. It follows from this 3.2. The production of global indicators as a collective endeavor literature that it is important to pay attention to both external and in­ ternal organizational effects. External effects refer to the fact that the Literature on benchmarking practices has arguably tended to exag­ production of global indicators affects the position of the promulgating gerate the power and intentionality of the international organizations organization vis-à-vis other international agencies (fostering relation­ responsible for global indicators. It is thus implicitly assumed that in­ ships of competition or cooperation). These dynamics have for instance ternational organizations operate as all-powerful agents deliberately been captured by Cooley (2015), who argues that rankings are used by 3 C. Fontdevila International Journal of Educational Development 96 (2023) 102684 both governmental and non-governmental international organizations Table 1 as a means to assert their authority over specific issues.4 Pool of interviews – disaggregated by categories. Internal effects capture the notion that the production of indicators Interview group Total also affects the relationships between different units operating within an UN agencies: UNESCO (including IIEP and the UIS), UNICEF, UNHCR (UN) 15 organization. Such a phenomenon has for instance been documented by Assessment producers (international organizations, NGOs, expert consortia) 16 Arndt (2008), who draws attention to the impact of quantification labor (LASS) on intra-organizational dynamics. In her discussion of the World Bank’s Multilateral and bilateral donors (DON) 12 governance indicators, the author notes that institutional reasons are Civil society organizations and non-governmental organizations (CSNGO) 5 Country representatives (CR) 3 one of the key drivers behind the rise of governance indicators and Private sector and experts (PRI) 8 observes that the production of indicators constitutes a means to raise Total 59 the status of particular units or divisions within a given organization Source: Author’s elaboration. (Arndt, 2008). Note: The bracketed acronyms correspond to the nomenclature used to quote This study engages directly with the collective nature of indicator interviewees. production, as well as with its organizational effects over institutions partaking in such a process. In this way, it contributes to a better un­ perspective of all those individuals displaying a feature that the derstanding of such theoretical constructs by providing an informed researcher judged to be of interest, rather than ensuring the represen­ illustration of the mechanics of global quantification – an area of study tativeness of different groups (Ritchie, Lewis and Elam, 2003). The table in which it is possible to identify a certain disconnect between theo­ below offers a breakdown of the pool of interviewees according to six retical and normative debates and empirical research. basic categories.5(Table 1). Complementarily, the research relies on the analysis of a broad 3.4. Analytical strategy and data sources corpus of documentation collected over the course of the research in an iterative way, including research reports, position papers, public state­ This paper applies the conceptual and theoretical remarks outlined ments made by different stakeholders, blog posts published by negoti­ above to the study of the UIS’ leading role in the production of SDG4 ators, meeting agendas and evaluation reports. The collection of such learning metrics. It focuses thus on the micro-social foundations of documents had an iterative nature and did not follow a pre-determined indicator-making processes, as well as on the effects of indicator-making protocol or search strategy. Hence, and on the basis of exploratory on the interactions between and within organizations engaged in the searches and pointers and recommendations given by interviewees, I production of data. This investigation is premised on the need to focus conducted regular screening of a range of institutional portals such as on tangible and observable instances of activity as key units of analysis – the UIS web pages dedicated to the Global Alliance to Monitor Learning in line with the tenets of the so-called practice turn as applied to the (GAML) and the Technical Cooperation Group (TCG), or the World study of international organizations. Introduced by the seminal works of Education Blog curated by the Global Education Monitoring Report Neumann (2002), and Adler and Pouliot (2011), the practice turn em­ (GEMR). A number of ad hoc searches were also conducted in response phasizes the need to focus on the more mundane unfolding of global to new developments of the debate and to imprecisions in the accounts social action, and particularly favors greater empirical attention to the offered by the interviewees. These searches were also to test and sub­ practices, routines and worldviews exhibited by international stantiate the emerging analytical insight. organizations. Finally, the study incorporated observation data in order to contex­ In order to capture these micro-processes, the analytical strategy tualize the insights gained on the basis of interview and documentary used in this investigation relies on the principles of thick description – data. Specifically, non-participation observation was carried out on the that is, an idiographic, contextualized and detail-oriented approach, occasion of three multi-day events directly related to the monitoring of characterized by a sequence-focused reasoning style. Popularized by the SDG4 agenda and the production of its associated metrics – namely Geertz (1973), the notion of thick description was originally used to (1) the 2nd Meeting of the Technical Cooperation Group on the In­ refer to interpretive work explicitly oriented towards capturing the in­ dicators for SDG4-Education 2030 (16–18 October 2016, Madrid); (2), sider views of the actors involved in a given process or phenomenon. the 3rd Meeting of the Global Alliance to Monitor Learning (11 and 12 However, and as noted by Merriam and Tisdell (2016), with the passage May 2017, Mexico City); and (3), the 5th Meeting of the Global Alliance of time ‘it has come to be used to refer to a highly descriptive, detailed to Monitor Learning (17 and 18 October 2018, Hamburg). presentation of the setting and in particular, the findings of a study’ (p. 257). This approach is adequate given the preoccupation of the study 4. The twists and turns of harmonizing learning data with the chain of events through which the SDG4 learning metrics came into being, and the organizational dynamics and micro-sociological This section is concerned with the first objective of the paper, forces that shaped such processes. namely, to gain insight into the political and technical strategies In terms of data sources, the study relies on the triangulation of three mobilized by the UIS to produce the learning metrics required by the main methods – namely, interview data, documentary analysis and non- SDG4 monitoring framework. The section shows that, in order to over­ participant observation. A corpus of 59 semi-structured interviews is the come the historical challenges associated to the creation of global primary source of data. Interviewees were recruited through a purposive learning data, the UIS relied on a transparency and inclusion strategy sampling strategy – a criterion-based approach oriented at capturing the oriented at maximizing political legitimacy, in combination with a bricolage strategy oriented at combining pre-existing approaches to data harmonization. In order to document this process, the section re­ 4 Note that these observations apply primarily to organizations not having a constructs three main episodes corresponding to three of the major formal mandate, and/or enjoying only limited legitimacy in the measurement challenges faced by the UIS in its most recent attempt to harmonize of a given area. This is ultimately reflective of the emphasis of quantification literature on comparative and ranking -like format – which has partially come at the expense of the study on the development of the international statistical 5 system, and UN statistical activity in particular (for an exception, see Ward, Note that, for the purposes of this table, representatives of IOs and NGOs 2004). Hence, the quantification practices of UN agencies with a formal mandate selected on account of their role in the production of learning data are included in the measurement and monitoring of specific themes remain largely in the Assessment producers category (LASS) rather than in the UN and CSNGO understudied. categories. 4 C. Fontdevila International Journal of Educational Development 96 (2023) 102684 global learning data – namely, the creation of a legitimate infrastructure, interests and new models, and not moving forward. (Interview the selection of data suppliers, and the design of an alignment strategy CSNGO6). allowing to equate existing learning assessments. In line with its self-professed open and participatory nature, GAML membership has, since the start, been open to any individual or orga­ nization interested in contributing to its work. However, the first GAML 4.1. The creation of a legitimate infrastructure meetings were mainly attended by international agencies, development partners, research organizations and foundations with a global scope As the negotiation of the SDGs progressed, the UIS started to antic­ (UIS, 2016b). A wide range of individuals with different institutional ipate the need to venture into the production of globally comparable affiliations were critical of the absence of country representatives at learning data. Thus, in April 2014, the UIS and the World Bank convened GAML meetings. In combination with such reliance on self-selection a meeting with the objective of defining criteria to monitor reading in dynamics, the limited formalization of internal decision-making pro­ primary education. This meeting eventually gave rise to the creation of cedures soon came under the critical scrutiny of a number of members. the Learning Metrics Partnership (LMP), a joint initiative of the UIS and Also, and as noted by Benavot and Smith (2020), donors have tended to the Australian Council for Educational Research Centre for Global Ed­ ‘dominate meetings and have played a significant role in directing the ucation Monitoring (ACER-GEM)6 supported by the Australian Depart­ focus toward Target 4.1’ (p. 254) – an observation that suggests that ment of Foreign Affairs and Trade, and which aimed to establish a power asymmetries remain within GAML, despite GAML’s commitment common scale against which existing assessment could be plotted. towards horizontality and inclusion. However, the LMP initiative was short-lived – a change in the lead­ These dynamics were particularly problematic for those attendees ership of the UIS in 2015 precipitated an early termination of the proj­ representing large international bureaucracies, and who were conse­ ect. According to different interviewees, the incoming UIS Director quently less willing or capable of making snap decisions (Interviews decided to put an end to the partnership with ACER in order to reaffirm DON6, CSNGO6, PRI3). In response to such concerns, and as a way to the centrality of the UIS and its visibility in the creation of globally sensitize and familiarize countries with the outcomes of the GAML-led comparable learning data. At the same time, the move was also technical negotiation, since 2018 an explicit effort has been made to prompted by reservations expressed by some representatives of regional incorporate country representatives. Similarly, a governance structure assessments, who perceived the initiative as excessively top-down in has been progressively clarified and refined. According to the Gover­ nature. A former UIS analyst noted that the LMP had fallen apart after a nance and organization note issued in 2017 (UIS, 2017), the technical particularly contentious meeting in which regional assessments work would fall on thematic task forces, but the inputs and recom­ perceived that they had been unduly sidelined (Interview UN7). Finally, mendation prepared by them should be discussed and endorsed by the limited support for the LMP within the learning measurement plenary meetings. This was paralleled by efforts to draw a clear sepa­ community was also a consequence of the leading role played by ACER ration between GAML’s role (making recommendations) and the in this venture. Some LMP participants were under the impression that mandate of the Technical Cooperation Group (TCG)7 (responsible for the LMP was being used by ACER as a ‘shop front’ to market its services, their discussion and adoption). Hence, it was made explicit that the and that ACER was instrumentalizing global targets to create a niche of Secretariat (served by UIS) was expected to prepare recommendations its own in the assessment market – especially after its central role as a and report to the TCG – the group ultimately responsible for its discus­ PISA contractor came to an end after the 2012 survey (Interview sion and approval (UIS, 2017). LASS10). Overall, an effort has been made to secure a more democratic envi­ It is precisely as a response to the failure of the LPM venture that the ronment – and despite some reservations, the move has proved suc­ Global Alliance to Monitor Learning (GAML) was created under the cessful in reinforcing both GAML’s and UIS’s authority. However, it is auspices of the UIS. Announced in late 2015, it was devised as a group important to bear in mind that the democratization of GAML remains an bringing together a wide range of stakeholders (most notably assessment unfinished process. Thus, in spite of the aforementioned governance agencies, education-related international organizations and NGOs, and innovations, some participants remain unclear on the real locus of national authorities) with the objective of reaching an agreement on the decision-making. Particularly within civil society circles, it has been specific measures and tools necessary to monitor learning in the context noted that despite the emphasis placed on deliberative forums and of the SDGs (Montoya, 2015). Since its formalization in 2016, GAML has GAML’s plenary meetings, the UIS frequently steers the debate so as to been meeting on a periodic basis. secure specific outcomes, and that much was being discussed behind The creation of GAML was largely welcomed by the education closed doors (Interview CSNGO5). Some even perceived the GAML as community on account of its inclusive character. The creation of a new operating as a rubber-stamping board in which the UIS was simply space was portrayed by a number of interviewees as a positive devel­ looking for a ‘seal of approval’ for previously negotiated agreements, opment – an opportunity to start with a clean slate. This was especially and argued that the inclusion of country representatives essentially true of those informants less well-versed in the decades-old infighting followed a tokenistic logic. between the IEA and the OECD, and who were very critical of the This episode suggests that the UIS has enhanced its legitimacy as a paralyzing effect of such disputes. Additionally, the advent of GAML was focal point for the negotiation of learning metrics by recourse to the perceived as creating more space for data producers other than large maximization of inclusion and transparency as a means to overcome the assessment consortia. As an NGO officer put it: limited confidence, misgivings and suspicion triggered by prior attempts It’s sort of a breath of fresh air because the rest of us have been sitting at the production of learning data. At the same time, this democratiza­ at these tables for years now, literally! […] We’ve been sitting here tion strategy pursued through the creation of GAML has not been for four years going around and around and around, with vested without costs and might be preventing the UIS from making progress at the pace required by the international statistical community. This 6 ACER is a not-for-profit, research-oriented organisation with a focus on education. Along with a handful of testing organizations and research centres 7 (e.g. ETS), it represents one of the few contractors with the necessary expertise A platform convened by the UIS (which serves as its secretariat), the TCG is to cater to the needs of international organizations and consortia administering tasked with the political mandate to develop and debate SDG4 thematic in­ cross-national assessments. Historically, ACER has played a key role in the dicators. The TCG is composed of regionally-representative UNESCO Member implementation and administration of large-scale assessments including States, and incorporates also representatives of civil society organization, and a OECD’s PISA and IEA’s TIMSS and PIRLS. range of multilateral partner agencies i.a. (UIS, 2022a). 5 C. Fontdevila International Journal of Educational Development 96 (2023) 102684 explains the fact that, as noted by different interviewees and confirmed NAs for global reporting purposes – including the fact that such data by non-participant observation, GAML meetings are increasingly scrip­ allows for greater coverage of Indicator 4.1.1; and that discarding NAs as ted – with attendees being presented with a narrow set of options rather a valid source of comparable data was a risky move since it runs against than invited to propose alternative routes. Ultimately, such dynamics country-ownership principles. Ruling out NAs might hence create a are indicative of the fact that, while the democratic imperative and the perverse incentive for resource-constrained countries to invest exclu­ inclusion expectations placed on GAML are key sources of legitimacy, if sively in CNAs (which, despite their capacity-building potential, are taken to extremes they can also operate as constraining elements that unlikely to realize the full potential of learning metrics, especially when might hamper the UIS’ ability to deliver its mandate – thus creating a it comes to education planning and policy design) (UIS, 2016a). perverse incentive for the UIS to preserve a formally democratic struc­ Taking stock of these trade-offs, since 2016 the UIS has been toying ture while moving the real locus of decision-making away from these with different options, and rather than favoring a particular course of democratic and participatory spaces. action, has explored a variety of reporting and harmonization strategies, commissioning work to a wide range of parties. The profusion of 4.2. The selection of data suppliers reporting protocols, mapping exercises, prospective studies and concept notes produced over the last 5 years is testimony to the multi-pronged Despite the legitimacy gains brought to the UIS by the GAML strategy pursued by the UIS. initiative, the production of globally comparable learning data con­ One of the first options discussed in early 2016 was the establishment tinues to present a number of technical difficulties that have historically of a new cross-national assessment specific to a target population and defied consensus. One of the most pressing challenges in this regard is learning domain, to be implemented in all countries – an idea originally related to the availability and quality of data sources. Such challenges pitched by a group of analysts affiliated with the Center for Global include the limited coverage of existing learning data (i.e., the fact that Development. Advocates of the single-test option portrayed the common for a large number of countries, data on learning outcomes remains assessment as a way of maximizing the robustness of a global dataset, absent), but also the limited comparability of existing metrics, and the equated to ex-ante comparability (Birdsall, Bruns and Madan, 2016). The existence of multiple datasets that require prioritization. It is important proposal, however, found limited resonance within the GAML commu­ to bear in mind that, by the mid-2010 s, there were several cross- nity. Part of the resistance faced by this proposal stemmed from the fact national assessments (CNAs) in place, but no consolidated methodol­ that it was not perceived as being sufficiently disinterested in nature. ogy to equate and harmonize them. In fact, the CNA category encompass According to a range of interviewees, it was not entirely clear to what both regional and international assessments and, as captured by Table 2, extent some of the original promoters of the idea were acting bona fide is notoriously heterogeneous in terms of targeted domain and grades, or driven by business interests.8 Concerns about the commercial impli­ design, sampling, methods for score estimation, etc. cations of a universal test were compounded by a sense of ‘expert The debate was further complicated by the discussion around the skepticism’ regarding the feasibility of such an ambitious project. The possibility of integrating data sources other than CNAs. While by the idea of an ex-novo, single-test was largely perceived as not grounded in mid-2010 s a considerable number of countries had a National Assess­ reality (inattentive to implementation costs and time challenges); and ment (NA) in place, this data is not particularly amenable to compara­ concerns were voiced on the risks on relying on a single tool. tive purposes. At the same time, the UIS had important incentives to use Once the idea of a new universal test started to lose steam, the idea of selecting an existing CNA and extending it to new countries was vigor­ ously pushed by some assessment agencies – the IEA and the OECD in Table 2 Overview of cross-national assessments. particular. These organizations seized SDG4 as an opportunity to make inroads into the development realm and expand their portfolio of Target potentially Grade or target Name of the Domain – Literacy countries.9 As summarised by a member of the GAML-SPC: informed by the population assessment (L) or Numeracy assessment (N) You know, particularly IEA was going ‘Well, we are already at the – 1 EGMA N primary level. We’re already all over the world. We should be the 1 EGRA L test’. And then PISA, they were trying to jump in to say, ‘Oh well, we PASEC L, N should be 4.1.c because we are at the end of secondary’. So they were 2 EGMA N EGRA L trying to occupy that space […] They were basically trying to argue, LLECE L, N ‘Look, those are valid, reliable, long-standing international assess­ 3 EGMA N ments. Why are we even debating it? These should be the tools’ […] 4.1.1a EGRA L So they were, I’d say, taking an arrogant approach. And they were trying PIRLS/ePIRLS L TIMSS N to intimidate the others based upon their technical prowess. (Interview 4 LaNA L, N DON6). PILNA L, N 5 SEA-PLM L, N As the quote above suggests, the behavior exhibited by OECD and LLECE L, N IEA representatives was largely perceived as opportunistic, particularly PASEC L, N on the part of some civil society and non-profit organizations. This was 4.1.1b 6 SACMEQ L, N compounded by the fact that, within the donor community, there was PILNA L, N limited appetite for this approach. Such reluctance was partially driven LaNA L, N 8 TIMSS N 4.1.1c 15 y.o. PISA L, N 14–16 y.o. PISA-D L, N 8 – – ASER, Uwezo L, N While the veracity of these misgivings is beyond the object of this study, the (5–16 year-olds) reservations expressed by a variety of individuals as to the true intent behind Source: Author’s elaboration on the basis of Treviño and Órdenes (2017) and UIS the single-test idea, even if fuelled by mere hearsay, are revelatory in their own (2016a). right and speak to the governance challenges that characterized the early days Notes: of GAML and that motivated the development of an increasingly refined governance structure. 9 • In bold: regional assessments; In Italics: international assessments; Regular font: An endeavor already initiated through IEA’s Literacy and Numeracy assessments of foundational skills and population-based assessments. Assessment for Developing Countries (LaNA) and the OECD’s PISA for Devel­ • Includes only those domains relevant for global reporting purposes. opment (PISA-D). 6 C. Fontdevila International Journal of Educational Development 96 (2023) 102684 by a perceived lack of alignment with country priorities, and especially existing assessments. A concept note published in 2019 described the by donors’ image concerns and certain preoccupation on the reputa­ rationale and main features of the project, devised as a tool that would tional risks associated with being perceived as imposing a rigid testing enable low- and middle-income countries to generate assessment data at program on recipient countries (Interviews DON7; PRI8). a comparatively low-cost while allowing them to report on SDG 4.1.1. Given the limited headway made by the CNA-extension option, in (UIS, 2019a). The project, currently in full swing as a collective 2017 the UIS centered its efforts in the development of an ensemble-like endeavor benefiting of the input of multiple partners, is indicative of the strategy allowing for the combination of multiple data sources, UIS’ willingness to reinforce its role as a provider of global goods and including CNAs and NAs. According to different interviewees, the idea of fortify its capacity-development function – by supporting countries in a common scale, against which different learning datasets could by their efforts to strengthen national statistical systems. plotted gradually, became attractive within UIS quarters. Not only did Overall, these developments suggest that, as a means to overcome the idea offer a way-out of the political divisiveness caused by those the political and technical impasses posed by the production of learning approaches privileging a limited set of CNAs, but it was also perceived as data, the UIS has relied on a bricolage strategy consisting of recombining better aligned with the principle of country ownership and a reporting a number of already available and legitimate models, recognizing the strategy at the service of countries’ statistical needs – rather that the limitations of each approach and emphasizing the potential for other way around (Interviews UN18, UN16, PRI4). The idea of a com­ complementarity. In other words, the UIS has accommodated the mul­ mon scale not imposing a specific data source enjoyed the support of a tiple (and contradictory) demands placed on the organization by variety of countries but also parties such as NGO and bilateral donors – resorting to a hybrid approach that maximizes data-source flexibility to whom in-built comparability was far from a deal-breaker, and who while recognizing the added-value of cross-national assessments. perceived the strategy as more conducive to data actionability and usability. 4.3. Bypassing the linking debate Despite the broad support enjoyed by the common-scale proposal, the idea was initially received with skepticism on the part of interna­ This bricolage strategy has, in fact, continued to drive UIS’ more tional assessment producers. In particular, the IEA and, to a lesser recent efforts in the learning measurement realm – with the so-called extent, the OECD, insisted upon the value of in-built comparability and ‘linking debate’ being one of the most illustrative examples of such the technical superiority of CNAs. Representatives of these organizations approach. Hence, by the end of 2017 a consensus had been attained argued vehemently that ex-post comparability was a problematic idea – regarding the convenience of using a common scale on which different and that the only way of producing reliable and accurate data was assessments could be mapped, and during 2019, this common metric, through the administration of a common test. As one OECD official put it known as Global Proficiency Framework (GPF) had been developed. ‘To be absolutely blunt, the only way you could compare results inter­ GPF was oriented at articulating ‘the minimum knowledge and skills nationally is if everybody takes the same test. There’s no other way’ that learners should be able to attain along their learning progressions at (Interview LASS3). each of the targeted grade levels in the two subject areas [reading and Anticipating that such emphasis on technical robustness and rigor mathematics]’ (GAML, 2019, p. 3). Importantly, and as remarked by risked having a paralyzing effect, the UIS responded by emphasizing the Smith and Benavot (2021), organizations such as USAID were need to avoid ‘letting the perfect be the enemy of the good’. This way, over-represented in the group behind the development of GPF. the notion of fit-for-purpose data started to gain traction within GAML In any case, while agreement had been reached regarding the content circles. In addition, and in order to appease the concerns of the leading and the minimum proficiency levels, there was still a need to decide on data producers (and the IEA in particular), the UIS coined the so-called an alignment strategy (i.e., a procedure to equate existing assessments to ‘stepping stone’ approach (Montoya, 2017). This strategy was support­ the common scale). Hence, the consensus on the GPF was paralleled by a ive of a variety of courses of action and was crucial in securing a new debate relative to the linking strategy. This soon proved to be a consensus among partners with different agendas and priorities. Hence, challenging endeavor for the UIS since, once again, different organiza­ this approach supported three different options, each of them associated tions favored different options and insisted upon the technical and/or with a different temporal horizon: political superiority of their own alternative. Discussed over the 2018–2020 period, such strategies not only differed in terms of technical • A short-term reporting strategy according to which countries are complexity and financial cost, but also in relation to more politically- allowed to submit data of their own choice; sensitive issues, including their item- and data-sharing implications, • A mid-term strategy oriented towards linking existing assessments to their potential to effectively inform education planning and contribute a common scale; to country capacity- building and, more importantly, the possibility to • Support for countries to join CNAs in order to create a critical mass of integrate NAs (cf. UIS, 2019b). comparable data – an objective with an undetermined horizon. Leaving aside the technicalities of the debate, the crucial point here is that, once again, such trade-offs (and in particular the possibility of Thus, by encouraging multiple streams of work rather than using NAs) have been strategically mobilized and exploited by the privileging a specific strategy, the UIS succeeded in breaking the dead­ different participating organizations. Thus, ACER has repeatedly called lock – with CNAs, NGOs and donors alike green-lighting the UIS strategy for the need to devise a system able to incorporate NAs, highlighting the (Crouch and Bernard, 2017; Montoya, 2017). In coherence with this benefits of such an approach in terms of coverage, country ownership all-encompassing approach, the UIS engaged in an effort to increase the and capacity-building – and arguing in favor of a less orthodox approach availability of learning data without privileging any source in particular. to comparability. Conversely, more heterodox methods such as policy- Thus, the UIS launched a series of publications prepared by consultants linking and item-based linking approaches have been harshly criti­ and oriented towards supporting both the establishment of NAs and cized by IEA and OECD specialists, who cast doubt on the validity and participation in CNAs. An example of this are the quick guides Imple­ reliability of this approach, and remain skeptical on the possibility of menting a National Learning Assessment and Making the Case for a Learning using NAs for global reporting purposes (Fontdevila, 2021). The Assessment – conceived respectively as a hands-on guide aimed to following excerpts express the contempt with which an analyst of a develop a NA; and to initiate a policy dialogue around the need for major assessment organization perceive policy-linking and item-based large-scale learning assessments (UIS, 2017; UIS, 2018a). linking: Also in line with this flexible, pragmatic approach, the UIS has been actively advocating and working to create a global bank of test items – What is being proposed, or what was proposed early on and still that is, a repository of test constructs crowdsourcing the items used in being talked about is, ‘Oh, we could use national assessments’. I’m 7 C. Fontdevila International Journal of Educational Development 96 (2023) 102684 sorry, that ain’t going to work. The national assessment in Honduras, 5. The UIS at a crossroads those results cannot really be compared to the results of national assessment in Australia, that’s just technically not feasible. But there The following section is concerned with the second objective of the are lots of people who argue that ‘No, you can draw up a common paper, namely, to analyze how the production of SDG4 learning data has scale’. You can take these assessment items and you can equate them impacted on the authority and credibility of the UIS within the global and link them. Okay, theoretically that could be done, but it is highly education field. To this end, the section delves first into the process of problematic. (Interview LASS3). organizational evolution through which the UIS managed to relatively enhance its authority within the learning measurement field; and then While the difficulty of reaching an expert consensus on the optimal turns to the circumstances that place the UIS in an ultimately fragile linking strategy risked having a paralyzing effect, the UIS has nurtured position and that appear to compromise its centrality. again a hybrid or ‘hedge-betting’ approach, insisting upon the fact that the alternatives are not mutually exclusive and can even reinforce one 5.1. The power of an honest broker another. Thus, the UIS has supported and commissioned technical work in relation to different options. Referred to as the portfolio approach, As discussed in the prior sections, the relative success of the UIS in this strategy emphasizes the idea that the different options ‘should be the area of learning measurement owes much to its democratization taken more as complementary routes than as alternative options in order strategy – as well as to its efforts to avoid a zero-sum approach and to minimize risk if some of the approaches prove to be too costly, the accommodate the use of different assessments and harmonization margin of error is too high, politically-unfeasible or a combination of methods, and to support countries in navigating the ‘assessment market’ these issues’ (UIS, 2018b, p. 19). This approach is also considered more (cf. Montoya and Crouch, 2019). Generally speaking, such strategies respectful of countries’ priorities and context-specificities – in that have been largely welcomed within the GAML community, for they have provides Member States with a menu of options. The rationale behind allowed actors to bypass a number of competitive dynamics and mis­ this pragmatist turn is captured by a GAML interviewee: givings that, in the past, had hindered global efforts towards the Silvia [Montoya, UIS Director] shifted into full pragmatics mode, and harmonization of learning data. just said, ‘I don’t have to pick a winner. So what we’re going to have In this sense, the UIS has made an effort to posit itself as an honest instead is a portfolio of options. So some countries want to do the broker driven by a public-service ethos, convening different parties and Rosetta Stone, fine’ – so she is giving something to the IEA. ‘And if interests and building consensus in a fraught arena riddled with vested they want to do the reporting scales, [she said] fine’. So in effect, she interests. While not all assessment partners are equally enthused by the pulled rank and said, ‘I’m not picking any of your models because it’s pragmatic turn pursued by the UIS, most GAML participants were for the countries to decide’. (Interview DON6). appreciative of the UIS’s role as a mediator driven by the common in­ terest, its efforts to create collaborative and inclusive spaces, and the At the moment of writing, the reporting strategy for Indicator 4.1.1, emphasis placed on the principle of country ownership. Such views are relies primarily on a consensus reached among major CNAs regarding captured in the words of an informant long-involved in the GAML space: the alignment of their respective proficiency levels aligned with the global minimum proficiency level.10 However, the UIS also supports the You cannot make progress in this work without involving organiza­ development of a much wider spectrum of strategies. These include the tions with high capacity. But then the question is how do you make so-called policy-linking method, which allows for the use of NA data; as sure that then the outputs of that do not privilege a particular or­ well as the psychometric test-based linking method – which builds on ganization? It’s a really delicate balancing exercise. and I think the the Rosetta Stone proposal advanced by the IEA and aligns existing UIS has given the credentials that they are not really favoring any regional assessments to IEA’s TIMSS and PIRLS achievement scales organization. I think they are trying to move with some people. But items (UNESCO, 2021; UIS, 2022b). More recently, and in the context of they also need to satisfy certain standards in terms of how they the COVID-19 crisis, the UIS has also launched the Monitoring Impacts collaborate, and what they make public, and what their agenda is. on Learning Outcomes (MILO) project in Africa, with the financial And [it] is not that easy. But from that point of view, I think the support of the GPE and ACER’s technical input. The MILO project makes GAML is trying to accommodate as many players as possible. it possible to link existing regional and national assessments with a (Interview UN17). global proficiency framework. Importantly, the project builds upon and Remarkably, the Institute has managed to turn a potential liability continuing to expand the aforementioned global bank of items, thus (namely, the imperfect character of global learning data and the polit­ maximizing country ownership and capacity development (UIS, 2022c). icized nature of the process) into an asset – an opportunity to affirm its As a final note of caution, and despite the significance of such de­ authority in the education measurement realm. Rather than casting the velopments and the growing consensus regarding data collection, reporting process as a purely technical challenge (or emphasizing expert reporting and harmonization standards, it is important to bear in mind knowledge as the most relevant attribute of the organization), the UIS that such agreements retain an unstable and fluctuating quality, and its has brought to the fore the political nature of the debate, and has tapped real strength will only be seen over the long haul. However, the objec­ into its aura of publicness, neutrality and commitment to the common tive of this paper is not to analyze the robustness or adequacy of the good as a means to bolster its credibility. indicator production efforts – let alone to speculate how the process will Similarly, rather than addressing technical rigor and ex-ante play out in the future. In this sense, the absence of a final agreement is comparability as supreme values, the UIS has emphasized the need to not judged to be an impediment but as an opportunity to set the focus on combine such principles with considerations for country ownership, and the structural dynamics shaping the process – rather than on an end- to accommodate a diversity of data sources. The emphasis on the need product whose specifics might simply reflect contingent circumstances. for fit-for-purpose data (as opposed to ‘perfectly accurate’ data) has ul­ timately contributed to reasserting UIS’ centrality. This is so as the 10 fitness-for-purpose criterion entails an element of judgement that cannot In this sense, and as observed by Smith and Benavot (2021), a certain hi­ rely exclusively on technical considerations – a role for which the UIS is erarchy of assessments seem to be emerging – presumably, as a temporary so­ lution. Hence, international assessments are prioritized over regional and ideally suited on account of its aura of neutrality. national assessments, and population-based assessments are seen as a last To summarize, it appears that when it comes to learning data, the UIS resort. Yet, as noted by the authors, a number of interrogation marks persist – derives its authority not from an appearance of scientific objectivity or for instance ‘it is unclear to what extent national assessments will be reshaped expertise, but from an explicit recognition of the ever-perfectibility of to meet the robust requirements for participation’ (p. 213). data, the necessarily provisional character of figures, and the political 8 C. Fontdevila International Journal of Educational Development 96 (2023) 102684 nature of the measurement debate. Such framing has proved functional the need for the UIS to comply with donors’ priorities and timeframes in that it has enhanced the centrality of the UIS in its role as honest makes it increasingly difficult for the Institute to rely on broker and standard-setter, rather than a mere data curator. consensus-building and GAML as the main decision-making strategies. In fact, the UIS has increasingly tended to rely on technical meetings (i. 5.2. A fragile position e., meetings bringing together donors and data producers) as a means to overcome political impasses. The reliance on such agreements is, how­ Although there is a certain agreement that the learning measurement ever, difficult to reconcile with the emphasis on participatory proced­ mandate associated with SDG4 has contributed to ensuring a much more ures from which the UIS derives its legitimacy. central role for the UIS, this position appears to be fragile. Likewise, UIS’ Finally, some measurement projects recently launched on the part of authority gains are more clearly or more explicitly recognized within the World Bank could also endanger UIS’ new-found centrality in the specific segments of the education-for-development field – such as learning measurement arena. This is for instance the case of the Human measurement experts and psychometricians within the GAML and TCG Capital Index (HCI), a new composite indicator launched in 2018 that circles. Thus, UIS’ new-found position is far from being a secure or combines metrics relative to different dimensions of human capital. The consolidated one. Indeed, as this section will discuss, a number of education component of the index aims precisely at capturing education intertwined factors appear to endanger the authority and centrality of quantity and quality and has motivated the development of a method­ the UIS. ology to harmonize assessment scores (Altinok, Angrist and Patrinos, The first concerns the UIS’ limited in-house expertise in the area of 2018). In addition, the launching of a Learning Target and a Learning learning measurement –which some deem a consequence of the com­ Poverty Indicator in 2019 have also resulted in the development of a bination of financial difficulties and/or UNESCO’s administrative ri­ new indicator combining measurements relative to school access and gidity. The UIS has thus tended to rely on external expertise to carry out learning into a single figure (World Bank, 2019). Overall, the production some of the technical work required to harmonize different assessments. of global learning data has gained considerable prominence in the World This explains UIS’s heavy reliance on organizations such as Brookings or Bank organizational agenda – a phenomenon perceived as an instance of ACER in the early days of the SDG4 indicator debate – even when such ‘mission creep’ within UNESCO and UNESCO-adjacent circles (PRI6). As collaborations came at a reputational cost. It also explains the fact that a UNESCO-affiliated interviewee noted: many UIS publications and initiatives launched over the last few years I’m quite concerned. So to me, this [HCI] didn’t bode well for the have been prepared in conjunction with consultancy firms, research Bank and its role. It really came across as a pretty half-baked idea. organizations and assessment consortia (e.g., IEA, ACER) and indepen­ What’s the sustainability here? Who’s all this for really? […] It dent researchers. While these collaborations ensure a certain degree of doesn’t seem to be in much of a leadership role, and my worry is it’s technical sophistication and allow the UIS to comply with the tight like other development agencies, they are more concerned about timeframe put in place by the global SDG reporting mechanisms, they having a branded product than actually having an impact. (Interview are also likely to turn into a double-edged sword in the long term. The UN9). ‘vicarious expertise’ acquired by the UIS by means of partnering with others may ultimately pose significant risks in terms of sustainability, Interestingly, the World Bank is not oblivious to the reputational and even legitimacy. risks associated with organizational overlap and has instituted a pre­ Secondly, the UIS’s rapport with UNESCO appears to be an uncertain emptive strategy to dissipate such fears – especially as duplications have one. On the one hand, the institutional distance between the UIS and come under public scrutiny (GEMR, 2018). Thus, in 2019, encouraged UNESCO has, to some extent, proved helpful to the UIS. On the other by some key donors, the World Bank and the UIS signed a Memorandum hand, UNESCO’s alignment and support to UIS’s renewed vision and of Understanding conceived as a means to give some stability to strategy in the learning measurement area remain an open question to inter-organization collaboration. While the impact of this partnership many. There is certainly an overall lack of clarity about the extent to will only be seen in the long-term, such dynamics are ultimately indic­ which UNESCO has been supportive of the UIS’ efforts in the learning ative that the learning assessment field is far from settled and retains an assessment domain, and some interviewees observed that the UIS’s ef­ unstable quality. forts to gain visibility have created some friction within UNESCO (UN18). This lack of alignment between the UIS and UNESCO’s prior­ 6. Conclusions ities appears to have ultimately had a detrimental effect on the activity of the Institute – especially since it has resulted in the UIS not receiving The study of the negotiation of SDG4 learning data offers an op­ the political and financial support necessary for the challenges entailed portunity to shed light on the intricacies and messy nature of global by the production of SDG4 indicators. indicator-making, as well as the impact that such processes may have on This brings us to the third factor – namely, the UIS’ economic situ­ the partaking organizations. The paper has first unpacked the data ation. An evaluation recently conducted by UNESCO’s Internal Over­ production stage. It has shown that, despite broad agreement on the sight Service (UNESCO, 2018) finds that the Institute has long been in a centrality of learning measurement, the production of Indicator 4.1.1 situation of economic distress, which has led to a significant downsizing has been far from straightforward and conflict-free. In line with the of its staff. Importantly, the bulk of UIS funding comes from bilateral findings advanced by the literature on global indicator-making, the organizations and private foundations. To be sure, UIS’s work on difficulties encountered in the process owe much to the collective nature learning outcomes stands out as one of the areas in which the Institute of data-collection processes. Thus, obstacles have not been exclusively has been more successful in terms of resource mobilization, securing technical in nature, but also stem from the difficulty of reconciling the considerable support on the part of the Bill & Melinda Gates Foundation multiple (and sometimes conflicting) interests and priorities of the data (Montoya, Beeharry and Woolf, 2019) and being one of the main ben­ suppliers engaged in such efforts. Specifically, the paper finds that the eficiaries of a DfID initiative oriented at improving education statistics11 increasing centrality of learning data has been seized by producers of (DfID, 2018). However, the overall lack of budget stability perpetuates cross-national assessments as an opportunity to consolidate and expand the Institute’s dependence on consultants to fill knowledge gaps or even their outreach and portfolio of activity– a process that inevitably creates to perform core tasks or mandates (e.g. leading in GAML). In addition, issues of rivalry and overlap, sometimes exacerbating decades-long conflicts. In consequence, the orchestration labor performed by the UIS has proved a particularly challenging enterprise. The UIS has been 11 Namely, the Better Education Statistics and global Action to improve learning tasked with keeping assessment producers on board (offering them (BESTA). reputational or material incentives powerful enough to get them to 9 C. Fontdevila International Journal of Educational Development 96 (2023) 102684 engage) while ‘taming’ the assessment industry – for instance, pre­ Conflict of interest venting certain data producers from imposing their methodological preferences, or from using SDG4 as a ‘product placement’ scheme. The author declares that there is no conflict of interest. The study has also shown how the production of SDG4 learning data has had a number of organizational effects (Freistein, 2016) on the UIS. Acknowledgements An immediate external effect has been the revitalization of the role of the UIS in the production of globally comparable data. The UIS has thus I am grateful to the three anonymous referees and to Oscar Valiente gained substantial visibility and centrality in an area in which it would for their insightful comments on earlier versions of this paper, and to have struggled to affirm its leadership. This reinvigoration appears to be Antoni Verger for his advice on the dissertation from which this paper largely the result of the UIS’ ability to posit itself as an honest broker derives. All omissions and errors remain my own. driven by a public-service ethos – one able to convene different parties and interests, and to build consensus in a fraught arena riddled with List of references vested interests. Remarkably, the UIS has succeeded in this by bringing to the fore the necessarily imperfect nature of global datasets. While the The references marked with an asterisk correspond to those docu­ construction of ‘good enough data’ typically occurs in the backroom, the ments gathered for the purposes of documentary analysis. Note however UIS has succeeded in bolstering its own credibility by doing exactly the this is not an exhaustive list – the complete relation can be found in opposite – that is, by exposing the messiness, complexity and political (Fontdevila, 2021). difficulties behind the production of globally comparable learning data. In this sense, the paper corroborates Rocha de Siqueira’s (2017) obser­ References vations that data producers recognize global datasets as inherently imperfect but accept such inaccuracies out of pragmatism. However, the Addey, C., 2018. The assessment culture of International Organisations: “From study also contends that the approximate and error-prone nature of philosophical doubt to statistical certainty” through the appearance and growth of Internatioinal Large-Scale Assessments. In: Alarcón, C., Lawn, M. (Eds.), Assessment global data is not simply accepted as a ‘lesser evil’ on the part of indi­ Cultures: Historical Perspectives. Peter Lang, Berlin, pp. 379–408. cator promulgators – imperfections can become an opportunity for in­ Adler, E., Pouliot, V., 2011. International practices. Int. Theory 3 (1), 1–36. ternational organizations to reassert their own authority. Altinok, N., Angrist, N., Patrinos, H.A., 2018. Global Data Set on Education Quality (1965–2015) (Policy Research Working Paper No. 8314). The World Bank, At the same time, the UIS’ position is by no means a solid one, largely Washington, D.C. as a consequence of the emergence of parallel measurement projects Arndt, C., 2008. The politics of governance ratings. Int. Public Manag. J. 11 (3), such as the World Bank’s Human Capital Index and Learning Poverty 275–297. Benavot, A., Smith, W.C., 2020. Reshaping quality and equity: global learning metrics as Indicator. This is compounded by the internal effects experienced by the a ready-made solution to a manufactured crisis. In: Wulff, A. (Ed.), Grading Goal UIS as a consequence of its reinvigorated role in the production of Four: Tensions, Threats, and Opportunities in the Sustainable Development Goal on learning data. Thus, the limited institutional alignment between the UIS Quality Education. Brill Publishing, Leiden, pp. 238–261. and UNESCO has translated into a lack of support that could ultimately Birdsall, N., Bruns, B., Madan, J., 2016. Learning Data for Better Policy: A Global Agenda (CGD Policy Paper No. 96). Center for Global Development, Washington, D.C. jeopardize both the success of the global reporting effort and the lead­ Cooley, A., 2015. The emerging politics of international rankings and ratings. A ership or pilotage capacity of the UIS. framework for analysis. In: Cooley, A., Snyder, J. (Eds.), Ranking the World. Grading Overall, the paper offers a number of empirically-informed insights States as a Tool of Global Governance. Cambridge University Press, Cambridge, pp. 1–38. into the production of global data – which remains a comparatively Fontdevila, C., 2021. Global governance as promise-making. Negotiating and Monitoring under-researched phase, in contrast with the conceptualization, use and Learning Goals in the Time of SDGs. PhD thesis. Universitat Autònoma de Barcelona. impact of global metrics. However, the results might also have impli­ Available at: https://www.tesisenred.net/handle/10803/672579#page1⁄41. Fontdevila, C., 2020. Learning assessments in the time of SDGs. New actors and evolving cations (or provide useful insight) for international development efforts. alliances in the construction of a global field. In: Wulff (Ed.), Grading Goal Four. Thus, the paper shows that the production of SDG4 learning data has Tensions, Threats, and Opportunities in the Sustainable Development Goal on Quality been very much shaped by a tension between the principles of in-built Education Brill | Sense, Leiden, The Netherlands/Boston, MA, pp. 262–279. Cussó, R., d’Amico, S., 2005. From development comparatism to globalization comparability and country ownership – the former being better served comparativism: Towards more normative international education statistics. Comp. by large cross-national assessments, the latter more likely to be fulfilled Educ. 41 (2), 199–216. by a hybrid approach that maximizes data-source flexibility. Such ten­ Custer, S., Sethi, T., 2017. Avoiding data graveyards: insights from data producers and users. In: Custer, S., Sethi, T. (Eds.), Avoiding Data Graveyards: Insights from Data sions are ultimately indicative of the fact that, in the absence of an Producers & Users in Three Countries. AidData at the College of William & Mary, honest broker, global reporting frameworks might end up trumping Williamsburg, VA, pp. 1–8. national statistical priorities. This is certainly the case when such Dahler-Larsen, P., 2014. Constitutive effects of performance indicators: getting beyond frameworks encourage ad-hoc measurement exercises exclusively unintended consequences. Public Manag. Rev. 16 (7), 969–986. Davis, K.E., Kingsbury, B., Merry, S.E., 2012. Introduction: Global governance by driven by comparability purposes, but lacking the granularity and fre­ indicators. In: Davis, K.E., Fisher, A., Kingsbury, B., Merry, S.E. (Eds.), Governance quency necessary to orient local and domestic policy-making. In this by Indicators Global Power through Quantification and Rankings. Oxford University sense, the findings of the paper echo an emerging literature concerned Press, Oxford, pp. 3–28. Wulff, A., 2020. The twists and turns in negotiating a global education goal: a civil with the risks of the disconnect between the supply and demand side of society perspective. In: Wulff, A. (Ed.), Grading Goal Four. Tensions, Threats and development data (Custer and Sethi, 2017; MacFeely and Barnat, 2017). Opportunities in the Sustainable Development Goal on Quality Education. Brill | The findings also suggest that, while the UIS has gone to great lengths to Sense, Leiden, The Netherlands/Boston, MA, pp. 28–64. DfID. 2018, Better Education Statistics and global Action to improve learning (BESTA). maximize data-source flexibility and ensure that SDG4 monitoring needs Annual Review. DfID. Retrieved from: https://devtracker.fcdo.gov.uk/projects/GB- do not trump domestic measurement efforts, many remain scarcely 1–204695/documents. informed of the opportunities and costs associated with different forms ECOSOC, 2017. Report of the High-level Group for Partnership, Coordination and Capacity-Building for Statistics for the 2030 Agenda for Sustainable Development of learning assessment. The responsibility to help countries navigate the (E/CN.3/2017/3). ECOSOC, New York. different assessment options cannot fall exclusively on the UIS or Erkkilä, T., Piironen, O., 2018. Rankings and Global Knowledge Governance. Higher UNESCO; efforts are also required from donors and development part­ Education, Innovation and Competitiveness. Palgrave Macmillan, Cham, ners who, through their technical and economic support, contribute Switzerland. Espeland, W.N., Stevens, M.L., 2008. A sociology of quantification. Eur. J. Sociol. /Arch. directly to shaping domestic priorities in the area of learning Eur. De. Sociol. 49 (3), 401–436. measurement. Freistein, K., 2016. Effects of indicator use: a comparison of poverty measuring instruments at the World Bank. J. Comp. Policy Anal.: Res. Pract. 18 (4), 366–381. 10 C. Fontdevila International Journal of Educational Development 96 (2023) 102684 Geertz, C., 1973. The interpretation of cultures: Selected essays. Basic Books,, New York. UIS, 2016b. Global Alliance to Monitor Learning. List of participants. UIS, Montreal, Kelley, J.G., & Simmons, B.A. (2014, August-September). The power of performance Canada (Retrieved from). http://gaml.uis.unesco.org/wp-content/uploads/sites indicators: rankings, ratings and reactivity in International Relations. Paper /2/2018/08/Confirmed-participants_GAML2-meeting_Website_20161101.pdf. presented at the Annual Meeting on the American Political Science Association, UIS, 2017. The Global Alliance to Monitor Learning (GAML): Governance and Washington D.C. Retrieved from: https://papers.ssrn.com/sol3/papers.cfm?abstrac organization. UIS, Montreal, Canada. http://gaml.uis.unesco.org/wp-content/uploa t_id=2451319. ds/sites/2/2018/10/gaml-governance-organization-2017-en.pdf. King, K., 2016. The global targeting of education and skill: policy history and UIS, 2018a. Making the Case for a Learning Assessment. UIS, Montréal. comparative perspectives. Comp.: A J. Comp. Int. Educ. 46 (6), 952–975. UIS, 2018b. Global Alliance to Monitor Learning (GAML): 2018 Progress report. UIS,, MacFeely, S., Barnat, N., 2017. Statistical capacity building for sustainable development: Montreal, Canada. http://gaml.uis.unesco.org/wp-content/uploads/sites/2/2018/1 developing the fundamental pillars necessary for modern national statistical systems. 0/GAML-5-Report.pdf. J. Int. Assoc. Off. Stat. 33 (4), 895–909. UIS, 2019b. SDG 4 Data Digest How to Produce and Use the Global and Thematic Merriam, S.B., Tisdell, E.J., 2016. Qualitative Research. A Guide to Design and Education Indicators. UIS, Montreal, Canada. Implementation, 4th ed. Jossey-Bass, Wiley, San Francisco, CA. UIS, 2022b. Rosetta Stone Policy Brief. Establishing a concordance between regional GEMR, 2018. Is global education data heading toward fragmentation? Retrieved from: (ERCE/PASEC) and international (TIMSS/PIRLS) assessments (UIS/UNESCO). UIS, https://world-education-blog.org/2018/09/19/is-global-education-data-heading-to Montréal. ward-fragmentation/*. UIS, 2022c. COVID-19 in Sub-Saharan Africa: Monitoring Impacts on Learning Montoya, S., 2017, July 12. A pragmatic and unified approach to measure learning Outcomes. Main report. UIS, Montréal. globally (Blog post – UIS). Retrieved from: http://uis.unesco.org/en/blog/pragmati UNESCO, 2018. Evaluation of the UNESCO Institute for Statistics (IOS/EVS/PI/170 REV). c-and-unified-approach-measure-learning-globally?=undefined&wbdisable=true. UNESCO, Paris. Montoya, S., 2015. Monitoring Education for All Global and Thematic Indicators. ETAG UNESCO, 2021. Global Education Monitoring Report. Non-state actors in education. Who Report to the EFA Drafting Group. Presentation delivered by the UIS Director. *. chooses? Who loses? UNESCO, Paris. Morgan, C., 2011. Constructing the OECD Programme for International Student UNGA, 2017. Resolution adopted by the General Assembly on 6 July 2017 - 71/313. Assessment. In: Pereyra, M.A., Kotthoff, H.-G., Cowen, R. (Eds.), PISA Under Work of the Statistical Commission pertaining to the 2030 Agenda for Sustainable Examination: Changing Knowledge, Changing Tests, and Changing Schools. Sense, Development (A/RES/71/313). United Nations General Assembly. *, New York. Rotterdam, pp. 47–59. Unterhalter, E., 2019. The many meanings of quality education: politics of targets and Mundy, K., 2010. Education for All and the global governors. In: D’Avant, D., indicators in SDG4. Global Policy 10 (S1), 39–51. Finnemore, M., Sell, S.K. (Eds.), Who Governs the Globe? Cambridge University Ward, M., 2004. Quantifying the World. UN Ideas and Statistics. Indiana University Press, Cambridge, pp. 333–355. Press, Bloomington, IN. Neumann, I.B., 2002. Returning practice to the linguistic turn: the case of diplomacy. World Bank, 2019. Learning Poverty: What Will It Take? The World Bank, Washington, Millenn.: J. Int. Stud. 31 (3), 627–651. D.C. (*). Ritchie, J., Lewis, J., Elam, G., 2003. Designing and selecting samples. In: Ritchie, J., GAML, 2019. Global Proficiency Framework: Reading and mathematics. Grades 2 to 6. Lewis, J. (Eds.), Qualitative Research Practice. A Guide for Social Science Students UIS. Retrieved from: https://gaml.uis.unesco.org/wp-content/uploads/sites/2/ and Researchers. SAGE, London, pp. 77–108. 2019/05/GAML6-REF-16-GLOBAL-PROFICIENCY-FRAMEWORK.pdf. Rocha de Siqueira, I., 2017. Development by trial and error: the authority of good Montoya, S., Crouch, L., 2019, April 26 and May, 20. The learning assessment market: enough numbers. Int. Political Sociol. 11 (2), 166–184. pointers for countries (Part 1 and 2). (Blog post – GEMR). Retrieved from: https Sachs-Israel, M., 2017. The SDG4-Education 2030 Agenda and its Framework For Action ://gemreportunesco.wordpress.com/2019/04/26/the-learning-assessment-marke – The process of its development and first steps in taking it forward. Bild. und Erzieh. t-pointers-for-countries-part-1/. 69 (3), 269–290. Montoya, S., Beeharry, G. & Woolf, E., 2019, January 22. A partnership for a global Sayed, Y., Ahmed, R., Mogliacci, R., 2018. The 2030 Global Education Agenda and the public good: data to improve learning (Blog post – UIS). Retrieved from: https://sdg. SDGs: process, policy and prospects. In: Verger, A., Novelli, M., Altinyelken, H.K. uis.unesco.org/2019/01/22/a-partnership-for-a-global-public-good-data-to-impr (Eds.), Global Education Policy and International Development, 2nd edition.,. ove-learning/. Bloomsbury, London/New York, pp. 185–208. Smith, W., Benavot, A., 2021. Quality education and global learning metrics. In: Crouch, L., Bernard, J.-M., 2017, July 6. Measure learning now to reduce inequality McCowan, T., Unterhalter, E. (Eds.), Education and International Development, 2nd tomorrow (Blog post – GPE). Retrieved from: https://www.globalpartnership.org/ ed. Bloomsbury, London, pp. 189–206. blog/measure-learning-now-reduce-inequality-tomorrow *. Treviño, E., Órdenes, M., 2017. Exploring Commonalities and Differences in Regional UIS 2022a. TCG Composition. Retrieved from: https://tcg.uis.unesco.org/tcg-compositio and International Assessments (UIS Information Paper No. 48). UIS, Montreal, n/*. Canada. UIS, 2019a. Global Alliance to Monitor Learning: Update on progress. Presentation UIE, 1962. Educational achievements of thirteen-year-olds in twelve countries. UNESCO delivered by the UIS Director in the context of the 6th GAML meeting. Retrieved Institute for Education, Hamburg (*). from: http://gaml.uis.unesco.org/wp-content/uploads/sites/2/2019/05/GAML6- UIS, 2016a. Proposal to develop global learning metrics. How to measure SDG 4.1. UIS, Session1-Progress.pdf *. Montreal, Canada (Retrieved from). http://uis.unesco.org/sites/default/files/do cuments/proposal-to-develop-global-learning-metrics.pdf. World Education Forum, 2000. The Dakar Framework for Action. UNESCO, Paris (*). 11