Psicothema (2023) 35(1) 05-20 Psicothema www.psicothema.com • ISSN 0214-9915 Colegio Oficial de Psicólogos del Principado de Asturias Article Coefficient Alpha: The Resistance of a Classic Eduardo Doval1, Carme Viladrich1 and Ariadna Angulo-Brunet2 1 Universitat Autònoma de Barcelona. 2 Universitat Oberta de Catalunya. ARTICLE INFO ABSTRACT Received: July 27, 2022 Background: During the 20th century the alpha coefficient (α) was widely used in the estimation of the internal Accepted: October 14, 2022 consistency reliability of test scores. After misuses were identified in the early 21st century alternatives became widespread, especially the omega coefficient (ω). Nowadays, α is re-emerging as an acceptable option for reliability estimation. Method: A review of the recent academic contributions, journal publication habits and recommendations Keywords: from normative texts was carried out to identify good practices in estimation of internal consistency reliability. Results: Internal consistency To guide the analysis, we propose a three-phase decision diagram, which includes item description, fit of the measurement Reliability model for the test, and choice of the reliability coefficient for test score(s). We also provide recommendations on the use Cronbach’s alpha of R, Jamovi, JASP, Mplus, SPSS and Stata software to perform the analysis. Conclusions: Both α and ω are suitable Omega for items with approximately normal distributions and approximately unidimensional and congeneric measures without Software extreme factor loadings. When items show non-normal distributions, strong specific components, or correlated errors, variants of ω are more appropriate. Some require specific data gathering designs. On a practical level we recommend a critical approach when using the software. Coeficiente Alfa: la Resistencia de un Clásico RESUMEN Antecedentes: Durante el siglo XX el coeficiente alfa (α) fue ampliamente utilizado en el cálculo de la consistencia Palabras clave: interna de las puntuaciones de los test. Después de identificar algunos malos usos, a principios del siglo XXI se Consistencia interna difundieron alternativas, especialmente el coeficiente omega (ω). Actualmente α resurge como una opción aceptable. Fiabilidad Método: Revisamos aportaciones académicas, hábitos de publicación en revistas y recomendaciones de textos Alfa de Cronbach Omega normativos con el fin de identificar buenas prácticas en la estimación de la fiabilidad de consistencia interna. Resultados: Software Para guiar el análisis, proponemos un diagrama de decisión en tres fases, a saber, descripción de los ítems, ajuste del modelo de medida del test y elección del coeficiente de fiabilidad de las puntuaciones. Para su ejecución proporcionamos recomendaciones sobre el uso de los programas R, Jamovi, JASP, Mplus, SPSS y Stata. Conclusiones: Tanto α como ω son adecuados para ítems que se distribuyen de forma aproximadamente normal y medidas aproximadamente unidimensionales y congenéricas sin cargas factoriales extremas. Cuando los ítems tienen otra distribución, un fuerte componente específico o sus errores están correlacionados, resultan más adecuadas variantes de ω. Algunas de ellas requieren diseños específicos de obtención de datos. A nivel práctico recomendamos un uso crítico del software. Cite as: Doval, E., Viladrich, C., & Angulo-Brunet, A. (2023). Coefficient Alpha: The Resistance of a Classic. Psicothema, 35(1), 05-20. https://doi.org/10.7334/ psicothema2022.321 Corresponding author: Carme Viladrich, carme.viladrich@uab.cat Doval et al. / Psicothema (2023), 35(1), 05-20 The use of questionnaires or test to score individuals on a independent of the true score, the total variance of the item sum construct or latent variable is common in social and health sciences or average is the sum of the systematic variance plus the error (test hereafter). Often, the test score is defined as the sum or the variance. Reliability coefficients are intended to quantify the average of each person’s responses to the test items and inferences proportion of systematic variance present in the total variance and about the construct must be based on sound psychometric properties therefore take values between 0 and 1, with high but not extreme of that score. Among other, evidence of test score reliability values (not close to one) being preferable. should be provided as stated in the standard 2.3 in Standards for The most widely used estimate of internal consistency reliability Educational and Psychological Testing (American Educational is Cronbach’s alpha coefficient (α), formulated by Cronbach (1951), Research Association [AERA] et al., 2014). One way to provide among other authors, in the framework of CTT (Gulliksen, 1950) this evidence is to calculate internal consistency reliability, which under the ideal scenario depicted above: unidimensional, essentia- is the focus of this paper. lly tau-equivalent measures with independent errors. Several equi- The internal consistency reliability of a test score is based on valent formulas can be used to compute α, but in essence it is a ratio the degree of association between the item responses obtained on a between the systematic variance and the total variance. single administration of the test to a group of persons. The calculus In the numerator, the average of covariances between items is very simple in an idealised measurement model, in which all quantifies the systematic variance. In the denominator, the sum of items assess a single construct (unidimensionality) with the same elements of the variance-covariance matrix of items quantifies the discriminating capacity (essentially tau-equivalent measures re- total variance. When the total score is the item sum (or average) flected in equal factor loadings). Moreover, measurement errors, and data are close to the idealised scenario described above, the considered present in all assessments, are random and unrelated resulting ratio is a good estimate of the proportion of systematic (independent errors). On the other hand, the persons assessed variance present in the observed score variance. conform a group with appreciable individual differences in the Coefficient α is also a particular case of the intra-class construct but homogeneous with respect to other characteristics. correlation coefficients derived from the Generalizability Theory Non missing responses to a long test composed by homogeneously (GT; Cronbach et al., 1963, see also Brennan, 2001). The main formatted items obtained from a big sample would bring this ideal objective of GT is to disentangle identifiable sources of error that scenario to completion. contribute to the error variance in CTT. In this case, different In realistic settings, measurement models are more complex. measures of reliability are derived from random effects repeated Test items may measure different constructs (multidimensionality), measures ANOVA. Among the best known intra-class correlation or measure the same main construct, reflected in a general factor, coefficients, the consistency coefficient for average measures with some of the items grouped into secondary factors or showing equals α. Based on covariation between repeated measures should correlated errors (essential unidimensionality; e.g., due to similarly not be confounded with the absolute agreement coefficient for worded items or to items that, in addition to the main dimension, average measures which includes the additional requirement of measure other minor dimensions). Moreover, items usually show equal means between repeated measures. different discrimination capacity (congeneric measures reflected In addition, internal consistency reliability can also be derived in different factor loadings) and may show specific variance from Factor Analysis (FA; Thurstone, 1947; see also Brown, 2015; not shared with other items and not assimilable to measurement Ferrando et al., 2022). If the items measure a single factor with error (e.g., items evaluating different facets of a single construct). uncorrelated errors, item responses can be explained by a common Furthermore, persons assessed may display other characteristics part (or factor loading) plus a unique part (or uniqueness). Assuming inducing heterogeneity or may even pertain to well-defined groups standardised factor scores, McDonald (1999) defined the omega or classes. In front of any of these complexities, the ideal scenario coefficient (ω) as the ratio between the common variance or the described above is no longer realistic, not to talk about missing square of the sum of the factor loadings, and the total variance data, short test length or small sample size. or the square of the sum of the factor loadings plus the sum of The best way to accommodate the estimation of reliability to the uniquenesses. This coefficient has also been referred to as the realistic settings has been a matter of intense debate since the end composite reliability or reliability of a composite score (Raykov, of the last century. To set the debate, the following few paragraphs 1997a), ωtotal (Revelle & Zinbarg, 2009) and ωu (Flora, 2020). outline the basic concepts of measurement theory posited by three Finally, the FA and the CTT measurement models are equivalent theories underpinning the calculation of an internal consistency when factor loadings are assimilated with the true variance and coefficient. uniquenesses are assimilated with the error variance (e.g., Green According to Classical Test Theory (CTT; Lord & Novick, & Yang, 2015). Then, α is a particular case of ω obtained in uni- 1968; see also Muñiz, 2018; Sijtsma & Pfadt, 2021), people’s dimensional data with uncorrelated errors and equal factor loadings responses to the test items are expected to correctly reflect for all items (essentially tau-equivalent measures). On the other individual differences in the construct by producing variability hand, unidimensional data with uncorrelated errors and different in the scores. This between-person variability is the focus of the factor loadings for some items (congeneric measures), will provide measurement and is referred to as systematic or true variance. The a value of α lower than that of ω. Both values, α and ω are equal responses of individuals will also depend on many other minor to or lower than the population reliability, and thus both are lower factors present in the assessment, such as fatigue or motivation, bounds of reliability except for sampling variability. Because both which are considered to produce unpredictable variability referred coefficients are estimates based on samples, their values must be to as error. If the item errors are independent of each other and accompanied by confidence intervals (Oosterwijk, et al., 2019). 6 Alpha Resists At the end of the 20th century, α had no clear competence as At the time of writing our text (September 2022) we found 696 an estimator of reliability, although misuses were already being citing papers of McNeish’s work in Web of Science. Among the reported (e.g., Cortina, 1993; Schmitt, 1996). From then on, a lively 598 that published empirical data, 46 (13.2%) reported only ω; debate about the best estimator of internal consistency reliability 207 (34.6%) reported α and another coefficient, generally ω; 251 has been growing. We will present the corners of the debate related (42.0%) reported only α; 21 (3.5%) reported ω and a coefficient to the methodological research, the publication habits of scientific other than α; 28 (4.7%) a coefficient other than ω or α; and 12 journals, and the position of normative institutions. (2.0%) did not report any reliability coefficient. That is, almost In the first decade of the 21st century, a lot of methodological the half of the McNeish’s work citing papers reported despite research was devoted to identifying and disseminating alternatives McNeish’s advice against it. for α as better (lower-bound) reliability estimators. A first group From normative positions, American Psychological Asso- of specialists advocated in favour of coefficients derived from ciation’s publication manual, indirectly recommended to report FA such as ω (Green & Yang, 2009; Raykov, 1997a; Yang & α up to its version 6. In version 7 (American Psychological Green, 2011), others argued in favour of coefficients not based Association, 2020), it is explicitly promoted to report the vali- on a specific measurement model, as the coefficient glb (greatest dation of the measurement model prior to the calculus of the lower bound; Sijtsma, 2009), while a third group argued the use reliability coefficient according to Appelbaum et al., (2018) and of several coefficients to express different aspects of internal con- Slaney et al., (2009) recommendations. Moreover, the possibility sistency reliability (Bentler, 2009; Zinbarg et al., 2005). Later, the of reporting α or other coefficients such as ω, jointly or separately, work of McNeish (2018) published in the journal Psychological has been adopted by European test evaluation commissions (e.g., Methods, advocated the use of several coefficients discouraging CET-R [The Questionnaire for the Assessment of Tests Revised explicitly α. Meanwhile, simulation studies have shifted the sponsored by the Test Commission of The National Association debate from the best lower bound estimate to the most accurate of Spanish Psychology], Hernández et al., 2016; EFPA model, estimate of the population reliability. No appreciable differences in [European Federation of Psychologists’ Associations], Evers et accuracy between α and ω have been reported for a large number al., 2013; COTAN model [Dutch Committee on Testing Affairs], of measurement models (e.g., Edwards et al., 2021; Gu et al., 2013; Evers et al., 2015). Also, in the methods for systematic reviews Raykov & Marcoulides, 2015). Additionally, some coefficients and meta-analyses, which can be considered normative for primary not based on a measurement model have shown unacceptable studies, the uncritical use of α is discouraged. The prior analysis of behaviour when tested for accuracy in simulation studies (Edwards the measurement model (Prinsen et al., 2018) and the publication et al., 2021; Sijtsma & Pfadt, 2021). Lastly, the coefficient ω has of the reliability coefficient best suited to the characteristics of the been criticised due to the large number of intermediate decisions data (Sánchez-Meca et al., 2021) are promoted instead. needed to obtain it (Davenport et al., 2016) and for the fact that Thus, the debate seems to have been settled in favour of α and ω does not refer to a single indicator but to a whole family of ω over coefficients not based on measurement models. A major coefficients, which may difficult comparison between studies advantage of ω would be its adaptability at first sight to more (Scherer & Teo, 2020; Viladrich et al., 2017). Based on some of sophisticated measurement models derived from FA, while the these reasons, a growing number of voices is calling for a return to greatest advantage of α would be its simplicity. However, whether α, even coming from authors who had previously advocated other α or other coefficient is used, the choice of coefficient(s) should not alternatives (Raykov et al., 2022; Sijtsma & Pfadt, 2021). Other be unreflective but justified. positions consider that the use of α or other coefficients should In light of what appears to be a new opportunity for α, we set depend on the fulfilment of their underlying assumptions (Green out to review our guidelines for using the more adequate reliability & Yang, 2015; Raykov & Marcoulides, 2016; Savalei & Reise, coefficient in different analytical scenarios (Viladrich et al., 2017). 2019; Viladrich et al., 2017). In addition, the joint publication of In that paper we distinguished the coefficient to be used according α and other alternative coefficient(s) has been proposed as a good to the nature of the data and the measurement model. We maintain practice (e.g. Bentler, 2021; Revelle & Condon, 2019). our alignment with the opinion that the choice of a reliability This debate among scholars has been ambiguously reflected coefficient depends on the measurement model that best fits the in the scientific journal publication habits. Flake et al. (2017) data (see also Green & Yang, 2015; Raykov & Marcoulides, 2016; analysed 301 papers and found that the 73% of them reported Savalei & Reise, 2019). Conversely, our position differs from those α. Some explanations were provided by the survey conducted who have recently defended the idea that should routinely replace by Hoekstra et al., (2019) of 664 researchers who published α α as an indicator of internal consistency reliability (e.g., Flora, in relevant journals in different fields. Although 88% reported 2020; Goodboy & Martin, 2020; Komperda et al., 2018). Our work knowing alternatives to α, 74% said they report α because that is also differs from those who suggest that the adequate coefficient standard practice in their field, 53% continue to report it because can be routinely obtained using point-and-click software (e.g., they believe they will be required to do so by the journal or the Kalkbrenner, 2021). review process, and 43% said it is the coefficient they were taught Thus, the first aim of this paper is to review criteria for decision- to calculate during their scientific training. making when estimating internal consistency reliability. For that Looking for extreme positions in reporting habits, we have reason, we examine the research showing when is adequate and reviewed the scientific papers citing the aforementioned McNeish’s what are the best alternatives, especially ω, when α is not adequate. (2018) work. Due to his position discouraging the use of α we In our view, the choice of the reliability coefficient is the outcome expected that these papers would mainly report other coefficients. of several successive decisions that we summarise in a flow chart 7 Doval et al. / Psicothema (2023), 35(1), 05-20 structured in three analytical steps. The second aim of this paper is acknowledges, their use is limited because they are based on to facilitate the application of these criteria when using statistical very demanding assumptions regarding the measurement model, software for reliability estimation. To accomplish that, we will and its efficiency in comparison with α and ω still needs further compare some of the most common or easy-to-use computer research. At this very moment, it would be safer to opt for a non- software for conducting the proposed three analytical steps ending linear measurement model based on item response theory (IRT) with an adequate estimation of the reliability coefficients. Finally, and deriving the reliability coefficient assimilable to those of some conclusions and recommendations will be derived with internal consistency derived from the TCT (e.g., Culpepper, 2013; importance for data analysts and scientific paper reviewers. Kim & Feldt, 2010; Raykov et al., 2010). On the other hand, if the irregularities in the distribution are due to low endorsement of When and Why to use α and/or ω some response categories, use can be made of the classical solution of grouping categories before starting the reliability analysis (e.g., The recommended uses of α and ω are based on different Agresti, 1996; DiStefano et al., 2020). aspects. These aspects are related to the nature of the variables or person groups, the factor loadings, the dimensionality of the test Homogeneous groups or the appropriate measurement models to describe the responses. The following is a review of the most recent literature on whether Regarding the homogeneity of respondents, when populations or not each of these characteristics conditions the use of α or ω. are structured in heterogeneous classes, parameter estimates may be biased and their standard errors incorrect, and it is Continuity and normality therefore recommended to identify the classes and calculate the reliability separately in each of them (Raykov et al., 2019). If the In the ideal scenario α can be used to estimate the reliability of heterogeneity is due to a multilevel structure, Lai (2021) proposes the sum or average of responses on a continuous scale. Considering using modifications of α and ω, although their behaviour in real that continuity is not a requirement, Chalmers (2018) proposes to data has not been sufficiently studied. use α also if the response scales are ordinal polytomous or even dichotomous. In contrast, the results of Xiao and Hau (2022) show Homogeneous factor loadings that in that case the biases may be high. Moreover, it should be kept in mind that with ordinal response scales, the measurement The diversity of factor loadings is the main source of difference model is often constructed invoking continuous latent variables between α and ω. Non homogenous factor loadings could for which discretised responses have been observed (Zumbo & derive from item content or simply from sizeable differences Kroc, 2019). For this reason, when faced with ordinal response between item variances (Graham, 2006). If the measurement formats, some authors choose to use the ordinal versions of α u model is unifactorial with uncorrelated errors, the coefficient α ω in which reliability is calculated in the metric of the continuous underestimates reliability even if only one factor loading is largely latent variables (Elosua & Zumbo, 2008; Gadermann et al., 2012; different from the others (Raykov, 1997b), especially when the Zumbo et al., 2007) and others opt for the non-linear or categorical number of items is small. However, when factor loadings are .70 version of ω (ωcategorical) in which the reliability coefficient is on average and discrepancies between factor loadings are below calculated in the metric of the discrete observed variables (Green & .20 in absolute value, the differences between α and ω are minimal Yang, 2009). Because of its metric, and as we argued in a previous (Raykov & Marcoulides, 2015). Recent simulation studies work (Viladrich et al., 2017), when the response scale is ordinal, published by Edwards et al. (2021) suggest that even with more in the present paper we are inclined to use either Cronbach’s α or extreme discrepancies, for example, with factor loadings between ωcategorical. .20 and .80 in samples of 100 or more cases, the underestimates are Moreover, in principle, the shape of the distribution of the slight, averaging .02 for 12 items and .04 for 6 items. Even a more responses of test items and, particularly, normality, is not a radical position can be seen in Raykov et al. (2022). necessary assumption for α (Raykov & Marcoulides, 2019). Additionally, the differences between the two coefficients are However, it is known that the distribution of items can affect the generally reflected from the third decimal place onwards if data estimation of covariances and correlations between them and was derived from a population reliability reasonable for practical thus the estimation of α. While in presence of positive kurtosis α purposes (close to .80). Larger differences are obtained only when underestimates reliability, in presence of negative kurtosis it may population reliability is extremely low. Furthermore, Deng and slightly overestimate reliability, biases attenuated in large samples Chan (2017), and Hussey and Hughes (2020), analysing real data, of for example of 1000 cases (Olvera et al., 2020). Additionally, report differences between α and ω in the third decimal place. That in presence of item skewness the estimation of reliability is is, despite calls of caution related to different factor loadings, in biased towards low values (Kim et al., 2020). Even worse, if the most cases there would be no practical difference between using departure from normality is remarkable, as happens with ceiling one coefficient or the other. or floor effects, all the results related to internal consistency reliability are affected, from the matrix of polychoric correlations Multidimensional tests (Foldnes & Grønneberg, 2020), through the determination of the test dimensionality (Christensen et al., 2022) to the calculation of Finally, the coefficients α and ω are not suitable for ω (Yang & Xia, 2019). Specific coefficients have been developed multidimensional tests measuring different constructs that do not to deal with these cases (Foster, 2021), but as the author himself share one general factor. Nevertheless, once the different factors 8 Alpha Resists have been identified, these coefficients can be calculated for each error. This may be achieved in several ways. In cross-sectional subscale separately (Bentler, 2021; Flora, 2020; Prinsen et al., designs, Bentler (2017) propose to measure auxiliary variables. 2018; Sijtsma & Pfadt, 2021). In longitudinal designs a preferrable approach would be using time-series concepts such as autocorrelated errors (Green & Measurement model Hershberger, 2000) or identifying item-specific factors based on repeated measures (Raykov, 2007). Anyhow, if the researcher The main concerns for using α and ω, arise when (a) the unique wants to differentiate variance components, the main decision variance of some items is not assimilable to measurement error, (b) no longer reduces to the choice of the best formula for reliability some item errors are correlated, or (c) minor factors are identified estimation but encompasses the data collection design recording in addition to the general factor. Put in other words, when the FA either auxiliary variables in a cross-sectional design or repeated results violate the CTT assumptions, and the two models are no measures in a longitudinal design. The reliability coefficient longer equivalent, not only α, but also ω, are in doubt. will be calculated as ω corrected (increased) by including the The first scenario is common in tests designed to measure broad specific predicted part as true variance in both the numerator concepts with few items, such as short personality tests. Item content and denominator (ω+ or specificity enhanced ω according to specificity, which will reflect in unshared variance or uniqueness, Bentler, 2017; ωi according to Raykov, 2007) or by including is necessary to achieve the measurement of the construct and the variability attributable to autocorrelated errors only in the therefore cannot be assimilated to measurement error. In this event, denominator (Green & Hershberger, 2000). both α and ω will markedly underestimate the reliability of the test As a summary of this section, Table 1 shows the reliability to the extent of invalidate the classical conclusion that reliability is coefficients α or ω that we recommend depending on the the upper limit of validity (McCrae, 2015). definition of true variance (rows) and quantitative (column 3) Even more concerning is the presence of correlated errors or ordinal (column 4) nature of the data. The recommendations between items or of minor factors. They are frequent in tests and are applicable to items with homogenous format, answered on a can be due to similarities in the meaning of some items, order scale of ordered categories, dichotomous or polytomous, for the effects, response format effects (e.g., Bandalos, 2021; Weijters et estimation of the reliability of the sum or average of the observed al., 2009) or the influence of specific identifiable factors beyond item responses, not of the hypothetical underlying continuous the common factor (Rodriguez et al., 2016a, 2016b). In these responses nor the factor scores. scenarios, the coefficients α and ω may sometimes underestimate The first setting is one of the most common: the analysis of and sometimes overestimate the population value of reliability, items measuring a single factor, even with factor loadings not thus losing the much appreciated guarantee of being conservative particularly homogeneous. In this situation, the use of α or ω estimates of reliability (Bentler, 2021; Raykov, 2001). for quantitative or categorical data would be perfectly justified, Three types of solutions have been proposed to deal with resulting in very similar values. In this setting, if some factor these concerns. All of them require judgment by the researcher. loadings show extreme values, the course of action is still a One option is to consider that the reliability of the construct matter of debate (Edwards et al., 2021; Raykov et al., 2022). The refers only to the common variance between all items. The other results obtained by Edwards et al. (2021) in a simulation study variance components, including the specific variance and the discourage the use of α with factor loadings out of the interval residual covariances or group factors, will be considered as part 0.2 – 0.8, although we did not find equivalent studies for ordinal of the measurement error and, consequently, the calculation of the data at present. All things considered, we think that the more internal consistency coefficients will be corrected (reduced) by conservative proposal would be the use of ω. including them only in the denominator. If this course of action The settings two to five provide solutions for other ways to is taken, formulas such as ωhierarchical (Zinbarg et al., 2005) or ω conceive the true variance. In the second setting, where some corrected for correlated errors (Raykov, 2004) will be useful. minor factors are found due to correlations between items not Note that the factor loadings for calculations should be derived explained by the general factor (essential unidimensionality) and from a FA with the appropriate measurement model, for example this variability is considered as true variance, ω should be derived the bifactor model with one common factor and some group from a bifactor model considering common and group variance factors, or the one-factor model with some correlated errors. as true variance. As can be seen in Table 1, this coefficient is Another option is to consider all shared variance as true developed for quantitative data and not yet for ordinal data. In variance, including the common variance and the covariances bet- the third setting we deal with the other option in front of essential ween some items or group factors. Consequently, them both will unidimensionality, where the group variance is considered error be included to the numerator and the denominator of the internal variance. In this case, we consider it more appropriate to use consistency reliability coefficient. If this option is accepted, the the ωhierarchical or ω corrected for correlated errors for quatitative formula for estimating internal consistency reliability would be ω data and ωh-cat for ordinal data. The fourth setting deals with with common and group factor loadings obtained from a bifactor items which specificity is considered as true variance within a model (Revelle & Zinbarg, 2009; Zinbarg et al., 2005) or even α. unidimensional model. In this case we think that ωi is the correct A third option is to differentiate these variance components option for estimating reliability, although this will only be by measuring predictors for variance and/or residual covarian- possible if it has been foreseen in the data collection design. Once ces. Variance accounted for by these covariates will turn into again, as far as we know, this type of coefficient has not been explained variance separated from random measurement developed for ordinal data. 9 Doval et al. / Psicothema (2023), 35(1), 05-20 Table 1. Recommended Use of α and ω Coefficients to Obtain Internal Consistency Reliability in Different Settings. Settings True variance Recommended coefficient for reliability of the item sum or average Quantitative: Five or more response categories and Ordinal: Four or less response categories linear relationship with normal errors and linearizable relationship 1 One-dimensional model (true variance = common variance) • α (Cronbach, 1951)** • α (Cronbach, 1951)** • ω (McDonald, 1999) • ωcategorical (Green y Yang, 2009) 2 Essentially one-dimensional model (true variance = common • α (Cronbach, 1951) • α (Cronbach, 1951) variance + minor factors variance) • ωtotal derived from bifactor (Zinbarg et al., 2005) • Categorical version not developed yet 3 Essentially one-dimensional model (true variance = common • ωhierarchical (Zinbarg et al., 2005) • ωh-cat (Flora, 2020) variance) • ωcorrected for correlated errors (Raykov, 2004) 4 One-dimensional model (true variance = common variance + • ωi (Raykov, 2007) • Categorical versión not developed yet specific variance) Note. α = Cronbach’s alpha; ω = omega (also ωu, ωtotal, or composite reliability); ωi = specificity-corrected omega (also ω+ or specificity enhanced); ωh-cat = omega hierarchical for categorical data. ** In setting 1, simulation studies with quantitative data favour the use of omega with extreme factor loadings. Equivalent studies for categorical data are not developed yet. The Choice of a Reliability Coefficient: A Three-Stage Analysis than to eliminate cases with missing data from the analysis (listwise), or treat missing data based on bivariate information (pairwise) From what was elaborated up to this point it should have which is the default in some software. It is quite a different matter to become clear that we discourage the analysis of the internal observe categories with low endorsement or no endorsement at all. consistency reliability of a test by choosing the default instruction There is no way to infer this kind of non-observed responses and that in the researcher’s preferred software. On the contrary, we share may pose a problem for further analysis. To further analyse these with other works the idea that this analysis involves a complex data as categorical or ordinal the researcher may choose to collapse but necessary process (Liddell & Kruschke, 2018; Savalei & some nearby categories (e.g., Agresti, 1996; DiStefano et al., 2020). Reise, 2019). We structure this process in three phases in which In previous phases of research, if the probability of endorsement of decisions are made successively: (1) the statistical description of some response categories is very low in the population, you may the items; (2) the fit of the measurement model for the test and consider gathering a very large sample of examinees or redesigning (3) the estimation of the internal consistency reliability of the test the response scale. score(s). The focus of this paper is on the third phase, but, as we have seen, the reasoned choice of the reliability coefficient in this Phase 1b: Homogeneity of Persons and Items third phase depends on the decisions made in the first two phases. Therefore, some guidelines for addressing the first two phases are Another task will be to assess whether the data come from a also given below. The three proposed phases are depicted in Figure homogeneous population. If so, we can proceed with the analysis 1. The vertical path highlighted and shaded in the left side of Figure as planned. On the other hand, if the data collection design has 1 depicts the analysis that leads to the calculation of the coefficient been multilevel, it is advisable to deal with heterogeneity using as recommended in the first line of Table 1, the most common multilevel analysis techniques (Cho et al., 2019; Hox et al., 2018). scenario. The more complex alternatives discussed through this If the heterogeneity stems from data coming from different po- paper are depicted in lighter colour and without shading. pulations, one option is to continue the analysis for each group separately. If the underlying populations are not known, they can Phase 1: Statistical Description of the Items be identified by cluster analysis or even by latent class analysis as proposed by Raykov et al. (2019). The aim of this first phase is to gain knowledge about the In addition, it is useful to inspect the homogeneity of the distribution of item responses, to detect the possible presence of relationships between items. Heterogeneous relationships an- missing data, and to inspect the subgroups of persons and items for ticipate possible deviations from unidimensionality that will possible patterns that may illuminate the modelling to be carried surface in the formal analysis in Phase 2. For quantitative data, out in the next phase. the variance-covariance matrix (or Pearson correlation matrix) can be examined. For categorical or ordinal data, the tetrachoric Phase 1a: Data Completeness (two response categories) or polychoric (more than two response categories) correlation matrix would be a better option. Visual The univariate description of the items provides information about inspection of these matrices may suffice if the number of items the response distribution including possible missing values. If data is moderate. More generally, the inspection can be conducted are complete, it is possible continue with the analysis as planned. If using multivariate statistical techniques such as exploratory factor some missing values are detected, it is recommended to use multiple analysis (EFA; e.g., Lloret-Segura et al., 2014), psychometric imputation whether the data are analysed as quantitative (Ferrando network analysis (e.g., Golino & Epskamp, 2017; see for practical et al., 2022) or as ordinal (Shi et al., 2020). Other possibilities are to application Pons et al., 2017), or multiple correspondence analysis use full information maximum likelihood estimation (FIML) during (e.g., Greenacre, 2017). Phase 2 or to further refine the analysis according to specialised texts The result of Phase 1 is a database for each population on which recommendations (e.g., Enders, 2010). All of these are better options the test measurement model will be formally studied during Phase 2. 10 Alpha Resists Data Phase 1: description matrix Complete data and No Impute data, group sufficient responses in all categories categories? Yes No: multilevel Multilevel analysis Homogeneous group? No: different populations Identify groups and analyse separately Yes Linear relationship with No IRT normal errors? Yes No: No: little deviation Linearisable Phase 2: measurement model ML/ULS* MLR/ULS* WLSM/ULSMV* FA (var-cov matrix) (var-cov matrix) (tetra/poli matrix) FIML FIML FIML Unidimensional No: essentially unidimensional measure? No: multidimensional Identify subscales and analyse separately Yes: error- Yes: specificity not assimilable assimilable to error specificity Phase 3: reliability α, α**, α**, ωi(?) ω (?), ω (ωcategorical) ω (ωcategorical) ωhierarchic (ωh-cat) Setting 1 Setting 4 Setting 1 Settings 2 & 3 Figure 1. Decision Making Diagram for Choosing a Reliability Coefficient. Note: IRT = Item response theory; FA = Factor analysis; ML = Maximum likelihood estimator; ULS = Unweigthed least squares estimator; * = use ULS in small samples, var-cov = Variance-covariance; FIML = Full information maximum likelihood estimator; MLR = Robust maximum likelihood estimator; WLSMV = Weighted least squares mean and variance corrected estimator; ULSMV = unweighted least squares mean and variance corrected estimator; tetra/pol = tetrachoric or polychoric correlations; Settings are defined in Table 1; α = Cronbach’s alpha; ω =omega (also ωu, ωtotal, or composite reliability); ωi = omega; ωi = specificity-corrected omega (also ω+ or specificity enhanced ); ωh-cat = omega hierarchical for categorical data. Coefficients that are separated by a comma may be reported together or one may be chosen in a reasoned manner. Between brackets are presented coefficients for ordinal data; ? represents coefficients that are currently not developed. 11 Doval et al. / Psicothema (2023), 35(1), 05-20 Phase 2: Analysis of the Measurement Model of the Test Phase 2b: Parameter Estimation and Model Fit The main objectives of this phase are to determine the For parameter estimation, either item FA or non-linear IRT dimensionality of the test as α and ω are only adequate for models can be used provided that data from large samples are unidimensional measures, and to estimate the parameters involved available. Many cases can be solved through FA using either in the calculation of ω. confirmatory (CFA) or exploratory (EFA) techniques (e.g., Bovaird & Koziol, 2012). In the simplest case, if the data are quantitative Phase 2a: Specification of the Measurement Model with item responses normally distributed, the use of the maximum likelihood (ML) estimator is recommended. As an alternative for The first step is to specify a reasonable relationship between slight deviations from normality, the use of the robust maximum the items and the latent variables or factors. If the relationships are likelihood estimator (MLR) is preferable. With ordinal data and assumed to be linear and the residuals normally distributed, limited a two-parameter model or a graded response model, the robust information estimation techniques can be used and the analysis in weighted least squares estimator with a χ2 statistic adjusted for Phase 2b is simplified. On the other hand, if the relationships are mean and variance (WLSMV) is considered a suitable option. specified as nonlinear, full information estimation techniques will The general solution of estimating the parameters through full be appropriate, the same techniques mentioned above for dealing information maximum likelihood (FIML) can always be chosen at with missing data. a higher computational cost. If response categories are five or more it is reasonable to treat If the sample size is small relative to the number of items, a items as quantitative and linearly related to the latent variables preferable option for FA for quantitative data may be the unweighted as long as the responses to the items follow a normal distribution least squares estimator (ULS; Ferrando et al., 2022) or for ordinal (Rhemtulla et al., 2012). In fact, they can be treated as normal if the data the unweighted least squares with a χ2 statistic adjusted for mean absolute values of skewness and kurtosis are not greater than 1 (e.g., and variance (ULSMV; Savalei & Rhemtulla, 2013). The number of Ferrando et al., 2022; Lloret-Segura et al., 2014). When moderate cases that is considered a small sample size is a difficult topic but, as deviations from normality are found, small corrections will suffice a guide, those analysed in the literature are in the order of 100 to 200 and will be discussed in Phase 2b. Otherwise, if extreme deviations cases (e.g., Forero et al., 2009; Savalei & Rhemtulla, 2013). are detected, such as those caused by floor or ceiling effects, a The output of Phase 2 is the measurement model of the test radical change of strategy should be considered. In this case, it will that (1) is theoretically sound, (2) displays good fit to the data, be advisable invoking other distributions of the residuals, such as and (3) displays better fit than alternative models compatible with the Poisson model (e.g., Foster, 2020; Muthén et al., 2016, cap. 7) or theory. Typically, the output will be either a unidimensional model, turn to non-linear models as described in the next paragraph. an essentially undimensional model or a multidimensional model. In contrast, if items are answered in a response scale of four or less categories, a linear relationship with the latent variables is no Phase 3: Estimation of the Internal Consistency Reliability of longer reasonable and therefore it is preferable to treat the data as the Score(s) categorical or ordinal (Rhemtulla et al., 2012). The relationship can take several forms, but in the common case that researchers As seen in the previous sections, the internal consistency are interested in a two-parameter model (item difficulty and item reliability of the score of a test with unidimensional structure discrimination), or the graded response model (difficulty of categories and specificity assimilable to measurement error can be obtained and item discrimination), then the relationships are linearizable either using α or ω, that will provide close values. Researchers by calculating polychoric or tetrachoric corre-lation coefficients. may also choose to report both types of coefficients. Conversely, Otherwise, if researchers are interested in more complex models, for if specificity is considered as true variance, the coefficient ωi will example, with more parameters, the alternative are the non-linear better reflect the reliability of the test score. IRT models (Culpepper, 2013; Kim & Feldt, 2010). On the other hand, if the measurement model is multidimensional, A good practice is to specify all measurement models α and/or ω can be calculated for each factor separately. When the compatible with the theory underlying the construct, analyse model is essentially unidimensional our recommendation would them one after the other and choose the one that best fits the data be to clarify whether to consider the entire non common part as and the purposes for which the test is to be used. When the test measurement error, which would be more consistent with reporting is intended to measure several constructs or factors, a typical the coefficient ωhierarchical, or whether to consider minor factors as sequence of nested models to check is (1) a flexible model allowing true variance which would be more consistent with reporting ω or item cross-loadings between factors, and (2) a restricted model even α. It may also be useful to report both types of coefficients where factors are congeneric measures with no cross-loadings. (e.g., Green & Yang, 2015). If the test measures only one construct, the sequence is reduced Finally, in all cases, it is good practice to report the confidence to step (2) and perhaps to checking (3) the model of essentially interval of the chosen internal consistency coefficient(s) or to report tau-equivalent measures. On the other hand, if heterogeneity is the Bayesian estimation of these coefficients (Pfadt et al., 2022). suspected in the relationships between items of one construct, a If the researcher chooses alternative coefficients that are beyond reasonable sequence of models to check would be (1) a bi-factor the objectives of this work, we recommend consulting specialized model, (2) the congeneric measurement model and, perhaps, (3) literature. This would happen, for example, with the coefficients the essentially tau-equivalent measurement model. derived from IRT or multilevel analysis, among other (Cho, 2022). 12 Alpha Resists Computer Software for Internal Consistency Reliability presents a guide and the syntax necessary to carry out Phase 2 and Estimation Phase 3 for unidimensional tests. Point and interval estimate of the omega coefficient are derived from CFA. Syntax of point and interval In this section, we present the current possibilities of widely used estimates for alpha coefficient is also provided. Complementarily, software to perform the three-stage analysis outlined above. In most Viladrich and Angulo-Brunet (2019) presents the syntax for Phase cases, the analysis can be fully developed using one or, at most, two 2 and Phase 3 to obtain ωhierarchical based on a confirmatory bifactor of them. We present open-source software R, Jamovi and JASP, and model. In all these syntaxes the reliability function, which is commercial software Mplus, SPSS and Stata. Jamovi, JASP, SPSS deprecated, can be replaced by the updated compRelSEM function. and Stata are menu-driven and can be complemented with syntax, As we have already said, if a multidimensional model is analysed, while syntax is always required in R and Mplus. Our comments the reliability of each factor can be obtained separately and thus, below refer to the analyses that can be managed through menus the procedure proposed in Viladrich et al. (2017) for unidimensional or syntax, explicitly ignoring the possibility of programming new tests can be applied to each factor. In addition to the confirmatory functions. Figure 2 summarises this information. analyses, the package psych (Revelle, 2022; Revelle & Condon, R (R Core Team, 2021). The data analyst can perform all the 2019) allows to obtain α, ω and ωhierarchical based on the exploratory analyses we have suggested for each of the three phases (i.e., bifactor model, which by default assumes three minor factors. This descriptive analysis of the items, fit of the measurement model of the exploratory option is not advisable (Savalei & Reise, 2019) due to test, and estimation of internal consistency reliability of test scores the fact that it can lead to reliability overestimation based on non- except coefficient ωi). The most convenient way to obtain results in R plausible model results. To the best of our knowledge, no R syntax is to adapt a ready-made syntax. The work of Viladrich et al. (2017) for ωi has been published so far. Phase 1: Data description Descriptives and frequencies Pearson correlations Tetracoric/polychoric correlations Latent profile analysis Cluster analysis Psychological networks Correspondence analysis EFA Phase II - Measurement model CFA Model Congeneric Correlated errors Bi-factor Multidimensional Estimator ML MLR WLSMV ULS FIML Phase III: reliability coefficient α CI α ω (EFA) CI ω (EFA) ω (CFA) CI ω (CFA) ωi CI ωi ω hierarchical CI ω hierarchical ω categorical CI ω categorical R Jamovi JASP Mplus SPSS Stata Available Not available Figure 2. Comparison of the Analytical Possibilities of Software to Complete the Three-stage Analysis for Reliability Estimation. Note: EFA = Exploratory factor analysis; CFA = Confirmatory factor analysis; ML = Maximum likelihood estimator; MLR = Robust maximum likelihood estimator; WLSMV = Weighted least squares mean and variance corrected estimator; ULS = Unweigthed least squares estimator; FIML = Full information maximum likelihood estimator; tetra/pol = tetrachoric or polychoric correlations; α = Cronbach’s alpha; ω = omega; ωi = specificity-corrected omega. CI = confidence interval. . 13 Doval et al. / Psicothema (2023), 35(1), 05-20 Jamovi (The Jamovi Project, 2021). All the analyses proposed Stata (StataCorp, 2021). All statistical techniques in the for Phase 1 can be carried out through menus. All the measurement descriptive phase are available except for psychological networks. The models we have dealt with in Phase 2 can be analysed downloading analysis of measurement models is generally performed with FIML the complementary module semlj (Gallucci & Jentschke, 2021) estimation. In the third phase, Viladrich et al. (2019) provide syntax which installs the SEM menu. In Phase 3, with the SEM menu, that facilitates the point and interval estimation of the coefficients you can get α, and the same ω options offered by the R package α and ω for unidimensional models, whereas Viladrich and Angulo- psych nowadays. Some particularities of this module are that it Brunet (2019) provide syntax for bifactor models and ωhierarchical. To does not implement the FIML estimator and that with categorical the best of our knowledge, the syntaxes for the calculation of the data it calculates the ordinal version of α (Zumbo et al., 2007). coefficients ωcategorical and ωi are not available at present. We discourage the routine use of the pre-installed Factor menu. Although it offers CFA and EFA for quantitative data, and the Discussion reliability analysis option calculates the α and ω coefficients, it should be kept in mind that omega in the output is only correct In this paper we have reviewed the current knowledge, for the congeneric unidimensional model, which is the default and practices and solutions concerning the estimation of the reliability cannot be assessed or modified by the user. of test scores based on an internal consistency design. The main JASP (JASP Team, 2022). All statistical techniques mentioned results from our review are presented as a flow-chart aimed to help in Phase 1 except correspondence analysis are available in JASP data analysts and paper reviewers. The flow-chart facilitates the menus. In Phase 2, all measurement models can be tested with the reasoned choice of the reliability coefficient for scores obtained by Factor menu if an exploratory strategy is chosen or with the SEM sum or average of items with dichotomous or polytomous ordered menu if a confirmatory strategy is prefered. For Phase 3, the menu category response scales. SEM provides point and interval estimation of α and ω. However, Our first conclusion is that the classical α derived from the ω is only correct for the unidimensional model with maximum variance-covariance matrix between items performs well in likelihood estimation which is the default and cannot be modified most cases. We are more optimistic than Bentler (2021) when he by the user. concludes on the uses of α by a laconic “That’s nice. But that’s it. Mplus (Muthén & Muthén, 2017). This commercial software And it’s not a lot” (p. 866). In our opinion it is quite a lot, at least offers the widest range of options for fitting the measurement by comparison with the uses of its best positioned competitor ω, model (Phase 2) and the first descriptive phase is also possible, although it is not enough because neither of the two coefficients except for multivariate techniques with no latent variables, such provide a correct estimation of reliability in all cases. In fact, there as cluster analysis, psychological network analysis and multiple is not a single coefficient that can do this job in all cases (Cho correspondence analysis. Again, the most convenient option is to 2022; Xiao & Hau, 2022). adapt ready-made syntaxes. Viladrich et al. (2019) provide a guide We consider it quite a lot due to compelling evidence in and the syntax for fitting the measurement model and estimating support of the close performance of α and ω when the data are the reliability of unidimensional tests using CFA. For bifactor approximately unidimensional the measures are congeneric with confirmatory models, see Viladrich and Angulo-Brunet (2019), for no extreme factor loadings and samples are large. Either coefficient bifactor exploratory models see García-Garzón et al. (2020). For would be correct in this scenario. And both would be incorrect for the computation of ωi see the syntax published by Sideridis et al. measurement models with correlated errors or items with a strong (2019). At the moment, we have not found published syntax to specific component. calculate directly ωcategorical in Mplus. Indirect options include copy- Studies comparing α and ω under different conditions show that, paste output values from Mplus to SAS (Yang & Xia, 2019) or in many cases, the difference between the two values is minimal. export output from Mplus to R using the function mplus2lavaan In our review we found that the conclusions in favour of ω in from the package MplusAutomation (Hallquist et al., 2022; syntax simulation studies are exaggerated as, under reasonable population available in Viladrich et al., 2019). reliabilities, the difference between the two coefficients reflects IBM SPSS (IBM Corp., 2021). All statistical techniques in form the third decimal place onwards. This adds to the Savalei and Phase 1 are available through menus except for psychological Reise’s (2019) conclusions that McNeish (2018) exaggerated the networks and latent class analysis. The extended command difference between the two coefficients and that the consequences of SPSSINC_HETCOR downloadable from IBM developerWorks the divergence for practical purposes would be trivial. allows the calculus of tetrachoric and polychoric correlations using In addition, using ω entails some dangers. The most serious an R package. Other options include the syntax by Lorenzo-Seva stems from the subjective decisions involved when fitting and Ferrando (2012; 2015). Measurement models in Phase 2 can the measurement model. Subjectivity can lead to much more be fitted using IBM SPSS AMOS (Arbuckle, 2014), an additional inadequate results and make replication difficult (Davenport et software for structural equation modelling with a limited number al., 2016; Edwards et al., 2021; Foster, 2021), not to mention the of the estimation methods mentioned in Phase 2. For Phase 3, bad practice of selecting an atheoretical but statistically fitting the command reliability of the core module provides the point measurement model obtained from refinement based on the results. estimation of α and, since version 27.0, also of ω for unidimensional In our opinion, the best way to deal with this danger is to make all models which is the default and cannot be modified by the user. As stages of the analysis transparent, including the availability of the an option in the reliability command, point and interval estimation database and the syntax used for the analysis. of α can be obtained as the intraclass correlation called consistency Thus, in front of proposals that only advise the publication of coefficient for average measures. some form of ω (e.g. Flora, 2020), we consider that α is appropriate 14 Alpha Resists in a wide variety of situations. We see coefficient α as simple to Savalei & Reise, 2019) and far from the point of view of other calculate, communicate and replicate, and differing from ω to the scholars who advocate for the dissemination of specific software third decimal place in simulation studies, so with practical utility that produces a proxi for ω in a few steps avoiding the assessment without a substantive loss in the rigour of reliability estimation. Let’s of the measurement model, as for example can be done the macro wait and see if this conclusion is further supported by the replication by Hayes and Coutts (2020) for SPSS. As simulation studies have of simulation studies in the future as proposed by Cho (2022). shown, in most cases, a good proxy for ω is simply α. For now, the most conservative position would be to report α In this area, our specific contribution consists of pointing out and ω, as proposed by Revelle and Condon (2019). The publication that not only the road is long but, in their curves, researchers will of α will facilitate direct comparison with other studies (in fact, find unexpected species in point-and-click-psychometrics such as α is still the most reported reliability coefficient). Additionally, cluster analysis, making decisions about the expected relationship the publication of ω will provide an estimation based on the between items and factors, about what parts of the variability of the measurement model. If the difference between the coefficient α responses are going to consider true or error variance, or what is a and ω was relevant, it would be worth discussing the reasons for reasonable form for the distribution of the residual variance. The this difference. reward will be a deep knowledge of their data, the human group It should also be clearly stated that both coefficients share who participated and the theory underlying their test. several limitations. To begin with, neither of them is useful for Another risk is to think that the empirical demonstrations on estimating the internal consistency reliability of scores derived the utility and efficiency of ω for unidimensional quantitative data from non-linear measurement models or with residual distributions are generalisable to any other version of the coefficient ω, as for largely departing from normality. For these cases, coefficients example, is implicit in the work by Flora (2020), in Lai (2021) or derived from linearised measurement models such as ωordinal in Bentler (2017). These papers introduce new coefficients based (Zumbo et al., 2007) and ωcategorical (Green & Yang, 2009) have on ω and usually provide a computer solution to calculate them. been dealt with in this text, but researchers should also consider On the one hand, the warning by Revelle and Condon (2019) coefficients derived from IRT models (Culpepper, 2013; Kim & against the temptation to apply reliability formulas to tetrachoric or Feldt, 2010) or Bayesian estimation applicable to a wide variety of polychoric correlation matrices and, for the other thing, the debate exponential distributions (Foster, 2020, 2021) which have not been about coefficient ωordinal (Chalmers, 2018; Yang & Green, 2015; discussed in this text. Zumbo & Kroc, 2019), has made us more cautious about jumping Furthermore, the use of α and ω is limited to the estimation to conclusions relative to new coefficients. That is why we have of the reliability of scores obtained by item sum or average. The changed our mind with respect to our previous work (Viladrich generalisation of these coefficients to estimate the reliability et al., 2017). We now believe that mathematical generalizations of factor scores can be seen in Rodriguez et al. (2016b) and in of ω to new analytical conditions should be accompanied by Ferrando and Lorenzo-Seva (2016, 2018). These papers address comparative empirical investigations, such as that by Yang and an even more important issue, namely the discussion of the Green (2015) or the more recent by Béland and Falk (2022) psychometric use-fulness of reliability coefficients compared showing their advantages. to other indicators of the quality of factor scores such as factor As far as the software is concerned, the implementation of the determinacy and the common variance explained by the general coefficient α is widespread while that of the coefficient ω is more factor. This is a very relevant practical issue because in popular restricted. If the data and model characteristics are aligned with analyses with structural equation models, the measure of latent the shaded path in Figure 1, the choice of software will not be a constructs is not obtained by item sum or average, but by factorial major problem for ω and even less so for α, since both coefficients combination of item responses. In short, although the extreme are generally available. As the characteristics of the data or model positions of the letters α and ω in the Greek alphabet suggest that move away from the ideal shaded in Figure 1 (e.g, the relationships they are coefficients located at antipodes, evidence show that they are non-linear, there are some correlated errors, data are ordinal) solve quite the same psychometric issues. the need to calculate a particular type of ω will also require access Another important point for practical purposes is that there to and knowledge of specialised software packages. We want to are no shortcuts to calculate α and ω. Indeed, one idea that has warn against the unthinking use of software under the heading survived the discussion of coefficients over the last few decades reliability or similar. Some of these, such as the Factor reliability is that whatever coefficient is used, the estimation of the internal menu of Jamovi, the reliability menu of JASP or the omega function consistency reliability comes after testing the measurement model. of the psych package in R, provide reliability results without the This idea is now well-established and included in normative user having control over the measurement model, whereas the texts such as the publication manual the American Psychological measurement model is of utmost importance as the numerator of Association (2020) or the methodological quality guidelines for ω is based on factor loadings. It should be noted that, for the time meta-analysis (Prinsen et al., 2018; Sánchez-Meca et al., 2021). In being, these solutions are based on unidimensional EFA and would other words, before calculating the internal consistency reliability only be appropriate when the data display the conditions shaded with α or ω, it must be checked that a FA of the items show results in the left of Figure 1. Additionally, documentation regarding compatible with the CTT. And we add that the right type of FA methods underlying an option in the menu is generally difficult to should be determined through the previous exploration of the follow, the exception being the well documented psych package data. Our view of the analysis as a three-stage journey is aligned (Revelle & Condon, 2019). Instead, we favour the use of functions with scholars that affirm that there are no quick ways to calculate such as compRelSEM from the semTools package in R that derive internal consistency reliability (e.g., Liddell & Kruschke, 2018; the calculation of reliability coefficients from the parameters 15 Doval et al. / Psicothema (2023), 35(1), 05-20 estimated when fitting the measurement model. In other words, known handicap of all reliability coefficients. These coefficients when data departs from the shaded path in Figure 1, researchers depend not only on the test but also on the group of people to and reviewers should only trust functions where ω is a subproduct whom it is applied and on the test correction procedure. According of a factor analysis defined by the data analyst and not obtained as to Ellis (2021) one way to deal with this situation is to explicitly a default in a statistical software. recognise that the same test can have multiple reliabilities. That Consistent with this, some comments on methods for reliability is, now that we have accepted that there is no single formula for generalisation meta-analysis are in order. As we have said, we estimating reliability, and that the best formula will depend on share the indication that the measurement model should be taken what is considered error for each intended use of a test, we have into account. However, once the unidimensional model has taken the first step in admitting that there is also no fixed number been fitted, the aggregation of reliability results is done without of designs to cover this purpose. For each proposed use of a test distinguishing between its estimators whether they are α or ω it will be necessary to justify what evidence of reliability would (Julio Sánchez-Meca et al., 2021) or only α (Prinsen et al., 2018). be compelling as recommended by Muñiz and Fonseca-Pedrero Perhaps not distinguishing between α and ωtotal might be a good (2019), Ziegler (2020) and, reflected in the different groups of idea since both coefficients share the definition of true variance and reliability evidence collected in the Standards for Educational thus, aim to estimate the same population parameter. On the other and Psychological Testing (AERA et al., 2014). This point of view hand, we consider results obtained with ω hierarchical or with ωi entails a clear extension of the three classic designs of internal as not aggregable either with each other or with α and ωtotal, since consistency, test-retest reliability and parallel measures, and the true variance is defined in a non-comparable way. Therefore, consequently also of the interpretation of the resulting coefficients. they should be treated separately, as it is common practice with A final recommendation for editors and reviewers would be other coefficients that not share with α the definition of true that in addition to assessing the choice of coefficient, importance variance such as the intraclass correlation coefficient of absolute should be given to the acceptable cut-off points and the report of agreement (Prinsen et al., 2018). We think that in all studies should confidence intervals. This message is not new, but we repeat it be explicitly reported which part of the variance of the responses because it seems to be difficult to apply. Although many authors has been considered as true variance. In this vein, Cho (2022), have provided cut-off points for reliability coefficients (see, for reaches a similar conclusion, and Scherer and Teo (2020) propose example, (DeVellis, 2003; Nunnally & Bernstein, 1994; or more the more drastic solution of performing reliability generalisation recently Kalkbrenner, 2021; Taber, 2018), mainly based on personal meta-analyses on the basis of the variance-covariance matrices and opinions (Streiner, 2003), the proposal of Nunnally and Bernstein not on the basis of the coefficients reported in the primary studies. (1994) is the most recognised. For example, the recommendations This type of analysis, named meta-analytic structural equation of the COTAN model are explicitly based on it COTAN (Evers et modelling or MASEM, is developing very fast for the study of al., 2013). Nunnally and Bernstein (1994) based their proposal on reliability generalization (Sánchez-Meca, 2022). the use of test scores and established two uses: using the scores Turning to the study design and data analysis, researchers must to obtain correlations with other variables or using them to make go beyond the mental framework of obtaining data with a single assessments of individuals. In the first case they set a reliability administration and consider a posteriori which is the best formula cut-off point at .80 to ensure that loss of reliability in the measures for estimating internal consistency reliability. In fact, it is necessary did not lead to a large attenuation in the correlations. In order to to be clear from the outset about all sources of variation to include obtain highly accurate measures for the second use they raised the them in the data collection design. For example, if the researcher minimum reliability value to .90. However, these authors are often wants to measure a conceptually broad construct with few items, cited to justify reliability values of .70, when they restrict this value those items will have sizeable specificity. This knowledge will to “early stages of predictive or construct validation research” (p. allow them to design the data collection in such a way that it is 264). Although it does not follow from their recommendations possible to estimate it and include it as true variance (Bentler, that any of these values should be taken as absolute benchmark, 2017; Raykov, 2007). Or perhaps a researcher may choose to nor are these recommendations supported by empirical studies, include some sources of error in the analysis of the measurement many researchers, reviewers and editors resort to them, especially model as is done for example in the work of Ferrando and Navarro- the lower criterion of .70, as absolute cut-off points (Cortina et González (2021) who, using a cross-sectional design, propose a al., 2020; see also Lance et al., 2006). It is also striking that it is data analysis model in which the error attributable to each person common to provide point estimates of the coefficients, either α or is estimated in order to quantify the role it plays in the reliability ω, without the confidence interval of the coefficient as an indicator of a test. of the level of precision of the estimate, which should be standard As novel as these proposals may seem, in our view, they add practice for sample estimates as claimed in normative texts (Evers to what was and continues to be the objective of GT since the et al., 2015; Prinsen et al., 2018; Sánchez-Meca et al., 2021). It fifties of the last century. As we have said, from this theory, the must be acknowledged that the value that should exceed the cut-off study of reliability is conceived as the identification and control of point is the lower limit of the confidence interval. An alternative possible sources of error in test scores. The designs and analyses treatment of uncertainty is that proposed in Pfadt et al. (2022) proposed by Bentler (2017), Ferrando and Navarro-González based on Bayesian estimation of these coefficients. (2021) Green and Hershberger (2000) or Raykov (2007) simply In a nutshell, if you plan to study the internal consistency promote the statistical control of the sources of error in front of the reliability of a test score, it would be advisable to (a) organize the experimental control initially adopted in GT. data gathering to include variables to account for all known sources We think that addressing the question of how to control or at of error; (b) analyse data exploring their completeness, shape, and least predict possible sources of error helps to address a well- relationships and test the measurement model fit; (c) report the 16 Alpha Resists interval estimation of the reliability, using α or other coefficients; example from psychiatry and a solution using mixture models. Applied and (d) gauge its value depending on the intended test use. Psychological Measurement, 46(3), 167–184. This paper does not intend to close the debate on the use of https://doi.org/10.1177/01466216211066602 internal consistency coefficients, much less on the estimation Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and of reliability. At present, the debate is so rich and wide-ranging applications. Journal of Applied Psychology, 78(1), 98–104. that addressing all its extremes would require much more space https://doi.org/10.1037/0021-9010.78.1.98 than is available in this paper. Moreover, as can be seen from the Cortina, J. M., Sheng, Z., Keener, S. K., Keeler, K. R., Grubb, L. K., Schmitt, references, this is a field in continuous development to which N., Tonidandel, S., Summerville, K. M., Heggestad, E. D., & Banks, G. much attention will have to be paid in the future. C. (2020). From alpha to omega and beyond! A look at the past, present, and (possible) future of psychometric soundness in the Journal of Applied References Psychology. Journal of Applied Psychology, 105(12), 1351–1381. https://doi.org/10.1037/apl0000815 American Educational Research Association, American Psychological Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Association & National Council on Measurement in Education. (2014). Psychometrika, 16(3), 297-334. The Standards for Educational and Psychological Testing. American Cronbach, L. J., Nageswari, R., & Gleser, G. C. (1963). Theory of Educational Research Association. generalizability: A liberation of reliability theory. The British Journal of Agresti, A. (1996). An introduction to categorical data analysis. Wiley. Statistical Psychology, 16(2), 137–163. American Psychological Association. (2020). Publication manual of the https://doi.org/10.1111/j.2044-8317.1963.tb00206.x American Psychological Association (7th ed.). Author. Culpepper, S. A. (2013). The reliability and precision of total scores and IRT https://doi.org/10.1037/000016S-000 estimates as a function of polytomous IRT parameters and latent trait Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., distribution. Applied Psychological Measurement, 37(3), 201–225. & Rao, S. M. (2018). Journal article reporting standards for quantitative https://doi.org/10.1177/0146621612470210 research in psychology: The APA publications and communications board Davenport, E. C., Davison, M. L., Liou, P-Y., & Love, Q. U. (2016). Easier task force report. American Psychologist, 73(1), 3–25. said than done: Rejoinder on Sijtsma and on Green and Yang. Educational https://doi.org/10.1037/amp0000191 Measurement: Issues and Practice, 35(1), 6–10. Arbuckle, J. L. (2014). Amos (Version 23.0) [Computer software]. IBM SPSS. https://doi.org/10.1111/emip.12106 Bandalos, D. L. (2021). Item meaning and order as causes of correlated Deng, L., & Chan, W. (2017). Testing the difference between reliability residuals in confirmatory factor analysis. Structural Equation Modeling, coefficients alpha and omega. Educational and Psychological Measurement, 28(6), 903–913. https://doi.org/10.1080/10705511.2021.1916395 77(2), 185–203. https://doi.org/10.1177/0013164416658325 Béland, S., & Falk, C. F. (2022). A comparison of modern and popular DeVellis, R. F. (2003). Scale development. Theory and applications. Sage. approaches to calculating reliability for dichotomously scored items. DiStefano, C., Shi, D., & Morgan, G. B. (2020). Collapsing categories is often Applied Psychological Measurement, 46(4), 321–337. more advantageous than modeling sparse data: Investigations in the CFA https://doi.org/10.1177/01466216221084210 framework. Structural Equation Modeling, 28(2), 237–249. Bentler, P. M. (2009). Alpha, dimension-free, and model-based internal https://doi.org/10.1080/10705511.2020.1803073 consistency reliability. Psychometrika, 74(1), 137–143. Edwards, A. A., Joyner, K. J., & Schatschneider, C. (2021). A simulation study https://doi.org/10.1007/s11336-008-9100-1 on the performance of different reliability estimation methods. Educational Bentler, P. M. (2017). Specificity-enhanced reliability coefficients. Psychological and Psychological Measurement, 81(6), 1–29. Methods, 22(3), 527–540. https://doi.org/10.1037/met0000092 https://doi.org/10.1177/0013164421994184 Bentler, P. M. (2021). Alpha, FACTT, and beyond. Psychometrika, 86(4), 861– Ellis, J. L. (2021). A test can have multiple reliabilities. Psychometrika, 86(4), 868. https://doi.org/10.1007/s11336-021-09797-8 869–876. https://doi.org/10.1007/s11336-021-09800-2 Bovaird, J. A., & Koziol, N. A. (2012). Measurement models for ordered- Elosua, P., & Zumbo, B. D. (2008). Reliability coefficients for ordinal response categorical indicators. In R. H. Hoyle (Ed.), Handbook of Structural scales. Psicothema, 20(4), 896–901. Equation Modeling (pp. 495–511). Guilford. Enders, C. K. (2010). Applied missing data analysis. Guilford. Brennan, R. L. (2001). Generalizability theory. Springer. Evers, A., Lucassen, W., Meijer, R., & Sijtsma, K. (2015). COTAN Review Brown, T. A. (2015). Confirmatory factor analysis for applied research. System for Evaluating Test Quality. Guilford. https://psynip.nl/wp-content/uploads/2022/05/COTAN-review-system-for- Chalmers, R. P. (2018). On misconceptions and the limited usefulness of ordinal evaluating-test-quality.pdf alpha. Educational and Psychological Measurement, 78(6), 1056–1071. Evers, A., Muñiz, J., Hagemeister, C., Hstmælingen, A., Lindley, P., Sjöberg, A., https://doi.org/10.1177/0013164417727036 & Bartram, D. (2013). Assessing the quality of tests: Revision of the EFPA Cho, E. (2022). The accuracy of reliability coefficients: A reanalysis of existing review model. Psicothema, 25(3), 283–291. simulations. Psychological Methods. Advance online publication. https://doi.org/10.7334/psicothema2013.97 https://doi.org/10.1037/met0000475 Ferrando, P. J., & Lorenzo-seva, U. (2016). A note on improving EAP trait Cho, S. J., Shen, J., & Naveiras, M. (2019). Multilevel reliability measures estimation in oblique factor-analytic and item response theory models. of latent scores within an item response theory framework. Multivariate Psicológica, 37, 235–247. Behavioral Research, 54(6), 856–881. Ferrando, P. J., & Lorenzo-Seva, U. (2018). Assessing the quality and https://doi.org/10.1080/00273171.2019.1596780 appropriateness of factor solutions and factor score estimates in exploratory Christensen, W. F., Wall, M. M., & Moustaki, I. (2022). Assessing dimensionality item factor analysis. Educational and Psychological Measurement, 78(5), in dichotomous items when many subjects have all-zero responses: An 762–780. https://doi.org/10.1177/0013164417719308 17 Doval et al. / Psicothema (2023), 35(1), 05-20 Ferrando, P. J., Lorenzo-seva, U., Hernández-Dorado, A., & Muñiz, J. (2022). Gu, F., Little, T. D., & Kingston, N. M. (2013). Misestimation of reliability Decalogue for the factor analysis of test items. Psicothema, 34(1), 7–17. using coefficient alpha and structural equation modeling when assumptions https://doi.org/10.7334/psicothema2021.456 of tau-equivalence and uncorrelated errors are violated. Methodology, 9(1), Ferrando, P. J., & Navarro-González, D. (2021). Reliability and external validity 30–40. https://doi.org/10.1027/1614-2241/a000052 of personality test scores: The role of person and item error. Psicothema, Gulliksen, H. (1950). Theory of mental tests. Wiley. 33(2), 259–267. https://doi.org/10.7334/psicothema2020.346 Hallquist, M., Willey, J., van Lissa, C., & Morillo, D. (2022). MplusAutomation: Flake, J. K., Pek, J., & Hehman, E. (2017). Construct validation in social an R package for facilitating large-scale latent variable analyses in Mplus and personality research: Current practice and recommendations. Social (1.1.0) [Computer software]. Psychological and Personality Science, 8(4), 370–378. https://michaelhallquist.github.io/MplusAutomation/ https://doi.org/10.1177/1948550617693063 Hayes, A. F., & Coutts, J. J. (2020). Use omega rather than Cronbach’s alpha for Flora, D. B. (2020). Your coefficient alpha is probably wrong, but which estimating reliability. But… Communication Methods and Measures, 14(1), coefficient omega is right? A tutorial on using R to obtain better reliability 1–24. https://doi.org/10.1080/19312458.2020.1718629 estimates. Advances in Methods and Practices in Psychological Science, Hernández, A., Ponsoda, V., Muñiz, J., Prieto, G., & Elosua, P. (2016). Revisión 3(4), 484–501. https://doi.org/10.1177/2515245920951747 del modelo para evaluar la calidad de los tests utilizados en España Foldnes, N., & Grønneberg, S. (2020). Pernicious polychorics: The impact [Assessing the quality of tests in spain: revision of the spanish test review and detection of underlying non-normality. Structural Equation Modeling, model]. Papeles Del Psicólogo, 37(3), 192–197. 27(4), 525–543. https://doi.org/10.1080/10705511.2019.1673168 Hoekstra, R., Vugteveen, J., Warrens, M. J., & Kruyen, P. M. (2019). An Forero, C. G., Maydeu-Olivares, A., & Gallardo-Pujol, D. (2009). Factor empirical analysis of alleged misunderstandings of coefficient alpha. analysis with ordinal indicators: A Monte Carlo study comparing DWLS International Journal of Social Research Methodology, 22(4), 351–364. and ULS estimation. Structural Equation Modeling, 16(4), 625–641. https://doi.org/10.1080/13645579.2018.1547523 https://doi.org/10.1080/10705510903203573 Hox, J., Moerbeek, M., & van de Schoot, R. (2018). Multilevel analysis: Foster, R. C. (2020). A generalized framework for classical test theory. Journal Techniques and applications. Routledge. of Mathematical Psychology, 96, Article 102330. Hussey, I., & Hughes, S. (2020). Hidden invalidity among 15 commonly used https://doi.org/10.1016/j.jmp.2020.102330 measures in social and personality psychology. Advances in Methods and Foster, R. C. (2021). KR20 and KR21 for some nondichotomous data (It’s Practices in Psychological Science, 3(2), 166–184. not just Cronbach’s alpha). Educational and Psychological Measurement, https://doi.org/10.1177/2515245919882903 81(6), 1172–1202. https://doi.org/10.1177/0013164421992535 IBM Corp. (2021). IBM SPSS Statistics for Windows, Version 28.0 (28.0) Gadermann, A. M., Guhn, M., & Zumbo, B. D. (2012). Estimating ordinal [Computer software]. IBM Corp. reliability for likert-type and ordinal item response data: A conceptual, JASP Team. (2022). JASP (Jeffreys’s Amazing Statistics Program) (0.16.2) empirical, and practical guide. Practical Assessment, Research and [Computer software]. https://jasp-stats.org/ Evaluation, 17(3), 1–13. Kalkbrenner, M. T. (2021). Alpha, omega, and H internal consistency reliability Gallucci, M., & Jentschke, S. (2021). Semlj: Jamovi SEM Analysis [Computer estimates: Reviewing these options and when to use them. Counseling software]. https://semlj.github.io Outcome Research and Evaluation, Published. Advance online publication, García-Garzón, E., Nieto, M. D., Garrido, L. E., & Abad, F. J. (2020). Bi-factor 1–12. https://doi.org/10.1080/21501378.2021.1940118 exploratory structural equation modeling done right: using the slidapp Kim, S., Lu, Z., & Cohen, A. S. (2020). Reliability for tests with items application. Psicothema, 32(4), 607–614. having different numbers of ordered categories. Applied Psychological https://doi.org/10.7334/psicothema2020.179 Measurement, 44(2), 137–149. https://doi.org/10.1177/0146621619835498 Golino, H. F., & Epskamp, S. (2017). Exploratory graph analysis: A new Kim, Seonghoon, & Feldt, L. S. (2010). The estimation of the IRT reliability approach for estimating the number of dimensions in psychological research. coefficient and its lower and upper bounds, with comparisons to CTT PLoS ONE, 12(6), 1–26. https://doi.org/10.1371/journal.pone.0174035 reliability statistics. Asia Pacific Education Review, 11(2), 179–188. Goodboy, A. K., & Martin, M. M. (2020). Omega over alpha for reliability https://doi.org/10.1007/s12564-009-9062-8 estimation of unidimensional communication measures. Annals of the Komperda, R., Pentecost, T. C., & Barbera, J. (2018). Moving beyond alpha: A International Communication Association, 44(4), 422–439. primer on alternative sources of single-administration reliability evidence for https://doi.org/10.1080/23808985.2020.1846135 quantitative chemistry education research. Journal of Chemical Education, Graham, J. (2006). Congeneric and (essentially) tau-equivalent estimates of score 95(9), 1477–1491. https://doi.org/10.1021/acs.jchemed.8b00220 reliability. What they are and how to use them. Educational and Psychological Lai, M. H. C. (2021). Composite reliability of multilevel data: It’s about Measurement, 66(6), 930–944. https://doi.org/10.1177/0013164406288165 observed scores and construct meanings. Psychological Methods, 26(1), Green, S. B., & Hershberger, S. L. (2000). Correlated errors in true score models 90–102. https://doi.org/10.1037/met0000287 and their effect on coefficient alpha. Structural Equation Modeling, 7(2), Lance, C., Butts, M. M., & Michels, L. C. (2006). The sources of four commonly 251–270. https://doi.org/10.1207/S15328007SEM0702_6 reported cutoff criteria: What did they really say? Organizational Research Green, S. B., & Yang, Y. (2009). Reliability of summed item scores using structural Methods, 9(2), 202–220. https://doi.org/10.1177/1094428105284919 equation modeling: An alternative to coefficient alpha. Psychometrika, 74(1), Liddell, T. M., & Kruschke, J. K. (2018). Analyzing ordinal data with metric 155–167. https://doi.org/10.1007/s11336-008-9099-3 models: What could possibly go wrong? Journal of Experimental Social Green, S. B., & Yang, Y. (2015). Evaluation of dimensionality in the assessment Psychology, 79, 328–348. https://doi.org/10.1016/j.jesp.2018.08.009 of internal consistency reliability: coefficient alpha and omega coefficients. Lloret-Segura, S., Ferreres-Traver, A., Hernández-Baeza, A., & Tomás-Marco, Educational Measurement: Issues and Practice, 34(4), 14–20. I. (2014). El análisis factorial exploratorio de los ítems: una guía práctica, https://doi.org/10.1111/emip.12100 revisada y actualizada [The exploratory factor analysis of the items: a Greenacre, M. (2017). Correspondence analysis in practice (3rd ed.). Chapman practical guide, revised and updated]. Anales de Psicología, 30(3), 1151– & Hall. https://doi.org/10.1201/9781315369983 1169. https://doi.org/10.6018/analesps.30.3.199361 18 Alpha Resists Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Raykov, T. (2004). Point and interval estimation of reliability for multiple- Addison Wesley. component measuring instruments via linear constraint covariance structure Lorenzo-Seva, U., & Ferrando, P. J. (2012). TETRA-COM: A comprehensive modeling. Structural Equation Modeling, 11(3), 452–483. SPSS program for estimating the tetrachoric correlation. Behavior Research https://doi.org/10.1207/s15328007sem1103 Methods, 44(4), 1191–1196. https://doi.org/10.3758/s13428-012-0200-6 Raykov, T. (2007). Reliability of multiple-component measuring instruments: Lorenzo-Seva, U., & Ferrando, P. J. (2015). POLYMAT-C: A comprehensive Improved evaluation in repeated measure designs. British Journal of SPSS program for computing the polychoric correlation matrix. Behavior Mathematical and Statistical Psychology, 60(1), 119–136. Research Methods, 47(3), 884–889. https://doi.org/10.1348/000711006X100464 https://doi.org/10.3758/s13428-014-0511-x Raykov, T., Anthony, J. C., & Menold, N. (2022). On the importance of McCrae, R. R. (2015). A more nuanced view of reliability: specificity in the coefficient alpha for measurement research: loading equality is not necessary trait hierarchy. Personality and Social Psychology Review, 19(2), 97–112. for alpha’s utility as a scale reliability index. Educational and Psychological https://doi.org/10.1177/1088868314541857 Measurement. Advance online publication. McDonald, R. P. (1999). Test theory: a unified treatment. Lawrence Erlbaum https://doi.org/10.1177/00131644221104972 Associates. Raykov, T., Dimitrov, D. M., & Asparouhov, T. (2010). Evaluation of scale McNeish, D. (2018). Thanks coefficient alpha, we’ll take it from here. reliability with binary measures using latent variable modeling. Structural Psychological Methods, 23(3), 412–433. Equation Modeling: A Multidisciplinary Journal, 17(2), 265–279. https://dx.doi.org/10.1037/met0000144 https://doi.org/10.1080/10705511003659417 Muñiz, J. (2018). Introducción a las Teorías Psicométricas [Introduction to Raykov, T., & Marcoulides, G. A. (2015). A direct latent variable modeling Psychometric Theories]. Pirámide. based method for point and interval estimation of coefficient alpha. Muñiz, J., & Fonseca-Pedrero, E. (2019). Ten steps for test development. Educational and Psychological Measurement, 75(1), 146–156. Psicothema, 31(1), 7–16. https://doi.org/10.7334/psicothema2018.291 https://doi.org/10.1177/0013164414526039 Muthén, B. O., Muthén, L. K., & Asparouhov, T. (2016). Regression and Raykov, T., & Marcoulides, G. A. (2016). Scale reliability evaluation mediation analysis using Mplus. Muthén & Muthén. under multiple assumption violations. Structural Equation Modeling: A Muthén, L. K., & Muthén, B. O. (2017). Mplus User’s Guide (8th edition). Multidisciplinary Journal, 23(2), 302–313. Muthén & Muthén. https://doi.org/10.1080/10705511.2014.938597 Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory. McGraw-Hill. Raykov, T., & Marcoulides, G. A. (2019). Thanks coefficient alpha, we still Olvera, O. L., Kroc, E., & Zumbo, B. D. (2020). The role of item distributions on need you! Educational and Psychological Measurement, 79(1), 200–210. reliability estimation: the case of Cronbach’s coefficient alpha. Educational https://doi.org/10.1177/0013164417725127 and Psychological Measurement, 80(5), 825–846. Raykov, T., Marcoulides, G. A., Harrison, M., & Menold, N. (2019). Multiple- https://doi.org/10.1177/0013164420903770 component measurement instruments in heterogeneous populations: Is there Oosterwijk, P. R., van der Ark, L. A., & Sijtsma, K. (2019). Using confidence a single coefficient alpha? Educational and Psychological Measurement, intervals for assessing reliability of real tests. Assessment, 26(7), 1207– 79(2), 399–412. https://doi.org/10.1177/0013164417733305 1216. https://doi.org/10.1177/1073191117737375 Revelle, W. (2022). psych: Procedures for personality and pscychological Pfadt, J. M., van den Bergh, D., Klaas, S., Moshagen, M., & Wagenmakers, research (2.2.5) [Computer software]. E.-J. (2022). Bayesian estimation of single-test reliability coefficients https://personality-project.org/r/psych/ bayesian estimation of single-test reliability coefficients. 57(4), 620–641. Revelle, W., & Condon, D. M. (2019). Reliability from α to ω: A tutorial. Multivariate Behavioural Research, 57(4). Psychological Assessment, 31(12), 1395–1411. https://doi.org/10.1080/00273171.2021.1891855 https://doi.org/10.1037/pas0000754 Pons, J., Viladrich, C., & Ramis, Y. (2017). Examining the big three of coping Revelle, W., & Zinbarg, R. E. (2009). Coefficient alpha, beta, omega, and the in adolescent athletes using network analysis. Revista de Psicologia Del GLB: Comment on Sitjsma. Psychometrika, 74(1), 145–154. Deporte, 26, 68–74. Rhemtulla, M., Brosseau-Liard, P. É., & Savalei, V. (2012). When can categorical Prinsen, C. A. C., Mokkink, L. B., Bouter, L. M., Alonso, J., Patrick, D. L., de variables be treated as continuous? A comparison of robust continuous Vet, H. C. W., & Terwee, C. B. (2018). COSMIN guideline for systematic and categorical SEM estimation methods under suboptimal conditions. reviews of patient-reported outcome measures. Quality of Life Research, Psychological Methods, 17(3), 354–373. https://doi.org/10.1037/a0029315 27(5), 1147–1157. https://doi.org/10.1007/s11136-018-1798-3 Rodriguez, A., Reise, S. P., & Haviland, M. G. (2016a). Applying bifactor R Core Team. (2021). R: A language and environment for statistical computing statistical indices in the evaluation of psychological measures. Journal of [Computer software]. R Foundation for Statistical Computing. Personality Assessment, 98(3), 223–237. https://www.r-project.org/ https://doi.org/10.1080/00223891.2015.1089249 Raykov, T. (1997a). Estimation of composite reliability for congeneric Rodriguez, A., Reise, S. P., & Haviland, M. G. (2016b). Evaluating bifactor measures. Applied Psychological Measurement, 21(2), 173–184. models: Calculating and interpreting statistical indices. Psychological https://doi.org/0803973233 Methods, 21(2), 137–150. https://doi.org/10.1037/met0000045 Raykov, T. (1997b). Scale reliability, Cronbach’s coefficient alpha, and Sánchez-Meca, J. (2022, July 20). Meta-análisis de generalización de la violations of essential tau-equivalence with fixed congeneric components. fiabilidad [Reliability generalization Meta-analysis][Simposium]. XVII Multivariate Behavioral Research, 32(4), 329–353. Congreso de Metodología de Las Ciencias Sociales y de La Salud. https://doi.org/10.1207/s15327906mbr3204_2 Sánchez-Meca, Julio, Marín-Martínez, F., López-López, J. A., Núñez- Raykov, T. (2001). Bias of coefficient alpha for fixed congeneric measures with Núñez, R. M., Rubio-Aparicio, M., López-García, J. J., López-Pina, J. A., correlated errors. Applied Psychological Measurement, 25(1), 69–76. Blázquez-Rincón, D. M., López-Ibáñez, C., & López-Nicolás, R. (2021). https://doi.org/10.1177/01466216010251005 Improving the reporting quality of reliability generalization meta-analyses: 19 Doval et al. / Psicothema (2023), 35(1), 05-20 The REGEMA checklist. Research Synthesis Methods, 12(4), 516–536. Viladrich, C., y Angulo-Brunet, A. (2019). Reliability of Essentially https://doi.org/10.1002/jrsm.1487 Unidimensional Measures Derived From Bifactor Modeling With R, Mplus Savalei, V., & Reise, S. P. (2019). Don’t forget the model in your model-based and Stata. [Data set and syntax]. Universitat Autònoma de Barcelona. reliability coefficients: A reply to McNeish (2018). Collabra: Psychology, https://ddd.uab.cat/record/205936 5(1), 36. https://doi.org/10.1525/collabra.247 Viladrich, C., Angulo-Brunet, A., & Doval, E. (2017). A journey around alpha Savalei, V., & Rhemtulla, M. (2013). The performance of robust test statistics with and omega to estimate internal consistency reliability. Annals of Psychology, categorical data. British Journal of Mathematical and Statistical Psychology, 33(3), 755–782. https://doi.org/10.6018/analesps.33.3.268401 66(2), 201–223. https://doi.org/10.1111/j.2044-8317.2012.02049.x Viladrich, C., Angulo-Brunet, A., & Doval, E. (2019). Mplus and stata tools to Scherer, R., & Teo, T. (2020). A tutorial on the meta-analytic structural equation calculate the internal consistency reliability coefficients alpha and omega modeling of reliability coefficients. Psychologycal Methods, 25(6), 747– [Data set and syntax]. Universitat Autònoma de Barcelona. 775. https://doi.org/10.1037/14262-002 https://ddd.uab.cat/record/205870 Schmitt, N. (1996). Uses and abuses of coefficient alpha. Psychological Weijters, B., Geuens, M., & Schillewaert, N. (2009). The proximity effect: The Assessment, 8(4), 350–353. role of inter-item distance on reverse-item bias. International Journal of Shi, D., Lee, T., Fairchild, A. J., & Maydeu-Olivares, A. (2020). Fitting Research in Marketing, 26(1), 2–12. ordinal factor analysis models with missing data: A comparison between https://doi.org/10.1016/j.ijresmar.2008.09.003 pairwise deletion and multiple imputation. Educational and Psychological Xiao, L., & Hau, K.T. (2022). Performance of coefficient alpha and its Measurement, 80(1), 41–66. https://doi.org/10.1177/0013164419845039 alternatives: Effects of different types of non-normality. Educational and Sideridis, G. D., Tsaousis, I., & Al-Sadaawi, A. (2019). An application of Psychological Measurement. Advance online publication. reliability estimation in longitudinal designs through modeling item-specific https://doi.org/10.1177/00131644221088240 error variance. Educational and Psychological Measurement, 79(6), 1038– Yang, Y., & Green, S. B. (2011). Coefficient alpha: A reliability coefficient for 1063. https://doi.org/10.1177/0013164419843162 the 21st Century? Journal of Psychoeducational Assessment, 29(4), 377– Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of 392. https://doi.org/10.1177/0734282911406668 Cronbach’s alpha. Psychometrika, 74(1), 107–120. Yang, Y., & Green, S. B. (2015). Evaluation of structural equation modeling https://doi.org/10.1007/s11336-008-9101-0 estimates of reliability for scales with ordered categorical items. Sijtsma, K., & Pfadt, J. M. (2021). Part II: On the use, the misuse, and the Methodology, 11(1), 23–34. https://doi.org/10.1027/1614-2241/a000087 very limited usefulness of Cronbach’s alpha: discussing lower bounds and Yang, Y., & Xia, Y. (2019). Categorical omega with small sample sizes via correlated errors. Psychometrika, 86, 843–860. bayesian estimation: An alternative to frequentist estimators. Educational https://doi.org/10.1007/s11336-021-09789-8 and Psychological Measurement, 79(1), 19–39. Slaney, K. L., Tkatchouk, M., Gabriel, S. M., & Maraun, M. D. (2009). https://doi.org/10.1177/0013164417752008 Psychometric assessment and reporting practices: Incongruence between Ziegler, M. (2020). Psychological test adaptation and development – How papers theory and practice. Journal of Psychoeducational Assessment, 27(6), 465– are structured and why. Psychological Test Adaptation and Development. 476. https://doi.org/10.1177/0734282909335781 Advance online publication. https://doi.org/10.1027/2698-1866/a000002 StataCorp. (2021). Stata statistical software. (Release 17) [Computer software]. Zinbarg, R. E., Revelle, W., Yovel, I., & Li, W. (2005). Cronbach’s α, Revelle’s StataCorp LLC. https://www.stata.com/ β, and Mcdonald’s ωH: their relations with each other and two alternative Streiner, D. L. (2003). Starting at the beginning: An introduction to coefficient conceptualizations of reliability. Psychometrika, 70(1), 123–133. alpha and internal consistency. Journal of Personality Assessment, 80(1), https://doi.org/10.1007/s11336-003-0974-7 99–103. Zumbo, B. D., Gadermann, A. M., & Zeisser, C. (2007). Ordinal versions of Taber, K. S. (2018). The use of Cronbach’s alpha when developing and reporting coefficients alpha and theta for Likert rating scales. Journal of Modern research instruments in science education. Research in Science Education, Applied Statistical Methods, 6(1), 21–29. 48(6), 1273–1296. https://doi.org/10.1007/s11165-016-9602-2 https://doi.org/10.1107/S0907444909031205 The jamovi project (2021). The jamovi project (v1.6) [Computer software]. Zumbo, B. D., & Kroc, E. (2019). A measurement is a choice and Stevens’ scales https://www.jamovi.org of measurement do not help make it: A response to Chalmers. Educational Thurstone, L. L. (1947). Multiple factor analysis. University of Chicago Press. and Psychological Measurement, 79(6), 1184–1197. https://doi.org/10.1177/0013164419844305 20