Open access Original research Method effects associated with negatively and positively worded items on the 12-­item General Health Questionnaire (GHQ-12): results from a cross-­sectional survey with a representative sample of Catalonian workers Maria F Rodrigo,1 J Gabriel Molina,1 Josep-­Maria Losilla,2 Jaume Vives,2 José M Tomás1 To cite: Rodrigo MF, Molina JG, Abstract Losilla J-­M, et al. Method Strengths and limitations of this study Objective Recent studies into the factorial structure of the effects associated with 12-­item version of the General Health Questionnaire (GHQ- negatively and positively ►► Sampling quality: a random and large representative 12) have shown that it was best represented by a single worded items on the 12-­item sample of workers and face-­to-­face administration substantive factor when method effects associated with General Health Questionnaire by professional interviewers. (GHQ-12): results from a negatively worded (NW) items are considered. The purpose ►► Comparison of confirmatory models for positively cross-­sectional survey with of the present study was to examine the presence of worded (PW) and/or negatively worded (NW) items a representative sample of method effects, and their relationships with demographic and the use of two different parameterisations. Catalonian workers. BMJ Open covariates, associated with positively worded (PW) and/or ►► There are no previous studies regarding the demo- 2019;9:e031859. doi:10.1136/ NW items. bmjopen-2019-031859 graphic correlates of wording effects on the 12-­item Design A cross-­sectional, observational study to compare version of the General Health Questionnaire. ►► Prepublication history for a comprehensive set of confirmatory factor models, ►► The different response scale used for the NW items this paper is available online. including method effects associated with PW and/or NW and the PW items in the questionnaire could be a To view these files, please visit items with GHQ-12 responses. confounding variable. the journal online (http://​dx.​doi.​ Setting Representative sample of all employees living in ►► The results might not be generalised to other specif- org/​10.​1136/​bmjopen-​2019-​ Catalonia (Spain). 031859). ic populations, for example, adolescents and elderly Participants 3050 participants (44.6% women) who retired people. responded the Second Catalonian Survey of Working Received 22 May 2019 Conditions. Revised 03 October 2019 Results A confirmatory factor analysis showed that the Accepted 29 October 2019 Health (GPH) in both community and non-­ best fitting model was a unidimensional model with two additional uncorrelated method factors associated with PW psychiatric clinical settings.2 The shortest and NW items. Furthermore, structural equation modelling 12-­item version (GHQ-12) is the most popular (SEM) revealed that method effects were differentially and has been employed on different settings © Author(s) (or their related to both the sex and age of the respondents. and in several countries, as well as part of employer(s)) 2019. Re-­use Conclusion Individual differences related to sex and multiple major national health, social well-­ permitted under CC BY-­NC. No commercial re-­use. See rights age can help to identify respondents who are prone to being and occupational surveys, achieving and permissions. Published by answering PW and NW items differently. Consequently, it is results which underline the fact that it is BMJ. desirable that both the constructs of interest as well as the highly reliable and valid.3–11 1 Methodology of behavioural effects of method factors are considered in SEM models as Despite its broad application, the factor sciencies, University of Valencia, a means of avoiding the drawing of inaccurate conclusions Valencia, Spain about the relationships between the substantive factors. structure underlying the responses to the 2 Psychobiology and GHQ-12 remains a controversial issue. In this methodology of health sciences, sense, although the GHQ-12 was originally Autonomous University of Introduction developed as a unidimensional scale, this Barcelona, Barcelona, Spain Originally developed by Goldberg,1 the one-­ factor latent structure has found little Correspondence to General Health Questionnaire (GHQ) empirical support and some alternative multi- Dr Jaume Vives; has been widely used as a screening instru- dimensional models have been proposed as ​Jaume.​Vives@​uab.​cat ment for measuring General Psychological more appropriate. Thus, the one with the most Rodrigo MF, et al. BMJ Open 2019;9:e031859. doi:10.1136/bmjopen-2019-031859 1 Open access from the statistical control of method biases, which has been mainly achieved through the correlated traits– correlated methods (CTCM) and the correlated traits– correlated uniquenesses (CTCU) confirmatory factor analysis models. Both procedures have been used in GHQ-12, to deal with method effects applying the CTCM model,30 44 the CTCU model29 31 39 40 or both CTCM and CTCU models.34–37 To date, we have not found any study about GHQ-12 that analyse the wording effects associated with either PW items alone, or with NW and PW items simulta- neously, comparing both CTCU and CTCM models. There are several multivariate statistical models for analysing method effect, and among them the CFA-­based approaches are the most popular ones,45 in particular the Figure 1 Competing models tested for the 12-­item General CFA with CTCM (CFA-­CTCM) and the CFA with CTCU Health Questionnaire. Underlined numbers identify negatively (CFA-­CTCU). On the one hand, the CTCM model speci- worded items. GPH: General Psychological Health; NW: method factor associated with negatively worded items; PW: fies that indicators’ variance can be explained by a linear method factor associated with positively worded items. combination of trait, method and error effects,46 with trait and method effects specified as latent variables. The CTCM model, when methods are specified independent empirical support is the three-­factor model proposed by (uncorrelated), directly translates into the well-­ known Graetz.5 12–22 It is important to note that the six positively bifactor model.47 48 On the other hand, the CTCU model worded (PW) items make up the first factor, whereas the specifies trait factors while method effects are modelled other two factors are made up of the six negatively worded correlating the uniqueness of items (indicators) sharing (NW) items (see figure 1, model 8). On the other hand, a common method.49 Both CTCM and CTCU models the bidimensional model, where the 6 NW and the 6 PW have strengths and shortcomings and therefore are items in the GHQ-12 are grouped into two factors, has usually employed simultaneously.50 This work extends also obtained wide support, especially in studies based on the previous work by Molina et al,36 which compares the exploratory factor analysis.5 10 23–28 The arguments against fit of the unidimensional model, the multifactor models these models and in favour of the unidimensional solu- and the CTCM and CTCU unidimensional models with tion are the high correlations between the factors13 and method effects for only the NW items. the low discriminant validity of the factor scores derived To clarify, figure 1 (models 1 to 9) shows the nine CFA from these models.16 29 30 models estimated to test the potential method effects As Hankins31 pointed out, multifactor models may just associated with either the PW or the NW or both. Model be the resulting artefact of the inclusion of PW and NW 1 is a one-­factor model of general health. This model items in the questionnaire, and so the controversy about also works as a baseline model against which to compare the factorial structure of the GHQ-12 might relate to the other more complex models. Models 2 and 3 are the effect of item wording on subjects’ response patterns CTCU and CTCM models that include method effects as part of a more general category called ‘method’.32 33 for the NW items. These were the best fitting models in Hankins31 found that, after modelling the wording effects Molina et al36 Models 4 and 5 are the CTCU and CTCM for the NW items, the unidimensional model fitted better models including method effects for the PW items. than both the two-­factor model (NW vs PW items) and Model 6 is the CTCM model including method factors Graetz’s three-­factor model. Other studies have called for both the NW and PW items (a CTCU model with into question the substantive meaning of the GHQ-12 method effects for both PW and NW items was not esti- multifactor solutions, suggesting that they might just be mated because it is not identified). Model 7 is a bifactor an artefact due to the wording effects associated with NW model with a general trait factor of general health and items.29 30 34–40 See Molina et al36 for a deeper review about two method factors associated to NW and PW items. The the dimensionality of GHQ-12. three factors are independent (uncorrelated). Addition- Some studies about other instruments, however, ally, considering the best fitting multidimensional model suggested considering the wording effects not only for in Tomás, Gutiérrez and Sancho51 based on the results the NW items but also for the PW items.41 42 Regarding by Graetz,12 models 8 and 9 were also tested. Model 8 GHQ-12, only a recent meta-­analysis modelled the pres- posited three substantive dimensions: social dysfunction, ence of method effects for NW and PW items concluding anxiety and depression and loss of confidence. Model 9 that positively keyed items explained incremental vari- included an additional method factor associated to NW ance beyond a general mental health factor.43 items. Models considering a method factor associated to Therefore, another source of variability in the results PW items made no sense as all PW items were indicators about the factor structure of the GHQ-12 could come of social dysfunction. 2 Rodrigo MF, et al. BMJ Open 2019;9:e031859. doi:10.1136/bmjopen-2019-031859 Open access As stressed by Marsh et al,52 it becomes necessary to official survey about the working conditions of employed consider this comprehensive set of competing models to Catalonian people commissioned by the Catalonian determine the relative importance and substantive nature Government Work Department. of the method effects. Results were published on the Catalonian Govern- Finally, there has been some research carried out on ment Work Department website60 and are available at the demographic correlates of method effects, such as https://​treball.​gencat.​cat/​ca/​ambits/​seguretat_​i_​salut_​ sex,53–57 age55 58 or educational level.41 59 With respect to laboral/​publicacions/​estadistiques_​estudis/​ci/​ii_​ecct/​ the GHQ-12, to date, we have not found any studies that treballadors/ analyse demographic correlates of method effects. Building on the previous studies, the first aim of this Measures study was to overcome the limitation pointed out in The GHQ-12 is a self-­report scale that contains 6 PW Molina et al36 and examine method effects associated with items (eg, ‘Have you been able to face up to problems?’) both PW and NW items. The second aim was to further and 6 NW items (eg, ‘Have you been losing confidence in understand the meaning of the method factors; there- yourself?’). The GHQ-12 was validated in Spain by Lobo fore, we evaluated the relationships between the method and Muñoz.61 Table 2 shows the statements of these items factors and three covariates (ie, sex, age, and educational in the same order as they were presented in the survey. It level) in the framework of a structural equation model- must be noted that the GHQ-12 has a different response ling (SEM). scale for the PW items (ie, more than usual; same as usual; less than usual and much less than usual) and the NW items (ie, not at all; no more than usual; rather more than Method usual and much more than usual). Accordingly, the four-­ Participants point scoring scheme was applied in our study, and so the The data used in this study came from the Second Cata- total scores in the GHQ-12 ranged from 0 to a maximum lonian Survey of Working Conditions60 and were based of 36, with higher scores indicating lower levels of GPH. on a representative random sample of all employees For the purposes of exploring the correlates of method living in Catalonia (Spain). Data were collected between effects (ie, item wording effects), we used the following September and November 2010 by professional inter- three covariates: (a) sex (0=men and 1=women); (b) age viewers in private households. The sample comprised a and (c) educational level, which was measured as a self-­ total of 3050 participants who responded to the GHQ-12 reported question with seven response graduated catego- included in the survey. Main sociodemographic charac- ries ranging from incomplete primary studies to postgraduate teristics of the sample are shown in table 1. studies. The educational level was scored as the highest level of education reached. Public involvement Respondents were not involved in any stage of the design Statistical analysis of the study and were only requested to respond the A set of competing confirmatory factor models were survey. In the selected households, interviewers identi- estimated using MPlus V.8.3.62 Figure 1 shows the spec- fied themselves personally and informed that this was an ification of all these CFA models. The goodness-­ of-­ fit indices computed were the χ2 statistic; the Comparative Fit Index (CFI); the Root Mean Square Error of Approx- Table 1 Main sociodemographic characteristics imation (RMSEA) with its 90% CI and the Standardised Mean (SD) n (%) Range Root Mean Square Residual (SRMR). Values greater than Gender 0.95 for CFI, and lower than 0.06 and 0.08 for RMSEA  Women 1361 (44.6) and SRMR, respectively, are considered to indicate good Age 40.46 (11.19) 17–82 model fit. As concerns the estimation of CFA models, most studies Education into the GHQ-12 factor structure have used maximum  Incomplete 90 (3.0) likelihood.16 31 35 40 44 This estimation method relies on primary studies several assumptions which should be met to be confident  Primary studies 541 (17.9) about the results obtained. This is the case of the assump-  Secondary 637 (21.0) tion of multivariate normality which implies, first, that studies: first the variables are continuous in nature and, second, that stage the joint distribution of the variables is normal. The first  Associate degree 763 (25.2) condition is unlikely to be met with the GHQ-12 Likert-­  High school 598 (19.8 type response data; nor is the second if the variables depart markedly from normality as is the case for the responses  Graduate studies 359 (11.9) to the NW items which were heavily positively skewed  Postgraduate 39 (1.3) (see figure 2). An alternative when these conditions studies are not met is to use the weighted least squares (WLS) Rodrigo MF, et al. BMJ Open 2019;9:e031859. doi:10.1136/bmjopen-2019-031859 3 Open access Table 2 Descriptive statistics, standardised factor loadings from model 7 and correlations between the model 7 factors and the covariates Model 7 Item Mean SD GPH PW NW Item 1. Able to concentrate 1.03 0.37 0.42* 0.49* Item 2. Lost sleep over worry 0.57 0.75 0.78* 0.07 Item 3. Playing a useful part in things 0.96 0.31 0.09* 0.59* Item 4. Capable of making decisions 0.96 0.30 0.14* 0.70* Item 5. Constantly under strain 0.71 0.79 0.83* 0.03 Item 6. Could not overcome difficulties 0.44 0.66 0.76* 0.25* Item 7. Enjoy day-­to-­day activities 1.01 0.40 0.53* 0.55* Item 8. Face up to problems 0.99 0.32 0.39* 0.60* Item 9. Feeling unhappy and depressed 0.37 0.66 0.78* 0.38* Item 10. Losing confidence in yourself 0.19 0.48 0.53* 0.70* Item 11. Thinking of yourself as a worthless person 0.12 0.40 0.48* 0.72* Item 12. Feeling reasonably happy 0.99 0.38 0.44* 0.72*  Relation between the model 7 factors and the socio-­demographic variables Sex 0.13* −0.08* −0.02 Age 0.11* 0.08* 0.01 Educational level 0.00 −0.02 −0.06 *P< 0.05. GPH, General Health Psychology;NW, negative wording factor; PW, positive wording factor. Figure 2 Bar charts of the response distributions for the 12-­item General Health Questionnaire. Responses were given on a different four-­point response scale for the positively worded items (0=better than usual, 1=same as usual, 2=less than usual, 3=much less than usual) and for the negatively worded items (0=not at all, 1=no more than usual, 2=more than usual, 3=much more than usual). 4 Rodrigo MF, et al. BMJ Open 2019;9:e031859. doi:10.1136/bmjopen-2019-031859 Open access items 2 and 5 in the method factor comprising the NW Table 3 Fit indexes for the alternative models of the 12-­ item General Health Questionnaire items. Finally, a statistical analysis of the relationships between Models df χ2 CFI RMSEA (90% CI) SRMR the latent factors in model 7 and the three covariates Model 1 54 5378.68 0.77 0.180 (0.176 to 0.119 considered in this study (ie, sex, age and educational 0.184) level) was performed through a Multiple Indicator Model 2 39 928.099 0.96 0.086 (0.082 to 0.049 Multiple Causes (MIMIC) SEM model in which the 0.091) effects between the three latent factors in model 7 and Model 3 48 1345.38 0.95 0.094 (0.090 to 0.061 the three covariates were freely estimated, the focus being 0.059) on the relationships between the method factors and the Model 4 39 934.690 0.96 0.087 (0.083 to 0.052 covariates. The model fit was excellent (RMSEA=0.040; 0.092) RMSEA 90% Confidence interval (CI) = (0.037, 0.049); Model 5 48 1275.28 0.95 0.092 (0.087 to 0.058 CFI=0.99; SRMR=0.029). As can be seen in table 2, the 0.096) relations of age with the method factors were near to 0 Model 6 41 497.520 0.98 0.060 (0.056 to 0.030 and statistically non-­significant for NW items, and posi- 0.065) tive and significant although small with PW items (0.08). Model 7 42 507.741 0.98 0.060 (0.056 to 0.030 Sex was significantly related to the method factor asso- 0.065) ciated with PW items (–0.08), whereas the educational Model 8 51 1142.88 0.95 0.084 (0.080 to 0.054 level was not significantly related to method factors. Thus, 0.088) men and women differ in the way they answer PW items, Model 9 45 960.388 0.96 0.082 (0.078 to 0.049 meaning that men are slightly more likely than women to 0.086) endorse PW items, and method effects associated with PW Models are specified in figure 1. items also increased by age. CFI, Comparative Fit Index; RMSEA, Root Mean Square Error of Approximation; SRMR, Standardised Root Mean Square Residual. Discussion This study focused on the examination of the latent struc- ture underlying the responses to the GHQ-12, considering estimator,63 which has already been used in some studies the role of method effects associated with both, PW and about the GHQ-12 factor structure13 18 20 29 and it will be NW items, and using two alternative parameterisations of the estimation method used here. Thus, the various CFA the CFA measurement models. What should first be noted models were estimated using diagonally WLS. is that the studies that have included method effects in Finally, correlates of the GHQ-12 factors were evaluated the measurement model of the GHQ-12 have been more using SEM through the inclusion in the finally selected the exception than the rule in previous research into the model of the three covariates considered in this study: sex factor structure of this questionnaire. was treated as categorical, whereas age and educational According to the results of the present study, we conclude level were treated as continuous variables. that the GHQ-12 factor structure is best characterised by introducing latent method factors that capture both the method effects associated with NW and PW items (model 7). These results support the conclusion from previous Results research that the good fit obtained by multidimensional The goodness-­of-­fit statistics and indices obtained for the models (mainly the two-­factor model and the three-­factor nine models compared here are shown in table 3. Graetz’s model) could simply be explained by the artificial Model 1, with a single factor of general health, and grouping of PW and NW items. However, the interpreta- model 8, with three substantive factors, had worse fit tion of the latent (method) factors as purely integrating than the models that include wording effects. That is, a method bias due to wording is not straightforward. It is careful look at fit indexes makes clear that the inclusion obvious that NW and PW items share the wording. It is of method effects always improves model fit. Indeed, both also clear that this three bifactor model (one trait and NW and PW method effects are needed to get the best two method factors) fitted the data best. And finally, there fitting models. These best fitting models were models is a lot of empirical evidence on these wording effects. 6 and 7. Their fit was practically indistinguishable and, However, it is also relevant to discuss the large loadings of given that they only differ in that model 7 is more parsi- many items on the method factors, being these loadings monious because constrains method factors correlation sometimes larger than their loadings in the trait factor. to zero, it will be retained as the best representation of The general factor explains a 52% of the shared variance, the observed data. but there are some items that deserve careful attention. An in-­depth inspection of the parameter estimates in For example, items 3 (‘playing useful part in things’) and model 7 (see table 2) showed that all factor loadings were 4 (‘capable of making decisions’) had very low loadings statistically significant for the three factors, except for on the trait factor. If we understand PW method factor as Rodrigo MF, et al. BMJ Open 2019;9:e031859. doi:10.1136/bmjopen-2019-031859 5 Open access the only method bias, then it follows that these two items biases have on individual measures and on the covaria- are purely method effects, but surely they must share tion between different constructs). Thus, it is desirable some trait variance. In the same vein, items 10 (‘losing that both the constructs of interest as well as the effects confidence in yourself’) and 11 (‘thinking of yourself as of method factors, like PW and NW, are considered in a worthless person’) load very high in the NW method SEM models as a means of controlling these systematic factor and, as a reviewer pointed out, a likely (post-­hoc) sources of bias, and thus avoiding the drawing of inac- explanation is that wording bias are still confounded with curate conclusions about the relationship between the a confidence/self-­image factor. Therefore, the interpre- substantive factors. tation of these effects as purely method and, accordingly, Previous research on the GHQ-1231 36 has outlined the the interpretation of an overall score for the scale diffi- asymmetry in the participants’ responses as a function of cult may be compromised. the wording of the items, as well as the different responses The second aim of this study was to examine the rela- scales for the PW and NW items. This asymmetry in the tionship between the method factors associated with both participants’ responses as a function of the wording of NW and PW items and three demographic variables, the items is consistent with results from previous research namely sex, age and educational level of the respon- into wording effects for contrastive survey questions.71 dents. Regarding the sex, we found a statistically signifi- The extent to which the presence of method effects is cant, but weak, relationship between PW and sex, so that linked to the asymmetric pattern of responses and/or to men were more likely than women to endorse PW items. the different response scales for the PW and NW items in These results are in line with previous works that, in the the GHQ-12 should be examined in future research. context of RSES, have found sex differences in wording Comparing the current work with previous studies effects.56 57 As for the explanatory role of age on method into the factorial structure of the GHQ-12, to our knowl- effects, we found that the relationship between age and edge, this is the first study that tests a comprehensive set the NW effect was not statistically significant, which of models including method effects associated with both supports previous research using other questionnaires PW and NW items and also explores some demographic (eg, self-­esteem scales,50 Hospital Anxiety & Depression correlates of these method effects. Another strength of Scale64). Moreover, our results give support to previous this work was the fact that it used a large representative studies which had stated that, in older adults, the stron- sample of workers, but the results might not be gener- gest method effects would be associated with PW items, alised to other specific populations, for example, adoles- rather than NW items.55 58 cents and elderly retired people. As to the educational level, we found that there was not a significant correlation of this variable on the two Twitter Josep-­Maria Losilla @jmlosilla and Jaume Vives @VivesJ_Research method factors. This result supports and extends the Contributors All authors meet the criteria recommended by the International evidence obtained in Tomás et al50 who found that the Committee of Medical Journal Editors (ICMJE). All authors made substantial contributions to conception and design, acquisition of data or analysis and educational level of the respondents had no effect on the interpretation of data. MFR and JGM: drafted the article. JV and JML: critically negative method factor using self-­esteem questionnaires. revised the draft for important intellectual content. JMT: worked in the statistical This results contradicts previous research on the relation- analysis and interpretation of data. All authors agreed on the final version. ship of the NW factor and the educational level/verbal Funding This work was supported by the Grant PGC2018-100675-­B-­I00, Spanish ability with different questionnaires and samples.41 64–69 Ministry of Science, Innovation and Universities (Spain). The funders had no role in Overall, the significant effects of sex and age on trait the study design, data collection and analysis, decision to publish or preparation of the manuscript. and method factors point out that women have a worse well-­being, but this effect is partly modified by a method Disclaimer All authors have agreed to authorship in the indicated order. All authors declare that this paper is an original unpublished work and it is not being submitted effect on the PW items, whereas the results for age suggest elsewhere. All authors do not have any financial interests that might be interpreted that older respondents have worse well-­being and this as influencing the research, and APA ethical standard were followed in the conduct effect is magnified by a method effect on the PW factor. of the study. The results on the individual differences related to the Competing interests None declared. demographic variables considered in this study cannot Patient consent for publication Not required. only help to understand the presence of wording method Ethics approval The research was not submitted to approval by an institutional effects but also to identify respondents who are prone to review board since this is not a requirement at our universities for this type of answering PW and NW items differently. In this sense, the study. Ethics approval was not sought for this study since this was a secondary relationship that appears as more evident is for the age analysis of anonymised data. and sex variables. Provenance and peer review Not commissioned; externally peer reviewed. Another practical consequence of our study concerns Data availability statement Data are available upon reasonable request. the relationship between the intended measure of the Open access This is an open access article distributed in accordance with GHQ-12 (ie, the GPH factor) and other constructs of the Creative Commons Attribution Non Commercial (CC BY-­NC 4.0) license, interest. Several studies have shown that method effects which permits others to distribute, remix, adapt, build upon this work non-­ commercially, and license their derivative works on different terms, provided the can inflate, deflate or have no effect at all on estimates original work is properly cited, appropriate credit is given, any changes made of the relationship between two constructs (see Podsa- indicated, and the use is non-­commercial. See: http://​creativecommons.​org/​ koff et al70 for a further review of the effects that method licenses/​by-​nc/​4.​0/. 6 Rodrigo MF, et al. BMJ Open 2019;9:e031859. doi:10.1136/bmjopen-2019-031859 Open access References the scales in a Turkish primary care sample. Soc Psychiatry Psychiatr 1 Goldberg DP. The detection of psychiatric illness by questionnaire. Epidemiol 1997;32:327-31. London: Oxford University Press, 1972. 27 Picardi A, Abeni D, Pasquini P. Assessing psychological distress in 2 Goldberg DP, Williams P. A user’s guide to the General Health patients with skin diseases: reliability, validity and factor structure of Questionnaire. Windsor, United Kingdom: NFER-­Nelson, 1988. the GHQ-12. J Eur Acad Dermatol Venerol 2001;15:410–7. 3 Bhui K, Bhugra D, Goldberg D. Cross-­cultural validity of the Amritsar 28 Werneke U, Goldberg DP, Yalcin I, et al. The stability of the factor depression inventory and the general health questionnaire amongst structure of the general health questionnaire. Psychol Med English and Punjabi primary care attenders. Soc Psychiatry Psychiatr 2000;30:823–9. Epidemiol 2000;35:248–54. 29 Aguado J, Campbell A, Ascaso C, et al. Examining the factor 4 Daradkeh TK, Ghubash R, el-­Rufaie OE. Reliability, validity, and structure and discriminant validity of the 12-­item general health factor structure of the Arabic version of the 12-­Item general health questionnaire (GHQ-12) among Spanish postpartum women. questionnaire. Psychol Rep 2001;89:85–94. Assessment 2012;19:517–25. 5 Gelaye B, Tadesse MG, Lohsoonthorn V, et al. Psychometric 30 Ye S. Factor structure of the general health questionnaire (GHQ-12): properties and factor structure of the general health questionnaire as the role of wording effects. Pers Individ Dif 2009;46:197–201. a screening tool for anxiety and depressive symptoms in a multi-­ 31 Hankins M. The factor structure of the twelve item general health national study of young adults. J Affect Disord 2015;187:197–202. questionnaire (GHQ-12): the result of negative phrasing? Clin Pract 6 Goldberg DP, Gater R, Sartorius N, et al. The validity of two versions Epidemiol Ment Health 2008;4. of the GHQ in the WHO study of mental illness in general health care. 32 Harvey RJ, Billings RS, Nilan KJ. Confirmatory factor analysis of the Psychol Med 1997;27:191–7. job diagnostic survey: good news and bad news. J Appl Psychol 7 Lundin A, Hallgren M, Theobald H, et al. Validity of the 12-­item 1985;70:461–8. version of the general health questionnaire in detecting depression in 33 Smith N, Stults DM. Factors defined by negatively keyed items: the the general population. Public Health 2016;136:66–74. results of careless respondents? Appl Psychol Meas 1985;9:367–73. 8 Rocha K, Pérez K, Rodríguez-­Sanz M, et al. Propiedades 34 Abubakar A, Fischer R. The factor structure of the 12-­Item general psicométricas y valores normativos del General Health Questionnaire health questionnaire in a literate Kenyan population. Stress Health (GHQ-12) en población general española. Int J Clin Heal Psychol 2012;28:248–54. 2011;11:125–39. 35 Fernandes HM, Vasconcelos-­Raposo J. Factorial validity and 9 Sánchez-­López MdelP, Dresch V. The 12-­item general health invariance of the GHQ-12 among clinical and nonclinical samples. questionnaire (GHQ-12): reliability, external validity and factor Assessment 2013;20:219–29. structure in the Spanish population. Psicothema 2008;20:839–43. 36 Molina JG, Rodrigo MF, Losilla J-­M, et al. Wording effects and the 10 Schmitz N, Kruse J, Tress W. Psychometric properties of the general factor structure of the 12-­Item general health questionnaire (GHQ- health questionnaire (GHQ-12) in a German primary care sample. 12). Psychol Assess 2014;26:1031–7. Acta Psychiatr Scand 1999;100:462–8. 37 Motamed N, Edalatian Zakeri S, Rabiee B, et al. The factor structure 11 Tait RJ, French DJ, Hulse GK. Validity and psychometric properties of of the twelve items general health questionnaire (GHQ-12): a the general health questionnaire-12 in young Australian adolescents. population based study. Appl Res Qual Life 2018;13:303–16. Aust N Z J Psychiatry 2003;37:374–81. 38 Rey JJ, Abad FJ, Barrada JR, et al. The impact of ambiguous 12 Graetz B. Multidimensional properties of the general health response categories on the factor structure of the GHQ-12. Psychol questionnaire. Soc Psychiatry Psychiatr Epidemiol 1991;26:132–8. Assess 2014;26:1021–30. 13 Campbell A, Knowles S. A confirmatory factor analysis of the GHQ12 39 Romppel M, Braehler E, Roth M, et al. What is the general health using a large Australian sample. European Journal of Psychological questionnaire-12 assessing? Compr Psychiatry 2013;54:406–13. Assessment, 2007: 2–8. 40 Smith AB, Oluboyede Y, West R, et al. The factor structure of the 14 Cheung YB. A confirmatory factor analysis of the 12-­item general GHQ-12: the interaction between item phrasing, variance and levels health questionnaire among older people. Int J Geriatr Psychiatry of distress. Qual Life Res 2013;22:145–52. 2002;17:739–44. 41 Marsh HW. Negative item bias in ratings scales for preadolescent 15 French DJ, Tait RJ. Measurement invariance in the general health children: a cognitive-­developmental phenomenon. Dev Psychol questionnaire-12 in young Australian adolescents. Eur Child Adolesc 1986;22:37–49. Psychiatry 2004;13:1–7. 42 Tomás JM, Oliver A. Rosenberg's self‐esteem scale: two factors or 16 Gao F, Luo N, Thumboo J, et al. Does the 12-­item general health method effects. Struct Equ Model A Multidiscip J 1999;6:84–98. questionnaire contain multiple factors and do we need them? Health 43 Gnambs T, Staufenbiel T. The structure of the general health Qual Life Outcomes 2004;2:63–7. questionnaire (GHQ-12): two meta-­analytic factor analyses. Health 17 Mäkikangas A, Feldt T, Kinnunen U, et al. The factor structure and Psychol Rev 2018;12:179–94. factorial invariance of the 12-­item general health questionnaire 44 Wang L, Lin W. Wording effects and the dimensionality of the general (GHQ-12) across time: evidence from two community-­based health questionnaire (GHQ-12). Pers Individ Dif 2011;50:1056–61. samples. Psychol Assess 2006;18:444–51. 45 Wothke W. Models for multitrait-­multimethod matrix analysis. In: 18 Padrón A, Galán I, Durbán M, et al. Confirmatory factor analysis of Marcoulides GA, Schumacker RE, eds. Advanced structural equation the general health questionnaire (GHQ-12) in Spanish adolescents. modeling: issues and techniques. Mahwah, NJ: Lawrence Erlbaum Qual Life Res 2012;21:1291–8. Associates, 1996. 19 Penninkilampi-­Kerola V, Miettunen J, Ebeling H. A comparative 46 Jöreskog KG. Analyzing psychological data by structural analysis assessment of the factor structures and psychometric properties of covariance matrices. In: Atkinson RC, Krantz DH, Luce RD, et al, of the GHQ-12 and the GHQ-20 based on data from a Finnish eds. Contemporary developments in mathematical psychology. San population-­based sample. Scand J Psychol 2006;47:431–40. Francisco: Freeman, 1974: 2. 1–56. 20 Shevlin M, Adamson G. Alternative factor models and factorial 47 Markon KE. Bifactor and hierarchical models: specification, invariance of the GHQ-12: a large sample analysis using confirmatory inference, and interpretation. Annu Rev Clin Psychol 2019;15:51–69. factor analysis. Psychol Assess 2005;17:231–6. 48 Reise SP. Invited paper: the rediscovery of Bifactor measurement 21 Martin CR, Newell RJ. The factor structure of the 12-­item general models. Multivariate Behav Res 2012;47:667–96. health questionnaire in individuals with facial disfigurement. J 49 Marsh HW, Bailey M. Confirmatory factor analyses of multitrait-­ Psychosom Res 2005;59:193–9. multimethod data: a comparison of alternative models. Appl Psychol 22 Tomás JM, Meléndez JC, Oliver A, et al. Efectos de método en las Meas 1991;15:47–70. escalas de Ryff: un estudio en población de personas mayores. 50 Tomás JM, Oliver A, Galiana L, et al. Explaining method effects Psicológica 2010;31:383–400. associated with negatively worded items in trait and state global and 23 Andrich D, van Schoubroeck L. The General Health Questionnaire: domain-­specific self-­esteem scales. Struct Equ Model A Multidiscip a psychometric analysis using latent trait theory. Psychol Med J 2013;20:299–313. 1989;19:469–85. 51 Tomás JM, Gutiérrez M, Sancho P. Factorial validity of the general 24 Gao W, Stark D, Bennett MI, et al. Using the 12-­item general health health questionnaire 12 in an Angolan sample. Eur J Psychol Assess questionnaire to screen psychological distress from survivorship to 2017;33:116–22. end-­of-­life care: dimensionality and item quality. Psychooncology 52 Marsh HW, Scalas LF, Nagengast B. Longitudinal tests of competing 2012;21:954–61. factor structures for the Rosenberg self-­esteem scale: traits, 25 Glozah FN, Pevalin DJ. Factor structure and psychometric properties ephemeral artifacts, and stable response styles. Psychol Assess of the general health questionnaire (GHQ-12) among Ghanaian 2010;22:366–81. adolescents. J Child Adolesc Ment Health 2015;27:53–7. 53 DiStefano C, Motl RW. Self-­esteem and method effects associated 26 Kilic C, Rezaki M, Rezaki B, et al. General health questionnaire with negatively worded items: investigating factorial invariance by (GHQ-12 & GHQ-28): psychometric properties and factor structure of sex. Struct Equ Model A Multidiscip J 2009;16:134–46. Rodrigo MF, et al. BMJ Open 2019;9:e031859. doi:10.1136/bmjopen-2019-031859 7 Open access 54 Gana K, Saada Y, Bailly N, et al. Longitudinal factorial invariance of 63 Browne MW. Asymptotically distribution-­free methods for the Rosenberg self-­esteem scale: determining the nature of method the analysis of covariance structures. Br J Math Stat Psychol effects due to item wording. J Res Pers 2013;47:406–16. 1984;37:62–83. 55 Lindwall M, Barkoukis V, Grano C, et al. Method effects: the problem 64 Wouters E, Booysen FleR, Ponnet K, et al. Wording effects and the with negatively versus positively keyed items. J Pers Assess factor structure of the hospital anxiety & depression scale in HIV/ 2012;94:196–204. AIDS patients on antiretroviral treatment in South Africa. PLoS One 56 Michaelides MP, Zenger M, Koutsogiorgi C, et al. Personality 2012;7:e34881–7. correlates and gender invariance of wording effects in the German 65 Bors DA, Vigneau F, Lalande F. Measuring the need for cognition: version of the Rosenberg self-­esteem scale. Pers Individ Dif item polarity, dimensionality, and the relation with ability. Pers Individ 2016;97:13–18. Dif 2006;40:819–28. 57 Urbán R, Szigeti R, Kökönyei G, et al. Global self-­esteem and method 66 Chen YH, Rendina-­Gobioff G, Dedrick RF. Factorial invariance of effects: competing factor structures, longitudinal invariance, and response a Chinese self-­esteem scale for third and sixth grade students: styles in adolescents. Behav Res Methods 2014;46:488–98. evaluating method effects associated with the use of positively and 58 Mullen SP, Gothe NP, McAuley E. Evaluation of the factor structure negatively worded items. Int J Educ Psychol Assess 2010;6:21–35. of the Rosenberg self-­esteem scale in older adults. Pers Individ Dif 67 Corwyn RF. The factor structure of global self-­esteem among 2013;54:153–7. adolescents and adults. J Res Pers 2000;34:357–79. 59 Marsh HW. Positive and negative global self-­esteem: a substantively 68 Rammstedt B, Goldberg LR, Borg I. The measurement equivalence meaningful distinction or artifactors? J Pers Soc Psychol 1996;70:810–9. of big-­five factor markers for persons with different levels of 60 Catalonian Labor Relations and Quality of Work Department. education. J Res Pers 2010;44:53–61. Segunda Encuesta Catalana de Condiciones de Trabajo [Second 69 Schmitt DP, Allik J. Simultaneous administration of the Rosenberg Catalonian Survey of Working Conditions. Author: Barcelona:, self-­esteem scale in 53 nations: exploring the universal and 2012. culture-­specific features of global self-­esteem. J Pers Soc Psychol 61 Lobo A, Muñoz PE. Versiones en lengua española validadas. In: 2005;89:623–42. Goldberg D, Williams P, eds. Cuestionario de Salud General GHQ 70 Podsakoff PM, MacKenzie SB, Podsakoff NP. Sources of method (General Health Questionnaire). Guia para el usuario de las distintas bias in social science research and recommendations on how to versiones [Guide for the use of the different validated versions in control it. Annu Rev Psychol 2012;63:539–69. Spanish language. Barcelona, Spain: Editorial Masson, 1996. 71 Kamoen N, Holleman B, Mak P, et al. Agree or disagree? cognitive 62 Muthén LK, Muthén BO. Mplus User’s Guide. 6th ed. Los Angeles, processes in answering contrastive survey questions. Discourse CA: Muthén & Muthén, 2011. Process 2011;48:355–85. 8 Rodrigo MF, et al. BMJ Open 2019;9:e031859. doi:10.1136/bmjopen-2019-031859