Current Psychology https://doi.org/10.1007/s12144-023-05180-1 Dimensionality and reliability of the online version of the Patient Health Questionnaire‑4 (PHQ‑4) in a large Colombian sample: Results from the PSY‑COVID study Juan P. Sanabria‑Mazo1,2,3 · Andrés Gómez‑Acosta4 · John Alexander Castro‑Muñoz5 · Yonatan Ferney Rojas5 · Albert Feliu‑Soler3,6 · Juan V. Luciano2,3,6 · Antoni Sanz1 Accepted: 28 August 2023 © The Author(s) 2023 Abstract The Patient Health Questionnaire-4 (PHQ-4) is an ultra-brief screening instrument to measure depressive and anxiety symptoms. This study evaluated the dimensionality and reliability of the online version of the PHQ-4 in a large sample of the general population in Colombia. Data were collected during the first phase of lockdown measures occasioned by the COVID-19 pandemic. A total of 18,061 adult participants completed the online version of the PHQ-4. The characteristics of the items and subscales were explored. Dimensionality was examined using confirmatory factor analysis (CFA), including an examination of invariance (configural, metric, and scalar) across socio-demographic characteristics. Reliability indices were computed and known-groups validity was addressed by estimating associations between PHQ-4 scores and socio- demographic characteristics. The CFA showed significantly adequate fit indices for the expected two-factor structure, being invariant across gender, age, income level, education level, and region. Internal consistency was satisfactory for the PHQ-2 (α = .83), the GAD-2 (α = .79), and the PHQ-4 (α = .86). Higher scores on depressive (PHQ-2), anxiety (GAD-2), and psy- chological distress (PHQ-4) symptoms in females and young people, and those respondents with lower income, unemployed, and lower level of education were observed. The findings indicate that the PHQ-4 is a reliable scale for depressive and anxiety symptoms among adult Colombian people, being recommendable this tool for online surveys. Keywords Patient Health Questionnaire-4 · Anxiety · Depression · COVID-19 pandemic · Psychometric properties Introduction Anxiety and depression are the most prevalent emotional Juan V. Luciano and Antoni Sanz share senior authorship. disturbances (Wittchen et al., 2011). The high comorbidity * Juan P. Sanabria‑Mazo between them, close to 50% of cases (Kessler et al., 2015), juanpablo.sanabria@uab.cat as well as the impact of both mood disorders on the func- 1 tioning and quality of life of affected individuals are priority Department of Basic, Developmental, and Educational Psychology, Autonomous University of Barcelona, problems for public health systems (Kroenke et al., 2009). Cerdanyola del Vallès, Spain The prevalence of anxiety disorders increased to 11% from 2 Teaching, Research, & Innovation Unit, Parc Sanitari Sant 1990 to 2010, growing from 200 to 272 million reported Joan de Déu, St. Boi de Llobregat, Spain cases worldwide (Baxter et al., 2014). A meta-analysis that 3 Centre for Biomedical Research in Epidemiology and Public examined 68 studies conducted in 30 countries, between Health (CIBERESP), Madrid, Spain 1994 and 2014, reported a prevalence of depression of 4 Department of Psychology, Universidad de Pamplona, around 13% in the general population (Lim et al., 2018). Pamplona, Colombia According to World Health Organization (2011) population 5 Department of Psychology, Fundación Universitaria Sanitas, reports estimate that by 2030 emotional disorders will be Bogotá, Colombia the most disabling mental health conditions worldwide. The 6 Department of Clinical and Health Psychology, Autonomous COVID-19 pandemic has increased the prevalence of mood University of Barcelona, Cerdanyola del Vallés, Spain disorders in the clinical and non-clinical population (Luo 13 Vol.:(0123456789) Current Psychology et al., 2020; Xiong et al., 2020), pointing to the relevance of the PHQ-9 using confirmatory factor analysis (CFA). The reliable screening tools for the assessment and monitoring bifactor model yielded an excellent fit to the data, being of symptoms severity. superior to that obtained in the one- and two-factor models. In recent years, it has been highlighted the importance Two types of latent factors are defined in bifactor models. of brief, rapid, and reliable screening tools to facilitate The first is a general factor in which all items are allowed to diagnosis of mood disorders in healthcare settings (Olariu load (i.e., PHQ-4) and the second is composed of specific et al., 2015). Different authors have proposed the use of brief factors in which the items are distributed by their content screening tools to reduce misdiagnosis (Castro-Rodríguez (i.e., PHQ-2 and GAD-2). In the case of the PHQ-4 –and et al., 2015), to optimize health system resources (Cano- following the Clark and Watson’s tripartite model (1991)–, Vindel et al., 2018), and to improve clinical outcomes (Gold- the general factor reflects the shared component of depres- berg et al., 2017). Specifically, the Patient Health Question- sion and anxiety (i.e., psychological distress; Drapeau et al., naire (PHQ-4) is one of the most widely used an ultra-brief 2012), whereas the specific factors (depression and anxiety screening instruments to measure depressive and anxiety after controlling for the general negative affect factor) rep- symptoms (Kroenke et al., 2009). This ultra-brief self-report resent low positive affect (for depression; Kroenke et al., instrument combines two items of the PHQ-9 (Kroenke 2009) and hyperarousal (for anxiety; Kroenke et al., 2009). et al., 2001) and two items of the Generalized Anxiety For all these reasons, it seems relevant to study the psy- Disorder (GAD-7; Spitzer et al., 2006). The psychometric chometric properties of the PHQ-4. Particularly, this study properties of the PHQ-4 has been explored in clinical (Cano- evaluated the dimensionality and reliability of the online Vindel et al., 2018; Ghaheri et al., 2020; Kroenke et al., version of the PHQ-4 in a large sample of the general popu- 2009; Renovanz et al., 2019; Weihs et al., 2018) and non- lation in Colombia collected during the first phase of lock- clinical samples (Fong et al., 2023; Kazlauskas et al., 2023; down measures occasioned by the COVID-19 pandemic Khubchandani et al., 2016; Kocalevent et al., 2014; Lari- (i.e., May to June 20, 2020). The four objectives and hypoth- onow & Mudło, 2023; Löwe et al., 2010; Meidl et al., 2023; eses explored in this research are presented below: Mendoza et al., 2022; Mills et al., 2015) in several countries (i.e., Colombia, Denmark, Ecuador, Germany, Greek, Iran, 1. Firstly, to examine the goodness-of-fit of the one-, Korea, Philippines, Poland, Spain, and United States, among two-, and bifactor model of the PHQ-4. In line with others), but mainly in paper-and-pencil format. the evidence reported in previous studies (Cano-Vindel Online screening tools are reliable for the detection of et al., 2018; Fong et al., 2023; Kocalevent et al., 2014; mood disorders (Muñoz-Navarro et al., 2017a, b). These Kroenke et al., 2009; Löwe et al., 2010; Meidl et al., questionnaires facilitate data collection and help to avoid 2023; Mendoza et al., 2022), it was speculated that the limitations of data loss in the classic paper-and-pencil for- two-factor correlated model would have a significantly mat or response bias in face-to-face interviews. The evalu- better fit to the data than the other models (hypothesis ation of the psychometric properties of an online version 1). is necessary, even if the paper version has been explored 2. Secondly, to test invariance (configural, metric, and sca- (Campbell et al., 2015; Coons et al., 2009; Mendoza et al., lar) of the best-fitting model across socio-demographic 2022). Recently, Cano-Vindel et al. (2018) tested the dimen- characteristics. As in previous validations (Kocalevent sionality, reliability, and validity of a computerized version et al., 2014; Larionow & Mudło, 2023; Löwe et al., of the PHQ-4 in a Spanish sample of 1052 patients from 2010; Mendoza et al., 2022), it was expected that the 28 primary care centers. Results indicated adequate inter- dimensions were invariant across gender, age, income nal consistency for depressive (α = .86) and anxiety (α = .76) level, education level, and region (hypothesis 2). symptoms. Even though the PHQ-4 has been standardized 3. Thirdly, to explore the reliability of the PHQ-4 subscales on a representative sample of 1500 people from the gen- through different reliability indexes (i.e., Cronbach's eral Colombian population through face-to-face interviews α, McDonald's ω, and Guttman's λ2). The PHQ-2, the (Kocalevent et al., 2014) –also with adequate properties–, GAD-2, and the PHQ-4 subscales were expected to have there is no evidence of the psychometric properties of an the capacity to reliably measure anxiety, depression, and online version of this instrument in other Spanish-speak- psychological distress beyond the reliability index exam- ing countries apart from Spain. This is the first study that ined (hypothesis 3). provides information on the online version of the PHQ-4 4. Fourthly, to explore the relationship between the PHQ-4 in a large sample of the Colombian population during the scores with socio-demographic characteristics of this COVID-19 pandemic. sample. Based on the results from previous psycho- As far as it is known, the goodness-of-fit of a bifacto- metric studies (Cano-Vindel et al., 2018; Fong et al., rial structure for the PHQ-4 has not been tested. Recently, 2023; Kocalevent et al., 2014; Löwe et al., 2010; Meidl Tibubos et al. (2021) evaluated the internal structure of et al., 2023; Mendoza et al., 2022), it was expected that 13 Current Psychology females, older individuals, those with lower incomes, Table 1  Socio-demographic characteristics of the sample employed, or with lower levels of education would Variables, n (%) Sample (n = 18,061) exhibit higher depressive, anxiety, and psychological distress symptoms (hypothesis 4). Gender Female 13,491 (75.0) Male 4,495 (25.0) Method Age groups 18–24 years 4,564 (25.3) Study design 25–34 years 5,426 (30.0) 35–44 years 3,865 (21.4) Data analyses of the online version of the PHQ-4 were 45–54 years 2,411 (13.3) conducted using the database of the PSY-COVID study in ≥ 55 years 1,795 (9.9) Colombia (Sanabria-Mazo & Sanz, 2021). PSY-COVID is Income level a cross-sectional study that aimed to assess the psychosocial Low 5,173 (28.6) impact of the COVID-19 pandemic in 30 countries. Specifi- Medium 11,186 (61.9) cally, this article explored data from the general population High 1,702 (9.4) residing in Colombia during the first phase of the lockdown Work status measures. Using the database of PSY-COVID, two previous Employed 10,872 (60.3) studies have been published on the impact of COVID-19 Unemployed 7,162 (39.7) lockdown measures on the mental health in the Colombian Education level population (see Sanabria-Mazo et al., 2021a, b). Primary 322 (1.8) Secondary 1,536 (8.5) Participants University 16,185 (89.7) Region In total, 18,833 people completed the online questionnaire Amazon 285 (1.6) in Colombia, of which 772 were excluded from this analy- Andean 9,347 (51.9) sis because they resided in other countries during the first Caribbean 1,395 (7.8) wave of COVID-19 pandemic. Finally, the sample consisted Orinoco 557 (3.1) of 18,061 participants from all regions of the country. As Pacific 6,420 (35.7) shown in Table 1, majority of the participants were female n = frequency, % = percentage (75%), adults between 25–34 years old (30%), with medium income levels (62%), with university education level (90%), This research was approved by the Ethical Committee on and resided in the Andean region (52%). Inclusion criteria Animal and Human Experimentation of the Autonomous were adults (≥ 18 years old) residing in Colombia during the University of Barcelona (CEEAH-5197). period in which the data were collected (see Table 1). No participants who met the eligibility criteria were excluded Measures from the analyses. The Patient Health Questionnaire-4 (PHQ-4) was used to Procedure measure depressive and anxiety symptoms (Löwe et al., 2010). The two items of PHQ-2 correspond to the symptoms Administration of an anonymous online questionnaire gener- of the Diagnostic and Statistical Manual of Mental Disor- ated with Google Forms® was carried out using a non-prob- ders-IV (DSM-IV) diagnostic criteria for major depressive abilistic sampling (snowball method) from May ­20th to 20 disorder (i.e., loss of interest and depressive mood) and the June ­20th, 2020. The survey was distributed through social two items of GAD-2 to the symptoms of generalized anxiety networks, media, and institutional contacts. A panel of 30 disorder (i.e., nervousness and worries). This version con- international experts in clinical and health psychology vali- tains 4 items with a 4-point Likert response format, where dated the online questionnaire. The instruments used in this 0 corresponds to "not at all" and 3 to “nearly every day", online survey were piloted prior to administration. No eco- and questions are asked in the time frame of the last two nomic incentives were offered to participants for responding weeks. The total score of the PHQ-4 (psychological distress) to this anonymous survey. The time to complete the socio- ranges from 0 to 12 and the specific score of its two sub- demographic and PHQ-4 items were approximately 3 min. scales (PHQ-2 and GAD-2) ranges from 0 to 6. The cut-off More information on the procedure of the PSY-COVID study points for detecting probable cases of depression (PHQ-2) in Colombia is available in Sanabria-Mazo et al., (2021a, b). or anxiety (GAD-2) is 3 or more for each subscale; and for 13 Current Psychology probable cases of psychological distress (PHQ-4) is 6 or more comparable subsamples with random assignment. The con- for total scale (Cano-Vindel et al., 2018; Kocalevent et al., figural invariance provides evidence on the consistency of 2014; Löwe et al., 2010; Mendoza et al., 2022). the factor structure of the model across groups, the metric In addition, a socio-demographic information question- invariance on the factor loadings of the items across groups, naire was included to collect data about gender (female and and the scalar invariance on the equality of the mean scores male), age, income level (low, medium, and high), work sta- across groups (Van de Schoot et al., 2012). In addition, to tus (employed and unemployed), educational level (primary, determine measurement invariance, the multigroup CFA was secondary, and university), and region of residence (Ama- conducted, observing a change of ΔCFI that is less than or zon, Andean, Caribbean, Orinoco, and Pacific). equal to .01, according with Chen’s (2007). To estimate reliability, both total internal consistency Data analyses of the scale (PHQ-4) and the subscales (PHQ-2 and GAD- 2) were assessed through Cronbach's α, McDonald's ω, Initially, descriptive statistics were conducted for socio- and Guttman's λ2. Furthermore, a known-groups validity demographic variables as frequencies (n) and percentages approach was used to estimate associations between PHQ-4 (%). Characteristics of the PHQ-4 were explored, including scores and socio-demographic characteristics that have been item means and standard deviations (SD), skewness and kur- reported in the literature as risk factors for depression and tosis, corrected item-total correlations, among items in each anxiety. For this purpose, univariate group comparisons subscale, and between items of different subscales. Given were performed with the PHQ-2, the GAD-2, and the PHQ-4 the brevity of the scales, these correlations were analyzed scores as dependent variables through t-tests and analysis of using the Spearman-Brown correction. No outliers were variance (ANOVA), considering the Bonferroni adjustment identified in the analyses and none of the participants were for multiple testing. Statistical analyses were performed with excluded due to missing data. The dimensionality of the SPSS-26®, AMOS-5, R Studio, and JASP®. PHQ-4 was examined through CFA, using maximum likeli- hood as the estimation method. Regarding dimensionality, it tested a (1) one-factor model with the four items loading Results on the latent factor; (2) two-factors model including two correlated dimensions; and (3) bifactor model with the four Item and scale characteristics items saturated with a global latent factor of psychological distress plus two uncorrelated specific factors of anxiety and Table 2 shows descriptive analyses of the items, subscales depression. (PHQ-2 and GAD-2) and total scale (PHQ-4). Mean (SD) The Tucker–Lewis’s Index (TLI), Normed Fit Index score of PHQ-2 was 2.28 (1.61), GAD-2 was 2.01 (1.67), (NFI), and Comparative Fit Index (CFI) were used to evalu- and PHQ-4 was 4.29 (3.01). Corrected item-total correla- ate goodness-off, with > . 90 confidence intervals and the tions ranged from r = .62 to r = .77. Correlation between Root Mean Square Error of Approximation (RSMEA) < .08, the two items of PHQ-2 was r = .64 and between the two according to Schermelleh-Engel et al. (2003). The invariance items of GAD-2 was .68, while correlation of the items (configural, metric, and scalar) of the models were tested with the items of the other subscale ranged from r = .46 by gender, age, income level, education level, and region in to r = .68. PHQ-2 and GAD-2 had a correlation of r = .64, Table 2  Characteristics of the items and subscales of the PHQ-4 Items M (95% CI) SD Skewness Kurtosis Reliability α ω λ2 Depression (PHQ-2) 1. Little interest or pleasure in doing things 1.24 (1.23–1.25) 0.88 .50 -.37 2. Feeling down, depressed, or hopeless 1.04 (1.03–1.05) 0.89 .66 -.21 PHQ-2 total score 2.28 (2.26–2.30) 1.61 .61 -.02 .79 .81 .80 Anxiety (GAD-2) 1. Feeling nervous, anxious, or on edge 1.04 (1.03–1.05) 0.89 .64 -.23 2. Not being able to stop or control worrying 0.97 (0.95–0.98) .92 .72 -.30 GAD-2 total score 2.01 (1.98–2.03) 1.67 .74 -.10 .83 .83 .82 Distress (PHQ-4) PHQ-4 total score 4.29 (4.24–4.33) 3.01 .73 -.03 .86 .86 .86 13 Current Psychology indicating high overlap between subscales. All the above Comparable subsamples with random assignment correlations were statistically significant (p < .01). were used to test the invariance (configural, metric, and scalar) of the two-factor correlated model by gen- Dimensionality der (female: n  = 4,305; male: n  = 4,295), age (≤ 32 year s: n  = 9,169; > 32 years: n  = 8,892), income level (low: As shown in Table 3, the fit indices for the correlated two- n = 1,173; medium: n = 1,186; high: n = 1,102), education factor model were significantly better (p < .001; X2 = 938.24) level (primary: n = 340; secondary: n = 436; university: than those obtained for the one-factor model [CFI (0.99 vs. n = 485), and region (Amazon: n = 285; Andean: n = 447; 0.94), TLI (0.99 vs. 0.83), NFI (0.99 vs. 94), and RMSEA Caribbean: n = 345; Orinoco: n = 328; Pacific: n = 420). (0.04 vs. 0.23)], which provides strong support for the Table 4 shows that no structural differences were identified adequacy of the original model proposed by Kroenke et al. the best-fitting model according to gender, age, income (2009). The bifactor structure was tested, although no con- level, education level, and region with a Δ CFI lower than vergence was found for this model. .01, which confirm hypothesis 2. Regarding factor loadings of the tested factor models, in the two-factor model ranged between .71 and .92, and Reliability those of the one-factor model between .68 and .83 (see Fig. 1). The results slightly differed between the specific Reliability of the PHQ-2 (α = .79, ω = .81, and λ2 = .80), factors of anxiety and depression. In line with hypoth- the GAD-2 (α = .83, ω = .83, and λ2 = .82), and the PHQ-4 esis 1, these results confirm that the two-factor correlated (α = .86, ω = .86, and λ2 = .86) was above .78 on all calculated model have a significantly better fit to the data than the indicators. The adequate reliability indices for depression, other models. anxiety, and psychological distress confirms hypothesis 3. Table 3  Confirmatory factor SB X2 df CFI TLI NFI RMSEA X2 p analysis (CFA) comparing fit indices of the one-factor and One-factor model 1960.335 1 .94 .83 .94 .23 938.24 < .001 two-factor model of PHQ-4 Two-factor model 33.528 6 .99 .99 .99 .04 The bifactor structure was tested, although no convergence was found for this model One-factor model Two-factor model .71 Loss of Loss of interest interest .68 PHQ-2 .92 Depressive .83 mood Depressive mood .83 PHQ-4 .81 .84 Nervousness Nervousness .81 GAD-2 .85 Worries Worries Fig. 1  Factor loadings for the one-and two-factor models of the PHQ-4 13 Current Psychology Table 4  Test for configural invariance across gender, age, income level, education level, and region using multi-group CFA X2 (df) ΔP CMIN/df CFI Δ CFI RMSEA Δ RMSEA Invariance Across gender Groups Female (n = 4,305) 13.772 (2) – 6.886 .999 – .036 – – Male (n = 4,295) 12.054 (2) – 6.027 .999 – .033 – – Multigroup analysis Configural model 32.289 (2) – – .999 .029 Invariant Metric model 33.688 (4) < .01 8.422 .999 .000 .020 .009 Invariant Scalar model 175.090 (8) < .01 21.886 .995 .004 .034 .014 Invariant Across age Groups ≤ 32 years (n = 9,169) 15.143(1) – 15.143 .999 – .039 – – > 32 years (n = 8,892) 13.543 (1) – 13.543 .999 – .038 – – Multigroup analysis Configural model 28.685 (2) – 14.343 .999 – .027 – Invariant Metric model 34.327 (4) < .01 8.582 .999 .000 .020 .007 Invariant Scalar model 1444.32 (8) < .01 180.540 .991 .008 .035 .015 Invariant Across income level Groups Low (n = 1,173) 17.424 (1) – 17.424 .998 – .056 – – Medium (n = 1,186) 11.049 (1) – 11.049 1.00 – .030 – – High (n = 1,102) 2.900 (1) – 2.900 .999 – .033 – – Multigroup analysis Configural model 31.370 (3) – – .999 – .023 Invariant Metric model 51.474 (7) < .01 7.35 .999 .000 .019 .004 Invariant Scalar model 586.068 (15) < .01 39.5071 .994 .005 .046 .027 Invariant Across education level Groups Primary (n = 340) 17.424 (1) – 17.424 .998 – .056 – – Secondary (n = 436) 11.049 (1) – 11.049 1.00 – .030 – – University (n = 485) 2.900 (1) – 2.900 .999 – .033 – – Multigroup analysis Configural model 31.370 (3) – – .999 .023 Invariant Metric model 51.474 (7) < .01 7.35 .999 .000 .019 .004 Invariant Scalar model 586.068 (15) < .01 39.5071 .989 .010 .046 .027 Invariant Across region Groups Amazon (n = 285) .005 (1) – 0.005 1.00 – .000 – – Andean (n = 447) 5.609 (1) – 5.609 1.00 – .022 – – Caribbean (n = 345) 8.369 (1) – 8.369 .997 – .073 – – Orinoco (n = 328) .123 (1) – .123 1.00 – .000 – – Pacific (n = 420) 28.147 (1) – 28.147 .998 – .065 – – Multigroup analysis Configural model 42.253 (5) – – .999 – .020 – Invariant Metric model 55.012 (13) < .01 4.232 .999 .000 .013 .007 Invariant Scalar model 157.242 (29) < .01 5,422 .996 .003 .016 .003 Invariant 13 Current Psychology Known groups validity paper-and-pencil format of the PHQ-4 in clinical (Ghaheri et al., 2020; Kroenke et al., 2009; Renovanz et al., 2019; As shown in Table 5, statistically significant differences Weihs et al., 2018) and non-clinical samples (Khubchan- were found in the PHQ-2, the GAD-2, and the PHQ-4 dani et al., 2016; Kocalevent et al., 2014; Löwe et al., scores according to gender, age, income level, work status, 2010; Mills et al., 2015), as well as non-clinical samples in and educational level, but with small effect sizes (d < 0.2 online version during the first few months of the COVID- and η2 < 0.12). Females, younger age, unemployed, and 19 outbroke in Philippines (Mendoza et al., 2022). How- those with lower incomes and educational levels reported ever, as far as it is known, this is the first study to evaluate the higher depression (PHQ-2), anxiety (GAD-2), and psy- the dimensionality and reliability of an online version of chological distress (PHQ-4) scores. The higher scores for the PHQ-4 in a large sample of the general population in females, and those with lower incomes and educational lev- Colombia during the COVID-19 pandemic. els were consistent with hypothesis 4. However, they were Consistent with previous research carried out in the inconsistent with the higher scores expected for older age classic paper-and-pencil and face-to-face interviews and employed. For more information, the prevalence of (Kocalevent et al., 2014; Kroenke et al., 2009; Löwe depressive (35%) and anxiety (29%) symptoms can be read et al., 2010), CFA indicates that the two-factor struc- in detail in Sanabria-Mazo et al. (2021a). ture (i.e., depression and anxiety) of the online PHQ-4 performs significantly better than the one-factor struc- ture (i.e., psychological distress), with excellent fit indi- Discussion ces on all parameters (CFI = .99, TLI = .99, NFI = .99, RMSEA = .04). Furthermore, these findings were con- The findings of this study provide evidence that the online sistent with those reported in another recent study of version of PHQ-4 is a reliable ultra-brief self-adminis- the PHQ-4 administered online (Mendoza et al., 2022). tered instrument for measuring depressive and anxiety The bifactorial model did not converge probably due symptoms in the general population in Colombia. Previ- to the small number of indicators per latent variable. ous studies have demonstrated the validity of the classic Therefore, these results demonstrate that the two-factor Table 5  Association PHQ-4 scores and socio-demographic characteristics Variables PHQ-2 (0–6) GAD-2 (0–6) PHQ-4 (0–12) Score Difference groups Score Difference groups Score Difference groups M (SD) p value (effect size) M (SD) p value (effect size) M (SD) p value (effect size) Gender < .001 (d = 0.15) < .001 (d = 0.19) < .001 (d = 0.19) Female 2.34 (1.60) 2.08 (1.68) 4.42 (3.00) Male 2.09 (1.63) 1.77 (1.63) 3.86 (2.99) Age groups < .001 (η2 = 0.09) < .001 (η2 = 0.03) < .001 (η2 = 0.07) 1. 18–24 years 2.98 (1.65) 1>2>3>4>5 2.43 (1.81) 1>2>3>4>5 5.41 (3.14) 1>2>3>4>5 2. 25–34 years 2.38 (1.55) 2.04 (1.68) 4.42 (2.96) 3. 35–44 years 1.99 (1.50) 1.86 (1.57) 3.85 (2.82) 4. 45–54 years 1.76 (1.45) 1.70 (1.54) 3.46 (2.75) 5. ≥ 55 years 1.53 (1.37) 1.55 (1.40) 3.08 (2.51) Income level < .001 (η2 = 0.03) < .001 (η2 = 0.01) < .001 (η2 = 0.02) 1. Low 2.69 (1.70) 1>2>3 2.27 (1.80) 1>2>3 4.95 (3.23) 1>2>3 2. Medium 2.15 (1.54) 1.92 (1.61) 4.08 (2.88) 3. High 1.88 (1.54) 1.76 (1.55) 3.63 (2.81) Work status < .001 (d = 0.15) < .001 (d = 0.08) < .001 (d = 0.13) Employed 2.18 (1.57) 1.95 (1.64) 4.13 (2.94) Unemployed 2.43 (1.65) 2.09 (1.72) 4.52 (3.09) Education level = .002 (η2 < 0.01) = .004 (η2 < 0.01) = .001 (η2 = 0.01) 1. Primary 2.25 (1.74) 2.06 (1.80) 4.32 (3.28) 2. Secondary 2.42 (1.71) 2>3>1 2.14 (1.82) 2>3>1 4.55 (3.26) 2>3>1 3. University 2.27 (1.60) 1.99 (1.65) 4.26 (2.98) Between-group differences were calculated through t-test and ANOVA, considering the Bonferroni adjustment for multiple testing 13 Current Psychology correlated model is the best fit to the data, confirming et al., 2014), results in line a vast amount of empirical hypothesis 1. evidence reflecting the negative effect of the COVID-19 The high correlation between the depression (PHQ-2) pandemic on the mental health (Hossain et al., 2020), a and anxiety (GAD-2) subscales was like those reported in phenomenon also observed in other studies on Colombian previous studies (Kroenke et al., 2009; Löwe et al., 2010; population (Caballero-Domínguez et al., 2022; Cénat et al., Mendoza et al., 2022). Comorbidity between these mood 2022; Gómez-Restrepo et al., 2022). disorders, close to 50% of cases (Kessler et al., 2015), theo- Regarding prevalence, 35% of the participants in this retically explains the high correlation identified between sample showed depressive symptoms and 29% anxiety both subscales of the PHQ-4 (Kocalevent et al., 2014; symptoms (see Sanabria-Mazo et al., 2021a, b). Consist- Kroenke et al., 2009; Löwe et al., 2010; Mendoza et al., ent with other research in the general population during the 2022). The structure of two factors (depression and anxi- COVID-19 pandemic (Luo et al., 2020; Xiong et al., 2020, ety) that share a higher order factor (psychological distress) it was found that about one third of this sample showed is consistent with the conception of two nosological entities depressive and anxiety symptoms. Specifically, the popula- clearly differentiated by the causal cognitive processes and tion groups most affected by the COVID-19 pandemic in their clinical manifestations (Clark & Watson, 1991) and Colombia were low-income people, students, and young with the extensive empirical evidence of the high comorbid- adults, with depressive symptoms between 46 and 56% and ity of both disorders (Kessler et al., 2015). anxiety symptoms between 36 and 40% (Sanabria-Mazo As in previous validations (Fong et al., 2023; Kocalevent et al., 2021a). Compared with the results of a national sur- et al., 2014; Löwe et al., 2010; Meidl et al., 2023; Men- vey, this study provided evidence that there were 2.5 to 2.8 doza et al., 2022), the two-factor structure of the PHQ-4 times more people with risk of anxiety and 1.5 to 1.9 times was invariant (configural, metric, and scalar) across gender, more with risk of depression in the first wave of the COVID- age, income level, education level, and region in this study, 19 outbreak (Sanabria-Mazo et al., 2021a). Although the supporting hypothesis 2. Given the geographically based above comparison is from a non-representative sample of the cultural heterogeneity of Colombia, regional invariance of Colombian population, the reported differences highlight the PHQ-4 seems to be a particularly relevant finding. Despite need to prioritize prevention, intervention, and monitoring the criticisms about the real relevance of the invariance of of symptoms related to emotional disorders. psychological assessment instruments (Welzel et al., 2021), there is certainly a great consensus that in cross-cultural studies it is necessary to guarantee its homogeneous behav- Limitations ior. Recently, the results of a cross-cultural research con- ducted in 7 European countries (Austria, Croatia, Georgia, These findings should be interpreted considering the fol- Germany, Lithuania, Portugal, and Sweden) have shown that lowing limitations. First, the analyses were conducted the PHQ-4 could be generalized to other countries and cul- based on a non-representative sample, which impedes tures (Kazlauskas et al., 2023). the generalizability of the results to the general popula- With regard to reliability, internal consistency values tion of Colombia or other Spanish-speaking languages. were slightly higher than those reported in other psycho- Second, due to the cross-sectional design, it was not pos- metric studies (Cano-Vindel et al., 2018; Ghaheri et al., sible to calculate the test–retest reliability of the instru- 2020; Khubchandani et al., 2016; Kocalevent et al., 2014; ment. Third, although the convergent and divergent of Löwe et al., 2010; Mendoza et al., 2022), with values close the PHQ-4 has been demonstrated in previous studies to α = .83 for depression (PHQ-2), α = .79 for anxiety (GAD- (Kocalevent et al., 2014; Kroenke et al., 2009; Löwe 2), and α = .86 for psychological distress (PHQ-4), which et al., 2010), no other instruments were used to provide supports hypothesis 3. Although small effect sizes were further evidence of construct validity. Fourth, diagnostic obtained, the findings of this study provide further evidence interviews were not considered as a procedure to verify about gender, age, income level, work status, and educa- criterion validity, making it not possible to provide fur- tional level role as risk factors for depression and anxiety. In ther evidence on specificity and sensitivity for the optimal line with other studies and hypothesis 4, it was identified that cut-off point (Mitchell et al., 2016; Plummer et al., 2016). people of female gender, low income, and low education lev- Fifth, it was not possible examine the responsiveness, the els reported higher scores on depression, anxiety, and psy- smallest detectable change, or the minimal clinical impor- chological distress (Khubchandani et al., 2016; Kocalevent tant difference for scoring the PHQ-4. Sixth, online data et al., 2014; Löwe et al., 2010; Mendoza et al., 2022; Mills collection can have a negative impact on the representa- et al., 2015). In contrast, it was found that younger people tion of population groups with internet connection diffi- and unemployed reported higher scores than people who culties, lack of knowledge in the use of new technologies, were older (Löwe et al., 2010) and employed (Kocalevent and low literacy. 13 Current Psychology Conclusions the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a In summary, this study provides further evidence on the copy of this licence, visit http://​creat​iveco​mmons.​org/​licen​ses/​by/4.​0/. dimensionality and reliability of an ultra-brief online screen- ing instruments for the detection of depressive and anxi- ety symptoms. It also shows that presentation in its online References format does not alter its psychometric properties and it is invariant across gender, age, income level, education level, Baxter, A. J., Vos, T., Scott, K. M., Ferrari, A. J., & Whiteford, and region. The existing results from the PHQ-2 and GAD-2 H. A. (2014). The global burden of anxiety disorders in 2010. Psychological Medicine, 44(11), 2363. https://​doi.​org/​10.​1017/​ denote similar psychometric behavior to the full versions of S0033​29171​30032​43 the PHQ-9 and GAD-2. Although the PHQ-2 and GAD-2 Caballero-Domínguez, C. C., Pedrozo-Pupo, J. C., & Campo-Arias, are reliable subscales for rapid screening of depression and A. (2022). Depression among COVID-19 survivors in Colom- anxiety, the use of their full versions is recommended when bia. Psychology, Health & Medicine, 28(6), 1–8. https://​doi.​org/​ 10.​1080/​13548​506.​2022.​20858​79 all DSM-IV diagnostic criteria need to be assessed. In line Cano-Vindel, A., Muñoz-Navarro, R., Medrano, L. A., Ruiz-Rod- with the proposal by Löwe et al. (2010), it is suggested to ríguez, P., González-Blanch, C., Gómez-Castillo, M. D., Capa- use the total scale as a global screening tool for psycho- fons-Bonet, A., Chacón, F., Santolayay, F., & PsicAP Research logical distress (PHQ-4), and the depression (PHQ-2) and Group. (2018). A computerized version of the Patient Health Questionnaire-4 as an ultra-brief screening tool to detect emo- anxiety (GAD-2) subscales for their discriminated detection. tional disorders in primary care. Journal of Affective Disorders, Finally, PHQ-4 is an ultra-brief screening tool that can help 234(247), 255. https://​doi.​org/​10.​1016/j.​jad.​2018.​01.​030 to optimize the time resources of health systems, especially Campbell, N., Ali, F., Finlay, A. Y., & Salek, S. S. (2015). Equiva- during health crises. lence of electronic and paper-based patient-reported outcome measures. Quality of Life Research: An International Journal of Acknowledgements JPS-M has a PFIS predoctoral contract from the Quality-of-Life Aspects of Treatment, Care, and Rehabilitation, Institute of Health Carlos III (ISCIII; FI20/00034). AF-S acknowl- 24(8), 1949–1961. https://​doi.​org/​10.​1007/​s11136-​015-​0937-3 edges the funding from the Serra Húnter program (Generalitat de Castro-Rodríguez, J. I., Olariu, E., Garnier-Lacueva, C., Martín- Catalunya; reference number UAB-LE-8015). The authors are grate- López, L. M., Pérez-Solà, V., Alonso, J., Forero, C. G., & ful to the CIBER of Epidemiology and Public Health (CIBERESP INSAyD Investigators. (2015). Diagnostic accuracy and ade- CB22/02/00052; ISCIII) for its support. quacy of treatment of depressive and anxiety disorders: A com- parison of primary care and specialized care patients. Journal Author contributions Juan P. Sanabria-Mazo: conceptualization, data of Affective Disorders, 172, 462–471. https://​doi.​org/​10.​1016/j.​ curation, software, formal analysis, methodology, validation, visualisa- jad.​2014.​10.​020 tion, and writing—original draft. Andrés Gómez-Acosta: conceptual- Cénat, J. M., Farahi, S. M. M. M., Dalexis, R. D., Darius, W. P., ization, formal analysis, methodology, validation, and writing—review Bekarkhanechi, F. M., Poisson, H., Broussard, C., Ukwu, G., & editing. John Alexander Castro-Muñoz: conceptualization, valida- Auguste, E., Nguyen, D. D., Sehabi, G., Furyk, S. E., Gedeon, A. tion, and writing—review & editing. Yonatan Ferney Rojas: method- P., Onesi, O., El Aouame, A. M., Khodabocus, S. N., Shah, M. ology, formal analysis, and writing—review & editing. Albert Feliu- S., & Labelle, P. R. (2022). The global evolution of mental health Soler: methodology and writing – review & editing. Juan V. Luciano: problems during the COVID-19 pandemic: A systematic review methodology and writing – review & editing. Antoni Sanz: funding and meta-analysis of longitudinal studies. Journal of Affective acquisition, project administration, methodology, and writing—review Disorders, 315, 70–95. https://​doi.​org/​10.​1016/j.​jad.​2022.​07.​011 & editing. Chen, F. F. (2007). Sensitivity of goodness of ft indexes to lack of measurement invariance. Structural Equation Modeling, 14(3), Funding Open Access Funding provided by Autonomous University 464–504. https://​doi.​org/​10.​1080/​10705​51070​13018​34 of Barcelona. This research was funded by the Agency for Management Clark, L. A., & Watson, D. (1991). Tripartite model of anxiety and of University and Research Grants (AGAUR; 2020PANDE00025). depression: Psychometric evidence and taxonomic implications. Journal of Abnormal Psychology, 100, 316–336. https://​doi.​org/​ Data availability The database of this study is available in Sanabria- 10.​1037/​0021-​843X.​100.3.​316 Mazo and Sanz (2021). Coons, S. J., Gwaltney, C. J., Hays, R. D., Lundy, J. J., Sloan, J. A., Revicki, D. A., Lenderking, W. R., Cella, D., & Basch, E. (2009). Recommendations on evidence needed to support Declarations measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO Good Conflict of interest All authors declare no conflict of interest. Research Practices Task Force report. Value in Health, 12(4), 419–429. https://​doi.​org/​10.​1111/j.​1524-​4733.​2008.​00470.x Open Access This article is licensed under a Creative Commons Attri- Drapeau, A., Marchand, A., & Beaulieu-Prévost, D. (2012). Epidemi- bution 4.0 International License, which permits use, sharing, adapta- ology of psychological distress. In L. Labate (Ed.), Mental ill- tion, distribution and reproduction in any medium or format, as long nesses - Understanding, prediction, and control (pp. 105–133). as you give appropriate credit to the original author(s) and the source, https://​doi.​org/​10.​5772/​30872 provide a link to the Creative Commons licence, and indicate if changes Fong, T. C., Ho, R. T., & Yip, P. S. (2023). Psychometric properties were made. The images or other third party material in this article are of the Patient Health Questionnaire-4 among Hong Kong young included in the article's Creative Commons licence, unless indicated adults in 2021: Associations with meaning in life and suicidal otherwise in a credit line to the material. If material is not included in 13 Current Psychology ideation. Frontiers in Psychiatry, 14, 1138755. https://​doi.​org/​ Lim, G. Y., Tam, W. W., Lu, Y., Ho, C. S., Zhang, M. W., & Ho, R. 10.​3389/​fpsyt.​2023.​11387​55 C. (2018). Prevalence of depression in the community from 30 Ghaheri, A., Omani-Samani, R., Sepidarkish, M., Hosseini, M., & countries between 1994 and 2014. Scientific Reports, 8(1), 2861. Maroufzadeh, S. (2020). The four-item patient health question- https://​doi.​org/​10.​1038/​s41598-​018-​21243-x naire for anxiety and depression: A validation study in infertile Löwe, B., Wahl, I., Rose, M., Spitzer, C., Glaesmer, H., Wingenfeld, patients. International Journal of Fertility and Sterility, 14(3), K., Schneider, A., & Brähler, E. (2010). A 4-item measure of 234–239. https://​doi.​org/​10.​22074/​ijfs.​2020.​44412 depression and anxiety: Validation and standardization of the Goldberg, D. P., Reed, G. M., Robles, R., Minhas, F., Razzaque, Patient Health Questionnaire-4 (PHQ-4) in the general popula- B., Fortes, S., de Jesus, J., Pong-Lam, T., García, J. A., Gask, tion. Journal of Affective Disorders, 122(1–2), 86–95. https://​doi.​ L., Dowell, A. C., Rosendal, M., Mbatia, J. K., & Saxena, S. org/​10.​1016/j.​jad.​2009.​06.​019 (2017). Screening for anxiety, depression, and anxious depres- Luo, M., Guo, L., Yu, M., Jiang, W., & Wang, H. (2020). The psycho- sion in primary care: A field study for ICD-11 PHC. Journal logical and mental impact of coronavirus disease 2019 (COVID- of Affective Disorders, 213, 199–206. https://​doi.​org/​10.​1016/j.​ 19) on medical staff and general public: A systematic review and jad.​2017.​02.​025 meta-analysis. Psychiatry Research, 291, 113190. https://​doi.​org/​ Gómez-Restrepo, C., Cepeda, M., Torrey, W. C., Suarez-Obando, 10.​1016/j.​psych​res.​2020.​113190 F., Uribe-Restrepo, J. M., Park, S., Jassir, M. P., Martínez, P., Meidl, V., Dallmann, P., Leonhart, R., Bretthauer, B., Busch, A., Castro, S. M., Aguilera, J., González, L., Chaparro, N., Gómez, Kubosch, E. J., Wrobel, N., & Hirschmüller, A. (2023). Valida- A. M., Bell, K., & Marsch, L. A. (2022). Perceived access to tion of the Patient Health Questionnaire-4 for longitudinal mental general and mental healthcare in primary care in Colombia health evaluation in elite Para athletes. PM&R. https://​doi.​org/​10.​ during COVID-19: A cross-sectional study. Frontiers in Public 1002/​pmrj.​13011 Health, 10, 3124. https://​doi.​org/​10.​3389/​fpubh.​2022.​896318 Mendoza, N. B., Frondozo, C. E., Dizon, J. I. W. T., & Buenconsejo, Hossain, M. M., Sultana, A., & Purohit, N. (2020). Mental health J. U. (2022). The factor structure and measurement invariance outcomes of quarantine and isolation for infection prevention: of the PHQ-4 and the prevalence of depression and anxiety in a A systematic umbrella review of the global evidence. SSRN, 42, Southeast Asian context amid the COVID-19 pandemic. Current 3561265. https://​doi.​org/​10.​2139/​ssrn.​35612​65 Psychology, 1, 1–10. https://d​ oi.o​ rg/1​ 0.1​ 007/s​ 12144-0​ 22-0​ 2833-5 Kazlauskas, E., Gelezelyte, O., Kvedaraite, M., Ajdukovic, D., Mills, S. D., Fox, R. S., Pan, T. M., Malcarne, V. L., Roesch, S. C., Johannesson, K. B., Böttche, M., Bondjers, K., Dragan, M., & Sadler, G. R. (2015). Psychometric evaluation of the patient Figueiredo-Braga, M., Grajewski, P., Anastassiou-Hadjich- health questionnaire–4 in Hispanic Americans. Hispanic Journal aralambous, X., Javakhishvili, J. D., Lioupi, C., Lueger-Schus- of Behavioral Sciences, 37(4), 560–571. https://​doi.​org/​10.​1177/​ ter, B., Mouthaan, J., Bagaric, I. R., Sales, L., Schäfer, I., Soy- 07399​86315​608126 das, S., … Lotzin, A. (2023). Psychometric properties of the Mitchell, A. J., Yadegarfar, M., Gill, J., & Stubbs, B. (2016). Case Patient Health Questionnaire-4 (PHQ-4) in 9230 adults across finding and screening clinical utility of the patient health question- seven European countries: Findings from the ESTSS ADJUST naire (PHQ-9 and PHQ-2) for depression in primary care: A diag- study. Journal of Affective Disorders, 335, 18–23. https://​doi.​ nostic meta-analysis of 40 studies. British Journal of Psychiatry org/​10.​1016/j.​jad.​2023.​05.​007 Open, 2(2), 127–138. https://d​ oi.o​ rg/1​ 0.1​ 192/b​ jpo.b​ p.1​ 15.0​ 01685 Kessler, R. C., Sampson, N. A., Berglund, P., Gruber, M. J., Al- Muñoz-Navarro, R., Cano-Vindel, A., Medrano, L. A., Schmitz, F., Hamzawi, A., Andrade, L., Bunting, B., Demyttenaere, K., Flo- Ruiz-Rodríguez, P., Abellán-Maeso, C., Font-Payeras, M. A., & rescu, S., de Girolamo, G., Gureje, O., He, Y., Hu, C., Huang, Y., Hermosilla-Pasamar, A. M. (2017a). Utility of the PHQ-9 to iden- Karam, E., Kovess-Masfety, V., Lee, S., Levinson, D., Medina tify major depressive disorder in adult patients in Spanish primary Mora, M. E., … Wilcox, M. A. (2015). Anxious and non-anxious care centers. BMC Psychiatry, 17(1), 291. https://d​ oi.o​ rg/1​ 0.1​ 186/​ major depressive disorder in the World Health Organization World s12888-​017-​1450-8 Mental Health Surveys. Epidemiology and Psychiatric Sciences, Muñoz-Navarro, R., Cano-Vindel, A., Moriana, J. A., Medrano, L. A., 24(3), 210–226. https://​doi.​org/​10.​1017/​S2045​79601​50001​89 Ruíz-Rodríguez, P., Agüero, L., Rodríguez-Enríquez, M., Pizà, Khubchandani, J., Brey, R., Kotecki, J., Kleinfelder, J. A., & Ander- M. R., & Ramírez-Manent, I. (2017b). Screening for generalized son, J. (2016). The psychometric properties of PHQ-4 depression anxiety disorder in Spanish primary care centers with the GAD- and anxiety screening scale among college students. Archives of 7. Psychiatry Research, 256, 312–317. https://​doi.​org/​10.​1016/j.​ Psychiatric Nursing, 30(4), 457–462. https://​doi.​org/​10.​1016/j.​ psych​res.​2017.​06.​023 apnu.​2016.​01.​014 Olariu, E., Forero, C. G., Castro-Rodriguez, J. I., Rodrigo-Calvo, M. Kocalevent, R. D., Finck, C., Jimenez-Leal, W., Sautier, L., & Hinz, A. T., Álvarez, P., Martín-López, L. M., Sánchez-Toto, A., Adro- (2014). Standardization of the Colombian version of the PHQ-4 in her, N. D., Blasco-Cubedo, M. J., Vilagut, G., Fullana, M. A., & the general population. BMC Psychiatry, 14(1), 1–8. https://​doi.​ Alonso, J. (2015). Detection of anxiety disorders in primary care: org/​10.​1186/​1471-​244X-​14-​205 A meta-analysis of assisted and unassisted diagnoses. Depression Kroenke, K., Spitzer, R. L., & Williams, J. B. (2001). The PHQ-9: and Anxiety, 32(7), 471–484. https://​doi.​org/​10.​1002/​da.​22360 Validity of a brief depression severity measure. Journal of Gen- Plummer, F., Manea, L., Trepel, D., & McMillan, D. (2016). Screening eral Internal Medicine, 16(9), 606–613. https://​doi.​org/​10.​1046/j.​ for anxiety disorders with the GAD-7 and GAD-2: A systematic 1525-​1497.​2001.​01600​9606.x review and diagnostic metaanalysis. General Hospital Psychiatry, Kroenke, K., Spitzer, R. L., Williams, J. B., & Löwe, B. (2009). An 39, 24–31. https://​doi.​org/​10.​1016/j.​genho​sppsy​ch.​2015.​11.​005 ultra-brief screening scale for anxiety and depression: The PHQ– Renovanz, M., Soebianto, S., Tsakmaklis, H., Keric, N., Nadji-Ohl, 4. Psychosomatics, 50(6), 613–621. https://​doi.​org/​10.​1176/​appi.​ M., Beutel, M., Ringel, F., Wollschlager, D., & Hickmann, A. K. psy.​50.6.​613 (2019). Evaluation of the psychological burden during the early Larionow, P., & Mudło, K. (2023). The Patient Health Question- disease trajectory in patients with intracranial tumors by the ultra- naire–4: Factor structure, measurement invariance, latent profile brief Patient Health Questionnaire for Depression and Anxiety analysis of anxiety and depressive symptoms and screening results (PHQ-4). Supportive Care in Cancer, 27(12), 4469–4477. https://​ in polish adults. Advances in Cognitive Research, 19(2), 123–137. doi.​org/​10.​1007/​s00520-​019-​04718-z https://​doi.​org/​10.​5709/​acp-​0384-9 Sanabria-Mazo, J. P., & Sanz, A. (2021). PSY-COVID database 1 (Colombia). https://​doi.​org/​10.​6084/​m9.​figsh​are.​13951​775.​v3 13 Current Psychology Sanabria-Mazo, J. P., Useche-Aldana, B., Ochoa, P. P., Rojas-Gualdrón, Psychology, 9(4), 486–492. https://​doi.​org/​10.​1080/​17405​629.​ D. F., Mateo-Canedo, C., Carmona-Cervelló, M., Crespo-Puig, 2012.​686740 N., Selva-Olid, C., Muro, A., Méndez-Ulrich, J. L., Feliu-Soler, Weihs, K. L., Wiley, J. F., Crespi, C. M., Krull, J. L., & Stanton, A. L. A., Luciano, J. V., & Sanz, A. (2021a). Social inequities in the (2018). Predicting future major depression and persistent depres- impact of COVID-19 lockdown measures on the mental health of sive symptoms: Development of a prognostic screener and PHQ-4 a large sample of the Colombian population (PSY-COVID study). cutoffs in breast cancer patients. Psycho-Oncology, 27(2), 420– Journal of Clinical Medicine, 10(22), 5297. https://​doi.​org/​10.​ 426. https://​doi.​org/​10.​1002/​pon.​4472 3390/​jcm10​225297 Welzel, C., Brunkert, L., & Inglehart, R.F. (2021). Non-invariance? Sanabria-Mazo, J. P., Useche-Aldana, B., Ochoa, P. P., Rojas-Gualdrón, An overstated problem with misconceived causes. Sociological D. F., & Sanz, A. (2021b). Impacto de la pandemia de COVID-19 Methods & Research, 52(3), 1368–1400. https://​doi.​org/​10.​1177/​ en la salud mental en Colombia. Editorial CES. https://w ​ ww.c​ olps​ 00491​24121​995521 ic.o​ rg.c​ o/w ​ p-c​ onten​ t/u​ pload​ s/2​ 021/0​ 9/L ​ ibro-I​ mpact​ o-d​ e-l​ a-p​ ande​ World Health Organization. (2011). Global burden of mental disor- mia-​de-​COVID-​19-​en-​la-​salud-​mental-​en-​Colom​bia.​pdf ders and the need for a comprehensive, coordinated response Schermelleh-Engel, K., Moosbrugger, H., & Müller, H. (2003). Evalu- from health and social sectors at the country level. World Health ating the fit of structural equation models: Tests of significance Organization. and descriptive goodness-of-fit measures. Methods of Psychologi- Wittchen, H. U., Jacobi, F., Rehm, J., Wittchen, H. U., Jacobi, F., cal Research Online, 8(2), 23–74. Rehm, J., Gustavsson, A., Svensson, M., Jonsson, B., Olesen, J., Spitzer, R. L., Kroenke, K., Williams, J. B., & Löwe, B. (2006). A brief Allgulander, C., Alonso, J., Faravelli, C., Fratiglioni, L., Jennum, measure for assessing generalized anxiety disorder: The GAD-7. P., Lieb, R., Maercker, A., van Os, J., Preisig, M., … Steinhausen, Archives of Internal Medicine, 166(10), 1092–1097. https://​doi.​ H. C. (2011). The size and burden of mental disorders and other org/​10.​1001/​archi​nte.​166.​10.​1092 disorders of the brain in Europe 2010. European Neuropsychop- Tibubos, A. N., Otten, D., Zöller, D., Binder, H., Wild, P. S., Fleischer, harmacology, 21, 655–679. https://​doi.​org/​10.​1016/j.​euron​euro.​ T., Johar, H., Atasoy, S., Schulze, L., Ladwig, H. K., Schomerus, 2011.​07.​018 G., Linkohr, B., Grabe, H. J., Kruse, J., Schmidt, C. O., Münzel, Xiong, J., Lipsitz, O., Nasri, F., Lui, L., Gill, H., Phan, L., Chen-Li, T., König, J., Brähler, E., & Beutel, M. E. (2021). Bidimensional D., Iacobucci, M., Ho, R., Majeed, A., & McIntyre, R. S. (2020). structure and measurement equivalence of the Patient Health Impact of COVID-19 pandemic on mental health in the general Questionnaire-9: Sex-sensitive assessment of depressive symp- population: A systematic review. Journal of Affective Disorders, toms in three representative German cohort studies. BMC Psy- 277, 55–64. https://​doi.​org/​10.​1016/j.​jad.​2020.​08.​001 chiatry, 21(1), 1–13. https://d​ oi.o​ rg/1​ 0.1​ 186/s​ 12888-0​ 21-0​ 3234-x Van de Schoot, R., Lugtig, P., & Hox, J. (2012). A checklist for test- Publisher's Note Springer Nature remains neutral with regard to ing measurement invariance. European Journal of Developmental jurisdictional claims in published maps and institutional affiliations. 13