ORIGINAL RESEARCH published: 19 April 2022 doi: 10.3389/fpsyg.2022.834843 Measuring Perceived Research Competence of Junior Researchers Sarah A. Marrs 1* , Carla Quesada-Pallarès 2,3 , Korinthia D. Nicolai 4 , Elizabeth A. Severson-Irby 4 and J. Reinaldo Martínez-Fernández 5 1 College of Health Professions, Virginia Commonwealth University, Richmond, VA, United States, 2 Applied Pedagogy Department, Universitat Autònoma de Barcelona, Barcelona, Spain, 3 Serra Hunter Fellow, Catalonia, Spain, 4 School of Education, Virginia Commonwealth University, Richmond, VA, United States, 5 Cognitive, Developmental, and Educational Psychology Department, Universitat Autónoma de Barcelona, Barcelona, Spain Graduates of doctoral (Ph.D.) programs are expected to be competent at designing and conducting research independently. Given the level of research competence needed to successfully conduct research, it is important that assessors of doctoral programs (e.g., faculty and staff) have a reliable and validated tool for measuring and tracking perceived research competence among their students and graduates. A high level of research competence is expected for all Ph.D. graduates worldwide, in addition to in all disciplines/fields. Moreover, graduates of Ph.D. programs may complete their studies Edited by: Tom Rosman, in one country but then obtain a research position in another country, emphasizing Leibniz Center for Psychological the need to ensure that all doctoral programs are fostering similar levels of research Information and Documentation competence. Thus, the purpose of this study was to gather additional evidence for (ZPID), Germany validity and reliability of the Research Competence (R-Comp) scale. Specifically, we Reviewed by: Peter Adriaan Edelsbrunner, sought to extend the findings of by adapting the scale, translating it to other languages, ETH Zürich, Switzerland and applying the tool with a sample of early stage researchers. Our findings provide Hassan Mohebbi, European Knowledge Development initial evidence that the adapted PR-Comp is appropriate for use in three languages and Institute (EUROKD), Turkey across a variety of disciplines/programs of study. Jana Groß Ophoff, Pädagogische Hochschule Keywords: research competence, doctoral programs, measurement, confirmatory factor analysis (CFA), validity, Vorarlberg, Austria reliability, perceived competence *Correspondence: Sarah A. Marrs marrssa@vcu.edu MEASURING PERCEIVED RESEARCH COMPETENCE OF JUNIOR RESEARCHERS Specialty section: This article was submitted to The degree doctor of philosophy (Ph.D.) is considered a research intensive degree, designed Educational Psychology, to foster the development of independent researchers. Upon completion of a Ph.D. program, a section of the journal individuals are expected to be not only experts in their chosen field but competent in designing and Frontiers in Psychology conducting research independently following the rules of science (De Jong, 2021); this expectation Received: 13 December 2021 is shared across the globe (e.g., Hambrick, 1997; Pole, 2000; Trotter, 2003; Jackson, 2013; Pinto Accepted: 02 March 2022 et al., 2013; Poh and Kanesan Abdullah, 2019; De Jong, 2021; Fairman et al., 2021; MacLeod Published: 19 April 2022 and Urquiola, 2021). However, and despite conventional beliefs, doctoral study alone may not be Citation: adequately preparing students for performing their roles as independent researchers. Given the Marrs SA, Quesada-Pallarès C, level of research competence needed for graduates of doctoral programs – particularly for those Nicolai KD, Severson-Irby EA and who go on to become academic researchers – it is important to have a reliable and validated tool Martínez-Fernández JR (2022) Measuring Perceived Research for measuring and tracking perceived research competence among junior researchers (Fairman Competence of Junior Researchers. et al., 2021). Thus, the purpose of this study is to gather evidence for validity and reliability Front. Psychol. 13:834843. of the Perceived Research Competence (PR-Comp) scale, which could then be used to inform doi: 10.3389/fpsyg.2022.834843 interventions for fostering greater research competence among early stage researchers. Frontiers in Psychology | www.frontiersin.org 1 April 2022 | Volume 13 | Article 834843 Marrs et al. Measuring Perceived Research Competence Research Competence (R-Comp) questionnaire. The R-Comp, a self-report measure of According to self-determination theory (SDT; Deci and Ryan, one’s research competence, was designed to measure research 2000), motivation stems from an individuals’ need to satisfy competence across multiple disciplines. It was created in three basic psychological needs, one of which is competence. alignment with the RMRC-K model, which posits research Competence is thought to satisfy one’s psychological need to competence as comprising five dimensions: skills in Reviewing master personally challenging tasks (Deci and Ryan, 2000). If the state of research, Methodological skills, skills in Reflecting on the psychological need of competence is met, students may be research findings, Communication skills, and content Knowledge able to work more effectively and maintain greater well-being; (Thiel and Böttcher, 2014). Their resulting instrument consisted conversely, if competence is not met, students may show signs of five factors, one for each dimension of research competence. of negativity (Deci and Ryan, 1985; Sheldon and Elliot, 1999; Though the R-Comp was intended to measure competence Andersen, 2000). Perceived competence is one’s perception of across multiple disciplines, it was developed using a sample of both their basic capability of performing a task and a personal students enrolled in a science program at either the Bachelor’s judgment of the importance of the task. The importance of (27.4%), Master’s (68.5%), or doctoral level (4.1%) at a German perceived competence has also been demonstrated in areas of life university. The R-Comp was also developed and administered such as academic achievement (e.g., Losier and Vallerand, 1994, in German then translated to English only for publication. As 1995), work (e.g., Gagné and Deci, 2005; Arshadi, 2010), and such, more work is needed to examine the R-Comp’s utility sports (e.g., Vansteenkiste et al., 2004; Wilson et al., 2006). for measuring research competence among doctoral students Research specifically focused on perceived research and early career researchers broadly. Since the expectations for competence is limited both in quantity and in scope. Regarding research competence are similar across fields globally, there is graduate student and early career researchers’ perceived also a need to create a tool that reliably and validly measures competence, research often focuses on broader perceived perceived research competence across multiple disciplines and professional competency (Graesser, 2014; Latorre, 2020), clinical languages. As such, the purpose of our study was to gather competency (Dozier, 2001), or competency utilizing specific additional evidence for validity and reliability to support the assessments (Ingram et al., 2020) instead of competence carrying R-Comp’s intended use of measuring research competence across out research independently. From what we do know, it seems multiple disciplines. Specifically, and since the original items and that students who participate in research activities as part of their factor structure were created and established in German and training report higher levels of research competence, particularly translated to English for publication purposes only, we sought in the areas of data analysis and applying results to practice to replicate Böttcher and Thiel’s (2018) findings by collecting (Olehnovica et al., 2015). Jung (2018) recently investigated what data using the English-translated version of the questionnaire. To factors contributed to doctoral students’ perceived research begin exploring the appropriateness of using the R-Comp across competence. Research-oriented learning environments positively multiple languages, we also translated the items into two other influenced task-oriented (e.g., critical thinking and problem languages to gather initial evidence for validity and reliability to solving) and idea-oriented (e.g., innovation and creativity) support broader application of the instrument. Henceforth we research competencies. Notably, Jung (2018) found that refer to the questionnaire as the PR-Comp as we believe this name participating in manuscript preparation and dissertation writing best captures the intended purpose of the instrument. did not have a strong influence on the students’ perceived research competence. Findings from other studies suggest that the more students are exposed to various aspects of the research process, such designing and carrying out studies, MATERIALS AND METHODS performing literature searches, and publishing manuscripts, the more confident students are in their ability to do research Participants (Phillips and Russell, 1994; Lambie and Vaccaro, 2011; Lambie Current enrollees in or recent graduates of Ph.D. programs were et al., 2014; Petko et al., 2020). Particularly important for recruited to participate in this study. Participants (N = 456) were academic researchers, especially those entering the tenure-track, primarily female (62.7%) and ranged in age from 19 to 64 with perceived competence in one’s research abilities may also be the average age being 33.1 (SD = 8.09) years of age. Our sample linked to interest in conducting research, research productivity, represented 28 nationalities (see Table 1) and 118 disciplines. innovation, and creativity for both graduate students and early The vast majority of participants (n = 405) were still enrolled in career researchers (Olehnovica et al., 2015; Jung, 2018; Petko a Ph.D. program; these students’ current year of study ranged et al., 2020). Given the influence perceived competence may have from first (21.5%) to five or more (7.9%). Approximately 12% on both academic and non-academic outcomes, coupled with the of the sample were recent graduates/early career researchers. expectation that graduates of doctoral programs in all fields can Most students (n = 365) were enrolled in or completed their effectively conduct research independently, developing a valid doctoral program full-time though 20% were enrolled part-time. and reliable tool for measuring perceived research competence Participants reported an average of 5.10 (SD = 3.14) years of across disciplines and settings is needed. research experience, including their Ph.D. experience; research Recognizing the need for programs to be able to assess the experience ranged from not having any experience at all (n = 9) degree to which they were producing competent researchers, to as much as 22 years of research experience (n = 1). Lastly, Böttcher and Thiel (2018) developed the Research Competence slightly more than half (54.40%) of the sample had not attended Frontiers in Psychology | www.frontiersin.org 2 April 2022 | Volume 13 | Article 834843 Marrs et al. Measuring Perceived Research Competence TABLE 1 | Nationalities represented by language. three items: “I can confidently analyze quantitative data,” “I can confidently analyze qualitative data,” and “I can confidently use a Frequency by language variety of methods for analyzing data (Excel, specialized software, Nationality Catalan English Spanish etc.).” Items were also reviewed and edited for clarity when needed. For instance, the item “I am able to plan a research Argentinian 2 process” was revised to “I am able to plan a research study.” Belgian 2 The final list of items were reviewed by native speakers of each Brazilian 2 2 language as well as by two individuals who were fluent in each Bulgarian 1 of the three languages. The adapted 36-items of the PR-Comp are Canadian 1 presented side-by-side with the original R-Comp items in Table 2. Chilean 1 17 We retained both the 5-point Likert response scale (“Strongly Chinese 3 1 Disagree” to “Strongly Agree”) as well as the five proposed Colombian 3 8 subscales (Böttcher and Thiel, 2018; Böttcher-Oschmann et al., Costa Rican 1 2021). Ecuadorian 1 1 2 Using an electronic recruitment campaign, a non-probabilistic French 1 1 1 sample of students and recent graduates from doctoral programs German 1 1 1 in Mexico, Spain, and the United States were recruited Ghanian 1 to participate in the study. Students were recruited from Greek 2 programs in these countries since the authors themselves Indian 1 worked in or had connections to programs in these countries. Iranian 4 Additionally, students currently and recent graduates of doctoral Italian 6 programs were recruited to participate using social media Mexican 115 advertisements. Consenting participants who received the link Paraguayan 4 to the questionnaire either via email or by seeing the study Peruvian 1 1 advertisement on social media then completed the questionnaire South African 1 online, on their own time, in one sitting. Participants were South Korean 1 informed their participation was completely voluntary and Spanish 134 4 44 anonymous. After consenting, participants could choose to Turkish 1 complete the questionnaire in either Catalan (n = 141), English United States, American 76 1 (n = 111), or Spanish (n = 204). Data collection took place from Uruguayan 1 1 late fall of 2019 to summer of 2020. Venezuelan 2 Total 141 111 204* Data Analysis *The total frequency of those who took the survey in Spanish (i.e., 204) does not Data were screened for response patterns prior to analyses. equal the total nationalities reported. One respondent did not share their nationality. Descriptive analyses were conducted and both Cronbach’s alpha and McDonald’s omega were calculated to examine the internal any additional education or training on research methods outside consistency of the five subscales in each language. Once internal of what their Ph.D. program offered. consistency and normality within each language were established, we then conducted confirmatory factor analysis (CFA) using all Procedure data to confirm the five-factor structure of the PR-Comp and We began with a forward and back translation that would provide added evidence for validity based on internal structure. result in equivalent versions of the scale in three different Model fit was assessed using the following indices: comparative fit languages: Catalan, Spanish, and English. These three languages index (CFI) > 0.95 (Hu and Bentler, 1999); Tucker-Lewis index were chosen as the authors were (1) native speakers of these (TLI) > 0.95 (Hu and Bentler, 1999); root mean square error of languages and (2) worked and recruited in countries where these approximation (RMSEA) < 0.06 (Hu and Bentler, 1999) and its languages are predominantly spoken. The original R-Comp items 90% confidence interval whereby (Kline, 2016). These guidelines were translated from German to English solely for publication and suggestions for “cutoff ” points were used to inform overall purposes (see Thiel and Böttcher, 2014). Our process began first evaluation of model fit (Marsh et al., 2004). We also compared by translating the English version of the items into Catalan and AIC values of each model, with smaller AICs indicating better Spanish by two native Catalan and Spanish speakers, respectively. model fit (Schermelleh-Engel et al., 2003). The chi-square Items were translated back into English by a non-native English goodness of fit test was examined to test the null hypothesis speaker. Lastly, the items were revised by a native English speaker, that a model fitting the data exactly exists (Kline, 2016). Once at which point revisions and inconsistencies were discussed by we ensured factorial validity of the model, and with the idea of the whole group. Part of this process included ensuring that items testing invariance across languages, we conducted CFA on all were unidimensional (i.e., not double-barreled). For example, the three language-models (Catalan, English, and Spanish), following R-Comp item “I can confidently apply even complex methods the recommendations of Sass and Schmitt (2013). Assuming to analyze data/sources/material” was split into the following factorial validity of the three language-models was obtained, we Frontiers in Psychology | www.frontiersin.org 3 April 2022 | Volume 13 | Article 834843 Marrs et al. Measuring Perceived Research Competence TABLE 2 | The adapted PR-Comp items compared to original R-Comp items by subscale. RMRC-K model of research Original R-Comp instrument (Böttcher and Thiel, 2018) Adapted PR-Comp instrument competence (Thiel and Böttcher, 2014) Skills in reviewing the state of research Systematically reviewing the state I know how and where to target a search of the state of research I know how to conduct a targeted search of the state of of research regarding a specific topic. research on a specific topic. I know where to target a search of the state of research on a specific topic. I am able to systematically review the state of research regarding a I am able to systematically review the state of research specific topic. regarding a specific topic. Critically evaluating the state of Based on the state of research, I am able to identify Based on the state of research, I am able to identify research gaps/unaddressed questions for further research. gaps/unaddressed questions for further research. I can evaluate the methodological quality of researched findings I can evaluate the methodological quality of research well. findings well. Methodological skills Systematic planning and I find it difficult to formulate specific research questions/hypotheses. I find it difficult to formulate specific research preparation of the research process (a) questions/hypotheses. (a) I am able to decide, which data/sources/materials I need to I am able to decide which data/sources/materials I need to address my research question. address my research question. I am able to plan a research process. I am able to plan a research study. I find it difficult to operationalize each step of the research process. I find it difficult to start/initiate each step of the research (a) process. (a) Selection and application of I find it easy to decide, which methods I need to use to examine a I find it easy to decide which methods I need to use to methods specific research topic. address a specific research question. I am good at judging which method is inappropriate to answer a I am good at judging which method is inappropriate to specific research question. answer a specific research question. I can apply different research methods appropriate to my research I can apply different research methods appropriate to my question. research question. I can confidently apply even complex methods to analyze I can confidently analyze quantitative data data/sources/materials. I can confidently analyze qualitative data I can confidently use a variety of methods for analyzing data (excel, specialized software, etc.) Skills in reflecting on research findings Theoretically and methodologically I am able to adequately interpret my own research findings by I am able to adequately interpret my research findings. reflecting on results relating them to key theories in the subject area. I am able to adequately relate my research findings to key theories in the subject area. I am able to critically reflect on methodological limitations of my own I am able to critically reflect on methodological limitations of research findings. my own research findings. Reflecting on scientific and practical I am able to reflect on the implications of my own research findings I am able to reflect on the implications of my own research reach on my discipline. findings in my discipline. I am able to discuss my research findings with regard to their I am able to discuss my research findings with regard to potential applications. their potential applications. Reflecting on ethical implications I am able to critically reflect on the social/ethical implications of my I am able to critically reflect on the social and ethical research. implications of my research. I am able to take a stand on social/ethical issues of research in my I am able to take a stand on social and ethical issues of discipline. research in my discipline. Communication skills Writing academic publications I can write up research findings in accordance with the current I can write up research findings in accordance with the conventions in my discipline. current conventions in my discipline. I am able to write a publication in accordance with the standards of I am able to write a publication in accordance with the my discipline. standards of my discipline. I find it difficult to write a report that meets the standards of I find it difficult to write a report that meets the standards of academic writing. (a) academic writing. (a) Presentation of research findings I am able to prepare research findings for a presentation at a I am able to prepare research findings for a presentation at research colloquium. a research colloquium. I am able to present my research at a scientific meeting in I am able to present my research at a scientific meeting in accordance with current standards in my discipline. accordance with current standards in my discipline. (Continued) Frontiers in Psychology | www.frontiersin.org 4 April 2022 | Volume 13 | Article 834843 Marrs et al. Measuring Perceived Research Competence TABLE 2 | (Continued) RMRC-K model of research Original R-Comp instrument (Böttcher and Thiel, 2018) Adapted PR-Comp instrument competence (Thiel and Böttcher, 2014) Content knowledge Knowledge of central theories and I have a good overview of the main (current) research findings in my I have a good overview of the main (current) research current findings discipline. findings in my discipline. I am informed about the main (current) theories in my discipline. I am informed about the main (current) theories in my discipline. I am informed about the history of theory/paradigm shifts in my I am informed about the history of theory/paradigm shifts in discipline. my discipline. Knowledge of central research I have a sound knowledge of the main research methods in my I have a sound knowledge of the main research methods in methods discipline. my discipline. I would describe my methodological knowledge as sophisticated I would describe my methodological knowledge as and comprehensive. sophisticated and comprehensive. I am very familiar with different research methods in my subject I am very familiar with different research methods in my area. subject area. Knowledge of communication I am informed about the most important national and international I am informed about the most important national and standards in academic research academic publication outlets in my discipline. international academic publication outlets in my discipline. I am informed about the standards for academic publications that I am informed about the standards for academic apply in my discipline. publications that apply in my discipline. I am informed about the standards that apply to the presentation of I am informed about the standards that apply to the research findings at congresses and meetings in my subject area. presentation of research findings at conferences and meetings in my subject area. (a) reversed items. then evaluated the configural invariance (as a baseline model) and with most being above 0.80, indicating good internal consistency measurement invariance (both metric and scalar invariance) of among subscales within each of the three languages (see Table 4). the factor model by applying the forward approach (sequentially One item on the Communication Skills subscale displayed a low adding more model constraints). To evaluate the model fit, we scale-item correlation. Specifically, item 3 showed a low item- used Chen’s (2007) cutoff criteria: reject 1CFI < −0.01 and scale correlation (0.27); alpha would increase from 0.84 to 0.92 1RMSEA < 0.01. We also considered the need of obtaining non- if item 3 was deleted. Three items on the Methodological Skills significant X2 for the language-models. All data were analyzed subscale yielded moderate scale-item correlations; for each of using SPSS and AMOS version 23. these items, alpha would either increase marginally or not at all if these items were removed. McDonald’s omega was also calculated to ensure the stability of alphas in case one of the assumptions RESULTS were not met. Omegas tended to be higher than alphas, providing added evidence that subscales produced reliable scores. Prior to analyses, we examined descriptive statistics and assessed Confirmatory factor analyses were conducted to examine our data for missingness; our data were complete and as whether the proposed five-factor structure could be confirmed such, there were no missing data patterns identified. However, with our data. According to the chi-square goodness of fit those completing the PR-Comp in English did tend to have test, we reject the null hypothesis that a model that fits the higher scores than those completing the PR-Comp in Catalan data exactly exists; X2 (5) = 12.58, p = 0.03. To assess model- and Spanish; all descriptive statistics are provided in Table 3. data fit, we examined the RMSEA and its 90% confidence Subscale scores were normally distributed within each language. interval, the CFI, the TLI, and AICs of a five-factor model Cronbach’s alpha was calculated for each of the five subscales including all 36 PR-Comp items and a reduced five-factor with within each language. All alphas were equal to or greater than 0.75 items having low or moderate scale-item correlations removed. TABLE 3 | Descriptive statistics for PR-Comp subscale. PR-Comp subscale Catalan n = 141 Mean (SD) English n = 111 Mean (SD) Spanish n = 204 Mean (SD) Reviewing the state of research 3.53 (0.87) 3.85 (0.89) 3.64 (0.86) Methodological skills 3.17 (0.69) 3.62 (0.73) 3.42 (0.78) Skills in reflecting on research findings 3.61 (0.82) 3.85 (0.66) 3.66 (0.82) Communication skills 3.37 (0.75) 3.94 (0.85) 3.49 (0.88) Content knowledge 3.35 (0.87) 3.65 (0.68) 3.43 (0.85) Frontiers in Psychology | www.frontiersin.org 5 April 2022 | Volume 13 | Article 834843 Marrs et al. Measuring Perceived Research Competence TABLE 4 | Reliability analyses of the PR-Comp subscales and instrument. PR-Comp subscales Catalan English Spanish Combined Observations Reviewing the state of research (5 items) α = 0.88 α = 0.93 α = 0.92 α = 0.91 No observations ω = 0.88 ω = 0.93 ω = 0.92 ω = 0.91 Methodological skills (10 items) α = 0.81 α = 0.89 α = 0.91 α = 0.87 Items 1, 4, and 9 showed moderate item-scale ω = 0.84 ω = 0.90 ω = 0.91 ω = 0.89 correlations (0.39,0.44, and 0.42); alpha would increase to 0.88 if item 1 was deleted and remain 0.87 if items 4 and 9 were deleted Skills in reflecting on research findings (7 items) α = 0.91 α = 0.87 α = 0.93 α = 0.91 No observations ω = 0.91 ω = 0.88 ω = 0.94 ω = 0.92 Communication skills (5 items) α = 0.75 α = 0.89 α = 0.88 α = 0.84 Item 3 showed a low item-scale correlation ω = 0.84 ω = 0.89 ω = 0.90 ω = 0.88 (0.27); alpha would increase to 0.92 if item 3 was deleted Content knowledge (9 items) α = 0.93 α = 0.91 α = 0.94 α = 0.93 No observations ω = 0.93 ω = 0.91 ω = 0.95 ω = 0.93 GLOBAL (36 items) α = 0.96 α = 0.96 α = 0.97 α = 0.97 ω = 0.97 ω = 0.96 ω = 0.97 ω = 0.97 α = Cronbach’s alpha; ω = McDonald’s omega. TABLE 5 | Model-fit indices for PR-Comp full versus reduced model. Model n Number of freely X2 (df), p Root mean RMSEA 90% Comparative Tucker-lewis Akaike estimated square error of Confidence interval fit index index information parameters approximation criterion Full five-factor 456 15 X2 (5) = 12.58, 0.06 90% CI [0.02–0.10] 0.99 0.99 42.58 model p = 0.03 Reduced five-factor 456 15 X2 (5) = 16.72, 0.07 90% CI [0.04–0.11] 0.99 0.99 46.72 model p = 0.01 Deleting items with low moderate scale-item correlations did not Model invariance across languages was first evaluated by result in improved model fit. The full model resulted in a more conducting factorial validity of the model in each language. As favorable RMSEA value and a marginally more favorable CFI seen in Table 6, model-fit indices were not adequate for any of and TLI values as well as a lower AIC. However, the confidence the language-models (M1–M3). Indeed, Hoelter’s value at 0.05 intervals for the RMSEA provided weaker evidence of good of significance, indicated that M2 and M3 (English and Spanish, model fit. Taking all of these indices together, the five-factor respectively) did not obtain the minimum sample size needed model including all 36 items was championed (see Table 5 and according to the complexity of the model, which should be above Figures 1, 2). 200 (Garson, 2015). Thus, we were not able to guarantee factorial FIGURE 1 | Five-factor model of 36-item PR-comp scale. Frontiers in Psychology | www.frontiersin.org 6 April 2022 | Volume 13 | Article 834843 Marrs et al. Measuring Perceived Research Competence FIGURE 2 | Five-factor model of reduced PR-comp scale. TABLE 6 | Model-fit indices for PR-Comp language-models. Model X2 (df), p Root mean square RMSEA 90% Comparative fix Tucker-lewis AIC Hoelter error of Confidence interval index index p = 0.05 approximation M1. Catalan (n = 141) X2 (5) = 1.62, p = 0.898 0.00 0.00–0.05 1.00 1.01 31.63 952 M2. English (n = 111) X2 (5) = 6.70, p = 0.244 0.06 0.00–0.15 0.99 0.99 36.70 182 M3. Spanish (n = 204) X2 (5) = 21.33, p = 0.001 0.127 0.08–0.184 0.98 0.96 51.33 106 TABLE 7 | Measurement invariance across PR-Comp language-models. Model (M) X2 df p CFI Root mean square error 1 X2a 1df p 1Comparative fix indexa 1RMSEAa of approximation M4. Configural invariance 29.63 15 0.013 0.99 0.05 M5. Metric invariance 56.28 23 0.000 0.98 0.06 26.65 8 0.013 −0.02 0.01 M6. Structural invariance 105.31 33 0.000 0.95 0.07 49.03 10 0.000 −0.03 0.01 a Indicates comparisons to the previous model, M5 with M4 and M6 with M5. validity as a first step for model invariance. Nonetheless, and fields of study in addition to confirming the five-factor structure with the idea of exploring the results of evaluating measurement underlying the items, as proposed by Böttcher and Thiel (2018). invariance, we continued with the analyses as if model-fit indices The results of our CFA were consistent with that of Böttcher were accepted. As expected, Table 7 showed that there was and Thiel (2018) and our analysis confirmed that a five-factor configural non-invariance (unconstrained baseline model) as well structure fits the data well. This provided validity evidence based as metric and structural non-invariance. It can be observed on the internal structure of the scale as well as supported the when looking at chi-square significance and the cutoff criteria scale’s theoretical alignment with the RMRC-K model of research suggested by Chen (2007) at CFI and RMSEA values. Thus, these competence (Thiel and Böttcher, 2014). Furthermore, our results results suggested that the data collected was not enough to ensure provided evidence for reliability of the scores produced by the model invariance among languages. PR-Comp both globally and within each language. All alphas were quite high and would either increase marginally or not at all with the deletion of any items. As previously mentioned, DISCUSSION one item, item 3 on the Communication Skills subscale, did appear problematic and would result in a notable increase in Our study aimed to further validate Böttcher and Thiel’s alpha if it were removed from the scale. Furthermore, this (2018) questionnaire as well as gather evidence for validity and finding was consistent across languages, suggesting this item reliability to support the use of this questionnaire across multiple should be examined further for clarity and relation to other disciplines and languages. Our findings provided added evidence variables in the scale. McDonald’s omega values were also that the PR-Comp can be applied across various disciplines and calculated for each subscale score and tended to be higher Frontiers in Psychology | www.frontiersin.org 7 April 2022 | Volume 13 | Article 834843 Marrs et al. Measuring Perceived Research Competence than alphas. Taken together, both alphas and omegas suggest original items were created in German and then translated for internal consistency of the subscale items is quite high. However, publication, including data from a German-speaking sample there are important limitations to our study that limit the would also be beneficial. generalizability of our findings. Having a tool for measuring research competence that is Our findings do provide some initial evidence for external appropriate across settings and languages that also produces validity as it appears using the PR-Comp scale may be appropriate reliable scores has implications for both practice and research. for use in and produces reliable scores when applied across Regarding practice, doctoral program faculty and staff could use countries, languages, and fields of study. However, due to the this tool to measure and track their students’ and graduates’ small sample size within each language group, we are not perceived research competence. For example, PR-Comp scores yet able to definitively demonstrate measurement invariance could be used to identify areas of strength and weakness of a across languages. Future studies should seek to gather added program which could then inform intervention efforts to boost evidence for external validity of the instrument. Another research competence in a more targeted way. A tool that can limitation to the broad applicability of our findings is that be completed by students would not only be a more efficient they are based on responses to a self-reported assessment means of collecting this data but would enable programs to track of one’s own competence, which may limit the accuracy of individual and cohort student growth over time. This tool could responses. Participants may have responded more favorably also be used as a means for assessing readiness for doctoral to make it seem as though they were more competent than study since a basic understanding of some research concepts is their true competence. Findings might be more accurate if needed prior to beginning doctoral study. Regarding research, scores represented another person’s assessment of an individual’s the PR-Comp scale is a tool that appears to produce stable and research competence rather than one’s own judgment of their generalizable findings that apply to the broader population of competence. For instance, doctoral program faculty and/or early career researchers and doctoral students. This could lead instructors’ ratings of individual students’ competence on us to greater understanding of the global landscape of research each of the PR-Comp items may provide a more objective competence of graduates of Ph.D. programs and, ultimately, rating of competence. Moreover, correlating instructors’ ratings support the global efforts of doctoral programs and their quest of research competence with students’ ratings of research to train competent researchers (e.g., Hambrick, 1997; Pole, 2000; competence could provide convergent validity evidence of PR- Trotter, 2003; Jackson, 2013; Pinto et al., 2013; Poh and Kanesan comp scores. As research competence and research self-efficacy Abdullah, 2019; Fairman et al., 2021; MacLeod and Urquiola, are often interchangeably used, future researchers could also 2021). consider using scores on a research self-efficacy measure as a means of gathering discriminant validity evidence for PR- Comp scores. This information would be useful to broaden DATA AVAILABILITY STATEMENT our understanding of the theoretical differences between these two constructs. Though the PR-Comp items were drafted and The raw data supporting the conclusions of this article will be edited by individuals having a Ph.D., it would be beneficial to made available by the authors, without undue reservation. further evidence for validity based on test content by having doctoral faculty from various fields review the items. This would ensure that items are relevant to all graduates of doctoral ETHICS STATEMENT programs. Another limitation to note is that our sample was recruited from doctoral programs and/or countries to which The studies involving human participants were reviewed and the authors had ties or in which the authors worked. While approved by Virginia Commonwealth University Institutional steps were taken to gather data from a large sample that Review Board (IRB). The patients/participants provided their represented various fields and countries as well as languages written informed consent to participate in this study. spoken, our findings might not represent all early career researchers across the globe. Similarly, the majority of our sample were still enrolled in their doctoral programs, meaning our AUTHOR CONTRIBUTIONS findings are less generalizable to early career researchers and recent graduates of doctoral programs. Finally, future studies CQ-P and SM were the leaders of this project and contributed should consider investigating measurement invariance (Davidov most to the project by designing the study and taking the lead et al., 2012) to ensure that we are measuring similar constructs on manuscript preparation. KN and ES-I contributed to data within each language group. While our primary purpose was collection efforts and drafted the initial introduction of the to expand evidence for validity and reliability of the PR- study. JM-F supported data collection and reviewed drafts of the Comp’s original purpose, more work is needed to be sure the manuscript. All authors contributed to data collection efforts and scale captures the same construct across languages. Since the manuscript preparation. Frontiers in Psychology | www.frontiersin.org 8 April 2022 | Volume 13 | Article 834843 Marrs et al. Measuring Perceived Research Competence REFERENCES Lambie, G. W., and Vaccaro, N. (2011). Doctoral counselor education students’ levels of research self-efficacy, perceptions of the research training environment, Andersen, S. (2000). Fundamental human needs: making social cognition relevant. and interest in research. Couns. Educ. Superv. 50, 243–258. doi: 10.1002/j.1556- Psychol. Inq. 11, 269–276. doi: 10.1207/S15327965PLI1104_02 6978.2011.tb00122.x Arshadi, N. (2010). Basic need satisfaction, work motivation, and Latorre, C. (2020). The Importance of Mindfulness and Self-Compassion in Clinical job performance in an industrial company in Iran. Procedia Training: Outcomes Related to Self-Assessed Competency and Self-Efficacy in Soc. Behav. Sci. 5, 1267–1272. doi: 10.1016/j.sbspro.2010.0 Psychologists-in-Training. Ph.D. thesis. Morgantown: West Virginia University. 7.273 Losier, G. A., and Vallerand, R. J. (1994). The temporal relationship between Böttcher, F., and Thiel, F. (2018). Evaluating research-oriented teaching: a new perceived competence and self-determined motivation. J. Soc. Psychol. 134, instrument to assess university students’ research competences. High. Educ. 75, 793–801. doi: 10.1080/00224545.1994.9923014 91–110. doi: 10.1007/s10734-017-0128-y Losier, G. F., and Vallerand, R. J. (1995). The development and validity Böttcher-Oschmann, F., Groß Ophoff, J., and Thiel, F. (2021). Preparing teacher of an interpersonal relations scale in sport. Int. J. Sport Psychol. 26, training students for evidence-based practice promoting students’ research 307–326. competencies in research-learning projects. Front. Educ. 6:642107. doi: 10.3389/ MacLeod, W. B., and Urquiola, M. (2021). Why does the United States have the feduc.2021.642107 best research universities? Incentives, resources, and virtuous circles. J. Econ. Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of Perspect. 35, 185–206. doi: 10.1257/jep.35.1.185 measurement invariance. Struct. Equ. Modeling 14, 464–504. doi: 10.1080/ Marsh, H. W., Hau, K. T., and Wen, Z. (2004). In search of golder 10705510701301834 rules: comment on hypothesis-testing approaches to setting cutoff values Davidov, E., Dülmer, H., Schlüter, E., Schmidt, P., and Meuleman, B. (2012). Using for fit indexes and dangers to overgeneralizing Hu and Bentler’s (1999) a multilevel structural equation modeling approach to explain cross-cultural findings. Struct. Equ. Modeling 11, 320–341. doi: 10.1207/s15328007sem1 measurement noninvariance. J. Cross Cult. Psychol. 43, 558–575. doi: 10.1177/ 103_2 0022022112438397 Olehnovica, E., Bolgzda, I., and Kravale-Paulina, , M. (2015). Individual potential De Jong, T. (ed.) (2021). Graduating around the globe: Protocols, principles, of doctoral students: structure of research competences and self-assessment. and traditions for PhD graduations - examples from the learning sciences. Procedia Soc. Behav. Sci. 174, 3557–3564. doi: 10.1016/j.sbspro.2015.01. Available online at: https://issuu.com/utwente/docs/graduating_around_the_ 1072 globe (accessed December 8, 2021). Petko, J., Sivo, S., and Lambie, G. (2020). The research self-efficacy, Deci, E. L., and Ryan, R. M. (1985). Intrinsic Motivation and Self-Determination in interest in research, and research mentoring experiences of doctoral Human Behavior. Chicago: Plenum Press. students in counselor education. J. Couns. Preparation Superv. 13, Deci, E. L., and Ryan, R. M. (2000). The ‘what’ and ‘why’ of goal pursuits: human 1–27. needs and self-determination of behavior. Psychol. Inq. 11, 227–268. doi: 10. Phillips, J. C., and Russell, R. K. (1994). Research self-efficacy, the research training 1207/S15327965PLI1104_01 environment, and research productivity among graduate students in counseling Dozier, S. L. (2001). On Becoming a Therapist: An Examination of Psychology psychology. Couns. Psychol. 22, 628–641. doi: 10.1177/001100009422 Doctoral Student’s Satisfaction With Training, Affective State, Functioning and 4008 Perceived Clinical Competence. Ph.D. thesis. Chicago: The Chicago School of Pinto, M., Fernández-Ramos, A., Sánchez, G., and Meneses, Professional Psychology. G. (2013). Information competence of doctoral students in Fairman, J. A., Giordano, N. A., McCauley, K., and Villarruel, A. (2021). information science in Spain and Latin America: a self-assessment. Invitational summit: re-envisioning research focused PHD programs of the J. Acad. Librariansh. 39, 144–154. doi: 10.1016/j.acalib.2012. future. J. Prof. Nurs. 37, 221–227. doi: 10.1016/j.profnurs.2020.09.004 08.006 Gagné, M., and Deci, E. L. (2005). Self-determination theory and work motivation. Poh, R., and Kanesan Abdullah, A. G. B. (2019). Factors influencing students’ J. Organ. Behav. 26, 331–362. doi: 10.1002/job.322 research self-efficacy: a case study of university students in Malaysia. Eurasian Garson, G. D. (2015). Structural Equation Modeling. Sunderland: Statistical J. Educ. Res. 82, 137–168. doi: 10.14689/ejer.2019.82.8 Associates Publishers. Pole, C. (2000). Technicians and scholars in pursuit of the PhD: some reflections Graesser, E. J. (2014). Serving Clients with Intellectual Disabilities: Clinical on doctoral study. Res. Pap. Educ. 15, 95–111. doi: 10.1080/0267152003 Psychology Training in APA-Accredited Doctoral Programs. Ph.D. thesis. New 62961 England: Antioch University. Sass, D. A., and Schmitt, T. A. (2013). “Testing measurement and structural Hambrick, R. (1997). The identity, purpose, and future of doctoral education. invariance: Implications for practice,” in Handbook of Quantitative Methods for J. Public Adm. Educ. 3, 133–148. doi: 10.1080/10877789.1997.12023423 Educational Research, ed. T. Timothy (Rotterdam: SensePublishers), 315–345. Hu, L., and Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance doi: 10.1007/978-94-6209-404-8_15 structure analysis: conventional criteria versus new alternatives. Struct. Equ. Schermelleh-Engel, K., Moosbrugger, H., and Muller, H. (2003). Evaluating the fit Modeling 6, 1–55. doi: 10.1080/10705519909540118 of structural equation models: tests of significance and descriptive goodness-of- Ingram, P. B., Schmidt, A. T., Bergquist, B. K., and Currin, J. M. (2020). fit measures. Methods Psychol. Res. Online 8, 23–74. Coursework, instrument exposure, and perceived competence in psychological Sheldon, K. M., and Elliot, A. J. (1999). Goal striving, need satisfaction, and assessment: a national survey of practices and beliefs of health service longitudinal well-being: the self-concordance model. J. Pers. Soc. Psychol. 76, psychology trainees. Train. Educ. Prof. Psychol. 16, 10–19. doi: 10.1037/ 482–497. doi: 10.1037/0022-3514.76.3.482 tep0000348 Thiel, F., and Böttcher, F. (2014). “Modellierung fächerübergreifender Jackson, D. (2013). Completing a PhD by publication: a review of Australian Forschungskompetenzen. Das RMKR-W-Modell als Grundlage der Planung policy and implications for practice. High. Educ. Res. Dev. 32, 355–368. doi: und Evaluation von Formaten forschungsorientierter Lehre,” in Neues 10.1080/07294360.2012.692666 Handbuch Hochschullehre. Lehren und Lernen effizient gestalten. [Teil] Jung, J. (2018). Learning experience and perceived competencies of doctoral I. Evaluation. Fachbereichs- /Studiengangsevaluation, eds B. Berendt, A. students in Hong Kong. Asia Pac. Educ. Rev. 19, 187–198. doi: 10.1007/s12564- Fleischmann, J. Wildt, N. Schaper, and B. Szczyrba (Berlin: Raabe), 109–124. 018-9530-0 Trotter, J. (2003). Researching, studying or jumping through hoops? Kline, R. B. (2016). Principles and Practice of Structural Equation Modeling, 4th Reflections on a PhD. Soc. Work Educ. 22, 59–70. doi: 10.1080/026154703 Edn. New York: The Guilford Press. 09132 Lambie, G. W., Hayes, B. G., Griffith, C., Limberg, D., and Mullen, P. R. (2014). Vansteenkiste, M., Simons, J., Soenens, B., and Lens, W. (2004). How to become a An exploratory investigation of the research self-efficacy, interest in research, persevering exerciser? Providing a clear, future intrinsic goal in an autonomy- and research knowledge of Ph.D. in education students. Innov. High. Educ. 39, supportive way. J. Sport Exerc. Psychol. 26, 232–249. doi: 10.1123/jsep.26. 139–153. doi: 10.1007/s10755-013-9264-1 2.232 Frontiers in Psychology | www.frontiersin.org 9 April 2022 | Volume 13 | Article 834843 Marrs et al. Measuring Perceived Research Competence Wilson, P. M., Longley, K., Muon, S., Rogers, W. T., Rodgers, W. M., and Wild, this article, or claim that may be made by its manufacturer, is not guaranteed or T. C. (2006). The psychological need satisfaction in exercise scale. J. Sport Exerc. endorsed by the publisher. Psychol. 28, 231–251. doi: 10.1123/jsep.28.3.231 Copyright © 2022 Marrs, Quesada-Pallarès, Nicolai, Severson-Irby and Martínez- Conflict of Interest: The authors declare that the research was conducted in the Fernández. This is an open-access article distributed under the terms of the absence of any commercial or financial relationships that could be construed as a Creative Commons Attribution License (CC BY). The use, distribution or potential conflict of interest. reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication Publisher’s Note: All claims expressed in this article are solely those of the authors in this journal is cited, in accordance with accepted academic practice. No and do not necessarily represent those of their affiliated organizations, or those of use, distribution or reproduction is permitted which does not comply with the publisher, the editors and the reviewers. Any product that may be evaluated in these terms. Frontiers in Psychology | www.frontiersin.org 10 April 2022 | Volume 13 | Article 834843