López‑del‑Hoyo et al. BMC Nursing (2022) 21:308 https://doi.org/10.1186/s12912-022-01089-5 STUDY PROTOCOL Open Access Effectiveness and implementation of an online intervention (MINDxYOU) for reducing stress and promote mental health among healthcare workers in Spain: a study protocol for a stepped‑wedge cluster randomized trial Yolanda López‑del‑Hoyo1,2,3, Selene Fernández‑Martínez1, Adrián Pérez‑Aranda1,4* , Alberto Barceló‑Soler1,5*, Jose Guzman‑Parra6,7, Esperanza Varela‑Moreno6,7,8, Daniel Campos1,2, Alicia Monreal‑Bartolomé1,3, María Beltrán‑Ruiz1,3, Berta Moreno‑Küstner7,8,9, Fermín Mayoral‑Cleries6,7 and Javier García‑Campayo1,3,10 Abstract Background: The World Health Organization has formally recognized that healthcare professionals are at risk of developing mental health problems; finding ways to reduce their stress is mandatory to improve both their quality of life and, indirectly, their job performance. In recent years, particularly since the COVID-19 pandemic outbreak, there has been a proliferation of online interventions with promising results. The purpose of the present study is twofold: to test the effectiveness of an online, self-guided intervention, MINDxYOU, to reduce the stress levels of healthcare workers; and to conduct an implementation study of this intervention. Additionally, an economic evaluation of the intervention will be conducted. Methods: The current study has a hybrid effectiveness-implementation type 2 design. A stepped wedge cluster randomized trial design will be used, with a cohort of 180 healthcare workers recruited in two Spanish provinces (Malaga and Zaragoza). The recruitment stage will commence in October 2022. Frontline health workers who provide direct care to people in a hospital, primary care center, or nursing home setting in both regions will participate. The effectiveness of the intervention will be studied, with perceived stress as the main outcome (Perceived Stress Scale), while other psychopathological symptoms and process variables (e.g., mindfulness, compassion, resilience, and psychological flexibility) will be also assessed as secondary outcomes. The implementation study will include analysis of feasibility, acceptability, adoption, appropriateness, fidelity, penetration, and sustainability. The incremental costs and benefits, in terms of quality-adjusted life years, will be examined by means of cost-utility and cost-effectiveness analyses. *Correspondence: adrian.peara@gmail.com; abarcelosoler@hotmail.com 1 Institute of Health Research of Aragon (IIS Aragón), Miguel Servet University Hospital, Saragossa, Spain Full list of author information is available at the end of the article © The Author(s) 2022. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://​creat​iveco​mmons.​org/​licen​ses/​by/4.​0/. The Creative Commons Public Domain Dedication waiver (http://​creat​iveco​ mmons.​org/​publi​cdoma​in/​zero/1.​0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. López‑del‑Hoyo et al. BMC Nursing (2022) 21:308 Page 2 of 12 Discussion: MINDxYOU is designed to reduce healthcare workers’ stress levels through the practice of mindfulness, acceptance, and compassion, with a special focus on how to apply these skills to healthy habits and considering the particular stressors that these professionals face on a daily basis. The present study will show how implementation studies are useful for establishing the framework in which to address barriers to and promote facilitators for accept‑ ability, appropriateness, adoption, feasibility, fidelity, penetration, and sustainability of online interventions. The ultimate goal is to reduce the research-to-practice gap. Trial registration: This study was registered in ClinicalTrials.gov on 29/06/2022; registration number: NCT05436717. Keywords: Healthcare workers, Stress, EHealth, Mindfulness, Compassion, Acceptance, Implementation, Cost-utility, Cluster randomized trial Background symptoms [11–15], including stress in healthcare pro- Although stress levels among healthcare workers are nor- fessionals [16–21]. The most common format for the mally high, they have been accentuated by the COVID- application of these psychotherapies has been face-to- 19 pandemic. Over the last two years, healthcare workers face, both in group and individual therapy. However, for have had to deal with the same pandemic-related stress- a few years, and particularly because of the pandemic, ors as any other citizens, in addition to their specific job- online psychotherapeutic programs have proliferated, related stressors: continued exposure to infection, longer making it possible to overcome obstacles to face-to-face shifts––which has led to inevitable physical exhaustion attendance. They are very well suited to the needs of and neglect of important aspects related to a healthy life- healthcare workers with schedules that are often diffi- style––and difficult decisions regarding how to prioritize cult to reconcile [22]. available resources in order to treat their patients, among There are a good number of online interventions (e.g., others [1, 2]. Subjection to this sustained pressure, which web-based, mobile applications, videoconference-deliv- is added to the normally high care burden of healthcare ered interventions, among others) aimed at treating workers, entails a risk factor for developing mental health stress, anxiety, depression, and insomnia, among others. problems, including anxiety disorders, depression, soma- Different studies have been published on the efficacy of toform disorders, post-traumatic stress disorders, and online interventions to improve the quality of life and even suicidal ideation [3, 4]. reduce the stress of healthcare workers. Some have veri- The World Health Organization (WHO) has formally fied that online CBT-based interventions are effective for recognized that healthcare professionals are at risk of reducing stress, suicidal ideation, or emotional distress in developing psychopathological symptoms, particularly this population [20, 23–26], and others have also studied those working in public health, primary care, emergency the efficacy of “third wave” online psychotherapies with departments, and intensive care units [5, 6], including positive results [27–30]. However, most of these pro- personnel employed by nursing homes [7]. It is essential grams have only achieved small effect sizes in terms of to take care of the mental health of healthcare workers, their effectiveness, and a fact that cannot be overlooked not only to improve their quality of life, but also because is that studies have also been published regarding online the success of their work largely depends on it. Hence, it interventions found to be ineffective for reducing stress is especially necessary in the current circumstances to of healthcare workers, which makes further study into offer these workers effective and highly adaptive strate- the characteristics of these programs and their possible gies that will enable them to cope with the numerous limitations necessary. Moreover, the interventions that stressors to which they are exposed in order to prevent are tested in effectiveness studies are mostly discontin- the appearance of mental health problems, also in the ued once the study ends, which poses a major problem long term. when it comes to the translation of efficacy studies on Many psychotherapeutic programs have been devel- evidence-based interventions to daily clinical practice, a oped to combat stress. Interventions based on cogni- period estimated to be 17–20 years [31]. tive behavioral therapy (CBT) have proven effective for reducing stress, anxiety, and depression in different clin- Methods ical populations [8–10]. Similarly, so-called “third wave” Study aims psychotherapies, such as mindfulness-based interven- This study aims to evaluate the effectiveness of a self- tions (MBIs), compassion-based programs, and accept- guided online program, MINDxYOU, based on “third ance and commitment therapy (ACT), have also proved wave” psychotherapy principles, for reducing the stress effective for reducing different psychopathological levels of healthcare workers through the practice of López‑del‑Hoyo et al. BMC Nursing (2022) 21:308 Page 3 of 12 different exercises that promote mindfulness, acceptance, be randomly assigned to one of them. The participants and self-compassion, in addition to indications on how from each site will all follow the same sequence. to apply these skills to a healthy lifestyle. Furthermore, SW designs are particularly recommended when the this project intends to emphasize the implementation of intervention being tested has strong evidence of posi- the intervention; implementation science is a discipline tive effects or is very unlikely to cause harm [35], which that is developing methods and procedures to ensure is the case of our study. The present work is considered that evidence-based interventions are incorporated more a closed cohort study, since all the study participants quickly into daily clinical practice [32]. The implementa- will start at the same time and will be evaluated at dif- tion study includes 3 types of evaluation: 1) evaluation of ferent time-points to be defined beforehand; the inclu- the process (characteristics of the use of the intervention; sion of new participants will not be allowed once the measures before, during, and after implementation); 2) trial has commenced. formative evaluation (adaptations for improvement based on participants’ feedback); and 3) summative evaluation (a compilation of the impact of the implementation strat- Participants egy, evaluating the impact on indicators such as increase The study sample will comprise healthcare workers cur- in the use of the intervention in the target context and rently working in clinical settings: a hospital, a primary economic impact). care (PC) center, and a nursing home in the provinces As secondary aims, the impact of the intervention on of Zaragoza (Aragon) and Malaga (Andalusia), Spain. other health-related outcomes such as psychopathologi- The following specific inclusion criteria will apply: 1) cal symptoms (depression, anxiety, and others) and pro- employment as a physician, nurse, physiotherapist, cess variables (resilience, mindfulness, compassion, and psychologist, nursing assistant, ambulance technician; psychological flexibility) will also be evaluated, and an a trainee student in any health profession; or employ- economic evaluation will be conducted. ment in a nursing home assisting patients; 2) aged between 18 and 70; 3) ability to understand Spanish; 4) Study design digital literacy and access to a smartphone, tablet, or An effectiveness-implementation type 2 design will be personal computer with Internet connection; 5) pros- used [33], since this study will assess both the effec- pects of continued employment at the same site for tiveness of the intervention and the implementation the following 6 months; and 6) provision of informed process in the same degree, as co-primary aims [34]. consent. The empirical study will take a stepped wedge (SW) The following exclusion criteria will apply: 1) present- cluster randomized controlled trial design; this design ing with a disorder that affects the central nervous sys- implies that every participant starts the trial in the con- tem; 2) diagnosis of a severe mental illness (including trol phase (no exposure to the intervention), and then severe depressive disorders, suicidal tendencies, bipo- undertakes the intervention in a staggered and sequen- lar disorders, panic disorders, anxiety or stress-related tial way. Once it is completed (8 weeks), the mainte- disorders, obsessive–compulsive disorders, and sub- nance phase takes place, during which the participant stance-related disorders); 3) presenting with a medical, can continue to use the program at will, but without infectious, or degenerative illness that is not under con- supervision. There will be three possible sequences trol; and 4) expertise in “third wave” psychotherapies (see Fig. 1), and each of the participating centers will (i.e., meditation, mindfulness, etc.). Fig. 1 The three possible sequences of the study phases. Note: A = Control phase; B = Intervention phase; C = Maintenance phase. Each cell represents a period of 8 weeks López‑del‑Hoyo et al. BMC Nursing (2022) 21:308 Page 4 of 12 Sample size Informed consent will then be documented via elec- The sample size was calculated following the recom- tronic signature, and complying candidates will undergo mendations for SW designs [36]. First, the sample size a basic screening process, which will consist of a check- for individual randomization needed to be determined; list of the inclusion criteria. Candidates who meet these in this case, with an expected effect size of d = 0.50 for self-reported criteria will be presented with a schedule the primary outcome (Perceived Stress Scale, PSS) [37], for the following 2 weeks and they will be asked to select which is a continuous variable, a statistical power of a time for a 20-min video call or telephone call from a 80%, and an alpha level of 0.050, the required sample size member of the research team who will conduct the com- would be 126 participants ­(N1). Subsequently, the follow- plete screening of the participant, including the assess- ing formula was considered: ment of psychiatric diagnoses (MINI v7.0.2). Candidates who meet all the criteria will formally be included as N1(1 − p) m= participants in the study and will be offered information K − N1p regarding the following procedures, which will start with where m is the necessary sample for each cluster and a baseline evaluation including different self-reported for each assessment moment (variable to be determined), questionnaires that the participant will complete using ­N1 is the already calculated sample size (126), p is the an online platform. The participant will receive via email intra-class correlation coefficient (assumed: 0.04), and K or text message the link to the evaluation platform. Once is the number of clusters (6). The result would be 24 par- the control phase is over, the participant will receive via ticipants per cluster, and assuming 20% attrition, 30 par- email or text message their credentials (i.e., user and ticipants should be recruited from each site. This would password) to access to the online intervention platform imply a total sample of 180 participants. so they can start the MINDxYOU program. An exter- nal researcher, not related to this study, will conduct Procedure the randomization process to one of the three possible The strategy to recruit study participants will commence sequences using EPIDAT 4.2 (see Fig. 1). in October 2022. It will include indirect techniques Clear indications will be given on how to use the plat- such as informative posters in the selected hospitals/PC form in the welcome module (see Table 1). Participants centers/nursing homes containing a brief explanation will be assessed for all the study variables every 8 weeks; of the study and the website link; in addition, the occu- the usability measure (System Usability Scale, SUS) [38] pational health service of the hospital and a member of will be completed after finishing module 1 and mod- staff at each site will contact their peers and send them ule 4. The research team will record the usage that each an informative email with a brief explanation of the study participant makes of the platform: the number of times and the website link. Consequently, any individuals inter- it was accessed, modules completed, and number of ested in taking part in the study will find on the website tasks delivered. Two external professionals, one in each all the details regarding the study procedures and aims, province and unrelated to the research group, will be in together with the rights of every participant and an expla- charge of monitoring the intervention, providing feed- nation of the way in which their data will be processed. back to the participants, and making weekly contact Table 1 Brief outline of the MINDxYOU program Module Main contents Main practices 0. Welcome module This module will introduce participants to the platform and Participants will be given the opportunity to share their feelings present the main aims of the intervention and needs, and will receive feedback from the research team 1. Mindfulness The concept of mindfulness will be explained, together with Body scan, sitting meditation, three-minute breathing medita‑ its applications for stress reduction and the promotion of good tion mental health 2. Compassion The concept of compassion will be explained, together with Compassionate breath meditation, compassionate body scan, its applications for stress reduction and the promotion of good safe place meditation mental health 3. Acceptance The concept of acceptance will be explained, together with its Meditation focused on acceptance, identification of personal applications for stress reduction and the promotion of good values mental health 4. Healthy lifestyle Instructions on how to apply the concepts of mindfulness, Mindfulness and compassion exercises applied to physical activ‑ compassion, and acceptance to the main aspects of a healthy ity, diet, sleep, and socialization lifestyle: physical activity, diet, sleep, and socialization López‑del‑Hoyo et al. BMC Nursing (2022) 21:308 Page 5 of 12 with participants via their preferred means (telephone questionnaire(s) they need to complete; nor will it be call, text message, or email); this contact will be used to used for counseling purposes. motivate participants to complete the intervention. Both The intervention begins with a welcome module that professionals will previously have been trained in the introduces the main aims and procedures of the plat- contents and procedures of the MINDxYOU program form. Participants are invited to share a written testi- by the research team. Once the intervention is complete, mony of their main stressors in their workplace, and their focus groups will be held in the provinces of Zaragoza feelings and personal needs. The external professional and Malaga to qualitatively assess the acceptance and will read each testimony and respond to each one by suitability of the intervention, and interviews will be con- offering a rationale for why the intervention can be ben- ducted with managers and directors from each site to eficial for their case, enhancing motivation to complete evaluate different aspects regarding implementation of the intervention. The testimonies will be analyzed and the program. used to improve the contents of the intervention wher- ever possible. Intervention After the welcome module, the program will be struc- The MINDxYOU online program is based on the prin- tured into 4 modules, each dedicated to a specific topic ciples of “third wave” psychotherapies, such as the pro- (see Table 1). Every module will contain animated vid- motion of wellbeing through the practice of mindfulness, eos with explanations of the aims and contents of the compassion, acceptance, and spirituality [39]. The effec- topic to be addressed, along with some examples of tiveness of “third wave” interventions has been widely exercises that participants will then be able to practice contrasted empirically, also in the case of health profes- on their own. Written material will also be provided to sionals [16], and the MINDxYOU program was designed give participants a deeper understanding of the contents based on the experience of the research group on previ- of the module, and audios with guided practices will be ous projects, such as the Attachment-Based Compassion included for each exercise. PDF files and audios will be Therapy [40] and Smiling is Fun [41] online programs, available for participants to download and save for use and also considering the results of the HEROES [42] in the future or for practice outside of the web platform. and the PSY-COVID projects, the latter being an inter- Practice logs will also be available, and participants will national research project that evaluated the psychosocial be able to complete them on the online platform to ena- impact of the COVID-19 pandemic on both the general ble the research team to check the completion of tasks. It population and specific populations (healthcare person- is recommended that a period of 2 weeks be dedicated to nel, university students, etc.). each module, although participants will be free to adapt MINDxYOU is a self-guided online program particu- this to their needs. The first module, on mindfulness, will larly designed for healthcare workers in order to both introduce most of the basic concepts of the intervention, monitor and support the individual’s mental health. This and these will also be taken up in the following modules. program will aid healthcare professionals to deal with The last module will focus on how to apply the concepts their daily, work-related stress through the practice of and practices of mindfulness and compassion concepts mindfulness, compassion, and acceptance exercises, to key aspects related to a healthy lifestyle (i.e., physical along with indications on how to apply these concepts to activity, diet, sleep, and socialization). The estimated time basic aspects of a healthy lifestyle (i.e., physical activity, required to complete the whole program is 8 weeks; after sleep hygiene, balanced diet, and socialization). week 8, participants will no longer be contacted by the The program is self-administered and will be deliv- member of the research team. However, the program will ered over the Internet and available via smartphone, continue to be available for use indefinitely. tablet, and personal computer. During the first 8 weeks, there will be minimum support provided by an external Assessment plan professional unrelated to the research group, who will All potential candidates will undergo an initial screening make weekly contact with participants via their pre- assessment conducted by a member of the research team ferred means (i.e., telephone, text message, email) to who is a certified psychiatrist or clinical/health psycholo- promote adherence to the intervention, encouraging the gist; this assessment will include MINI v7.0.2 to identify participant to keep on using the program and strength- psychiatric disorders. Later, a baseline evaluation will be ening their commitment to the exercises. However, this conducted, including an ad hoc sociodemographic ques- contact will neither be used to add new educational con- tionnaire, primary and secondary outcomes, process tent nor for assessment purposes, which will be entirely variables, and cost-utility measures. After the baseline covered by the online platform with online links to the evaluation, the participants will be assessed again every López‑del‑Hoyo et al. BMC Nursing (2022) 21:308 Page 6 of 12 8 weeks. During the intervention, the participants will GAD-7 in different populations, also for the Spanish ver- complete the SUS twice. sion [48]. The Brief Symptoms Inventory-18 (BSI-18) [49] is an Diagnostic interview: Mini International Neuropsychiatric adapted version of the original 53-item, self-report ques- Interview (MINI) Version 7.0.2 tionnaire [50] that is designed to offer rapid screening MINI v7.0.2 [43] is a brief structured diagnostic interview for the symptoms of psychological disorders. Each item for DSM-5 psychiatric disorders. It assesses the 17 most is scored on a 5-point Likert-type scale, reflecting on the common disorders in mental health, those considered past 7 days. Scores on the 18 items are summarized on the most important to identify in clinical and research the Global Severity Index (GSI). The BSI-18 also includes settings: major depressive episode, major depressive three symptom scales: Somatization, Depression, and disorder, suicidal tendencies, suicidal behavior disorder, Anxiety, each comprising six items. The Spanish version maniac episode, hypomanic episode, bipolar disorder [51] shows reliable and valid psychometric properties. type I, bipolar disorder with psychotic traits, bipolar dis- order type II, other bipolar disorder, panic disorder, ago- Process variables raphobia, social anxiety disorder, obsessive–compulsive The Connor-Davidson Resilience Scale (CD-RISC) [52] disorder, post-traumatic stress disorder, alcohol abuse is a 10-item, self-report measure addressed at assessing disorder, and substance abuse disorder. It takes approxi- resilience. Each item is scored on a 5-point Likert scale, mately 15 min to administer, and the present study will and the total score, which ranges between 1 and 5, is cal- use the Spanish version to verify that the participants do culated by averaging the scores of the items; higher scores not present with any severe conditions. MINI has shown indicate higher levels of resilience. The questionnaire has excellent psychometric properties, including high inter- shown good psychometric properties in people with anx- rater reliability (k = 0.90 or higher) [43]. iety or stress-related disorders. The Spanish version [53] also presented good internal consistency (α = 0.86) and Primary outcome: Perceived Stress Scale (PSS) test–retest reliability (r = 0.87). PSS [37] consists of 10 items in which participants are The Five Facets of Mindfulness Questionnaire -15 item asked to rate how unpredictable, uncontrollable, and version (FFMQ-15) [54] is an adaptation of the origi- overloaded they have found their life over the past month nal 39-item FFMQ, a questionnaire measuring the five on a 5-point Likert-type scale (e.g., “how often have you facets of mindfulness: observing (i.e., noticing internal felt that you were unable to control the important things and external experiences such as sensations, thoughts, in your life?”). Scores range between 0 and 40, and higher or emotions), describing (i.e., labeling internal experi- scores reflect higher levels of perceived stress. PSS was ences with words), acting with awareness (i.e., focusing validated in Spanish [44], and this version presented good on one’s activities in the present moment as opposed to psychometric properties, including internal consistency behaving automatically), nonjudging of inner experience (α = 0.81), test–retest reliability (r = 0.77), and sensitivity (i.e., taking a non-evaluative stance toward thoughts to change. and feelings), and non-reacting to inner experience (i.e., allowing thoughts and feelings to come and go, without Secondary outcomes getting caught up by them). Each subscale of FFMQ-15 The Patient Health Questionnaire (PHQ-9) [45] is a includes 3 items, scored on a Likert-type scale (1–5). A 9-item scale aimed at screening for depression in primary score for each subscale, ranging between 5 and 15, can care and other medical settings. It assesses the frequency be computed by summing the items, where higher scores of depressive symptoms during the last 2 weeks using a indicate higher levels of the mindfulness facet. FFMQ-15 scale between 0 (not at all) and 3 (nearly every day). The presents good psychometric properties, similar to those total score ranges between 0 and 27, with higher scores of the original questionnaire, and the Spanish version indicating higher severity of depression. PHQ-9 has pre- also presented good internal consistency for every sub- sented excellent psychometric properties, including reli- scale [55]. ability, validity, and responsiveness [46]. The Sussex-Oxford Compassion Scales (SOCS) [56] The Generalized Anxiety Disorder-7 (GAD-7) [47] is a consist of two self-report measures designed to evalu- 7-item, self-report measure to assess the intensity of anx- ate self-compassion and other-compassion. Each meas- iety symptoms. It refers to the period of the past 2 weeks; ure consists of 20 items that are answered on a 5-point each item is scored on a 4-point Likert-type scale, result- Likert-type scale. A total score of each scale can be cal- ing in a total score that can range between 0 and 21, with culated by summing the scores of the items; each total higher values reflecting more severe anxiety symptoms. score ranges between 20 and 100, with higher scores Good psychometric properties have been reported for indicating higher levels of compassion. The psychometric López‑del‑Hoyo et al. BMC Nursing (2022) 21:308 Page 7 of 12 properties of the SOCS have proved to be good [56], and With regard to service use, patients will be asked about the Spanish adaptation is currently being validated by our total visits to emergency departments, total days of gen- research group. eral inpatient hospital admissions, number of diagnostic The Acceptance and Action Questionnaire-II (AAQ- tests administered, and total visits to primary care phy- II) [57] is a common measure for assessing experiential sician, nurse, social worker, psychologist, psychiatrist, avoidance, a key element of “third wave” psychotherapies group psychotherapy, and other community healthcare that refers to the tendency to avoid thoughts, feelings, professionals, specifying in each case if these services memories, sensations, and other internal experiences, were provided by the public or by the private sector. and which is very commonly associated with worse men- tal health outcomes, as opposed to psychological flex- Implementation outcomes ibility. The scale presents 7 items scored on a 7-point In accordance with the Consolidated Framework for Likert-type scale, and the total score is calculated by Implementation Research (CFIR) model, and taking into summing all the items. Higher scores mean higher expe- consideration the guidelines by Proctor et al. [64] and riential avoidance. AAQ-II shows good psychometric recommendations for implementation studies of online properties, including internal consistency (α = 0.84) and interventions by Hermes et al. [65], the present study test–retest reliability (r = 0.79). The Spanish version has will assess the following outcomes for the implementa- presented similar psychometric qualities [58]. tion process: acceptability, adoption, appropriateness, feasibility, fidelity, penetration, sustainability, and imple- Cost‑utility variables mentation costs, which will be measured along with the The five-level version of the EuroQol five-dimensional cost-utility variables. Treatment expectancies will be classification system (EQ-5D) [59] is a widely used assessed before commencing the intervention, and par- health-related quality of life measure. First, partici- ticipants will again be asked the same questions after pants are asked to report the severity (1 = no problems, completing the last module to assess their opinion [66]. 2 = mild, 3 = moderate, 4 = severe, 5 = extreme) of the In addition, focus groups will be held with the key players problems they may present on the day of reporting with involved in the study in each province (i.e., Zaragoza and regard to each of the following five domains: mobility, Malaga) once the intervention is complete (posttreat- self-care, usual activities, pain, and anxiety/depression. ment) to qualitatively evaluate different domains related The combination of the answers given to these domains to the implementation. results in 3,125 different health states. The utility scores The first of these, acceptability, defined as the percep- are obtained from the EQ-5D classification system and tion among implementation stakeholders that the inter- are used to rate patients’ health-related quality of life, vention is useful or satisfactory, will be measured using which normally range between 0 (although it is possible the following measures: to present negative scores) and 1 (i.e., “perfect health”). These utility values will be calculated using the Spanish – The System Usability Scale (SUS) [67], which is a tariffs of EQ-5D-5L [60]. EQ-5D utility values will be 10-item questionnaire to measure the usability which used to estimate quality-adjusted life-years (QALYs), a is qualitatively related to the quality and acceptabil- common measure to assess the outcomes associated with ity of the intervention [68]. Usability is defined as the different treatments, both in terms of patients’ quality of ease of use perceived by the users of the implemented life and survival [61]. EQ-5D also asks participants to use technology. Use will be made of the Spanish version a visual analog scale (EQ-VAS) to record their current of SUS, which has presented good internal consist- overall health status, ranging between 0 (worst imagina- ency (α = 0.81) [69]. ble health) and 100 (best imaginable health). – The Attitudes towards Psychological Online Interven- The Client Service Receipt Inventory (CSRI) [62] is one tions (APOI) [70], which includes 16 items to assess of the most widely used resource-use measurement tools; attitudes toward online interventions. APOI explores it describes and measures service utilization patterns as four dimensions; “Skepticism and Perception of a basis for estimating associated costs across healthcare, Risks,” “Confidence in Effectiveness,” “Technologi- social care, and community settings. In the present study, zation Threat,” and “Anonymity Benefits”. APOI has the Spanish version [63] will be used to collect retrospec- shown good psychometric properties with acceptable tive data (last 2 months) on medication and service use. to good internal consistency (α = 0.77) [70]. For medication intake, patients will be asked to bring – The Client Satisfaction Questionnaire adapted to their daily medication prescriptions, and the following Internet-based interventions (CSQ-I) [71] is an information will be recorded: drug name, dosage, total 8-item, 4-point Likert scale questionnaire that number of prescription days, and daily dosage consumed. assesses the general satisfaction of participants López‑del‑Hoyo et al. BMC Nursing (2022) 21:308 Page 8 of 12 regarding the received intervention. The total score the 5 modules of the intervention) will be performed. ranges between 8 and 32. The scale has shown excel- The analyses will include the description and elemen- lent internal consistency (α = 0.93) [72]. tary head-to-head comparisons between both phases (i.e., control vs. intervention). Specifically, the variables Another construct that will be assessed in this regard for each of the phases will be described by using descrip- is appropriateness, which is defined as perceived fit, rel- tive statistics (means and 95% confidence intervals in the evance, or compatibility of the evidence-based practice case of quantitative variables with normal distribution, for a given practice setting. We will use the Intervention medians, and interquartile range in the case of quantita- Appropriateness Measure (IAM) [73], which includes 4 tive variables with non-normal distribution). To confirm items designed to measure appropriateness of the inter- the main hypothesis, all of the effectiveness variables (i.e., vention. The scale has shown good psychometric proper- PSS and secondary outcomes) will be compared between ties with high levels of internal consistency (α = 0.85 to the intervention and control phases. A multilevel regres- 0.91) and test–retest reliability coefficients (r = 0.73 to sion analysis will be performed including the cluster as 0.88). a random effect, and multivariate multilevel regression The third domain that will be assessed in terms of analysis will be performed to adjust for possible con- implementation is adoption, defined as the intention, founders. The magnitude of the effect of the improve- initial decision, or action to try or employ an evidence- ment and the Number Needed to Treat (NNT) will based practice. In our study, adoption will be measured also be reported, using the cut-off points already estab- by the number of participants who agreed to use the pro- lished in the primary outcome as criteria for considering gram (considering the total number that were invited to whether a participant is a responder. use it) and actually accessed the online program. We will gather this information from the passive data collected by the online platform. Economic evaluation The feasibility of the intervention will also be studied. The economic evaluation will be carried out following This is defined as the extent to which a new treatment the recommendations of the Spanish guidelines for the can be successfully used or carried out within a given economic evaluation of health technologies [74]. A soci- agency or setting. Feasibility will be measured with pas- etal perspective will be adopted, considering both direct sive data gathered by the online platform regarding use of and indirect costs. The cost-utility analyses will con- the program, specifically the frequency of use during the sider the effect of the intervention on the QALYs, and study. Use will also be made of the Feasibility of Interven- the results will be expressed in terms of the incremental tion Measure (FIM) [73], which includes 4 items designed cost-utility ratio (ICUR), calculated by dividing the dif- to measure the feasibility of the intervention. The scale ference in total costs between the intervention phase and has shown good psychometric properties with high levels the control phase by the difference in QALYs between of internal consistency (α = 0.85 to 0.91) and test–retest both phases. A cost-effectiveness analysis will be also reliability coefficients (r = 0.73 to 0.88). conducted, in this case considering the effect of the inter- The fifth domain of implementation that will be ana- vention on perceived stress (PSS). These results will be lyzed is fidelity. In the context of online interventions, expressed in terms of the incremental cost-effectiveness fidelity is defined as expected clinically meaningful use. ratio (ICER), calculated by dividing the difference in total This dimension will be measured with passive data col- costs between the treatment phase and the control phase lected by the online platform, specifically the number of by the difference in PSS scores between both phases. modules and tasks completed. Penetration, in turn, will be measured considering the number of participants who agreed to use the intervention and completed at least Implementation analysis 4 of the 5 modules of the program. Finally, sustainabil- In order to perform the analysis related to the implemen- ity, defined by the extent to which a new intervention is tation process of the intervention, different outcomes maintained, will be evaluated during the interviews and/ will be considered according to the adopted model [64, or focus groups with the managers or directors of the dif- 65]. The analysis of the passive data consists of objective ferent sites. and direct counts (e.g., logins, frequency of use, mod- ules completed, task completed, etc.). The quantitative Data analysis data extracted from the questionnaires will be analyzed Effectiveness analysis according to the appropriate methodology in each case. Both intention-to-treat analysis and per-protocol analy- Some data will be collected qualitatively through inter- ses (i.e., those participants who complete at least 4 of views with the different stakeholders. López‑del‑Hoyo et al. BMC Nursing (2022) 21:308 Page 9 of 12 Ethics and dissemination in order to bring about a reduction in their risk of All procedures performed in this study will adhere to the developing mental health disorders and, consequently, 1964 Declaration of Helsinki and its most recent amend- enhancement of their quality of life and job perfor- ments (7th revision, adopted by the 64th World Medical mance. The pandemic has helped to raise the visibility Association General Assembly, Fortaleza, Brazil). Signed of healthcare workers as a risk group for developing informed consent will be obtained from all participants, mental health problems, which has facilitated the pro- who will not receive any compensation, once they have liferation of different interventions in the recent years been informed of the study procedures, potential risks, to address the needs of this population. and their right to withdraw at any time from the study. In this regard, online interventions have become par- The Research Ethics Committee of the Autonomous ticularly popular, both during and after the pandemic, Community of Aragon (CEICA) and the Ethics and as they offer a solution when face-to-face treatments Research Committee of Northeast Malaga evaluated and are not available or recommended. Different programs approved the study protocol in July 2022 (PI22/341). Any have been adapted to the online format, mainly those important protocol modifications (e.g., changes to eligibil- based on classic CBT and “third wave” psychothera- ity criteria, outcomes, analyses) will need to be approved pies [20, 23–30]. Both types of interventions have been by the ethics committees. To guarantee the confidential- tested, with generally positive effects in stress reduction, ity of the information, all the data collected by the plat- although the effect sizes in many cases were small, which form will comply with the provisions of the Spanish Data indicates that there is still room for improvement. The Protection and Digital Rights Act 3/2018 (LOPD), which problem with some of these interventions might be that adapts Spanish legislation to the European Union’s Gen- they deal with stress in a more generic way, rather than eral Data Protection Regulation (GRDP). A specific plan focusing on the kind of stressors that healthcare workers to guarantee maximum security of all collected data will face on a daily basis. In this regard, a systematic review be implemented for this project, and an assessment will be [76] suggested that the next step when developing online made of any possible risks and impacts on the information interventions targeting healthcare professionals was to flow. The technology platforms will be separated into two adjust them to the particular needs of this population; completely independent systems, which will access two therefore, an intervention that considers this aspect also independent databases. This will provide complete could generate both a higher degree of adherence and disaggregation of patient data, i.e., users’ personal data greater impact in terms of stress reduction. will be separated from their clinical record, multiplying by MINDxYOU is a self-guided, Internet-delivered pro- 2 all the security measures protecting end users. The data gram that has been designed following the path drawn will remain separate on 2 different servers, with clinical by previous studies offering online interventions to data hosted on server 1 and personal data on server 2. deal with different psychopathological symptoms (e.g., Once the study is complete, we will publish our depression, anxiety, stress) [25, 26, 40, 41] and, using results in international peer-reviewed biomedical jour- the feedback provided by different healthcare work- nals and present them at national and international ers (i.e., doctors, nurses, psychologists), it adds a focus conferences. The implementation study will be con- customized to the needs of these professionals, with ducted and reported according to the Standards for examples that relate to common problems in their daily Reporting Implementation Studies (StaRI) Statement, lives. The program offers different modules that enable indicated for reporting implementation studies focus- the practice mindfulness, acceptance, and compassion, ing on enhancing the adoption and sustainability of which constitute the core of the intervention, and a the intervention [75]. A report will also be sent to the final module focusing on how to apply such skills to the Instituto de Salud Carlos III (main funding body). The main aspects of a healthy lifestyle (i.e., diet, sleep, exer- lead researcher will organize an end-of-study knowl- cise, socialization). This is justified by the importance edge translation seminar. The main aim of this activity of such aspects in mental health, and considering that will be to share the study findings with stakeholders in many healthcare professionals have neglected their per- order to discuss how to maximize uptake of the find- sonal needs owing to their high levels of stress during ings and to determine future research directions. the pandemic [2]. On the whole, MINDxYOU aims to cover the main needs of healthcare workers with regard to their stress management, and the present study Discussion will test whether the program is both effective and Healthcare workers are in need of effective interven- cost-effective. tions to reduce their stress levels, which have been Furthermore, the problem with many psychological very significantly affected by the COVID-19 pandemic, interventions is the gap between research and practice. López‑del‑Hoyo et al. BMC Nursing (2022) 21:308 Page 10 of 12 This means that, despite being proven effective, interven- DC: Methodology, Writing – review & editing. AMB: Visualization, Resources, Writing – review & editing. MBR: Resources, Writing – review & editing. BMK: tions generally take quite a number of years to be applied Conceptualization, Supervision, Writing – review & editing. FMC: Concep‑ in the real-world practice [31]. Thus, implementation sci- tualization, Supervision, Writing – review & editing. JGC: Conceptualization, ence focuses on the development of studies to reduce the Supervision. All authors have read and approved the final manuscript. know-do gap, focusing on the barriers to and facilitators Funding for the implementation process [77]. Based on previous The study has been externally funded, after undergoing peer review, by the RCTs, we expect to adapt the intervention according to Instituto de Salud Carlos III (ISCIII), dependent on the Spanish Ministry of Science and Innovation, through a grant for health-related research projects users’ experience and opinions [78, 79]. The implemen- (PI21/01338) and co-financed with European Union ERDF funds. The funder tation study will be conducted in order to assess the plays no role in study design, data collection and analysis, decision to publish, impact of the intervention on participants’ stress levels, or manuscript preparation. The funding organization will audit trial conduct once a year. the direct and indirect costs of the intervention, and the different factors influencing the implementation process: Availability of data and materials acceptability, appropriateness, adoption, feasibility, fidel- Upon completion of the study, the datasets generated and/or analyzed during the current study will be made available to researchers on request to the cor‑ ity, penetration and sustainability. responding authors: Adrián Pérez-Aranda (aparanda@iisaragon.es) and Alberto If our hypotheses are confirmed, MINDxYOU will turn Barceló-Soler (abarcelosoler@hotmail.com). No manuscripts related to this out to be both effective and cost-effective, and our results study have yet been published or submitted to any journal. will provide information on the potential use of informa- tion and communications technologies (ICTs) related to Declarations the cost-effectiveness of Internet-based psychological Ethics approval and consent to participate interventions. Moreover, the study will show how imple- The Research Ethics Committee of the Autonomous Community of Aragon mentation studies are useful to establish the framework (CEICA) and the Ethics and Research Committee of Northeast Malaga evalu‑ ated and approved the study protocol in July 2022 (PI22/341). All study with which to deal with the barriers to online psycho- participants will sign an informed consent prior to randomization. therapies and promote their implementation. In sum- mary, this study could make a significant contribution Consent for publication Not applicable. to promote the use of online psychotherapy for reducing healthcare workers’ stress and reduce the gap between Competing interests research and practice. The authors declare that they have no competing interests. Author details 1 Institute of Health Research of Aragon (IIS Aragón), Miguel Servet University Abbreviations Hospital, Saragossa, Spain. 2 Departamento de Psicología Y Sociología, Facultad AAQ: Acceptance and Action Questionnaire; ACT​: Acceptance and Commit‑ de Ciencias Humanas Y de La Educación, Universidad de Zaragoza, Saragossa/ ment Therapy; APOI: Attitudes towards Psychological Online Interventions; Huesca, Spain. 3 Research Network On Chronicity, Primary Care and Health BSI: Brief Symptom Inventory; CBT: Cognitive-Behavioral Therapy; CD-RISC: Promotion (RICAPPS) RD21/0016/0005, Saragossa, Spain. 4 Department Connor-Davidson Resilience Scale; CFIR: Consolidates Framework for Imple‑ of Basic, Developmental and Educational Psychology, Autonomous University mentation Research; CSQ-I: Client Satisfaction Questionnaire adapted to of Barcelona, Cerdanyola Del Vallès, Spain. 5 Navarra Medical Research Institute Internet-Based interventions; CSRI: Client Service Receipt Inventory; EQ-5D: (IdiSNA), Pamplona, Spain. 6 Mental Health Clinical Management Unit, Regional EuroQoL 5 Dimensions; FFMQ: Five Facet Mindfulness Questionnaire; FIM: University Hospital of Malaga, Málaga, Spain. 7 Malaga Biomedical Research Feasibility of Intervention Measure; GAD: Generalized Anxiety Disorder; IAM: Institute (IBIMA), Málaga, Spain. 8 Department of Personality, Evaluation Intervention Appropriateness Measure; ICER: Incremental Cost-Effectiveness and Psychological Treatment, Faculty of Psychology, University of Malaga, Ratio; ICT: Information and communications technology; ICUR​: Incremental Málaga, Spain. 9 MARISTAN Network, Málaga, Spain. 10 Department of Psychia‑ Cost-Utility Ratio; MBI: Mindfulness-Based Interventions; NNT: Number Needed try, Faculty of Medicine, University of Zaragoza, Saragossa, Spain. to Treat; PC: Primary Care; PHQ: Patient Health Questionnaire; PSS: Perceived Stress Scale; QALY: Quality-Adjusted Life Year; RCT​: Randomized Controlled Received: 3 August 2022 Accepted: 27 October 2022 Trial; SOCS: Sussex-Oxford Compassion Scales; SUS: System Usability Scale; SW: Stepped Wedge; WHO: World Health Organization. Acknowledgements APA holds a “Sara Borrell” postdoctoral contract from the Instituto de Salud Carlos III (ISCIII; CD20/00181), not directly related to this project. We wish References to thank the Mental Health Research Group (B17_20R) of the Government 1. Pfefferbaum B, North CS. Mental Health and the Covid-19 Pandemic. N of Aragon (Spain) Department of Innovation, Research and University; Engl J Med. 2020;383:510–2. https://​doi.​org/​10.​1056/​nejmp​20080​17. the Research Network on Chronicity, Primary Care, and Health Promotion 2. Greenberg N, Docherty M, Gnanapragasam S, Wessely S. Managing (RICAPPS, RD21/0016/0005), which is part of the Results Oriented Cooperative mental health challenges faced by healthcare workers during covid-19 Research Networks in Health (RICORS) (Carlos III Institute of Health); and Feder pandemic. BMJ. 2020;368:m1211. Funds “Another way to make Europe.” 3. Karasu F, Öztürk Çopur E, Ayar D. The impact of COVID-19 on healthcare workers’ anxiety levels. J Public Heal. 2021;:1–11. doi:https://​doi.​org/​10.​ Authors’ contributions 1007/​s10389-​020-​01466-x. YLDH: Conceptualization, Resources, Project administration. SFM: Resources, 4. Williamson V, Stevelink SAM, Greenberg N. Occupational moral injury Methodology. APA: Conceptualization, Methodology, Writing – original draft. and mental health: systematic review and meta-analysis. Br J Psychiatry. ABS: Visualization, Resources, Writing – review & editing. JGP: Methodology, 2018;212:339–46. https://​doi.​org/​10.​1192/​bjp.​2018.​55. Writing – review & editing. EVM: Methodology, Writing – review & editing. 5. Cullen W, Gulati G, Kelly BD. Mental health in the COVID-19 pandemic. QJM. 2020;113:311–2. https://​doi.​org/​10.​1093/​qjmed/​hcaa1​10. López‑del‑Hoyo et al. BMC Nursing (2022) 21:308 Page 11 of 12 6. World Health Organization (WHO). Mental health considerations during versus control group on work functioning of hospital employees: a COVID-19 outbreak. Geneva, Switzerland.; 2020. Cluster-RCT. PLoS One. 2013;8(9):e72546. 7. Sarabia-Cobo C, Pérez V, de Lorena P, Hermosilla-Grijalbo C, Sáenz-Jalón 26. Smoktunowicz E, Lesnierowska M, Carlbring P, Andersson G, Cieslak R. M, Fernández-Rodríguez A, et al. Experiences of geriatric nurses in Resource-based internet intervention (med-stress) to improve well-being nursing home settings across four countries in the face of the COVID-19 among medical professionals: randomized controlled trial. J Med Internet pandemic. J Adv Nurs. 2021;77:869–78. https://​doi.​org/​10.​1111/​jan.​14626. Res. 2021;23(1):e21445. 8. Lewis C, Roberts NP, Andrew M, Starling E, Bisson JI. Psychological thera‑ 27. Nourian M, Nikfarid L, Khavari AM, Barati M, Allahgholipour AR. The impact pies for post-traumatic stress disorder in adults: systematic review and of an online mindfulness-based stress reduction program on sleep quality meta-analysis. Eur J Psychotraumatol. 2020;11:1729633. https://​doi.​org/​ of nurses working in covid-19 care units: a clinical trial. Holist Nurs Pract. 10.​1080/​20008​198.​2020.​17296​33. 2021;35:257–63. https://​doi.​org/​10.​1097/​HNP.​00000​00000​000466. 9. Kowalik J, Weller J, Venter J, Drachman D. Cognitive behavioral therapy 28. Morrison Wylde C, Mahrer NE, Meyer RML, Gold JI. Mindfulness for novice for the treatment of pediatric posttraumatic stress disorder: a review and pediatric nurses: smartphone application versus traditional intervention. J meta-analysis. J Behav Ther Exp Psychiatry. 2011;42:405–13. Pediatr Nurs. 2017;36:205–12. https://​doi.​org/​10.​1016/j.​pedn.​2017.​06.​008. 10. Cuijpers P, Sijbrandij M, Koole S, Huibers M, Berking M, Andersson G. 29. Hersch RK, Cook RF, Deitz DK, Kaplan S, Hughes D, Friesen MA, et al. Psychological treatment of generalized anxiety disorder: a meta-analysis. Reducing nurses’ stress: a randomized controlled trial of a web-based Clin Psychol Rev. 2014;34:130–40. stress management program for nurses. Appl Nurs Res. 2016;32:18–25. 11. Chiesa A, Serretti A. Mindfulness-based stress reduction for stress man‑ https://​doi.​org/​10.​1016/j.​apnr.​2016.​04.​003. agement in healthy people: a review and meta-analysis. J Altern Comple‑ 30. Lee D, Lee WJ, Choi SH, Jang JH, Kang DH. Long-term beneficial effects ment Med. 2009;15:593–600. https://​doi.​org/​10.​1089/​acm.​2008.​0495. of an online mind-body training program on stress and psychological 12. Ledesma D, Kumano H. Mindfulness-based stress reduction and cancer: a outcomes in female healthcare providers: a non-randomized controlled meta-analysis. Psychooncology. 2009;18:571–9. https://​doi.​org/​10.​1002/​ study. Medicine (Baltimore). 2020;99: e21027. https://​doi.​org/​10.​1097/​MD.​ pon.​1400. 00000​00000​021027. 13. Carrière K, Khoury B, Günak MM, Knäuper B. Mindfulness-based interven‑ 31. Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the ques‑ tions for weight loss: a systematic review and meta-analysis. Obes Rev. tion: understanding time lags in translational research. J R Soc Med. 2018;19:164–77. https://​doi.​org/​10.​1111/​obr.​12623. 2011;104:510–20. https://​doi.​org/​10.​1258/​jrsm.​2011.​110180. 14. Wakelin KE, Perman G, Simonds LM, Wakelin K. Effectiveness of self- 32. Bauer MS, Damschroder L, Hagedorn H, Smith J, Kilbourne AM. An intro‑ compassion-related interventions for reducing self-criticism: A systematic duction to implementation science for the non-specialist. BMC Psychol. review and meta-analysis. 2021. 2015;3:1–12. https://​doi.​org/​10.​1186/​S40359-​015-​0089-9. 15. Phillips WJ, Hine DW. Self-compassion, physical health, and health behav‑ 33. Green BB, Coronado GD, Schwartz M, Coury J, Baldwin LM. Using a con‑ iour: a meta-analysis. Health Psychol Rev. 2021;15:113–39. https://​doi.​org/​ tinuum of hybrid effectiveness-implementation studies to put research- 10.​1080/​17437​199.​2019.​17058​72. tested colorectal screening interventions into practice. Implement Sci. 16. Kriakous SA, Elliott KA, Lamers C, Owen R. The effectiveness of mindful‑ 2019;14:1–9. ness-based stress reduction on the psychological functioning of health‑ 34. Geoffrey M. Curran, PhD*, Mark Bauer, MD†, Brian Mittman, PhD‡, Jeffrey care professionals: a systematic review. Mindfulness (N Y). 2021;12:1–28. M. Pyne M, and Cheryl Stetler P. Effectiveness-implementation Hybrid https://​doi.​org/​10.​1007/​s12671-​020-​01500-9. Designs: Combining Elements of Clinical Effectiveness and Implementa‑ 17. Rudaz M, Twohig MP, Ong CW, Levin ME. Mindfulness and acceptance- tion Research to Enhance Public Health Impact Geoffrey. Account Patient based trainings for fostering self-care and reducing stress in men‑ Saf Policy Reform. 2012;50:159–84. tal health professionals: a systematic review. J Context Behav Sci. 35. Hemming K, Haines TP, Chilton PJ, Girling AJ, Lilford RJ. The stepped 2017;6:380–90. wedge cluster randomised trial: Rationale, design, analysis, and reporting. 18. Asuero AM, Queraltó JM, Pujol-Ribera E, Berenguera A, Rodriguez-Blanco BMJ. 2015;350:h391. T, Epstein RM. Effectiveness of a mindfulness education program in pri‑ 36. Hemming K, Taljaard M. Sample size calculations for stepped wedge mary health care professionals: a pragmatic controlled trial. J Contin Educ and cluster randomised trials: A unified approach. J Clin Epidemiol. Health Prof. 2014;34:4–12. https://​doi.​org/​10.​1002/​chp.​21211. 2016;69:137–46. https://​doi.​org/​10.​1016/j.​jclin​epi.​2015.​08.​015. 19. Aranda Auserón G, Elcuaz Viscarret MR, Fuertes Goñi C, Güeto Rubio V, 37. Cohen S, Kamarck T, Mermelstein R. A global measure of perceived stress. Pascual Pascual P, Sainz de Murieta García de Galdeano E. Evaluation of the J Health Soc Behav. 1983;24:385. https://​doi.​org/​10.​2307/​21364​04. effectiveness of a Mindfulness and Self-Compassion program to reduce 38. Brooke J. SUS: A “Quick and Dirty” Usability Scale. In: Jordan PW, Thomas B, stress and prevent burnout in Primary Care health professionals. Aten Weerdmeester AB, McClelland IL, editors. Usability Evaluation In Industry. Primaria. 2018;50:141–50. doi:https://​doi.​org/​10.​1016/j.​aprim.​2017.​03.​009. London, UK: Taylor & Francis; 1996. p. 189–94. 20. Mistretta EG, Davis MC, Temkit M, Lorenz C, Darby B, Stonnington CM. 39. Kahl KG, Winter L, Schweiger U. The third wave of cognitive behavioural Resilience training for work-related stress among health care workers. J therapies: What is new and what is effective? Curr Opin Psychiatry. Occup Environ Med. 2018;60:559–68. https://​doi.​org/​10.​1097/​JOM.​00000​ 2012;25:522–8. https://​doi.​org/​10.​1097/​YCO.​0b013​e3283​58e531. 00000​001285. 40. Campos D, Navarro-Gil M, Herrera-Mercadal P, Martínez-García L, Cebolla 21. Brinkborg H, Michanek J, Hesser H, Berglund G. Acceptance and com‑ A, Borao L, et al. Feasibility of the internet attachment⇓based compas‑ mitment therapy for the treatment of stress among social workers: a sion therapy in the general population: protocol for an open-label randomized controlled trial. Behav Res Ther. 2011;49:389–98. https://​doi.​ uncontrolled pilot trial. JMIR Res Protoc. 2020;9(8):e16717. org/​10.​1016/j.​brat.​2011.​03.​009. 41. Montero-Marín J, Araya R, Pérez-Yus MC, Mayoral F, Gili M, Botella C, et al. 22. Gruber J, Prinstein MJ, Clark LA, Rottenberg J, Abramowitz JS, Albano An internet-based intervention for depression in primary care in Spain: a AM, et al. Mental health and clinical psychological science in the time of randomized controlled trial. J Med Internet Res. 2016;18(8):e231. COVID-19: Challenges, opportunities, and a call to action. Am Psychol. 42. Mascayano F, van der Ven E, Moro MF, Schilling S, Alarcón S, Al Barathie 2021;76:409–26. https://​doi.​org/​10.​1037/​amp00​00707. J, et al. The impact of the COVID-19 pandemic on the mental health of 23. Imamura M, Chen J, Matsubayashi SR, Targino RA, Alfieri FM, Bueno DK, healthcare workers: study protocol for the COVID-19 HEalth caRe wOrk‑ et al. Changes in pressure pain threshold in patients with chronic non‑ ErS (HEROES) study. Soc Psychiatry Psychiatr Epidemiol. 2022;57:633–45. specific low back pain. Spine (Phila Pa 1976). 2013;38:2098–107. https://​doi.​org/​10.​1007/​s00127-​021-​02211-9. 24. Fiol-DeRoque MA, Serrano-Ripoll MJ, Jiménez R, Zamanillo-Campos R, 43. Sheehan D V., Lecrubier Y, Sheehan KH, Amorim P, Janavs J, Weiller E, et al. Yáñez-Juan AM, Bennasar-Veny M, et al. Effectiveness of a Mobile Phone- The Mini-International Neuropsychiatric Interview (M.I.N.I.): The develop‑ Based Intervention to Reduce Mental Health Problems in Healthcare ment and validation of a structured diagnostic psychiatric interview Workers During the COVID-19 Pandemic: A Randomized Controlled Trial for DSM-IV and ICD-10. In: Journal of Clinical Psychiatry. 1998. p. 22–33. (PsyCovidApp Trial). SSRN Electron J. 2020;:In press. doi:https://​doi.​org/​10.​ https://​pubmed.​ncbi.​nlm.​nih.​gov/​98815​38/. Accessed 3 Feb 2022. 2139/​ssrn.​37492​14. 44. Remor E. Psychometric properties of a European Spanish version of the 25. Ketelaar SM, Nieuwenhuijsen K, Gärtner FR, Bolier L, Smeets O, Sluiter Perceived Stress Scale (PSS). Span J Psychol. 2006;9:86–93. https://​doi.​org/​ JK. Effect of an E-mental health approach to workers’ health surveillance 10.​1017/​S1138​74160​00060​04. López‑del‑Hoyo et al. BMC Nursing (2022) 21:308 Page 12 of 12 45. Kroenke K, Spitzer RL. The PHQ-9: A New Depression Diagnostic and 66. Boß L, Lehr D, Reis D, Vis C, Riper H, Berking M, et al. Reliability and validity Severity Measure. Psychiatry Ann. 2002;32. of assessing user satisfaction with web-based health interventions. J Med 46. Furukawa TA. Assessment of mood: guides for clinicians. J Psychosom Internet Res. 2016;18(8):e234. Res. 2010;68:581–9. https://​doi.​org/​10.​1016/J.​JPSYC​HORES.​2009.​05.​003. 67. Bangor, Aaron, Kortum P, T. P, Miller, T. J. The System Usability Scale (SUS): 47. Spitzer RL, Kroenke K, Williams JBW, Löwe B. A brief measure for an Empirical evaluation. Int J Hum Comput Interact. 2008;24:574. assessing generalized anxiety disorder: The GAD-7. Arch Intern Med. 68. Brooke J. „SUS-A quick and dirty usability scale (in" Usability Evaluation 2006;166:1092–7. in Industry", PW Jordan, B Thomas, I McLelland, BA Weerdmeester (eds))“. 48. García-Campayo J, Zamorano E, Ruiz MA, Pardo A, Pérez-Páramo M, Usability Eval Ind (PW Jordan, B Thomas, BA Weerdmeester, AL Mcclell. 1996. López-Gómez V, et al. Cultural adaptation into Spanish of the generalized 69. Sevilla-Gonzalez MDR, Moreno Loaeza L, Lazaro-Carrera LS, Bourguet anxiety disorder-7 (GAD-7) scale as a screening tool. Health Qual Life Ramirez B, Vázquez Rodríguez A, Peralta-Pedrero ML, et al. Spanish Ver‑ Outcomes. 2010;8:8. sion of the system usability scale for the assessment of electronic tools: 49. Derogatis LR. Brief Symptoms Inventory (BSI-18). Administration, scoring development and validation. JMIR Hum Factors. 2020;7: e21161. https://​ and procedures. Minneapolis: NCS Pearson; 2001. doi.​org/​10.​2196/​21161. 50. Derogatis LR, Melisaratos N. The brief symptom inventory: an introduc‑ 70. Schröder J, Sautier L, Kriston L, Berger T, Meyer B, Späth C, et al. Develop‑ tory report. Psychol Med. 1983;13:595–605. ment of a questionnaire measuring Attitudes towards Psychological 51. Miaja Ávila M, Moral de la Rubia J. Validación del Inventario Breve de Sínto‑ Online Interventions-the APOI. J Affect Disord. 2015;187:136–41. mas (BSI-18) en mujeres Mexicanas diagnosticadas con cáncer de mama. 71. Bo L, Lehr D, Reis D, Vis C, Riper H, Berking M, et al. Reliability and validity Psicooncologia. 2017;14:307–24. doi:https://​doi.​org/​10.​5209/​PSIC.​57088. of assessing user satisfaction with web-based health interventions. J Med 52. Campbell-Sills L, Stein MB. Psychometric analysis and refinement of the Internet Res. 2016;18. doi:https://​doi.​org/​10.​2196/​jmir.​5952. Connor-Davidson Resilience Scale (CD-RISC): Validation of a 10-item 72. Attkisson CC, Zwick R. The client satisfaction questionnaire. Psychometric measure of resilience. J Trauma Stress. 2007;20:1019–28. https://​doi.​org/​ properties and correlations with service utilization and psychotherapy 10.​1002/​jts.​20271. outcome. Eval Program Plann. 1982;5:233–7. 53. Soler Sánchez MI, Meseguer de Pedro M, García Izquierdo M. Propiedades 73. Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, et al. psicométricas de la versión española de la escala de resiliencia de 10 Psychometric assessment of three newly developed implementation ítems de Connor-Davidson (CD-RISC 10) en una muestra multiocupa‑ outcome measures. Implement Sci. 2017;12:1–13. cional. Rev Latinoam Psicol. 2016;48:159–66. 74. López Bastida J, Oliva J, Antoñanzas F, García-Altés A, Gisbert R, Mar J, 54. Gu J, Strauss C, Crane C, Barnhofer T, Karl A, Cavanagh K, et al. Examin‑ et al. Propuesta de guía para la evaluación económica aplicada a las tec‑ ing the factor structure of the 39-item and 15-item versions of the Five nologías sanitarias. Gac Sanit. 2010;24:154–70. https://​doi.​org/​10.​1016/j.​ Facet Mindfulness Questionnaire before and after mindfulness-based gaceta.​2009.​07.​011. cognitive therapy for people with recurrent depression. Psychol Assess. 75. Pinnock H, Barwick M, Carpenter CR, Eldridge S, Grandes G, Griffiths CJ, 2016;28:791–802. https://​doi.​org/​10.​1037/​pas00​00263. et al. Standards for Reporting Implementation Studies (StaRI): explanation 55. Cebolla A, García-Palacios A, Soler J, Guillen V, Baños R, Botella C. Psycho‑ and elaboration document. BMJ Open. 2017;7: e013318. metric properties of the Spanish validation of the Five Facets of Mindful‑ 76. Pospos S, Young IT, Downs N, Iglewicz A, Depp C, Chen JY, et al. Web- ness Questionnaire (FFMQ). Eur J Psychiatry. 2012;26:118–26. https://​doi.​ based tools and mobile applications to mitigate burnout, depression, org/​10.​4321/​S0213-​61632​01200​02000​05. and suicidality among healthcare students and professionals: a system‑ 56. Gu J, Baer R, Cavanagh K, Kuyken W, Strauss C. Development and Psy‑ atic review. Acad Psychiatry. 2018;42:109–20. https://​doi.​org/​10.​1007/​ chometric Properties of the Sussex-Oxford Compassion Scales (SOCS). S40596-​017-​0868-0. Assessment. 2020;27:3–20. https://​doi.​org/​10.​1177/​10731​91119​860911. 77. Bauer MS, Kirchner JA. Implementation science: What is it and why 57. Bond FW, Hayes SC, Baer RA, Carpenter KM, Guenole N, Orcutt HK, et al. should I care? Psychiatry Res. 2020;283. doi:https://​doi.​org/​10.​1016/J.​ Preliminary psychometric properties of the acceptance and action PSYCH​RES.​2019.​04.​025. questionnaire-II: a revised measure of psychological inflexibility and 78. Castro A, López-del-Hoyo Y, Peake C, Mayoral F, Botella C, García-Campayo experiential avoidance. Behav Ther. 2011;42:676–88. https://​doi.​org/​10.​ J, et al. Adherence predictors in an Internet-based Intervention program 1016/j.​beth.​2011.​03.​007. for depression. Cogn Behav Ther. 2018;47:246–61. https://​doi.​org/​10.​ 58. Ruiz FJ, Langer Herrera ÁI, Luciano C, Cangas AJ, Beltrán I. Midiendo la 1080/​16506​073.​2017.​13665​46. evitación experiencial y la infl exibilidad psicológica: Versión española del 79. Jesús MM, Prado-Abril J, Botella C, Fermin MC, Rosa B, Paola HM, et al. cuestionario de aceptación y acción - II. Psicothema. 2013;25:123–9. Expectations among patients and health professionals regarding Web- 59. Herdman M, Gudex C, Lloyd A, Janssen M, Kind P, Parkin D, et al. Develop‑ based interventions for depression in primary care: a qualitative study. J ment and preliminary testing of the new five-level version of EQ-5D Med Internet Res. 2015;17(3):e67. (EQ-5D-5L). Qual Life Res. 2011;20:1727–36. https://​doi.​org/​10.​1007/​ s11136-​011-​9903-x. 60. Badia X, Roset M, Montserrat S, Herdman M, Segura A. [The Spanish ver‑ Publisher’s Note sion of EuroQol: a description and its applications. European Quality of Springer Nature remains neutral with regard to jurisdictional claims in pub‑ Life scale]. Med Clin (Barc). 1999;112 Suppl:79–85. http://​www.​ncbi.​nlm.​ lished maps and institutional affiliations. nih.​gov/​pubmed/​10618​804. Accessed 23 Apr 2019. 61. Stone G, Hutchinson A, Corso P, Teutsch S, Fielding J, Carande-Kulis V, et al. Understanding and using the economic evidence. In: The Guide to Commu‑ nity Preventive Services: What works to promote health? 2005. p. 449–63. Ready to submit your research ? Choose BMC and benefit from: 62. Beecham J, Knapp M. Costing psychiatric interventions. In: Thronicroft G, Bewin C, Wing J, editors. Measuring Mential Health Needs. Oxford: Oxford • fast, convenient online submission University Press; 1992. 63. Vázquez-Barquero J, Gaite L, Cuesta M, Garcia-Usieto E, Knapp M, • thorough peer review by experienced researchers in your field Beecham J. Spanish version of the CSRI: a mental health cost evaluation • rapid publication on acceptance interview. Arch Neuobiology. 1997;60:171–84. • support for research data, including large and complex data types 64. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, • gold Open Access which fosters wider collaboration and increased citations measurement challenges, and research agenda. Adm Policy Ment Health. • maximum visibility for your research: over 100M website views per year 2011;38:65–76. 65. Hermes EDA, Lyon AR, Schueller SM, Glass JE. Measuring the implemen‑ At BMC, research is always in progress. tation of behavioral intervention technologies: Recharacterization of established outcomes. J Med Internet Res. 2019;21: e11752. https://​doi.​ Learn more biomedcentral.com/submissions org/​10.​2196/​11752.