Internet Interventions 30 (2022) 100581 Contents lists available at ScienceDirect Internet Interventions journal homepage: www.elsevier.com/locate/invent Implementation of a psychological online intervention for low to moderate depression in primary care: study protocol A R T I C L E I N F O A B S T R A C T Keywords Background: Depression affects millions of people all over the world and implies a great socioeconomic burden. Implementation Despite there are different effective evidence-based interventions for treating depression, only a small proportion Psychological treatment of these patients receives an appropriate treatment. In this regard, information and communication technologies Depression Primary care (ICTs) can be used with therapeutic aims and this can contribute to make interventions more accessible. One Health settings example is “Smiling is fun”, an internet-based treatment which has proved to be effective and cost-effective for Evidence-based treatment treating depression in Spanish Primary Care (PC). However, the “know-do gap” between research and clinical settings implies that the actual implementation of such interventions could last up to 20 years. To overcome this obstacle, the implementation research establishes the methodology to implement the advances developed in the laboratories to the health care services maintaining the validity of the intervention and offering specific stra­ tegies for the implementation process. Objective: This is the protocol of an implementation study for the Internet-based program “Smiling is fun”, which will be conducted on patients with mild-to-moderate depression of Spanish PC settings. In the implementation study, the feasibility, efficacy, cost-efficacy, acceptability, adoption, appropriateness, fidelity, penetration, normalization, and sustainability will be assessed. Methods: The current investigation is a Hybrid Effectiveness-Implementation Type II design. A Stepped Wedge randomized controlled trial design will be used, with a cohort of 420 adults diagnosed with depression (mild-to- moderate) who will undergo a first control phase (no treatment) followed by the intervention, which will last 16 weeks, and finishing with an optional use of the intervention. All patients will be assessed at baseline, during the treatment, and at post-treatment. The study will be conducted in three Spanish regions: Andalusia, Aragon, and the Balearic Islands. Two primary care centers of each region will participate, one located in the urban setting and the other in the rural setting. The primary outcome will be implementation success of the intervention assessing the reach, clinical effect, acceptability, appropriateness, adoption, feasibility, fidelity, penetration, implementation costs and sustainability services. Discussion: “Smiling is Fun”, which has already been established as effective and cost-effective, will be adapted according to users' experiences and opinions, and the efficacy and cost-efficacy of the program will again be assessed. The study will point out barriers and facilitators to consider in the implementation process of internet- based psychological interventions in health services. The ultimate goal is to break the research-to-practice split, which would undoubtedly contribute to reduce the high burden of depression in our society. Trial registration: ClinicalTrials.gov, Identifier: NCT05294614. 1. Introduction prevalent mental disorder in this setting (Roca et al., 2009). Pharmacological and psychological interventions (or the combina­ Depression is one of the most prevalent disabling psychological tion of both) are effective in the treatment of depression (Cuijpers et al., disorders with significant personal and social costs. In fact, it is the 2011; Cuijpers and Gentili, 2017). Furthermore, Evidence-Based Psy­ second cause of disability and is expected that in 2030 will be the first chological Treatments (EBPT) such as Cognitive Behavior Therapy cause of disease burden worldwide (Mathers and Loncar, 2006; Ferrari (CBT) or Behavioral Activation are recommended to address depressive et al., 2013; Haro et al., 2014). Specifically, in Spain, the annual cost of symptomatology (Nathan and Gorman, 2015). Despite the fact that depressive disorders is estimated to be 5.348 million of euros and the there are effective treatments for depression, access to these treatments lifetime prevalence rate in the general population in Spain is estimated is limited (Kazdin and Blase, 2011). Less than half of depressed people is at 10.55 (Salvador-Carulla et al., 2011). Moreover, the prevalence rate is treated by a professional and only a quarter receive an appropriate specially problematic in Primary Care (PC) (13.9–29 %), being the most treatment. Elevated costs of face-to-face treatment, the required time for https://doi.org/10.1016/j.invent.2022.100581 Received 29 June 2022; Received in revised form 13 October 2022; Accepted 25 October 2022 Available online 1 November 2022 2214-7829/© 2022 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by- nc-nd/4.0/). R. Lorente-Català et al. Internet Interventions 30 (2022) 100581 the application of the intervention, adherence problems, or the lack of et al., 2018). The determinants were grouped as (1) the acceptance of well-trained professionals are some of the causes of this situation the intervention perceived by different agents involved, including de­ (Fernández et al., 2010). In PC settings in Spain, this problem is specially terminants such as expectations and preferences or technical aspects (ex. dramatic, not only because of the high rate of prevalence, but also for the Technical support, complexity of the platform), (2) the appropriateness lack of accessibility to psychological treatment (Aragonès et al., 2004; of the intervention including determinants such as the professional- Gabilondo et al., 2011; Serrano-Blanco et al., 2010). patient interaction, the patients' needs and characteristics, (3) the par­ Traditional psychological face-to-face therapy is not responding to ticipants' engagement considering the organization, policies, leadership, the population's mental health needs, underlying the necessity of alter­ collaboration etc., (4) the resources for the implementation including native cost-effective ways of administering interventions. Information personnel, funds and the infrastructure, (5) work processes comprising and Communication Technologies (ICTs) offer cost-effective and acces­ primary and facilitating processes and (6) the leadership in the imple­ sible resources to address mental health needs (Titov, 2011; Kazdin and mentation process attaching the management, strategies, etc. Another Rabbitt, 2013). Systematic reviews and meta-analysis have supported results from a review focusing on the implementation components on the effectiveness and acceptability of Internet-based treatments (IBT) for Internet intervention for depression through the Active Implementation depression (Karyotaki et al., 2018; Kaltenthaler et al., 2008; Cuijpers Framework (AIF) show congruency with the determined factors (Drozd et al., 2009; Andrews et al., 2010; Andersson and Cuijpers, 2009; Kar­ et al., 2016). yotaki et al., 2017; Richards and Richardson, 2012). Furthermore, ICT In congruence, other studies also underline the importance of the tools are pointed as a solution to address dissemination, accessibility, professional's role. It has been observed that the general motivation and and delivery problems of psychological interventions (Kazdin and beliefs about the intervention and ICTs act as mediators on the imple­ Rabbitt, 2013). mentation process (Titzler et al., 2018). Although these characteristics Our research group has developed an Internet-delivered CBT (ICBT) act as facilitators, the lack of time, the negative beliefs about the (“Smiling is Fun”), that has shown efficacy in different RCTs (i.e., Botella intervention and ICT, and the lack of knowledge are some of the et al., 2016; Mira et al., 2019, 2018, 2017). The results showed a sig­ observed barriers (Folker et al., 2018; Vis et al., 2018; Wilhelmsen et al., nificant reduction of depressive symptoms and an increase in positive 2014). affect (Mira et al., 2019). Furthermore, the intervention was cost- Albeit these results may shed light on the factors influencing the effective, highlighting the economical appropriateness of it in Spanish implementation of interventions in daily settings, not all factors and PC settings along with its clinical benefits (Romero-Sanchiz et al., 2017). strategies are significant in all contexts. In fact, the current knowledge However, although “Smiling is Fun” has demonstrated efficacy and about the implementation process to address depression is scare stun­ cost-effectiveness, there is still a way to go until a program like this is ning the translation process of an intervention to daily practice (Drozd implemented and used in routine PC. In this sense, it is necessary to take et al., 2016). Therefore, it is necessary to explore the factors related to into account the know-do gap between the scientific world and the the implementation process in the specific real-world setting (PC set­ clinical settings, which is translated into 17 to 20 years for the popula­ tings in Spain), the implementation costs, and the necessary strategies to tion to benefit from the advantages of psychological research (Van Den implement the specific intervention (Smiling is fun). Driessen Mareeuw et al., 2015; Pakenham-Walsh, 2004; Morris et al., In view of the above, there is a need to determine the barriers and 2011). Implementation Research (IR) was developed to address this strategies that may help in implementation process of the intervention issue, studying the strategies and factors that affect the translation Smiling is Fun. The present work describes the protocol of a Hybrid- process of evidence based treatments to the daily routine (Bauer and effectiveness implementation study of the IBT “Smiling is Fun” to Kirchner, 2020). One of the essentials aspects of IR is his interaction with address mild-to-moderate depression in Spanish PC settings. the context. Despite a growing body of literature on the therapeutic The protocol will be reported according to the Consolidated Stan­ changes experienced by the population with different mental disorders dards of Reporting Trials (CONSORT) adapted for Stepped Wedge Trials (Grol et al., 2007), almost no research attempts to understand the dy­ (SWT) corresponding to the selected design for the study (Hemming namic interactions between individuals and the context where they et al., 2018). receive the treatment, and how these interactions may influence indi­ In particular, the main research question of the study is: vidual improvement (Kazdin, 2008). Unlike laboratory simulations conducted far from clinical settings, implementation studies are context - Which are the barriers impeding the implementation process in PC of specific. These studies focus on determining the reasons for the effec­ a psychological intervention applied through ICTs to address mild to tiveness of a treatment and the factors influencing it considering the moderate depression and which strategies are useful to overcome uniqueness of the implementation settings (May et al., 2016). Therefore, these barriers and promote the implementation? the main objective is to determine the barriers and facilitators in each context and develop specific implementation strategies for the desired Furthermore, the following specific aims are stablished: intervention. Consequently, IR establishes the methodology and the scientific approaches to implement the advances from the laboratories to - To assess the implementation results considering the dimensions of the health care services maintaining the validity of the intervention acceptability, adoption, appropriateness, feasibility, fidelity, pene­ (King et al., 2019). tration, and sustainability, perceived and experienced by the Fixsen and their colleagues elaborated a synthesis of the IR literature different agents involved. determining the core components for implementation practice (Fixsen - To assess the efficacy of the intervention. et al., 2005). These components, which act in an interactive and - To assess the implementation results based on an economic compensatory way, included preservice and inservice training, consultant/ evaluation. support, staff evaluation, facilitative administration and systems interven­ tion. Although these components might be considered to drive an 2. Methods implementation study, as IR is context-specific, in our case a review of the related literature for PC settings, depressed population, and internet 2.1. Hypothesis interventions is needed. Vis et al., identified 37 determinants clustered in six groups in the According to the literature and the established objectives the implementation process of eMental Health (eMH) interventions to following hypotheses are determined: address mood disorders in routine practice using the RE-AIM framework Implementation: (Reach, Effectiveness, Adoption, Implementation and Maintenance) (Vis 2 R. Lorente-Català et al. Internet Interventions 30 (2022) 100581 - Regarding barriers and facilitators of the implementation process, no followed in this study (Copas et al., 2015; Hemming et al., 2018). Spe­ specific hypotheses are established. Although there is some literature cifically, three sequences will be applied into 6 implementation sites. that could lead to some hypotheses, considering the context speci­ The assignment of the six clusters (two centers for each region in Spain ficity of the study, a more constructivist and unbiased understanding participating in the study) to one of the three sequences will be will be developed. randomly performed (Thompson et al., 2017). In these sequences, three - The data on the use of the platform and the questionnaire Feasibility steps or crossover can be differentiated. A step is defined as the change of Intervention Measure (FIM) will demonstrate that the intervention from the control condition to the intervention condition, which in this is adopted, feasible, and has high fidelity. study will be done at 2, 4 and 6 weeks with no transition period estab­ - The intervention will show high acceptability measured by the Sys­ lished. In the three sequences each cluster-period (b) (a group of obser­ tem Usability Scale (SUS). vations by time of measurement and cluster) will have an extension of two - The perceived appropriateness of the intervention will be high ac­ weeks leading to 30 weeks of duration of the trial (15 clusters × 2 weeks cording to the Intervention Appropriateness Measure (IAM). cluster period). Within these 30 weeks, participants will be 16 weeks - There will be preliminary indicators of sustainability of the inter­ under the treatment condition. In this case, the measurement points (Tx) vention measured by the Program Sustainability Assessment will be linked to trial steps, therefore the individuals will be assessed (PASAT) and the barriers and facilitators questionnaire. each two weeks. - The implementation of the intervention will be successful, under­ These designs are especially recommended when there is evidence standing that Smiling is fun will be normalized in the Primary Care that a treatment produces beneficial effects or it is difficult to cause settings assessed through the Normalization MeAsure Development harm (Hemming et al., 2015). SWT has the advantage that it does not Questionnaire (NoMAD). require the use of control groups, such us clusters act as own controls (Beard et al., 2015). Efficacy and cost-effectiveness. Considering the previous evidence about the efficacy and effective­ ness of the implemented intervention (Mira et al., 2019), ethical benefits - The intervention will be effective in reducing depression symptoms are found as control group receive also the intervention. Addressing severity measured by the Patient Health Questionnaire (PHQ9) and depression in PC is an imperative urgency (Haro et al., 2014), therefore improving Quality of Life (EuroQuol-5D-5L) preventing a group of people from receiving a beneficial evidence-based - The psychological intervention applied through ICTs will be cost- intervention would be unethical (Prost et al., 2015). Furthermore, SWT effective. prevent dropping out and promote higher motivation to participate as all participants benefit from the intervention (Beard et al., 2015). Be­ 2.2. Design sides promoting a greater willingness to participate, it offers more in­ formation about the effectiveness of the intervention due to all clusters The current investigation is a Hybrid Effectiveness-Implementation participate. Type II design, which allow the dual assessment of the clinical in­ Despite all the enumerates advantages, SWT designs present also tervention's effectiveness and the implementation process in different some limitations that need to be considered in the study development. degrees, adopting two co-primary aims (Green et al., 2019). Due to the control condition they present longer durations which can A Stepped Wedge Randomized controlled trial (SWT) design will be lead to an increase on the drop-out rates or the loss of motivation used. SWT represent a variation of the clinical trials randomized by (Dreischulte et al., 2013). However, some strategies could be imple­ clusters. The clusters are the unit of randomization which are defined mented to prevent it, such as professional contact in the step between into different sequences that determine the order to switch to the conditions (Prost et al., 2015). intervention condition from the control condition. In SW all the groups These designs are also influenced by temporal trends, within-cluster receive the treatment (intervention condition) with an initial period contamination and contextual and policy changes (Hemming et al., with no exposure to the intervention (control condition), followed by the 2018). One example of these contextual factors that could influence the application in a randomized order of the treatment in a staggered and study is the COVID-19 pandemic (Thome et al., 2021). These designs are sequential way. Fig. 1 shows the schema of the design that will be context-specific in a real-setting, which leads to a more complexity Fig. 1. Design of the study; stepped wedge cluster randomized trial design. 3 R. Lorente-Català et al. Internet Interventions 30 (2022) 100581 needing more coordination at logistical level and higher level of adap­ 2.3. Study population, recruitment, and eligibility criteria tation. Although one of the benefits of SWT is the continuous learning along the process and the consequent adjustments (Beard et al., 2015), According to the CONSORT guidelines, eligibility criteria for clusters these continuous adaptations can undermine the standardization of (implementations sites) and participants need to be reported (item 4a) application and the replicability of the study. To overcome this limita­ (Hemming et al., 2018). tion and promote long-term sustainability, it is necessary to register and protocolize all the actions performed during the study (Hemming et al., - Clusters level 2018; Prost et al., 2015). At cluster level the implementation sites and the cluster-level par­ 2.2.1. Implementation model ticipants are differentiated. In Fig. 2 the eligibility criteria for this study The implementation model of the study will have an especial focus to and the interaction with the designed SWT are offered. Specifically, the Hermes et al. (2019) recommendations which are focused on psy­ implementation sites focus on PC settings of three pre-determined chological interventions supported by ICTs. Autonomous Communities (Andalusia, Aragon and Balearic Islands) at Hermes and colleagues adapted the taxonomy of outcomes of Proctor urban and rural settings. The cluster-level participants are composed by (E. Proctor et al., 2011) to the field of psychological interventions sup­ the different professionals of PC such as doctors, nurses and adminis­ ported by ICTs (Hermes et al., 2019). The Proctor's taxonomy high­ trators that will be responsible for the local implementation work. All lighted the following outcomes for the implementation process: implementers will give the informed consent to participate considering acceptability, adoption, appropriateness, feasibility, fidelity, imple­ the protection of privacy, voluntary participation, and the right to mentation costs, penetration, and sustainability. However, Hermes et al. withdraw from the study at any given time. There are not exclusion (2019) pointed out the lack of successful implementation of Behavioral criteria for the cluster-level participants. Intervention Technologies, defined as those electronic devices and ser­ vices developed to address behaviors, cognitions, and emotional states. - Participants level Consequently, in order to promote their use in care settings they developed a specific theoretical framework and measurement tool for e- At individual-level participants specific inclusion and exclusion health services implementation. In this study we will follow these criteria are established. Participants (patients) will be included in the recommendations. study in case of (1) being 18 year of age or older, (2) with an established diagnosis of major depression according to the Diagnostic and Statistical Manual of Mental Disorder (DSM-5), (3) the severity of the depression is mild to moderate with a punctuation of 14 and lower on the PHQ-9, (4) Fig. 2. Cluster level eligibility process. 4 R. Lorente-Català et al. Internet Interventions 30 (2022) 100581 the depressive symptomatology has been present during two or more a = − 2k*(t − 1/t)*ρ*(1 + t/2) months, (5) had regular access to computer with Internet connection (6) understand spoken and written Spanish and (7) the participant has given b = 3N1*(1 − ρ)*ρ(1 + t) − 2k*(t − 1/t)*(1 − ρ) the informed consent. Participants will be excluded if they (1) present a disease affecting the Central Nervous System functioning, (2) suffer other psychiatric c = 3N1*(1 − ρ)2 diagnosis with the exception of anxiety pathology, or suffer severe In this study a sample m is obtained for each cluster and each psychiatric illnesses (substance dependence and abuse, psychosis, eating moment of evaluation of 57 participants. Assuming an experimental loss disorders, etc.), (3) presence of severe medical illness, uncontrolled se­ of 20 %, it will be necessary to recruit 70 participants per each center vere degenerative or infectious disease that could interfere with the and maintain at least 57 of them throughout the study. Consequently, it administration of the psychological treatment, (4) presence of delusions is needed a sample of 140 participants per region and 420 participants in or hallucinations in the time of study and (5) presence of risk of suicide. total. Regarding professionals the inclusion criteria were being health professionals in the primary care centers included in the study, accept­ 2.6. Ethics ing to participate and signing a consent form. We did not include any exclusion criteria. The study will be conducted following the guidelines of the Helsinki Regarding managers, the inclusion criteria were being the manager and Tokyo Declaration (World Medical Association, 1975, 2013). The of the primary care centers and accepting to participate in the study and participation will be completely voluntary, and the participants will not signing a consent form. No exclusion criteria were established. receive any compensation for their participation. Participants will be informed about the possibility of withdrawal with no consequences. The 2.4. Recruitment informed consent will be signed once the participants have all the in­ formation about the study. The patients will be recruited by the participating professionals in The study has been approved by the Ethics Committee of the Inves­ the PC settings. Health professionals will identify potential participants tigation of Balearic Islands (CEI), Ethics Committee of Investigation of within their patients and will explain the characteristics of the study. the Autonomous Community of Aragon (CEICA) and the Ethics Com­ Any questions will be addressed in order to ensure that they have un­ mittee of Investigation of Malaga and Northeast. In addition, the trial is derstood everything correctly and the patients interested in partici­ registered at ClinicalTrials.gov as NCT05294614. pating will sign the informed consent. The health professional will assess The implementation study will be conducted and reported according if the potential participants meet the inclusion criteria. to the Standards for Reporting Implementation Studies Statement Regarding the recruitment of the implicated agents in the imple­ (StaRI) (Pinnock et al., 2017). mentation process; health professionals, managers, and the investigators in each region will conduct this process. For the recruitment a specific session with information about the intervention and the study will be 2.7. Intervention held in each PC center to ask for participation. “Smiling is Fun” (“Sonreír es divertido” in Spanish) is a manualized 2.5. Sample size intervention protocol of ICBT using multimedia material in an interac­ tive and self-administered way for the prevention and treatment of Participants will be 420 (140 in each region AC) patients with low to depression and adjustment disorders (Botella et al., 2016). “Smiling is moderate depression. Required sample size was calculated following Fun” is made up around six main components including some trans­ recommendations for SW designs (Hemming and Taljaard, 2016). First, diagnostic components (motivation, psychoeducation, cognitive therapy it is needed to calculate the sample size required in case of an individual and relapse prevention) (Barlow et al., 2016), behavioral activation randomization. In our case, with a conservative effect size of 0.25 on the (Lejuez et al., 2001), and positive psychology to offer strategies to continuous variable, a statistical power of 80 % and an alpha of 0.05, the promote and enhance positive mood (Algoe and Fredrickson, 2011; Sin necessary size calculated with the Student's t-test would be N1 = 506 and Lyubomirsky, 2009). The program is composed of the most effective participants (394 per experimental condition). Secondly, a correction psychological techniques for stress management, to promote coping, factor (DEsw) or design effect needs to be calculated, which is multiplied emotion regulation, and resilience to learn to deal with depressive by the sample size obtained in case of individual randomization, to symptoms and daily problems. obtain the sample size necessary for the study using the following The program consists of nine interactive modules and two initial formula: modules that are presented in a pre-established sequential way and has an estimated duration between of 8 and 16 weeks. The information N1 DEsw = m(t + 1)k about the objective, contents and exercises of each module are reflected in Table A.1. In the formula, m is the necessary sample for each cluster and for each assessment moment (variable to determine), t is the number of sequences (in this case: 3), and k is the number of groups or clusters (in 2.8. Implementation study procedure this case: 6). As implementation strategies are one of the main focus of this study, DESW = 3(t + 1)(1 − ρ)[1 + ρ(tm + m − 1) ]/2(t − 1/t)[1 + p(tm/2 + m − 1) priori battery of implementation strategies has been established ac­ cording to the detected barriers by the literature (explained below). Considering the described formula for Desw, ρ is the expected cor­ However, prior to define the implementation strategies for this study, is relation between the measurements, which in this study is expected to necessary to establish the different agents in the implementation be between moderate and large (ρ = 0'25). It is possible to determine the process. numbers of clusters of each sequence (m) isolating the equation as Following the CONSORT guide the cluster and participants levels follows: have been defined. However, as this is a hybrid-implementation study different role of implementers have been determined with different m = ( − b ± √(b2 − 4ac) )/2a, where : levels of interaction. Fig. 3, reflect the multiple levels of implementation influences and implications and the interaction between the involved 5 R. Lorente-Català et al. Internet Interventions 30 (2022) 100581 Fig. 3. Implementation influences flowchart. Notes: The arrows in the flow chart establish the level of intensity in the implementation of the strategies and the interaction between the agents involved. Spe­ cifically, the continuous arrows mark a greater interaction and the dashed arrows a more punctual or secondary interaction. agents in inner and outer settings. implementation, and sustainability of EBT in a specific setting (Proctor In the outer settings, the external change agents affiliated to the et al., 2013). academic institution facilitate the implementation process by coordi­ The implementation strategies that will be applied in this study are nating the study (Damschroder et al., 2009). The main coordination reported following Proctor et al. (2013) recommendations: (Proctor tasks are conducting with the local researchers, although there is et al., 2013) and focus on the core components for implementation punctual contact with health professionals to conduct the qualitative process determined by Fixsen et al. (2005). and quantitative assessment as to offer general training. According to the barriers determined by the literature in these con­ In the implementation setting, the local researchers and the health texts, target and settings a total of 13 strategies, addressing one or professionals are the main implementers and the patients are those who several determinants will be implemented. These strategies has been receive the implemented intervention. determined following the compilation of 68 implementation strategies Local researchers have the greater weight in the implementation of grouped in six key processes (planning, educating, financing, restructuring, strategies. They are responsible of the recruitment of mental health managing quality and attending to policy context) developed by Powell and professional and the dissemination tasks of the study. On a second level, colleagues for clinical innovations in health (Powell et al., 2012). they also assess the recruited patients by health professionals and in case It should be mentioned that the strategies can suffer variation along that they met the established criteria, local researchers give access to the implementation process based on health professionals' feedback patient to the platform. Furthermore, local researchers will select the from qualitative interviews. The possible adaptations will be reported champions between health professionals. and protocolized (Hemming et al., 2018; Prost et al., 2015). Champions are the professionals that will act as a defenders and The package of the strategies designed for this study include the supporters of the study. They will be selected according to the beliefs components in Table A.2. and behaviors demonstrated towards the intervention (Damschroder et al., 2009). 2.9. Assessment plan Finally, the health professionals are those that will implement the intervention and will have the higher impact on the normalization 2.9.1. Efficacy and cost effectiveness process of the intervention. Most of the implementation strategies will Patient Health Questionnnaire-9 (PHQ-9) (Kroenke et al., 2001). be directed to impact on them (for example: attitudes, incentives, The PHQ-9 is a 9-item self-report scale questionnaire to assess the training, technical support, etc.) and therefore promote the imple­ severity of depressive symptoms. The Spanish version will be used as a mentation process. primary outcome, this presents comparable diagnostic validity to the The literature has determined some factors impeding and promoting original version, with levels of 88 % of sensitivity and 88 % of specificity the implementation process, for example, support staff, acceptance, (Diez-Quevedo et al., 2001). technical support, perceived patient's needs, leadership, policies, etc. In EuroQol-5D-5L (Ramos-Goñi et al., 2018). This questionnaire de­ accordance, different strategies have been established in order to over­ scribes health status in terms of five dimensions (mobility, self-care, come and promote the different factors. In IR implementation strategies usual activities, pain/discomfort, and anxiety/depression) according are pre-established actions and methods used to enhance the adoption, to five levels of severity. 6 R. Lorente-Català et al. Internet Interventions 30 (2022) 100581 Client Service Receipt Inventory (CSRI) (Beecham and Knapp, Fidelity: In the context of internet interventions, fidelity is defined 2001). The CSRI is a questionnaire to collect retrospective information by the expected use of clinically meaningful use. This dimension will be about the use of the healthcare resources and variables related with the measured with passive data from the online platform; number of mod­ indirect impact. The applied version is designed to collect retrospec­ ules completed, and number of tasks completed. tively the data of the use of services during the previous six months Penetration: Following Hermes et al., we will measure this dimen­ (Vázquez-Barquero et al., 1997). sion with the number of managers who actually delivered the IBT and the number of patients who accepted to use the internet intervention and 2.9.2. Implementation outcome measures completed at least 80 % of the program. Feasibility: defined as the extent to which a new treatment can be Sustainability: This dimension is defined by the extent to which a successfully used or carried out within a given agency or setting. new intervention is maintained. Feasibility of Intervention Measure (FIM) (Weiner et al., 2017). Program Sustainability Assessment Tool (PASAT) (Luke et al., This scale includes four items designed to measure feasibility of the 2014). The PASAT assess the capacity for a program sustainability in intervention. The scale has shown good psychometric properties public health during the implementation process. The test has shown with high levels of internal consistency (alpha from 0.85 to 0.91) and excellent psychometric properties with high internal consistency test-retest reliability coefficients ranged from 0.73 to 0.88. (alpha = 0.70 to 0.92). Feasibility will be also measured with passive data gathered by the Barriers and facilitators of the implementation (FBI). The FBI is a online platform about use of the program, specifically the frequency questionnaire developed specifically for this study according to the of use during the study. systematic review about the barriers and facilitators in the process implementation of evidence based intervention among the third Acceptability: defined as the perception among implementation sector (Bach-Mortensen et al., 2018). According to this review, 28 stakeholders that the intervention is useful or satisfactory. The in­ items were established to address the construct of barriers and 15 struments to assess acceptability are: items to address the facilitators. System Usability Scale (SUS) (Bangor et al., 2008). The SUS is a 10- The study variables assessed, and the instruments used are summa­ item questionnaire to measure the usability which qualitatively is rized in Table A.2. related with the quality and acceptability of the system. The usability is defined as the facility of use perceived by the users of the imple­ mented technology. 2.10. Data analysis Acceptability of Intervention Measure (AIM) (Weiner et al., 2017). This measure is composed by 12 items that measure three 2.10.1. Efficacy implementation constructs; acceptability, the appropriateness, and For the analysis of treatment efficacy, both an intention-to-treat the feasibility of the intervention. The three constructs have shown analysis and an analysis for the patients that have completed the pro­ good psychometric properties with high levels of internal consis­ tocol (80 % of the treatment) will be conducted. The analysis will tency (alpha from 0.85 to 0.91) and test-retest reliability coefficients include the description and elementary head-to-head comparisons ranged from 0.73 to 0.88. across time. Specifically, the variables will be described using descrip­ Client Satisfaction Questionnaire adapted to Internet-Based in­ tive statistics (means and 95 % confidence intervals in the case of terventions (CSQ-I) (Boß et al., 2016). The CSQ-I is an 8-item quantitative variables with a normal distribution and medians and questionnaire that assesses the general satisfaction of the partici­ interquartile ranges in the case of quantitative variables with non- pants regarding the received intervention. The internet version has normal distributions). To confirm the main hypothesis, all the vari­ shown excellent reliability (Omega = 0.93 and 0.95) (Boß et al., ables (t0-tk) will be used for the ANOVA test with the appropriate post- 2016). hoc contrasts if we compare variables with a normal distribution, or using the Kruskal-Wallis H test if variables are not normally-distributed. Appropriateness: This construct is defined as perceived fit, rele­ Finally, multivariate analyses including multi-level regression will be vance or compatibility of the EBP for a given practice setting. We will performed incorporating measures such as patients and time, if use the following measures: necessary. Normalization MeAsure Development Questionnaire (NoMAD) 2.10.2. Economic evaluation (Rapley et al., 2018; Finch et al., 2018). NoMAD is a 23-items The economic evaluation will be carried out following the recom­ questionnaire that assess the process of normalization of an inter­ mendations of the Spanish guideline for economic evaluation of health vention according to the Normalization Process Theory (May et al., technologies (López Bastida et al., 2010). The economic evaluation will 2009) focusing on four dimensions: coherence, cognitive participa­ be conducted in terms of a Cost-Effectiveness Analysis (CEA) and Cost- tion, collective action and reflexive monitoring. The NoMAD has Utility Analysis (CUA), and a societal perspective will be adopted. For shown good psychometric properties, showing high levels of internal the CEA, the effectiveness of the intervention will be estimated as consistency along the four dimensions (20 items) (alpha = 0.89). improvement of the depression, measured with the PHQ-9. Results of the Intervention Appropriateness Measure (IAM) (Weiner et al., CEA will be expressed in terms of the incremental cost-effectiveness 2017). The IAM measure appropriateness of the intervention. The ratio (ICER), calculated by dividing the difference in total costs scale has shown good psychometric properties with high levels of through the Client Service Receipt Inventory (CSRI) between the treat­ internal consistency (alpha from 0.85 to 0.91) and test-retest reli­ ment phase and the control phase by the difference in PHQ-9 scores ability coefficients ranged from 0.73 to 0.88. between both phases. For the CUA, the effectiveness of the intervention will be estimated as Quality-Adjusted Life Years (QALYs). Results of the Adoption: Defined as the intention, initial decision or action to try or CUA will be expressed in terms of the Incremental Cost-Utility Ratio employ an EBP. Adoption will be measure by the number of participants (ICUR), calculated by dividing the difference in total costs between the who accepted to use the program and actually accessed the online treatment phase and the control phase by the difference in QALYs be­ program. We will gather this information from passive data from the tween both phases. To analyze the uncertainty of the ICER and ICUR online platform. results, we will perform sensitivity analyses. 7 R. Lorente-Català et al. Internet Interventions 30 (2022) 100581 2.10.3. Implementation study the future will be determined. In congruency, our findings will support One of the main focus of this study are the implementation results, the efficacy of Smiling is Fun for reducing depressive symptomatology which are understood as the effects of the implementation strategies to and the intervention will be successfully normalized in Spanish PC set­ translate the intervention in the specific context (Proctor et al., 2011). tings, being the implementation process successful. Furthermore, our As a consequence, the implementation data of the study will be analyzed results will provide information about the potential use of ICTs related in order to understand the specific barriers and strategies for this to the cost-effectiveness of online-based psychological interventions. context. Different outcomes will be considered for implementation an­ The study will show how implementation studies are useful to establish alyses according to the adopted model (Hermes et al., 2019; Proctor the framework to address the barriers and promote the facilitators to et al., 2011). The analysis of the passive data will consist of objective and promote the acceptability, appropriateness, adoption, feasibility, fidel­ direct counts (e.g., logins, frequency of use, modules completed, task ity, penetration, and sustainability of internet psychological in­ completed, etc.). The self-report questionnaires will be scored according terventions in health services. to the questionnaires' instructions. Qualitatively data will be recorded To sum up, the present investigation could establish the first step to through interviews with the different health professional. These in­ promote the use of ICBT in PC in Spain and improve the access of the terviews will pretend to identify the difficulties, facilities and opinions population to EBPT for depression. The final goal is to reduce the high of the implementers. The interviews will be conducted using brain­ burden of this important health problem, seeking the integration of an storming and focus group method in order to detect determinants of EBPT into actual care setting. This study will help to understand the practice. Due to the changing nature of the context qualitative data will factors impeding and promoting the gap between the EBPT and the be assessed in different moments of the implementation process and implementation of these into routine practice. reported in a specific protocol (Hemming et al., 2018). Funding 3. Discussion R.L.-C. is holder of PhD grant from “Consellería de Innovación, Mental health needs of the population are not being adequately Universidades, Ciencia y Sociedad Digital, Generalitat Valencia (Spain). addressed due to the gap between research and practice and the lack of Subvenciones para la contratación de personal investigador de carácter financial and personal resources (Bauer and Kirchner, 2020). In Spain, predoctoral (ACIF/2020)”. Reference: ACIF/2020/332. this is specially problematic in PC, where it is the most prevalent dis­ Coordinated project grant from Ministerio de Ciencia, Innovación y order (Roca et al., 2009; Serrano-Blanco et al., 2010) leading to Universidades (Spain). Instituto de Salud Carlos III. Convocatoria de important personal and economical costs (Ferrari et al., 2013; Salvador- Ayudas a la acción estratégica de salud. Reference: P19/00723. Carulla et al., 2011). A.P.-A. is holder of Sara Borrell postdoctoral contract from the IR focuses on the development of implementation studies to address Instituto de Salud Carlos III. Reference: CD20/00181. the know-do gap and bring the scientific advances to health services. These studies are context-specific and focus on the barriers and facili­ tators of the implementation process (Bauer and Kirchner, 2020). Declaration of competing interest Consequently, implementation studies offer the theoretical framework to specifically address the problem found in PC, facilitating the factors The authors declare that they have no known competing financial and strategies to focus on bringing EBPT to this setting and population interests or personal relationships that could have appeared to influence (King et al., 2019). the work reported in this paper. Consequently, the intervention “Smiling is Fun” developed to address depressive symptomatology (Mira et al., 2019) is presented as Appendix A. Supplementary data an opportunity to increase the number of people receiving treatment for depression. As its efficacy and cost-effectiveness have been already Supplementary data to this article can be found online at https://doi. proven in PC, the following step is to succeed in the implementation of org/10.1016/j.invent.2022.100581. this intervention in routine practice. The advantages of IBT such as Smiling is fun have been widely References demonstrated. Nevertheless, it is necessary to answer the question how; How to translate them to daily care settings? How to get the imple­ Algoe, S.B., Fredrickson, B.L., 2011. Emotional fitness and the movement of affective mentation in public services? science from lab to field. Am.Psychol. 66 (1), 35–42. https://doi.org/10.1037/ In the study protocol, we describe a hybrid type II effectiveness- a0021720. Andersson, G., Cuijpers, P., 2009. Internet-based and other computerized psychological implementation study with a SW randomized controlled trial design to treatments for adult depression: a meta-analysis. Cogn. Behav. Ther. 38 (4), assess the implementation of an ICBT (“Smiling is Fun”) to address mild 196–205. https://doi.org/10.1080/16506070903318960. to moderate depression in PC in Spain. Andrews, G., Cuijpers, P., Craske, M.G., McEvoy, P., Titov, N., 2010. Computer therapy for the anxiety and depressive disorders is effective, acceptable and practical health The implementation study will be conducted in order to assess the care: a meta-analysis. PLOS ONE 5 (10), e13196. impact of the intervention in participants' depressive symptomatology, Aragonès, E., Piñol, J.L., Labad, A., Folch, S., Mèlich, N., 2004. Detection and manage­ the direct and indirect costs of the intervention and the different factors ment of depressive disorders in primary care in Spain. Int.J.PsychiatryMed. 34 (4), 331–343. https://doi.org/10.2190/N835-FDYX-2E2E-V8XM. (barriers and facilitators) influencing the implementation process. Note Bach-Mortensen, A.M., Lange, B.C.L., Montgomery, P., 2018. Barriers and facilitators to that the main goal of the study is not to prove the effectiveness of the implementing evidence-based interventions among third sector organisations: a intervention, as it has already been proven (Mira et al., 2019). The ul­ systematic review. Implement. Sci. 13 (1), 103. https://doi.org/10.1186/s13012- 018-0789-7. timate goal is to break the research-practice split (Kazdin, 2008), Bangor, Aaron, Kortum, P.T., Miller, J.T., 2008. The System Usability Scale (SUS): an implementing an EBPT to address the depressive symptomatology in PC empirical evaluation. Int. J. Hum. Comput. Interact. 24, 574. https://doi.org/ settings and determining the factors influencing in this process. In order 10.1080/10447310802205776. Barlow, D.H., Allen, L.B., Choate, M.L., 2016. Toward a unified treatment for emotional to assess the implementation, process the different moderator factors disorders – republished article. Behav. Ther. 47 (6), 838–853. https://doi.org/ (acceptability, appropriateness, adoption, feasibility, fidelity, penetra­ 10.1016/j.beth.2016.11.005. tion and sustainability), the implementation strategies designed and the Bauer, M.S., Kirchner, J.A., 2020. Implementation science: what is it and why should I specific contextual factors will be assessed. care? Psychiatry Res. 283 (March 2019) https://doi.org/10.1016/j. psychres.2019.04.025. If our hypothesis is confirmed, the designed implementation strate­ Beard, E., Lewis, J.J., Copas, A., Davey, C., Osrin, D., Baio, G., Thompson, J.A., gies will facilitate the implementation process and the barriers to deal in Fielding, K.L., Omar, R.Z., Ononge, S., Hargreaves, J., Prost, A., 2015. Stepped 8 R. Lorente-Català et al. Internet Interventions 30 (2022) 100581 wedge randomised controlled trials: systematic review of studies published between Hemming, K., Haines, T.P., Chilton, P.J., Girling, A.J., Lilford, R.J., 2015. The stepped 2010 and 2014. Trials 16 (1). https://doi.org/10.1186/s13063-015-0839-2. wedge cluster randomised trial: rationale, design, analysis, and reporting. BMJ 350, Beecham, Knapp, 2001. Costing psychiatric interventions. In: Measuring Mental Health 1–7. https://doi.org/10.1136/bmj.h391 (Online). Needs, pp. 200–224. Hemming, K., Taljaard, M., 2016. Sample size calculations for stepped wedge and cluster Boß, L., Lehr, D., Reis, D., Vis, C., Riper, H., Berking, M., Ebert, D.D., 2016. Reliability randomised trials: a unified approach. https://doi.org/10.1016/j. and validity of assessing user satisfaction with web-based health interventions. jclinepi.2015.08.015. J. Med. Internet Res. 18 (8), e234 https://doi.org/10.2196/jmir.5952. Hemming, K., Taljaard, M., McKenzie, J.E., Hooper, R., Copas, A., Thompson, J.A., Botella, C., Mira, A., Moragrega, I., García-Palacios, A., Bretón-López, J., Castilla, D., Dixon-Woods, M., Aldcroft, A., Doussau, A., Grayling, M., Kristunas, C., Goldstein, C. López, Riera, del Amo, A., Soler, C., Molinari, G., Quero, S., Guillén-Botella, V., E., Campbell, M.K., Girling, A., Eldridge, S., Campbell, M.J., Lilford, R.J., Weijer, C., Miralles, I., Nebot, S., Serrano, B., Majoe, D., Alcañiz, M., Baños, R.M., 2016. An Forbes, A.B., Grimshaw, J.M., 2018. Reporting of stepped wedge cluster randomised internet-based program for depression using activity and physiological sensors: ef­ trials: extension of the CONSORT 2010 statement with explanation and elaboration. ficacy, expectations, satisfaction, and ease of use. Neuropsychiatr. Dis. Treat. 12, BMJ 363. https://doi.org/10.1136/bmj.k1614 (Online). 393–406. https://doi.org/10.2147/NDT.S93315. Hermes, E.D.A., Lyon, A.R., Schueller, S.M., Glass, J.E., 2019. Measuring the imple­ Copas, A.J., Lewis, J.J., Thompson, J.A., Davey, C., Baio, G., Hargreaves, J.R., 2015. mentation of behavioral intervention technologies: recharacterization of established Designing a stepped wedge trial: three main designs, carry-over effects and ran­ outcomes. J. Med. Internet Res. 21 (1) https://doi.org/10.2196/11752. domisation approaches. Trials 16 (1), 1–12. https://doi.org/10.1186/s13063-015- Kaltenthaler, E., Parry, G., Beverley, C., Ferriter, M., 2008. Computerised cognitive- 0842-7. behavioural therapy for depression: systematic review. Br.J.Psychiatry 193 (3), Cuijpers, P., Andersson, G., Donker, T., Van Straten, A., 2011. Psychological treatment of 181–184. https://doi.org/10.1192/bjp.bp.106.025981. depression: results of a series of meta-analyses. Nordic J.Psychiatry 65 (6), 354–364. Karyotaki, E., Kemmeren, L., Riper, H., Twisk, J., Hoogendoorn, A., Kleiboer, A., https://doi.org/10.3109/08039488.2011.596570. Mira, A., Mackinnon, A., Meyer, B., Botella, C., Littlewood, E., Andersson, G., Cuijpers, P., Gentili, C., 2017. Psychological treatments are as effective as pharmaco­ Christensen, H., Klein, J.P., Schroeder, J., Breton-Lopez, J., Scheider, J., Griffiths, K., therapies in the treatment of adult depression: a summary from randomized clinical Farrer, L., Huibers, M.J.H., Phillips, R., Gilbody, S., Moritz, S., Berger, T., Pop, V., trials and neuroscience evidence. Res.Psychother.: Psychopathol.ProcessOutcome 20 Spek, V., Cuijpers, P., 2018. Is self-guided internet-based cognitive behavioural (2), 147–152. https://doi.org/10.4081/ripppo.2017.273. therapy (iCBT) harmful? An individual participant data meta-analysis. Psychol. Med. Cuijpers, P., van Straten, A., van Schaik, A., Andersson, G., 2009. Psychological treat­ 48 (15), 2456–2466. https://doi.org/10.1017/S0033291718000648. ment of depression in primary care: a meta-analysis. Br. J. Gen. Pract. 59 (559), Karyotaki, E., Riper, H., Twisk, J., Hoogendoorn, A., Kleiboer, A., Mira, A., 120–127. https://doi.org/10.3399/bjgp09X395139. Mackinnon, A., Meyer, B., Botella, C., Littlewood, E., Andersson, G., Christensen, H., Damschroder, L.J., Aron, D.C., Keith, R.E., Kirsh, S.R., Alexander, J.A., Lowery, J.C., Klein, J.P., Schröder, J., Bretón-López, J., Scheider, J., Griffiths, K., Farrer, L., 2009. Fostering implementation of health services research findings into practice: a Huibers, M.J., Phillips, R., Gilbody, S., Moritz, S., Berger, T., Pop, V., Spek, V., consolidated framework for advancing implementation science. Implement. Sci. 4 Cuijpers, P., 2017 Apr 1. Efficacy of Self-guided Internet-Based Cognitive Behavioral (1), 1–15. https://doi.org/10.1186/1748-5908-4-50. Therapy in the Treatment of Depressive Symptoms: A Meta-analysis of Individual Diez-Quevedo, C., Rangil, T., Sanchez-Planell, L., Kroenke, K., Spitzer, R.L., 2001. Vali­ Participant Data. JAMA Psychiatry 74 (4), 351–359. https://doi.org/10.1001/ dation and utility of the patient health questionnaire in diagnosing mental disorders jamapsychiatry.2017.0044. PMID: 28241179. in 1003 general hospital Spanish inpatients. Psychosomatic medicine 63 (4), Kazdin, A.E., 2008. Evidence-based treatment and practice: new opportunities to bridge 679–686. https://doi.org/10.1097/00006842-200107000-00021. clinical research and practice, enhance the knowledge base, and improve patient Dreischulte, T., Grant, A., Donnan, P., Guthrie, B., 2013. Pro's and con's of the stepped care. Am. Psychol. 63 (3), 146–159. https://doi.org/10.1037/0003-066X.63.3.146. wedge design in cluster randomised trials of quality improvement interventions: two Kazdin, A.E., Blase, S.L., 2011. Rebooting psychotherapy research and practice to reduce current examples. Trials 14 (S1), 2013. https://doi.org/10.1186/1745-6215-14-s1- the burden of mental illness. Perspect.Psychol.Sci. 6 (1), 21–37. https://doi.org/ o87. 10.1177/1745691610393527. Drozd, F., Vaskinn, L., Bergsund, H.B., Haga, S.M., Slinning, K., Bjørkli, C.A., 2016. The Kazdin, A.E., Rabbitt, S.M., 2013. Novel models for delivering mental health services and implementation of internet interventions for depression: a scoping review. J. Med. reducing the burdens of mental illness. Clin. Psychol. Sci. 1 (2), 170–191. https:// Internet Res. 18 (9), 1–18. https://doi.org/10.2196/jmir.5670. doi.org/10.1177/2167702612463566. Fernández, A., Pinto-Meza, A., Bellón, J.A., Roura-Poch, P., Haro, J.M., Autonell, J., King, K.M., Pullmann, M.D., Lyon, A.R., Dorsey, S., Lewis, C.C., 2019. Using imple­ Palao, D.J., Peñarrubia, M.T., Fernández, R., Blanco, E., Luciano, J.V., Serrano- mentation science to close the gap between the optimal and typical practice of Blanco, A., 2010. Is major depression adequately diagnosed and treated by general quantitative methods in clinical science. J. Abnorm. Psychol. 128 (6), 547–562. practitioners? Results from an epidemiological study. Gen. Hosp. Psychiatry 32 (2), https://doi.org/10.1037/abn0000417. 201–209. https://doi.org/10.1016/j.genhosppsych.2009.11.015. Kroenke, K., Spitzer, R.L., Williams, J.B., 2001. The PHQ-9: validity of a brief depression Ferrari, A.J., Charlson, F.J., Norman, R.E., Patten, S.B., Freedman, G., Murray, C.J.L., severity measure. J. Gen. Intern. Med. 16 (9), 606–613. https://doi.org/10.1046/ Vos, T., Whiteford, H.A., 2013. Burden of depressive disorders by country, sex, age, j.1525-1497.2001.016009606.x. and year: findings from the global burden of disease study 2010. PLoS Med. 10 (11), Lejuez, C.W., Hopko, D.R., Hopko, S.D., 2001. A brief behavioral activation treatment for e1001547 https://doi.org/10.1371/journal.pmed.1001547. depression.Treatment manual. Behav. Modif. 25 (2), 255–286. https://doi.org/ Finch, T.L., Girling, M., May, C.R., Mair, F.S., Murray, E., Treweek, S., McColl, E., 10.1177/0145445501252005. Steen, I.N., Cook, C., Vernazza, C.R., Mackintosh, N., Sharma, S., Barbery, G., López Bastida, J., Oliva, J., Antoñanzas, F., García-Altés, A., Gisbert, R., Mar, J., Puig- Steele, J., Rapley, T., 2018. Improving the normalization of complex interventions: Junoy, J., 2010. Propuesta de guía para la evaluación económica aplicada a las part 2 - validation of the NoMAD instrument for assessing implementation work tecnologías sanitarias. Gac. Sanit. 24 (2), 154–170. https://doi.org/10.1016/j. based on normalization process theory (NPT). BMC Med. Res. Methodol. 18 (1), 135. gaceta.2009.07.011. https://doi.org/10.1186/s12874-018-0591-x. Luke, D., Calhoun, A., Robichaux, C., Elliott, M., Moreland-Russell, S., 2014. The pro­ Fixsen, D.L., Naoom, S., Blase, K., Friedman, R., Wallace, F., 2005. Implementation gram sustainability assessment tool: a new instrument for public health programs. research: a synthesis of the literature. In: The National Implementation Research Prev. Chronic Dis. 11, E12. https://doi.org/10.5888/pcd11.130184. Network. Mathers, C.D., Loncar, D., 2006. Projections of global mortality and burden of disease Folker, A.P., Mathiasen, K., Lauridsen, S.M., Stenderup, E., Dozeman, E., Folker, M.P., from 2002 to 2030. PLoS Med. 3 (11), 2011–2030. https://doi.org/10.1371/journal. 2018. Implementing internet-delivered cognitive behavior therapy for common pmed.0030442. mental health disorders: a comparative case study of implementation challenges May, C.R., Johnson, M., Finch, T., 2016. Implementation, context and complexity. perceived by therapists and managers in five European internet services. Internet Implement.Sci. 11 (1), 141. https://doi.org/10.1186/s13012-016-0506-3. Interv. 11 (December 2017), 60–70. https://doi.org/10.1016/j.invent.2018.02.001. May, C.R., Mair, F., Finch, T., MacFarlane, A., Dowrick, C., Treweek, S., Rapley, T., Gabilondo, A., Rojas-Farreras, S., Rodríguez, A., Fernández, A., Pinto-Meza, A., Ballini, L., Ong, B.N., Rogers, A., Murray, E., Elwyn, G., Légaré, F., Gunn, J., Vilagut, G., Haro, J.M., Alonso, J., 2011. Use of primary and specialized mental Montori, V.M., 2009. Development of a theory of implementation and integration: health care for a major depressive episode in Spain by ESEMeD respondents. Psy­ normalization process theory. Implement. Sci. 4 (1), 1–9. https://doi.org/10.1186/ chiatr. Serv. 62 (2), 152–161. https://doi.org/10.1176/ps.62.2.pss6202_0152. 1748-5908-4-29. Green, B.B., Coronado, G.D., Schwartz, M., Coury, J., Baldwin, L., 2019. In: Using a Mira, A., Bretón-López, J., Enrique, Á., Castilla, D., García-Palacios, A., Baños, R., continuum of hybrid effectiveness- implementation studies to put research- tested Botella, C., 2018. Exploring the incorporation of a positive psychology component in colorectal screening interventions into practice, pp. 1–9. a cognitive behavioral internet-based program for depressive symptoms. Results Grol, R.P.T.M., Bosch, M.C., Hulscher, M.E.J.L., Eccles, M.P., Wensing, M., 2007. Plan­ throughout the intervention process. Front. Psychol. 9, 2360. https://doi.org/ ning and studying improvement in patient care: the use of theoretical perspectives. 10.3389/fpsyg.2018.02360. Milbank Q. 85 (1), 93–138. https://doi.org/10.1111/j.1468-0009.2007.00478.x. Mira, A., Bretón-López, J., García-Palacios, A., Quero, S., Baños, R.M., Botella, C., 2017. Haro, J.M., Ayuso-Mateos, J.L., Bitter, I., Demotes-Mainard, J., Leboyer, M., Lewis, S.W., An internet-based program for depressive symptoms using human and automated Linszen, D., Maj, M., McDaid, D., Meyer-Lindenberg, A., Robbins, T.W., support: a randomized controlled trial. Neuropsychiatr. Dis. Treat. 13, 987–1006. Schumann, G., Thornicroft, G., Van Der Feltz-Cornelis, C., Van Os, J., Wahlbeck, K., https://doi.org/10.2147/NDT.S130994. Wittchen, H.-U., Wykes, T., Arango, C., Bickenbach, J., Brunn, M., Cammarata, P., Mira, A., Soler, C., Alda, M., Banos, R., Castilla, D., Castro, A., Garcia-Campayo, J., Chevreul, K., Evans-Lacko, S., Finocchiaro, C., Fiorillo, A., Forsman, A.K., Hazo, J.- Garcia-Palacios, A., Gili, M., Hurtado, M., amp, check, et al., 2019. B., Knappe, S., Kuepper, R., Luciano, M., Miret, M., Obradors-Tarragó, C., Exploring the relationship between the acceptability of an internet-based interven­ Pagano, G., Papp, S., Walker-Tilley, T., 2014. ROAMER: roadmap for mental health tion for depression in primary care and clinical outcomes: secondary analysis of a research in Europe. Int. J. Methods Psychiatr. Res. 23 Suppl 1 (Suppl. 1), 1–14. randomized controlled trial. Front. Psychiatry 10 (MAY). https://doi.org/10.3389/ https://doi.org/10.1002/mpr.1406. fpsyt.2019.00325. 9 R. Lorente-Català et al. Internet Interventions 30 (2022) 100581 Morris, Z.S., Wooding, S., Grant, J., 2011. The answer is 17 years, what is the question: trial designs. Clin.Trials 14 (6), 639–647. https://doi.org/10.1177/ understanding time lags in translational research. J.R.Soc.Med. 104 (12), 510–520. 1740774517723921. https://doi.org/10.1258/jrsm.2011.110180. Titov, N., 2011. Internet-delivered psychotherapy for depression in adults. Curr.Opin. A guide to treatments that work, 4th ed. In: Nathan, P.E., Gorman, J.M. (Eds.), 2015. Psychiatry 24 (1), 18–23. https://doi.org/10.1097/YCO.0b013e32833ed18f. A Guide to Treatments That Work, 4th ed. Oxford University Press, p. xxvii. Titzler, I., Saruhanjan, K., Berking, M., Riper, H., Ebert, D.D., 2018. Barriers and facili­ 956–xxvii, 956. tators for the implementation of blended psychotherapy for depression: a qualitative Pakenham-Walsh, N., 2004. Learning from one another to bridge the “know-do gap”. pilot study of therapists' perspective. Internet Interv. 12, 150–164. https://doi.org/ BMJ 329 (7475), 1189. https://doi.org/10.1136/bmj.329.7475.1189. 10.1016/j.invent.2018.01.002. Pinnock, H., Barwick, M., Carpenter, C.R., Eldridge, S., Grandes, G., Griffiths, C.J., Van Den Driessen Mareeuw, F., Vaandrager, L., Klerkx, L., Naaldenberg, J., Koelen, M., Rycroft-Malone, J., Meissner, P., Murray, E., Patel, A., Sheikh, A., Taylor, S.J.C., 2015. Beyond bridging the know-do gap: a qualitative study of systemic interaction 2017. Standards for Reporting Implementation Studies (StaRI): explanation and to foster knowledge exchange in the public health sector in the Netherlands. BMC elaboration document. BMJ Open 7 (4), e013318. https://doi.org/10.1136/ Public Health 15 (1), 1–15. https://doi.org/10.1186/s12889-015-2271-7. bmjopen-2016-013318. Vázquez-Barquero, J., Gaite, L., Cuesta, J., Garcia Usieto, E., Knapp, M., Beecham, J., Powell, B.J., McMillen, J.C., Proctor, E.K., Carpenter, C.R., Griffey, R.T., Bunger, A.C., 1997. Versión española del CSRI: una entrevista Para la evaluación de costes en Glass, J.E., York, J.L., 2012. A compilation of strategies for implementing clinical salud mental. Arch.Neurobiol. 60 (2), 171–184. innovations in health and mental health. Med. Care Res. Rev. 69 (2), 123–157. Vis, C., Mol, M., Kleiboer, A., Bührmann, L., Finch, T., Smit, J., Riper, H., 2018. https://doi.org/10.1177/1077558711430690. Improving implementation of eMental health for mood disorders in routine practice: Proctor, E.K., Powell, B.J., McMillen, J.C., 2013. Implementation strategies: recom­ systematic review of barriers and facilitating factors. JMIR Ment. Health 5 (1). https:// Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., doi.org/10.2196/mental.9769. Hensley, M., 2011. Outcomes for implementation research: conceptual distinctions, Weiner, B.J., Lewis, C.C., Stanick, C., Powell, B.J., Dorsey, C.N., Clary, A.S., Boynton, M. measurement challenges, and research agenda. Admin. Pol. Ment. Health 38 (2), H., Halko, H., 2017. Psychometric assessment of three newly developed imple­ 65–76. https://doi.org/10.1007/s10488-010-0319-7. mentation outcome measures. Implement. Sci. 12 (1), 1–13. https://doi.org/ Prost, A., Binik, A., Abubakar, I., Roy, A., De Allegri, M., Mouchoux, C., Dreischulte, T., 10.1186/s13012-017-0635-3. Ayles, H., Lewis, J.J., Osrin, D., 2015. Logistic, ethical, and political dimensions of Wilhelmsen, M., Høifødt, R.S., Kolstrup, N., Waterloo, K., Eisemann, M., Chenhall, R., stepped wedge trials: critical review and case studies. Trials 16 (1). https://doi.org/ Risør, M.B., 2014. Norwegian general practitioners' perspectives on implementation 10.1186/s13063-015-0837-4. of a guided web-based cognitive behavioral therapy for depression: a qualitative Ramos-Goñi, J.M., Craig, B.M., Oppe, M., Ramallo-Fariña, Y., Pinto-Prades, J.L., Luo, N., study. J. Med. Internet Res. 16 (9), e208 https://doi.org/10.2196/jmir.3556. Rivero-Arias, O., 2018. Handling data quality issues to estimate the Spanish EQ-5D- World Medical Association, 1975. WMA Declaration of Tokyo - Guidelines for Physicians 5L value set using a hybrid interval regression approach. ValueHealth 21 (5), Concerning Torture And Other Cruel, Inhuman Or Degrading Treatment or Punish­ 596–604. https://doi.org/10.1016/j.jval.2017.10.023. ment in Relation to Detention And Imprisonment, Adopted by the 29th World Rapley, T., Girling, M., Mair, F.S., Murray, E., Treweek, S., McColl, E., Steen, I.N., May, C. Medical Assembly, Tokyo, Japan. October 1975, 1. R., Finch, T.L., 2018. Improving the normalization of complex interventions: part 1 - World Medical Association, 2013. World Medical Association Declaration of Helsinki: development of the NoMAD instrument for assessing implementation work based on ethical principles for medical research involving human subjects. JAMA 310 (20), normalization process theory (NPT). BMC Med. Res. Methodol. 18 (1), 133. https:// 2191–2194. https://doi.org/10.1001/jama.2013.281053. doi.org/10.1186/s12874-018-0590-y. Richards, D., Richardson, T., 2012. Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin. Psychol. Rev. 32 (4), Rosa Lorente-Catalàa,*, Margalida Gilib,c,d, Yolanda López-Del-Hoyoe,f, 329–342. https://doi.org/10.1016/j.cpr.2012.02.004. Fermín Mayoral-Cleriesg, Adrián Perez-Arandaf,h, Adoración Castrob,c, Roca, M., Gili, M., Garcia-Garcia, M., Salva, J., Vives, M., Garcia Campayo, J., Comas, A., Esperanza Varela-Morenog, Rosa M. Bañosi,j, Miquel Rocab,c,d, 2009. Prevalence and comorbidity of common mental disorders in primary care. J. Affect. Disord. 119 (1–3), 52–58. https://doi.org/10.1016/j.jad.2009.03.014. Alicia Monreal-Bartoloméd,f, Azucena García-Palaciosa,j a Romero-Sanchiz, P., Nogueira-Arjona, R., García-Ruiz, A., Luciano, J.V., Campayo, J.G., Department of Basic and Clinical Psychology, and Psychobiology, Gili, M., Botella, C., Baños, R., Castro, A., López-Del-Hoyo, Y., Ara, M.Á.P., Modrego- Universitat Jaume I, Castellon, Spain Alarcón, M., Cleríes, F.M., 2017. Economic evaluation of a guided and unguided b internet-based CBT intervention for major depression: results from a multicenter, Health Research Institute of the Balearic Islands (IdiSBa), Hospital three-armed randomized controlled trial conducted in primary care. PLoS ONE 12 Universitario Son Espases, Edificio S, 07120 Palma de Mallorca, Spain c (2), 1–15. https://doi.org/10.1371/journal.pone.0172741. Research Institute of Health Sciences (IUNICS), University of Balearic Salvador-Carulla, L., Bendeck, M., Fernandez, A., Albertí-Casas, C., Sabes-Figuera, R., Islands, 07122 Palma de Mallorca, Spain Molina, C., Knapp, M., 2011. Cost of depression in Catalonia (Spain). J. Affect. d Disord. 132, 130–138. https://doi.org/10.1016/j.jad.2011.02.019. Primary Care Prevention and Health Promotion Research Network, Serrano-Blanco, A., Palao, D.J., Luciano, J.V., Pinto-Meza, A., Luján, L., Fernández, A., RedIAPP, 28029 Madrid, Spain Roura, P., Bertsch, J., Mercader, M., Haro, J.M., 2010. Prevalence of mental disor­ e Department of Psychology and Sociology, University of Zaragoza, Spain ders in primary care: results from the diagnosis and treatment of mental disorders in f primary care study (DASMAP). Soc. Psychiatry Psychiatr. Epidemiol. 45 (2), Institute for Health Research Aragón (IIS Aragón), 50009 Zaragoza, Spain g 201–210. https://doi.org/10.1007/s00127-009-0056-y. Mental Health Department, Biomedical Research Institute of Malaga Sin, N.L., Lyubomirsky, S., 2009. Enhancing well-being and alleviating depressive (IBIMA), University Regional Hospital of Malaga, Spain symptoms with positive psychology interventions: a practice-friendly meta-analysis. h J. Clin. Psychol. 65 (5), 467–487. https://doi.org/10.1002/jclp.20593. Department of Basic, Developmental and Educational Psychology, Thome, J., Deloyer, J., Coogan, A.N., Bailey-Rodriguez, D., da Cruz E Silva, O.A.B., Autonomous University of Barcelona, 08913 Cerdanyola del Vallès, Spain Faltraco, F., Grima, C., Gudjonsson, S.O., Hanon, C., Hollý, M., Joosten, J., i Instituto Polibienestar, University of Valencia, Valencia, Spain Karlsson, I., Kelemen, G., Korman, M., Krysta, K., Lichterman, B., Loganovsky, K., j Marazziti, D., Maraitou, M., Mertens deWilmars, S., Reunamen, M., Rexhaj, S., The Spanish Biomedical Research Centre in Physiopathology of Obesity and Sancaktar, M., Sempere, J., Tournier, I., Weynant, E., Vis, C., Lebas, M.C., Fond- Nutrition (CIBERObn), Instituto de Salud Carlos III, Madrid, Spain Harmant, L., 2021 Sep. The impact of the early phase of the COVID-19 pandemic on mental-health services in Europe. World J. Biol. Psychiatry 22 (7), 516–525. https:// * doi.org/10.1080/15622975.2020.1844290. Epub 2020 Dec 23. PMID: 33143529. Corresponding author. Thompson, J.A., Fielding, K., Hargreaves, J., Copas, A., 2017. The optimal design of E-mail address: rlorente@uji.es (R. Lorente-Català). stepped wedge trials with equal allocation to sequences and a comparison to other 10