Ruiz Díaz et al. BMC Health Services Research (2020) 20:354 https://doi.org/10.1186/s12913-020-05216-3 RESEARCH ARTICLE Open Access Patient satisfaction with remote monitoring of cardiac implantable electronic devices: the VALIOSA questionnaire Miguel A. Ruiz Díaz1*, Marta Egea García2, Roberto Muñoz Aguilera3, Xavier Viñolas Prat4, Jorge Silvestre García5, María Álvarez Orozco2, José Martínez Ferrer6 and On behalf of the VALIOSA study group Abstract Background: Remote monitoring of cardiac implantable electronic devices (CIEDs) has demonstrated substantial benefits. Treatment guidelines have therefore endorsed its use and is being increasingly adopted in the clinical setting, but the level of satisfaction they convey remains still unknown. We developed and validated a questionnaire to measure patient satisfaction with remote monitoring using Medtronic CareLink® Network and assessed its internal reliability and dimensional validity. Methods: After a thorough literature review, cognitive debriefing of 18 patients, and an expert panel discussion, a 30-item instrument was proposed and grouped into 5 dimensions (items): 1- Information on cardiac condition (3), 2- Device convenience (3), 3- Transmission process (6), 4- Satisfaction with medical monitoring (8), and 5- General opinions (10). Correlation with the visual analog scale (VAS), overall health related quality of life (HRQoL) measured by the EuroQoL-5D accompanied by the VAS as well as with the Medical Outcomes Study (MOS) SF-36 were assessed. Psychometric properties, exploratory factor analysis and a second order confirmatory factor analysis (a hierarchical CFA with a general common factor explaining the relations between the first order common factors, See Figure 1) were estimated. Models were assessed based on item loading size, sign and statistical significance, and goodness-of-fit statistics. Results: A total of 186 patients (77% male) with a mean age of 66.03 (SD = 13.94) years were assessed. 48% had implantable cardioverter-defibrillators, 24% had pacemakers, and 29% had cardiac resynchronization therapy devices. An overall Cronbach’s α = 0.893 was achieved, with acceptable reliabilities for isolated dimensions. Correlations with corresponding VAS scales were meaningful and significant (p < 0.01). The second order factor solution yielded good goodness-of-fit indexes (χ2/df = 1.44, CFI = 0.96, TLI = 0.95, RMSEA = 0.05). Satisfaction with remote monitoring was not related to HRQoL (r < 0.05), except for the correlation between the SF-36 mental component and the information on cardiac condition dimension (r = 0.263, p < 0.001). Conclusions: The 30-item questionnaire showed good reliability and validity to assess satisfaction with remote monitoring in patients with CIEDs. Keywords: Patient satisfaction, Patient reported outcomes, Remote monitoring, Cardiac implantable electronic devices, Patient health questionnaire * Correspondence: miguel.ruiz@uam.es 1 Department of Methodology, School of Psychology, Universidad Autónoma de Madrid, Madrid, Spain Full list of author information is available at the end of the article © The Author(s). 2020 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. Ruiz Díaz et al. BMC Health Services Research (2020) 20:354 Page 2 of 9 Background Despite the increasing interest in patient opinions and Technology available for managing cardiac implantable concerns, to date there is no validated questionnaire spe- electronic devices (CIED) has advanced considerably. In cifically designed to measure patient satisfaction with this sense, remote monitoring (RM) represents a com- CIED RM systems. We believe that patient satisfaction plement to routine in-office care that provides access to measurement is not only a compromise with quality a tremendous wealth of information recorded and stored management for health service providers [13] but also a by CIEDs such as device performance, including history need towards identifying problematic users, and particu- of cardiac arrhythmias, battery and lead parameters larly among those who do not make automated trans- without face-to-face interaction. Remote monitoring missions (like pacemaker users). enables remote feedback from device to physician and The aim of the present study was to develop a new allows for continuous remote monitoring, which im- questionnaire with proven psychometric properties that proves patient safety and care by allowing any device measures patient satisfaction with RM in patients with problems to be detected immediately [1]. In addition to CIEDs, focusing on a specific RM system, CareLink® Net- clinical benefits, RM has been shown to allow for longer work (Medtronic, Minneapolis, MN), currently used by intervals between in-office visits and a shorter visit dur- more than 1,300,000 patients from more than 1200 health ation as querying the device is no longer necessary, thus centers worldwide in about 80 different countries [14]. lightening the burden of in-hospital follow-up of patients with implanted devices, with the consequent cost reduc- Methods tion [2, 3] Thanks to demonstrated substantial benefits, The present research was designed as an observational, treatment guidelines have endorsed its use for all eligible two-stage, cross-sectional, multicenter study to develop patients and RM is being increasingly adopted in the a specific instrument for the measurement of patient sat- clinical setting [4]. isfaction with RM of CIED. Patients with implanted Despite the existing clinical and economic evidence pacemakers (PM), ICD, and CRT devices who had been available, patient reported outcomes have received little using CareLink® Network for at least 2 months before attention, even though they are expected to play a more recruitment were included. The study was conducted prominent role in assessing performance and determining under actual treatment conditions for their disorder in the comparative effectiveness of treatment alternatives, in clinical practice. part because of a growing emphasis on patient-centered The Medtronic CareLink® Network is the Internet- care and value-based procurement initiatives. based remote monitoring service for patients with Med- In particular, patient satisfaction represents an import- tronic implanted cardiac devices. It allows patients a ant measure of the extent to which a patient is satisfied convenient timely connection to their clinic using the with all aspects of healthcare delivery that are of rele- person’s monitor/ smartphone/ tablet to collect and vance to health, which could be related with the quality transmit device data. A hand-held smart reader is used of care patients receive. It has been shown that patient to collect data from the implanted device. The smart satisfaction affects patients’ health-related decisions and reader communicates with the monitor and, through a treatment-related behaviors (like appropriate use of ser- software application, it transfers the data to the CareLink® vices, correct medication use or treatment continuation), Network. All the information is recorded and reviewed by and consequently, impacting substantially the success of the nurses through the online platform. In case there is a treatment outcomes [5]. critical event, the patient is called by the medical staff to To date, few publications evaluating patient satisfac- visit the physician (Figure 6 Additional file 3). tion associated with RM of patient with CIEDs have The study was performed following the Helsinki- been published. Large registries like ALTITUDE and Tokyo-Venice guidelines for human research. PREDICT-RM did not include patient reported out- comes [6, 7]; besides these registries, certain observa- Development phase: study design and participants tional studies have considered measures of health status, The aim of the development phase was to create and de- satisfaction and/or acceptation, remote patient monitor- sign the VALIOSA questionnaire incorporating the pa- ing experiences, and preferences for follow-up but using tient perspective. a self-designed questionnaire that did not undergo a For that purpose, a thorough literature review and pa- strict validation process [8–11]. There is currently a tient focus groups were conducted to identify relevant European randomized controlled trial ongoing (RE- contents and topics addressing treatment satisfaction MOTE-CIED) designed to examine the patient perspec- and patient management. tive of RM but it only covers patients with implantable Two focus groups were held at two different hospitals. cardioverter-defibrillators (ICD) and cardiac resynchro- The first one took place at Hospital Universitario de nization therapy devices (CRT) [12]. Araba (Vitoria) with 10 patients and the other one at Ruiz Díaz et al. BMC Health Services Research (2020) 20:354 Page 3 of 9 Hospital Infanta Leonor (Madrid) with 8 patients. Patients Sample size was determined based on the anticipated were recruited at random from those with a scheduled number of dimensions, the number of items per dimen- visit in the selected week and balancing for gender and sion (3:1) [17, 18], the number of patients per item (4:1), age. The topics discussed included symptoms and disease [19, 20] and the sample representativeness (n > 150) [21]. burden, daily limitations, device-related discomfort, life- A sample size of 180 patients with complete information style changes, impact on social relationships, family and was deemed necessary. specialized support, and appropriate follow-up. Patient contributions were summarized following a semantic re- Psychometric properties of final version duction process by two independent content specialists. Dimension and overall scale reliability (internal consistency) As a result, a preliminary questionnaire of 52 items was was estimated by computing Cronbach’s alpha and 95% proposed. confidence interval intra-class correlation coefficient (ICC) This raw version underwent a discussion and semantic for internal consistency. refinement process by an expert panel consisting of four To test the construct validity of the questionnaire, a cardiologists, one psychometrician and one pharmacist. confirmatory factor analysis was performed including all Redundant and inappropriate items were discarded and items, assuming a second order factor structure with replaced by equivalent items and was reduced to 37 items. items as indicators of the proposed first order dimen- This initial set of 37 items comprised the following sions and with all dimension loading in an overall satis- RM aspects: information on monitor use, training and faction second order dimension. The robust weighted convenience of monitor use, problems with the trans- least squares estimation method was used. mission process, satisfaction with medical care and RM, The following psychometric properties were studied. and benefits and impact on activities of daily living. Feasibility: response time, floor-ceiling effect, and miss- The reviewed version was administered to a pilot sam- ing values distribution; Internal Consistency/Reliability: ple consisting of 10 patients to check comprehension Cronbach’s alpha, item correlations, adjusted item-total and acceptance, and the information gathered was used correlation, ICC and omega [22]; Construct Validity: to refine the items. The resulting version was composed Confirmatory factor analysis; Concurrent Validity: Correl- of 30 items grouped into 5 dimensions (number of items): ation between the new questionnaire (VALIOSA) and the information (3), monitor convenience (3), monitor hand- VAS measures; Convergent and construct Validity: Correl- ling and transmission process (6), disease follow-up (7), ation between similar/dissimilar dimensions of the VALIOSA, general impression and benefits (11). A panel composed MOS SF-36 dimensions and corresponding VAS measures. by 9 specialists (cardiologists, nurses and psychometri- cians) assessed content validity, scoring item adequacy for Instruments measuring each content dimension and the unidimen- The final refined questionnaire, named VALIOSA (see sional [15] and multidimensional [16] item-domain con- Additional files 1 & 2), was included in a clinical record gruence indexes were computed. form along with questions about patient social and demographic characteristics, relevant clinical informa- Validation phase: study design and participants tion, the MOS SF-36, and 6 VAS. The aim of the validation phase was to administer the 30- The VALIOSA questionnaire is a 30-item, self-completed item version to a representative sample of patients to check questionnaire measured on a 5-point Likert scale (0 = No, item fit to the proposed construct, check psychometric nothing, 4 = Yes, a lot) to measure patient satisfaction with properties of the preliminary questionnaire designed in RM. It is structured in 5 dimensions: 1- Information on the development phase, and reduce the number of items cardiac condition (3 items), 2- Device convenience (3 (if necessary) in accordance with the proposed construct. items), 3- Transmission process (6 items), 4- Satisfaction with medical monitoring (8 items), and 5- General opin- Patients ions (10 items). Dimension scores are obtained by adding Three samples of patients were recruited at 4 hospitals item scores for that dimension and transforming the ob- in 3 regions of Spain. A pilot sample of 10 patients was tained score to a 0–100 common metric. The rescaled recruited to check item comprehension and to identify score is obtained using the following transformation: possible response problems. Finally, a sample of 186 patients was recruited to measure psychometric proper- Y obs −Y min Y0 ¼  100 ties. All patients had to meet selection criteria: being Y max −Y min older than 18 years of age, being implanted with a CIED, having more than 2 months of experience using RM, Where Ymax = maximum dimension score, Ymin = 0 being able to understand and answer questionnaires in (minimum score dimension), Yobs = observed dimension Spanish, and having no cognitive impairment. score (algebraic summation), and Y′ = transformed score. Ruiz Díaz et al. BMC Health Services Research (2020) 20:354 Page 4 of 9 Scores for items 9, 10, 12, and 19 need to be reversed be- 48% had ICD, 29% had PM, and 24% had CRT devices. fore summation. An overall score can be calculated by Nearly 77% were experienced in the use of CareLink® averaging the dimension scores. Network, and 23.3% were naïve to this RM system. Additionally, five 15-cm VAS (0 = Completely unsatis- Other descriptive variables are presented in Table 1. fied, 10 = Completely satisfied), with 10 marks along the scale, were used as convergent measures of the same Acceptance and responsiveness concepts contained in the dimensions. The HRQoL VAS Most patients (78.5%) fully understood all items and left accompanying the EuroQoL-5D [23] was used to meas- no blank responses, while 14.5% left one item blank and ure overall health related quality of life and as a distinct 3.8% left two. Two patients were excluded, because they measure from satisfaction. HRQoL was also measured did not answer any question or left more than 80% using the MOS SF-36 [24]. This is a generic instrument blank. Items with the highest non-response rates were measuring 8 dimensions arranged in 2 summary compo- item 11 (13%) about getting support for technical prob- nents (physical and mental) using 36 items. lems and item 12 (5%) about having trouble on holidays. All analyses were performed using IBM SPSS 20, We assumed that these items were left blank due to in- AMOS 20, and Mplus 7.31 software. applicability. The full response rate was slightly lower (75.8%) for the MOS SF-36. Results Almost all items presented a unimodal skewed distri- Development phase bution of responses across response categories with a Focus groups marked ceiling/floor effect (more than 50% of responses Of the 18 participants in the 2 focus groups, 78% were in the extreme category), and only two items did not ob- male, 56% had PM, and 44% had ICD. Mean age was 67.5 tain responses in both range extreme categories. Dimen- years (SD = 7.1, min. = 54, max. = 80), mean time since im- sion average scores ranged within 82.0 and 91.4 with a plantation was 2.03 years (SD = 5.8, min. = 2 months, total average of 88.8 (see Table 2). Differences between max. = 20 years), and mean use of CareLink® Network was mean dimension scores were statistically significantly 2.03 years (SD = 1.7, min. = 2 months, max. = 5 years). (p < 0.05), except for Disease Information vs. Conveni- Most patients reported living a normal life for their ence (p = 0.97) and vs. Monitoring (p = 1.0). age, although they tried to avoid physical exertion. Some reported they had adopted healthier habits (no smoking Reliability or alcohol consumption) since diagnosis, although they The overall scale reliability was good (α = 0.893, ω = thought that other diseases (e.g., diabetes) need more 0.95) while dimension reliabilities were good for General stringent care. Opinions (α = 0.836, ω = 0.94) and acceptable for Con- PM patients showed some distrust of the transmission venience (α = 0.772, ω = 0.92), Monitoring (α = 0.614, process. They wished they could have made the first ω = 0.86), Information (α = 0.588, ω = 0.79), and Trans- transmission under supervision at the hospital. One pa- mission (α = 0.576, ω = .76) (see Table 2). Since the scale tient called in 20 min after each manual transmission, is thought to be multidimensional, α may be underesti- even if the modem beeped at the end. ICD patients felt mating the true reliability and ω values should be con- they had tight supervision by their nurse, even receiving sidered more informative. phone calls when transmission errors occurred or when fluid retention was observed during regular monitoring. Validity Intervals between face-to-face visits were perceived as Construct validity too long. One patient visited the medical team bringing Content validity results showed that experts were able to sweets to keep in touch. Self-support groups were not identify the different domains measured but with a rela- welcome, only the clinician was trusted for delivering in- tive overlap between medical care and overall benefits formation. Patients did not manifest any other needs or dimensions (See Additional file 3). concerns and the saturation for information was reached Correlations between item scores fit the proposed sec- in each focus group. ond order factor model with an acceptable goodness-of- fit: χ2/df = 1.63, CFI = 0.94, TLI = 0.94, RMSEA = 0.064. Validation phase All first-order loadings were significant (except for item Sample description 19) as were second order loadings (see Fig. 1). However, A sample of 186 patients was recruited between 2014 several modifications had to be incorporated into the and 2015; 53% at Hospital Universitario de Araba, 30% model. Item 8 loaded on the Convenience dimension, at Hospital Infanta Leonor, 15% at Hospital Sant Pau, item 14 showed cross-loading on the General Benefits and 3% at Hospital Universitario La Paz. On average, pa- dimension, item 19 showed cross-loading on the Trans- tients were 66.03 (SD = 13.94) years old, 77% were men, mission dimension, and items 9 and 10 exhibited a Ruiz Díaz et al. BMC Health Services Research (2020) 20:354 Page 5 of 9 Table 1 Sample description of social/demographic and clinical variables Employment Status (%) Marital Status (%) Working 13.6 Single 9.5 Homemaker 8.9 Married 75.0 Retired 59.2 Divorced 6.0s Unemployed 2.4 Widowed 8.3 Disabled 16.0 Other 1.2 Education Level (%) Clinical (%) None 8.1 ICD 47.8 Primary 45.3 PM 28.6 Secondary 12.2 CRT 23.6 Professional Degree 16.9 Higher 17.4 Risk Factors (%) Heart Disease Hypertension 59.5 Ischemic 53.4 Hypercholesterolemia 53.7 Dilated cardiomyopathy (non-ischemic) 33.6 History of AFib 38.8 Valvular 6.9 Smoking 37.2 Brugada syndrome 2.3 Diabetes 28.1 Other 6.1 History of renal insufficiency 17.4 None 3.8 History of sudden death 8.3 Stroke 8.3 CareLink use (%) Sex (%) Experienced 76.7 Women 23.0 First implantation: Naïve 21.0 Men 77.0 Replacement/upgrade: Naïve 2.3 Sample size (N) 186 ICD Implantable Cardioverter-Defibrillator, PM Pacemaker, CRT Cardiac Resynchronization Therapy device significant correlation between their error terms. After tended to correlate more highly with Convenience, accepting these additional specifications, the model pro- Transmission, and General Opinions, while information vided a substantially better fit: χ2/df = 1.44, CFI = 0.96, exhibited a similar correlation with all other VAS scales TLI = 0.95, RMSEA = 0.05. (differences in correlation below 0.110 were not statisti- cally significant). Convergent validity Correlations with divergent measures, i.e., the two All dimensions significantly correlated (p < 0.01) with MOS SF-36 HRQoL and with the HRQoL VAS measure, the corresponding VAS measure, although Monitoring were negligible (Tables 3 & 4), except for the correlation Table 2 Summary and reliability estimates of the VALIOSA questionnaire by dimension Dimension Initial Mean Standard Cronbach’s 95% ICC Omega Items number Deviation Alpha Lower Upper of items 1. Disease Information 3 91.4 13.6 .588 .473 .682 .79 1, 2, 3 2. Convenience 3 93.1 12.2 .772 .707 .824 .92 4, 5, 6 3. Transmission 6 82.0 16.1 .576 .467 .669 .76 7, 8, 9, 10, 11, 12 4. Monitoring 7 90.1 10.7 .614 .521 .695 .86 13, 14, 15, 16, 17, 18, 19 5. General Benefits 11 86.8 14.2 .836 .797 .870 .94 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30 Total Scale 30 88.8 9.9 .893 .866 .917 .96 1–30 ICC Intra-class correlation coefficient Ruiz Díaz et al. BMC Health Services Research (2020) 20:354 Page 6 of 9 Fig. 1 Confirmatory Factor Analysis standardized solution between the General Opinions dimension and the experienced users (M = 89.0) and replacement patients HRQoL VAS measure (r = 0.157, p < 0.05); while the naïve to CareLink® Network (M = 92.3). HRQoL VAS measure correlated more with the Physical Significant differences were found in mean dimension component (r = 0.550) than with Mental component (r = scores between patients from different hospitals (F3,171 = 0.272). 3.65, p = 0.014) with an interaction term close to signifi- cance (F12,684 = 1.68, p = 0.068), with average satisfaction Known groups validity at one of the treatment sites significantly worse than the VALIOSA dimensions were shown to be sensitive between others. After excluding that site from the analysis, more different types of patients. Significant differences were statistically significant differences were found (which found in the dimension mean scores between patients could be explained by additional information received depending on the type of implanted device (F2,179 = 14.0, from the sites). p < 0.001), with no interaction with the dimension score Although small, a significant correlation was found be- factor (F8,716 = 1.09, p = 0.368). Overall, patients with PM tween patient age and the Health Monitoring dimension were significantly more dissatisfied than those with ICD (r = 0.183; p = 0.013), but not for the remaining dimen- (d = − 6.8, p < 0.001) and CRT devices (d = − 9.3, p < sions nor for the overall score. 0.001), but the latter did not differ from one another (d = 2.4; p = 0.477). This pattern was repeated for all di- Discussion mensions (Fig. 2), except for Convenience, where ICD To our knowledge, this is the first validated question- and PM patients did not reach significance (d = − 4.7, naire developed to measure patient satisfaction with re- p = 0.078). mote monitoring of patients with CIED. This 30-item No significant differences were found in mean dimen- questionnaire showed good reliability and validity to as- sion scores between patients based on their experience sess satisfaction in patients implanted with all types of with CareLink® Network (F2,163 = 0.77, p = 0.462), al- CIED, suggesting that the questionnaire is valid across though initial implantation patients naïve to CareLink® different groups of patients. Patients who were naïve in Network (M = 87.2) were slightly less satisfied than the use of CareLink® Network along with those who had Table 3 Correlations among VALIOSA dimensions (rows) and visual analog scales (columns) Dimension VAS Measure Information Convenience Transmission Monitoring General Opinion Disease Information .333** .258** .376** .376** .337** Convenience .112 .478** .408* .306** .470** Transmission .104 .405** .406** .323** .351* Monitoring .206** .501** .504** .422** .534** General Benefits .259** .463** .404** .486** .488** HRQoL Health-related Quality of Life. Significance: * p < 0.05, ** p < 0.01 Ruiz Díaz et al. BMC Health Services Research (2020) 20:354 Page 7 of 9 Table 4 Correlation between VALIOSA dimensions and health- problem for me”), initially assigned to the Monitoring di- related quality of life measures mension, showed significant cross-loading with the Trans- Dimension SF-36 Component mission dimension. Item 14 (“Using the transmitter makes Physical Mental HRQoL me feel better cared for”), initially assigned to the Moni- Disease Info. .030 .263** .094 toring dimension, showed significant cross-loading with Convenience .110 .043 .088 the General Benefits dimension. Nevertheless, moving these items to the dimensions they cross-load with does Transmission −.006 −.036 .019 not improve dimension reliability, suggesting they should Monitoring .090 .043 .061 remain in their proposed dimensions pending on other General Benefits .106 .035 .157* additional evidence. While the overall scale exhibited good Physical 1.000 .012 .550** reliability, some individual subscales showed only accept- Mental .012 1.000 .272** able reliability when accounting for the multidimensional HRQoL .550** .273** 1.000 structure, suggesting that they should be used isolated with care. Mean 37.2 52.1 73.1 The second order factor structure did fit our data Standard Deviation 8.68 10.54 19.88 properly. Items (first order indicators) loaded with a sta- ** p < 0.01, * p < 0.05 tistically significant weight and most with a loading (in absolute value) above 0.6. The exception was the rela- more than 1 year of experience were included, and this tively low loading for items 9 and 10, which also showed classification factor was not shown to have an influence correlation between their error terms. This behavior on the level of satisfaction, suggesting that the system is could be associated with a response bias for PM users, well accepted from very early stages in its use. The level since they transmit only once a year, and most did not of satisfaction of the study population was found to be have the opportunity to experience any transmission very high, with scores of more than 80 in all dimen- problem. First order dimension loadings on the Overall sions. Initial results were reported at the 23rd Inter- Satisfaction second order dimension were all high and national Society for Quality Of Life Research Annual had the expected sign, corroborating the high correlation Conference [25]. among all dimensions. Correlations between dimension scores and conver- Study findings gent VAS measures were as expected, although the Three items proved somewhat problematic. Item 8 (“It is Health Monitoring dimension correlated more highly easy for me to make the transmission”), was initially pro- with the Convenience and Transmission VAS measures. posed as an indicator of the Transmission Process di- These results may reflect the fact that some sites per- mension, but it was ultimately better allocated to the form active patient monitoring, personally calling them Convenience dimension, and cross-loading in both di- when any health problem needs to be addressed after mensions was proposed. Item 19 (“The fact that the con- automatic transmissions (e.g. due to fluid retention). sultation is done at home instead of at the hospital is a This being the case, medical monitoring may affect the Fig. 2 VALIOSA mean dimension score by device type. Note: ICD=Implantable Cardioverter-Defibrillator, PM=Pacemaker, CRT=Cardiac Resynchronization Therapy device. Ruiz Díaz et al. BMC Health Services Research (2020) 20:354 Page 8 of 9 patient perception of the adequateness of the transmis- questionnaire in a telephone interview, since most of sion process and the convenience of the monitoring sys- them are written in first person. tem. Furthermore, VAS measures were found to be less sensitive to patient differences than dimension scores, Conclusions behaving as overly holistic measures. In the last years, remote monitoring of patients implanted Patient satisfaction, as measured by the VALIOSA ques- with CIED has grown as an alternative to in-office follow- tionnaire, was shown to have little relation to HRQoL, up. Rigorous studies have demonstrated its clinical and supporting proper construct validity. economic benefits to patients and healthcare institutions. As for known group validity, the different dimensions The VALIOSA questionnaire is the first validated ques- were shown to be able to discriminate between types of tionnaire developed to assess patient satisfaction with RM patients. As expected, PM patients were less satisfied with which has shown good reliability and validity. The ques- the monitoring system, and perhaps setting up the moni- tionnaire may be adapted to other cardiac RM systems by tor and making the transmission may even be a burden changing the name of the system, although maintenance for them, as we learned from the focus groups. This could of psychometric properties should be assessed. be explained because PM patients do not transmit regu- larly as ICD and CRT patients do because of their medical condition, and hence they might find transmission prob- Supplementary information Supplementary information accompanies this paper at https://doi.org/10. lems more often that decreases their level of satisfaction 1186/s12913-020-05216-3. with the follow up. The questionnaire dimensions were shown to be sensitive to differences in patient manage- Additional file 1. VALIOSA questionnaire, Spanish version. ment at different sites, with lower satisfaction for those Additional file 2. VALIOSA Questionnaire, English version. patients treated more impersonally. Additional file 3. Additional Materials: Further Psychometric Evidences. Benefits and applications of the VALIOSA questionnaire Abbreviations CFA: Confirmatory Factor Analysis; CFI: Comparative fit index; d: Difference; The present patient satisfaction questionnaire could be- df: Degrees of freedom; F: F statistic; ICC: Intra-class correlation coefficient; come a useful instrument in quality-of-care assessment, RMSEA: Root mean square error of approximation; SD: Standard Deviation; either being used by clinicians or hospital administra- TLI: Tucker-Lewis fit index tions. In this way, this tool can be used to evaluate pa- Acknowledgments tient’s impressions of RM at different times at hospitals VALIOSA group investigators: Socorro Sorbet, Hospital Universitario Araba, Julia and therefore benchmark between different healthcare Díaz, Hospital Universitario Araba, Celia Hijosa, Hospital Universitario Infanta institutions. The success of telemedicine programs must Leonor, Carmen Varela, Hospital Santa Creu i Sant Pau, Dr. Francisco Méndez, Hospital Santa Creu i Sant Pau, Olga Calvo, Hospital Universitario La Paz, Dr. include patient’s perspectives to understand how well Omar Al-Rizzo, Hospital Universitario La Paz. they serve the needs and address the concerns of the pa- The authors would like to thank B. Martí, employed by Medtronic Ibérica, tients. Therefore, investigating patients’ impressions with S.A., at the time the work for this manuscript was planned. Initial results of this study were reported at the poster presentation session accuracy and precision could help in the design and im- of the 23rd International Society for Quality Of Life Research Annual plementation of telemedicine systems that can improve Conference. the quality of healthcare delivery. Authors’ contributions All authors have contributed substantially in the manuscript preparation, Limitations interpretation of results or study design and management. MAR, ME and MA researched literature and conceived the study. RM, XV, JS and JM-F were Present results may not represent patients using other responsible of the data acquisition, validated the design and results of the RM systems without further study. The brand name of model. MAR performed the analysis and wrote the first draft of the the RM system was included in the questionnaire since manuscript. All authors reviewed and edited the manuscript, and approved the final version of the manuscript. Authors take full responsibility for the elder patients showed problems to differentiate between data presented in this study, analysis of the data, conclusions, and conduct the RM system and the CIED, but replacing the name of of the research, and had full access to those data and has maintained the the system used in the study with other names should right to publish any and all data independent of any third party. not be a problem. The questionnaire may be adapted to Funding other e-health monitoring systems but important par- This research has been conducted with an unrestricted grant from ticularities, such as web usability or time to response Medtronic Ibérica, S.A. to the Universidad Autónoma de Madrid. could be missing. Adaptation to other RM systems may need the inclusion of additional items when special Availability of data and materials prophylactic measures are customary, such as continu- The datasets generated and analyzed during the current study are not publicly available due to the fact that Ethics Committee approvals were not ous glucose monitoring systems in insulin pumps for obtained for sharing of datasets outside of the research team, but are diabetics. Items should be adapted for use of the available from the corresponding author on reasonable request. Ruiz Díaz et al. BMC Health Services Research (2020) 20:354 Page 9 of 9 Ethics approval and consent to participate 14. Medtronic Inc. CareLink network statistics, 2016. Data on file. The study protocol was approved (CEI 48–925) by the Clinical Research 15. Rovinelli RJ, Hambleton RK. On the use of content specialists in the Commitsstees of the Universidad Autónoma de Madrid and the participating assessment of criterionreferenced test item validity. Dutch J Educ Res. 1977; hospitals. This study was registered at ClinicalTrials.gov with the Identifier 2:49–60. NCT02401659 on March 17, 1015. 16. Turner RC, Carlson L. Indexes of Item-Objective Congruence for Signed informed consent and permission to use personal health information Multidimensional Items. Int J Testing. 2003;3(2):163–71. were obtained from all participating patients. 17. Wiley DE. The identification problem for structural equation models with unmeasured variables. In: Goldberger AS, Duncan OD, editors. Structural Consent for publication equation models in the social sciences. New York: Seminar; 1973. p. 69–83. Not applicable. All results are reported as aggregated data. 18. Anderson TW, Rubin H. Statistical inference in factor analysis. In: Neyman J, editor. Proceedings of the third Berkeley symposium on mathematical statistics and probability. Berkeley: University of California Press; 1956. p. Competing interests 111–50. M. Ruiz, R. Muñoz, X. Viñolas, J. Silvestre, J. Martínez acknowledge research 19. Cattel RB. Factor analysis. An introduction and manual for the Psychologist grants from Medtronic Ibérica, S.A. M. Egea and M. Álvarez are employees of and Social Scientist. New York: Harper and Row; 1952. Medtronic Ibérica, S.A. 20. Everitt BS. Multivariate analysis: the need for data, and other problems. Br J Psychiatry. 1975;126:237–40. Author details 1 21. MacCallum RC, Widaman KF, Zhang S, Hong S. Sample size in factor Department of Methodology, School of Psychology, Universidad Autónoma analysis. Psychol Methods. 1999;4:84–99. de Madrid, Madrid, Spain. 2Health Economics & Outcomes Research 22. McDonald, Test theory. A unified treatment. Hillsdale: Erlbaum; 1999. Department, Medtronic Ibérica, S.A, Madrid, Spain. 3Department of 23. Badía X, Roset M, Montserrat S, Herdman M, Segura A. La versión española Cardiology, Hospital Universitario Infanta Leonor, Madrid, Spain. 4Department del EuroQol: descripción y aplicaciones. Medicina Clínica. 1999;112(Supl. 1): of Cardiology, Hospital Santa Creu i Sant Pau, Barcelona, Spain. 5Department 79–86. of Cardiology, Hospital Universitario La Paz, Madrid, Spain. 6Department of 24. Ware JE, Snow KK, Kosinski M, Grandek B. SF-36 health survey: manual and Cardiology, Hospital Universitario de Araba, Vitoria, Spain. interpretation guide. Boston: The Health Institute, New England Medical Center; 1993. Received: 8 October 2018 Accepted: 13 April 2020 25. Ruiz MA, Egea M, Muñoz R, et al. Behavior of VAS measures of patient satisfaction with remote monitoring of cardiac implantable electronic devices. the valiosa questionnaire. 23rd Annual Conference of the References International Society for Quality of Life Research. Qual Life Res. 2016;19(7): 1. Burri H, Senouf D. Remote monitoring and follow-up of pacemakers and A700-A701. https://doi.org/10.1016/j.jval.2016.09.2031. implantable cardioverter defibrillators. Europace. 2009 Jun;11(6):701–9. 2. Brugada P. What evidence do we have to replace in-hospital implantable cardioverter defibrillator follow-up? Clin Res Cardiol. 2006;95(Suppl 3):III–9. Publisher’s Note 3. Theuns DA, Res JC, Jordaens LJ. Home monitoring in ICD therapy: future Springer Nature remains neutral with regard to jurisdictional claims in perspectives. Europace. 2003;5(2):139–42. published maps and institutional affiliations. 4. Slotwiner D, Varma N, Akar JG, Annas G, Beardsall M, Fogel RI, et al. HRS expert consensus statement on remote interrogation and monitoring for cardiovascular implantable electronic devices. Heart Rhythm. 2015;12:e69–100. 5. Atkinson MJ, Sinha A, Hass SL, Colman SS, Kumar RN, Brod M, et al. Validation of a general measure of treatment satisfaction, the Treatment Satisfaction Questionnaire for Medication (TSQM), using a national panel study of chronic disease. Health Qual Life Outcomes. 2004;2:12. 6. Akar JG, Bao H, Jones P, Wang Y, Chaudhry SI, Varosy P, et al. Use of remote monitoring of newly implanted cardioverter-defibrillators: insights from the patient related determinants of ICD remote monitoring (PREDICT RM) study. Circulation. 2013;26(128):2372–83. 7. Saxon LA, Hayes DL, Gilliam FR, Heidenreich PA, Day J, Seth M, et al. Long-term outcome after ICD and CRT implantation and influence of remote device follow-up: the ALTITUDE survival study. Circulation. 2010; 122:2359–67. 8. Joseph GK, Wilkoff BL, Dresing T, Burkhardt J, Khaykin Y. Remote interrogation and monitoring of implantable cardioverter defibrillators. J Interv Card Electrophysiol. 2004;11:161–6. 9. Laurent G, Amara W, Mansourati J, Bizeau O, Couderc P, Delarche N, et al. Role of patient education in the perception and acceptance of home monitoring after recent implantation of cardioverter defibrillators: the EDUCAT study. Arch Cardiovasc Dis. 2014;107:508–18. 10. Morichelli L, Porfili A, Quarta L, Sassi A, Ricci RP. Implantable cardioverter defibrillator remote monitoring is well accepted and easy to use during long-term follow-up. J Interv Card Electrophysiol. 2014;41:203–9. 11. Ricci RP, Morichelli L, Quarta L, Sassi A, Porfili A, Laudadio MT, et al. Long- term patient acceptance of and satisfaction with implanted device remote monitoring. Europace. 2010;12:674–9. 12. Versteeg H, Pedersen SS, Mastenbroek MH, Redekop WK, Schwab JO, Mabo P, et al. Patient perspective on remote monitoring of cardiovascular implantable electronic devices: rationale and design of the REMOTE-CIED study. Neth Heart J. 2014;22:423–8. 13. Quintana JM, González N, Bilbao A, Aizpuru F, Escobar A, Esteban C, et al. Predictors of patient satisfaction with hospital health care. BMC Health Serv Res. 2006;6(1):102.