International Journal of Environmental Research and Public Health Article The WOMEN-UP Solution, a Patient-Centered Innovative e-Health Tool for Pelvic Floor Muscle Training: Qualitative and Usability Study during Early-Stage Development Sònia Anglès-Acedo 1, * , Lorena López-Frías 1 , Vicenç Soler 2 , Joan Francesc Alonso 3 , Arnoud W. Kastelein 4 , Boris C. de Graaf 4 , Eva. V. Vodegel 4 , Jaana Tervo 5 , Adriana Baban 6 , Montserrat Espuña-Pons 1 and on behalf of the WOMEN-UP Consortium † 1 Department of Obstetrics and Gynecology, Hospital Clinic de Barcelona, Universitat de Barcelona, 08036 Barcelona, Spain; llopez3@clinic.cat (L.L.-F.); mespuna@clinic.cat (M.E.-P.) 2 Department of Microelectronics and Electronic Systems, Universitat Autònoma de Barcelona, 08193 Bellaterra, Spain; vicenc.soler@uab.cat 3 Department of Automatic Control, Biomedical Engineering Research Centre, Universitat Politècnica de Catalunya, 08028 Barcelona, Spain; joan.francesc.alonso@upc.edu  4 Department of Obstetrics and Gynecology, Amsterdam UMC Location Academic Medical Center,  1105 AZ Amsterdam, The Netherlands; a.w.kastelein@amsterdamumc.nl (A.W.K.); Citation: Anglès-Acedo, S.; borisc.degraaf@gmail.com (B.C.d.G.); e.v.vodegel@amsterdamumc.nl (E.V.V.) 5 Department of Physical and Rehabilitation Medicine, Kuopio University Hospital, 70210 Kuopio, Finland; López-Frías, L.; Soler, V.; Alonso, J.F.; Kastelein, A.W.; Graaf, B.C.d.; jaana.tervo@kuh.fi 6 Department of Psychology, Babes-Bolyai University of Cluj-Napoca, 400015 Cluj-Napoca, Romania; Vodegel, E..V.; Tervo, J.; Baban, A.; adrianababan@psychology.ro Espuña-Pons, M.; et al. The * Correspondence: sangles@clinic.cat; Tel.: +34-932775436 WOMEN-UP Solution, a † WOMEN-UP Consortium are listed in acknowledgments. Patient-Centered Innovative e-Health Tool for Pelvic Floor Muscle Training: Abstract: e-Health may enhance self-management of pelvic floor muscle training (PFMT) to treat Qualitative and Usability Study stress urinary incontinence (SUI). It is crucial to involve patients in planning, developing and during Early-Stage Development. Int. monitoring the optimal e-Health solution. This research aims to describe patient-centered innovation J. Environ. Res. Public Health 2021, 18, in an early developmental stage of the WOMEN-UP solution. We conducted a qualitative study 7800. https://doi.org/10.3390/ ijerph18157800 through a self-developed questionnaire in 22 women with SUI, to define system requirements from a patient’s perspective. The first prototype of the WOMEN-UP solution was developed. It was tested by Academic Editors: Maria 9 patients in a usability study (think-aloud protocol and retrospective interviews). Patient preferences Luisa Sanchez Ferrer, Ines Moran regarding the possible use of an e-Health solution with serious games for PFMT were: (1) to receive Sanchez and Julián J. Arense Gonzalo feedback about PFMT; (2) convenient home-use; (3) increasing motivation; (4) available in medical centers. Identified usability aids (31) reassured our design-development plan, which considered the Received: 25 June 2021 biofeedback and serious games as key factors. Patient’s perspective detected some unexpected issues Accepted: 17 July 2021 related to the calibration and serious games, involving a change in the ongoing development to get an Published: 23 July 2021 improved WOMEN-UP solution; the value of patient-centered innovation during the development of an e-Health solution for PFMT (WOMEN-UP solution). To identify patients’ unmet needs, we Publisher’s Note: MDPI stays neutral proposed a longitudinal approach for the future eHealth-related patient-centered innovations. with regard to jurisdictional claims in published maps and institutional affil- Keywords: patient-centered innovation; qualitative study; usability study; pelvic floor muscle iations. training; urinary incontinence; serious games; biofeedback; eHealth Copyright: © 2021 by the authors. 1. Introduction Licensee MDPI, Basel, Switzerland. This article is an open access article It is estimated that one out of three women may be affected by pelvic floor dysfunction: distributed under the terms and urinary incontinence (UI), pelvic organ prolapse and anal incontinence [1]. Urinary inconti- conditions of the Creative Commons nence can be experienced as a major burden on quality of life. Stress urinary incontinence Attribution (CC BY) license (https:// (SUI), characterized by involuntary loss of urine during physical activity, coughing and creativecommons.org/licenses/by/ sneezing, is the most common type (10–39%) [1]. Physical, sexual, social and emotional 4.0/). well-being can be disturbed in women with SUI. Int. J. Environ. Res. Public Health 2021, 18, 7800. https://doi.org/10.3390/ijerph18157800 https://www.mdpi.com/journal/ijerph Int. J. Environ. Res. Public Health 2021, 18, 7800 2 of 15 Conservative management with lifestyle modifications and supervised pelvic floor muscle training (PFMT) is the recommended first-line treatment [2], especially in mild or moderate SUI. In standard care, this treatment usually involves regular visits to a clinic or hospital (e.g., 1/week) for a multidisciplinary approach provided by different clinicians (physiotherapists/physical therapists, nurses, midwifes or medical doctors). e-Health has gained exponential popularity during recent years in urogynecology, specifically regarding pelvic floor muscle training (PFMT) for urinary incontinence (UI). Mobile applications, vaginal devices and telemedicine (remote monitoring, videoconferenc- ing, phone call, asynchronous store and forward systems record) have been demonstrated as potential alternatives to usual care, with growing interest and availability in high-income countries [3]. During the COVID-19 pandemic, as patients were not allowed to attend to face-to-face interventions, such innovative ways to deliver healthcare have become the key for UI conservative management, to guarantee the first-line treatment according to guideline recommendations [3,4]. Innovation in medicine aims to find new alternatives to improve the health of patients. Nevertheless, new does not always mean improved. Some new practices are taken up and diffused throughout health systems [5], even when they are of limited benefit or unproven efficacy, or represent risks to patients, as many women’s experiences with transvaginal mesh for prolapse. Consequently, we should be cautious with innovation implementation, considering the possibility of both positive and negative impacts. Focusing on conservative management for UI, e-Health may improve access to care. Often, women encounter difficulties to perform PFMT because of waiting lists, costs or limited access due to time constraints. Moreover, other frequently occurring barriers such as trouble remembering to exercise, doubts regarding PFMT performance and difficulty finding time [6] may be overcome by empowering self-management and improving patient adherence through an e-Health solution. However, it is unclear which patients are willing and able to accept e-Health. Some factors have been proposed as predictors of patient acceptance or resistance to consumer e-health services, including sociodemographic vari- ables, device usability, awareness of the e-health innovations, and the user’s computer skills [7]. Implementation of eHealth interventions often fails, due to poor or lacking usability evaluation, which can lead to misuse of the intervention [8,9]. Therefore, new interventions must undergo a rigorous, suitable and user-based usability evaluation first [10–12]. Reliable and widely accepted usability testing methods, such as the think-aloud protocol and retro- spective interviews, facilitate insight into usability issues (barriers) through observations of interface interaction [13]. Specifically, user-centered testing has been used for gamified health interventions [14] and biofeedback-assisted systems [15]. Therefore, to ensure if an e-health solution meets the needs of patients, it is crucial to involve patients in the development of the e-Health solution: patient-centered innova- tion. [16]. The WOMEN-UP solution is a patient-centered innovation developed from 2015 to 2019 in a project funded by the European Union’s Horizon 2020 research and innovation programme. The WOMEN-UP project aims to improve quality of life in women with SUI through an e-Health self-management solution for PFMT and lifestyle modification. In short, the WOMEN-UP solution consists of biofeedback devices (vaginal and abdominal probes) to record EMG data, a mobile application to make the user perform exercises using serious games (gamification) and a web platform that collects all data and enables personalized supervision and an engaging home-training environment. Biofeedback in PFMT, a direct audiovisual feedback system on pelvic floor muscle activity, can improve clinical outcomes [17]. The gamification of motor function rehabilitation may contribute to this physiotherapeutic intervention by improving user engagement and learning experi- ence [18]. The WOMEN-UP consortium, composed of hardware and software engineers, psy- chologists and healthcare professionals, was guided by patient perspectives, in early- stage Int. J. Environ. Res. Public Health 2021, 18, 7800 3 of 15 design as well as development of the WOMEN-UP solution. Accordingly, patients were involved in one of the 8 work packages to ensure patient-centered innovation. The aim of this study was to describe patient-centered innovation in an early develop- mental stage of the WOMEN-UP solution, prior to future evaluation of functionality in a pilot study and future evaluation of efficacy in a randomized controlled trial. 2. Materials and Methods The WOMEN-UP project was approved by ethical committees in all participating medical centers (HCB/2016/0480, NL56562.018.16, 324/13.02.00/2016). Informed consent was obtained from all subjects involved in the study. In the present study, we focused on the first three stages of the project: qualitative assessment (stage 1), development of the first prototype (stage 2) and usability testing (stage 3). 2.1. Stage 1. Qualitative Assessment: Identify Key Stakeholders’ Needs This task included identifying patients’ preferences for the conservative treatment of UI, aiming to define system requirements from a patient’s perspective. The WOMEN- UP consortium developed a set of 29 standard questions (Supplementary Material S1), open-ended and with answers on Likert scales. The questions covered issues like: global impressions about device functionality; physical characteristics of the device including materials, interface, batteries; instructions; serious games; feedback received (current and desired feedback, areas of improvement). Participants answered the questionnare after a self-experience with available commercial devices for PFMT: Neurotrack® Myotrac Infinity Pro (2 Channels) (Thought Technology Ltd., Montreal, Quebec, Canada) and Femiscan® (Mega Electronics, Kuopio, Finland). This information provided valuable input for the development of a new device. 2.1.1. Participants Women with mild or moderate SUI able to perform PFMT, who attended a urogynae- cological department, were recruited in university hospitals in The Netherlands, Spain and Finland. Participants were invited for an intake visit in an outpatient clinic and were instructed to use the available commercial devices: Neurotrack® , Femiscan® or both. 2.1.2. Data Analysis Quantitative data from questionnaires (numerical data, Likert-scale answers) were entered separately and analyzed with a quantitative analytic tool (SPSS 20, IBM, Armonk, NY, USA). Because of the low number of participants in each group, only basic descriptive analyses were conducted and reported (frequency of a particular answer and means together with SD). As for qualitative data (answers in open-ended format), we analyzed and manually coded every interview form, according to the general themes covered by the questionnaire. Frequency analyses of answer categories (codes) were conducted. 2.2. Stage 2. WOMEN-UP Solution: First Prototype The information provided in stage 1 was used to developed an early, operational prototype of the intervention. This system was comprised of a vaginal and abdominal biofeedback device, a serious gaming application for a smartphone, a therapist–patient web-based communication platform and a personalized training schedule. Figure 1 shows the components of the intervention and their integration. ject, this first prototype used a commercial vaginal probe, which was connected by wires to the abdominal device. The devices measured electromyographic activity of the pelvic floor and the abdominal muscles. Muscular activity recorded from both muscles will be compared to the exercise programmed by the therapist (in terms of achievement of the Int. J. Environ. Res. Public Health 2021, 18, number of 7800 contractions, their duration and level, and the relaxation periods). The connec‐ 4 of 15 tion with the smartphone enabled the games to provide real‐time biofeedback and de‐ tailed insight into the progress and results of the training. Int. J. Environ. Res. Public Health 2021, 18, x FOR PEER REVIEW 5 of 15 (a) (b) (c) Figure 1. First Figure prototype 1. First of the prototype WOMEN‐UP of the solution. WOMEN-UP (a) (a) solution. Elements of the Elements WOMEN‐UP of the intervention, WOMEN-UP intervention, (b) Vaginal biofeedback device (pelvic floor muscles), (c) Abdominal biofeedback device (rectus ab‐ (b) Vaginal biofeedback device (pelvic floor muscles), (c) Abdominal biofeedback device (rectus dominis muscle). abdominis muscle). TheThesecure online platform, smartphone also underbiofeedback and the abdominal development during device this study, connect facilitated wirelessly via Blue- written tooth. Pending the development of a wireless vaginal device in a future stage oftothe communication with the therapist. Both user and therapist had access theproject, de‐ tails of the completed and planned exercises. Data protection regulations for security this first prototype used a commercial vaginal probe, which was connected by wires to theand privacy were considered carefully. All patient data were encrypted and stored in a data‐ base in the web platform server independently of personal data. Patient data recorded by the personal devices were encrypted with the SSL certificate (SHA‐256) and stored se‐ curely hashed with an encryption algorithm. Communication between the smartphone Int. J. Environ. Res. Public Health 2021, 18, 7800 5 of 15 abdominal device. The devices measured electromyographic activity of the pelvic floor and the abdominal muscles. Muscular activity recorded from both muscles will be compared to the exercise programmed by the therapist (in terms of achievement of the number of contractions, their duration and level, and the relaxation periods). The connection with the smartphone enabled the games to provide real-time biofeedback and detailed insight into the progress and results of the training. The secure online platform, also under development during this study, facilitated written communication with the therapist. Both user and therapist had access to the details of the completed and planned exercises. Data protection regulations for security and privacy were considered carefully. All patient data were encrypted and stored in a database in the web platform server independently of personal data. Patient data recorded by the personal devices were encrypted with the SSL certificate (SHA-256) and stored securely hashed with an encryption algorithm. Communication between the smartphone and the web platform was secured with encrypted transmission via the HTTPS protocol. 2.3. Stage 3. Usability Testing The first prototype developed in stage 2 was used for user-centered usability eval- uation through a think-aloud method. It is vital that participants feel confident and comfortable with the think-aloud session and the investigators [19,20], so we decided that the investigators should be usability-trained healthcare professionals with expertise on PFMT. Additionally, the healthcare professional acted as spokesperson for usability issues and performed both think-aloud and retrospective interviews. In phase one of the usability testing, a therapy simulation with elements of standard PFMT was performed in which participants tested the usability of the WOMEN-UP solution while thinking aloud with a healthcare provider. In phase two of the usability testing, participants gained instructions to use the first prototype of the WOMEN-UP solution in a home setting for two weeks. After two weeks’ use, a second think-aloud testing and a retrospective semi-structured interview were performed. One usability-trained healthcare professional conducted the interviews with all par- ticipants, guided by a retrospective questionnaire on usability, functionality, satisfaction and preference. The interviews and think-aloud sessions were audio-recorded and obser- vational notes were taken. All audio-recorded data and written notes were transcribed verbatim and imported into MAXQDA software (VERBI Software, Berlin, Germany) (ver- sion 12.2.1). 2.3.1. Participants Volunteers were recruited in university hospitals in The Netherlands, Spain and Finland. Participants fluent in English were invited for an intake visit in an outpatient clinic and received instructions on PFMT and a brief introduction to the eHealth solution by a healthcare professional. 2.3.2. Qualitative Analysis Computer-assisted qualitative analysis was performed using a thematic label frame- work based on the heuristics for medical device evaluation [21]. After data familiarization and iterative coding in data of two participants, the framework was complemented with additional relevant labels of the relevant themes from the model for user acceptance of technology [22] and serious gaming components [14,18]. The data were consolidated through the deletion of duplicate coded text segments. The unique text segments were manually summarized into written positive usability comments relating to user experience and usability issues that violated prior determined heuristic themes. Text segments with positive usability comments were manually separated from the issues, summarized and categorized. Usability issues were sorted according to the place of occurrence in the WOMEN-UP solution and observation frequency among participants. Int. J. Environ. Res. Public Health 2021, 18, 7800 6 of 15 Subsequently, all issues were scored according to the Nielsen severity scale. The scale ranges from 0 to 4, representing: nihil (0), trivial (1), minor (2), major (3) and severe (4) problems [23]. 2.3.3. Multidisciplinary Solution Optimization Hardware and software engineers, psychologists and healthcare professionals com- bined their expertise in a multidisciplinary panel discussion. The usability aids and issues were discussed, and the consortium partners devised possible system improvements, in addition to already planned features and functionalities. The possible improvements were discussed after informal assessment of the state of development of system elements, the available resources, and the feasibility of the proposed improvements. 3. Results 3.1. Stage 1. Qualitative Assessment: Identify Key Stakeholders’ Needs Overall, 22 patients—10 in the Netherlands (45.4%), 6 in Spain (27.3%), 6 in Finland (27.3%)—took part at this survey. Fourteen of these patients used both devices, whereas five used only Neurotrack® and three only used Femiscan® . Due to the small sample sizes in different countries, we present the overall findings and will not make inter-country comparisons. The mean age of participants was 47 years old (SD: 12 years, range 29–71 years). The mean number of vaginal deliveries was 1.85 (SD 1.22, range 0–5). The mean duration of UI was 59.23 months (SD: 151.69 months, range: 2–444 months). The medical supervision of the patients varied from 0.5 to 360 months (mean = 12.83, SD = 277.06). The majority of patients (59.1%) had not been treated before for UI. Those treated previously with PFMT reported being moderately satisfied with a mean satisfaction of 3.68 (SD = 1.11), on a scale varying from 1 to 5. Nearly three out of four patients indicated that the e-Health solution should be available in medical centers. Table 1 shows the quantitative indicators for the pelvic floor muscle device features and integrated system. Overall, the device should be reliable and easy to use, and when it is used it should provide clear feedback on how the exercises are performed. Table 1. Summary of quantitative indicators for the pelvic floor muscle device and the integrated system. Feature Option Responses % Participants • oval and conical with stop 11 50.0% • cylindrical with stop 5 22.7% Vaginal probe shape • cylindrical without a stop 4 18.2% • other shape with stop 2 9.1% Valid responses 22 100% • smooth surface 19 86.4% • neutral 18 81.8% • soft 12 54.5% • flexible 10 45.5% Vaginal probe • hard 9 40.9% material • rigid 9 40.9% • warm 7 31.8% • rugged 1 4.5% • cold 1 4.5% Int. J. Environ. Res. Public Health 2021, 18, 7800 7 of 15 Table 1. Cont. Feature Option Responses % Participants Total responses 87 • comfortable 18 81.8% • easy to use 17 77.3% • inexpensive 17 77.3% Device perception • motivating 16 72.7% • reliable 15 68.2% • versatility 14 63.6% • good design 12 54.5% Total responses 110 • they help to increasemy 18 81.8% Motivation to exercise motivation • they do not modifymy 2 9.1% motivation Valid responses 20 90.9% Empty responses 2 9.1% • any time needed 6 27.3% • monthly 5 22.7% • at the end of thetraining period 2 9.1% Feedback preferences • every 15 days 2 9.1% Valid responses 15 68.2% Empty responses 7 31.8% Note that the participants were allowed to select more than one option regarding some of the features; in these cases, the percentages corresponding to valid and empty responses are not shown. The qualitative indicators for pelvic floor muscle device features and integrated system are summarize below (Table 2). Table 2. Summary of qualitative indicators for the pelvic floor muscle device and the integrated system. Topic Recommendations Risks • finding the right • convenient use at home time/place(tiredness, busy global functionality • valuable feedback schedule) • enhanced motivation • use difficulties • too simple/not stimulating • oval/conical with stop • several sizes • wrong size mechanical design • light and smooth • hard to reach buttons • wireless • hard/cold materials • simple interface (few • rough materials buttons) • reliability system features(in order • ease of use • too simple/not stimulating ofimportance) • motivating/entertaining • too difficult to use or follow • clear feedback/guidance • clear • with pictures/diagrams • too simple instructions device instructions • with safety/hygiene • too complex instructions information • too many program • describing the training alternatives program • exercises correctly performed • ambiguous feedback feedback • information on strength • monotonous feedback • summary after exercising • too complicated feedback • improvement over time Int. J. Environ. Res. Public Health 2021, 18, 7800 8 of 15 Table 2. Cont. Topic Recommendations Risks • necessary, even if minimal • incorrect use if there is patient– (messages after first visit) nointeraction professionalinteractionpreferences • provide feedback • poor adherence because of onperformance lack of interaction • adjust treatment • smartphone or tablet-based system interface • providing feedback during • not familiar withtechnology andafter training • data safety • clear instructions/guidance serious games • should enhance motivation • overtraining • training for 10 to 20 min • incorrect training The most common comments from patients’ perspectives were related to: • Adherence improvement: “Having the home device allows greater availability and gives them autonomy, which facilitates the adherence to the PFMT.” “The verbal and visual instructions facilitate the successful accomplishment of the PFMT, and consequently, improve concentration and adherence to treatment.” “Having a biofeedback device at home helps them to motivate themselves to continue doing the exercises, thus creating more adherence to PFMT.” “The possibility to train home increased the amount of exercises done during the study period.” • Performance improvement: “A key issue was also the easy-to-use technique with easily found start-buttons and no wires and very clear instructions.” “Devices help to improve the self-awareness of pelvic floor muscles by vaginal probe and by control over the use of the abdominal muscles, and allow better integration about the instructions given by the therapist.” “Self-assessment in each session may increase the motivation to overcome the results of the training itself.” 3.2. Stage 3. Usability Testing Nine women, three in each center, volunteered to participate in the usability testing. Seven out of nine were healthy volunteers, two volunteers had complaints of SUI. The mean age was 41 years old (range: 25–61 years). Seven women were parous. Fifty-six percent were familiar with pelvic floor exercises. All participants completed the two weeks of training. Two out of nine women reported having limited experience with smartphone applications. The think-aloud interviews of the first phase failed in three participants due to the absence of the interviewer at the start of the study. 3.2.1. Usability Aids Analysis of labeled text segments showed 31 usability aids that appeared to positively affect user experience (Table 3). These predominantly concerned remarks on the games and direct visual feedback of muscular contractions. Participants expressed that those exercises were more fun than dry exercises, or even that exercises would not have been done at all without the WOMEN-UP solution. The interactive design and the challenge of games were appreciated by multiple participants. The direct visual feedback without time lag provided the participants with valuable information, because of the difficulty of dry exercises. Also, it was reported by users that these features contribute positively to performing PFMT. Int. J. Environ. Res. Public Health 2021, 18, 7800 9 of 15 Table 3. Summary of usability aids. Category Summarized Facilitator Description Number of Facilitators Positive attitude towards the Mobile Positive attitude toward the system, application and the 7 Application general system flexibility provided by the solution The calibration of the devices Calibration Easy-to-learn calibration 1 has a fast learning curve Users appreciate the usefulness of direct visual Good biofeedback feedback on the muscular 7 contractions and the final scores provided Additional feedback given to the user (next actions to be Good control during games performed, remaining time) 3 helps the feeling of being in control of exercising Serious games Games are well designed for Good game design these kind of rehabilitation 1 exercises Exercises could not have been Games facilitate exercising 4 done without the games Users can choose from different games, based on Game choice 2 their preferences, to perform the same exercise Games make exercising fun Enjoyable gaming 4 and enjoyable Users feel comfortable with Devices Functioning and comfort 2 the use of the wearable device 3.2.2. Usability Issues Analysis of labeled text segments showed 80 usability issues. Figure 2 shows the role of the first and second phase of the usability study. The non-persisting issues (12 issues only detected in phase 1) concerned 10 trivial issues and 2 minor issues (15% of all detected issues). These issues were most often related to either the calibration of the devices or the navigation of the mobile application such as simple navigation buttons. Among all identified issues in phase 1, 64% (21/33) were also identified in phase 2, meaning that most of the issues were persistent. On the other hand, newly identified issues in the second phase accounted for 59% (47/80) of all issues (Figure 2. Among them, the majority were nihil (17%), trivial (42%) or minor (28%) issues, although new major and severe issues were also identified (13%). These severe and major issues were related to communication, calibration and serious games. These usability issues could predominantly be classified into 11 categories: mobile application general, mobile application navigation, calibration, serious games, gaming biofeedback, training evaluation, web platform, devices, device connectivity, training schedule, communications with the user. Among all identified issues, two-thirds are expected barriers due to the early-stage of development of the first prototype and all of them are already considered in the ongoing design and development of the WOMEN-UP prototype (e.g., device design and connectivity, communication with the user, web-platform and navigation app functionalities . . . ). However, one-third of the detected issues were unexpected, mainly related to the calibration which required a new adjustment of the Int. J. Environ. Res. Public Health 2021, 18, 7800 10 of 15 software or related to the serious games which required design of new type of games, as Int. J. Environ. Res. Public Health 2021, 18, wellx FOR PEER REVIEW of the patient’s manual or improvement of the feedback at the10end as improvement of 15 of the performance (e.g., stars). Table 4 shows a summary of the usability issues. Figure 2. Colored blocks represent the usability issues with corresponding severity score. The order Figure 2. Colored blocks represent the usability issues with corresponding severity score. The order shows the value of each study phase. shows the value of each study phase. 4. Summary of the usability issues (barriers). TheTable non‐persisting issues (12 issues only detected in phase 1) concerned 10 trivial Category issues Issues and 2 minor issues Type (15% of all detected issues). These issues wereComments Description most often related to either the calibration of the devices or the navigation of the mobile application such as As this is a pre-product test simple navigation buttons. Among all identified issues in phase 1, 64% (21/33) were also step, the online platform was identified in phase 2, meaning that most of the issues were persistent. chosen On the other hand, to summarize all newly identified issues in the second Notphase all the accounted functionalities for 59% (47/80) of all issues functionalities since it(Figure was Mobile were in the App, outdated Application 2. 9 Among them, 3 major,the majority 2 minor 1 were nihil (17%), trivial (42%)easier design and sometimes used or minor (28%)the to increase issues, software alt‐ general hough newtrivial majorand and3 nihil severe issues were also identified medical jargon abbreviations functions (13%). These without severe needing and to major install new App versions issues were related to communication, calibration (PFMT) and serious games. constantly. In addition, design These usability issues could predominantly be classifiedwas into 11 categories: mobile applied for functionality application general, mobile application navigation, calibration, serious abovegames, gaming bi‐ cosmetics. ofeedback, training evaluation, web platform, devices, device connectivity, training User could not use back Mobile schedule, communications with thebutton user. and Among all identifiedSome the design issues issues, were due are two‐thirds to anex‐ 7 6 trivial and 1 minor pected barriers due to the early‐stage of development of the first prototype and all ofat incomplete development the them Application navigation (visibility) of some buttons time of testing. are already considered in the ongoing design could and development of the WOMEN‐UP pro‐ be improved. totype (e.g., device design and connectivity, communication The major issue was due to with the user, web‐platform and navigation app functionalities…). However, an absence ofone‐third of the detected issues were un‐ expected, mainly related to the calibration which required assistance/messages in case a new adjustment of the soft‐ ware or related to the serious games of error which in required the process. In design of new type of games, as well as 4 trivial, 4 minor and 1 addition, the display of the Improvement implemented by Calibration 9 improvement of the patient’s manual or improvement of the feedback at the end of the major. calibration process can be a simpler calibration method. performance (e.g., stars). Table 4 shows a summary of the usability issues. improved with more information and clear Table 4. Summary of the usability issues visibility (barriers). along the whole process to guide the user. Category Issues Type Description Comments As this is a pre‐product test step, the online platform was chosen to sum‐ Not all the functionalities were in the marize all functionalities since it was Mobile 3 major, 2 minor 1 trivial App, outdated design and sometimes easier to increase the software func‐ Application 9 and 3 nihil used medical jargon abbreviations tions without needing to install new general (PFMT) App versions constantly. In addition, design was applied for functionality above cosmetics. Int. J. Environ. Res. Public Health 2021, 18, 7800 11 of 15 Table 4. Cont. Category Issues Type Description Comments Major and minor issues Some games and menus information was used to help were unclear and with lags. improving the application for 5 major, 3 minor and 5 Absence of encouragement Serious games 13 future versions. The absence of trivial. and levels in some games. description was improved by The visualization of some means of implementing a elements could be improved. demo screen before playing, Most of the issues are The score issues were referred to the score improved by showing the visualization, since it information graphically and complicates the visualization concentrated in the post-game Gaming biofeedback 6 2 minor and 4 trivial. of the game and it is screen. Respecting the uninformative. They also abdominal muscle activity, refer to the abdominal instructions were added to muscle activity: visualization explain to the user how to and artifacts. avoid artifacts. The online platform contains Major and Minor issues the feedback information since referred to lack of feedback it should also be accessible by Training 1 major, 2 minor and 2 information after performing the therapist, as mentioned in 5 evaluation trivial the exercises and also on the Mobile Application how to perform the exercises General section. Other issues (relaxation and contraction). were used to help improving the application. In the major issue, the The major issue is referring visualization was more to the poor visualization of intended to be useful for the Web platform 4 1 major and 4 trivial. the results. Trivial issues study outcomes. The other refer to some issues on the issues were used to help information display. improve the web platform. Most of the issues referred to the feeling of the devices. Vaginal: unpleasant, rigid, In most cases, the issues 1 minor, 9 trivial and 2 lack of low-battery warning, referred to the design, which Devices 12 nihil. only 1 size; abdominal: maintained unaltered for narrow belt, hook-in system. measurability reasons. Also lack of available instructions. Lack of assistance and Improvements implemented messages when connecting through the manual, increasing Device 1 major, 2 minor and 1 4 or having troubles the visual pop-up, when connectivity trivial connecting. Also, names of possible. Issues concerning the devices were unclear. Android were unaltered. In general, the user was confused with the names given to the training sessions when having to select to In general, they were expected 1 minor, 2 trivial and 1 Training schedule 4 recover not-performed issues due to an incomplete nihil. sessions. They were marked development at time of testing. as uncompleted and the user was disappointed with such labels. Int. J. Environ. Res. Public Health 2021, 18, 7800 12 of 15 Table 4. Cont. Category Issues Type Description Comments User reported some errors concerning not being aware of receiving messages since they were shown at the moment the user uses the Some of them were expected App, such as Communications 1 severe, 1 minor, 2 since uncomplete development 7 communications of the with the user trivial and 3 nihil. at time of testing and were therapist, reminders improved later on. (questionnaires, exercises . . . ), etc. Other reported issues were the web accessibility of the instructions (only on paper). 3.2.3. Multidisciplinary Solution Optimization A list of improvements was formed by the team of hardware and software engineers, psychologists and healthcare professionals after a collaborative review of issues, review of proposed improvements and discussion on the impact and feasibility of these proposed improvements. Afterwards, most issues were classified to ensure urgent implementation, and were implemented and solved during the next phase of the project: an evaluation of the func- tionality of the three components of the solution in a pilot study. Overall, urgent issues to be solved were identified in relation to the web platform, the mobile application and the serious games. Few issues were classified as non-urgent, and were implemented and solved in a later phase, prior to evaluation of efficacy in a randomized controlled trial. Some of those issues were related to vaginal device design or device connectivity. 4. Discussion The present study demonstrated the value of patient-centered innovation during the development of an e-Health solution for the conservative treatment of SUI (WOMEN-UP solution) in two different stages: (1) qualitative study, and (2) usability study. 4.1. Stage 1. Qualitative Assessment: Identify Key Stakeholders’ Needs During an early-stage of the design (stage 1), we identified patients’ needs through a qualitative study. We included women aged 29 to 71 years old, which is a good representation of the female population affected by SUI, permitting us to design an e-health solution suitable for a wide range of age. According to the participants, the main benefits of using an e-Health solution with serious games for PFMT were: (1) to receive feedback about the performed pelvic floor contractions; (2) convenient home-use; (3) increasing motivation to perform the PFMT. These data were used by the WOMEN-UP consortium to design a web platform, serious games and a vaginal device. Most participants reported they would like to receive feedback on performance as well as on progress and adjustment. This may explain why three-quarters of participants preferred to use the WOMEN-UP solution in a medical center. Therefore, such supervision by a healthcare professional is one of the key points in the design and development of the WOMEN-UP solution to provide a solution to a patient’s unmet needs. Supervised PFMT is also part of all international recommendations [2] Definition of the new WOMEN-UP solution from patient’s point of view was consid- ered as key information to understand how the first prototype should be built. According to the methods to achieve person-centered care proposed by Bhattacharyya et al. [24], we Int. J. Environ. Res. Public Health 2021, 18, 7800 13 of 15 followed a need-focused approach (customers have unmet needs which are not well de- fined but could be met by new services with new processes that have yet to be developed), providing us with a clear definition of a patient’s needs and of a new solution. Afterwards, to best way to understand what patients really want is by rapidly building a prototype (stage 2) that has enough functionality for potential patients to use, and give feedback, which was done in stage 3 of the present study. 4.2. Stage 3. Usability Testing According to the usability aids found in our study, the use of the WOMEN-UP solution provided autonomy, feedback on performance and made exercises more fun. These findings confirmed the findings from our previous qualitative study (stage 1) and demonstrated the importance of the serious games and the feedback devices within the WOMEN-UP system. In addition, it reassured our design-development plan, which considered the biofeedback and serious games as key factors. Accordingly, feedback was found to be one of the most important game elements in a systematic review focused on the effect of gamification [25]. The longitudinal design of the usability testing provided us with interesting informa- tion about patient’ needs in different moments. Phase 1 provided insight in the usability issues that participants experienced after first-time use and usability issues that positively influenced user experience. Phase 2 provided insight in usability issues that positively influenced user experience and usability issues that persisted to obstruct task performance after prolonged use at home, and potential new issues. Understanding patient perspectives is critical to developing tailored care approaches that address patients’ needs and wants, with the ultimate goal of improving patient satisfaction and quality of life [26]. Most identified usability issues were already planned to be solved in the ongoing de- sign and development of the WOMEN-UP prototype, but such expected barriers confirmed the agreed design by the multidisciplinary team and reassured the investment plan of the resources. In addition, the patient-centered innovation assessment permitted us to identify some unexpected issues, which allowed us to implement new functionalities according to our patients’ needs. Therefore, most of the issues indicated by the study participants were implemented urgently to guarantee a new and improved WOMEN-UP solution (second prototype). This second prototype will be extensively tested for functionality of the three components in a pilot study (will be reported in a follow-on paper). 5. Conclusions In conclusion, this qualitative study and usability testing generated important sug- gestions for the prototype development of an e-Health tool for PFMT. In the context of future eHealth-related patient-centered innovations, we demonstrated the added value of a longitudinal approach that involved patients in an early stage of the design and development. Supplementary Materials: The following are available online at https://www.mdpi.com/article/10 .3390/ijerph18157800/s1, S1: The WOMEN-UP consortium developed a set of 29 standard questions during the qualitative study (stage 1). Author Contributions: Conceptualization, M.E.-P. and S.A.-A.; methodology, M.E.-P., S.A.-A., A.B., A.W.K., B.C.d.G., E.V.V. and J.T.; software, J.F.A., V.S. and B.C.d.G.; formal analysis, A.B., B.C.d.G., A.W.K., E.V.V., J.F.A., V.S. and S.A.-A.; investigation, L.L.-F., J.T. and A.W.K.; resources, B.C.d.G., E.V.V., L.L.-F. and J.T.; data curation, S.A.-A.; writing—original draft preparation, S.A.-A., L.L.-F., M.E.-P., J.F.A. and V.S.; writing—review and editing, S.A.-A., M.E.-P., J.F.A., V.S., A.W.K., B.C.d.G. and A.B.; supervision, S.A.-A. and M.E.-P.; project administration, S.A.-A. and M.E.-P.; funding acquisition, WOMEN-UP Consortium. All authors have read and agreed to the published version of the manuscript. Funding: This project received funding from the European Union’s Horizon 2020 Research and Innovation Programme under grant agreement No. 643535 (WOMEN-UP project). The funder had Int. J. Environ. Res. Public Health 2021, 18, 7800 14 of 15 no involvement in the design and conduct of the survey, in collection, management, analysis and interpretation of the data, or in preparation, review and approval of the manuscript. Institutional Review Board Statement: The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Ethics Committee of Hospital Clínic de Barcelona (protocol code HCB/2016/0480 and date of approval 29 September 2016), of Kuopio University Hos- pital (protocol code 324/13.02.00/2016 and date of approval 10 October 2017) and of the Amsterdam Academic Medical Center (protocol code NL56562.018.16). Informed Consent Statement: Informed consent was obtained from all subjects involved in the study. Data Availability Statement: The data presented in this study are available on request from the corresponding author. The data are not publicly available due to privacy reasons. Acknowledgments: The authors acknowledge Linda Peute for supporting the consortium in analysing the usability data in line with current innovative techniques in eHealth Learning Lab of the Ams- terdam Med-ical centers, AMC supported by the department of Medical Informatics. We thank the clinical partners of the WOMEN-UP consortium for their contribution in participants recruitment, we acknowledge the technical partners of the WOMEN-UP consortium for their contribution to the system development and all members of the WOMEN-UP consortium for their comments on an earlier draft of this article. Finally, the authors acknowledge the patients who have participated in this study. Conflicts of Interest: The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results. The authors declare no conflict of interest. References 1. Milsom, I.; Altman, D.; Cartwrigth, R.; Lapitan, M.C.; Nelson, R.; Sjöström, S.; Tikkinen, K.a.O. Epidemiology of urinary incontinence (UI) and other lower urinary tract symptoms (LUTS), pelvic organ prolapse (POP) and anal (AI)incontinence. In International Consultation on Incontinence, 6th ed.; Abrams, P., Cardozo, L., Wagg, A., Wein, A., Eds.; Health Publications Ltd.: London, UK, 2017; pp. 1–142, ISBN 978–0–9569607-3-3. 2. Dumoulin, C. Adult Conservative Management. Committee 12. In International Consultation on Incontinence, 6th ed.; Abrams, P., Cardozo, L., Wagg, A., Wein, A., Eds.; Health Publications Ltd.: London, UK, 2017; pp. 1443–1628, ISBN 978–0–9569607-3-3. 3. Novara, G.; Checcucci, E.; Crestani, A.; Abrate, A.; Esperto, F.; Pavan, N.; De Nunzio, C.; Galfano, A.; Giannarini, G.; Gregori, A.; et al. Telehealth in Urology: A Systematic Review of the Literature. How Much Can Telemedicine Be Useful during and after the COVID-19 Pandemic? Eur. Urol. 2020, 78, 786–811. [CrossRef] 4. Grimes, C.L.; Balk, E.M.; Crisp, C.C.; Antosh, D.D.; Murphy, M.; Halder, G.E.; Jeppson, P.C.; Lebrun, E.E.W.; Raman, S.; Kim-Fine, S.; et al. A guide for urogynecologic patient care utilizing telemedicine during the COVID-19 pandemic: Review of existing evidence. Int. Urogynecol. J. 2020, 31, 1063–1089. [CrossRef] [PubMed] 5. Dixon-Woods, M.; Amalberti, R.; Goodman, S.; Bergman, B.; Glasziou, P. Problems and promises of innovation: Why healthcare needs to rethink its love/hate relationship with the new. BMJ Qual. Saf. 2011, 20, i47–i51. [CrossRef] [PubMed] 6. Borello-France, D.; Burgio, K.L.; Goode, P.S.; Ye, W.; Weidner, A.C.; Lukacz, E.S.; Jelovsek, J.-E.; Bradley, C.; Schaffer, J.; Hsu, Y.; et al. Adherence to Behavioral Interventions for Stress Incontinence: Rates, Barriers, and Predictors. Phys. Ther. 2013, 93, 757–773. [CrossRef] [PubMed] 7. Or, C.K.L.; Karsh, B.-T. A Systematic Review of Patient Acceptance of Consumer Health Information Technology. J. Am. Med. Inform. Assoc. 2009, 16, 550–560. [CrossRef] 8. Zapata, B.C.; Fernández-Alemán, J.L.; Idri, A.; Toval, A. Empirical Studies on Usability of mHealth Apps: A Systematic Literature Review. J. Med. Syst. 2015, 39, 1. [CrossRef] 9. Gurupur, V.P.; Wan, T.T.H. Challenges in implementing eHealth interventions: A technical perspective. eHealth 2017, 3, 32. [CrossRef] 10. Schnall, R.; Rojas, M.; Bakken, S.; Brown, W.; Carballo-Dieguez, A.; Carry, M.; Gelaude, D.; Mosley, J.P.; Travers, J. A user-centered model for designing consumer mobile health (mHealth) applications (apps). J. Biomed. Inform. 2016, 60, 243–251. [CrossRef] 11. Brown, W.; Yen, P.-Y.; Rojas, M.; Schnall, R. Assessment of the Health IT Usability Evaluation Model (Health-ITUEM) for evaluating mobile health (mHealth) technology. J. Biomed. Inform. 2013, 46, 1080–1087. [CrossRef] 12. Agarwal, S.; LeFevre, A.; Lee, J.; L’Engle, K.; Mehl, G.; Sinha, C.; Labrique, A. Guidelines for reporting of health interventions using mobile phones: Mobile health (mHealth) evidence reporting and assessment (mERA) checklist. BMJ 2016, 352, i1174. [CrossRef] 13. Jaspers, M.W. A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. Int. J. Med. Inform. 2009, 78, 340–353. [CrossRef] Int. J. Environ. Res. Public Health 2021, 18, 7800 15 of 15 14. Giunti, G.; Mylonopoulou, V.; Romero, O.R. More stamina, a gamified eHealth solution for persons with multiple sclerosis: Research through design. J. Med. Internet Res. 2018, 6, e51. [CrossRef] 15. O’Reilly, M.A.; Slevin, P.; Ward, T.; Caulfield, B. A Wearable Sensor-Based Exercise Biofeedback System: Mixed Methods Evaluation of Formulift. JMIR eHealth uHealth 2018, 6, e33. [CrossRef] 16. Thakar, R.; Cundiff, G.W. In pursuit of patient-centered innovation: The role of professional organizations. Int. Urogynecol. J. 2020, 31, 423–428. [CrossRef] 17. Herderschee, R.; Hay-Smith, E.J.C.; Herbison, G.P.; Roovers, J.P.; Heineman, M.J. Feedback or biofeedback to augment pelvic floor muscle training for urinary incontinence in women. Cochrane Database Syst. Rev. 2011, CD009252. [CrossRef] [PubMed] 18. Tăut, D.; Pintea, S.; Roovers, J.-P.W.R.; Mañanas, M.A.; Băban, A. Play seriously: Effectiveness of serious games and their features in motor rehabilitation. A meta-analysis. NeuroRehabilitation 2017, 41, 105–118. [CrossRef] [PubMed] 19. Hertzum, M.; Holmegaard, K.D. Thinking Aloud Influences Perceived Time. Hum. Factors J. Hum. Factors Ergon. Soc. 2014, 57, 101–109. [CrossRef] [PubMed] 20. Nielsen, J.; Clemmensen, T.; Yssing, C. Getting access to what goes on in people’s heads? In Proceedings of the Second Nordic Conference on Human-Computer Interaction-NordiCHI ’02, New York, NY, USA, 19–23 October 2002; Volume 101. [CrossRef] 21. Zhang, J.; Johnson, T.; Patel, V.L.; Paige, D.L.; Kubose, T. Using usability heuristics to evaluate patient safety of medical devices. J. Biomed. Inform. 2003, 36, 23–30. [CrossRef] 22. Venkatesh, V.; Thong, J.Y.; Xu, X. Consumer Acceptance and Use of Information Technology: Extending the Unified Theory of Acceptance and Use of Technology. MIS Q. 2012, 36, 157. [CrossRef] 23. Nielsen, J. Usability Engineering; Morgan Kaufmann Publishers Inc.: San Francisco, CA, USA, 1994; ISBN 9780080520292. 24. Bhattacharyya, O.; Blumenthal, D.; Stoddard, R.; Mansell, L.; Mossman, K.; Schneider, E.C. Redesigning care: Adapting new improvement methods to achieve person-centred care. BMJ Qual. Saf. 2018, 28, 242–248. [CrossRef] 25. Vermeir, J.F.; White, M.J.; Johnson, D.; Crombez, G.; Van Ryckeghem, D.M.L. The Effects of Gamification on Computerized Cognitive Training: Systematic Review and Meta-Analysis. JMIR Serious Games 2020, 8, e18644. [CrossRef] [PubMed] 26. Tarver, M.E.; Neuland, C. Integrating Patient Perspectives into Medical Device Regulatory Decision Making to Advance Innovation in Kidney Disease. Clin. J. Am. Soc. Nephrol. 2021, 16, 636–638. [CrossRef] [PubMed]