Meneses‑Echavez et al. BMC Med Inform Decis Mak (2021) 21:169 https://doi.org/10.1186/s12911-021-01532-8 RESEARCH Open Access Users’ experiences with an interactive Evidence to Decision (iEtD) framework: a qualitative analysis Jose Francisco Meneses‑Echavez1*, Sarah Rosenbaum1, Gabriel Rada2,3, Signe Flottorp1,4, Jenny Moberg1 and Pablo Alonso‑Coello5 Abstract Background: Evidence to Decision (EtD) frameworks bring clarity, structure and transparency to health care decision making. The interactive Evidence to Decision (iEtD) tool, developed in the context of the DECIDE project and pub‑ lished by Epistemonikos, is a stand-alone online solution for producing and using EtD frameworks. Since its develop‑ ment, little is known about how organizations have been using the iEtD tool and what characterizes users’ experi‑ ences with it. This missing information is necessary for any teams planning future developments of the iEtD tool. Methods: This study aimed to describe users’ experiences with the iEtD and identify main barriers and facilitators related to use. We contacted all users registered in the iEtD via email and invited people who identified themselves as having used the solution to a semi-structured interview. Audio recordings were transcribed, and one researcher conducted a directed content analysis of the interviews guided by a user experience framework. Two researchers checked the content independently for accuracy. Results: Out of 860 people contacted, 81 people replied to our introductory email (response rate 9.4%). Twenty of these had used the tool in a real scenario and were invited to an interview. We interviewed all eight users that accepted this invitation (from six countries, four continents). ‘Guideline development’ was the iEtD use scenario they most commonly identified. Most participants reported an overall positive experience, without major difficulties navigating or using the different sections. They reported having used most of the EtD framework criteria. Participants reported tailoring their frameworks, for instance by adding or deleting criteria, translating to another language, or rewording headings. Several people preferred to produce a Word version rather than working online, due to the burden of completing the framework, or lack of experience with the tool. Some reported difficulties working with the exportable formats, as they needed considerable editing. Conclusion: A very limited number of guideline developers have used the iEtD tool published by Epistemonikos since its development. Although users’ general experiences are positive, our work has identified some aspects of the tool that need improvement. Our findings could be also applied to development or improvement of other solutions for producing or using EtD frameworks. Keywords: Decision-making, Clinical decision support, Evidence-based health care, GRADE approach *Correspondence: jose.meneses@fhi.no Background 1 Division for Health Services, Norwegian Institute of Public Health, Oslo, Decision-making in healthcare can be very complex, Norway involving interactions between numerous actors and Full list of author information is available at the end of the article different kinds of information, including evidence from © The Author(s) 2021, corrected publication 2021. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://​creat​iveco​mmons.​org/​licen​ses/​by/4.​0/. The Creative Commons Public Domain Dedication waiver (http://​creat​iveco​mmons.​org/​publi​cdoma​in/​zero/1.​0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. Meneses‑Echavez et al. BMC Med Inform Decis Mak (2021) 21:169 Page 2 of 11 research [1, 2]. Although decisions about treatments, The iEtD is a stand-alone interactive solution for pre- diagnostic tests, coverage, health system or public health paring and facilitating use of GRADE-EtD frameworks interventions involve different types of people and differ- by decision-making groups (e.g., guidelines techni- ent sets of information, they share a common set of fac- cal teams, panels, clinicians, and researchers). It was tors that needs to be taken into account by groups making developed on a technical platform provided by Episte- those decisions: the desirable and undesirable effects, monikos during the DECIDE project, with the aim of values and preferences, acceptability, feasibility, and costs making it intuitive to use for people without much pre- [3, 4] and the certainty of the available evidence. If the vious experience using the technical platform [4, 7, 10]. decision-making process is not well structured, decision The iEtD provides functionality for the needs of three makers may neglect some of these factors or give some main groups: people preparing evidence summaries for of them too much emphasis, resulting in unbalanced use by decision making panels, people making deci- judgments. sions/recommendations, and target audiences for the Decision making also needs to be transparent. If deci- decision or recommendation output. This includes sions are going to be useful, they must be communicated functionality for creating, tailoring and editing frame- to target audiences (e.g., health care professionals, pol- works, for individual or group voting, for document- icy makers or patients) in an easily understandable way ing draft or final judgments, for exporting interim [5, 6]. But people may need to know more than just key and final reports, or for reconsidering evidence, judg- messages; in order to understand the relevance of a deci- ments and final recommendations in other settings. A sion for themselves or their context, they may need to central design feature is the structure, which enables know what considerations and evidence underlie a deci- placement of concise key messages from summarised sion. A systematic approach that helps decision making evidence in close proximity to each decision criterion, groups consider all the relevant factors can facilitate a making it easier for people with different levels of back- more structured and explicit process [3]; a framework ground knowledge to access and interpret the evidence documenting this process can render the decisions and while considering that criterion. The clear separation of underlying considerations more transparent and useful criteria, evidence and judgments facilitates structured, for their target audiences. balanced panel discussions and enhances the transpar- The Grading of Recommendations Assessment, Devel- ency of the final decision/recommendation. By laying opment and Evaluation (GRADE) Working Group the ground for a complete summary of the best avail- started, in 2000, work towards the development of a sys- able evidence to inform judgements, iEtD can facilitate tematic and transparent approach to grading quality of both dissemination and adoption of recommendations evidence and strength of recommendations in healthcare [3, 7, 11]. [7]. As part of the GRADE Working Group’s efforts, the Despite widespread use [4, 8, 12], little is known DECIDE project—a 5-year European Union funded pro- about current utilisation and the user experience of ject aiming to improve the dissemination of evidence- the GRADE-EtD frameworks, nor about the experi- based recommendations—facilitated the development ences of users of the online tools that include them. of different presentations of research evidence for use An evaluation of the GRADE-EtD frameworks in real in decision making, including the GRADE Evidence to guideline panels conducted by members of the GRADE Decision (EtD) frameworks [3, 4]. group (using early paper prototypes of the frameworks) The GRADE-EtD frameworks help to ensure the showed that methodologists and panel members per- important factors that determine a decision (the cri- ceived the frameworks as positive and useful tools that teria) are considered and add structure to discussions. simplified the process of moving from evidence to deci- They can be used to identify reasons for disagreements sions. However, participants identified some sections in a decision-making group. Their structure can help of the frameworks that needed improvements [13]. render decisions transparent to those affected [3, 4, 7]. Panel chairs have also valued the frameworks as a use- The frameworks include: (a) criteria, (b) judgements that ful tool for managing discussions and reaching conclu- must be made in relation to each criterion, (c) evidence sions; keeping guideline panels on track, and dealing to inform each judgement and (d) conclusions based on with disagreements [7]. Since the development of the an overall judgement across all of the criteria [3, 4, 7]. In iEtD online tool for preparing and using frameworks tackling decision making-related complexity, tailorable was completed, no published studies exploring users’ GRADE-EtD framework templates have been prepared experiences have been carried out. Our study was con- for clinical practice guidelines [3], diagnostics [4], health ducted to assess users’ experiences with the iEtD tool, system and public health decisions [8], and coverage and to identify potential tool related barriers, and fac- decisions [9]. tors for improvement. Meneses‑Echavez et al. BMC Med Inform Decis Mak (2021) 21:169 Page 3 of 11 Methods or by Norwegian institutions, in accordance with ACT We used methods from a descriptive qualitative research 2009-06-20 no. 44: Act on medical and health research [14]. We retrospectively described participants’ subjec- (the Health Research Act). This study falls outside of tive experiences of using the iEtD tool, collecting qualita- REC’s remit, because it is not medical or health research tive data through semi-structured interviews. In order to aiming to generate new knowledge about health and dis- analyze findings in a way that would be useful for future eases. Therefore, it does not require submission for REC users and developers, we mapped data onto a framework approval. According to REC, it therefore falls upon the of user experience categories and to a list of tool features, Norwegian Institute of Public Health (NIPH) (employer and ranked these according to their seriousness. We fol- of the lead author) to oversee that the project is carried lowed the standards for reporting qualitative research out responsibly. NIPH does not have formalized ethical [15] (Additional file 1). approval routines but leaves oversight up to the individ- The research team was made up of members of the ual research departments. We have followed the guide- DECIDE project [11] that were involved in the develop- lines for humanities research as laid out by the national ment of both the iEtD and the GRADE-EtD frameworks research ethics foundation in Norway [20]. (GR, PAC, JM, SR). The sum of these experiences shaped the research methods of this study, especially when it Data collection came to the design of the interview guide, and the con- We organized the interviews by inviting participants to tent analysis used. In addition to the preexisting knowl- log into the iEtD and open an EtD framework, either an edge about the iEtD, we judged the research methods actual framework (e.g. from a guideline the participant in this study to be fit for purpose as they are systemati- was involved in) or one used for training purposes that cally developed methods previously used in other stud- the participant was familiar with. Then, using a semi- ies, including the development of the EtD frameworks [7, structured interview guide, we encouraged the partici- 16–18]. pant to engage in ‘think aloud’ while exploring the tool. This is a method where a participant verbalizes their Recruitment thoughts and impressions while exploring a product [21, We sent an introductory email to people registered in 22]. We encouraged participants to be honest about their the iEtD platform, asking if they had used the tool and reactions, looked and listened for signs that they were in what kind of context (e.g. workshop or guideline). experiencing barriers or difficulties, and posed questions We sent two reminders two weeks after the introduc- when they became silent (e.g., how was your experience tory email. We included people who were registered as with formulating the question? did you find any trouble in users in the iEtD platform and who confirmed that they doing so?). Further details are described in the interview had actually used the tool, as opposed to just explored it. guide (Additional file 2). Once eligible participants were identified, they received an email with a brief description of the study’s aim and Interview guide their potential involvement. We adapted an interview guide used by the DECIDE pro- ject [17, 23] that included: Ethical considerations All methods were carried out in accordance with rel- (1) Background questions (education, current work, evant guidelines and regulations. Before the interviews, previous knowledge of systematic reviews/sum- we informed the participants about the voluntary basis of mary of findings/GRADE) their enrolment, which implied that they were allowed to (2) Free exploration (unguided) of iEtD based on sce- withdraw at any time without giving a reason, and that nario text all information collected would be used only for research (3) iEtD walk through, one section at a time with spe- purposes and treated anonymously. We did not collect cial attention to some specific pre-determined any sensitive information from participants and stored sections that we suspected were problematic (e.g., their contact information separately from the data. We export formats) obtained consent to audio record sessions, and erased (4) General impressions recordings after we finished transcribing. Transcriptions (5) Improvement suggestions were rendered and stored anonymously. The Regional Committee for Medical Research Ethics The interview guide (Additional file 2) was based on (REC) [19] has the authority for approving or disapprov- Rosenbaum’s adaptation of Morville’s “honeycomb” ing medical research studies conducted within Norway, framework of user experience (pages 60–62, 108– 116) [2]. We covered seven of the eight facets in this Meneses‑Echavez et al. BMC Med Inform Decis Mak (2021) 21:169 Page 4 of 11 1. Accessibility: are there physical barriers to actually gaining access, also for people with handicaps? 2. Findability: can users locate what they are looking for? 3. Usefulness: does this tool have praccal value for the user? 4. Usability: how easy and satisfying is this product to use? 5. Understandability: do users understand the product and the content correctly? 6. Credibility: is this product/content trustworthy? 7. Desirability: is this product something the user wants/responds posively to? 8. Iden fica on: does this feel like it was designed for “someone like me (user)”? Adapted from Rosenbaum (2). Fig. 1 Honeycomb framework used to guide the interviews and explore users’ experiences with the iEtD. Adapted from Rosenbaum [2]. framework (see Fig. 1). Accessibility was not evalu- meeting software GoToMeeting (https://​www.​gotom​ ated in this study, as it involves checking the applica- eeting.​com) to conduct and record the interviews. tion against a set of technical requirements, but can be assessed through other user experience testing methods We obtained transcripts of all interviews, and one [2]. This framework has been used and adapted based researcher (JM) checked the accuracy and completeness on findings from several similar studies exploring par- of the transcripts compared to the original recordings. ticipants’ experiences of technology designed to facilitate We based our data analysis on the interview transcrip- use of research evidence in health decision making [17, tions, informed by our observational notes. 23, 24]. The framework provides a way of understanding users’ experiences of this kind of information in a way Data analysis that makes direct sense to developers or designers of the We followed a directed content approach [25] to analyze information technology. participants’ experiences in the context of the predefined We did not set any limited time for the interviews, and iEtD structure. We chose this analytical approach due the participants were always prompted to freely provide to its capability to expand conceptually the knowledge as much information as possible about their experience about a phenomenon that has been previously studied, of use. Hence, we probed for more in-depth explanation but that would benefit from more in-depth research [25, of places where participants had problems, frustrations 26]. Existing research shaped the analytical strategy and or were confused. Follow-up questions covered overall its coding framework. impressions and suggestions for improvement. Addition- The analysis started with a deductive content approach. ally, we followed a checklist to ask questions about spe- Two researchers (JM and PAC) read all transcripts and cific pre-determined sections and functionalities of the highlighted data that could be mapped onto a list of fea- iEtD, including presentation formats. Our main interest tures of the iEtD tool (e.g., formulating the question, was to understand user’s experiences of the iEtD. Data assessment, or voting), which was our first coding frame- collection involved documenting the user’s interaction work [11]. This information was complemented with with the iEtD and his/her experiences/reactions to it. observational notes and exported into an Excel spread With participants’ permission, we collected data in two sheet. Then we categorized data according to the facets ways: of the honeycomb framework (e.g. usability, usefulness, understandability, etc.), which constituted our second • Audio recording (recording what the participant said coding framework [2, 27]. while navigating through the iEtD) Rounds of conversations between the researchers were • Observational notes (recording participants’ behav- used to ensure completeness and accuracy of the data. iour and actions, in context with what they said, and Furthermore, we followed an inductive content analysis describing problems we observed). We used remote of the data to uncover findings that expressed explicit or Meneses‑Echavez et al. BMC Med Inform Decis Mak (2021) 21:169 Page 5 of 11 implicit need or desire for features or content that did Council). Participants used the iEtD mostly for guide- not currently exist in the iEtD (e.g., tailoring and sugges- line development, but also for educational purposes (i.e., tions for improvement). training workshops of panels). Two participants reported Throughout the deductive data analysis, we identified expertise in both the GRADE approach and the iEtD; and rated findings according to the severity of the prob- two attended workshops before starting to use the iEtD, lem as expressed by the participant or observed by the and four did not receive any training. All participants researchers: were methodologists who were members of guidelines’ technical teams, not decision-makers or panelists. Most • High (show-stopper, causes incorrect interpretation, participants had sole responsibility within their teams critical errors or high degree of uncertainty or dissat- for completing GRADE-EtD frameworks using the iEtD isfaction) solution. • Medium (causes much frustration or unnecessarily slow use) Main findings • Low (minor or cosmetic problems) We organized findings according to users’ general impressions of the iEtD and the tool-specific tasks users We assigned codes based on our interpretation of the carry out using iEtD. quotes highlighted in the deductive analysis. The codes helped us to understand in which task or location the participant experienced difficulty, and how this related Participants’ general impressions to the facet(s) of the honeycomb framework when inter- Overall, participants had positive experiences working preting the experience of use. We also registered features with the iEtD. They gave several reasons for this, such that participants explicitly liked, participants’ sugges- as the simplicity of the tool, that it was easy to work tions for improvement and nice-to-haves. Finally, we with, and that it was free. Users liked the way the iEtD sorted findings according to their severity (e.g. how much is organized, felt that the tool was designed for some- importance a finding represented for the participant’s one like them, and that it was a useful tool for their ability to use the tool successfully), and corresponding organization(s). Regarding the interaction with panelists iEtD section. and other members of the guideline development group, All the researchers drafted separate lists of problems they perceived the iEtD as logical and easy to follow tool users demonstrated or expressed explicitly, as well as during meetings: participant suggestions for improvements. Any disa- Yes. It was really helpful both for the people compil- greements or discrepancies were resolved by discussion. ing the evidence-to-decision framework, but also as Looking across all the findings, we searched for a more a way [for us] to share it with the people making the coherent way of grouping them that would be prag- decisions. So…we shared them with the guideline matically useful for users or developers. After dialogue groups, and they used the decision-making frame- among project team members, we decided to organize works as they were presented in this format. them according to the main set of tasks that users com- plete when using the iEtD (e.g. tailoring and exporting Nevertheless, some drawbacks emerged from the inter- frameworks). views. Some participants said that due to the amount of information and type of evidence available they had to conduct additional work to synthesize and present Results research evidence (e.g. prepare new tables). Participants Participants working in large groups expressed that it was difficult to We contacted 860 people registered in the iEtD data- coordinate framework completion work across the group. base. The project team sent introductory emails and two reminders in the summer of 2017. Eighty-one registrants I think the difficulty is using it in a group situation. responded to these emails (9.5% response rate), and 61 I think you have to have a very motivated team who (7%) were excluded as they did not complete an entire have all been trained in using the online version to framework for a real group decision context or for edu- be able to really use it well. So I think the challenge cational purposes. Twenty participants were considered for us is that we had a big group with quite a num- iEtD users and were invited to the interviews. Eight par- ber of different people, often from different depart- ticipants agreed to be interviewed. ments, all developing their evidence profiles. So, lots Participants worked in international or national organi- of different people putting the evidence in. So, if it’s zations that developed guidelines (e.g. World Health a very small team I can see that it’s much easier to Organization, Australian Health and Medical Research use the online version compared to a larger team of Meneses‑Echavez et al. BMC Med Inform Decis Mak (2021) 21:169 Page 6 of 11 people who may not be able to use it. However, this difficulty did not stop them using the tool and no other major problems were identified. On the ‘values’, the options are, "important”, "uncer- Getting help to use the tool tainty" or "variability"; “possibly important”, “uncer- Two of the participants expressed they would have liked tainty or variability”; “probably not important or no access to online help or support; however, this did not important”. But the question was: “Is there impor- stop them when using the iEtD. Despite being one of the tant uncertainty about, or variability in how much least commonly used sections of the iEtD as reported by people value the main outcome? That is a hard participants, some considered the help files as useful. question, and everyone had trouble with reading it. Well, first of all the little drop pin boxes that give The way the question is phrased is the variability you instructions are very helpful. So we kept refer- and how much people value it; nearly everybody had ring back to those. problems understanding what that means. Desirable and undesirable effects Despite the overall feed- back being positive, participants consistently expressed Creating GRADE‑EtD frameworks their wish to have both desirable and undesirable effects Formulating the question and background in only one section rather than in two separate sections. Participants did not report any difficulties with the PICO Equity Most participants expressed favorable experi- question section and expressed that the structure of this ences with the use of this criterion. However, some par- section was clear. ticipants referred no clear understanding of its definition: Ah, I think we had trouble with the definition around Assessment "equity". The way that is written and defined… and This section, which includes all the different criteria to how you define is... it wasn’t nicely articulated so be considered by a panel, was the most used section of people had often difficulties with it. Otherwise, most the iEtD. However, not all teams used all of the criteria, things were reasonable. for example when conducting rapid health technology assessments that had no formal health economic analy- They pointed out that there is no information about sis. Participants’ general impressions about this section whether it refers to the intervention or the comparison, and its structure were positive. Moreover, they appreci- and at the time of this judgment, the panel does not know ated being able to distinguish between research evidence about the direction or strength of the recommendation. and additional considerations by placing them in separate To solve this conundrum, two participants suggested cells. moving recommendations right before these three cri- Some participants criticized aspects of this section, teria. Some of the comments from some of the partici- although we observed that this was often coupled with pants reflected suboptimal understanding of the GRADE basic misunderstandings. For instance, some participants approach. demonstrated a poor understanding of some of the cri- I would definitely put "recommendation" under the teria (described below), the purpose of some of the fea- "desirable" and "undesirable effects". In fact, if it were tures (e.g. the rationale behind Additional Considerations up to me, I would do desirable effects, undesirable cells), and more fundamentally, the GRADE approach for effects, and after that I would put the draft recom- formulating recommendations that underlies iEtD struc- mendation. And then I worked through values ture and functionality. For six criteria in the assessment section (Problem, Certainty of the evidence, Balance of effects, Resources Conclusions section required, Cost-effectiveness, Acceptability and Feasibility), Overall experiences with the Conclusions section were participants reported having only positive experiences. positive. For three criteria (Values, Desirable and undesirable effects and Equity), participants had mixed experiences Embedding tabulated summaries that we describe below. Some participants found difficult to insert tables (e.g. Values Some participants found confusing the term Summary of Findings tables) to present the research evi- “Values” (how people value outcomes) in the Assessment dence within the different criteria. This led them to stop section menu, and others found confusing the signaling using the iEtD and moved to Excel. question (Is there important uncertainty about or vari- ability in how much people value the main outcomes?). So it was an easy way for me to use the tool for Meneses‑Echavez et al. BMC Med Inform Decis Mak (2021) 21:169 Page 7 of 11 tables, to do my own tables. And it was too much participant said: “I sent them, together with an instruc- work and it was not fitting because we couldn’t tion document explaining how to use the iEtD. Explain- really... I’m trying to remember exactly what the ing what I did, what we did, and the way they would have issue was but I think the problem is that any study... to interpret what I did”. So I decided to frame the table, the evidence-to- summary table as GRADE does, so starting from Exporting frameworks the outcomes. But then for the same kind of outcome We asked participants about their experience with the we did too many different studies recording the out- (vertical) Word document format that is generated when comes in different ways. So even for the same kind exporting a framework in iEtD. We also showed them of outcome I couldn’t put anything. So eventually I another format from the GRADEpro system, which was decided to use the Excel. horizontal. Participants were also asked to share their experiences with their own formats, which they had pro- duced and tailored. Most of the participants perceived Use of the Export‑to‑Word function the horizontal format as clearer and more logical; they The iEtD was designed to facilitate users to complete deemed the vertical format exported from the iEtD as GRADE-EtD frameworks in a both online and interac- harder to read. Moreover, participants expressed that tive way. The tool was intended to allow people to create the vertical format demanded a lot of further formatting tailored templates for making decisions or recommenda- once it was in Word: tions as well as interactive end user summaries. However, It is repetitive; you see the same tables several such online use was not common among participants we times...messy interviewed. Many participants referred completing their It is not friendly,…., and requires too much editing to work with the GRADE-EtD frameworks in a Word for- be able to generate a document that is easily usable mat rather than online. They logged on to the tool, cre- and readable by decision makers ated a framework and exported it as Word document. I think, while the information is the same it doesn’t Overall, participants reported that other members of the feel like I can see things so well but I think is just guideline development group were satisfied with using because it’s all... it’s feels like it’s more text, which is the iEtD just as a guide to structure the work that then rare because it’s the same text, but it’s not as appeal- continued in Word. ing to me. So for both of those guidelines we downloaded the sheets and used them in Word format. So we used Tailoring frameworks the tool as a template and that’s what we used for Some participants tailored the frameworks. It was com- both guideline meetings, to fill-in for quite a number mon for people to translate and modify the wording, par- of different PICO’s. ticularly of the judgment options. But there are always people that are not confident with online tools. So I asked them, please use the I think it was felt that it was too... introduced too Word file if you want to send me comments much uncertainty, to have the options as they are... some of them we took out the "various" option, so The main reasons why participants preferred to work that we just had, "don’t know", "no", "probably no", with a Word format were lack of confidence in using a "probably yes", "yes". new tool among members of the guideline development group, and their familiarity and perceived ease of use of Participants viewed tailoring as a valuable functionality. Word. It gave them the possibility to modify the frameworks to their specific needs, such as limiting the number of crite- It was easier to get everyone else in the team to use ria for rapid health technology assessments or modifying Word than to use it online. the order of the criteria to improve understanding. People tended to find very difficult to...they were all experts in the field but they are not necessarily Motivations to use the iEtD familiar with that sort of platform Despite some difficulties, participants still expressed Honestly there were also technical issues that I had motivation to use the iEtD. Some chose it because it is to face. Not everyone is so comfortable working on part of the GRADE approach, and they were famil- these things iar with it. The attendance to iEtD workshops was also The non-online use of the iEtD implied extra work for highlighted as a facilitator. Participants said that the sys- the person in charge of completing the frameworks. One tematic and comprehensive structure of the iEtD was a Meneses‑Echavez et al. BMC Med Inform Decis Mak (2021) 21:169 Page 8 of 11 crucial aspect for deciding to use it. They considered it kept crashing, so what we actually moved into was... a suitable tool for producing systematic and transparent we printed the relevant document note, extracted guidelines, as it provided a comprehensive overview of the relevant document, and tables and headings, the different factors involved. Most of the participants and send them to people on an email, and they com- expressed that they would like to receive further training pleted the framework. They send it back to us on the tool. We decided to take the iEtD because we it was a Participants’ suggestions good match between the dimensions considered in Two main suggestions for improvement emerged from framework, to assess the effectiveness and feasibility the interviews: (1) need to provide more guidance, I went through the criteria for the evidence-to-deci- including examples, about what type of information sion framework and I found that it fit quite well with should be include in each of the criteria; (2) need to what I was looking for, a kind of framework or meth- improve the wording of some domain headings, signaling odological system that could allow me to include questions as well as more detailed definitions. We com- everything. So criteria-like values, equity, feasibility, piled a list of problems and potential suggestions for fur- acceptability, were all criteria that we were consid- ther improvement of the iEtD tool (Table 1). ering in our guidelines. So that was eventually the reason.... Discussion In addition, participants expressed they chose the iEtD Main findings partly due to the online voting function that could be Our study explored users’ experiences with the iEtD tool used during meetings. in real scenarios. The majority of participants reported an overall positive experience, without major difficul- Well, we wanted to do real time voting with in the ties navigating or using the different sections. They also panel meeting and so because that feature was reported having used most of the framework criteria available, and because it was easy to migrate from satisfactorily. Participants reported tailoring the frame- MAGIC into this, we decided to go with it works, for instance by adding or deleting criteria, trans- lating to another language, or by rewording headings. Some participants reported concerns with the having to Using iEtD in guideline meetings edit the exportable formats. In the context of guideline meetings, voting was one of We also uncovered some difficulties people experi- the features most valued by participants and received enced using the iEtD. The tool did not work well in large positive feedback from most. However, the ways that panels, probably due to the panelists’ lack of familiarity groups used the voting function varied. For instance, with the tool as well as some technical instability. A more some collected votes manually outside the iEtD, then structured process and group dynamics could facilitate a compiled results and entered them into the system. more favorable experience. A non-online use of the iEtD We did the voting two ways. We started by asking was also common in this study, as participants preferred each panel member to go in and register their vote to export frameworks to a Word format and then work and comment, and that provided a baseline. We with them on paper. Findings revealed that panels had extracted all that information and circulated it to highly varying workflows, technical environments, and the whole group. Then, we met and put the informa- degrees of familiarity with the tool; this demonstrates the tion up on the screen –and did it live–and we read importance of building tools that are highly flexible in through and amended it, and then we all voted. use, a finding with implications for other EtD interactive …only one person in the room had the iEtD frame- technologies. But it also underscores that many technical work opened, projected on the screen, and counted teams prefer to complete their work in a more standard out the votes and recorded them in the iEtD. format. Improving export formats would help those users who prefer to work in Word. A few participants expressed connectivity issues when working online. Our results in the context of previous research The system could not take all ten of us working on Consistent with the findings of the development process the same iEtD, at the same time voting in the same of the iEtD [7], participants in our study found the iEtD way, so we stopped doing that intuitive and easy to work with. Our participants also When we used it live, when everyone was online at highlighted the framework structure as a positive factor the same time and they were all voting together it for facilitating structured discussions among panelists. Meneses‑Echavez et al. BMC Med Inform Decis Mak (2021) 21:169 Page 9 of 11 Table 1 Participants’ suggestions and potential solutions Suggestions Potential solutions Preference for different order and number of sections and criteria Making explicit the already available possibility of collapsing several criteria Make possible to change the order of sections and criteria (e.g., the recom‑ mendation stands at the top rather than at the bottom) Large amount of work and time invested when completing frameworks Raise awareness regarding the iEtD’s flexibility (e.g., that it is not mandatory in the iEtD to include all criteria or to prepare systematic reviews for each crite‑ rion). The amount of work needs to be tailored to the resources of each organization Difficulties when working with large groups Small technical teams might be optimal size; explore what kind of extra guidance might be needed for larger technical teams Make sure good Internet connectivity is available for the work with large groups Rigorous technical testing needed, simulating use by large panels to resolve stability issues Unclear wording of terminology and of the signaling questions in the Improve guidance, both general and contextual Assessment section Improve wording of the criteria. For example, specify for equity whether it refers to the intervention or the comparison Increase training possibilities. For example, providing tutorials or related resources Difficulties when inserting Summary of Findings (SoF) tables Possible integration with GRADE-Pro or to facilitate a more flexible way of including tables in the iEtD Further training on how to use the iEtD and other resources, such as Inter‑ active Summary of Findings (iSoF) tables Preference of some users for the horizontal presentation format (rather Include an additional horizontal presentation format, both for visualization than vertical) of the cells for each criterion and for its exportation Li and colleagues, recently reported similar experi- they had. However, during the interviews, we discov- ences with the use of the GRADE-EtD frameworks in ered that the participants varied much more regarding face-to-face panel meeting discussions for guidelines familiarity with the tool and the GRADE approach than for the management of venous thromboembolism [12]. we anticipated. Earlier research gives us reason to believe In their study, authors found that the frameworks not that degree of familiarity with the GRADE approach will only provided structure but also ensured that the pan- impact people’s experience of the EtD frameworks [7]. elists considered all relevant criteria for making deci- Therefore, we can’t be confident that we have uncovered sions. Guideline panelists also experienced the use of the all the main problems, either for people who are famil- GRADE-EtD frameworks as to be more straightforward iar with GRADE, or likewise, for people who are less when the evidence available was sufficient and clear [12], familiar with GRADE. Our study would likely have been In a study about the development of WHO guidelines for strengthened by focusing on a more homogenous group task shifting, authors describe valuing use of EtD frame- of participants. works to structure discussions about a large and complex This study exhibits some limitations inherent to both body of evidence, including qualitative evidence. How- the methodological design and its analytical strategy. Fol- ever, although they valued expanding the evidence base lowing a directed content analysis means that researchers for decision making, they found that summarizing evi- will approach the data with preconceived ideas about the dence for multiple criteria was very resource demanding phenomenon of interest [25, 26]. This makes them more [28]. Likewise, we observed that some users complained prone to find and communicate findings that are support- about the magnitude of the work involved completing the ive rather than non-supportive for those previous ideas. framework, given the number of criteria included, the This limitation could be reinforced by an overemphasis in implied expectation that they needed to provide evidence the predefined categories of the coding frameworks that for all criteria, and the difficulty of creating succinct evi- guided the analysis. Additionally, some participants may dence summaries. have provided feedback intended to please the research- ers. Furthermore, and despite the flexibility of the Limitations and strengths research methods used, the deductive analysis preceded We carried out in-depth interviews with eight people, the inductive analysis, which could have constrained the providing us with rich data about their experiences with possibilities of identifying additional, unexpected insights iEtD that helped us identify several significant problems from the participants about their experience of use. Meneses‑Echavez et al. BMC Med Inform Decis Mak (2021) 21:169 Page 10 of 11 A potential additional limitation is that the majority of real world testing of the iEtD, for instance with techni- the authors were involved in the development of the iEtD. cal teams who are familiar with the EtD frameworks but However, the iEtD is an expert tool, and it would not be not necessarily with the iEtD. Finally, we also recommend possible to fully comprehend participants’ user experi- exploring experiences of others beyond technical teams, ences without having a comprehensive prior understand- such as panel members, chairs, and end users of the dif- ing of how the tool works and what it can do. None of the ferent presentation formats. members of the had any previous professional relation- ship with the participants interviewed. To the best of our knowledge, this is the first study Conclusions addressing the technical teams’ user experiences with A very limited number of guideline developers have used the use of electronic EtD frameworks. Our findings are the iEtD tool published by Epistemonikos since its devel- applicable for both methodologists and technical team opment. Although users’ general experiences are posi- members of guideline development groups who have tive, our work has identified some aspects of the tool that experience in completing GRADE-EtD frameworks in need improvement. Our findings could be also applied to real scenarios. We followed research rigorous methods development or improvement of other solutions for pro- that have been used in previous studies in the field of ducing or using EtD frameworks. evidence-informed decision-making [7, 13, 17, 23]. Time from actual use to the time of the interview ranged from Abbreviations 3 years in one participant to a couple of months in two GRADE approach: The Grading of Recommendations Assessment, Develop‑ of them. The interview guide helped us ensure a compre- ment and Evaluation (GRADE) approach; GRADE-EtD frameworks: Evidence to Decisions frameworks; GRADEPro GDT: The GRADEPro Guideline Development hensive exploration of the iEtD and facilitated appropri- Tool; iEtD: Interactive Evidence-to-Decision. ate recall by participants. Nevertheless, we do not discard the likelihood of recall bias. Supplementary Information The online version contains supplementary material available at https://​doi.​ org/​10.​1186/​s12911-​021-​01532-8. Implications for practice and research Guideline developers may use our findings to improve their experiences with the iEtD, for instance by using Additional file 1. a trained technical team, helping them develop skills Additional file 2. to create succinct evidence summaries, and providing training for panels, both in the GRADE approach and Acknowledgements the GRADE-EtD frameworks. Panels using the iEtD and Jose F. Meneses-Echavez is a doctoral candidate at the Pediatrics, Obstetrics and Gynecology and Preventive Medicine Department, Universitat Autònoma EtD frameworks in general will benefit from the use of de Barcelona, Barcelona, Spain. a highly structured process and optimal group dynam- ics. Developers of other online tools that include the Authors’ contributions JFME and PAC conceptualized the idea for this study. All authors (JFME, PAC, GRADE-EtD frameworks may also find our results and SR, GR, SF, and JM) contributed to the development of the methods. JFME and suggestions for improvement valuable. PAC conducted the interviews. JFME drafted the manuscript. All authors read In this study, we identified important facilitators of a and approved the final manuscript. positive user experience. Further research will help a bet- Funding ter understanding in the use of online tools for evidence- No funding was received for this research project. informed decision-making processes. Findings from our Availability of data and materials study might serve as a starting point to explore the extent Not applicable. to which guideline development groups and panelists dis- cuss and use multi-criteria frameworks for decision-mak- Declarations ing processes. Further research is also needed on which are the factors that influence the use of online tools for Ethics approval and consent to participate All methods were carried out in accordance with relevant guidelines and decision making, such as group composition, communi- regulations. Before the interviews, we informed the participants about the cation styles, and contextual/situational factors. Various voluntary basis of their enrolment, which implied that they were allowed to methods can be used in further research, such as par- withdraw at any time without giving a reason, and that all information col‑ lected would be used only for research purposes and treated anonymously. ticipatory and non-participatory observation of guide- Informed consent was obtained from all subjects involved in this study. Ethical line panels and workshops, prototype sketching, testing approval was not needed for this study, as recommended by the Norwegian examples, user-test interviews, stakeholder feedback, Committees for Medical and Health Research Ethics. surveys, questionnaires, and discussions in face-to-face Consent for publication meetings [7]. Further research would also benefit from Not applicable. Meneses‑Echavez et al. BMC Med Inform Decis Mak (2021) 21:169 Page 11 of 11 Competing interests 13. Neumann I, Brignardello-Petersen R, Wiercioch W, Carrasco-Labra A, The authors declare that they have no competing interests. Cuello C, Akl E, et al. The GRADE evidence-to-decision framework: a report of its testing and application in 15 international guideline panels. Author details Implement Sci IS. 2016;11:93. 1 Division for Health Services, Norwegian Institute of Public Health, Oslo, 14. Sandelowski M. Whatever happened to qualitative description? Res Nurs Norway. 2 Epistemonikos Foundation, Santiago, Chile. 3 UC Evidence Center, Health. 2000;23(4):334–40. Cochrane Chile Associated Center, Pontificia Universidad Católica de Chile, 15. O’Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for report‑ Santiago, Chile. 4 Institute of Health and Society, University of Oslo, Oslo, Nor‑ ing qualitative research: a synthesis of recommendations. Acad Med. way. 5 Iberoamerican Cochrane Centre, Biomedical Research Institute (IIB Sant 2014;89(9):1245–51. Pau-CIBERESP), Barcelona, Spain. 16. Rosenbaum SE, Glenton C, Cracknell J. User experiences of evidence- based online resources for health professionals: user testing of The Received: 19 January 2021 Accepted: 13 May 2021 Cochrane Library. BMC Med Inform Decis Mak. 2008;8(1):34. 17. Rosenbaum SE, Glenton C, Nylund HK, Oxman AD. User testing and stakeholder feedback contributed to the development of understand‑ able and useful Summary of Findings tables for Cochrane reviews. J Clin Epidemiol. 2010;63(6):607–19. References 18. Rosenbaum SE, Glenton C, Oxman AD. Summary-of-findings tables in 1. Lavis JN, Ross SE, Hurley JE, Hohenadel JM, Stoddart GL, Woodward CA, Cochrane reviews improved understanding and rapid retrieval of key et al. Examining the role of health services research in public policymak‑ information. J Clin Epidemiol. 2010;63(6):620–6. ing. Milbank Q. 2002;80(1):125–54. 19. REK: Norwegian Committees for Medical and Health Research Ethics. 2. Rosenbaum SE. Improving the user experience of evidence. A design Examples of activities that do not require approval from REC. Oslo [cited]. approach to evidence-informed health care. Oslo: Oslo College of Archi‑ Available from https://​helse​forsk​ning.​etikk​om.​no/​regle​rogru​tiner/​sokna​ tecture and Design; 2010. dspli​kt/​soker​ikker​ek?p_​dim=​34999​&_​ikbLa​nguag​eCode=​us. 3. Alonso-Coello P, Schünemann HJ, Moberg J, Brignardello-Petersen R, Akl 20. The National Committee for Research Ethics in the Social Sciences and EA, Davoli M, et al. GRADE Evidence to Decision (EtD) frameworks: a sys‑ the Humanities (NESH). Guidelines for Research Ethics in the Social tematic and transparent approach to making well informed healthcare Sciences, Humanities, Law and Theology [cited]. Available from: https://​ choices. 1: Introduction. BMJ. 2016;353:i2016. www.​forsk​nings​etikk.​no/​en/​guide​lines/​social-​scien​ces-​human​ities-​law-​ 4. Schunemann HJ, Mustafa R, Brozek J, Santesso N, Alonso-Coello P, Guyatt and-​theol​ogy/​guide​lines-​for-​resea​rch-​ethics-​in-​the-​social-​scien​ces-​ G, et al. GRADE Guidelines: 16. GRADE evidence to decision frame‑ human​ities-​law-​and-​theol​ogy/. works for tests in clinical practice and public health. J Clin Epidemiol. 21. Jaspers MWM, Steen T, Bos C, Geenen M. The think aloud method: a 2016;76:89–98. guide to user interface design. Int J Med Inform. 2004;73(11):781–95. 5. Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE. Knowledge transla‑ 22. Eccles DW, Arsal G. The think aloud method: what is it and how do I use tion of research findings. Implement Sci IS. 2012;7:50. it? Qual Res Sport Exerc Health. 2017;9(4):514–31. 6. Liang Z, Howard PF, Leggat SG, Murphy G. A framework to improve 23. Rosenbaum SE, Glenton C, Cracknell J. User experiences of evidence- evidence-informed decision-making in health service management. Aust based online resources for health professionals: user testing of The Health Rev Publ Aust Hosp Assoc. 2012;36(3):284–9. Cochrane Library. BMC Med Inform Decis Mak. 2008;8:34. 7. Rosenbaum SE, Moberg J, Glenton C, Schünemann HJ, Lewin S, Akl E, 24. Rosenbaum SE, Glenton C, Wiysonge CS, Abalos E, Mignini L, Young T, et al. Developing evidence to decision frameworks and an interactive et al. Evidence summaries tailored to health policy-makers in low- and evidence to decision tool for making and using decisions and recom‑ middle-income countries. Bull World Health Organ. 2011;89(1):54–61. mendations in health care. Glob Chall. 2018;2(9):1700081. 25. Hsieh H-F, Shannon SE. Three approaches to qualitative content analysis. 8. Moberg J, Oxman AD, Rosenbaum S, Schunemann HJ, Guyatt G, Flottorp Qual Health Res. 2005;15(9):1277–88. S, et al. The GRADE Evidence to Decision (EtD) framework for health 26. Assarroudi A, Heshmati Nabavi F, Armat MR, Ebadi A, Vaismoradi M. system and public health decisions. Health Res Policy Syst. 2018;16(1):45. Directed qualitative content analysis: the description and elaboration 9. Parmelli E, Amato L, Oxman AD, Alonso-Coello P, Brunetti M, Moberg J, of its underpinning methods and data analysis process. J Res Nurs. et al. GRADE Evidence to Decision (EtD) framework for coverage deci‑ 2018;23(1):42–55. sions. Int J Technol Assess Health Care. 2017;33(2):176–82. 27. US Dept of Health & Human services User Experience Basics. [Cited]. 10. Alonso-Coello P, Oxman AD, Moberg J, Brignardello-Petersen R, Akl EA, Available from https://​www.​usabi​lity.​gov/​what-​and-​why/​user-​exper​ Davoli M, et al. GRADE Evidence to Decision (EtD) frameworks: a sys‑ ience.​html. tematic and transparent approach to making well informed healthcare 28. Glenton C, Lewin S, Gülmezoglu AM. Expanding the evidence base for choices. 2: clinical practice guidelines. BMJ. 2016;353:i2089. global recommendations on health systems: strengths and challenges of 11. DECIDE 2011–2015. Interactive GRADE Evidence to Decision (iEtD) frame‑ the OptimizeMNH guidance process. Implement Sci. 2016;11(1):98. work [cited]. Available from http://​www.​decide-​colla​borat​ion.​eu/​inter​ active-​evide​nce-​decis​ion-​ietd-​frame​work. 12. Li SA, Alexander PE, Reljic T, Cuker A, Nieuwlaat R, Wiercioch W, et al. Evi‑ Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in pub‑ dence to Decision framework provides a structured “roadmap” for making lished maps and institutional affiliations. GRADE guidelines recommendations. J Clin Epidemiol. 2018;104:103–12.