F1000Research 2020, 8:2018 Last updated: 17 APR 2023 STUDY PROTOCOL Learning to make informed health choices: Protocol for a pilot study in schools in Barcelona [version 3; peer review: 2 approved] Laura Martínez García 1,2, Pablo Alonso-Coello1,2, Laia Asso Ministral 3, Clara Ballesté-Delpierre4, Carlos Canelo Aybar1,2, Carol de Britos5, Ana Fernández Rodríguez6, Ana Gallego Iborra7, Victoria Leo Rosas1, Paloma Llaquet5, Ena Pery Niño de Guzmán Quispe 1, Giordano Pérez-Gaxiola 8, Carolina Requeijo 9, Karla Salas-Gama9, Laura Samsó Jofra 9, Jordi Terres10, Iratxe Urreta 11, Sarah Rosenbaum 12 1Centre for Informed Health Choices, Norwegian Institute of Public Health, Oslo, Norway 2Iberoamerican Cochrane Centre (IbCC) - Sant Pau Biomedical Research Institute (IIB-Sant Pau), Barcelona, Spain 3CIBER of Epidemiology and Public Health (CIBERESP), Barcelona, Spain 4Maternal and Child Health Service, General Subdirectorate of Health Promotion, Maternal and Child Health Service. General Subdirectorate of Health Promotion. Public Health Agency of Catalonia, Barcelona, Spain 5ISGlobal, Hospital Clínic, ISGlobal, Hospital Clínic - University of Barcelona, Barcelona, Spain 6Escola Virolai, Barcelona, Spain 7Escola Sant Martí, Barcelona, Spain 8Andalusian Health Service, Malaga, Spain 9Paediatric Hospital of Sinaloa, Sinaloa, Mexico 10Epidemiology and Public Health Department, Hospital de la Santa Creu i Sant Pau, Barcelona, Spain 11Institut Escola Antaviana, Barcelona, Spain 12Clinical Epidemiology and Research Unit, Clinical Epidemiology and Research Unit - University Hospital of Donostia, Donostia, Spain v3 First published: 28 Nov 2019, 8:2018 Open Peer Review https://doi.org/10.12688/f1000research.21292.1 Second version: 12 May 2020, 8:2018 https://doi.org/10.12688/f1000research.21292.2 Approval Status Latest published: 15 Jun 2020, 8:2018 https://doi.org/10.12688/f1000research.21292.3 1 2 version 3 Abstract (revision) Introduction: The Informed Health Choices (IHC) project has 15 Jun 2020 developed learning resources to teach primary school children (10 to 12-year-olds) to assess treatment claims and make informed health version 2 choices. The aim of our study is to explore both the students’ and (revision) view view teachers’ experience when using these resources in the context of 12 May 2020 Barcelona (Spain). Methods: During the 2019-2020 school year, we will conduct a pilot version 1 study with 4 th and 5 th-year primary school students (9 to 11-year- 28 Nov 2019 view view olds) from three schools in Barcelona. The intervention in the schools will include: 1) a workshop with the teachers, and 2) lessons to the Page 1 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 students. The data collection will include: 1) assessment of the IHC resources by the teachers before the lessons, 2) non-participatory 1. Loai Albarqouni , Bond University, Gold observations during the lessons, 3) semi-structured interviews with Coast, Australia the students after a lesson, 4) assessment of the lessons by the teachers after a lesson, 5) treatment claim assessment by the students 2. Luis Gabriel Cuervo , Pan American at the end of the lessons, and 6) assessment of the IHC resources by Health Organization, Washington, USA the teachers at the end of the lessons. We will use ad hoc questionnaires and guides to register the data. We will perform a Sofía Giraldo-Hoyos , Pan American quantitative and qualitative analysis of the data to explore Health Organisation, Washington, USA understandability, desirability, suitability, usefulness, facilitators and barriers of the resources. The most relevant results will be discussed Any reports and responses or comments on the and some recommendations on how to use, how to adapt (if needed), article can be found at the end of the article. and how to implement the IHC resources to this context will be agreed. The findings of the contextualization activities could inform the design of a cluster-randomised trial, to determine the effectiveness of the IHC resources in this context prior to scaling-up its use. Ethical considerations: The study protocol has obtained an approval exemption from the Ethics Committee of the Hospital de la Santa Creu i Sant Pau (Barcelona, Spain). Keywords Children’s health, critical thinking, evidence-based medicine, health education, health promotion, public health. This article is included in the Global Public Health gateway. Page 2 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 Corresponding author: Laura Martínez García (lmartinezg@santpau.cat) Author roles: Martínez García L: Conceptualization, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing; Alonso-Coello P: Conceptualization, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing; Asso Ministral L: Methodology, Writing – Review & Editing; Ballesté-Delpierre C: Methodology, Writing – Review & Editing; Canelo Aybar C: Methodology, Writing – Review & Editing; de Britos C: Methodology, Writing – Review & Editing; Fernández Rodríguez A: Methodology, Writing – Review & Editing; Gallego Iborra A: Methodology, Writing – Review & Editing; Leo Rosas V: Methodology, Writing – Review & Editing; Llaquet P: Methodology, Writing – Review & Editing; Niño de Guzmán Quispe EP: Methodology, Writing – Review & Editing; Pérez-Gaxiola G: Methodology, Writing – Review & Editing; Requeijo C: Methodology, Writing – Review & Editing; Salas-Gama K: Methodology, Writing – Review & Editing; Samsó Jofra L: Methodology, Writing – Original Draft Preparation, Writing – Review & Editing; Terres J: Methodology, Writing – Review & Editing; Urreta I: Methodology, Writing – Review & Editing; Rosenbaum S: Conceptualization, Methodology, Writing – Review & Editing Competing interests: No competing interests were disclosed. Grant information: LMG has a Miguel Servet research contract from the Institute of Health Carlos III [CP18/00007](Cofunded by European Regional Development Fund/European Social Fund, 'Investing in your future'). Dr. Antoni Esteve Foundation has funded the Spanish translation and production of the IHC resources. The funders have not participated in the pilot study design nor will they participate in its development. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Copyright: © 2020 Martínez García L et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. How to cite this article: Martínez García L, Alonso-Coello P, Asso Ministral L et al. Learning to make informed health choices: Protocol for a pilot study in schools in Barcelona [version 3; peer review: 2 approved] F1000Research 2020, 8:2018 https://doi.org/10.12688/f1000research.21292.3 First published: 28 Nov 2019, 8:2018 https://doi.org/10.12688/f1000research.21292.1 Page 3 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 as citizens or as health decision-makers, to develop and imple- REVISED Amendments from Version 2 ment health policies14. In addition, the IHC project focused on We would like to thank the reviewers for their comments to our the child population of low-income countries because making manuscript. In this new version, we have included only minor informed health choices can contribute to a more efficient amendments to the text. use of resources in contexts with higher social and economic Any further responses from the reviewers can be found at the inequality8. end of the article The IHC Working Group has developed several resources to help people understand the differences between trustworthy and untrustworthy health claims, and how to use reliable Introduction information to make informed health choices8. The main In our day-to-day, we hear and make claims about treatments resources are: 1) key concepts, 2) learning resources, and 3) a tool that can improve or worsen our health (“treatment” can be to evaluate the ability to assess treatment claims. defined broadly as any action to improve or maintain the health of individuals). Claims we make, or are exposed to, may be IHC key concepts. Using the principles of a spiral cur- about therapeutic interventions (take drugs, undergo surgery riculum, the IHC Working Group has compiled a list of or use medical devices), changes in lifestyle (follow dietary concepts that individuals need to understand and apply when guidelines, do exercise), interventions involving alternative assessing claims about treatment effects and making health medicine (use medicinal herbs), public health or environmen- choices3,15. tal interventions, or changes in how health care is provided, funded or managed1,2. The list of concepts is reviewed and updated periodically. The list currently includes 44 concepts divided into three capability Many of these claims, regardless of whether they are groups: 1) identify when the treatment claim has an untrust- well-intentioned or driven by various interests, can be wrong, worthy basis, 2) recognise when evidence from comparisons inadequate or untrustworthy3. When people make decisions based of treatments is trustworthy and when it is not, and 3) make on untrustworthy treatment claims, or when they ignore trust- well-informed choices about treatments. Table 1 shows the list worthy claims, they may harm their health and use resources of key concepts13. inadequately3. IHC learning resources. Using a human-centred design In order for people to make informed health choices, they approach16–18, the IHC Working Group has produced various learn- need to be able to obtain, process and understand the relevant ing resources (IHC resources) to teach children and their families health information (health literacy) and use that information to understand and apply some of the key concepts8. from a critical perspective (critical thinking)4–6. Unfortunately, many people lack that ability. A European survey showed that The following resources were produced for primary school chil- 58.3% of the Spanish population has a limited level of health dren (10 to 12-year-olds): a book (that includes and explains literacy7. 12 key concepts), an exercise book, a teachers’ guide, some activity cards, a poster and a song (Figure 1)8,19. The book Informed Health Choices project tells a story, narrated as a comic, about a brother and a sister, The main objective of the Informed Health Choices (IHC) John and Julie, who know two teachers and health research- project is to teach people to assess treatment claims and make ers, professor Compare and professor Fair. The professors teach informed health choices. John and Julie: 1) what questions they should ask when someone says something about a treatment; 2) what ques- The IHC project has a focus on enabling people to learn these tions health researchers ask to find out more about treatment skills at a young age and began their first work in develop- effects; and 3) what questions they should ask when deciding to ing learning resources for primary school children (10 to use a treatment or not19. 12-year-olds) from low-income countries (Uganda)8. There are several reasons the IHC project started with primary school The effect of the resources was assessed in a cluster children: 1) children can learn about fair comparisons (control- randomised trial conducted in Uganda14. In the trial, 120 schools led research) and critical appraisal (in some countries, teaching were assigned randomly to receive the intervention with the these basic capabilities is already part of the curriculum)9,10; resources (60 schools, 76 teachers and 6,383 children) or not 2) primary school interventions can reach a large population receive it (60 schools, 67 teachers and 4,430 children)14. The study group, before many of them leave school11; 3) compared to adults, showed that the children who used the resources improved their children have more time to learn and show less resistance to ability to assess treatment claims in comparison with the group change with regard to their beliefs, attitudes or behaviours12; without resources (69% of the children who use the learning 4) teaching children to think critically improves their academic resources got a passing score vs. 27% of children in the con- performance13; and 5) learning how to think critically about trol group)14. A follow-up study one year later showed that claims about treatment effects can help them, once they become children retained this knowledge and, in fact, the proportion adults, to make decisions about their health and to contribute, of children with a passing score increased from 69% to 80%20. Page 4 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 Table 1. List of key concepts from the Informed Health Choices project13. 1. Beware of treatment claims like these: We hear claims about the effects of treatments all the time. Many of these are not trustworthy. When you hear someone use one of these reasons to support a claim about the effects of a treatment, you should beware and ask where the evidence is. 1.1. Beware of claims that are too good to be true 1. “100% safe!”* 2. “100% effective!” 3. “100% certain!” 1.2. Beware of claims based on faulty logic 4. “Treatment needed!” 5. “It works like this!” 6. “Associated with!” 7. “Real world data!” 8. “No comparison needed!”* 9. “A study shows!”* 10. “Old is better!” * 11. “New is better!”* 12. “More is better!” 13. “Early is better!” 14. “Personalised medicine!” 1.3. Beware of claims based on trust alone 15. “As advertised!”* 16. “It worked for me!”* 17. “Recommended by experts!”* 18. “Peer reviewed!” 2. Check the evidence from treatment comparisons A treatment has to be compared to something else to know what would happen without the treatment. For treatment comparisons to be FAIR, the only important difference between comparison groups should be the treatments they receive. Unfair treatment comparisons and unsystematic summaries of treatment comparisons can be misleading. The way that treatment effects are described can also be misleading. 2.1. Don’t be misled by unfair comparisons! 19. Dissimilar comparison groups* 20. Indirect comparisons 21. Dissimilar attention and care 22. Dissimilar expectations or behaviours* 23. Dissimilar assessment of outcomes 24. Unreliable assessment of outcomes 25. Lots of people not followed-up 26. Outcomes counted in the wrong group 2.2. Don’t be misled by unreliable summaries of 27. Unsystematic summaries treatment comparisons! 28. Selective reporting 29. Unfounded assumptions 2.3. Don’t be misled by how treatment effects are 30. Just words described! 31. Relative effects 32. Average effects 33. Few people or events* 34. Subgroup analyses 35. Statistically significant 36. No confidence interval 37. No evidence Page 5 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 3. Make well-informed treatment choices Deciding what to do requires judgements about the relevance of the evidence, how important the good and bad outcomes are to you, and how sure you can be about the treatment effects. 3.1. What is the problem and what are the options? 38. What is your health problem and what are your options? 3.2. Is the evidence relevant? 39. What outcomes matter to you? 40. Are the people (or animals) very different from you? 41. Are the treatments different from those available to you? 42. Are the circumstances different from yours? 3.3. Do the advantages outweigh the 43. Do the advantages outweigh the disadvantages for you?* disadvantages? 44. How sure are you about the treatment effects? * The 12 concepts included in the learning resources of the IHC project for primary school children. The IHC Key Concepts’ explanations are available from the “That’s a Claim” website. This table has been reproduced with permission from Oxman et al. (Box 3)13. Figure 1. Learning resources from the Informed Health Choices project for primary school children translated into Spanish. Page 6 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 Additionally, the IHC project team developed and evalu- • E  xplore the feasibility of implementing the IHC resources ated a podcast with several episodes for parents (that introduce in this context. and explain nine key concepts)8,18,21,22. • E  valuate the ability of the students to assess treatment claims Tool to evaluate the ability to assess treatment claims. The and make informed health choices after using the IHC IHC Working Group has created a database with questions resources in this context. to assess people’s understanding and ability to apply the key concepts; the CLAIM Evaluation Tools23. Each question is Methods based on a scenario that involves a claim about a treatment. There During the 2019–2020 school year, we will conduct a pilot are two types of questions: 1) individual multiple-choice questions study with 4th and 5th-year primary school students (9 to and 2) several true or false statements23. 11-year-olds) from three schools in Barcelona, based on methods proposed by the IHC Working Group28. Table 2 shows and This tool is a flexible resource, since people may design a describes the different steps of the pilot study. questionnaire according to the key concepts that they wish to evaluate, selecting the questions that are most relevant for their Participants objectives19. For example, teachers can design questionnaires Establishment of the IHC-Barcelona Working Group. We to assess children, and researchers can design questionnaires to will establish a “coordination group” to lead and coordi- assess interventions or to describe a population’s ability to make nate the pilot study and to ensure it is completed according to informed health choices23. All the questions have been designed the established work plan. We will establish a multidiscipli- to be answered by children over 10 years of age as well as nary “advisory group” (researchers, teachers, paediatricians, by adults23. The CLAIM Evaluation Tools can be found on student representatives, family representatives, education and the Testing Treatments international website. health stakeholders, and translators) to review and advise on the development of the different steps of the pilot study. In the previously cited trials that assessed the effect of the IHC resources, the researchers used questions from this We will aim for profile representativeness of the IHC- database14,22. Barcelona Working Group members. We will identify researchers from CIBER of Epidemiology and Public Health Contextualization of the Informed Health Choices project (CIBERESP) and expert colleagues; teachers, student repre- The IHC resources have proven to be effective in the sentatives, and family representatives from selected schools; Ugandan trial, but it is still unknown whether they may be paediatricians from Asociación Española de Pediatría de useful in other contexts24. Different working groups from more Atención Primaria (AEPap); education and health stakehold- than 20 countries are adapting, or planning adaptation, of the ers from Catalan Education and Health Departments; and IHC resources to their context25,26. translators who participated in the IHC resources translation into Spanish. Potential members will be contact and invite to The IHC Working Group has proposed the following contex- participate by email. We will request and register the conflicts tualization activities to explore how these resources can be used of interest of all the members of the IHC-Barcelona Working in a context different from the one that they were originally Group. designed for: 1) context analysis, 2) translation of the resources, 3) pilot study, 3) content adaptation, 5) resource production, and 6) validation of the tool to assess treatment claims24. Selection of the schools. To achieve the objective, we will select a convenience sample of three schools in Barcelona29. Currently, we do not have any specific learning resources to teach The IHC-Barcelona Working Group reach a consensus on eli- primary school children to think critically about their health in gibility criteria of the schools: 1) schools included in the the context of Barcelona (Spain). The working group from the school directory from the Department of Education from the Iberoamerican Cochrane Centre (CCIb) - Sant Pau Biomedical Government of Catalonia (2018–2019); 2) schools that have par- Research Institute (IIB Sant Pau) has translated the IHC resources ticipated in a health promotion programme (2016–2017)30; and into Spanish Spanish based on methods proposed by the IHC 3) schools that take part in the initiative Escola Nova 21 (alli- Working Group27. The Spanish IHC resources were included on ance of schools and civil society institutions for an advanced the IHC website. The next step is to explore how to use and, if education system, carried out between 2016 – 2019, and respond- needed, how to adapt the IHC resources in this context. ing to United Nations and UNESCO’s call for the participation of all sectors in an inclusive process to make possible the Objectives education paradigm shift). We will also take into consideration Primary objective whether the schools include students that are representative of • E  xplore the students’ and teachers’ experience when using the neighbourhood, if they are in different neighbourhoods of the the learning resources of the IHC project in the context of city, and their type of funding (two public schools and one Barcelona (Spain). private or charter school). Secondary objectives Selection of the students and teachers. We will select 4th and • E  xplore potential changes to the IHC resources to adapt 5th-year primary school students (9 to 11-year-olds) in all the them to this context. lines from the selected schools (in this context, the number of Page 7 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 Table 2. Pilot study tasks. Tasks Participants Activities 1. Establishment of the IHC-Barcelona Working Group 1.1. Establishment of the coordination - Researchers Group responsible for planning, coordinating and monitoring the group different steps of the pilot study. 1.2. Establishment of the advisory group - Researchers Group responsible for reviewing and advising during the - Teachers development of the different steps of the pilot study. - Paediatricians - Student representatives - Family representatives - Education and health stakeholders - Translators 2. Protocol development 2.1. Protocol development - IHC-Barcelona Working Develop and publish the pilot study protocol. Request the approval Group of Ethics Committee of the Hospital de la Santa Creu i Sant Pau (Barcelona, Spain). 3. Preparation of the activities in the schools 3.1. Selection of the schools - Coordination group Select three schools in Barcelona (convenience sample). 3.2. Selection of the students and the - Coordination group Select 4th and 5th-year primary school students (10 to 11-year-olds) teachers and teachers. 3.3. Introduction of the pilot study to - Researchers Introduce the IHC project and the pilot study in a meeting with the the families - Teachers families (first meeting of the school year and/or specific meeting about the project). 3.4. Compilation of the informed - Teachers Request the families and the teachers to give their informed consent forms consent (Extended data 1, 2 and 3)31. 3.5. Delivery IHC resources to the - Coordination group Send a book for each student. Send a book, a teachers’ guide, schools activity cards, and a poster for each teacher. 4. Intervention in the schools 4.1. Workshop with the teachers - Researchers Introduce and review the IHC project, the pilot study, and the IHC - Teachers resources with the teachers (Extended data 5)31. 4.2. Lessons to the students - Students Teach students to assess treatment claims and make informed - Teachers health choices using the IHC resources. 5. Data collection 5.1. Assessment of the IHC resources - Teachers Explore the teachers’ initial perception of the IHC resources by the teachers before the lessons (Extended data 6)31. 5.2. Non-participatory observations - Researchers Assess (objectively) the degree of implementation of the IHC during the lessons resources and explore the students’ experience when using the IHC resources (Extended data 7)31. 5.3. Semi-structured interviews with - Students Explore the students’ experience when using the IHC resources the students after a lesson - Researchers (Extended data 8)31. - Teachers 5.4. Assessment of the lessons by the - Teachers Assess (self-reportedly) the degree of implementation of the IHC teachers after a lesson resources and explore the teachers’ experience when using the resources (Extended data 9)31. 5.5. Treatment claim assessment by the - Students Evaluate the ability of the students to assess treatment claims and students at the end of the lessons make informed health choices after using the IHC resources in this context (the questionnaire is accessible upon request from the Testing Treatments website to preserve the validity of the questions). 5.6. Assessment of the IHC resources - Teachers Explore the teachers’ final experience when using the IHC by the teachers at the end of the resources and compare their initial perception with the final lessons experience (Extended data 10)31. 6. Data analysis 6.1. Data analysis - Researchers Quantitative and qualitative analysis of the data. 7. Formulation of the recommendations 7.1. Formulation of the recommendations - IHC-Barcelona Working Suggest and agree some recommendations on how to implement Group the IHC resources in this context. 8. Dissemination of the results 8.1. Dissemination of the results - IHC-Barcelona Working Publish in a peer-reviewed journal, publish in several internet Group resources and introduce to the different users of interest. Page 8 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 lines means the number of student groups per academic level). • Resource format (Spanish and/or English, printed and/or We expect to include a convenience sample of approximately digital) 150 students (25 students per class * two lines per school * • Completion of extra activities three schools). We will request written informed consent from the families (Extended data 1 and 2)31. • The teachers will reach an agreement with the IHC- Barcelona Working Group regarding their proposal for We will select one teacher from every 4th or 5th year class in adaptation. the selected schools. We expect to include six teachers (one Data collection teacher per class * two lines per school * three schools). The The data collection will include: 1) assessment of the IHC profile of the participatory teachers, as well as the subject resources by the teachers before the lessons, 2) non-participatory where the lessons will be included (for example, in Science, observations during the lessons, 3) semi-structured interviews Ethics or even Spanish) will depend on the education plan and with the students after a lesson, 4) assessment of the lessons by the availability of the resources in each school. We will request the teachers after a lesson, 5) treatment claim assessment by informed consent from the teachers (Extended data 1 and 3)31. the students at the end of the lessons, and 6) assessment of the IHC resources by the teachers at the end of the lessons. Each Intervention in the schools of the activities is summarised below: The intervention in the schools will include: 1) a workshop with the teachers, and 2) lessons to the students (Extended data 4 1.       Assessment of the IHC resources by the teachers before the provides a description of the intervention using the TIDieR lessons checklist)31,32. Each of the activities is summarised below:  he objective is to explore the teachers’ initial perception T of the IHC resources. 1.       Workshop with the teachers  e will explore the teachers’ initial perception of the W The objective is to introduce and review the IHC IHC resources using an ad hoc self-administered ques- project, the pilot study, and the IHC resources with the tionnaire after the workshop. The questionnaire will teachers. include: teacher’s impression of the students’ expected experience with the IHC resources (understandability, Before the workshop, a paper copy of the IHC resources desirability, suitability, and usefulness), the teach- translated into Spanish will be sent to the teachers for ers’ experience with the IHC resources (understand- their review. During the workshop, a researcher from ability, desirability, suitability, and usefulness), examples the IHC-Barcelona Working Group will introduce the of treatment claims, and comments (Table 3; Extended IHC project and the pilot study. In addition, a mock data 6)31. lesson will be taught as an example (previously selected by the teachers). Finally, a teacher from each 2.       Non-participatory observations during the lessons school will explain the plan to teach the lessons to the  he objectives are to assess (objectively) the degree T student body. The workshop will last approximately five of implementation of the IHC resources and explore and a half hours (Table 3; Extended data 5)31. the students’ experience when using the IHC resources. 2.       Lessons to the students  researcher from the IHC-Barcelona Working Group A The objective is to teach students to assess treat- will make the non-participatory observations during ment claims and make informed health choices using the lessons. For convenience, each lesson will be the IHC resources. observed in two classes (18 observations). Which lesson is going to be observed in each class will be assigned The IHC resources were designed to be used over nine randomly. Each non-participatory observation will be weeks, with one double period (80 min) per week, during audio-recorded and transcribed. The researcher will a single term, and one hour to complete the test at the register his or her observations in an ad hoc guide that end of the term14. In the pilot study, we will require will include: researcher’s impression of the students’ and to read and discus the story during each lesson. teachers’ experience with the IHC resources (understand- Although the teacher will be able to adapt the lessons to ability, desirability, suitability, and usefulness), technique their students depending on the education plan of each used to teach the lesson, the facilitators and barriers to school. The criteria that the teachers must take into teach the lesson, examples of treatment claims, questions, consideration are: and comments (Table 3; Extended data 7)31. Another researcher will check the notes with the recorded audios.  ontinuity of lessons (number of lesson/week and • C The two researchers will resolve potential disagreements number of weeks) by discussion, and if necessary, by consulting a third researcher. • Duration of lessons (number of minutes/lesson) 3.       Semi-structured interviews with the students after a lesson  ompletion of some or all activities and/or exercises • C  he objective is to explore the students’ experience T proposed in the lessons when using the IHC resources. Page 9 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 Table 3. Pilot study variables. Semi- Treatment Assessment Assessment Non- structured Assessment claim of the IHC of the IHC participatory interviews of the assessment resources by resources by Variables observations with the lessons by by the the teachers the teachers during the students the teachers students at before the at the end of lessons after a after a lesson the end of lessons the lessons lesson the lessons 1. Questionnaire identification X X X X X X 2. Students’ experience with the IHC resources (understandability, X X X X X desirability, suitability, and usefulness) 3. Teachers’ experience with the IHC resources (understandability, X X X X desirability, suitability, and usefulness) 4. Technique used to teach the X X lesson 5. Facilitators and barriers to teach X X the lesson 6. Examples of treatment claims X X X 7. Suggestions to improve the lesson X X 8. Questions X X X 9. Comments X X X X X 10. Treatment claim assessment X A researcher from the IHC-Barcelona Working Group  fter teaching each lesson, the teachers will assess A will hold, with the support of a teacher, semi-structured it in an ad hoc self-administered questionnaire. The individual interviews with a selection of students questionnaire will include: teacher’s impression of the after a lesson. For convenience, two interviews will students’ experience with the lesson (understandability, be held per lesson (18 interviews). Which student is desirability, suitability, and usefulness), the teachers’ going to be interviewed in each class will be assigned experience with the lesson (understandability, desirability, randomly (using the alphabetical attendance sheet). suitability, and usefulness), the technique used to teach In the event that any of the selected students does not the lesson, the facilitators and barriers to teach the wish to participate, the next student will be selected lesson, suggestions to improve the lesson, questions, and from the list. Each interview will last approximately comments (Table 3, Extended data 9)31. 30 minutes, and its audio will be recorded and transcribed. The researcher will hold the semi-structured interview 5.       Treatment claim assessment by the students at the end of the using an ad hoc guide that will include: the students’ lessons experience with the IHC resources (understandabil-  he objective is to evaluate the ability of the students T ity, desirability, suitability, and usefulness), examples of to assess treatment claims and make informed health treatment claims, suggestions to improve the lesson, choices after using the IHC resources in this context. questions, and comments (Table 3, Extended data 8)31. Another researcher will check the notes with the recorded  fter completing all the lessons, the students will take A audios. The two researchers will resolve potential disa- a self-administered test (CLAIM questionnaire test) to greements by discussion, and if necessary, by consulting a evaluate their ability to apply the concepts discussed third researcher. during the lessons. The test will include 24 questions 4.       Assessment of the lessons by the teachers after a lesson (15 multiple-choice questions and nine true or false state- The objectives are to assess (self-reportedly) the degree ments) from the CLAIM Evaluation Tools (Table 3; of implementation of the IHC resources and explore the questionnaire is accessible upon request from the the teachers’ experience when using the resources. Testing Treatments website to preserve the validity of Page 10 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 the questions). The evaluation will be in Spanish (even if Dissemination of the results the resources were used in English), on a paper copy, The dissemination activities of the pilot study results will and with a duration of approximately 60 minutes. include: 1) publication in a peer-reviewed journal, 2) publication in several Internet resources (for example, related web pages, 6.       Assessment of the IHC resources by the teachers at the end electronic bulletins and social media), and 3) introduction to of the lessons the different users of interest (researchers, teachers, paediatri- The objectives are to explore the teachers’ final experi- cians, student representatives, family representatives, education ence when using the IHC resources and compare their and health stakeholders, and translators) in conferences, work- initial perception with the final experience. shops and meetings. The implementation activities will include: 1) offering support to the schools that have participated in After completing all the lessons, we will explore the the pilot study and that are interested in including the IHC teachers’ final experience with the IHC resources using resources in the following school years, 2) giving support to an ad hoc self-administered questionnaire. The question- other schools that are interested in including the IHC resources naire will include: teacher’s impression of the students’ in their education plan. experience with the IHC resources (understandability, desirability, suitability, and usefulness), the teachers’ experience with the IHC resources (understandability, User participation desirability, suitability, and usefulness), and comments Representatives from all the different areas of interest (research- (Table 3; Extended data 10)31. ers, teachers, paediatricians, student representatives, family representatives, education and health stakeholders, and trans- Data analysis lators) will be invited to be members of the IHC-Barcelona Quantitative analysis. We will perform a descriptive analysis Working Group. of the categorical variables (absolute and relative frequencies), and the continuous variables (mean and standard deviation or Ethical considerations median and range). The study protocol has obtained an approval exemption (does not include patients, biological specimens or clinical With regard to the treatment claim assessment by the students, data) from the Ethics Committee of the Hospital de la we will show the mean score and the standard deviation, the pro- Santa Creu i Sant Pau (Barcelona, Spain)35. We will inform portion of the students with a passing score (basic knowledge participants about the pilot study and will request their written of the concepts and how to apply them, 13 points or more informed consent (Extended data 1–3)31. If a family does not over 24), and the proportion of the students with a high score want to participate, the student will participate in the lessons (clear knowledge of the concepts and how to apply them, as a curricular activity but will not participate in any of the 20 points or more over 24)33. study data collection activities. We will anonymise the data removing participant and school name. Qualitative analysis. We will register in an excel sheet the feedback from: 1) the initial assessments by the teachers, 2) the Study status non-participatory observations, 3) the semi-structured interviews Figure 2 is a Gantt chart illustrating the schedule of the pilot with the students, 4) the assessment of the lessons by the study. Currently, we have started the intervention in schools teachers, and 5) the final assessments by the teachers. with the teachers’ workshop. We will perform a thematic analysis based on the catego- Discussion ries previously used in the IHC project (seriousness, user It is important that people learn how to think critically about experience, facilitators and barriers, and potential changes) their health and how to make informed choices. The IHC (Table 4)28,34. One researcher will identify, codify, and summa- project tackles this challenge from an innovative perspective rise the feedback using these categories and search for emerging because: 1) it focuses in children and 2) uses learning resources categories; another researcher will check the codification. designed and assessed to facilitate the teaching and learning They will discuss and review the definitions and limits of each process. By introducing the IHC resources in a new context, category. Finally, using the summarised data, they will explore we hope to contribute to the global effort to help people the nature of the phenomena (understandability, desirability, make informed choices regarding their health. suitability, usefulness, facilitators and barriers) and the possible explanations of the results. Our study in the context of current knowledge Formulation of the recommendations Introducing the IHC resources in schools can be considered The IHC-Barcelona Working Group will discuss the most as a health promotion and education intervention36. Accord- relevant results from the qualitative analysis. They will reach ing to the World Health Organization (WHO), the concept of a consensus on the potential changes of the IHC resources (dra- health promotion comprises “the process of enabling people matic changes, major changes, or minor changes) (Table 4). Finally, to increase control over, and to improve their health”37. Addi- they will suggest and agree on recommendations - both for tionally, health education comprises “consciously constructed practice and research purposes - on how to use, how to adapt (if opportunities for learning involving some form of com- needed), and how to implement the IHC resources to this context. munication designed to improve health literacy, including Page 11 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 Table 4. Categories of the qualitative analysis for the pilot study28,34. Categories Description Seriousness for the user Severe problem XXX Issues associated with incorrect (or a lack of) understanding, critical errors, severe lack of interest, or any issue that may result in abandoning the whole exercise, task or lesson Serious problem XX Issues associated with frustration, unnecessarily slow use, or deviation from the lesson guide/plan but that are either resolved or do not interfere with the learning/teaching/use in a critical way Minor problem X Minor or cosmetic issues that probably don’t have consequences for use, such not liking some detail in the drawing Positive feedback with 00 Praise where we should consider changes in the resources changes Positive feedback without 0 Praise that do not involve a change in the resources changes Suggestions i A suggestion made by the participant User experience Understandability Easy for participant to comprehend (content) and recognize (type of product) Desirability Something the participant wants, likes, or has a positive emotional response Suitability Something the participant feels is for “someone like me”, is suitable for use in her context Usefulness Helpful to participant in achieving her goals/tasks/needs Facilitators and barriers Teachers Profiles and competences Teacher’s education and experience in relation to the lessons being taught Understanding of the content being Teachers’ understanding of the context taught Sufficient training The extent to which the teachers received sufficient training in teaching the lessons Self-efficacy Teacher’s confidence in teaching the lessons Fit to the teacher’s teaching style and Teachers’ comfort or ability to adapt the instructions to their style and context (e.g., class size) context Attitudes Teachers’ attitude towards new resources (change), science, critical thinking and independent thinking by the student body (or their role as authorities in the classroom) Beliefs Teachers’ beliefs about the methods or content (e.g., what treatments work or the concepts) Emotions Teachers’ emotions, such as stress or anxiety Motivation Teachers’ motivation to teach the material Positive learning environment Teachers’ ability to create a positive learning environment; for example, encourage discussion, respond positively to questions, engage students Students Literacy Students’ ability to read and understand the material Attendance Students’ attendance or reasons for poor attendance (e.g., long distance to school or inability to pay school fees) Motivation to learn Students’ motivation to learn the new material Attitudes Students’ attitudes towards learning, towards authorities, towards science, towards critical thinking Beliefs Students’ beliefs about the content (e.g., what treatments work or the concepts) Page 12 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 Categories Description Home environment The extent to which the student’s home environment encourages or discourages learning from the lessons Differentiated instruction The extent to which students different learning needs are met Peer influence Positive or negative attitudes of other students towards the material Learning resources Value of the material The extent to which the materials are valued by the teachers and students Compatibility with the curriculum The extent to which the resources fits with the rest of the curriculum and how it is taught Appropriateness of the material The extent to which the resources are relevant, challenging and engaging Credibility of the material The extent to which the teachers and students perceive the resources as credible School system and Time constraints The extent to which there is sufficient time to accommodate environment introducing the new material Competing priorities The extent to which other priorities for the school, teachers or students limit introducing the resources (e.g., preparing for exams) School organisation and The extent to which the school provides an environment that supports management adoption of new subjects, resources and teaching methods School resources, particularly human The extent to which the school has adequate resources to introduce resources the new resources (e.g., human resources, student/teacher ratio, teacher workload, classroom space and classroom resources, such as blackboards and acoustics) Attitudes and beliefs of head teacher Attitudes or beliefs of colleagues that influence the teacher’s interest and other teachers in and ability to teach the material Parent and community involvement Parents’ attitudes towards the new resources or how things are done at the school Regulations Regulations (e.g., Ministry of Education policies and regulations) that affect introducing the new material Political environment Elements of the political environment that affect introducing the new material; for example, authoritarianism or teacher strikes Bureaucracy Bureaucratic arrangements that delay or limit introduction of the new materials, or facilitate introducing them Incentives and disincentives Incentives or disincentives to introduce the new resources for teachers or head teachers Potential changes Dramatic changes Involve creating new IHC resources Major changes Involve changing the IHC resources drawings Minor changes Involve changing the IHC resources text The ‘Facilitators and barriers’ section of this table has been reproduced with permission from Nsangi et al. (Table 1)34. improving knowledge, and developing life skills which are cluster randomised trial14,22. Thus far, there are few studies conducive to individual and community health”37. Health edu- that assess the effect of the learning resources when acquiring cation and promotion interventions in schools have proven to be competences43. Moreover, the available studies show that beneficial for the health of the population38,39. the evaluated textbooks provide little learning support44–46. We must start demanding the same standards for evaluating There are several definitions of critical thinking, as well as educational interventions that are used for evaluating health several strategies to teach how to think critically9,40. In 1990, interventions47. a Delphi panel of experts defined this ability as a “purposeful, self-regulatory judgment which results in interpretation, Study strengths and limitations analysis, evaluation, and inference, as well as explanation of Our proposal has several strengths. Firstly, before this study, the evidential, conceptual, methodological, criteriological, we have translated the IHC resources into Spanish. A translator, or contextual considerations upon which that judgment is researchers, students, teachers, and medical doctors participated based”41. Therefore, promoting critical thinking at schools in the translation process and fit the text of the IHC resources can be not only useful in the health area22, but also in other to this context. Secondly, we have expanded the profile of the curricular areas (e.g., Mathematics, Science, Literacy)42. users of interest (researchers, teachers, paediatricians, student representatives, family representatives, education and health The IHC project offers several learning resources that were stakeholders, and translators) to establish the pilot study’s created accurately and explicitly, and have been assessed in a multidisciplinary working group. Thirdly and lastly, we will pilot Page 13 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 Figure 2. Gantt chart of the pilot study. an intervention in the schools that has already been shown to Data availability be effective in a cluster randomised trial in Uganda, where over Underlying data 100 schools participated (100 teachers and 10,000 children). No data are associated with this article. Our proposal also has some limitations. The main limitation Extended data is using a convenience sample (small, geographically limited, Extended data is available at: https://figshare.com/articles/ and non-representative sample). However, we will provide a IHC_BCNPilotStudy/12221189/1 DOI: https://doi.org/10.6084/ detailed description of our data collection and research context m9.figshare.12221189.v131. to help other stakeholders consider transferring our results to their settings. It is also worth noting that we will not be Data are available under the terms of the Creative Commons assessing the impact of the IHC resources in this study; due Attribution 4.0 International license (CC-BY 4.0). to this, we will not include a control group and we will not have a questionnaire validated to assess treatment claims for students. Acknowledgements Laura Samsó Jofra is a doctoral candidate at the Paediatrics, Implications for practice and research Obstetrics and Gynaecology and Preventive Medicine Department, The next contextualization activities will be: 1) content Universitat Autònoma de Barcelona, Barcelona, Spain. adaptation - if needed, 2) context analysis (exploring factors that can impact scaling up), and 3) validation of the CLAIM ques- We would like to thank the three schools in Barcelona (Escola tionnaire test into Spanish for use in this context. The findings Sant Martí, Escola Virolai and Institut Escola Antaviana) for of the contextualization activities could inform the design of a their participation in the pilot study. Additionally, we would cluster-randomised trial, to determine the effectiveness of the like to thank Escola Virolai for their participation in the IHC resources in this context prior to scaling-up its use. translation of the IHC resources into Spanish. Page 14 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 We would like to thank Dr Andrew Oxman (Centre for Health, Oslo, Norway) for his advice and feedback on an earlier Informed Health Choices, Norwegian Institute of Public version of this protocol. References 1. Chalmers I, Oxman AD, Austvoll-Dahlgren A, et al.: Key Concepts for Informed Working Paper, Norwegian Institute of Public Health, 2016. Health Choices: a framework for helping people learn how to assess treatment Reference Source claims and make informed choices. BMJ Evid Based Med. 2018; 23(1): 29–33. 20. Nsangi A, Semakula D, Oxman AD, et al.: Effects of the Informed Health Choices PubMed Abstract | Publisher Full Text primary school intervention on the ability of children in Uganda to assess 2. Schwitzer G: A guide to reading health care news stories. JAMA Intern Med. the reliability of claims about treatment effects, 1-year follow-up: a cluster- 2014; 174(7): 1183–6. randomised trial. Trials. 2020; 21(1): 27. PubMed Abstract | Publisher Full Text PubMed Abstract | Publisher Full Text | Free Full Text 3. Austvoll-Dahlgren A, Oxman AD, Chalmers I, et al.: Key concepts that people 21. The Informed Healthcare Choices Group: The Health Choices programme need to understand to assess claims about treatment effects. J Evid Based podcast Kampala. Makerere University. 2016. Med. 2015; 8(3): 112–25. 22. Semakula D, Nsangi A, Oxman AD, et al.: Effects of the Informed Health Choices PubMed Abstract | Publisher Full Text podcast on the ability of parents of primary school children in Uganda to 4. Moseley D, Baumfield V, Elliott JG, et al.: Frameworks for Thinking: A Handbook assess claims about treatment effects: a randomised controlled trial. Lancet. for Teaching and Learning. Cambridge: Cambridge University Press, 2005. 2017; 390(10092): 389–398. Publisher Full Text PubMed Abstract | Publisher Full Text 5. Sørensen K, Van den Broucke S, Fullam J, et al.: Health literacy and public 23. Austvoll-Dahlgren A, Semakula D, Nsangi A, et al.: Measuring ability to assess health: a systematic review and integration of definitions and models. BMC claims about treatment effects: the development of the ‘Claim Evaluation Public Health. 2012; 12: 80. Tools’. BMJ Open. 2017; 7(5): e013184. PubMed Abstract | Publisher Full Text | Free Full Text PubMed Abstract | Publisher Full Text | Free Full Text 6. World Health Organization: Health literacy. The solid facts. World Health 24. Martínez García L, Rosenbaum S: Informed Health Choices Group: Organization: Geneva, 2013. Contextualising Informed Health Choices primary school resources for use in Reference Source different countries. 25th Cochrane Colloquium; Edinburgh, UK. 2018. 7. Sørensen K, Pelikan JM, Röthlin F, et al.: Health literacy in Europe: comparative Reference Source results of the European health literacy survey (HLS-EU). Eur J Public Health. 25. The Informed Health Choices Group: Informed Health Choices Newsletter. 2015; 25(6): 1053–8. Norwegian Institute of Public Health. 2019; [Accessed April 16, 2020]. PubMed Abstract | Publisher Full Text | Free Full Text Reference Source 8. The Informed Healthcare Choices Group: Supporting informed healthcare 26. The Informed Health Choices Group: Informed Health Choices Newsletter. choices in low-income countries - final report. IHC Working Paper, Norwegian Norwegian Institute of Public Health, 2020; [Accessed April 16, 2020]. Institute of Public Health, 2018. Reference Source Reference Source 27. The Informed Health Choices group: Guide for translating the Informed Health 9. Oxman A, Martínez García L: Comparison of the Informed Health Choices Key Choices school resources. Informed Health Choices Working Paper, Norwegian Concepts Framework to other frameworks relevant to teaching and learning Institute of Public Health, 2019. how to think critically about health claims and choices: a systematic review Reference Source [version 1; peer review: awaiting peer review]. F1000Res. 2020. 28. The Informed Health Choices group: Guide for piloting the Informed Health Publisher Full Text Choices (IHC) learning resources. IHC Working Paper. Norwegian Institute of 10. Health Research Board – Trials Methodology Research network (HRB-TMRN), Public Health. 2017. National University of Ireland Galway: START – Schools Teaching Awareness of Reference Source Randomised Trials. [Accessed April 16, 2020]. 29. Etikan I, Abubakar Musa S, Sunusi Alkassim R: Comparison of Convenience Reference Source Sampling and Purposive Sampling. American Journal of Theoretical and Applied 11. UNESCO Institute for Statistics: New Methodology Shows that 258 Million Statistics. 2016; 5(1): 1–4. Children, Adolescents and Youth Are Out of School. [Accessed April 16, 2020]. Publisher Full Text Reference Source 30. Salvador M, Bastida A, Martínez SN, et al.: Programes de promoció de la salut 12. Vosniadou S: International handbook of research on conceptual change. 2nd en centres d’educació primària i secundària de Barcelona. Avaluació del curs edition. Oxford: Routledge. 2013. 2016-2017. Barcelona: Agència de Salut Pública de Barcelona. 2018. Publisher Full Text Reference Source 13. Oxman AD, Chalmers I, Austvoll-Dahlgren A, et al.: Key Concepts for assessing 31. Martínez García L: IHC@BCNPilotStudy. figshare. 2020; [Accessed April 30, 2020]. claims about treatment effects and making well-informed treatment choices http://figshare.com/articles/IHC_BCNPilotStudy/12221189/1 [version 2; peer review: 3 approved]. F1000Res. 2019; 7: 1784. 32. Hoffmann TC, Glasziou PP, Boutron I, et al.: Better reporting of interventions: PubMed Abstract | Publisher Full Text | Free Full Text template for intervention description and replication (TIDieR) checklist and 14. Nsangi A, Semakula D, Oxman AD, et al.: Effects of the Informed Health Choices guide. BMJ. 2014; 348: g1687. primary school intervention on the ability of children in Uganda to assess the PubMed Abstract | Publisher Full Text reliability of claims about treatment effects: a cluster-randomised controlled 33. Davies A, Gerrity M, Nordheim LV, et al.: Measuring ability to assess claims trial. Lancet. 2017; 390(10092): 374–388. about treatment effects: establishment of a standard for passing and mastery. PubMed Abstract | Publisher Full Text IHC Working Paper, 2017. 15. The Informed Health Choices Group: Spiral curriculum. [Accessed April 16, 2020]. Reference Source Reference Source 34. Nsangi A, Semakula D, Glenton C, et al.: Informed health choices intervention to 16. Giacomin J: What is human centred design? Des J. 2014; 17(4): 606–623. teach primary school children in low-income countries to assess claims about Publisher Full Text treatment effects: process evaluation. BMJ Open. 2019; 9(9): e030787. 17. Nsangi A, Semakula D, Rosenbaum S, et al.: Development of the Informed Health PubMed Abstract | Publisher Full Text | Free Full Text Choices resources to teach primary school children to assess claims about 35. Sant Pau Biomedical Research Institute (IIB-Sant Pau): Ethics Committee for treatment effects in four countries. IHC Working Paper, Norwegian Institute of research with medicinal products (CEIm). 2020; [Accessed April 16, 2020]. Public Health, 2017. Reference Source Reference Source 36. Sharples JM, Oxman AD, Mahtani KR, et al.: Critical thinking in healthcare and 18. Semakula D, Nsangi A, Oxman M, et al.: Development of mass media resources education. BMJ. 2017; 357: j2234. to improve the ability of parents of primary school children in Uganda PubMed Abstract | Publisher Full Text to assess the trustworthiness of claims about the benefits and harms of treatments. IHC Working Paper, Norwegian Institute of Public Health, 2018. 37. World Health Organization: Health Promotion Glossary. World Health Reference Source Organization: Geneva, 1998. Reference Source 19. Informed Health Choices Group: The Health Choices Book: Learning to think carefully about treatments. A health science book for primary school children. IHC 38. Caan W, Cassidy J, Coverdale G, et al.: The value of using schools as Page 15 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 community assets for health. Public Health. 2015; 129(1): 3–16. 43. Behnke Y: Textbook Effects and Efficacy. The Palgrave handbook of textbook PubMed Abstract | Publisher Full Text studies. New York: Palgrave Macmillan, 2018. 39. Langford R, Bonell CP, Jones HE, et al.: The WHO Health Promoting School Publisher Full Text framework for improving the health and well-being of students and their 44. Stern L, Roseman JE: Can Middle-School Science Textbooks Help Students academic achievement. Cochrane Database Syst Rev. 2014; (4): CD008958. Learn Important Ideas? Findings from Project 2061’s Curriculum Evaluation PubMed Abstract | Publisher Full Text Study: Life Science. J Res Sci Teach. 2004; 41(6): 538–68. 40. Abrami PC, Bernard RM, Borokhovski E, et al.: Strategies for teaching students Publisher Full Text to think critically a meta-analysis. Rev Educ Res. 2015; 85: 275–314. 45. Pudas AK: Investigating Possibilities to Develop Textbooks to Implement Publisher Full Text Global Education in Basic Education Instruction. IARTEM e-Journal. 2013; 41. Facione PA: Critical thinking: A statement of expert consensus for 5(2): 1–22. purposes of educational assessment and instruction. Research findings and Reference Source recommendations. Newark, DE: American Philosophical Association. 1990. 46. Vojir K, Rusek M: Science education textbook research trends: a systematic Reference Source literature review. Int J Sci Educ. 2019; 41(11): 1496–1516. 42. Ruddock G, Sainsbury M: Comparison of the core primary curriculum in Publisher Full Text England to those of other high performing countries. Research Report 47. Ferrer S: Educación basada en la evidencia: ¿qué pedagogías han probado DCSF-RW048. London: Department for Children, Schools and Families, 2008. que funcionan? Agencia SINC, la ciencia es noticia. 2018. Reference Source Reference Source Page 16 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 Open Peer Review Current Peer Review Status: Version 2 Reviewer Report 02 June 2020 https://doi.org/10.5256/f1000research.26302.r63344 © 2020 Albarqouni L. This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Loai Albarqouni 1 The Institute for Evidence-Based Healthcare, Faculty of Health Sciences and Medicine, Bond University, Gold Coast, Qld, Australia 2 The Institute for Evidence-Based Healthcare, Faculty of Health Sciences and Medicine, Bond University, Gold Coast, Qld, Australia Thanks for this opportunity to review this revised manuscript. The authors have revised the manuscript taking into considerations the raised comments. No further comments on this protocol. Competing Interests: No competing interests were disclosed. Reviewer Expertise: Evidence-based practice I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Reviewer Report 02 June 2020 https://doi.org/10.5256/f1000research.26302.r63343 © 2020 Cuervo L. This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Luis Gabriel Cuervo 1 Pan American Health Organization, Washington, DC, USA 2 Pan American Health Organization, Washington, DC, USA Thanks for providing a revised version with substantial improvements including enhanced details Page 17 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 and a Gantt Chart, the revised version1 is an improvement over the original2. The authors need to clarify if this protocol if for an ongoing study or the protocol they intend to implement. Once this has been clarified, the text needs to be amended accordingly. Currently, parts are written in the future tense (abstract, methods, most of the article), parts in the past tense (e.g. acknowledgments), and the implementation dates suggest that they study is way underway (2019-2020 School year, Gantt Chart). That some of the recommendations in the earlier review were not addressed also suggests that perhaps they were received after the fact. Given the disruption COVID-19 has brought, including to the school systems in Spain, it makes sense to point out if the implementation is/was affected and how this will affect the protocol. The paragraph entitled “Formulation of the recommendations” needs to be copy edited for clarity and grammar. Congratulations on your protocol for this important study, I wish you success with the prompt publication of the findings, and the implementation and scaling up of the intervention in Barcelona and beyond, as appropriate. Developing effective critical thinking skills from young ages seems increasingly important and I commend the authors for building on existing knowledge to further advance and expand its implementation. Well done! References 1. Martínez García L, Alonso-Coello P, Asso Ministral L, Ballesté-Delpierre C, et al.: Learning to make informed health choices: Protocol for a pilot study in schools in Barcelona. F1000Research. 2020; 8. Publisher Full Text 2. Martínez García L, Alonso-Coello P, Asso Ministral L, Ballesté-Delpierre C, et al.: Learning to make informed health choices: Protocol for a pilot study in schools in Barcelona. F1000Research. 2019; 8. Publisher Full Text Competing Interests: The peer reviewer did this review in his personal time and he alone is responsible for the views expressed in this comment. His views do not necessarily represent the decisions or policies of the Pan American Health Organization. Reviewer Expertise: Clinical Epidemiology and Biostatistics, Family Medicine, Research for Health, Policies on Research for Health, Health Systems Research, Public Health I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Author Response 09 Jun 2020 Laura Martínez García Comment 1 - Ongoing study Response 1 Page 18 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 We already included the state of the study in the “Study status” section; the text reads: “Figure 2 is a Gantt chart illustrating the schedule of the pilot study. Currently, we have started the intervention in schools with the teachers’ workshop”. We have reviewed the verb tense consistency. According to study status, we wrote the text in the future tense, except the “Introduction”, “Objectives”, and “Acknowledgements” sections. Comment 2 - COVID-19 Response 2 The first version of the “Learning to make informed health choices: Protocol for a pilot study in schools in Barcelona” was published on 28 Nov 2019. We will describe details about adjustments to the COVID19 in the publication of the results of the pilot study. Comment 3 - Formulation of the recommendations Response 3 We have modified the text in the “Methods - Formulation of the recommendations” section according to the suggestion of the reviewers. The text reads: “The IHC-Barcelona Working Group will discuss the most relevant results from the qualitative analysis. They will consensus the potential changes of the IHC resources (dramatic changes, major changes, or minor changes) (Table 4). Finally, they will suggest and agree on recommendations - both for practice and research purposes - on how to use, how to adapt (if needed), and how to implement the IHC resources to this context.” Competing Interests: No competing interests were disclosed. Version 1 Reviewer Report 26 March 2020 https://doi.org/10.5256/f1000research.23446.r61277 © 2020 Cuervo L et al. This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Luis Gabriel Cuervo 1 Pan American Health Organization, Washington, DC, USA 2 Pan American Health Organization, Washington, DC, USA 3 Pan American Health Organization, Washington, DC, USA Sofía Giraldo-Hoyos 1 Pan American Health Organisation, Washington, DC, USA 2 Pan American Health Organisation, Washington, DC, USA 3 Pan American Health Organisation, Washington, DC, USA Page 19 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 We congratulate the authors for developing this protocol for a pilot study to adapt a tested intervention with a lasting effect in education of benefit to children, teachers and families. This is especially important at a time when pseudoscience and false statements frequently confuse people. There is great potential and value in people being able to critically assess information and distinguish valid information from unreliable one. Well done. We were pleased to review this protocol and to provide recommendations to enhance it ahead of the study. Is the rationale for, and objectives of, the study clearly described? The manuscript provides a detailed description of the study previously done in Uganda1,2,3 and needs to provide a similar level of detail of the protocol to be implemented in Barcelona. The rationale and objectives are described under "Contextualization of the Informed Health Choices project" yet the descriptions need to provide in-depth detail of how the pilot is to be prepared, implemented, analysed, reported and used to inform future developments. Although it is a pleasure to read the description they do of the above cited research conducted in Uganda, if length were an issue, the authors can summarize some details by citing the studies and instead offering more details from their own study. The objectives need to explicitly state their intent to to assess the effects of the intervention and replicability of the study in a IHC setting in a Spanish speaking country, or to prepare the IHC project for its scaling up in Barcelona, Spain. This appears to be a central objective to the study. Also describe their aims regarding the scaling-up of the intervention beyond the pilot schools. The information in the abstract needs to be matched to the key elements of the study, thus highlighting the ultimate purpose of the pilot and subsequent studies. Is the study design appropriate for the research question? Please describe the target and study population in detail elaborating on the characteristics of the school districts and system in Barcelona and provide descriptive details of the target population (children, teachers, parents) disaggregating the information where possible by gender and other relevant variables (e.g. socioeconomic stratum, ethnicity, religion, etc.)4,5,6. This way it will be clear if the pilot relates to the target population and/or elicit any limitations that need to be considered in the analysis or conclusions, as well as considerations for subsequent research. Elaborate on the criteria they will be using to select the participating schools in the convenience sample. The sample of teachers will be small thus it is important to illustrate any efforts to determine if these are representative of the broader population of teachers, to deal with imprecision or bias arising from the small sample, and to elaborate on considerations for their subsequent implementation research. How will they control if the selected teachers are over or under performers? We believe there are important conceptual issues to address under Ethical considerations. First, the decision to waive an ethical review falls with the Ethics Committee; apparently this was the case. However, we have some concerns with the rationale presented: 1. Describe what review process is being undertaken to ensure that their is compliance with Page 20 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 local laws, with the assent of children (if appropriate), and whether the informed consent process (and corresponding documents) are appropriate for each of the groups of participants so that there is an informed decision; 2. Elaborate on the processes to protect the privacy of participants considering that this is a small sample size making feasible that identities can be exposed; 3. The rationale for the exemption is flawed because this is research in human subjects. If there was a reason for conducting an expedited review (ideal) or waiving it would be this being a “less than minimal risk” or “minimal risk study with the prospect of direct benefit to individual subjects”. Definitions are provided and surely there is equivalent European regulation that applies.4,5. Clarify what the management will be if a student, teacher or parent decline to participate, considering the whole grade is participating. Will informed consent be managed at the individual or class level? Are sufficient details of the methods provided to allow replication by others? There are aspects that need more detail or perhaps a visual aid, such as: ○ Specify the times when measurements will be done illustrating them with a PERT Chart complementing their timetable (or GANTT). This will make the information clear and also help to determine if the evaluations will be done at an appropriate time, avoiding for example these extending into the holiday season. ○ Describe how the sample was chosen for the pilot and what implications this may have in the preparation of the larger implementation study. ○ Details of the specific skills sought in the IHC-Barcelona Working Group and the Advisory Group are missing. It is of relevance to know what skills and attitudes are sought in the advisory group members to perform a proper job, and how they will be assessed. What criteria will be used in selecting the researchers and experts from the institutions listed? Will education policy makers be involved from the outset (advisable) and what positions do they have in Barcelona’s education system? Kindly describe what is the Escola Nova 21 initiative and how it relates to Barcelona’s education structure and standards. Elaborate on the implications -including bias that can result from the selection, and how this will be controlled. ○ Describe what efforts will be made to determine the replicability and adaptations needed for the study in different settings in Barcelona and beyond. ○ Clarify the criteria that will be used to select the convenience sample as well the the justification for the sample size. Describe the process to be followed in selecting the schools meeting criteria 1, 2 and 3, and how these schools represent the broader education district of Barcelona, and the population of students. Elaborate how you will analyse representation. ○ Provide specific dates for data collection and their place within the school calendar. Page 21 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 ○ Clarify if the selected classes will be co-ed (see note on PROGRESS) and if groups turn out to have >25 students, how will this be managed. Are the data sets clearly presented in a usable and accessible format? ○ The data analysis section would benefit from a table with the variables that will be analysed, and the analysis strategies (e.g. summary statistics, dispersion assessment, etc.) for each variable. ○ Consider ways to minimize the variability of the assessments from teachers. ○ Consider having double data capturing and whether and the thought process behind having one researcher identifying, codifying and summarizing feedback instead of having a duplicate data-capturing with a third person as tie breaker, or other methods to enhance precision and assess consistency (page 10.) ○ In section “Dissemination of the results” elaborate on the engagement and appropriation by education and science authorities as a strategy to enhance implementation and sustainability in Barcelona and beyond. Elaborate on the participation (if any) of science communicators and journalists, and potential advocates for education. ○ Elaborate on the inclusion and exclusion criteria under “User participation” and describe the characteristics and more detailed relevant information of the group. ○ In section “Discussion” highlight any preparations to adjust to the COVID19 (and if this may provide suitable material) and if possible frame the study on relevant policies that promote critical appraisal learning and partnerships with civil society to increase the value and use of research for health. We list a couple of policy documents and surely the authors can illustrate some local and European policies that apply as well6,7 We were delighted to read this study and how it will build on the success of a previous study to facilitate the implementation of an effective intervention in a different setting. Congratulations to the authors! It is important for these new studies to uphold the methodological rigour of the original studies and we have made some recommendations towards this end. A significant part of the protocol elaborates on the earlier studies referenced and some additional enhancements and details are needed to level this protocol with previous studies. We have provided actionable recommendations and we look forward to a revised version. This study has a tremendous potential to address with a key population, the current crisis that pseudo-science, false news, and misleading claims have brought upon societies, and we believe it needs to be advanced reported and developed in collaboration with education and science and technology leaders, seeking to strengthen appropriation and uptake. We also believe that equity considerations need to be reflected in every aspect of this study, and we have made some recommendations towards this. Well done, we look forward to a revised version! References 1. Nsangi A, Semakula D, Glenton C, Lewin S, et al.: Informed health choices intervention to teach primary school children in low-income countries to assess claims about treatment effects: process evaluation. BMJ Open. 2019; 9 (9). Publisher Full Text 2. Nsangi A, Semakula D, Oxman AD, Austvoll-Dahlgren A, et al.: Effects of the Informed Health Choices primary school intervention on the ability of children in Uganda to assess the reliability of claims about treatment effects, 1-year follow-up: a cluster-randomised trial.Trials. 2020; 21 (1): 27 Page 22 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 PubMed Abstract | Publisher Full Text 3. Nsangi A, Semakula D, Oxman A, Austvoll-Dahlgren A, et al.: Effects of the Informed Health Choices primary school intervention on the ability of children in Uganda to assess the reliability of claims about treatment effects: a cluster-randomised controlled trial. The Lancet. 2017; 390 (10092): 374-388 Publisher Full Text 4. WHO: Standards and operational guidance for ethics review of health-related research with human participants. Reference Source 5. Electronic Code of Federal Regulations, Part 46—Protection of Human Subjects. Reference Source 6. WHO: The WHO strategy onresearch for health. Reference Source 7. Pan American Health Organisation World Health Organisation 49th Directing Council and 61st Session of the Regional Committee of WHO. Reference Source Is the rationale for, and objectives of, the study clearly described? Partly Is the study design appropriate for the research question? Partly Are sufficient details of the methods provided to allow replication by others? Partly Are the datasets clearly presented in a useable and accessible format? Not applicable Competing Interests: Luis Gabriel Cuervo is employed by the Pan American Health Organization / World Health Organization (PAHO/WHO) as Senior Advisor on Research for Health. Sofia Giraldo Hoyos is doing a volunteer internship at PAHO/WHO; this review was done in the reviewers’ personal time. Their opinions and contributions are their own and do not necessarily reflect the decisions or policies of PAHO/WHO. We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however we have significant reservations, as outlined above. Author Response 01 May 2020 Laura Martínez García Comment 1.1 - Detailed description of the study Response 1.1 We assume that the details requested by the reviewers in this comment are disaggregated below. We describe our responses to each of the comments below. Comment 1.2 - Objectives Response 1.2 The primary objective of the pilot study is to “explore the students’ and teachers’ experience Page 23 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 when using the learning resources of the IHC project in the context of Barcelona (Spain).” To achieve this objective, we will conduct a pilot study in a convenience sample of three schools in Barcelona. Our principal results will be recommendations - both for practice and research - on how to use, how to adapt (if needed), and how to implement the IHC resources in this context. To evaluate the effect of the intervention, we would have to conduct a cluster-randomised trial [Nsangi 2017]. We already described this limitation in the “Discussion - Study strengths and limitations” section; the text reads “It is also worth noting that we will not be assessing the impact of the IHC resources in this study; due to this, we will not include a control group and we will not have a questionnaire validated to assess treatment claims for students.” Comment 1.3 - Scaling-up the intervention Response 1.3 We have modified the text in the “Discussion - Implications for practice and research” section according to the suggestion of the reviewers. The text reads: “The next contextualization activities will be: 1) content adaptation - if needed, 2) context analysis (exploring factors that can impact scaling up), and 3) validation of the CLAIM questionnaire test into Spanish for use in this context. The findings of the contextualization activities could inform the design of a cluster-randomised trial, to determine the effectiveness of the IHC resources in this context prior to scaling-up its use.”. Comment 1.4 - Abstract Response 1.4 We have modified the text in the “Abstract” section according to the suggestion of the reviewers. The text reads: “The findings of the contextualization activities could inform the design of a cluster-randomised trial, to determine the effectiveness of the IHC resources in this context prior to scaling-up its use.”. Comment 2.1 - Study population Response 2.1 We have included, as extended data, the description of the intervention in the schools using TIDieR checklist. The text now reads “The intervention in the schools will include: 1) a workshop with the teachers, and 2) lessons to the students (Extended data 4 provides a description of the intervention using the TIDieR checklist) [Hoffmann 2014, Martínez García 2020].”. We will describe the study population (children, teachers, parents) and relevant related variables (e.g. gender, socioeconomic stratum, ethnicity, or religion, etc.) in the publication of the results of the pilot study. Comment 2.2 - Eligibility criteria of the sample Response 2.2 Relating the eligibility criteria of the sample, please see “Response 2” for Reviewer 1. Comment 2.3 - Generalisation and implementation considerations Response 2.3 Page 24 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 The primary objective of the pilot study is to “explore the students’ and teachers’ experience when using the learning resources of the IHC project in in the context of Barcelona (Spain).” To achieve this objective, we will conduct a pilot study in a convenience sample of three schools in Barcelona. Our principal results will be recommendations - both for practice and research - on how to use, how to adapt (if needed), and how to implement the IHC resources in this context. To consider to generalise the results, we would have to include a representative sample [Nsangi 2017]. We have included this limitation in the “Discussion - Study strengths and limitations” section; the text reads “The main limitation is using a convenience sample (small, geographically limited, and non-representative sample). However, we will provide a detailed description of our data collection and research context to help other stakeholders consider transferring our results to their settings.”. To evaluate implementation considerations, we would have to conduct a context analysis [IHC 2020]. The mail objectives of a context analysis would be: 1) explore what demand there is for learning resources for teaching critical thinking about health in schools, 2) map where teaching critical thinking about health best fits in the curriculum, 3) identify and examine relevant resources already in use, and 4) explore conditions for introducing new learning resources [IHC 2020]. The development of context analysis in our setting is already considered in the “Discussion - Implications for practice and research” section, the text reads: “The next contextualization activities will be: 1) content adaptation - if needed, 2) context analysis, and 3) validation of the CLAIM questionnaire test into Spanish for use in this context. Finally, a cluster randomised trial with the adapted IHC resources could be conducted to evaluate their effect on the students’ ability to assess treatment claims in this context.”. Comment 2.4- Ethical considerations Response 2.4 We have modified the text in the “Ethical considerations” section according to the suggestion of the reviewers. The text now reads: “The study protocol has obtained an approval exemption (does not include patients, biological specimens or clinical data) from the Ethics Committee of the Hospital de la Santa Creu i Sant Pau (Barcelona, Spain) [CEIm 2020]. We will inform participants about the pilot study and will request their written informed consent (Extended data 1-3) [Martínez García 2020]. If a family does not want to participate, the student will participate in the lessons as a curricular activity but will not participate in any of the study data collection activities. We will anonymise the data removing participant and school name.” Comment 3.1 - Gantt chart Response 3.1 We have modified the text and Figure 2 according to the suggestion of the reviewers. The text now reads “Figure 2 is a Gantt chart illustrating the schedule of the pilot study.” Comment 3.2 - Eligibility criteria of the sample Response 3.2 Relating the eligibility criteria of the sample, please see “Response 2” for Reviewer 1. Page 25 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 Relating the implementation considerations, please see “Response 2.3” above. Comment 3.3 - IHC-Barcelona Working Group Response 3.3 To establish the IHC-Barcelona Working Group, we only considered profession criteria (researchers, teachers, paediatricians, education and health stakeholders, and translators) or family-related criteria (student and family representatives). We will describe detail about IHC-Barcelona Working Group in the publication of the results of the pilot study. Comment 3.4 - Escola Nova 2 Response 3.4 Relating the eligibility criteria of the sample, please see “Response 2” for Reviewer 1. Comment 3.5 - Contextualising IHC resources Response 3.5 The primary objective of the pilot study is to “explore the students’ and teachers’ experience when using the learning resources of the IHC project in the context of Barcelona (Spain).” To achieve this objective, we will conduct a pilot study in a convenience sample of three schools in Barcelona. Our principal results will be recommendations - both for practice and research - on how to use, how to adapt (if needed), and how to implement the IHC resources in this context. The IHC project team has developed several guidelines for contextualising IHC resources [IHC 2019a, IHC 2019b, IHC 2017a, IHC 2017b]. Different working groups from more than 20 countries are adapting, or planning adaptation, of the IHC resources to their context [IHC Newsletter 2019, IHC Newsletter 2020]. Comment 3.6 - Sample and eligibility criteria Response 3.6 Relating the sample and eligibility criteria, please see “Response 2” for Reviewer 1. Comment 3.7 - Gantt chart Response 3.7 The schedule of the pilot study is illustrated in Figure 2. We will describe detail about data collection in the publication of the results of the pilot study. Comment 3.8 - Study population Response 3.8 We will describe detail about co-educational classes or the number of students in the publication of the results of the pilot study. Comment 4.1 - Study variables Response 4.1 The variables of the pilot study are already listed in “Table 3. Pilot study variables”. The reviewers' proposal “to include a data analysis plan” [Simpson 2015] is not a request of the F1000Research journal [https://f1000research.com/for-authors/article-guidelines/study- protocols]. Page 26 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 Comment 4.2 - Variability of the assessments Response 4.2 We will use ad hoc questionnaires and guides to collect the data to minimize the variability of the assessments. We included questionnaires and guides as extended data, available at: https://figshare.com/articles/IHC_BCNPilotStudy/12221189/1. Comment 4.3 - Data collection Response 4.3 We have included a review process to improve the precision of the data that we will collect from “Non-participatory observations during the lessons” and “Semi-structured interviews with the students after a lesson”. The included text reads: “Another researcher will check the notes with the recorded audios. The two researchers will resolve potential disagreements by discussion, and if necessary, by consulting a third researcher.” Comment 4.4 - Dissemination of the results Response 4.4 We already included education and science authorities in the “Dissemination of the results” section; the text reads: “The dissemination activities of the pilot study results will include: 1) publication in a peer-reviewed journal, 2) publication in several Internet resources (for example, related web pages, electronic bulletins and social media), and 3) introduction to the different users of interest (researchers, teachers, paediatricians, student representatives, family representatives, education and health stakeholders, and translators) in conferences, workshops and meetings.”. Relating the implementation considerations, please see “Response 2.3” above. Comment 4.5 - IHC-Barcelona Working Group Response 4.5 Relating the description of IHC-Barcelona Working Group, please see “Response 3.3” above. Comment 4.6 - COVID19 Response 4.6 The first version of the “Learning to make informed health choices: Protocol for a pilot study in schools in Barcelona” was published on 28 Nov 2019 [Martínez García 2019]. We will describe detail about preparations to adjust to the COVID19 in the publication of the results of the pilot study. References ○ [CEIm 2020] Ethics Committee for research with medicinal products (CEIm). 2020 [online]. Available at: http://www.recercasantpau.cat/en/clinical-research/ceic/. [Accessed April 16, 2020]. ○ [Hoffmann 2014] Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348:g1687. ○ [IHC 2017a] The Informed Health Choices group. Guide for piloting the Informed Health Choices (IHC) learning resources. IHC Working Paper, Norwegian Institute of Page 27 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 Public Health, 2017. ○ [IHC 2017b] The Informed Health Choices group. Guide for translating and adapting the Informed Health Choices (IHC) podcast. Informed Health Choices Working Paper, Norwegian Institute of Public Health, 2017. ○ [IHC 2019a] The Informed Health Choices group. Guide for translating the Informed Health Choices school resources. Informed Health Choices Working Paper, Norwegian Institute of Public Health, 2019. ○ [IHC 2019b] The Informed Health Choices group. Resource production guide for translating and adapting Informed Health Choices learning resources. Informed Health Choices Working Paper, Norwegian Institute of Public Health, 2019. ○ [IHC 2020] IHC CHOICE team. Protocol for a context analysis: Exploring the considerations for introducing digital learning resources for critical thinking about health in secondary schools. Informed Health Choices Working Paper, Norwegian Institute of Public Health, 2020. ○ [IHC Newsletter 2019] The Informed Health Choices Group. Informed Health Choices Newsletter 2019 Norwegian Institute of Public Health. 2019 [online]. Available at: https://www.informedhealthchoices.org/wp-content/uploads/2019/03/IHC- Newsletter-2019.pdf [Accessed April 16, 2020]. ○ [IHC Newsletter 2020] The Informed Health Choices Group. Informed Health Choices Newsletter 2020. Norwegian Institute of Public Health. 2020 [online]. Available at: https://www.informedhealthchoices.org/wp-content/uploads/2020/02/2020- Informed-Health-Choices-newsletter-1.pdf. [Accessed April 16, 2020]. ○ [Martínez García 2019] Martínez García L, Alonso-Coello P, Asso Ministral L, Ballesté- Delpierre C, Canelo Aybar C, de Britos C, et al. Learning to make informed health choices: Protocol for a pilot study in schools in Barcelona [version 1; peer review: awaiting peer review]. F1000Research. 2020 (https://f1000research.com/articles/8- 2018). ○ [Martínez García 2020] Martínez García L. IHC@BCNPilotStudy [Internet]. figshare; 2020. Available at: https://figshare.com/articles/IHC_BCNPilotStudy/12221189/1 [Accessed April 30, 2020]. ○ [Nsangi 2017] Nsangi A, Semakula D, Oxman AD, Austvoll-Dahlgren A, Oxman M, Rosenbaum S, et al. Effects of the Informed Health Choices primary school intervention on the ability of children in Uganda to assess the reliability of claims about treatment effects: a cluster-randomised controlled trial. Lancet. 2017;390(10092):374-388. ○ [Simpson 2015] Simpson SH. Creating a Data Analysis Plan: What to Consider When Choosing Statistics for a Study. Can J Hosp Pharm. 2015; 68(4): 311–317. Competing Interests: No competing interests were disclosed. Reviewer Report 10 February 2020 https://doi.org/10.5256/f1000research.23446.r58609 Page 28 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 © 2020 Albarqouni L. This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Loai Albarqouni 1 The Institute for Evidence-Based Healthcare, Faculty of Health Sciences and Medicine, Bond University, Gold Coast, Qld, Australia 2 The Institute for Evidence-Based Healthcare, Faculty of Health Sciences and Medicine, Bond University, Gold Coast, Qld, Australia 3 The Institute for Evidence-Based Healthcare, Faculty of Health Sciences and Medicine, Bond University, Gold Coast, Qld, Australia Thanks for inviting me to review this protocol. The author proposes a pilot study to evaluate the IHC learning resources in 3 primary schools in Barcelona. the protocol is well -designed and - written. Few comments below: Introduction ○ I would cite a few references to support each of the 5 arguments for starting the IHC project in primary school. Methods ○ I would justify the sampling methods & eligibility criteria of the schools and describe if these 3 schools are representatives to the average school in Barcelona (e.g. a number of students). ○ I would consider TIDier Checklist to better describe intervention details. ○ I would describe the time points for each of the outcome measures. ○ Data analysis – ‘qualitative’ variables should be replaced by ‘categorical’ and ‘quantitative’ variables by ‘continuous’ – Both quantitative measures. ○ Authors should describe whether the resources have been translated and validated into Spanish. Is the rationale for, and objectives of, the study clearly described? Yes Is the study design appropriate for the research question? Yes Are sufficient details of the methods provided to allow replication by others? Partly Are the datasets clearly presented in a useable and accessible format? Not applicable Page 29 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 Competing Interests: No competing interests were disclosed. Reviewer Expertise: Evidence-based practice I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. Author Response 01 May 2020 Laura Martínez García Comment 1 - References in the introduction Response 1 We have included relevant references to support the reasons for starting the IHC project in primary school children. The text now reads: “There are several reasons the IHC project started with primary school children: 1) children can learn about fair comparisons (controlled research) and critical appraisal (in some countries, teaching these basic capabilities is already part of the curriculum) [Oxman 2020, START 2020]; 2) primary school interventions can reach a large population group, before many of them leave school [UNESCO 2019]; 3) compared to adults, children have more time to learn and show less resistance to change with regard to their beliefs, attitudes or behaviours [Vosniadou 2013]; 4) teaching children to think critically improves their academic performance [Oxman 2019]; and 5) learning how to think critically about claims about treatment effects can help them, once they become adults, to make decisions about their health and to contribute, as citizens or as health decision-makers, to develop and implement health policies [Nsangi 2017].” Comment 2 - Sample, eligibility criteria, and description of the schools Response 2 Sample We have included a relevant reference to justify the sampling. The text now reads: “To achieve the objective, we will select a convenience sample of three schools in Barcelona [Etikan 2016].” On the other hand, we already highlight this limitation in the “Discussion - Study strengths and limitations” section, the text reads: “The main limitation is using a convenience sample (small, geographically limited, and non-representative sample).” Eligibility criteria We have clarified that the IHC-Barcelona Working Group made a consensus to establish the eligibility criteria of the sample. The text now reads: “The IHC- Barcelona Working Group reach a consensus on eligibility criteria of the schools: 1) schools included in the school directory from the Department of Education from the Government of Catalonia (2018-2019); 2) schools that have participated in a health promotion programme (2016-2017) [Salvador 2018]; and 3) schools that take part in the initiative Escola Nova 21 (alliance of schools and civil society institutions for an advanced education system, carried out between 2016 - 2019, and responding to United Nations and UNESCO’s call for the participation of all sectors in an inclusive process to make possible the education paradigm shift). We will also take into consideration whether the schools include students that are representative of the neighbourhood, if they are in different neighbourhoods of the city, Page 30 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 and their type of funding (two public schools and one private or charter school). Description of the schools We will describe the included schools and discuss their representativeness in the publication of the results of the pilot study. Comment 3 - TIDieR Checklist Response 3 We have included, as extended data, the description of the intervention in the schools using TIDieR checklist. The text now reads “The intervention in the schools will include: 1) a workshop with the teachers, and 2) lessons to the students (Extended data 4 provides a description of the intervention using the TIDieR checklist) [Hoffmann 2014, Martínez García 2020]. Each of the activities is summarised below:” Also, we have combined all extended data in a single document to facilitate its availability to readers. The text in the "Data availability" section now reads: “Extended data is available at: https://figshare.com/articles/IHC_BCNPilotStudy/12167622 [Martínez García 2020].” We have modified the reference according “Martínez García L. IHC@BCNPilotStudy [Internet]. figshare; 2020. Available at: https://figshare.com/articles/IHC_BCNPilotStudy/12221189/1 [Accessed April 30, 2020].” Comment 4 - Time points of outcomes measures Response 4 We have clarified the time points for data collection throughout the manuscript. The activities now read: 1. Assessment of the IHC resources by the teachers before the lessons 2. Non-participatory observations during the lessons 3. Semi-structured interviews with the students after a lesson 4. Assessment of the lessons by the teachers after a lesson 5. Treatment claim assessment by the students at the end of the lessons 6. Assessment of the IHC resources by the teachers at the end of the lessons Comment 5 - Data analysis Response 5 We have amended the text, now it reads: “We will perform a descriptive analysis of the categorical variables (absolute and relative frequencies), and the continuous variables (mean and standard deviation or median and range).” Comment 6 - Translation and validation into Spanish of IHC resources Response 6 We have included a relevant reference to support the method used to translate into Spanish the IHC resources. The text now reads “The working group from the Sant Pau Biomedical Research Institute (IIB Sant Pau) - Iberoamerican Cochrane Centre (CCIb) has translated the Page 31 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 IHC resources into Spanish based on methods proposed by the IHC Working Group [IHC 2019].” On the other hand, we already highlight the strengths and limitations of translation process in the “Discussion - Study strengths and limitations” section, the text reads: ○ Strength: “Firstly, before this study, we have translated the IHC resources into Spanish. A translator, researchers, students, teachers, and medical doctors participated in the translation process and fit the text of the IHC resources to this context.” ○ Limitation: “It is also worth noting that we will not be assessing the impact of the IHC resources in this study; due to this, we will not include a control group and we will not have a questionnaire validated to assess treatment claims for students.” References ○ [Etikan 2016] Etikan I, Abubakar Musa S, Sunusi Alkassim R. Comparison of Convenience Sampling and Purposive Sampling. American Journal of Theoretical and Applied Statistics. 2016;5(1):1-4. ○ [Hoffmann 2014] Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348:g1687. ○ [IHC 2019] The Informed Health Choices group. Guide for translating the Informed Health Choices school resources. Informed Health Choices Working Paper, Norwegian Institute of Public Health, 2019. ○ [Martínez García 2020] Martínez García L. IHC@BCNPilotStudy [Internet]. figshare; 2020. Available at: https://figshare.com/articles/IHC_BCNPilotStudy/12221189/1 [Accessed April 30, 2020]. ○ [Nsangi 2017] Nsangi A, Semakula D, Oxman AD, Austvoll-Dahlgren A, Oxman M, Rosenbaum S, et al. Effects of the Informed Health Choices primary school intervention on the ability of children in Uganda to assess the reliability of claims about treatment effects: a cluster-randomised controlled trial. Lancet. 2017;390(10092):374-388. ○ [Oxman 2020] Oxman A, Martínez García L. Comparison of the Informed Health Choices Key Concepts Framework to other frameworks relevant to teaching and learning how to think critically about health claims and choices: a systematic review [version 1; peer review: awaiting peer review]. F1000Research. 2020. ○ [Salvador 2018] Salvador M, Bastida A, Martínez SN, Juárez O, Artazcoz L. Programes de promoció de la salut en centres d’educació primària i secundària de Barcelona. Avaluació del curs 2016-2017. Barcelona: Agència de Salut Pública de Barcelona; 2018. ○ [START 2020] Health Research Board – Trials Methodology Research network (HRB- TMRN), National University of Ireland Galway. START – Schools Teaching Awareness of Randomised Trials [online]. Available at: https://www.hrb-tmrn.ie/public- engagement/start-competition [Accessed April 16, 2020] ○ [UNESCO 2019] UNESCO Institute for Statistics. New Methodology Shows that 258 Million Children, Adolescents and Youth Are Out of School [online]. Available at: http://uis.unesco.org/sites/default/files/documents/new-methodology-shows-258- million-children-adolescents-and-youth-are-out-school.pdf [Accessed April 16, 2020] ○ [Vosniadou 2013] Vosniadou S. International handbook of research on conceptual change. 2nd edition. Oxford: Routledge, 2013. Page 32 of 33 F1000Research 2020, 8:2018 Last updated: 17 APR 2023 Competing Interests: No competing interests were disclosed. The benefits of publishing with F1000Research: • Your article is published within days, with no editorial bias • You can publish traditional articles, null/negative results, case reports, data notes and more • The peer review process is transparent and collaborative • Your article is indexed in PubMed after passing peer review • Dedicated customer support at every stage For pre-submission enquiries, contact research@f1000.com Page 33 of 33