Powered by EHA HemaTopics (EHA Only) The European Hematology Exam: The Next Step toward the Harmonization of Hematology Training in Europe José-Tomás Navarro1, Gunnar Birgegård2, Janet Strivens3, Wietske W.G. Hollegien4, Naomi van Hattem4, Manon T. Saris4, Marielle J. Wondergem5, Cheng-Hock Toh6, Antonio M. Almeida7 Correspondence: José-Tomás Navarro (e-mail: jnavarro@iconcologia.net). Introduction education experience and educationalists, and had its first meeting in November 2015. This group recommended that the Following the implementation of the European Hematology aim of the European Exam should be to test knowledge according Curriculum,1 the creation of an exam based on it emerged as the to the topics and recommended levels of knowledge outlined in next logical step on the way to harmonize Hematology training the European Hematology Curriculum. The first step was throughout Europe. In 2015, the project of the European benchmarking among national hematology societies and Euro- Hematology Exam was launched by the Education and pean societies of other medical specialties. A survey performed in Curriculum committees of the European Hematology Associa- 2015 among the presidents of 25 European national hematology tion (EHA). The purpose was not to substitute for the national societies showed that 52% of countries conducted an end-of- authorities’ certification role but, instead, to produce a tool to training exam. Additionally, the majority (72%) were willing to assess knowledge and provide a stamp of quality for those who, endorse a European hematology knowledge test. Given their long by passing the exam, reach the standards of knowledge as experience in knowledge assessments, the Swedish and the determined in the European Curriculum. Belgian exams were used for more in-depth benchmarking. A As with all EHA harmonization programs, the final aim of the second benchmarking step focused on European exams carried exam is to ensure high quality patient care throughout Europe. In out by the other specialty societies. The structure and addition, as the exam gains recognition and prestige in different implementation of final exams held by the European Respiratory countries, professional mobility will be facilitated in Europe. The Society, the European Society of Cardiology, and the European exam is open to all professionals with an interest in hematology Society of Anesthesiology were reviewed and information was but the main target group is those who have just finished their exchanged with those responsible for these exams. specialty training. With the above-mentioned objectives, the European Exam Working Group was created, consisting of hematologists with The exam preparation As a result of the above preliminary research, the Exam working The authors have indicated they have no potential conflicts of interest to group decided that the most appropriate approach was to create a disclose. 1 ICO-German Trias i Pujol Hospital. Josep Carreras Research Institute. knowledge test with multiple-choice questions (MCQs). The Universitat Autònoma de Barcelona, Spain strength of this method is that it allows a large sample size of test 2 Institute for Medical Sciences, Uppsala University, Sweden items even within a limited testing time, testing knowledge in 3 Centre for Higher Education Studies, The University of Liverpool several areas with high reproducibility and it can be used in a 4 European Hematology Association, Executive Office, The Hague, The Netherlands web-based format with automatic objective correcting and 5 VU University Medical Center, Amsterdam, the Netherlands scoring.2,3 The questions were to be in English as this is the 6 Roald Dahl Haemostasis & Thrombosis Centre, Royal Liverpool University official language of EHA and the recognized scientific language Hospital, UK 7 worldwide. Additionally, mobility requires a good level of Hospital da Luz, Lisboa, Portugal. English and, on the other hand, translating questions into Copyright © 2019 the Author(s). Published by Wolters Kluwer Health, Inc. on behalf of the European Hematology Association. This is an open access article different languages would raise important interpretation difficul- distributed under the terms of the Creative Commons Attribution-Non ties. Following careful consideration, it was felt that competence Commercial-No Derivatives License 4.0 (CCBY-NC-ND), where it is permissible assessment would be beyond the remit of this project and is best to download and share the work provided it is properly cited. The work cannot assessed by national training programs. be changed in any way or used commercially without permission from the journal. Nevertheless, to ensure high quality of the exam, the MCQ’s HemaSphere (2019) 3:5(e291) should test the highest classification level that can be tested with Citation: Navarro JT, Birgegård G, Strivens J, Hollegien W, van Hattem N, this type of examination. For this purpose, question-writing Saris M, Wondergem M, Toh CH, Almeida A. The European Hematology guidelines were developed by the educational experts of the Exam Exam: The next step toward the harmonization of Hematology training in working group. The Guidelines for the development of Questions Europe. HemaSphere 2019;3:5. http://dx.doi.org/10.1097/ HS9.0000000000000291 & Answers for the European Exam in Hematology gives the 1 HemaTopics HemaTopics guidance on how to construct questions capable of testing skills served as a successful pilot for the implementation of the European other than just simple data recall, such as applying knowledge to Hematology Exam as part of the official Swiss end-of-training exam. practice. Existing guidelines for MCQ writing were studied and In 2019, 4 more national hematology societies organized a validated recommendations taken into account.4 The questions parallel session, to give local participants, who could not access should depict a realistic context and consist of 3 parts: the congress, the possibility to participate. At the same time as in Amsterdam, candidates in Bern (Switzerland), Madrid (Spain), 1. The stem: Typically comprises a clinical scenario that provides Lisbon (Portugal), Athens (Greece) and Ankara (Turkey) took the the background of the knowledge item being tested. The stem exam. must be clinically plausible and comprise a few lines or Two hundred ninety-three candidates have so far participated paragraphs providing all the information required for the in the 3 iterations of the exam, including the parallel sessions. correct answer. It should also include information that makes These candidates have come from 48 countries representing 5 each of the wrong answer options plausible. It may continents (Table 1). incorporate laboratory results, or still images (such as morphology slides, scans or blood films). Questions without Table 1 a clinical stem may be used sparsely for specific knowledge List of Participants in the first 3 European Hematology Exams recall testing. According to the Country of Origin 2. The question: The question should be direct, unambiguous and should address a single issue. The knowledgeable Country 2017 2018 2019 Total % participant should be able to answer the question from the Spain 21 9 41 71 24% stem before reading the options. Turkey 5 6 18 29 10% 3. The answer options: Five options for the participant to choose Switzerland 4 14 10 28 10% from, one of which is the ‘best’ answer (clearly correct) and the Portugal 9 5 13 27 9% Greece 4 4 12 20 7% others of which are plausible alternatives (‘distractors’). United Kingdom 3 7 7 17 6% The guidelines are the pivotal tool for the question writing India 1 7 4 12 4% process, following a set format that was designed to be compliant Netherlands 4 1 5 10 3% and in line with current thinking on best practice. Germany 2 3 2 7 2% Bulgaria 2 3 1 6 2% The next step was to recruit a group of question writers for Belgium 2 3 5 2% producing questions based on the above-mentioned guidelines. Italy 1 1 2 4 1.4% Following suggestions from the Scientific Working Groups of Saudi Arabia 1 3 4 1.4% EHA, 20 experts on the different sections of the European Sweden 1 2 1 4 1.4% Curriculum were selected to participate in the creation of a Ireland 1 2 3 1.0% database of MCQs. The question-writers group was trained in Mexico 2 1 3 1.0% the writing process by the educational experts of the working Qatar 1 2 3 1.0% group in face-to-face workshops. To ensure the quality and United States 1 2 3 1.0% reliability of the questions, a peer review system was set up. Argentina 1 1 2 0.7% Question writers were paired according to sections of the Austria 2 2 0.7% Bahrain 2 2 0.7% Curriculum. These 2 writers exchanged and reviewed each Brazil 2 2 0.7% other’s questions as many times as needed, submitting a “final” Croatia 2 2 0.7% version to the database for later review by an expert panel. Estonia 2 2 0.7% In order to produce each exam, questions from the database Poland 2 2 0.7% were selected semi-randomly, keeping a balance between question Australia 1 1 0.3% difficulty and different curriculum sections. At a final review Belarus 1 1 0.3% meeting, 120 to 130 selected questions were reviewed by a panel of Bosnia and Herzegovina 1 1 0.3% 6 experts. The final exams comprised 100 questions representing Egypt 1 1 0.3% all (sub)sections of the European Hematology Curriculum V3. Finland 1 1 0.3% Examinees were given 2.5 hours to answer all questions. France 1 1 0.3% Hong Kong 1 1 0.3% The exam was run on a secure online platform and each applicant Indonesia 1 1 0.3% answered the exam questions on an individual computer provided Iraq 1 1 0.3% for this purpose. The informatic tool allowed simple statistics to be Japan 1 1 0.3% collected during the exam and a subsequent deeper analysis. Jordan 1 1 0.3% Kazakhstan 1 1 0.3% Kuwait 1 1 0.3% The results of European Hematology Exam Luxembourg 1 1 0.3% Nigeria 1 1 0.3% Participation and profile of cohort. Norway 1 1 0.3% The exam has had 3 sittings to date. The first took place at the Pakistan 1 1 0.3% 22nd Congress of EHA, in Madrid in 2017, the second in Paraguay 1 1 0.3% Romania 1 1 0.3% Stockholm during the 23rd EHA Congress in 2018, and the third, Slovenia 1 1 0.3% recently in Amsterdam at the 24th Congress of EHA. South Africa 1 1 0.3% In addition, in 2018 and 2019, parallel sessions were organized Sri Lanka 1 1 0.3% by national societies. In 2018, the Swiss Society of Hematology United Arab Emirates 1 1 0.3% (SSH) took the initiative to organize a parallel session in Bern, Total 64 80 149 293 100% Switzerland. This session was open to all Swiss candidates and 2 (2019) 3:5 www.hemaspherejournal.com Figure 1. (A) Profession, (B) main field of interest, and (C) main work activity of the participants in the first 3 iterations of the European Exam of Hematology. The median age of the applicants was 36 years (27,73) and exam in the end (42% in 2017 and 41% in 2018). EHA aims to 54.5% were female. The profession, main field of interest and provide access to the exam to as many people as possible. main work activity are shown in the graphics (Fig. 1). Therefore, after the 2018 exam, the reasons for cancellation/not- In the first 2 years, it was noted that a relatively high number of attending were investigated through a survey, to define any causes registered candidates did not subsequently participate in the that could be mitigated. The majority of no-show candidates did 3 HemaTopics HemaTopics have the intention to participate. Reasons for cancellation/not The future of the European Exam project showing were mostly related to not being able to travel to the congress, such as lack of funding, travel/visa issues or inability to Annual exam and National parallel sessions take the time off. A small majority of candidates who could not After 3 successful iterations of the Exam, the EHA Curriculum finally take the exam expressed the view that they would prefer to Committee and the European Exam Working Group were have the exam later in the congress. encouraged to pursue this project further. The fourth iteration The first 2 iterations took place on the day before the start of will be in Frankfurt, Germany, during the 24th Congress of EHA, the Congress. The third iteration took place on the second day and the aim is to hold a yearly exam during the annual EHA of the Congress, as the expectation was that this might make it meeting. easier for candidates to attend, taking into account their After the successful pilot with the Swiss Society of Hematology opportunities for taking days off and receiving travel funding (SSH) in 2018, EHA has invited the national societies of other for visiting the congress. As expected, there was a decrease in European countries to organize a parallel session as well. This the percentage of registered candidates that did not participate will give candidates who cannot travel to the EHA Congress the in the exam (28%). possibility to still participate in the exam. Five national societies have decided to team up with EHA to offer this opportunity in Quality control and methodology their country in 2019, and more societies are considering this for future iterations, including societies from countries outside of To guarantee independent judgment of the quality of the exam Europe. and the passing score definition, EHA collaborates with CITO, an With the backbone of the European Curriculum, EHA has internationally recognized professional research and knowledge been developing an extensive educational program with the final institute in the field of testing and educational measurement. aim of harmonizing Hematology training throughout Europe. To define the cut-off score for the first exam, the Angoff The European Exam is the next step to reach this objective, which method was used.5,6 Applying this method means that 6 judges ultimately will generate higher quality patient care. The strong (the same experts that made the final review of the questions) interest in participation from hematologists inside and outside of evaluated the difficulty of each question, leading to the Europe will give firm support to a wider harmonization. calculation of a final passing score of 64. Fifty-two candidates (81.2%) passed the exam. For the 2018 exam, 2 standard-setting methods were Further development – the progress test combined. First, the Angoff method, and second, an equating The development of trainees’ knowledge over the years of procedure based on item response theory, where the results of the training can be monitored by progress testing, a method used in 2018 exam could be placed on the same ability scale as the 2017 undergraduate medical training for many years. A progress test exam by means of a number of overlapping questions used in can be used as an examination7 but also as a learning tool, testing both exams. This means that independent of the year in which a knowledge and giving feedback to trainees about their knowledge candidate participates, the chance that they pass the exam is development. Such a progress test has been used successfully in equal. For 2018, this resulted in a cut-off score of 58 (or more) Sweden since 2013. Using MCQs from the large question items correct to pass the exam. This means that 64 candidates databank developed for the EHA exam such a learning tool will (80.0%) passed. now be implemented on the EHA Campus on-line platform and CITO’s Psychometric Research- and Knowledge Center performed a psychometric analysis of the results of both iterations, testing the discriminatory power of questions and other markers of test quality. In both cases CITO deemed the exam to be a robust method of knowledge testing. At the time of writing this article, the outcomes of the 2019 exam are still being analyzed. These will be published in the Exam report on the EHA website. Questionnaires Directly after having finished the exam questions, the participants were asked to participate in a short survey on the same platform as the exam. In 2018, 97% found the exam relevant (78%) or partly relevant (19%) to their hematology training. The 87% experienced the exam setup as good or very good, and 95% of candidates were able to finish the exam within the 2.5 hours. To get insight in the impact of achieving success in the European Hematology Exam, the candidates who passed the 2017 and 2018 editions were asked to complete a short survey by e-mail. 39 candidates (33.6%) responded. 62% of the candidates who responded expected that passing this would help in their career in the future, and 10% reported that it had helped already. The 72% indicated that passing the exam helped in their personal Figure 2. The chart shows the main reasons for participating in the development. Figure 2 shows the main reasons for participating exam reported by the candidates. in the exam. 4 (2019) 3:5 www.hemaspherejournal.com will help trainees to follow their development of knowledge, to 3. Pham H, Trigg M, Wu S, et al. Choosing medical assessments: Does the multiple-choice question make the grade? Educ Health. 2018;31:65–71. inform themselves of areas in need of extra study and to prepare 4. Case SM, Swanson DB (2002) Constructing Written Test Questions for for the final exam. the Basic and Clinical Sciences (3rd Ed.) Philadelphia: National Board of Medical Examiners. 2002 Retrieved from: https://www.nbme.org/pdf/ References itemwriting_2003/2003iwgwhole.pdf (30/04/190. 5. 1971;Angoff WH. Thorndike RL. Scales, norms, and equivalent scores. 1. Almeida A, Ar C, Hellström-Lindberg E, et al. The European hematology Educational, curriculum: an electronic passport promoting professional competence 6. Ricker KL. Setting cut-scores: a critical review of the angoff and modified and mobility. Hemasphere. 2018;2:e49. angoff methods. Albert J EducatRes. 2006;52:53–64. 2. Little JL, Bjork EL, Bjork RA, et al. Multiple-choice tests exonerated, at 7. Muijtjens AM, Timmermans I, Donkers J, et al. Flexible electronic least of some charges: fostering test-induced learning and avoiding test- feedback using the virtues of progress Medical Teacher. 2010;32: induced forgetting. Psychol Sci. 2012;23:1337–1344. 491–495. 5