Human Reproduction Open, pp. 1–15, 2020 doi:10.1093/hropen/hoaa026 ESHRE PAGES ESHRE Clinical Embryologist certification: the first 10 years† Borut Kovacic 1,*, Fernando J. Prados2, Catherine Plas3, Bryan J. Woodward4, Greta Verheyen5, Liliana Ramos6, Sirpa Mäkinen7, Susanna Jamina Apter8, Francesca Vidal9, Søren Ziebe10, M. Cristina Magli11, Kersti Lundin12, Arne Sunde13, and Carlos E. Plancha14 1 Department of Reproductive Medicine, University Medical Centre Maribor, Maribor, Slovenia 2Independent researcher, Madrid, Spain 3 ESHRE Central Office, Grimbergen, Belgium 4X&Y Fertility, Leicester, UK 5University Hospital, Vrije Universiteit Brussel, Centre for Reproductive Medicine, Brussels, Belgium 6Department of Obstetrics and Gynaecology, Radboud University Medical Centre, Nijmegen, The Netherlands 7Ovumia Fertinova, Helsinki, Finland 8Livio Fertilitetscentrum, Stockholm, Sweden 9Unitat de Biologia Cellular, Universitat Autonoma de Barcelona, Barcelona, Spain 10The Fertility Clinic, Section 4071, Rigshospitalet, University of Copenhagen, Copenhagen, Denmark 11SISMER Reproductive Medicine Unit, Bologna, Italy 12Reproductive Medicine, Sahlgrenska University Hospital, Gothenburg, Sweden 13Department of Clinical and Molecular Medicine, Norwegian University of Science and Technology, Trondheim, Norway 14Instituto de Histologia e Biologia do Desenvolvimento, Faculdade de Medicina, Universidade de Lisboa, Portugal and CEMEARE, Lisboa, Portugal *Correspondence address. Department of Reproductive Medicine, University Medical Centre Maribor, Maribor, Slovenia.Tel: þ386 2 321 2192; E-mail: borut.kov@ukc-mb.si https://orcid.org/0000-0003-3911-2408 Submitted on March 16, 2020; editorial decision on April 20, 2020 STUDY QUESTION: What has the ESHRE programme ‘ESHRE Certification for Clinical Embryologists’ achieved after 10 years? SUMMARY ANSWER: The post-exam analysis showed a pass rate of 60% for Clinical and 50% for Senior Clinical Embryologists and a high level of internal consistency of all exams, leading to a total of 773 certified Clinical and 493 Senior Clinical Embryologists over the decade. WHAT IS KNOWN ALREADY: In an ESHRE survey on the educational and professional status of Clinical Embryology in Europe, it was found that education of laboratory personnel working in the field of assisted reproduction is highly variable between countries. In 2008, ESHRE introduced a programme, curriculum and certification in the field of Clinical Embryology. Knowledge gained by postgraduate study of recommended literature, following a clear curriculum, is verified by a written two-level exam for obtaining a certificate for Clinical (basic) or Senior Clinical (advanced) Embryologists. With a total of 1266 certificates awarded over a period of 10 years and recognition by the Union Européenne des Médecins Spécialistes and their Council for European Specialists Medical Assessment, the ESHRE Clinical Embryology exams have become an internationally recognized educational standard in the field of Clinical Embryology. STUDY DESIGN, SIZE, DURATION: A retrospective analysis of all applications for ESHRE Clinical (2009–2018) and Senior Clinical Embryologist Certification (2008–2018) and exam results of the first decade was carried out by the Steering Committee for Clinical Embryologist Certification. PARTICIPANTS/MATERIALS, SETTING, METHODS: A total of 2894 applications for ESHRE Certification for Clinical Embryologists and the results of 10 exams for the Clinical (1478 candidates) and 11 exams for Senior Clinical (987 candidates) levels were analysed. A detailed post-exam retrospective analysis was performed regarding difficulty, discrimination and reliability levels of 1600 multiple-choice questions (MCQs) with a single best answer among four options, from eight different curriculum topics (Basic cell biology, Genetics, Developmental biology, Female reproduction, Male reproduction, IVF laboratory, Cryopreservation and Laboratory manage- ment), representing the core theoretical knowledge of Clinical Embryology. Difficulty levels of the MCQs were subsequently compared re- garding each topic and each yearly exam. The participation and success rates in the ESHRE Clinical Embryology exams were also assessed in terms of the educational and geographic backgrounds of candidates. MAIN RESULTS AND THE ROLE OF CHANCE: Over the 10 years studied, the mean pass rate for the Clinical Embryologist exam was 60% (range 41–86%), and for the Senior Clinical Embryologist exam was 50% (range 34–81%). On average, 63% European candidates † ESHRE pages content is not externally peer reviewed. The manuscript has been approved by the Executive Committee of ESHRE. C The Author(s) 2020. Published by Oxford University Press on behalf of European Society of Human Reproduction and Embryology. V This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/), which per- mits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact journals.permissions@oup.com 2 Kovacic et al. and 35% non-European candidates passed the Clinical Embryologist exam, while 52% European candidates and 31% non-European candi- dates passed the Senior Clinical Embryologist exam. The candidates’ educational level impacted on the success of the Clinical Embryologist exam but not of the Senior Clinical Embryologist exam. The mean difficulty indices by study topic showed that in the period of 10 years, there were no statistically significant differences between topics, for either the Clinical or Senior Clinical Embryologist exams. However, the overall exam difficulty varied between years. Reassuringly, the exam MCQ discrimination and reliability indices always showed a high level of internal consistency in all exams. LIMITATIONS, REASONS FOR CAUTION: Some data from the initial ESHRE certification programme were not obtained electroni- cally, in particular data for education, implying tables and figures reflect the specified valid data periods. Several countries exhibit different study profiles for those working in ART laboratories, such that laboratory technicians/technologists predominate in some countries, while in others only biologists and medical doctors are allowed to work with human embryos. Such differences could consequently affect the exam performance of candidates from specific countries. WIDER IMPLICATIONS OF THE FINDINGS: The ESHRE exams on Clinical Embryology are the most widely, internationally accepted tests of knowledge in the rapidly growing area of human reproduction. Clinical Embryology is increasingly recognized as a specific discipline for scientific staff who are collaborating closely with clinicians in managing human infertility through medically assisted reproduction. The analysis of the first 10 years of application of a two-level exam for Clinical Embryology shows a consistent high quality and reliability of the exam and MCQs used. These results represent an important follow-up of the quality of the ESHRE Certification programme for Clinical Embryologists, and convincingly position Clinical Embryology in the wider group of health disciplines that are harmonized through profes- sional bodies such as ESHRE and European Board & College of Obstetrics and Gynaecology. The exams provide a clear step towards the increasing professional recognition and establishment of Clinical Embryology within health systems at both European and international level. STUDY FUNDING/COMPETING INTEREST(S): No competing interest. All costs of the Steering Committee meetings were covered by ESHRE. Key words: clinical embryology / education / embryologist certification / exam / post-exam item analysis / ESHRE .. Introduction .. Trávnı́k et al., 2013); however, in most countries, no formal recogni- .. tion of this profession exists. In order to bring further clarification and From the late 1970s/early 1980s, professionals experienced in gam- .. .. harmonization, ESHRE decided to establish a European certification/ ete/embryo culture or cell culture systems co-operated with clinicians .. qualification programme with a comprehensive scientific curriculum. In to set up the first clinical ART laboratories. These pioneers included .. .. fact, the internationally recognized technical and scientific competences scientists from basic biological or biomedical fields and veterinarians .. of ESHRE positioned the society to develop such a system, aimed at with practical experience in animal reproduction, as well as individuals .. .. providing wide and formal recognition of the status of Clinical with several other backgrounds. Reproductive medicine has since ex- .. Embryologists. In 2006, it was decided to create an ESHRE Task Force perienced an exponential development and growth, particularly due to .. .. for the Certification of Clinical Embryologists. improvements in the laboratory aspects of medically assisted repro- .. duction (MAR). .. The Certification Task Force established contact with several profes- .. sional and scientific bodies in order to seek input in the curriculum It soon became evident that the quality of laboratory work was criti- .. cal to the success of MAR treatment. The need for specific knowledge .. topics. ESHRE consulted Alpha Scientists in Reproductive Medicine .. (ALPHA), Asociación para el Estudio de la Biologı́a de la Reproducción and skills for manipulating human gametes and embryos, performing .. high-technology procedures and consistently ensuring high quality cul- .. (ASEBIR, Spain), Arbeitsgemeinschaft Reproduktionsbiologie des .. Menschen (AGRBM, Germany), Association of Clinical Embryologists ture conditions, led to the establishment of a new discipline in labora- .. .. (ACE, UK) and Nordic IVF Laboratory Society (NILS), Vereniging voor tory medicine: Clinical Embryology. .. The need for wider recognition of the field of Reproductive .. Klinisch Embryologen (KLEM, The Netherlands), Vlaamse Vereniging .. voor Klinische Embryologie (VVKE, Belgium), Association Belge des Medicine encouraged ESHRE together with the European Board & .. College of Obstetrics and Gynaecology (EBCOG) to develop an edu- .. Embryologistes Francophones (ABEF, Belgium), Panhellenic Association .. of Clinical Embryologists (PEKE, Greece), Embryologenforum Austria cational system enabling medical clinicians to be granted the title of ... (EFA) and key embryologists from Portugal. This consultation resulted subspecialists in Reproductive Medicine (Calhaz-Jorge et al., 2015). For .. Clinical Embryologists working in MAR centres; however, the situation .. in the establishment of the ESHRE Embryology Certification Committee .. (EmCC) who developed a two-level self-educational certification pro- was more complex, particularly given the high heterogeneity of aca- .. demic levels (BSc, MSc or PhD) and scientific backgrounds of these .. gramme for Clinical Embryologists (basic level) and Senior Clinical .. professionals working in healthcare (Alpha Scientists in Reproductive .. Embryologists (advanced level). Medicine, 2015; Kovacic et al., 2015). .. The proposed application system included a background curriculum, .. Professional bodies in some European countries have provided na- .. evidence of academic levels, a logbook showing a minimum number of tional recommendations regarding qualifications for Clinical .. performed key procedures in the ART laboratory, and a formal exam .. Embryologists (Hughes and Association of clinical embryologists, 2012; . to test the candidates’ theoretical knowledge. The major objective Clinical embryologists: 10 years of ESHRE certification 3 .. was to test the Clinical Embryologists’ understanding of the fundamen- .. Multiple-choice questions (MCQ) of the single best answer (SBA) tal biology underpinning the technical and clinical aspects of MAR. .. type were developed for the exam, reviewed and approved by all .. The ESHRE Certification for Clinical Embryologists programme .. EmCC members, and requiring one correct answer to be selected started in 2008 with a fast-track certification for Senior Clinical ... among the four answer options. All MCQs were divided into three Embryologists having served for at least 10 years, being head of an .. groups according to the estimated level of knowledge required: specific .. ART laboratory and holding a PhD in a relevant discipline. Following .. for Clinical Embryologists, specific for Senior Clinical Embryologists and this, the first ESHRE exam was held at the ESHRE Annual Meeting in .. common to for both (Clinical and Senior Clinical Embryologists). The .. Barcelona 2008, where only candidates for the Senior Clinical .. MCQs have been developed and updated over time, based on evolv- Embryologist exam were allowed to participate. From 2009 onwards, .. ing knowledge and post-exam performance analysis. .. the system was also open to Clinical Embryologists. .. All questions were secured electronically, each of them containing In the period 2008–2018, the certification programme underwent .. information about its topic, predicted difficulty and unique serial num- .. several improvements by introducing electronic exam evaluation .. ber. Once used, each question was linked also with data about the (2011), external validation of the academic diploma (2012) and an on- .. year of use, percentages of chosen answers and difficulty index (see .. line application platform (2015). Certification was also opened for .. below) based on post-exam analysis. .. non-European embryologists in 2012. Furthermore, in 2015, ESHRE in- .. troduced an online system for monitoring Continuing Professional .. Curriculum and logbook .. Development (CPD) for all certified embryologists. In a pilot project, .. The Clinical Embryologist exam was set for laboratory staff who ful- one long-distance on-site exam for a small group of 22 candidates was .. .. filled the following requirements at the date of the application dead- organized in 2018 in India, simultaneously with the main exam that .. line: at least a BSc degree in natural/life sciences, at least 3 years was held in Geneva. In 2016, ESHRE applied for validation of the en- .. .. hands-on experience with human gametes and embryos in an ART tire certification process by the Union Européenne des Médecins .. laboratory, and a minimum of 50 hands-on procedures per laboratory Spécialistes (UEMS) and their Council for European Specialists Medical .. .. topic, as specified below. The Senior Clinical Embryologist exam was Assessment (CESMA). As a result, UEMS-CESMA monitored the .. set for candidates with either a MSc or PhD, at least 6 years hands-on ESHRE Certification for Clinical Embryologists exams in Geneva in .. .. experience in a ART laboratory, and a minimum of 50 hands-on pro- 2017 and the ESHRE exams for embryologists became officially recog- .. cedures per laboratory topic, as specified below. nized in accordance with the UEMS standards. .. .. Applicants were informed about the curriculum, requirements, ap- The current report presents a detailed analysis of the first decade of .. plication procedure and type of exam via dedicated webpages added certification of Clinical Embryologists including the number of candi- .. .. to the ESHRE website. Applications were open for 2 months from dates, their educational background and the pass rate for the Clinical .. mid-October until mid-December. From 2015 onward, an online ap- .. or Senior Clinical Embryologist level exams. Data regarding the diffi- .. plication platform was developed and the applicants’ highest university culty, discrimination and reliability levels of exams and questions from .. .. degree was verified by an independent European organization: the the various curriculum topics of Clinical Embryology are also provided. .. Netherlands Universities Foundation For International Cooperation In addition, the variation in participation and success rate between dif- .. .. (NUFFIC). The validation process was co-ordinated via ESHRE. ferent educational and geographic backgrounds is presented. This arti- .. Diplomas that did not fit the criteria of an official national university cle aims to pave the way for further improvements in the ESHRE .. .. degree were not accepted. Certification for Clinical Embryologists programme. .. A logbook was required to provide a record of the number of per- .. .. sonally performed procedures in nine laboratory tasks: oocyte re- .. trieval, semen analysis, semen preparation, IVF insemination, ICSI, .. Material and methods .. zygote and embryo evaluation, embryo transfer, cryopreservation of .. oocytes/embryos, thawing/warming of oocytes/embryos. A minimum .. Steering committee and exam questions .. number of 50 cases per topic was required in a period of 3 years for .. Clinical Embryologist applicants or 6 years for Senior Clinical A team of experts selected by ESHRE formed the EmCC, with re- .. sponsibility for the self-educational Clinical Embryology programme. .. Embryologist applicants (see ESHRE link for further specifications). .. The information from the application and logbook(s) had to be veri- The EmCC presented a detailed curriculum, recommended appropri- .. .. fied and signed by the supervisor(s) from the respective MAR centres. ate teaching materials, prepared exam questions, regularly opened a .. Two references were required to support the application, preferably call for exam participation, verified the candidates’ eligibility, organized .. .. being ESHRE members and/or ESHRE Certified Senior Clinical exams, evaluated the exam results and addressed complaints. The .. Embryologists. If the candidate had been working at several clinics, EmCC comprised eight members, representing different areas of .. .. separate signed logbooks were required. Europe and speciality topics, and one ESHRE administrative officer. .. The basis of the self-educational programme was a curriculum that .. .. Exam and marking identified eight main topics from Clinical Embryology: Basic cell biology, .. Genetics, Developmental biology, Female reproduction, Male reproduc- .. The exams were held at every ESHRE Annual Meeting. Each exam .. tion, IVF laboratory, Cryopreservation and Laboratory management. .. comprised 100 MCQs of the SBA type. Candidates received 50 level- Appropriate updated/revised literature was recommended to candi- .. specific questions and 50 questions common to both levels. The official .. dates preparing for the exam. Exam questions were based on the rec- .. exam language was English; however, for the first three exams only, ommended literature and good laboratory and clinical practice. .. the exam was also provided in the languages of the host country of 4 Kovacic et al. .. the ESHRE Annual Meeting (Spanish, Dutch and Italian). The exam .. the field, the difficulty index therefore results from the proportion of time was 2 h and 15 min. The applicants were notified in advance of .. correctly answered MCQs. It therefore reflects the ‘easiness’ of the .. the exam that they needed to pass a threshold of 66% of the answers .. MCQ: an MCQ with a higher difficulty index value means an easier correctly to pass the exam (see Discussion section). ... question. Ideally, MCQs should have p-values (difficulty indices) that Marking was performed electronically and verified manually by .. range between 0.2 (20% correct answers) and 0.8 (80% correct .. ESHRE officers to ensure accuracy. The marking software provided in- .. answers) (Dixon, 1994). One-way ANOVA allowed for comparison formation on candidate performance for each question. The EmCC .. of the mean difficulty indices of different topics and between exams. .. performed a post-exam analysis (see below). Questions with a high .. Questions from each curriculum topic were categorized into 10 proportion of incorrect answers were reconsidered and, if appropri- .. groups according to difficulty, from very easy (level 1, difficulty index .. ate, accepted with more than one correct answer, or removed. A final .. range 0.9–1) to very difficult (level 10, difficulty index range 0–0.1). In check was always made prior to informing candidates of their .. detail, the difficulty groups were: 1 (0.9–1), 2 (0.81–0.9), 3 (0.71–0.8), .. performance. .. 4 (0.61–0.7), 5 (0.51–0.6), 6 (0.41–0.5), 7 (0.31–0.4), 8 (0.21–0.3), 9 .. (0.11–0.2) and 10 (0–0.1). .. Study design and outcome measurements .. The post-exam discrimination analysis was computed from equal- .. sized upper and lower scoring candidates (27% highest and lowest All applications and exam results for ESHRE Clinical (2009–2018) and .. .. scoring candidates) according to Kelley (1939). Values above 0.2 were Senior Clinical Embryologist certification (2008–2018) were analysed .. retrospectively. The demographic data of candidates were collected .. regarded as optimal (Ebel, 1954). .. KR-20 is the variance of the total scores of all the candidates taking from their application files. From 2015 onwards, the application plat- .. form was updated to collect more detailed information on study fields .. the test. KR-20 values range from 0 to 1. A high KR-20 indicated reli- .. able (consistent) student scores, with a KR-20 score of 0.60 or higher and academic degrees. The online application system simplified confir- .. mation of the eligibility of applicants. A maximum of 300 candidates .. being desirable. .. To compare pass rates between candidates of different educational was permitted to apply per exam. .. Information about exam performance was taken from the EmCC .. background, the v2 test was used. A P-value of <0.05 was considered .. as statistically significant. Difference between countries in number of database, in which relevant data (number of applications, valid applica- .. tions, pass rate, mean score, median score and mean difficulty level) .. failed and successful exam attempts are presented graphically. .. were collected annually for quality control. .. The aim of this retrospective study was to present applicants’ edu- .. .. cational background and the pass rates according to participant group .. Results (new applicants vs resits, Europeans vs non-Europeans and academic .. .. degrees). From post-exam analysis, the difficulty, discrimination and re- .. Exam analysis: applications, pass rates and .. liability indices were calculated. Answer options that should not be .. mean scores marked by the candidate are defined as distractors. A distractor analy- .. .. The data for Fig. 1 were obtained from the online application platform sis was also performed (see below). Additional outcome measure .. for the last three exams (2016–2018). Prior to 2016, the educational parameters included difficulty levels of specific topics and number of .. .. background of candidates was verified from the submitted documenta- successful/failed exam attempts according to the registered country of .. tion, but this information was not analysed. The applicants from both candidates. All study outcomes are presented separately for Clinical .. .. exam levels had university degrees from different life science study and Senior Clinical Embryologist exams. .. fields (Fig. 1). The majority studied biology (43.5%), biotechnology .. .. (15.1%) or biomedical sciences (10.1%), accepting that different coun- Statistical analysis .. tries may use different nomenclature for the topic of a study field. The .. For exams from 2009 to 2010, the evaluation and the analysis were .. representation of study fields did not differ between applicants for performed manually. For exams from 2011 onwards, the analysis was .. Clinical and Senior Clinical Embryologist exams, with the exception of .. performed electronically using the relevant computer software .. Laboratory Technology, which was more frequently represented in the .. Clinical Embryologist exam (6.1% vs 1.7%, respectively P ¼ 0.005). (Remark Office OMR 2014, Gravic, Inc. Malvern, PA, USA). The IBM .. SPSS Statistics (version 22) software package was used for v2 test and .. Data regarding applications for the Clinical Embryologist exam are .. analysis of variance (ANOVA). Post-exam analysis of 1600 MCQs .. presented in Table I. In the period from 2009 to 2018, a total of 1611 from both types of exams was based on ‘item analysis’ indices. This .. applications were received, of which 92% (1478 candidates) met the .. provided a post-exam p-statistic (difficulty index) (Downing et al., .. application criteria. Altogether, 1289 candidates attended the exam, 1995), a generalized upper-lower item discrimination index .. with 89% participants from European and 11% from non-European .. (Richardson and Stalnaker, 1933; Ebel, 1954) for each question, and .. countries. The mean pass rate for the Clinical Embryologist exam was general reliability of assessment (also known as internal consistency re- .. 60%, ranging from 41% to 86%. On average, 80% of the candidates .. liability) presented as Kuder–Richardson (KR-20) parameter (Kuder .. attended the exam for the first time and reached the mean pass rate and Richardson, 1937) (see below). .. of 63%. Twenty percent of applicants were resits, with the lower .. The p-statistic was a proxy for the difficulty of each question. Each .. mean pass rate of 49%. On average, 63% European candidates and p-value was calculated as the percentage of candidates providing a cor- .. 35% non-European candidates passed the Clinical Embryologist exam. .. rect response to a question. The percentage was then divided by 100 .. The mean score achieved by all participants in the Clinical Embryology and converted to a ‘difficulty index’. Using the classical terminology in .. exams was 68/100 and by those who passed exams was 76/100. Clinical embryologists: 10 years of ESHRE certification 5 .. .. and for the Senior Clinical Embryologist exam came from Spain .. (n ¼ 251), Italy (n ¼ 96), Poland (n ¼ 69), Germany (n ¼ 65) and .. .. Greece (n ¼ 54). The success rate per country did not correlate with ... the number of attempts. .. .. .. Question analysis .. .. From 2011 to 2018, the number of MCQs with optimal difficulty index .. (range from 0.2 to 0.8) increased from around 60% in the Clinical .. .. Embryologist exam and 70% in the Senior Clinical Embryologist exam .. .. to around 80% in both (Table III). The overall mean difficulty indices .. of Clinical Embryologist and Senior Clinical Embryologist exams were .. .. 0.68 § 0.19 (range from 0.63 to 0.74 between years) and 0.64 § 0.21 .. (range from 0.59 to 0.7), respectively (Table III). .. .. The specific questions for Clinical Embryologists and Senior Clinical .. Embryologists showed mean difficulty indices of 0.71 and 0.56, respec- .. .. tively. The common questions used for both the Clinical Embryologist .. and Senior Clinical Embryologist exams demonstrated mean difficulty .. Figure 1 Frequency distribution (%) of examinees’ basic .. indices of 0.62 and 0.65, respectively (data not shown in tables). study fields, 2016–2018. Clinical Embryology* (University Master; .. Supplementary Table SI presents the mean difficulty indices of both .. not titulo propio—a degree, offered by a single university, which is .. types of exams, per years and per specific topic. The analysis of the not officially recognized by the national educational system), other** .. mean difficulty indices per study topic showed that in the period of 10 .. (Bioanalytical Science, Bioengineering, Environmental Science, .. years, there were no statistically significant differences between topics, Forensic Science, Human Biology, Medical Informatics, Molecular .. Biology and Genetics, Physiology, Zoology, etc.). .. for either the Clinical or Senior Clinical Embryologist exams. However, .. the overall exam difficulty varied between years (P < 0.001). .. .. Most MCQs from Clinical Embryologist exams had difficulty levels .. between 2 and 4, while most Senior Clinical Embryologist exam ques- .. .. tions were between 2 and 5 (Fig. 4a and b). The second part of Table I presents the data of the Senior Clinical .. The majority of MCQs from the Clinical Embryologist exams (70%) Embryologist exams. In total, 1283 candidates applied, 90% (1158 .. .. and the Senior Clinical Embryologist exams (68%) demonstrated a suf- applicants) were accepted and finally 987 applicants attended the .. ficiently high discrimination index (0.2). The mean discrimination indi- exam. The mean pass rate was 50%, ranging from 34% to 81%. On .. .. ces were 0.27 § 0.12 (Clinical Embryologist exam) and 0.26 § 0.14 average, 77% were first-attempt candidates and 23% were resit candi- .. (Senior Clinical Embryologist exam). The annual discrimination index dates (50% and 39% mean pass rate, respectively). Altogether, there .. .. never dropped below 0.2 (Table III). were 89% European and 11% non-European participants (52% and .. The relation between the difficulty and discrimination indices is .. 31% mean pass rate, respectively). The mean score achieved in Senior .. shown in scatterplots for 1600 MCQs from both exams (Fig. 5a and Clinical Embryology exams was 59/100 and in those who passed .. .. b). The blue area indicates the optimal discrimination and difficulty exams it was 73/100. .. ranges. MCQs in the pink area are very difficult with low discrimina- The frequency of scores achieved in all Clinical Embryology and .. .. tion, whilst MCQs in the green area are very easy with low discrimina- Senior Clinical Embryology exams is presented graphically (Fig. 2a and .. tion. Most MCQs are in the blue area. In the Senior Clinical b). .. .. Embryologist exams, many MCQs showed low discrimination, which is The candidates’ educational level impacted on the success of the .. characteristic of making a random choice by the candidates in choosing Clinical Embryologist exam (Table II). Candidates with a PhD obtained .. .. the right answer. MCQs in the red area or having a negative discrimi- a higher pass rate (67%) than candidates with a MSc (55%) (P ¼ 0.022) .. nation index were set aside for future improvement. Only 1.8% of .. or BSc (48%) (P < 0.001). However, for the Senior Clinical .. MCQs in the Clinical Embryologist exams and 3.8% in the Senior Embryologist exam, there was no difference in pass rates among candi- .. Clinical Embryologist exams had negative discrimination indices. .. dates with a PhD or MSc (42% and 48%, P ¼ 0.14). In line with the .. There were 4800 answer options that should not have been chosen expected higher difficulty of the Senior Clinical versus the Clinical .. by the candidate from 1600 MCQs (Tables III). These answer options .. Embryology exams, the detected difference in pass rates between PhD .. are termed ‘distractors’. If a MCQ is good quality, then all distractors groups (67% vs 42%, P < 0.001) was expected. However, candidates .. .. should be selected by some candidates. If a MCQ is poorer quality, with an MSc degree obtained similar pass rates in both types of exams .. then one or more of the distractors should not be selected by all can- .. (55% vs 48%). .. didates. In such instances, poorer quality MCQs are less discriminative, The number of attempts (tries by the candidates, that can be suc- .. as the answer options are ‘low-quality’ distractors, also known as .. cessful or not) according to country, the exam level and performance .. ‘non-distractors’. Thus, the quality of each distractor can modulate the is shown in Fig. 3a and b. The highest number of attempts for the .. difficulty of an MCQ. In both levels of exam, a trend for an increasing .. Clinical Embryologist exam came from Poland (n ¼ 131), Italy .. number of better quality distractors was observed from 2011 to 2018. (n ¼ 120), Spain (n ¼ 100), Denmark (n ¼ 99) and Sweden (n ¼ 81); .. Consequently, the number of MCQs with ‘non-distractors’ has 6 Kovacic et al. Table I Results of ESHRE exams. Exams for Clinical Embryologists Where exam held Amsterdam Rome Stockholm Istanbul London Munich Lisbon Helsinki Geneva Barcelona Total 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 ............................................................................................................................................................................................................................ Applications Applications received 115 161 156 135 143 171 184 149 176 221 1611 Valid applications (%) 99 (86) 143 (89) 141 (90) 122 (90) 134 (94) 160 (94) 167 (91) 137 (92) 168 (95) 207 (94) 1478 (92) Participants All participants 88 127 121 107 120 135 149 121 142 179 1289 New applicants (%) 88 (100) 116 (91) 95 (79) 88 (82) 79 (66) 103 (76) 96 (64) 99 (82) 117 (82) 149 (83) 1030 (80) Resits (%) 11 (9) 26 (21) 19 (18) 41 (34) 32 (24) 53 (36) 22 (18) 25 (18) 30 (17) 259 (20) European (%) 88 (100) 127 (100) 121 (100) 107 (100) 120 (100) 123 (91) 120 (81) 102 (84) 108 (76) 130 (73) 1146 (89) Non-European (%) 12 (9) 29 (19) 19 (16) 34 (24) 49 (27) 143 (11) Pass rate All participants (%) 76 (86) 97 (76) 87 (72) 55 (51) 77 (64) 65 (48) 108 (72) 50 (41) 70 (49) 88 (49) 773 (60) New applicants (%) 76 (86) 87 (75) 71 (75) 46 (52) 51 (65) 57 (55) 69 (72) 49(49) 65 (46) 76 (51) 647 (63) Resits (%) 10 (91) 16 (62) 9 (47) 26 (63) 8 (25) 39 (74) 1 (5) 5 (20) 12 (40) 126 (49) European (%) 76 (86) 97 (76) 87 (72) 55 (51) 77 (64) 62 (50) 94 (78) 43 (42) 60 (56) 72 (55) 723 (63) Non-European (%) 3 (25) 14 (48) 7 (37) 10 (29) 16 (33) 50 (35) Scores Mean score (%) 76 73 71 66 69 65 72 63 65 65 68 Mean score of 79 77 77 73 76 74 78 75 75 75 76 participants passed (%) Exams for Senior Clinical Embryologists Where exam held Barcelona Amsterdam Rome Stockholm Istanbul London Munich Lisbon Helsinki Geneva Barcelona Total 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 ............................................................................................................................................................................................................................ Applications Applications received 170 140 101 72 91 120 110 116 102 111 150 1283 Valid applications (%) 152 (89) 115 (82) 89 (88) 62 (86) 81 (89) 111 (92) 107 (97) 101 (87) 93 (91) 105 (95) 142 (95) 1158 (90) Participants All participants 145 100 74 51 66 94 91 80 78 85 123 987 New applicants (%) 145 (100) 89 (89) 49 (66) 30 (59) 44 (67) 53 (56) 65 (71) 48 (60) 63 (81) 72 (85) 103 (84) 761 (77) Resits (%) 11 (11) 25 (34) 21 (41) 22 (33) 41 (44) 26 (29) 32 (40) 15 (19) 13 (15) 20 (16) 226 (23) European (%) 145 (100) 100 (100) 74 (100) 51 (100) 61 (92) 75 (80) 75 (82) 73 (91) 62 (79) 91 (76) 101 (82) 882 (89) Non-European (%) 5 (8) 19 (20) 16 (18) 7 (9) 16 (21) 20 (24) 22 (18) 105 (11) Pass rate All participants (%) 118 (81) 40 (40) 37 (50) 19 (37) 26 (39) 43 (46) 31 (34) 58 (73) 28 (36) 44 (52) 49 (40) 493 (50) New applicants (%) 118 (81) 38 (43) 29 (59) 12 (40) 24 (55) 22 (42) 19 (29) 17 (77) 27 (43) 35 (41) 43 (35) 384 (50) Resits (%) 2 (18) 8 (32) 7 (33) 2 (9) 21 (51) 12 (46) 21 (66) 1 (7) 9 (69) 6 (30) 89 (39) European (%) 118 (81) 40 (40) 37 (50) 19 (37) 24 (39) 36 (48) 26 (35) 53 (73) 24 (39) 35 (38) 48 (48) 460 (52) Non-European (%) 2 (40) 7 (37) 5 (31) 5 (71) 4 (25) 9 (45) 1 (5) 33 (31) Scores Mean score (%) 72 63 64 62 59 64 61 70 62 64 63 59 Mean score of 74 74 72 72 70 75 73 76 72 73 74 73 participants passed (%) Clinical embryologists: 10 years of ESHRE certification 7 Figure 2 Frequency of scores achieved in exams: a pass threshold of 66% was defined since the first exam. (a) Clinical embryologist exams (2009–2018). (b) Senior clinical embryologist exams (2008–2018). .. broader developmental biology. Specializations in laboratory techni- Table II The impact of candidate’s academic degree on .. .. ques and skills required in human reproductive embryology now rep- success of exam for Clinical and Senior Clinical .. Embryologists (data from 2013 to 2018). .. resent a new clinical laboratory discipline and, as such, have developed .. as a post-graduate educational competence. In 2013, most of Europe’s Clinical Embryologistsa Senior Clinical Embryologistsb .. ... 1700 ART laboratories were led by clinically oriented scientists, named Participants, Passed, Participants, Passed, .. ‘Clinical Embryologists’, who worked closely with gynaecologists to .. provide MAR (Kovacic et al., 2015). In 2016, ESHRE presented the N N (%) N N (%) .. ...................................................................................................... .. status of Clinical (Reproductive) Embryology and Clinical Embryologists BSc 302 146 (48.3) .. in European countries to the UEMS, who recognized the ESHRE certi- .. MScc 437 239 (54.7) 379 184 (48.5) .. fication in Clinical Embryology and appraised the Clinical Embryologist PhDd 107 72 (67.3) 168 70 (41.7) .. and the Senior Clinical Embryologist exams. .. Total 846 457 (54) 547 254 (46.4) .. .. The v2 test: .. Participants’ profile a .. Significant difference in pass rates between MSc and PhD (P ¼ 0.022) and between .. The ESHRE certification programme started 30 years after the birth of BSc and PhD (P < 0.001) in clinical embryologist group. .. the world’s first IVF baby and had to adapt to the existing highly het- b No difference in pass rates between MSc and PhD degrees in Senior Clinical .. Embryologist group. .. erogeneous staffing situation in ART laboratories, which has developed c No difference in pass rates between MSc degrees of Clinical and Senior Clinical .. over decades. Staff composition, level of education, experience, etc. Embryologist exams. .. d Significant difference in pass rates between PhD degrees in Clinical and Senior .. often depend on the size of MAR centres. Only in a few European .. Clinical Embryologist exams (P < 0.001). .. countries are the educational standards and competences in ART lab- .. oratories defined in quality guidelines, national recommendations and/ .. .. or legislations (Lamb, 2005; Practice Committees of ASRM et al., .. 2014; Alpha Scientists in Reproductive Medicine, 2015; Go, 2015; decreased. The questions in which distractors were chosen more of- .. ten than the correct answer were considered as more difficult ques- .. Kovacic et al., 2015; ESHRE Guideline Group on Good Practice in IVF .. Labs, 2016). Scientists from the natural/life sciences remain the preva- tions. The Clinical Embryologist exams included 6.9% and the Senior .. Clinical Embryologist exams 11.7% of such questions (data not .. lent staff in ART laboratories and up to 2008 no specific certification .. programme for Clinical Embryologists existed across Europe. One shown). .. The reliability index KR20 was always higher than 0.80 in all exams, .. clear advantage of the ESHRE certification programme for Clinical .. showing a high level of internal consistency and that the exam pro- .. Embryologists is that the exams are open to all candidates with a back- duced similar results over subsequent years. .. ground in natural/life sciences and with relevant laboratory expertise .. .. and postgraduate practical experience in the field of MAR. From the .. collected data about exam participants’ education, biologists strongly .. .. predominate, followed by biotechnologists and biomedical scientists. Discussion .. The educational background of personnel in ART laboratories differs .. Clinical Embryology has rapidly developed in the last three decades as .. between European countries. Laboratory technicians/technologists a new discipline differentiated from medical embryology or the .. predominate in some countries, while in others only biologists and 8 Kovacic et al. (a) Failed aempts Passed aempts -110 -100 -90 -80 -70 -60 -50 -40 -30 -20 -10 0 0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 Spain Spain Italy Italy PolandPoland Sweden Sweden DenmarkDenmark Non Non Europe Europe Portugal Portugal GermanyGermany Norway Norway Russia Russia Belgium Belgium GreeceGreece Czech Republic Czech Republic BulgariaBulgaria RomaniaRomania Serbia Serbia AustriaAustria Switzerland Switzerland IrelandIreland United United Kingdom Kingdom Slovenia Slovenia FinlandFinland North Macedonia Macedonia Latvia Latvia TurkeyTurkey Netherlands Netherlands HungaryHungary Ukraine Ukraine Montenegro Montenegro FranceFrance CyprusCyprus AlbaniaAlbania SlovakiaSlovakia BelarusBelarus Bosniaand Bosnia andHerzegovina Herzegovina CroaaCroaa Lithuania Lithuania Kazakhstan Kazakhstan IcelandIceland Azerbaijan Azerbaijan Georgia Georgia EstoniaEstonia KosovoKosovo (b) Failed aempts Passed aempts Spain Italy Non Europe Greece Germany Poland Czech Republic Belgium United Kingdom Switzerland Portugal Romania Russia Ireland Turkey Austria Sweden Finland Bulgaria Norway France Serbia Denmark Ukraine Slovakia Slovenia Hungary Cyprus Montenegro North Macedonia Macedonia Luxembourg Albania Latvia Netherlands Lithuania Kosovo Estonia Figure 3 Number of failed and pass attempts of exams by country. (a) Clinical Embryologists (2009–2018). (b) Senior Clinical Embryologists (2008–2018). medical doctors are allowed to work with human embryos (Kovacic .. candidates with higher degrees (MSc and PhD) passed the exam at .. et al., 2015). Such differences could consequently influence the exam .. higher rates than those with a BSc only. performance of candidates from specific countries, since the analysis of .. The ESHRE certification exams are typical criterion-referenced tests, .. the performance per country and educational level shows that .. where the main goal is for candidates to demonstrate a level of Clinical embryologists: 10 years of ESHRE certification 9 Table III Post-exam analysis (2011–2018). Exams for Clinical Embryologists Year 2011 2012 2013 2014 2015 2016 2017 2018 Total ............................................................................................................................................................................................................................ Difficulty Mean difficulty index (p-statistic) 0.72 0.66 0.69 0.65 0.71 0.63 0.65 0.65 0.67 §SD 0.19 0.20 0.18 0.20 0.16 0.20 0.18 0.17 0.19 Median difficulty index 0.75 0.68 0.71 0.67 0.72 0.62 0.68 0.66 0.68 MCQs with optimal difficulty index (%) 60 71 67 73 72 81 79 82 73 Discrimination Mean discrimination index 0.27 0.23 0.27 0.25 0.29 0.28 0.28 0.28 0.27 §SD 0.12 0.11 0.11 0.12 0.15 0.13 0.12 0.12 0.12 Median discrimination index 0.28 0.23 0.27 0.25 0.28 0.31 0.29 0.28 0.27 MCQs with optimal discrimination index (%) 74 63 75 71 73 66 67 71 70 Distractor analysis Optimal distractors (%) 57 55 65 62 63 81 73 70 65 MCQs containing non-distractors (%) 23 26 18 16 10 4 7 8 14 Internal consistency Reliability index (KR-20) 0.87 0.82 0.86 0.85 0.89 0.87 0.87 0.88 0.86 Exams for Senior Clinical Embryologists Year 2011 2012 2013 2014 2015 2016 2017 2018 Total ............................................................................................................................................................................................................................ Difficulty Mean difficulty index (p-statistic) 0.62 0.59 0.64 0.61 0.70 0.62 0.64 0.63 0.63 §SD 0.23 0.21 0.18 0.21 0.17 0.22 0.18 0.17 0.20 Median difficulty index 0.65 0.61 0.65 0.64 0.74 0.61 0.65 0.63 0.65 MCQs with optimal difficulty index (%) 70 81 77 79 67 72 84 84 77 Discrimination Mean discrimination index 0.25 0.27 0.27 0.26 0.27 0.25 0.29 0.24 0.26 §SD 0.15 0.14 0.13 0.12 0.14 0.15 0.15 0.12 0.14 Median discrimination index 0.27 0.29 0.27 0.26 0.28 0.26 0.28 0.26 0.27 MCQs with optimal discrimination index (%) 65 72 70 69 73 63 69 62 68 Distractor analysis Optimal distractors (%) 66 53 67 66 62 75 72 70 66 MCQs containing non-distractors (%) 27 16 12 20 22 8 13 11 16 Internal consistency Reliability index (KR-20) 0.84 0.87 0.87 0.85 0.87 0.82 0.88 0.84 0.85 Optimal difficulty index ranges from 0.2 to 0.8; optimal discrimination index 0.2; optimal reliability index 0.6; optimal distractors: answers chosen by 5% of examinees; non-dis- tractors: not chosen answers. KR-20—Kuder–Richardson 20 is a reliability index, a special case of Cronbach’s A. practical experience and theoretical knowledge. The purpose of intro- .. years, which is similar to many medical specialities (ESHRE Guideline .. ducing two-level exams was to ensure that also holders of a BSc, who .. Group on Good Practice in IVF Labs 2016). acquired the skills to perform ART laboratory techniques through ... During the initial period of the ESHRE certification, concern was work experience, could access the Clinical Embryologist certificate. .. raised that the certificate only assesses theoretical knowledge and .. The field of Clinical Embryology remains poorly represented in the uni- .. does not demonstrate the candidate’s practical skills. Although this last .. versity curricula (Hamilton and Carachi, 2014). To gain sufficient com- .. point is highly relevant, formal practical training programmes for petence in practical tasks in an ART laboratory takes 3–4 years .. Clinical Embryology are largely non-existent at national level, with .. (Hughes and Association of clinical embryologists, 2012). According to .. some valuable exceptions like the UK. In general, practical training pro- European standards, to gain advanced knowledge of ART laboratory .. grammes are left to the institutions themselves. They cover the pre- .. techniques, management experience and the ability to actively partici- .. scribed programme that should be carried out in certified learning pate in clinical consultations, training usually takes a minimum of 6 .. centres under the supervision of formal instructors using ethically . 10 Kovacic et al. Figure 4 Difficulty level of exam questions by curriculum topic. (a) Clinical Embryologist exams (2009–2019). (b) Senior Clinical Embryologist exams (2008–2019). .. acceptable biological material. The practical training must be recorded .. level, and has performed at least 50 procedures of each of the nine in predefined logbooks and qualifications through the verification of .. basic laboratory MAR working fields. While it is acknowledged that .. knowledge and issue of a certificate. Most European medical specialist .. this is not a guarantee that the candidate has mastered the practical exams operate in line with this principle and consist of theoretical and .. ART laboratory work, the completed logbook represents only a mini- .. practical work. .. mal proficiency entry criterion for acceptance to take the exam. A completed ESHRE logbook, signed by the supervisor, is compul- .. On average, 8% of applications were rejected. The most common .. sory and serves as confirmation that the candidate has been active in .. reasons were an inadequate level of education, an incomplete logbook the laboratory for at least 3 or 6 years according to the proposed .. and/or inadequate working experience in an ART laboratory. . Clinical embryologists: 10 years of ESHRE certification 11 Figure 5 Post-exam analysis of multiple-choice questions. Scatterplot of difficulty and discrimination indices. Blue area: optimal discrimina- tion and difficulty ranges. Pink area: suboptimal ranges (very difficult, low discrimination). Green area: suboptimal ranges (very easy, low discrimina- tion). (a) Clinical Embryologist exams (2011–2018). (b) Senior Clinical Embryologist exams (2011–2018). 12 Kovacic et al. .. Exam questions background .. Number of applicants and pass rates .. MCQs with only one SBA from four answer options have been shown .. While in most undergraduate and university exams, the pass threshold in previous analyses of medical specialist exams to be appropriate and .. is set a priori at 50% (10/20), in the postgraduate and professional .. objective. They allow for quick and simple manual or computer-based .. exams, the pass threshold may also be set a posteriori at variable levels .. verification of the answers (Tenore et al., 2015). In comparison with .. after each exam, based on post-exam analysis, such as by using the the false-true question type, where there is a 50% chance of obtaining .. Angoff method (Tenore et al., 2015). .. the correct answer by guessing, MCQs give more reliable results .. In the ESHRE Clinical Embryology exams a pass threshold of 66% (Mathysen et al., 2013). Although MCQs also allow for guessing .. was set prior to the first exam, meaning that candidates had to select .. (Skakun et al., 1979), this leads to lower results since one correct op- .. the correct answer for two-thirds of the questions in order to pass. tion (key) and three wrong options (distractors) decrease the chance .. Candidates were notified of this threshold prior to the exam, and this .. for correct answer by guessing to only 25%. In order to prepare high .. has been maintained for all exams. quality questions with the optimal type of distractors, this type of .. Such a pass threshold, although arbitrarily set a priori, was chosen to .. question not only requires a good level of knowledge from the experts .. ensure that only well-prepared candidates succeed in our Clinical .. Embryology exams. Furthermore, a detailed post-exam analysis has al- preparing the questions but also extensive discussions in order to at- .. tain the desired standards. The plausibility of distractors is a prerequi- .. ways been performed each year, allowing correction for eventual vari- .. abilities linked to a specific population of candidates and allowing site for good MCQ building. Constructing plausible distractors .. represents one of the most difficult tasks in preparation of MCQs. .. results to be comparable to those obtained through a posteriori proce- .. dures, such as the Angoff method. There are diverse views on what makes a high quality exam ques- .. tion. Assembling good questions is a task that needs to be developed .. The analysis of applications revealed a great variability in the motiva- .. over the years by the exam board members. In recent years, the .. tion for certification, number of attempts and the pass rates per coun- EmCC has followed the guidelines for the composition of exam ques- .. try. It could be hypothesized that the number of applications per .. tions (The Australian Medical Assessment Collaboration, 2014; .. country depends both on the number of ART laboratory staff involved Tenore et al., 2015), and members have attended workshops dedi- .. in national MAR programmes and recognition of the ESHRE Clinical .. cated to this issue, organized by CESMA-UEMS. This has provided re- .. Embryologist certificate by national professional bodies. .. Several factors, such as the educational, legal or economical situation assurance that the question structure meets the highest standards and .. has allowed for continual improvement in exam quality. .. for each specific country, can play important roles in pass rates. Of rele- .. vance could be the access to scientific literature, meetings or research, Inclusion of low quality questions can reduce overall exam reliability and .. validity. Problems may arise for the following reasons: the leading state- .. and lack of experience with some techniques due to national legislation .. or clinic policy. For example, some clinics do not offer conventional in- ment/question may be poorly written causing candidates to be confused; .. there may not be a clear correct answer, and one or more distractors .. semination for IVF (preferring ICSI), preimplantation genetic testing .. (PGT) or cryopreservation. The lack of experience with the use of could potentially qualify as the correct answer; there may be distractors .. that most candidates can understand as obviously wrong, increasing the .. European standards and legislation for MAR centres (European Tissue .. and Cells Directives) for embryologists from non-EU countries might be odds of selecting the correct answer; inserted figures may not be clearly .. depicted (The Australian Medical Assessment Collaboration, 2014). .. one of several explanations for significant differences in pass rates be- .. In order to minimize most of these issues in ESHRE Clinical .. tween European and non-European candidates, respectively, in Clinical Embryology exams, proactive measures included shortening the ques- .. Embryologist (63% vs 33%) and Senior Clinical Embryologist (52% vs .. tion length and ensuring that all answers were of similar length. In addi- .. 31%) exams. However, the exam passing rate per country might be .. best interpreted by the national or regional professional bodies. tion, all abbreviations or uncommon English words were removed. .. However, there may be some additional factors that can add to the .. Another significant difference in pass rates was observed between .. first time and resit candidates, respectively, who participated in the MCQ difficulty, which are unavoidable. For example, a single MCQ .. may test knowledge from more than one area of the curriculum, such .. Clinical Embryologist exams (63% vs 49%) and the Senior Clinical .. Embryologist exams (50% vs 39%). Initially candidates who failed the as a MCQ testing a combined knowledge of Male Reproduction, .. Genetics and Developmental Biology. This could convey difficulty to .. exam were allowed to re-apply to sit the next exam in the following .. year and the number of resits were increasing with years. However, a those candidates who only work in specific areas of Clinical .. Embryology, and/or did not study the whole curriculum content. .. rule was introduced in 2016 to allow a maximum of three consecutive .. attempts. If the candidate failed on the third attempt, this person Furthermore, candidates from non-European countries could experi- .. ence difficulty related to specific questions regarding EU legislation. .. would have to miss the following year’s exam, but would be eligible to ... apply for the exam in 2 years’ time. This rule was intended to allow Given that the exam in the past 7 years was only in English, non- .. the candidate further time to study for the exam, and excluded candi- English-speaking participants may have been at a disadvantage. One .. .. dates who might gamble on success due to the MCQ format. By free- might postulate that some Dutch, Spanish and Italian-speaking partici- .. pants in the first three exams (provided in their native languages) may .. ing up this candidate space, it also allowed other candidates the .. opportunity to take the exam. have had a better chance and that this may have resulted in a higher .. pass rate (Table I). .. .. MCQ quality assessment However, all of these factors were deliberate, aimed at approaching .. interdisciplinarity of curricular areas, addressing the EU legislation and .. Preparing exams to a consistent level of quality is challenging. If the .. using English as ESHRE’s official language. . exam is considered ‘too easy’, this could lead to degradation of the Clinical embryologists: 10 years of ESHRE certification 13 .. hard-earned reputation of ESHRE certification. On the contrary, if the .. possible answers instead of just one, and extended matching questions exam is considered ‘too difficult’, this could reduce candidate .. may provide an increased assessment of knowledge. All these new .. motivation. .. question types may reduce the chance of getting the right answer by When preparing new questions, the difficulty level of a question can- ... guessing. However, there are also disadvantages with their use, and not be determined until it is used. This is well known by psychometric .. further discussion will be required. .. science and for this reason specific statistical methods for monitoring .. the questions after use have been developed (Kuder and Richardson, .. Discrimination index .. 1937; Cronbach and Warrington, 1951 ; Ebel, 1954). Such feedback is .. Reviewing the answers to each individual MCQ by the best and the also important for the group setting the questions. .. .. worst exam performers has provided information on the discriminating The number of candidates participating in the exam (>250 per .. .. potential of the question. If a MCQ had a low difficulty index but a year) has become sufficiently large to allow reliable statistical analysis .. high discrimination index, this meant that the MCQ was difficult but of exam performance, including question-response analysis (Kehoe, .. .. relevant. MCQs that demonstrated good discriminating potential were 1995). Such systematically performed analysis after each exam has .. usually moderately difficult, while very easy or very difficult MCQs usu- allowed for continuous progressive improvement throughout the .. .. ally had poor or even negative discrimination potential (Sim and years. This has allowed for appropriate support in identifying those .. Rasiah, 2006). Where candidates answered a MCQ correctly but curriculum topics which need better or more detailed description for .. .. scored low in the exam overall, or they answered an MCQ incorrectly future exams. .. but scored high in the exam overall, both scenarios describe a low dis- .. .. crimination index. If a MCQ had a negative discrimination index, this Difficulty index .. usually indicated that the MCQ required removal or correction for fu- .. The exam difficulty level should help to correctly differentiate candi- .. ture use. The exam analysis showed that only 1.8–3.8% of MCQs had dates regarding curriculum knowledge unless the MCQs are out of the .. a negative discrimination index. Altogether, 68–70% of MCQs passed .. optimal range of difficulty. An MCQ demonstrates low discrimination if .. the criteria for good discrimination. .. it is either too difficult or too easy. The acceptable range of MCQ diffi- .. culty indices is usually arbitrarily defined, but this also depends on the .. Distractor analysis .. question type and on the number and specific content of possible dis- .. An optimal distractor must look like a correct answer. It is considered tractors (Crocker and Algina, 1986). .. .. a non-distractor if it is only selected by <5% candidates (Haladyna and The optimal mean difficulty index for a 4-response MCQ is around .. Downing, 1993). A trend for improving the quality of MCQs was 0.7, which is defined as 70% candidates selecting the correct answer .. .. achieved for both exam levels over several years by increasing the pro- (Oermann and Gaberson, 2017), but the optimal difficulty index range .. portion of optimal distractors. By increasing the number of answer is 0.3–0.7 (The Australian Medical Assessment Collaboration, 2014; .. .. options per MCQ from 4 to 5, the negative effect of one non- Oermann and Gaberson, 2017) or even larger, at 0.2–0.8 (Dixon, .. distractor on the quality of MCQ is minimized. 1994). MCQs with difficulty indices outside the acceptable ranges .. .. need attention and should either be removed from subsequent exams .. .. Reliability coefficient KR-20 or revised. .. The ESHRE MCQs had a mean difficulty index of 0.7 and 0.6 in .. The KR-20 is a coefficient for the assessment of exam quality and rep- .. resents an estimation of its internal consistency. The KR-20 depends Clinical Embryologist and Senior Clinical Embryologist exams, respec- .. tively, with the majority of them in ranges of 0.3 to 0.95 and 0.2 to .. on the difficulty, the total number of MCQs and the number of MCQs .. that did not discriminate (Kuder and Richardson 1937). A low KR-20 0.95, respectively. The distribution of the difficulty indices of MCQs .. from all curriculum topics were spread across the entire acceptable .. indicates that many MCQs are too difficult, too easy, or poorly written .. (McGahee and Ball, 2009). All ESHRE Clinical Embryology exams had range and very close to the desired range, but showed a deviation .. from normal distribution (right skewed) towards an easier area. .. a KR-20 in the range of 0.8–0.9 and were therefore considered as .. very good to almost the best standardized tests, according to previ- Overall, this is a further characteristic demonstrating the reliability of .. the exams. .. ously described criteria (Nunnally, 1967). .. However, the upper range limit of 0.95 shows that the exams have .. included several easy MCQs. These could subsequently be revised for .. Curriculum analysis .. future exams to bring them into the acceptable range (Dixon, 1994; .. The MCQs were divided in eight curriculum topics; however, there is .. The Australian Medical Assessment Collaboration, 2014; Oermann .. inevitably some overlap. For example, the topic of Basic Cell Biology and Gaberson, 2017). The mean pass rates of the Clinical .. MCQs may contain questions on meiosis, which overlap the topic of .. Embryologist and Senior Clinical Embryologist exams, 60% and 50%, .. Genetics. respectively, demonstrated that these exams have achieved a status of .. In addition, certain curriculum topics have changed significantly over .. moderately difficult to difficult exams. Difficulty indices of most of the .. time. For example, the topic of Cryopreservation has undergone sub- MCQs were considered optimal, with 73% and 77% of the Clinical .. stantial change with the introduction of vitrification. Furthermore, labo- .. Embryologist and Senior Clinical Embryologist MCQs, respectively, in .. ratory techniques of embryo biopsy and PGT, as well as the views on the range between 0.2 and 0.8 (Dixon, 1994). .. the relevance of their use, have changed considerably over the last .. Inclusion of other types of question types, such as MCQs of the .. 10 years. Consequently, some MCQs were removed, modified or SBA type with five answer options instead of four, MCQs with several .. newly introduced in order to keep up with the new trends in our field. 14 Kovacic et al. .. The new ESHRE guidelines, the Atlas of Human Embryology, and con- .. .. Acknowledgements sensus publications (e.g. Magli et al., 2012; ESHRE Guideline Group .. The authors thank Kay Elder for providing initial pool of MCQs and to on Good Practice in IVF Labs2016 ; ESHRE SIG-E and Alpha, 2017) .. has resulted in the introduction of new questions derived from these ... all past EmCC members: Joyce Harper, Etienne van den Abbeel, Julius documents. Attempts were made to include schemes and photo ma- .. Hreinsson, Maria José de los Santos, Giovanni Coticchio and Heidi .. Rojiemans for their valuable contribution to ESHRE Clinical terial in the exam, but due to the altered quality of the photographs .. on the printed sheets, this practice did not expand as anticipated. The .. Embryologist certification programme. For the pilot exam in India, we .. use of electronic exams in the coming years will facilitate and possibly .. thank Kuldeep Jain and Jayant Mehta from the Indian Fertility Society. expand this possibility. Depending on the specific modifications, the list .. Thanks also to Ioannis Messinis for critical reading of the manuscript, .. of recommended literature was regularly and accordingly updated. It is .. to Wil van Groesen for development of on-line application platform reassuring that, despite being 10 years old, the curriculum still conve- ..  Kovacic for his contribution to the statistical analysis. and to Ziga .. niently covers all major areas. However, it may benefit from updates .. in the near future. .. .. Authors’ roles Over this 10-year period, various questions have been asked by .. candidates about the application process and the exams. To assist .. .. B.K. and C.E.P. co-ordinated the analysis and writing the manuscript. with this, a ‘Frequently Asked Questions’ section was added the .. F.J.P., as MCQ bank manager, provided raw data and methodological .. ESHRE website, which is updated as required. Complaints have been .. support for further data analysis. C.P. provided administrative and tech- marginal, with no complaints received from candidates who passed the .. nical support in preparation and organization of exams and their evalu- .. exams. .. ations. All other authors contributed to the interpretation of the .. analysis and critically revised the manuscript. .. .. .. Conclusion .. .. Funding Both the Clinical Embryologist and the Senior Clinical Embryologist .. .. The study has no external funding. exams are demanding and have adequate discrimination power. The .. difficulty grade is evenly distributed across the eight topics of the cur- .. .. riculum. Furthermore, the exams demonstrated high reliability and .. Conflict of interest allowed for appropriate scrutiny of the knowledge of Clinical .. .. The authors reported no conflicts of interest. Embryologists. .. The interest and performance in the ESHRE Clinical Embryology .. .. References exams varies among candidates from different countries. Academic .. level and educational background were factors associated with differ- .. .. Alpha Scientists in Reproductive Medicine. The Alpha Consensus ences in performance. .. Meeting on the professional status of the clinical embryologist: pro- The ESHRE exams on Clinical Embryology are the most widely ac- .. .. ceedings of an expert meeting. Reprod Biomed Online 2015;30: cepted tests of knowledge from laboratory science in MAR. These .. .. 451–461. exams provide the recognition that minimum standards have been .. Calhaz-Jorge C, Feki A, Farquharson R. European view of subspecialty achieved. The acquired certificate allows ESHRE certified Clinical and .. .. training on behalf of the European Society of Human Reproduction Senior Clinical Embryologists to be included in the ESHRE CPD pro- .. and Embryology (ESHRE). Fertil Steril 2015;104:8–11. gramme and registered on the list of ESHRE certified and CPD active .. .. Crocker L, Algina J. Introduction to Classical and Modern Test Theory. Clinical Embryologists. .. New York: Holt, Rinehart and Winsto, 1986. Clinical Embryology is increasingly recognized as a specific discipline .. .. Cronbach LJ. Warrington WG. Time-limit tests: estimating their reli- for scientific staff collaborating closely with clinicians in managing hu- .. ability and degree of speeding. Psychometrika 1951;16:167. man infertility by MAR. ESHRE Certification for Clinical Embryologists, .. .. 10.1007/BF02289113 14844557 and its current recognition by CESMA-UEMS, adds clarification to this .. health-staff postgraduate examination, and conveniently adds Clinical .. Dixon RA. Evaluating and improving multiple choice papers: true– .. false questions in public health medicine. Med Educ 1994;28: Embryology into the wider group of health disciplines harmonized .. through professional societies such as ESHRE and EBCOG. .. 400–408. .. Downing SM, Baranowski RA, Grosso LJ, Norcini JJ. Item type and The EmCC will continue its efforts to reinforce ESHRE’s position at .. UEMS-CESMA and to follow its advice. The authors hope that the cer- .. cognitive ability measured: the validity evidence for multiple true- .. false items in medical specialty certification. Appl Measur Educ tification results shown will enhance recommendations towards deliver- .. ing learning programmes of knowledge in Clinical Embryology and will .. 1995;8:187–197. .. increase the interest for ESHRE certification programmes worldwide. .. Ebel RL. Procedures for the analysis of classroom tests. Educ Psychol .. Meas 1954;14:352–364. .. .. ESHRE Special Interest Group of Embryology and Alpha Scientists in .. Reproductive Medicine. The Vienna consensus: report of an expert Supplementary data .. .. meeting on the development of ART laboratory performance indi- Supplementary data are available at Human Reproduction Open online. .. cators. Reprod Biomed Online 2017;35:494–510. Clinical embryologists: 10 years of ESHRE certification 15 .. ESHRE Guideline Group on Good Practice in IVF Labs, De los .. European Board of Ophthalmology Diploma examination. Acta Santos MJ, Apter S, Coticchio G, Debrock S, Lundin K, Plancha .. Ophthalmol 2013;91:e573–e577. .. CE, Prados F, Rienzi L, Verheyen G, Woodward B, Vermeulen N. .. McGahee Ball TW, J. How to read and really use an item analysis. Revised guidelines for good practice in IVF laboratories (2015). .. Nurse Educ 2009;34:166–171. .. Hum Reprod 2016;31:685–686. .. Nunnally JC. Psychometric Theory. New York: McGraw-Hill, 1967, pp. Go KJ. ‘By the work, one knows the workman’: the practice and .. 172–235. .. profession of the embryologist and its translation to quality in the .. Oermann M, Gaberson K (eds). Evaluation and Testing in Nursing embryology laboratory. Reprod Biomed Online 2015;31:449–458. .. Education. New York, Springer Publishing Company, 2017, p. 403. .. Haladyna TM, Downing SM. How many options is enough for a .. Practice Committee of the American Society for Reproductive multiple-choice test item? Educ Psychol Meas 1993;53:999–1010. .. Medicine; Practice Committee of the Society for Assisted .. Hamilton J, Carachi R. Clinical embryology: is there still a place in .. Reproductive Technology; Practice Committee of the Society of medical schools today? Scot Med J 2014;59:188–192. .. Reproductive Biologists and Technologists. Recommended practices .. Hughes C, Association of clinical embryologists. Association of clini- .. for the management of embryology, andrology, and endocrinology cal embryologists—guidelines on good practice in clinical embryol- .. laboratories: a committee opinion. Fertil Steril 2014;102:960–963. .. ogy laboratories 2012. Hum Fertil (Camb) 2012;15:174–189. .. Richardson MW, Stalnaker JM. A note on the use of bi-serial r in Kehoe J. Basic item analysis for multiple choice tests. Pract Assess J .. test research. J Gen Psychol 1933;8:463–465. .. Res Eval 1995;4:10. .. Sim SM, Rasiah RI. Relationship between item difficulty and discrimi- .. Kelley TL. The selection of upper and lower groups for the validation .. nation indices in true/false multiple choice questions of a para- of test Items. J Educ Psychol 1939;30:17–24. .. clinical multidisciplinary paper. Ann Acad Med Singapore 2006;35: .. Kovacic B, Plas C, Woodward BJ, Verheyen G, Prados FJ, Hreinsson .. 67–71. J, De Los Santos MJ, Magli MC, Lundin K, Plancha CE. The educa- .. Skakun EN, Nanson EM, Kling S, Taylor WC. A preliminary investiga- .. tional and professional status of clinical embryology and clinical .. tion of three types of multiple choice questions. Med Educ 1979; embryologists in Europe. Hum Reprod 2015;30:1755–1762. .. 13:91–96. .. Kuder GF, Richardson MW. The theory of the estimation of test reli- .. Tenore A, Mathysen DGP, Mills P, Westwood M, Rouffet J, Papalois ability. Psychometrika 1937;2:151–160. .. V, Sparrow O, Goldik Z. A guide to successfully writing MCQs: .. Lamb VK. National occupational standards in clinical embryology: im- .. Executive summary. UEMS-CESMA Publication 2015. plementation in 2005? Hum Fertil 2005;8:1–4. .. The Australian Medical Assessment Collaboration (AMAC). .. Magli MC, Jones GM, Lundin K, van den Abbeel E. Atlas of human .. Determining the Quality of Assessment Items in Collaborations: embryology: from oocytes to preimplantation embryos. Preface. .. Aspects to Discuss to Reach Agreement. Australian Government .. Hum Reprod 2012;27(Suppl 1):i1. .. Department of Education, Canberra, Australia 2014, 26 p. Mathysen DG, Aclimandos W, Roelant E, Wouters K, Creuzot- .. Trávnı́k P, Hampl A, Hüttelová R, Malenovská A, Priesnitz J, Rejthar .. Garcher C, Ringens PJ, Hawlina M, Tassignon MJ. Evaluation of ..  D, Záková J. Theoretic and practical content of the clinical adding item-response theory analysis for evaluation of the .. Embryology. Ceska Gynekol 2013;78:400–401.