ORIGINAL RESEARCH published: 18 February 2020 doi: 10.3389/fvets.2020.00056 Cross-Validation of Generic Risk Assessment Tools for Animal Disease Incursion Based on a Case Study for African Swine Fever Clazien J. de Vos 1*, Rachel A. Taylor 2 , Robin R. L. Simons 2 , Helen Roberts 3 , Cecilia Hultén 4 , Aline A. de Koeijer 1 , Tapani Lyytikäinen 5 , Sebastian Napp 6 , Anette Boklund 7 , Ronald Petie 1 , Kaisa Sörén 4 , Manon Swanenburg 1 , Arianna Comin 4 , Leena Seppä-Lassila 5 , Maria Cabral 1 and Emma L. Snary 2 1 Department of Bacteriology and Epidemiology, Wageningen Bioveterinary Research (WBVR), Wageningen University & Research, Lelystad, Netherlands, 2 Department of Epidemiological Sciences, Animal and Plant Health Agency (APHA), Weybridge, United Kingdom, 3 Department for Environment, Food & Rural Affairs (Defra), London, United Kingdom, 4 National Veterinary Institute (SVA), Uppsala, Sweden, 5 Finnish Food Authority (Ruokavirasto), Helsinki, Finland, 6 Centre de Recerca en Sanitat Animal (CReSA IRTA-UAB), Bellaterra, Spain, 7 Department of Veterinary and Animal Sciences, Section for Animal Welfare and Disease Control, University of Copenhagen, Frederiksberg, Denmark In recent years, several generic risk assessment (RA) tools have been developed that can be applied to assess the incursion risk of multiple infectious animal diseases allowing Edited by: for a rapid response to a variety of newly emerging or re-emerging diseases. Although Thibaud Porphyre, these tools were originally developed for different purposes, they can be used to answer University of Edinburgh, United Kingdom similar or even identical risk questions. To explore the opportunities for cross-validation, Reviewed by: seven generic RA tools were used to assess the incursion risk of African swine fever Solenne Costard, (ASF) to the Netherlands and Finland for the 2017 situation and for two hypothetical EpiX Analytics, United States Maria Eleni Filippitzi, scenarios in which ASF cases were reported in wild boar and/or domestic pigs in Sciensano, Belgium Germany. The generic tools ranged from qualitative risk assessment tools to stochastic *Correspondence: spatial risk models but were all parameterized using the same global databases for Clazien J. de Vos disease occurrence and trade in live animals and animal products. A comparison of clazien.devos@wur.nl absolute results was not possible, because output parameters represented different Specialty section: endpoints, varied from qualitative probability levels to quantitative numbers, and were This article was submitted to expressed in different units. Therefore, relative risks across countries and scenarios Veterinary Epidemiology and Economics, were calculated for each tool, for the three pathways most in common (trade in live a section of the journal animals, trade in animal products, and wild boar movements) and compared. For Frontiers in Veterinary Science the 2017 situation, all tools evaluated the risk to the Netherlands to be higher than Received: 05 June 2019 Finland for the live animal trade pathway, the risk to Finland the same or higher as Accepted: 22 January 2020 Published: 18 February 2020 the Netherlands for the wild boar pathway, while the tools were inconclusive on the Citation: animal products pathway. All tools agreed that the hypothetical presence of ASF in de Vos CJ, Taylor RA, Simons RRL, Germany increased the risk to the Netherlands, but not to Finland. The ultimate aim of Roberts H, Hultén C, de Koeijer AA, Lyytikäinen T, Napp S, Boklund A, generic RA tools is to provide risk-based evidence to support risk managers in making Petie R, Sörén K, Swanenburg M, informed decisions to mitigate the incursion risk of infectious animal diseases. The case Comin A, Seppä-Lassila L, Cabral M study illustrated that conclusions on the ASF risk were similar across the generic RA and Snary EL (2020) Cross-Validation of Generic Risk Assessment Tools for tools, despite differences observed in calculated risks. Hence, it was concluded that the Animal Disease Incursion Based on a cross-validation contributed to the credibility of their results. Case Study for African Swine Fever. Front. Vet. Sci. 7:56. Keywords: African swine fever, cross-validation, livestock diseases, generic model, introduction risk, model doi: 10.3389/fvets.2020.00056 uncertainty, risk assessment Frontiers in Veterinary Science | www.frontiersin.org 1 February 2020 | Volume 7 | Article 56 de Vos et al. Cross-Validation of Generic RA Tools INTRODUCTION ASF is a viral disease of pigs and wild boar caused by the ASF virus, the only member of the family Asfarviridae (21–23). Increasing globalization and international trade contribute to ASF virus (genotype 2) was introduced into the Caucasus region rapid expansion of infectious animal diseases, as illustrated in 2007 and has subsequently spread into Belarus, Russia and by the recent outbreaks of bluetongue (BT), African swine Ukraine and then to the European Union in 2014 with infection fever (ASF), lumpy skin disease (LSD), and peste des petits having been reported from Estonia, Latvia, Lithuania, Poland, ruminants (PPR) in Europe (1–3). Introduction of exotic animal Czech Republic, Romania, Hungary, Bulgaria, and Belgium in diseases into naive livestock populations can result in large-scale recent years (2, 3). Prevention of ASF introduction is a high epidemics with serious economic and socio-ethical impact (4–6). priority for European countries still free from the disease as Hence, preparedness is warranted to prevent, detect, and control introduction of the disease can have severe consequences for outbreaks of exotic animal diseases. To make decisions on risk the domestic pig sector and the wild boar population due to management of exotic animal disease threats, it is necessary to an extremely high case fatality rate (up to 100%) (24). Rigorous know which animal diseases pose the highest threats and should measures are needed to control the disease including culling of therefore deserve more attention. infected herds and movement bans, and there are difficulties in Risk assessment is a useful tool for prioritization of diseases eradicating the disease once it has established itself in the wild with respect to their incursion risk, the results of which can boar population (24, 25). There is currently no vaccine available be used to assign resources for prevention and surveillance against ASF infection and infection continues to spread through to those diseases posing the highest risk or to identify targets various pathways, including movement of infected wild boar and for additional research. Most commonly, risk assessments are human mediated routes. developed to assess the risk for a single disease and introduction Results of risk assessment studies are an important input into pathway. In recent years, however, several generic risk models or risk management decisions to prevent ASF spread to new regions frameworks have been developed that can be applied to assess the and several bespoke models for ASF have been developed in the incursion risk for multiple animal diseases (7–15). In contrast to last decade (26–31). All generic RA tools of this study can also be bespoke models, these generic risk assessment (RA) tools allow used to assess the ASF incursion risk with the major advantage for a more rapid response to a variety of newly emerging or that results can easily be updated when outbreaks are reported re-emerging diseases. from new regions in Europe. However, because of their generic Generic RA tools, however, tend to have a lower resolution character, validation is an even more essential step to ensure in their algorithms to allow the assessment of risk over multiple plausibility of results. diseases that differ with respect to the species of animals To cross-validate the generic RA tools of this study, all tools affected, transmission modes, and epidemiological and economic were used to assess the incursion risk of ASF to two European impact. Furthermore, uncertainty and variability are not always countries, the Netherlands and Finland. These countries were considered in much detail. To parameterize generic RA tools, chosen because of their opposite risk profiles when considering global databases are preferred, as these contain information their ASF risk, with the Netherlands being a trading country over multiple countries, diseases, and/or introduction pathways. not only exporting millions of pigs annually, but also importing Application of results from generic RA tools vary from a over a million of live pigs each year (32), whereas Finland rapid response to new emergencies to horizon scanning and has hardly any international trade in live pigs (33). Finland, prioritization of diseases, pathways, or regions. on the contrary, was geographically much closer to observed A shared problem of generic tools with bespoke models is outbreaks of ASF in Europe at the time this study was the validation of their results. Most import risk assessments initiated (before the ASF outbreaks in wild boar in Belgium in estimate the probability of rare events occurring in an ever- September 2018). changing world, leaving the use of a long range of historical data for this purpose useless. Validation of import risk assessments is, nevertheless, an important step in their development to MATERIALS AND METHODS ensure the plausibility of results. Sensitivity analysis to address parameter uncertainty by varying the values of uncertain input Definitions parameters in a plausible range contributes to the internal Although an assessment of the incursion risk would ideally validation of risk assessments (16, 17). With several generic RA address both the probability of incursion of disease and tools having been developed in recent years in Europe that all can its subsequent consequences, not all generic RA tools of be used to assess the incursion risk of notifiable animal diseases this study had incorporated a consequence assessment. such as ASF, LSD, and BT, opportunities arise to address model Therefore, in this study incursion risk is used as a generic uncertainty by comparing results obtained with different tools. term to indicate any metric to estimate the risk of exotic The objective of this study was to explore the opportunities for diseases entering a new territory, varying from entry only cross-validation of generic RA tools, where cross-validation was to a full risk assessment including epidemiological and defined as the validation of model results by comparing them economic consequences. with results of other models that addressed the same question. Regarding the probability of incursion, four separate steps ASF was selected as a case study given its rapid spread in Europe are distinguished in this study. In the entry step the probability in recent years (18–20). that an infectious agent enters a new territory (hereafter called: Frontiers in Veterinary Science | www.frontiersin.org 2 February 2020 | Volume 7 | Article 56 de Vos et al. Cross-Validation of Generic RA Tools TABLE 1 | Brief outline of seven generic RA tools. RA tool Objective Prioritization Data Number of Number of Variability/ Software Developer More available diseases in pathways in uncertainty information in tool tool tool included SPARE Early warning of Target areas Yes 4 5 None R APHA, UK (14, 35) disease incursion risks COMPARE Identification of Target areas, No 3 5 Variability R APHA, UK (15, 36–38) hotspots for pathways risk-based surveillance RRAT Identification of high Pathways, Yes 10 3 None R, SQLite WBVR, NL (39) priority exotic diseases notifiable diseases MINTRISK Comparison and Target areas, No NAa NAb Uncertainty C#, Visual WBVR, NL (10, 11, 40, 41) prioritization of diseases Studio vector-borne diseases IDM Identification of high Diseases Yes 34 7 None Excel Defra, UK (8, 11) priority exotic notifiable diseases NORA Rapid risk assessment Pathways No NAa 9 None Excel Ruokavirasto, (12) to respond to new FI disease events SVARRA Rapid risk assessment Pathways No NAa 8 Uncertainty Word, Excel SVA, SE (11) to respond to new disease events a These tools have no underlying database with disease-specific data and can evaluate any disease. b MINTRISK can evaluate any pathway. target area) by any pathway is evaluated, without assessing Generic Risk Assessment Tools subsequent exposure of susceptible animals in the target area Seven generic RA tools, all developed by the G-RAID1 via this pathway. In the exposure step the probability that a consortium (34), were included in the cross-validation study. susceptible native host animal in the target area is exposed to Selection criteria for inclusion were (i) the tool was developed the infectious agent is evaluated given its entry into the target to assess the incursion risk of multiple diseases rather than area, without assessing the probability that such exposure would a single disease, and (ii) the tool focused on the veterinary result in infection. In the first infection step the probability risk of animal diseases rather than public health. Although all that contact with the infectious agent results in infection seven tools can be used to address the incursion risk of exotic of a first native host in the target area (= index case) is animal diseases, they were originally developed for different evaluated given its entry into the target area and exposure purposes ranging from immediate response to new disease of the host animal. In the establishment step the probability events to prioritization of diseases and horizon scanning. As that the infectious agent will start spreading in the target a consequence, input, algorithms, and endpoints of the tools area is evaluated given it has entered the target area and differed. The seven generic RA tools included two quantitative resulted in a first infection of native host animals. Establishment tools (SPARE, COMPARE), four semi-quantitative tools (RRAT, is an important step when evaluating the incursion risk of MINTRISK, IDM, NORA) and one qualitative tool (SVARRA). vector-borne diseases, where an infection of a native host A brief overview of all tools is given in Table 1. More details are animal can be a dead-end if, e.g., no competent vectors are available in Supplementary Material 1. present in the target area or if climatic conditions impede All seven RA tools were built to be flexible with respect to subsequent transmission. the animal diseases to be evaluated, although MINTRISK was Consequences of disease incursion have been separated into primarily designed to assess the risk of vector-borne diseases. The epidemiological and economic consequences. Epidemiological total number of diseases evaluated so far with each of the tools consequences have been defined as the expected spread of the varies greatly, as does the level of resources (expertise, data, time) infectious agent in the native (susceptible) population in the needed to complete a risk assessment. For all tools, the assessment target area or further geographical spread from the target is less rapid if the disease has not been evaluated before with the area to new regions, considering, e.g., the epidemic size and tool, because additional data collection and parameterization is geographic area affected. Economic consequences have been required. The RRAT and the IDM tool have the data required defined as the expected monetary losses resulting from an to perform the risk assessment readily available in the tool for outbreak with the infectious agent in the target area due to, e.g., morbidity and mortality, production losses, control measures 1 Generic approaches for Risk Assessment of Infectious animal Disease and trade restrictions. introduction. Frontiers in Veterinary Science | www.frontiersin.org 3 February 2020 | Volume 7 | Article 56 de Vos et al. Cross-Validation of Generic RA Tools TABLE 2 | Introduction pathways embedded in each of the generic RA tools. PATHWAY SPARE COMPARE RRAT MINTRISKa IDM NORA SVARRA Live animals X X X X X X X Products of animal origin X X X X X X Germplasm X X X X Vectors X X X X X Wildlife X X X X X X Human travel X X X X Transport X X X Laboratory material and samples X Feed and bedding X X Airborne spread X a MINTRISK can deal with any pathway, but none are embedded in the tool; these are the pathways that were evaluated for the ASF case study. FIGURE 1 | Outline of the steps involved in assessing disease incursion risks indicating the entirety of the generic RA tools. a multitude of diseases. SPARE and COMPARE have only been considered, either related to vertebrate host animals and their parameterized for a few diseases, although the data available on products, vectors, or humans. In general, the tools with relatively disease prevalence worldwide in these tools can theoretically be uncomplicated algorithms, such as the qualitative tool SVARRA used to assess the risk of any OIE-listed disease. MINTRISK, and the semi-quantitative tools IDM and MINTRISK, are most NORA, and SVARRA come mostly without underlying databases flexible to include additional pathways. and have to be filled by the risk assessor. The outline of all seven generic RA tools is primarily based The tools also differ widely with respect to the number of on the OIE import risk assessment framework (42). However, the introduction pathways that can be evaluated (Table 2). Legal RA tools differ widely with respect to the steps that are included trade in live animals and imports of products of animal origin to assess the disease incursion risk (Figure 1). Endpoints of are considered by each tool, although these pathways were the tools differ alike. MINTRISK is the most complete tool not always consistently defined across the tools. For example, considering entry up to establishment, and epidemiological and some tools consider trade in livestock animals only, whereas economic consequences. SPARE on the contrary only considers other tools also consider trade in pets and exotic animals. entry. All tools use the basic principles of the Binomial model Most tools also address windborne vector spread and wild (43) to assess entry of pathogens into new areas, combining animal dispersion including migratory birds. All tools but information on pathway numbers (N) with probabilities of MINTRISK have predefined pathways built in. MINTRISK asks infection (p) based on prevalence levels. All tools that evaluate the risk assessor to define relevant pathways for the disease epidemiological consequences (COMPARE, MINTRISK, IDM) Frontiers in Veterinary Science | www.frontiersin.org 4 February 2020 | Volume 7 | Article 56 de Vos et al. Cross-Validation of Generic RA Tools FIGURE 2 | ASF cases in domestic pigs and wild boar in 2017 as reported to the OIE (2). Target countries of the case study (the Netherlands and Finland) have been colored yellow. Inset: location of hypothetical ASF cases in Germany. do so by estimating the basic reproductive number R0 (44) and well as trade patterns in 2017, what is the predicted incursion implementing it into their model calculations. risk of ASF to (a) the Netherlands and (b) Finland from any Only a few of the generic RA tools have embedded uncertainty country where the virus strain circulates? In addition, the same and/or variability in their risk assessment. COMPARE is the question was answered considering two hypothetical scenarios only tool addressing variability in calculating the incursion risk in which ASF cases were reported in Germany. In the first using stochastic calculations, including primarily the variability hypothetical scenario (HS1), it was assumed that on 30/12/2017 in disease prevalence in traded animals/products but also 10 separate cases of wild boar found dead, infected with ASF, variability in other disease parameters. MINTRISK and SVARRA were reported from the Munster region of Germany at a distance are the only tools that explicitly ask the risk assessor for of ∼50 km from the Dutch border (Figure 2). In the second his/her uncertainty in estimating the input parameter values, hypothetical scenario (HS2), it was assumed that on 30/12/2017, and only MINTRISK uses stochastic simulation to address this as per scenario HS1, ten separate cases of ASF in wild boar were uncertainty. NORA also acknowledges that risk assessors cannot reported from the Munster region of Germany, and that one be expected to know everything and offers the “I don’t know” outbreak on a single commercial mixed (breeding and fattening) option in answering the questions. This uncertainty is reported farm had been reported in the same region with 2,500 pigs on it, in the results of the tool by counting the number of questions 18 of which were found infected and all 2,500 were immediately that were given this answer. Despite the fact that uncertainty culled (Figure 2). In both hypothetical scenarios, the history of is not embedded in the other RA tools, most of them offer the ASF outbreaks and reported trade patterns were assumed to be opportunity to consider uncertainty via scenario analysis. the same as in the 2017 scenario. Furthermore, hypothetical ASF cases in scenarios HS1 and HS2 were assumed to behave in a similar way to the other cases in Europe in 2017 with regards to Risk Question and Scenarios characteristics such as infectious period and transmission rate. To explore the opportunities for cross-validation of the generic RA tools, all seven tools were used to assess the risk for a selected case study considering several African swine fever scenarios. The Input Data for the African Swine Fever risk question considered the ASF virus strain responsible for Case Study European cases in 2017 as the hazard and was formulated as: The majority of the data required for the generic RA tools can Given the history of ASF cases reported in Europe in 2017, as be broken down into four categories: (1) pathway movements Frontiers in Veterinary Science | www.frontiersin.org 5 February 2020 | Volume 7 | Article 56 de Vos et al. Cross-Validation of Generic RA Tools from source areas to the target area; (2) disease prevalence from WAHIS. The RA tools used, however, different approaches in source areas; (3) susceptible animals in the target area; to convert the WAHIS data into disease prevalence estimates. and (4) disease-specific parameters. For the case study, input Whereas some of them included all historical data available from data used by the tools were harmonized wherever possible the OIE website, others only considered information on disease to ensure that differences in results would result from model outbreaks in a recent period (e.g., a 1-year period or the high- uncertainty rather than input parameter uncertainty. An risk period, i.e., the period from introduction of disease into an overview of input data shared between the tools is available in area until its detection). Furthermore, some tools had built-in Supplementary Material 2. algorithms to correct for non-reporting or underreporting, or to include a probability of disease presence despite current absence, Pathway Movements whereas other tools based their prevalence estimates on the actual Data on pathway movements are relevant to the entry stage situation reported to OIE only. In addition, the more qualitative of the risk assessment and focus on how many units (animals, tools had the ability to include additional information derived products etc.) will reach the target area from different source from, e.g., ADNS and reports and mailings from the European areas, regardless of whether or not they are infected. All generic Commission, especially when assessing the infection probabilities RA tools derive data on pathway movements predominantly of wild boar populations. from global databases on international trade. For the legal trade pathways, all of the tools can use one or more of TRACES (45), Susceptible Animals Comext (33), or Comtrade (46). For movement of wild animals Data on susceptible animals relate to the probabilities of disease and illegal trade, global datasets are obviously not possible to transmission to susceptible populations in the target area. For obtain. However, most tools incorporate global datasets of travel this category of data input, more variation is observed between statistics (47) or population abundance maps in order to either the seven tools due to contrasting choices made to model contact model or estimate a score for how much illegal trade or wild between susceptible animals and infected animals/products. animal movements are to be expected. Whereas some tools include detailed data on number of animals For the case study, all of the generic RA tools used Comext and farms or livestock densities in the target area derived from trade data for 2017 (33) to assess the numbers of animals, global databases such as WAHIS (2) and FAOSTAT (53), other products (including germplasm and laboratory material), and tools distinguish between different farm types based on size and travelers entering the Netherlands and Finland. However, the biosecurity level or include information on disease susceptibility selection of CN codes (Combined Nomenclature) (48, 49) only. One of the tools, SPARE, has not included information included in the tools depended on how pathways were defined on susceptible animals in the target area at all, because it only and was not the same across all tools, especially when considering evaluates entry of the pathogen, and not subsequent exposure the legal trade in animal products. Population abundance maps or infection. (50) were used to either model or estimate a score for how As a result, harmonization of data on susceptible animals in many wild boar movements were to be expected. The quantitative the Netherlands and Finland was difficult. COMPARE used maps tools (SPARE, COMPARE) applied mechanistic approaches (in on wild boar abundance (50), whereas most tools did not need which a complex problem is broken down to the underlying data at this spatial scale and used data on wild boar presence and mechanisms) to determine the movement of wild boars and abundance from Dutch and Finnish sources (54–56). harmonized the parameters within these tools as far as possible, e.g., percentage of wild boars that move long distances and how Disease-Specific Parameters far they move. The category of disease-specific input parameters includes all parameters specific to the disease, such as duration of the latent Disease Prevalence and infectious period, transmission probabilities, severity of Data on disease prevalence are also relevant to the entry stage of clinical signs, test sensitivities, and decay rates in products. For the risk assessment and are used to estimate the probability that this category of data input, the tools differ in whether and how animals/products are infected. Disease prevalence in source areas they incorporate these parameters; however, all of them primarily is primarily based on information on disease occurrence derived used published literature and expert opinion to find relevant from the World Animal Health Information System (WAHIS) parameter values. (2), although Empres-i (51) and the Animal Disease Notification For the case study, disease-specific parameters were sourced System (ADNS) (52) are also used across the tools. Furthermore, individually by each tool from published literature [e.g., (57– the more qualitative tools might also include information derived 61)] and expert opinion. Parameter values were shared to enable from reports and mailings from the European Commission. harmonization of input data over the RA tools if the same input For the case study, all generic RA tools were fed with data parameters were used (Supplementary Material 2). from WAHIS (2) on the number of ASF outbreaks and the number of cases by species (domestic pigs, wild boar) on the Comparison of Results country level for 2017 and previous years. An overview of ASF Although ideally both the probability of incursion of disease cases reported to OIE in 2017 is given in Figure 2. To estimate and its consequences are evaluated in assessing the incursion disease prevalence in the source countries, information on pig risk of exotic diseases, results of the generic RA tools for the and wild boar populations in those countries were also derived ASF case study could only be compared for the probability Frontiers in Veterinary Science | www.frontiersin.org 6 February 2020 | Volume 7 | Article 56 de Vos et al. Cross-Validation of Generic RA Tools TABLE 3 | Risk of ASF incursion as evaluated by the seven generic RA tools. RA tool Endpoint Output type Output parameter SPARE Entry Quantitative Number of entries per year COMPARE First infection Quantitative Annual probability RRAT First infection Semi-quantitative Risk score (from 0 to 1) MINTRISK Establishment Semi-quantitative Annual rate IDM Exposure Semi-quantitative Risk score (from 0 to 60) NORA First infection Semi-quantitative Risk score (from 0 to 1) SVARRA Exposure Qualitative Qualitative probability level of incursion, because consequences were only assessed by some tools. A comparison of absolute results obtained by the seven generic RA tools was not possible though, because endpoints for the probability of incursion varied from entry to establishment (Table 3). In addition, the tools had different output parameters (Table 3) and evaluated different numbers and types of introduction pathways (Table 2). All tools assessed the ASF risk from legal trade in live animals and six out of seven tools assessed trade in animal products (all but MINTRISK) and wild boar movements (all but RRAT). Therefore, for each tool, relative risks were calculated by country and scenario for the three pathways most in common to enable comparison of results. These were compared to see if the tools agreed on the directions and magnitudes of the relative risks resulting in a similar prioritization of countries and scenarios. In addition the pathways within each RA tool were compared against each other to identify the pathways contributing most to the ASF risk to the Netherlands and Finland. The relative risk across the two countries (RRcij ) for each tool FIGURE 3 | Relative risk of introducing ASF into the Netherlands compared to Finland in the baseline scenario (2017 situation) by (A) trade in live animals, (B) i was calculated for each pathway j as: trade in animal products, and (C) movement of wild boar. A relative risk above 1 (bold line) denotes the Netherlands has a higher risk than Finland, while a RRcij = RNLDij /RFINij relative risk below 1 denotes Finland has a higher risk. Please note the different scales used on the y-axes. where RNLDij is the calculated risk by tool i to the Netherlands for pathway j and RFINij the calculated risk by tool i to Finland for pathway j. Calculations were done for the baseline scenario (2017 situation) only. RA tools provided numerical results, either representing absolute Relative risks across scenarios (RRHS1ijk and RRHS2ijk ) for each risk estimates or semi-quantitative risk scores (see Table 3), tool i were calculated for each pathway j and each country k as: making it possible to calculate relative risks. The only exceptions were scenarios resulting in a negligible or zero result. To enable RRHS1ijk = RHS1ijk /RBaseijk and RRHS2ijk = RHS2ijk /RBaseijk the calculation of relative risks, negligible and zero results were set equal to 10−10 , based on the lowest results that were calculated where RBaseijk is the calculated risk by tool i for pathway j and by the tools (7 × 10−10 for the risk of the live animal trade country k for the baseline scenario (2017 situation), RHS1ijk is the pathway to Finland as calculated by RRAT). calculated risk by tool i for pathway j and country k for the first hypothetical scenario (ASF in wild boar in Germany), and RHS2ijk RESULTS is the calculated risk by tool i for pathway j and country k for the second hypothetical scenario (ASF in wild boar and domestic Relative Risks Across Countries pigs in Germany). For each of the generic RA tools, the ASF incursion risk to the In order to calculate relative risks for the qualitative RA Netherlands was compared to Finland for the pathways (a) trade tool SVARRA, qualitative probability levels were converted to in live animals, (b) trade in animal products, and (c) wild boar numerical values using a log-scale, where negligible = 1, very movements. Results for the baseline scenario (2017 situation) are low = 10, low = 100, etc. The in-between probability√ level given in Figure 3. A calculated relative risk above 1 indicates negligible/very low was given a numerical value of 10. All other that the ASF risk was higher to the Netherlands than Finland. Frontiers in Veterinary Science | www.frontiersin.org 7 February 2020 | Volume 7 | Article 56 de Vos et al. Cross-Validation of Generic RA Tools FIGURE 4 | Relative risk of introducing ASF into (A) the Netherlands and (B) Finland in the hypothetical scenario with ASF reported in wild boar in Germany (HS1) and (C) the Netherlands and (D) Finland with ASF reported in wild boar and domestic pigs in Germany (HS2) compared to the baseline scenario (2017 situation). A relative risk of 1 (bold line) denotes no differences in risks among the scenarios. Please note the different scales used on the y-axes. From Figure 3A it can be seen that, for each of the RA tools, The hypothetical situation in which ASF cases were reported the evaluated risk of the live animal trade pathway was higher to in wild boar in Germany (HS1) resulted in an increased risk to the Netherlands than Finland. In particular SPARE, COMPARE, the Netherlands compared to the baseline scenario, especially RRAT and NORA predicted a much higher risk (i.e., over 107 for the wild boar pathway. In particular SPARE and COMPARE times higher). predicted a much higher risk (i.e., over 103 times higher). NORA Six of the seven RA tools evaluated the animal products was the only tool of those evaluating the wild boar pathway pathway (MINTRISK did not) (Figure 3B). COMPARE, RRAT, that did not predict an increased risk for this pathway in HS1 and NORA predicted a higher incursion risk to the Netherlands to the Netherlands. The addition of a single ASF outbreak at a than Finland for this pathway, although differences in risks were domestic pig farm (HS2) resulted in a slightly increased risk to much smaller than for the live animal trade pathway. IDM the Netherlands for the live animal trade pathway compared to and SVARRA predicted an equivalent risk for both countries, HS1 for MINTRISK, IDM and NORA, and a slightly increased whereas SPARE uniquely predicted that the risk was lower to the risk for both the live animal trade pathway and the animal Netherlands than Finland. products pathway for SPARE, COMPARE and RRAT. SVARRA Again, six of the seven RA tools evaluated the wild boar indicated that the ASF outbreak at a domestic farm in Germany pathway (RRAT did not) (Figure 3C). COMPARE and NORA (HS2) did not further increase the ASF risk to the Netherlands predicted an equivalent, very low to negligible incursion risk compared to HS1. resulting from this pathway to both countries. All other tools had Most RA tools agreed that the presence of ASF in wild boar in a relative risk below 1, which indicates that the predicted ASF risk Germany (HS1) did not increase the risk to Finland compared was higher to Finland than the Netherlands. to the baseline scenario. Only IDM and NORA predicted a slightly increased risk for the live animal trade pathway and Relative Risks Across Scenarios the animal products pathway, respectively. The addition of a For each of the generic RA tools, the ASF incursion risk of both single ASF outbreak at a domestic pig farm (HS2) resulted in hypothetical scenarios was compared to the baseline scenario. a slightly increased risk to Finland for the live animal trade Results for the Netherlands and Finland are given in Figure 4. pathway compared to HS1 for SPARE and IDM, and a slightly A calculated relative risk above 1 indicates that the ASF risk was increased risk for the animal products pathway for SPARE, higher in the hypothetical scenario than in the baseline scenario. COMPARE, and RRAT. MINTRISK and SVARRA indicated Frontiers in Veterinary Science | www.frontiersin.org 8 February 2020 | Volume 7 | Article 56 de Vos et al. Cross-Validation of Generic RA Tools TABLE 4 | Pathways contributing most to the ASF incursion risk for each toola per country and scenario (indicated in red)b . RA TOOL Scenario The Netherlands Finland Live animals Animal products Wild boar Live animals Animal products Wild boar COMPARE Baseline HS1 HS2 RRAT Baseline HS1 HS2 MINTRISK Baseline HS1 HS2 NORAc Baseline HS1 HS2 SVARRAd Baseline HS1 HS2 a Itwas not possible to compare pathways in the SPARE and IDM tools. b The gray cells indicate that this pathway was not evaluated by the tool. c In NORA, the human travel pathway had the highest risk for Finland in all three scenarios; the animal products pathway ranked second. d Equal contribution of several pathways in one scenario. that the presence of ASF in wild boar and/or domestic pigs in qualitative approach when risk estimates are close. The presence Germany did not increase the ASF risk to Finland compared to of ASF in Germany resulted in an increased risk estimate for the the baseline scenario. wild boar pathway in SVARRA, such that all three pathways had an equal risk level in the hypothetical scenarios. MINTRISK was Pathway Contribution the only tool not to consider animal products and concluded that Table 4 presents an overview of the three most commonly from those that were included (trade in live animals and wild investigated pathways in the tools (i.e., trade in live animals, boar movements), wild boar was the pathway associated with the trade in animal products and wild boar movements), indicating highest risk, for both the Netherlands and Finland. the pathway that contributed most to the ASF incursion risk Some of the tools included more pathways than the three to the Netherlands and Finland in the baseline scenario and investigated here (Table 2). When taking into account these the two hypothetical scenarios. Results for SPARE and IDM are additional pathways, only in NORA the pathway ranking top for not included here, because a relative comparison of estimated Finland was changed from trade in animal products to human pathway risks was not possible for these tools. In SPARE the travel for all three scenarios (Supplementary Material 3). units in which the risks are expressed differ over the pathways being numbers of infected animals for the live animal trade DISCUSSION and wild boar pathway and numbers of infected kilograms for the animal products pathway (Supplementary Material 3). In Cross-Validation Based on the ASF Case IDM the semi-quantitative risk scores assigned to each of the Study pathways have not been scaled to allow for a comparison between Validation of generic RA tools is a challenging task for which the pathways. no gold standard is available. In this study, we explored Overall, most tools agreed that (when considered) animal the opportunity to cross-validate seven generic RA tools by products constituted the highest risk to Finland for all three universally applying them to a predefined case study on the scenarios, and to the Netherlands in the baseline scenario. For incursion risk of ASF. Comparison of the absolute results both hypothetical scenarios the animal products pathway was from the tools was not possible for several reasons, including still predicted to have the highest risk to the Netherlands by their differing objectives, endpoints, outputs, and risk pathways COMPARE and RRAT. However, NORA indicated a change in considered. However, by comparing relative risks it was possible risk such that trade in live animals became the pathway with the to cross-validate the generic RA tools across the different highest risk for ASF incursion when the disease was assumed to pathways, between the two countries of interest (Netherlands and be present in Germany. Under all scenarios the SVARRA tool Finland) and between the three scenarios that were evaluated. did not have a single pathway constituting the highest risk to the In general, the tools agreed on the ranking of the target Netherlands resulting from the lack of resolution available with a countries for the pathways evaluated, although the magnitude Frontiers in Veterinary Science | www.frontiersin.org 9 February 2020 | Volume 7 | Article 56 de Vos et al. Cross-Validation of Generic RA Tools of relative risks calculated differed widely (Figure 3). All of evaluated the risk of the live animal trade pathway equal to the the tools estimated that the live animal trade pathway posed animal products pathway for the Netherlands. a (much) higher risk to the Netherlands than Finland in the Differences in calculated relative risks over countries and baseline scenario (2017 situation). The large differences in risk scenarios between the tools can be largely explained from (i) relate to the extremely low or even negligible incursion risk of differences in endpoints considered when evaluating the risk of trade in live animals to Finland rather than to a high risk to ASF incursion, (ii) different quantitative scales on which the the Netherlands (absolute results for each tool are provided in risk estimates were scored, and (iii) differences in the definition Supplementary Material 3). In 2017, only 300 live pigs were of pathways. transported to Finland compared to 1.9 million pigs to the SPARE was the only tool that evaluated the entry of ASF into Netherlands (33). All of the tools which included the animal the target area without considering exposure or first infection of products pathway agreed that the risk of this pathway to the native animals. This has probably contributed to the differences Netherlands was higher than or equal to Finland in the baseline observed between SPARE and the other RA tools when evaluating scenario with the exception of SPARE. This might be explained by the relative risk from the animal products and the wild boar the relatively large amount of pork and pork products imported pathways to the Netherlands compared to Finland in the baseline by Finland from Estonia (33) and SPARE being the only tool scenario (Figure 3). No such differences between SPARE and the not considering exposure, first infection or establishment, having other tools were observed for the live animal trade pathway, entry as an endpoint. Hence, these imports from Estonia, which where entry is more directly linked to exposure, first infection is an ASF-infected country, contributed largely to the estimated and establishment if animals are imported for life. incursion risk by SPARE. For the wild boar pathway, all of the All tools but SVARRA produced numbers to express the tools including this pathway agreed that the risk to Finland was incursion risk of ASF to the Netherlands and Finland, although higher than or equal to the Netherlands, although differences in the quantitative scales differed among the tools (see Table 3). risks between the countries were less pronounced than for the Absolute results from SPARE and MINTRISK could in theory live animal trade pathway. SPARE calculated a high relative risk run from zero to infinity, even though most estimates were of this pathway to Finland if compared to the other tools, which below 1 given the relatively low risk of disease incursion. Again, again might be explained by the fact that this tool had entry as an as SPARE evaluated entry rather than exposure, first infection endpoint. A lot of uncertainty was included in the predictions for or establishment, its results were relatively high for the animal the wild boar pathway, especially in the estimated risk to Finland, products pathway (Supplementary Material 3). Absolute results due to uncertainty on the presence of wild boar in the border from COMPARE, RRAT and NORA were bounded by 0 and 1. region of Finland and Russia and the spatial distribution of ASF Thus, for all these five tools, whether absolute results were given cases in wild boar in Russia. from 0 to 1 or from 0 to infinity, calculated relative risks could run When comparing the hypothetical scenarios to the baseline all from zero to infinity and some tools indeed had extremely high of the RA tools indicated an increased risk to the Netherlands due relative risk scores (Figures 3, 4), especially if the denominator to presence of ASF in wild boar and/or domestic pigs in Germany, was estimated to be negligible or zero. Results from IDM, on especially for the wild boar pathway, whereas most tools agreed the contrary, are semi-quantitative risk scores ranging from 0 that the risk to Finland would stay at the same level or increase to 60 for the overall risk, with maximum scores for individual slightly. However, differences were observed between the tools on pathways being < 60. As risk scores of IDM are discrete values the extent to which the live animal trade pathway contributed to (at an interval of 0.5), calculated relative risks were in general the increased risk to the Netherlands. relatively low compared to those from other tools. Qualitative Investigating the ranking of risk pathways, comparisons could probability levels from SVARRA were converted into quantitative only be made for five of the seven tools, as the risk estimates numbers using a log10 scale. Although this would have allowed for individual pathways in SPARE and IDM were given in the calculated relative risks to vary from 10−4 to 104 with a different units and at a different scale, respectively. Comparisons scale consisting of five levels (ranging from very low to very of rankings were further complicated by the different numbers high), the relative risks estimated by SVARRA never exceeded and types of pathways evaluated by each of the tools. Although 10, since differences in risk were always equal to or less than one Table 4 indicates that most tools agreed that the animal products probability level. pathway constituted the highest risk to both countries in the Differences in calculated relative risks over countries and baseline scenario, this is actually only true for three out of scenarios might also have originated from how pathways were the five tools that could be compared (COMPARE, RRAT, and defined in the tools. Trade in live animals was most uniformly SVARRA), with MINTRISK not having evaluated this pathway defined, although some tools (SPARE, RRAT, IDM) not only and NORA estimating a higher risk for the human travel pathway included trade in domestic livestock animals, but also trade in for Finland (Supplementary Material 3). In NORA, the human horses, pets, and exotic mammals and birds. This will, however, travel pathway includes the risk of ASF incursion via animal not have resulted in differences for the ASF risk, since the only products carried for own consumption. Bringing products from animals affected by ASF are porcine species (including wild boar Estonia was assumed to be common practice in Finland due to and warthogs) (21, 22, 59, 62). Trade in animal products differed easy accessibility of wild boar products in Estonia at low prices, to a larger extent, with some tools only including pork and pork contributing largely to the high estimate of the risk of the human products for human consumption (SPARE, COMPARE), whereas travel pathway to Finland. Lastly, it should be noted that SVARRA other tools also included other products derived from slaughtered Frontiers in Veterinary Science | www.frontiersin.org 10 February 2020 | Volume 7 | Article 56 de Vos et al. Cross-Validation of Generic RA Tools pigs, such as hides and bristles (RRAT, IDM). In IDM, NORA contingency planning. While absolute risk estimates contribute and SVARRA, this pathway even included illegal trade, whereas to the decision on whether any preventive actions are required that was separated out in the other tools. The different definitions or not, relative risk estimates are useful for prioritization of of the animal products pathway might have lead to different risk management options. Prioritization of diseases, pathways estimates of the absolute incursion risk of this pathway by the and/or target areas is an important output of all seven different tools, especially from source countries from which generic RA tools in this study (Table 1). Comparing the imports of fresh and frozen meat are not allowed. Also when results of the tools for the ASF case study indicated that considering the relative risks, the generic tools agreed least on the tools largely agreed upon the direction of the relative the animal products pathway (Figure 3). Although the wild risks and thus on prioritization of countries and scenarios. boar pathway was uniformly defined among the tools to be the All tools concluded that the ASF risk of trade in live animals incursion of ASF due to wild boar movements, this pathway was was lower to Finland than the Netherlands in the baseline modeled differently among the tools, with some tools estimating scenario (2017), and that the risk of wild boar movements the risk based on geographic proximity of infected wild boar to Finland was equal to or higher than the Netherlands. populations at the country level (SPARE, MINTRISK, IDM), Furthermore, all tools concluded that the presence of ASF in whereas one tool modeled the spatial distribution of wild boar Germany (hypothetical scenarios) had little or no impact on and their movements based on habitat suitability (COMPARE). the ASF risk to Finland, but did increase the ASF risk to MINTRISK is the only tool in which pathways are not the Netherlands. Thus, we concluded that the cross-validation predefined. The risk assessor can thus decide upon the pathways contributed to the credibility of the results obtained with the to include in the risk assessment and will preferably include generic RA tools. those that are deemed most important given the transmission Further validation of generic RA tools could be achieved mechanisms of the disease. Although trade in animal products by comparing their outcome for a specific risk question with was considered an important route for ASF, the risk assessors results from bespoke models that were specifically developed decided not to include it because of lack of information on to address this risk question. Although several bespoke models whether animal products were derived from domestic pigs or were developed for ASF in recent years (26–31), results of these wild boar. The other tools did not offer the opportunity to models could not be used directly for comparison purposes distinguish between products from domestic pigs and wild boar as they addressed the ASF incursion risk for different years and included both in the animal products pathway. The asset and/or countries than we did. Updating, re-parameterizing and of having user-defined pathways in MINTRISK thus created re-running these models for the ASF case study was beyond another level of uncertainty in the results of the risk assessment the scope of this study. Two bespoke models were, however, which is beyond parameter and model uncertainty. developed that addressed the same risk question as the generic Differences in (implicit) modeling assumptions between the RA tools, one model assessing the probability of a first ASF generic RA tools might have further contributed to the observed infection in a new territory resulting from trade in live animals differences in the evaluated ASF incursion risk between the tools, and the other model assessing the probability of entry of ASF although it is difficult to predict their impact on the calculated virus in a new territory due to wild boar movements (63). relative risks. For instance, the level of detail at which contact Both models were parameterized to assess the ASF risk to with susceptible animals in the target area was modeled varied the Netherlands and Finland in the 2017 situation and for widely among the tools, with some of them only using an the two hypothetical scenarios with ASF present in Germany. overall probability estimate (RRAT, MINTRISK, IDM, NORA), Results for trade in live animals indicated a higher risk to the while others explicitly modeled how different animal species Netherlands than Finland in the baseline scenario, with the (COMPARE) or livestock farms (SVARRA) would be exposed. calculated relative risk being in the same order of magnitude as Another example is that RRAT and NORA used worst case those of the generic RA tools (Supplementary Material 4). The assumptions to evaluate the ASF incursion risk whereas the other bespoke model for trade in live animals predicted an increased tools used more realistic assumptions. Furthermore, the time risk only to the Netherlands for HS2, i.e., when ASF cases were period for which the risk assessment was performed differed also reported in domestic pigs, with a calculated relative risk among the tools. Most tools used an annual timescale and thus just above 1 if compared to the baseline scenario. Again, this assessed the ASF incursion risk using data from a 1-year period is in agreement with the results from the generic RA tools (here: 2017). However, NORA and SVARRA considered a 3- (Supplementary Material 4). Results of the bespoke model for month period to account for the high-risk period of newly the wild boar pathway indicated an equally negligible risk to the infected territories, when no trade restrictions are in force yet. For Netherlands and Finland in the baseline scenario. In HS1, the risk some tools, decisions on these issues were embedded in the model of this pathway was increased to the Netherlands with no further structure, and could thus be attributed to model uncertainty, increase in HS2. The hypothetical scenarios did not result in an whereas for other tools, decisions were taken by the risk assessor increased risk to Finland for the wild boar pathway. These results performing the ASF case study. are in agreement with results from the generic RA tools, with The ultimate aim of these generic RA tools is to provide risk- none of them predicting an increased risk to Finland for the wild based evidence to support risk managers in making informed boar pathway under the hypothetical scenarios and all of them decisions on reducing the incursion risk of infectious animal but one predicting an equally increased risk under HS1 and HS2 diseases by, e.g., preventive actions, targeted surveillance, and to the Netherlands (Supplementary Material 4). Frontiers in Veterinary Science | www.frontiersin.org 11 February 2020 | Volume 7 | Article 56 de Vos et al. Cross-Validation of Generic RA Tools Application of the tools for a prolonged period might create come with a prefilled database for specific diseases and pathways an opportunity for external validation using field data. Some and can easily be used to perform updates of risk assessments of the generic tools have been up and running for at least 5 for these diseases (SPARE, COMPARE, RRAT, IDM), whereas years now. IDM was first released in 2011 and has intensively other tools do not come with an underlying database and have been used in the UK by Defra and the Scottish Government’s to be filled by the risk assessor (MINTRISK, NORA, SVARRA). Center of Expertise on Animal Disease Outbreaks (EPIC) to For these tools, disease expertise is a prerequisite to perform prioritize their risk levels for incursion of disease at different the risk assessment, whereas some of the prefilled tools mainly times of the year. SVARRA was first used in 2013 and has been require computing expertise. Generally speaking, SVARRA and used for several rapid risk assessments including ASF, BT, LSD, MINTRISK are most flexible as to which pathways and diseases and avian influenza (AI). However, one of the difficulties in to include. SVARRA is very well-suited for rapid risk assessments validating models evaluating the incursion risk of exotic diseases in response to disease events such as the incursion risk of ASF is that the adverse events being modeled have a low probability in wild boar in Belgium in September 2018 (2). MINTRISK was of occurrence resulting in too few data points for validation, even primarily developed to evaluate the incursion risk of vector- when used for a prolonged period. borne diseases. Although ASF can be transmitted by soft ticks of the genus Ornithodoros, these vectors do not seem to play a role Model Uncertainty in ASF virus transmission in Europe (23). Hence, MINTRISK This study clearly illustrates the impact of model uncertainty on is not a preferred tool to evaluate the risk of ASF. Some of the risk assessment results. Although the generic RA tools agreed generic RA tools can rapidly assess the risk of multiple diseases on the direction of the relative risk of ASF to the Netherlands for a target area given the diseases have been included in the and Finland, the magnitudes of these relative risks varied largely, tool (SPARE, COMPARE, RRAT, IDM). Results of these tools especially for the live animal trade and wild boar pathways. The can be used to evaluate the relative risk of ASF compared to the range of results obtained when considering the results of all risk of other notifiable diseases that might threaten the target generic RA tools could be interpreted as an indication of the area on a regular basis. Most of the generic RA tools can break uncertainty included in the risk estimates. The results of the down results to provide more detail on source areas and pathways bespoke models fell well within this uncertainty range. contributing most to the risk or to indicate regions in the target Several methods exist to combine the results of different area most at risk of incursion of a new disease, all of which models that predict similar metrics, such as ensemble modeling, is essential information for disease prevention and surveillance structured decision making and model averaging (64). Although purposes. Communication of results from generic RA tools to widely applied in, for example, weather prediction (65), ensemble risk managers should therefore aim at a proper understanding of modeling is still at its infancy in veterinary epidemiology (64, 66). the risks and the uncertainties involved by indicating underlying Methods like structured decision making and ensemble modeling mechanisms rather than at communicating the absolute value or can only be applied to models that produce similar metrics level of risk predicted. to compare results accross models. It was thus not possible to combine the model outputs of the generic RA tools involved in DATA AVAILABILITY STATEMENT this study to produce an uncertainty distribution of the modeled ASF risk as output parameters represented different endpoints, All datasets generated for this study are included in the varied from qualitative to quantitative, and were expressed in article/Supplementary Material. different units. To make an integrated risk estimate from the generic RA tools possible, output parameters would need to be AUTHOR CONTRIBUTIONS harmonized first. A further impediment to the integration of risk estimates obtained by the generic RA tools is the difference CV and ES initiated and conceptualized the research. RS provided in pathways evaluated by each of the tools. Nevertheless, a harmonized data from Eurostat and WAHIS. RS performed the mere comparison of results of different tools can already be case study in SPARE. RT and ES performed the case study helpful in obtaining a more complete picture of the risk and the in COMPARE. AK, MC, and AB performed the case study in uncertainties involved as illustrated by this study. MINTRISK. RP, MS, and CV performed the case study in RRAT. The choice for one or more generic RA tools to answer a HR performed the case study in IDM. TL and LS-L performed the specific risk question will depend on the primary objective of case study in NORA. CH, KS, and AC performed the case study the risk assessment, the diseases and pathways that need to be in SVARRA. SN developed the bespoke models to evaluate the evaluated, the resources and expertise available, and the timescale ASF incursion risk. CV did the analysis of results and drafted the at which the risk assessment has to be completed (see Tables 1, manuscript. All authors reviewed and edited the manuscript. 2 and Supplementary Material 5). Some of the tools allow for a rapid risk assessment in response to a new disease event FUNDING (NORA, SVARRA), whereas others can be used for a continuous assessment of incursion risks over time making them suitable for horizon scanning (SPARE, COMPARE, RRAT, IDM). The latter This research was funded by the CoVetLab tools could, for instance, be used to monitor the incursion risk consortium (CoVetLab 2017-5) and an EFSA Partnering Grant of ASF for a specific target area by comparing results obtained GP/EFSA/AFSCO/2017/01. RP, MS, and CV also received for previous years with the current situation. Some of the tools funding from the Dutch Ministry of Agriculture, Nature Frontiers in Veterinary Science | www.frontiersin.org 12 February 2020 | Volume 7 | Article 56 de Vos et al. Cross-Validation of Generic RA Tools and Food Quality (KB-21-006-028). Responsibility for the Switzerland), Dr. Sharon Calvin (Canadian Food Inspection information provided is with the authors. Results do not express Agency, Canada), and Dr. Yves van der Stede (European Food EFSA’s view. Safety Authority, Italy) are gratefully acknowledged for reviewing an intermediate report on the ASF case study. ACKNOWLEDGMENTS SUPPLEMENTARY MATERIAL The authors would like to thank Dr. Helen Crooke (Animal and Plant Health Agency, United Kingdom) for insightful comments The Supplementary Material for this article can be found that have helped to improve this work. Prof. Katharina Stärk online at: https://www.frontiersin.org/articles/10.3389/fvets. (Bundesamt für Lebensmittelsicherheit und Veterinärwesen, 2020.00056/full#supplementary-material REFERENCES lumpy skin disease case study. Transbound Emerg Dis. (2019) 66:131–43. doi: 10.1111/tbed.12993 1. Allepuz A, Casal J, Beltrán-Alcrudo D. Spatial analysis of lumpy skin disease 16. Kleijnen JPC. Validation of models: statistical techniques and data availability. in Eurasia—Predicting areas at risk for further spread within the region. In Farrington PA, Nembhard HB, Sturrock DT, Evans GW, editors. Transbound Emerg Dis. (2019) 66:813–822. doi: 10.1111/tbed.13090 Proceedings of the 1999 Winter Simulation Conference. Phoenix, AZ (1999). 2. OIE. World Animal Health Information System. (2019). Available online: p. 647–54. https://www.oie.int/wahis_2/public/wahid.php/Wahidhome/Home (accessed 17. De Vos CJ, Saatkamp HW, Nielen M, Huirne RBM. Sensitivity April 18, 2019). analysis to evaluate the impact of uncertain factors in a scenario 3. SCOPAFF. Standing Committee on Plants, Animals, Food and Feed. tree model for classical swine fever introduction. Risk Anal. (2006) Animal Health – Regulatory Committee – presentations (2019). Available 26:1311–22. doi: 10.1111/j.1539-6924.2006.00816.x online at: https://ec.europa.eu/food/animals/health/regulatory_committee/ 18. EFSA AHAW Panel. Scientific opinion on African swine fever. EFSA J. presentations_en (accessed May 13, 2019). (2015) 13:4163. doi: 10.2903/j.efsa.2015.4163 4. Horst HS, De Vos CJ, Tomassen FHM, Stelwagen J. The economic evaluation 19. EFSA, Depner K, Gortazar C, Guberti V, Masiulus M, More S, et al. Scientific of control and eradication of epidemic livestock diseases. Rev Sci Tech. (1999) report on the epidemiological analyses of African swine fever in the Baltic 18:367–79. doi: 10.20506/rst.18.2.1169 States and Poland (Update September 2016-September 2017). EFSA J. (2017) 5. Thompson D, Muriel P, Russel D, Osborne P, Bromley A, 15:5068. doi: 10.2903/j.efsa.2017.5068 Rowland M, et al. Economic costs of the foot and mouth disease 20. EFSA, Boklund A, Cay B, Depner K, Földi Z, Guberti V, et al. Scientific outbreak in the United Kingdom in 2001. Rev Sci Tech. (2002) report on the epidemiological analyses of African swine fever in the 21:675–87. doi: 10.20506/rst.21.3.1353 European Union (November 2017 until November 2018). EFSA J. (2018) 6. Velthuis AGJ, Saatkamp HW, Mourits MCM, De Koeijer AA, 16:5494. doi: 10.2903/j.efsa.2018.5494 Elbers ARW. Financial consequences of the Dutch bluetongue 21. Sánchez-Vizcaíno JM, Arias Neira M. African swine fever virus. In: serotype 8 epidemics of 2006 and 2007. Prev Vet Med. (2010) Zimmerman JJ, Karriker LA, Ramirez A, Schwartz KJ, Stevenson GW, editors. 93:294–304. doi: 10.1016/j.prevetmed.2009.11.007 Diseases of Swine. 10th ed. Chicester: John Wiley & Sons (2012). p. 396–404. 7. Havelaar AH, van Rosse F, Bucura C, Toetenel MA, Haagsma JA, Kurowicka 22. Sánchez-Vizcaíno JM, Mur L, Martinez-Lopez B. African swine fever: An D, et al. Prioritizing Emerging Zoonoses in The Netherlands. PLoS ONE. epidemiological update. Transbound Emerg Dis. (2012) 59(Suppl. 1):27– (2010) 5:e13965. doi: 10.1371/journal.pone.0013965 35. doi: 10.1111/j.1865-1682.2011.01293.x 8. Roberts H, Carbon M, Hartley M, Sabirovic M. Assessing the risk of disease 23. Sánchez-Vizcaíno JM, Mur L, Gomez-Villamandos JC, Carrasco L. An update introduction in imports. Vet Rec. (2011) 168:447–8. doi: 10.1136/vr.d1784 on the epidemiology and pathology of African swine fever. J Comp Path. 9. ANSES. Hiérarchisation de 103 maladies animales présentes dans les filières (2015) 152:9–21. doi: 10.1016/j.jcpa.2014.09.003 ruminants, équidés, porcs, volailles et lapins en France métropolitaine. 24. Arias M, Jurado C, Gallardo C, Fernández-Pinero J, Sánchez-Vizcaíno JM. Avis de l’Anses. Rapport d’expertise collective. Maisons-Alfort (2012). Gaps in African swine fever: Analysis and priorities. Transbound Emerg Dis. Available online at: https://www.anses.fr/fr/system/files/SANT2010sa0280Ra. (2018) 65 (Suppl. 1):235–47. doi: 10.1111/tbed.12695 pdf (accessed May 13, 2019). 25. Guinat C, Vergne T, Jurado-Diaz C, Sánchez-Vizcaíno JM, Dixon L, Pfeiffer 10. De Vos CJ, Van Roermund HJW, De Koeijer AA, Fischer EAJ. Risk assessment DU. Effectiveness and practicality of control strategies for African swine fever: of seven emerging vector-borne animal diseases for The Netherlands: a what do we really know? Vet Rec. (2017) 180:97. doi: 10.1136/vr.103992 structured approach. In: Lindberg A, Thulke HH, editors. Society for 26. Mur L, Martínez-López B, Sánchez-Vizcaíno JM. Risk of African swine fever Veterinary Epidemiology and Preventive Medicine. Proceedings of a meeting introduction into the European Union through transport-associated routes: held in Elsinore. Denmark (2016). p. 215–28. returning trucks and waste from international ships and planes. BMC Vet Res. 11. EFSA. EFSA Webinar: Rapid Risk Assessment Tools for Animal Disease (2012) 8:149. doi: 10.1186/1746-6148-8-149 Outbreaks. (2017). Available online: http://www.efsa.europa.eu/en/events/ 27. Mur L, Martinez-Lopez B, Martinez-Aviles M, Costard S, Wieland B, Pfeiffer event/171127 (accessed March 6, 2019). DU, et al. Quantitative risk assessment for the introduction of African swine 12. Kyyrö J, Sahlström L, Lyytikäinen T. Assessment of the risk of African fever virus into the European Union by legal import of live pigs. Transbound swine fever introduction into Finland using NORA – a rapid tool for Emerg Dis. (2012) 59:134–44. doi: 10.1111/j.1865-1682.2011.01253.x semiquantitative assessment of the risk. Transbound Emerg Dis. (2017) 28. Costard S, Jones BA, Martínez-López B, Mur L, de la Torre A, Martínez 64:2113–25. doi: 10.1111/tbed.12633 M, et al. Introduction of African Swine Fever into the European Union 13. Roelandt S, Van der Stede Y, D’hondt B, Koenen F. The assessment through Illegal Importation of Pork and Pork Products. PLoS ONE. (2013) of African swine fever risk to Belgium early 2014 using the quick and 8:e61104. doi: 10.1371/journal.pone.0061104 semiquantitative Pandora screening tool. Transbound Emerg Dis. (2017) 29. De la Torre A, Bosch J, Iglesias I, Muñoz MJ, Mur L, Martínez-López 64:237–49. doi: 10.1111/tbed.12365 B, et al. Assessing the risk of African swine fever introduction into the 14. Simons RRL, Horigan V, Ip S, Taylor RA, Crescio MI, Maurella C, et al. European Union (EU) by wild boar. Transbound Emerg Dis. (2015) 62:272– A spatial risk assessment model framework for incursion of exotic animal 9. doi: 10.1111/tbed.12129 disease into the European Union Member States. Microbial Risk Anal. (2019) 30. Mur L, Martínez-López B, Costard S, De la Torre A, Jones BA, 13:100075. doi: 10.1016/j.mran.2019.05.001 Martínez M, et al. Modular framework to assess the risk of African 15. Taylor RA, Berriman ADC, Gale P, Kelly LA, Snary EL. A generic swine fever virus entry into the European Union. BMC Vet Res. (2014) framework for spatial quantitative risk assessments of infectious diseases: 10:145. doi: 10.1186/1746-6148-10-145 Frontiers in Veterinary Science | www.frontiersin.org 13 February 2020 | Volume 7 | Article 56 de Vos et al. Cross-Validation of Generic RA Tools 31. Herrera-Ibatá DM, Martínez-Lopez B, Quijada D, Burton K, Mur 52. EC (European Commission). Animal Disease Notification System (ADNS). L. Quantitative approach for the risk assessment of African swine (2019). Available online at: https://ec.europa.eu/food/animals/animal- fever and Classical swine fever introduction into the United States diseases/not-system_en (accessed April 18, 2019). through legal imports of pigs and swine products. PLoS ONE. (2017) 53. FAOSTAT. FAO Global Statistical Yearbook, FAO Regional Statistical 12:e0182850. doi: 10.1371/journal.pone.0182850 Yearbooks. (2018). Available online at: http://www.fao.org/faostat/en/#data/ 32. CBS. Statline. (2019). Available online at: https://opendata.cbs.nl/statline/#/ QA (accessed February 16, 2018). CBS/en/ (accessed May 29, 2019). 54. Evira. Afrikkalaisen sikaruton mahdollisia maahantuloreittejä – riskiprofiilin 33. Eurostat. Comext Bulk Download. (2019). Available online at: https://ec. päivitys 2017 (Possible routes of entry into the country for African swine fever – europa.eu/eurostat/estat-navtree-portlet-prod/BulkDownloadListing?sort= an updated risk profile 2017). Evira Research reports 4/2011 (2017). Available 1&dir=comext (accessed April 19, 2019). online at: https://www.ruokavirasto.fi/globalassets/tietoa-meista/julkaisut/ 34. De Vos CJ, Taylor RA, Simons RRL, Roberts H, Hultén C, De julkaisusarjat/julkaisuja/elaimet/paivitetty_2017_eviran_tutkimuksia_4_ Koeijer AA, et al. Generic approaches for risk assessment of infectious 2011.pdf (accessed June 3, 2019). animal disease introduction (G-RAID). EFSA Support Publ. (2019). 55. Nature Resources Institute Finland. Arvio Suomen villisikakannan doi: 10.2903/sp.efsa.2019.EN-1743 koosta tammikuussa 2018 (Estimation of wild boar population size in 35. Simons RRL, Horigan V, De Nardi M, Ru G, Pena AE, Adkin A. Mighty Finland in January 2018). (2018) Available online at: http://wordpress1. models from little data grow: Estimating animal disease prevalence. In: luke.fi/riistahavainnot-hirvielaimet/wp-content/uploads/sites/5/2018/02/ Nielsen LR, Lindberg A, editors. Society for Veterinary Epidemiology and Villisikakanta2018.pdf (accessed June 3, 2019). Preventive Medicine. Proceedings of a meeting held in Inverness. Scotland 56. VWV. Jaarrapportage grofwild 2017–2018. Nieuwsbrief Nr. 44. (2017). p. 166–75. Vereniging Wildbeheer Veluwe (2018). Available online at: http://www. 36. Taylor RA, Gale P, Kelly LA, Snary EL. One for all: a generic risk verenigingwildbeheerveluwe.nl/Portals/24/Downloads/Jaarverslagen/VWV framework for all pathways of disease introduction. In: The 15th International %20Jaarverslag%202018.pdf (accessed October 17, 2018). Symposium of Veterinary Epidemiology and Economics. Chiang Mai (2018). 57. Gabriel C, Blome S, Malogolovkin A, Parilov S, Kolbasov D, Teifke JP, et al. p. 650. Characterization of African swine fever virus Caucasus isolate in European 37. Taylor RA, Podgórski T, Simons RRL, Ip S, Gale P, Kelly LA, et al. Predicting wild boars. Emerg Infect Dis. (2011) 17:2342–5. doi: 10.3201/eid1712. spread and effective control measures for African swine fever– should we 110430 blame the boars? bioRxiv. (2019) 654160. doi: 10.1101/654160 58. Gulenkin VM, Korennoy FI, Karaulov AK, Dudnikov SA. Cartographical 38. Taylor RA, Condoleo R, Simons RRL, Gale P, Kelly LA, Snary EL. The risk analysis of African swine fever outbreaks in the territory of the Russian of infection of African swine fever virus in European swine through boar Federation and computer modeling of the basic reproduction ratio. Prev Vet movement and legal trade of pigs and pig meat. Front Vet Sci. (2020) 6:486. Med. (2011) 102:167–74. doi: 10.1016/j.prevetmed.2011.07.004 doi: 10.3389/fvets.2019.00486 59. Gallardo MC, De la Torre Reoyo A, Fernández-Pinero J, Iglesias I, Munoz MJ, 39. De Vos CJ, Petie R, Van Klink E, Swanenburg M. Rapid risk assessment of Arias ML. African swine fever: a global view of the current challenge. Porcine exotic animal disease introduction. In: The 15th International Symposium of Health Manage. (2015) 1:21. doi: 10.1186/s40813-015-0013-y Veterinary Epidemiology and Economics. Chiang Mai (2018). p. 253. 60. Guinat C, Gogin A, Blome S, Keil G, Pollin R, Pfeiffer DU, et al. Transmission 40. De Vos C, Hoek M, Fischer E, De Koeijer A, Bremmer J. Risk Assessment routes of African swine fever virus to domestic pigs: current knowledge and Framework for Emerging Vector-Borne Livestock Diseases. Report 11-CVI0168 future research directions. Vet Rec. (2016) 178:262. doi: 10.1136/vr.103593 (2012). Available online at: http://edepot.wur.nl/198115 (accessed June 4, 61. Probst C, Globig A, Knoll B, Conraths FJ, Depner K. Behaviour of 2019). free ranging wild boar towards their dead fellows: potential implications 41. EFSA AHAW Panel. Scientific opinion on vector-borne diseases. EFSA J. for the transmission of African swine fever. R Soc Open Sci. (2017) (2017) 15:4793. doi: 10.2903/j.efsa.2017.4793 4:170054. doi: 10.1098/rsos.170054 42. OIE. Handbook on Import Risk Analysis for Animals and Animal Products. 2nd 62. Costard S, Mur L, Lubroth J, Sanchez-Vizcaino JM, Pfeiffer DU. ed. Paris: World Organization for Animal Health (2010). p. 88. Epidemiology of African swine fever virus. Virus Res. (2013) 43. Vose D. Risk Analysis. A Quantitative Guide. 3rd ed. Chicester: John Wiley & 173:191–7. doi: 10.1016/j.virusres.2012.10.030 Sons (2008). p. 735. 63. Napp S. G-RAID Case Study Scenarios Using a Specific (Non-Generic) Model. 44. Keeling M, Rohani P. Modeling Infectious Diseases in Humans and Poster Presented at the G-RAID Symposium. Amsterdam (2019). Available Animals. Princeton, NJ: Princeton University Press (2008). p. 366. online at: https://www.wur.nl/upload_mm/b/a/3/441d2e1b-a9aa-4c6b-871f- doi: 10.1515/9781400841035 6283bd164313_IRTA.pdf (accessed May 29, 2019). 45. EC (European Commission). TRACES: TRAde Control and Expert System. 64. Webb CT, Ferrari MF, Lindström T, Carpenter T, Dürr S, Garner (2019). Available online at: https://ec.europa.eu/food/animals/traces_en G, et al. Ensemble modelling and structured decision-making to (accessed April 18, 2019). support Emergency Disease Management. Prev Vet Med. (2017) 46. UN. UN Comtrade Database, United Nations. (2019). Available online at: 138:124–33. doi: 10.1016/j.prevetmed.2017.01.003 https://comtrade.un.org/ (accessed April 19, 2019). 65. Bauer P, Thorpe A, Brunet G. The quiet revolution of numerical weather 47. Eurostat. European Statistics. (2019). Available online at: https://ec.europa.eu/ prediction. Nature. (2015) 525:47–55. doi: 10.1038/nature14956 eurostat/data/database (accessed April 19, 2019). 66. Probert WJM, Shea K, Fonnesbeck CJ, Runge MC, Carpenter TE, Dürr S, 48. EC (European Commission). The Combined Nomenclature. (2019). et al. Decision-making for foot-and-mouth disease control: objectives matter. Available online at: https://ec.europa.eu/taxation__customs/business/ Epidemics. (2016) 15:10–9. doi: 10.1016/j.epidem.2015.11.002 calculation-customs-duties/what-is-common-customs-tariff/combined- nomenclature__en (accessed April 18, 2019). Conflict of Interest: The authors declare that the research was conducted in the 49. EC (European Commission). Commission Implementing Regulation (EU) absence of any commercial or financial relationships that could be construed as a 2018/1602 of 11 October 2018 amending Annex I to Council Regulation potential conflict of interest. (EEC) No 2658/87 on the tariff and statistical nomenclature and on the Common Customs Tariff. Off J Eur Union. (2018) L273:1–960. Copyright © 2020 de Vos, Taylor, Simons, Roberts, Hultén, de Koeijer, Lyytikäinen, 50. Alexander NS, Massei G, Wint W. The European Distribution of Sus Scrofa. Napp, Boklund, Petie, Sörén, Swanenburg, Comin, Seppä-Lassila, Cabral and Snary. Model Outputs from the Project Described within the Poster – Where are All This is an open-access article distributed under the terms of the Creative Commons the Boars? An attempt to gain a continental perspective. Open Health Data. Attribution License (CC BY). The use, distribution or reproduction in other forums (2016) 4:e1. doi: 10.5334/ohd.24 is permitted, provided the original author(s) and the copyright owner(s) are credited 51. FAO. EMPRES Global Animal Disease Information System (EMPRES-i). and that the original publication in this journal is cited, in accordance with accepted (2019). Available online: http://empres-i.fao.org/eipws3g/ (accessed April 18, academic practice. No use, distribution or reproduction is permitted which does not 2019). comply with these terms. Frontiers in Veterinary Science | www.frontiersin.org 14 February 2020 | Volume 7 | Article 56