IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 40, NO. 12, DECEMBER 2021 3543 Multi-Centre, Multi-Vendor and Multi-Disease Cardiac Segmentation: The M&Ms Challenge Víctor M. Campello , Polyxeni Gkontra , Cristian Izquierdo, Carlos Martín-Isla, Alireza Sojoudi, Peter M. Full , Klaus Maier-Hein, Yao Zhang , Zhiqiang He, Jun Ma , Mario Parreño , Alberto Albiol , Fanwei Kong, Shawn C. Shadden , Jorge Corral Acero , Vaanathi Sundaresan , Mina Saber, Mustafa Elattar , Hongwei Li , Bjoern Menze, Firas Khader, Christoph Haarburger, Cian M. Scannell , Mitko Veta , Adam Carscadden, Kumaradevan Punithakumar , Senior Member, IEEE, Xiao Liu, Sotirios A. Tsaftaris , Xiaoqiong Huang, Xin Yang , Lei Li, Xiahai Zhuang , David Viladés , Martín L. Descalzo , Andrea Guala , Lucia La Mura , Matthias G. Friedrich, Ria Garg , Julie Lebel, Filipe Henriques, Mahir Karakas, Ersin Çavuş, Steffen E. Petersen , Sergio Escalera , Santi Seguí , José F. Rodríguez-Palomares , and Karim Lekadir Abstract — The emergence of deep learning has consider- I. I NTRODUCTION ably advanced the state-of-the-art in cardiac magnetic res- onance (CMR) segmentation. Many techniques have been proposed over the last few years, bringing the accuracy of automated segmentation close to human performance. A CCURATE segmentation of cardiovascular magnetic res- onance (CMR) images is an important pre-requisite in clinical practice to reliably diagnose and assess a number of However, these models have been all too often trained and major cardiovascular diseases [1], [2]. Currently, the process validated using cardiac imaging samples from single clin- typically requires the clinician to provide a significant amount ical centres or homogeneous imaging protocols. This has of manual input and correction to accurately and consistently prevented the development and validation of models that are generalizable across different clinical centres, imaging annotate the cardiac boundaries across all image slices and conditions or scanner vendors. To promote further research cardiac phases. The automation of such a tedious and time- and scientific benchmarking in the field of generalizable consuming task has been pursued for a long time by using mul- deep learning for cardiac segmentation, this paper presents tiple approaches, such as statistical shape models [3] or cardiac the results of the Multi-Centre, Multi-Vendor and Multi- atlases [4]. In the last few years, the advent of the deep Disease Cardiac Segmentation (M&Ms) Challenge, which was recently organized as part of the MICCAI 2020 Confer- learning paradigm has motivated the development of many ence. A total of 14 teams submitted different solutions to the neural network based techniques for improved CMR segmen- problem, combining various baseline models, data augmen- tation, as listed in a recent review [5]. However, most of these tation strategies, and domain adaptation techniques. The techniques have been all too often trained and evaluated using obtained results indicate the importance of intensity-driven cardiac imaging samples collected from single clinical centres data augmentation, as well as the need for further research using similar imaging protocols. While these works have to improve generalizability towards unseen scanner ven- dors or new imaging protocols. Furthermore, we present a advanced the state-of-the-art in deep learning based cardiac new resource of 375 heterogeneous CMR datasets acquired image segmentation, their high performances were reported on by using four different scanner vendors in six hospitals samples with relatively homogeneous imaging characteristics. and three different countries (Spain, Canada and Germany), As an example, the CMR datasets from the Automated which we provide as open-access for the community to Cardiac Diagnosis Challenge (ACDC) dataset [6] have been enable future research in the field. extensively used to build and test new implementations of Index Terms — Cardiovascular magnetic resonance, deep neural networks for cardiac image segmentation. The image segmentation, deep learning, generalizability, data top performing technique in the ACDC challenge, proposed augmentation, domain adaption, public dataset. by Isensee et al. [7], obtained a very high segmentation Manuscript received March 26, 2021; revised June 8, 2021; accepted accuracy for both the left and right ventricles. However, June 11, 2021. Date of publication June 17, 2021; date of current version the ACDC datasets were compiled from 150 subjects scanned November 30, 2021. This work was supported in part by the European at a single clinical centre using the same imaging protocol, Union’s Horizon 2020 Research and Innovation Program (euCan- SHare Project) under Grant 825903. (Corresponding author: which limits the ability of the researchers to develop and test Víctor M. Campello.) models that can generalize suitably across multiple centres This work involved human subjects or animals in its research. The and scanner vendors. Other researchers attempted to encode authors confirms that all human/animal subject research procedures and protocols are exempt from review board approval. higher variability by building and testing their models based Please see the Acknowledgment section of this article for the author on much larger datasets obtained from the UK Biobank [8]. affiliations. For instance, Bai et al. [9] implemented a fully convolutional This article has supplementary downloadable material available at https://doi.org/10.1109/TMI.2021.3090082, provided by the authors. network that achieved highly accurate results on this large Digital Object Identifier 10.1109/TMI.2021.3090082 dataset (over 4,875 cases), but the authors concluded that their This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ 3544 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 40, NO. 12, DECEMBER 2021 model might not generalize well to other vendor or sequence TABLE I datasets. I NFORMATION F ROM C ENTRES I NCLUDED IN T HIS W ORK Some researchers proposed to improve CMR segmenta- tion by training neural networks with images from multiple cohorts [10], [11], but these works do not include methods for addressing domain shifts between training and new unseen cohorts. Other works used data augmentation on models built from single cohorts such as the ACDC [12] or the UK Biobank [13], then tested their techniques on other existing public cohorts, including the Sunnybrook Cardiac Data [14], LV Segmentation Challenge Dataset (LVSC) [15] or RV Segmentation Challenge Dataset (RVSC) [16]. However, these studies are limited by the fact that these different CMR cohorts have been annotated with distinct standard operating procedures (SOPs), which makes it difficult to draw con- clusions from the multi-cohort comparative results. Further- more, such an approach requires a large training dataset from the single centre to model high variability across subjects. Another multi-centre and multi-vendor study conducted by Tao et al. [11] relied solely on private data, which makes it difficult to replicate the results and perform community- driven benchmarking. While these recent works confirmed the difficulties encountered by deep learning models to generalize beyond the training samples, they also support the need for well-defined heterogeneous public datasets that can be used by the community to improve model generalizability through scientific benchmarking. In this context, the Multi-Centre, Multi-Vendor and Multi- Disease Cardiac Segmentation (M&Ms) Challenge was pro- Fig. 1. Visual appearance of a CMR short axis middle slice for posed and organized as part of the Statistical Atlases and anatomically similar subjects in the four different vendors considered. Computational Modelling of the Heart (STACOM) Workshop, held in conjunction with the MICCAI 2020 Conference. The TABLE II M&Ms challenge was set up as part of the euCanSHare inter- D ISTRIBUTION OF THE M OST F REQUENT PATHOLOGIES AND H EALTHY VOLUNTEERS B ETWEEN C ENTRES . T HE A BBREVIATIONS national project,1 which is aimed at developing interoperable C ORRESPOND TO H YPERTROPHIC C ARDIOMYOPATHY (HCM), D ILATED data sharing and analytics solutions for multi-centre cardiovas- C ARDIOMYOPATHY (DCM), H YPERTENSIVE H EART D ISEASE (HHD), cular research data. Together with clinical collaborators from A BNORMAL R IGHT V ENTRICLE (ARV), ATHLETE H EART six different hospitals in Spain, Canada and Germany, a public S YNDROME (AHS), I SCHEMIC H EART D ISEASE (IHD), CMR dataset was established from 375 participants, scanned AND L EFT V ENTRICLE N ON -C OMPACTION (LVNC) with four different scanners (Siemens, Philips, General Electric (GE) and Canon) and annotated using a consistent contouring SOP across centres. To our knowledge, this dataset is the most diverse resource of CMR studies, which is provided as open-access2 to promote further research and scientific benchmarking in the devel- opment and evaluation of future generalizable deep learning models in cardiac image segmentation. In this paper, we also present and discuss the results of the M&Ms challenge in detail, to which a total of 14 international teams submitted a range of solutions, including different strategies of transfer learning, domain adaptation and data augmentation, to accom- modate for the differences in scanner vendors and imaging translated reliably and deployed in routine clinical practice protocols. The obtained results show the extent of the problem, across the globe. the promise of the proposed solutions, as well as the need for further research to build fully generalizable tools that can be II. C HALLENGE F RAMEWORK A. Data Preparation 1 euCanSHare project website: www.eucanshare.eu A total of six clinical centres from Spain, Canada and 2 The dataset is publicly available at www.ub.edu/mnms Germany (numbered 1 to 6 in this work) contributed to this CAMPELLO et al.: MULTI-CENTRE, MULTI-VENDOR AND MULTI-DISEASE CARDIAC SEGMENTATION 3545 TABLE III AVERAGE S PECIFICATIONS FOR THE I MAGES ACQUIRED IN THE D IFFERENT C ENTRES challenge by providing a different number of CMR studies TABLE IV from different scanner vendors, as detailed in Table I. In total, N UMBER OF S TUDIES FOR E ACH S TEP OF THE C HALLENGE P RESENTED BY C ENTRE AND S CANNER V ENDOR 375 studies were included in this challenge. The subjects considered for this multi-disease study were selected among groups of various cardiovascular diseases, such as hypertrophic cardiomyopathy, dilated cardiomyopathy, coronary heart dis- ease, abnormal right ventricle, myocarditis and ischemic car- diomyopathy as well as healthy volunteers (see Table II for more details on the distribution of these cases). The specific scanner manufacturers are: 1) Siemens (Siemens Healthi- neers, Germany), 2) Philips (Philips Healthcare, Netherlands), 3) General Electric (GE, GE Healthcare, USA) and 4) Canon (Canon Inc., Japan). These four manufacturers were coded as A, B, C and D during the challenge, respectively. The CMR images derived from these four vendors are illustrated in Fig. 1. More specific details on the studies are given in Table III. Every CMR study was annotated manually by an expert clinician from the centre of origin, with experiences ranging from 3 to more than 10 years. Following the clinical protocol, short-axis views were annotated at the end-diastolic (ED) and end-systolic (ES) phases, as they correspond to the phases used to compute the relevant clinical biomarkers for cardiac diagnosis and follow-up. Three main regions were considered: the left and right ventricle (LV and RV, respectively) cavities and the left ventricle myocardium (MYO). In order to reduce Fig. 2. Degree of generalizability of models trained from the four vendors. the inter-observer and inter-centre variability in the contours, Four 2D UNet models [17] were trained with datasets from the four vendors separately (rows) and subsequently tested their segmentation in particular at the apical and basal regions, a detailed revi- performance on datasets from all vendors (columns). The heatmap sion of the provided segmentations was performed by four shows the Dice similarity coefficient, with a color scale that goes from blue researchers in pairs. They applied the same SOP across all (good generalizability) to red (poor generalizability). The results are the average of 5 models cross-validated on subsets of 30 training subjects. CMR datasets to obtain the final ground truth. To generate consistent annotations for the research community, we chose to apply the SOP that was already used by the ACDC challenge, the NIFTI format, representing the final files delivered to the as follows: challenge participants. a) The LV and RV cavities must be completely covered, including the papillary muscles. b) No interpolation of the MYO boundaries must be per- B. Model Training formed at the basal region. The 375 CMR studies were divided into three sets, namely c) The RV must have a larger surface at the ED time-frame training, validation and testing, as detailed in Table IV. compared to ES. To decide on a particular subdivision, we first estimated the d) The RV does not include the pulmonary artery. degree of generalizability of models trained from the four ven- Clinical delineations as well as later corrections were dors, as shown in Figure 2. We have thus decided to combine performed using CVI42 software (Circle Cardiovascular Imag- the datasets from vendors A, which generalize relatively well, ing Inc., Calgary, Alberta, Canada). All studies were pro- with datasets from B, which generalize poorly to new vendors, vided in DICOM format and contours were extracted in as training datasets. The participants received the 175 training cvi42 workspace format (.cvi42ws). An in-house software was cases on 1st May 2020, including 75 annotated CMRs from then used to extract the contours and transform the images into vendor A, 75 annotated CMRs from vendor B, 25 CMRs from 3546 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 40, NO. 12, DECEMBER 2021 vendor C but without any annotations (only the raw images) (iv) Hausdorff distance (HD):   and no datasets from vendor D, in order to test generalizability to different situations (e.g. image protocol included or not H D(P, G) = max sup d( p, G), sup d(g, P) (4) included in the training). Note that in the case of vendor A, p∈ P g∈G the 75 CMRs were included from centre 1 but none from that measures the largest disagreement between the volumes centre 6, to test generalizability across vendors but also across and it is useful for identifying small outliers. All these metrics centres for the same vendors. Regarding vendor B, we included were computed using the public library medpy.5 more training datasets from centre 2 (50 cases) than from These metrics were computed for the three target labels: centre 3 (25 cases) to assess the impact of imbalanced training LV, RV, and MYO, resulting in a total of 12 measures. In case data and fairness in multi-centre cardiac image segmentation. one participant had a prediction missing for a specific subject, For optimizing the models, the participants were allowed to a value of zero was assumed for DSC and JI and maximum remotely validate against 40 additional CMRs, i.e. 10 from values of 150 and 50 milimetres were assumed for HD and each of the four vendors. A maximum of 7 submissions ASSD, respectively, based on the worst results obtained by were allowed per team during the validation process. Note the participating methods. Any value above the thresholds on that during training, it was not allowed to use any external surface distances was set to the maximum value. datasets or pre-trained models, to enable a fair comparison To obtain the final ranking for each team, a weighted between the proposed solutions. average was computed giving a greater importance to the unlabelled and unseen scanner vendors. Therefore, if v A and C. Model Evaluation v B are defined as the labelled vendors, v C , the unlabelled one The testing period for the challenge started on 8th and v D , the unseen one, the weighted sum for a metric M is June 2020 and concluded on 15th July 2020. The participants obtained as follows: had to evaluate their models remotely to ensure the unseen 1 1 1 1 datasets were totally hidden from the segmentation methods. M = Mv A + Mv B + Mv C + Mv D (5) 6 6 3 3 As such, for example, the participants had no prior information Then, a min-max normalization was applied across participants on the images provided by vendor D. In order to evaluate for each measure and a final average over the normalized the models, the participants were asked to build a Singularity metrics yielded the performance (P) ranging from 0 to 1, image3 and share it with the organizers via a MEGA4 folder being 1 the value that a team would obtain if it had the best shared by the organizers or by any other secure cloud storage results for every metric. service. This Singularity image allows its execution on a similar architecture machine without the need to install all the III. PARTICIPATING M ETHODS diversity of used libraries. The necessary computing power was sponsored by NVIDIA, who provided the organizers with In total, 80 teams registered to download the M&Ms training access to an NVIDIA V100 GPU card with 16GB of memory, dataset, 16 submitted a solution for the final testing phase as well as the Barcelona Supercomputing Center (BSC) who and 14 teams submitted their methodology as a paper to the provided access to two K80 NVIDIA GPU cards. STACOM Workshop (see Table V for details on these teams). In order to assess the quality of the automatically segmented All participants used deep learning as their segmentation masks P with respect to the ground truth G, four measures approach. Table VI summarizes the main characteristics of were proposed, namely: the submitted techniques, including the backbone architectures (i) Dice similarity coefficient (DSC): and domain adaptation strategies, which are described in more 2|P ∩ G| detail in the following subsections. Furthermore, details on DSC(P, G) = (1) the hardware used during training and the times that each |P| + |G| method took for training and inference as well as the number that measures the degree of overlapping of two volumes. of parameters for each model are presented in Table VII. (ii) Jaccard index (JI): |P ∩ G| |P ∩ G| A. Backbone Architectures J I (P, G) = = (2) |P ∪ G| |P| + |G| − |P ∩ G| There is a degree of variability in the backbone architectures that measures overlapping as well but is more sensitive to used between the different participants, as shown in Table VI. results with average performance. Four teams used the nnUNet [33] (which includes UNet (iii) Average symmetric surface distance (ASSD): architectures in 2D and 3D as well as a cascaded UNet) ⎛ ⎞   as their baseline segmentation model (P1-P3 & P9). Four 1 ASS D(P, G) = ⎝ d( p, G) + d(g, P)⎠ participants used a traditional UNet [17] (P6, P10, P13, P14), |P| + |G| while other variants of UNets were adopted by the rest p∈P g∈G d( p, G) := inf d( p, g) (3) of the teams. In particular, UNets combined with residual g∈G connections were applied by three teams (P4, P8, P11), that measures the average distance between the two volumes. with P8 preferring a residual UNet with dilated convolutions (DRUNet) [34]. P5 proposed the use of an attention UNet [35], 3 https://sylabs.io 4 https://mega.nz 5 https://github.com/loli/medpy CAMPELLO et al.: MULTI-CENTRE, MULTI-VENDOR AND MULTI-DISEASE CARDIAC SEGMENTATION 3547 TABLE V L IST AND D ETAILS OF THE PARTICIPATING T EAMS IN THE C HALLENGE TABLE VI C HARACTERISTICS OF PARTICIPATING M ODELS . A BBR : R OTATIONS (R), F LIPPING (F), S CALING (S), D EFORMATIONS (D), H ISTOGRAM M ATCHING (HM), G AUSSIAN N OISE (GN), B RIGHTNESS (B), G AMMA (G), T EST T IME AUGMENTATION (TTA) TABLE VII As pre-processing techniques, all models that provided T RAINING AND I NFERENCE T IME , AND H ARDWARE U SED, FOR A LL detailed information about this step performed either image PARTICIPATING M ETHODS . H , M , S AND M IL . S TAND FOR H OURS , M INUTES , S ECONDS AND M ILLIONS , R ESPECTIVELY normalization to a unit Gaussian distribution or pixel value rescaling to the range [0,1] (only P6 chose the range [0,255] instead). With regards to image resolution, images were resized based on target size or pixel resolution values in 10 out of 14 methods, while the other methods preferred to keep the original image resolution (P4, P7, P8, P11). In order to obtain squared images, cropping and zero padding were used depending on the desired image size for each case. Additionally, some methods applied intensity clipping between varying ranges to get rid of bright artifacts (P5, P6, P11). Finally, P8 was the only method to apply also a non-local means denoising filter prior to the training process. B. Data Augmentation All participants in the challenge (except P11) used some form of data augmentation to enhance their models. Specif- ically, two families of data augmentations were considered: while P7 developed a modified UNet based on multi-gate and (1) spatial transformations to increase sample size through dilated inception blocks to extract multi-scale features. Lastly, rotation, flipping, scaling or deformation of the original one team (P12) proposed a modified Spatial Decomposition images; (2) intensity-driven techniques, which maintain the Network (SDN) [36] with an AdaIN [37] decoder. spatial configuration of the anatomical structures but modify 3548 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 40, NO. 12, DECEMBER 2021 classification loss is high while the segmentation loss is low, generating features that are robust to vendor-specific variations as well as optimal for segmentation. IV. R ESULTS As shown in Table IV, a balanced dataset across the four vendors was prepared for evaluating the final submissions (40 CMRs per vendor, total 160 datasets). In this section, we analyze the obtained results per (1) team, (2) vendor, (3) clinical center, and (4) show some qualitative results. For analysing the obtained results, we also implemented two Fig. 3. The effect of data augmentation on a single CMR slice. In the baseline models to better appreciate the added value of the top row, the original image and spatial augmentations are shown. In the data augmentation and domain adaptation techniques used in bottom row, intensity-based augmentations. this challenge: B1: A 2D UNet without any data augmentation as described in the original reference [17], trained with weighted their image appearance. The second type of augmentation cross entropy loss. seems particularly relevant for the M&Ms as it may increase B2: The nnUNet pipeline, with a 2D UNet module and the variability in image appearance, with the hypothesis that default parameters as given in [33] (the best fold accord- this may lead to improved adaptation to varying imaging ing to the validation set was selected). protocols and scanner vendors. Two teams performed data aug- In particular, B2 differed from those in P1-P3 in that it only mentation using only spatial transformations (P4, P6). Eleven included one architecture type [2D UNet] and ±180 degrees teams additionally implemented intensity-based transforma- rotations, flippings, scalings, deformations, gamma transfor- tions using one of two main approaches: (i) standard image mations and test-time augmentation as data augmentation. transformations such as histogram matching, blurring, change In contrast, P1, P2 and P3 methods included further augmen- in brightness, gamma and contrast, or addition of Gaussian tation techniques such as histogram matching, noise addition, noise (P1-P3, P7-P8, P10, P13) (see 3 for a visualization brightness modification, contrast modification and pseudo- of a subset of these transformations on a training slice); label generation by label propagation in time space. (ii) advanced image synthesis by using generative adversarial networks (GANs) (P5, P8, P14) or variational auto-encoders (VAE) (P12). For the latter one, the generation of synthetic A. Analysis per Team images for the unseen vendor D is not feasible since it was Fig. 4 displays the results of the challenge for all partici- not included in the training. Note that the majority of the pants and according to two evaluation metrics (DSC and HD). teams participating in the challenge (10 out of 14) relied solely It can be seen that the curves are flat for about half of the on data augmentation of the training sample to address the participating teams, which indicates comparable performances domain-shift problem posed by the M&Ms challenge. overall. Note that these methods (P1 to P7) are also the Additionally, some teams (P1-P3, P9, P13) applied test- ones that performed better than the baseline methods and time augmentation techniques, which consist of passing to the we hypothesize that the other models (P8 to P14) suffered model two or more transformed versions of the same inference from some form of over-fitting (see also the shapes of the image to obtain several predictions. These predictions are then curves in Fig. 4). Team P1 provided the most consistent combined to obtain one final outcome, usually by averaging results across all metrics. However, the difference with respect them. This method has been shown to improve the final per- to other teams was relatively small and in many cases not formance in small data size scenarios and a net improvement statistically significant, as presented in Table VIII. The three with a scale effect that depends on the model architecture [38]. best performing teams, P1 to P3, used nnUNet as the baseline pipeline, as well as standard intensity-based data augmentation (e.g. blurring, noise addition, histogram matching), but no C. Domain Adaptation domain adaptation, showing a significative improvement with Of all participants, only three teams (P4, P6, P10) imple- respect to the standard nnUNet implementation B2. For a sim- mented a method to explicitly address the differences in the ilar performance, P5 used an Attention UNet as the backbone image distributions between the unseen and trained vendors. architecture and CycleGANs for data augmentation through At training, P4 constructed a classifier to distinguish between image synthesis. P4 and P6 also obtained similar performances scanner vendors and used it to modify the training images overall, but implemented instead domain adaptation methods (through error propagation) until the classifier could not dis- and no image-driven data augmentation. tinguish between the domain. In other words, this method Fig. 5 displays the average DSC for all participating teams resulted in training images and a trained model that are less organised this time per pathology, showing better segmentation dependent on the specific vendors. P6 and P10 proposed to performance for healthy cases and dilated cardiomyopathy train two models simultaneously with shared features, one (DCM), followed by hypertrophic cardiomyopathy (HCM) and for segmentation and one for classification, such that the other pathologies. It can be seen that the performances of CAMPELLO et al.: MULTI-CENTRE, MULTI-VENDOR AND MULTI-DISEASE CARDIAC SEGMENTATION 3549 Fig. 4. Weighted average DSC and HD for all participating methods, according to equation (5). TABLE VIII DSC AND HD FOR THE F INAL S UBMISSIONS OF A LL PARTICIPANTS AND THE T WO B ASELINE M ODELS . B OLD FACE N UMBERS A RE THE B EST R ESULTS FOR E ACH C OLUMN AND B LUE N UMBERS A RE N ON -S IGNIFICANTLY L OWER R ESULTS W HEN C OMPARED TO THE P1 R ESULTS (p-VALUE > 0.01 FOR THE W ELCH ’ S t-T EST ). HD I S M EASURED IN M ILLIMETERS Fig. 5. Average DSC for all participants for the most common pathologies in the dataset. HCM and DCM stand for hypertrophic and dilated Fig. 6. Boxplots with vendor-wise results for DSC and HD when all cardiomyopathy, respectively. participants predictions are considered. Vendors are presented in order: Siemens (A), Philips (B), GE (C) and Canon (D). the 14 techniques relative to each other do not change when analysed per pathology. can be seen that for the baseline methods there is a loss of accuracy of up to −6% in the segmentation of images from B. Analysis per Vendor vendors C and D compared to A and B. However, this loss Fig. 6 summarizes the segmentation results for all teams is reduced, for example, to −1.5% for P1 (e.g. from DSC = for each vendor separately (A, B, C & D). It can be seen that 0.92 for vendor A to 0.90 in vendor C and D, for the LV), overall, the differences in the segmentation errors between the −2.1% for P2 (e.g. from DSC = 0.87 in vendor B to 0.82 vendors are reduced with respect to the results obtained by the in vendor D, for the RV), and almost to 0% for P7. This two baseline methods as detailed in Table IX. Specifically, it indicates that while there is a need for further research to 3550 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 40, NO. 12, DECEMBER 2021 TABLE IX DSC R ESULTS S TRATIFIED BY V ENDOR AND H EART S UBSTRUCTURE . T HE L AST T WO C OLUMNS A RE THE AVERAGE DSC L OSS FOR V ENDORS C AND D W ITH R ESPECT TO THE C OMBINED AVERAGE DSC R ESULTS F ROM V ENDORS A AND B Fig. 7. Boxplots with centre-wise results for DSC and HD when all Fig. 9. Boxplots for DSC results for the top 3 performing methods participants predictions are considered. Same color-coding as in Fig. 6 depending on different cardiac structures (LV, MYO and RV) and different is used for scanner vendors. slice position for both ED and ES. The apex and the base are defined as the last and first annotated slices, respectively. The middle slice is the slice located in between the apex and base slices. The remaining slices are defined based on their relative position with respect to the middle slice. using the same scanner may be overstated, making it necessary to consider also Fig. 7, where the segmentation results are summarized according to the six clinical centres. Here too, it can be seen that there remains some degree of variation in the segmentation of the CMR images from the different centres. In more detail, there is a decrease in segmentation accuracy between centres 1 and 6 even though their images Fig. 8. Boxplots for DSC and HD results for centres that had labelled are from the same scanner vendor A. However, this difference samples in the training set, unlabelled samples in the training set and no can be explained by two facts: 1) the scanners in these two samples at all. centres are different models and have different field strengths, as shown in Table III, and 2) all the 75 datasets included during bring segmentation accuracy in unseen and unlabelled vendors training for vendor A were from centre 1 (Spain) and none at the same level of the one obtained in trained vendors, data from centre 6 (Canada). In this case, even though the images augmentation and data adaptation enable to close the gap and are from the same vendor, differences in scanner specifications improve the generalizability of deep learning models. resulted in the lack of generalizability. In contrast, images from both centres 2 and 3 were included in the training of vendor C. Analysis per Centre B, which resulted in segmentation accuracies for these two In the previous subsection, centres were combined in centres that are comparable. Finally, the datasets from centres the analysis despite having different machines or scanning 4 and 5 correspond to vendors C and D, respectively, which protocols. In doing so, possible variabilities between centres were not included in the training, which explain the loss of CAMPELLO et al.: MULTI-CENTRE, MULTI-VENDOR AND MULTI-DISEASE CARDIAC SEGMENTATION 3551 Fig. 10. Prediction examples for method P1 for vendors C (GE) and D (Canon). Top two rows show satisfactory results, while the two bottom rows present some error in the final contours. Color correspondence: left ventricle endocardium (red), left ventricle epicardium (green) and right ventricle endocardium (yellow). Ground truth is drawn in white color. accuracy compared to centres 1, 2 and 3. In Fig. 8, the results challenge developed varying workflows combining a base- are grouped for all centres according to their inclusion (or not) line neural network, intensity-based and/or spatial data aug- in the training. Clearly, it can be seen that the segmentation mentation, and in some cases a data adaptation strategy. accuracy is the highest for centres that are part of the training In addition to a relatively large sample of 175 cases for together with their labels, followed by those with images but training, the authors were given a total of seven attempts for no labels, and finally the performance is the lowest and most optimising the parameters and characteristics of their models variable for images from fully unseen centres. This result during the validation process, to ensure an optimal design of confirms the need for further developments to optimize the the solutions. generalizability of deep learning solutions in future tools for cardiac image segmentation. A. Analysis of the Methods The obtained results, first of all, indicate that data aug- D. Qualitative Results mentation, though its primary purpose is to increase training Fig. 9 presents the effect of the slice position in the size and reduce over-fitting, can perform well in addressing final segmentation DSC for the top three performing teams, some of the differences in image appearance between vendors. quantifying the loss of accuracy, especially prominent in the In particular, by varying the parameters and types of intensity apical and basal slices. To illustrate this, Fig. 10 provides some transformations (e.g. histogram matching, contrast modifi- visual examples from team P1 to further show the added value cation, noise addition, image synthesis), one can generate of the implemented techniques, as well as their limitations new training images that enhance the generalizability of the when applied to unseen vendors. In the two examples above, models. As an example, one can look at the performance the segmentation techniques enabled to accurately identify the of the baselines models B1 and B2 and augmented models, cardiac boundaries even though these imaging protocols were such as P1, P2 and P3. While for the baseline models, not included in the training set. However, in the two examples the results do not differ significantly for specific cases, such as below, despite the use of data augmentation and domain at ES, P1-P3 used many more data augmentation types, such adaptation, the models were unsuccessful in the segmentation as histogram matching, noise addition, brightness modifica- of these unseen cases and diverged more notably from the tion and contrast modification, and obtained a more marked ground truth in basal slices. These examples illustrate the need improvement (e.g. the DSC for the myocardium at ES for future work to further improve the generalizability of deep increased from 0.84 for B1 to 0.86 for P1, the DSC for the learning models in cardiac image segmentation. RV at ES increased from 0.81 for B1 to 0.84 for P3). This indicates the added value of more advanced image-driven data V. D ISCUSSION augmentation for multi-vendor image segmentation as well as In this paper, we presented a comprehensive analysis that the domain shift between different scanners or protocols of a range of deep learning solutions for the automated can be potentially solved by using an exhaustive set of image segmentation of multi-centre, multi-vendor and multi-disease transformations during training. However, the results also CMR datasets. Roughly speaking, the 14 participants in the clearly show that the obtained segmentations remain generally 3552 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 40, NO. 12, DECEMBER 2021 more stable in trained vendors compared to unseen vendors, c) Complex workflows did not always lead to better results, as intensity-driven data augmentation alone cannot enable a resulting sometimes in an excessive use of computing full coverage of the variety of imaging protocols that can exist resources. across clinical centres. As for domain adaptation, while it is theoretically suitable B. Analysis of the Segmentation Results for multi-vendor image segmentation, as it can adapt on the spot to the imaging distribution of the unseen images, it did Compared to other publicly available and annotated multi- not result in better segmentations than when using exhaustive structure (LV, MYO, RV) datasets in the field of CMR seg- data augmentation alone. In fact, the three first techniques in mentation, M&Ms is the largest as well as the most diverse the ranking did not use any domain adaptation, though it is (375 cases from four vendors, six centres and three countries, important to reiterate that the first seven solutions obtained vs. 150 cases for ACDC from one centre). However, given relatively similar results overall. It is worth noting that the that ACDC is an established database, we selected to use choice of the baseline model may play a role, as again its contouring SOP in this challenge to derive standardized the first three techniques used the same model, namely the annotations for the community, as well as to enable the nnUNet. Finally, while the results indicate the potential of combination of these datasets in future studies. data augmentation and domain adaption, they also show that Note that our study, while it focuses on multi-scanner there is still a loss in segmentation accuracy when segmenting generalizable segmentation, confirms several of the results labelled versus unlabelled or unseen image samples. Note also already obtained by the ACDC challenge and other previous that training and testing a model on two datasets from the works. Specifically: same vendor does not guarantee a good generalizability. This a) The segmentations at ED were more accurate than at is particularly true if the two sets of images are from two ES for LV and RV cavities, but not for the myocardium, different centres and scanner types, such as 1.5T (e.g. centre 1) which becomes thicker and therefore easier to segment and 3T (e.g. centre 6) as shown in Figure 7. when the heart contracts. The results also show that advanced workflows integrating, b) The segmentation accuracy according to the DSC was for instance, data augmentation or generative adversarial net- the highest for the LV blood pool, followed by the RV works, are not guaranteed to lead to robust segmentations. and MYO, in this order, but it was the lowest for the In fact, half of the submitted techniques had a lower perfor- RV for the distance-based measures, given its shape mance than the two baselines implemented for comparison. complexity. This shows that over-fitting remains a challenge that requires c) The segmentation accuracy was at its maximum at the special attention during the calibration and validation of com- mid-ventricular slices, while the performance decreased plex deep learning solutions for cardiac image segmentation, for the apical and basal slices, where there is higher in particular in the presence of highly heterogeneous data. variability and complexity. Lastly, the presented methods show a vast diversity in On average, the best performing method in this challenge hardware performance, with training times ranging from obtained 0.88 as DSC and 11 mm as HD versus the values 6 to 100 hours and inference times from tenths of seconds 0.93 and 9 mm obtained in the ACDC challenge, respectively, to almost half a minute. However, the amount of training with the greatest difference shown at ES. This gap can be and inference time do not correlate well with the final accu- easily explained by the single-centre nature of the ACDC racy, indicating an excessive use of computational power for studies in comparison to a multi-centre scenario in this work, some techniques. For example, the methods implemented by although other effects such as the training size may play a role P1 and P2, despite using the same baseline model than P3, and should be assessed (150 vs. 100 studies, respectively). needed around half the time for training and obtained slightly better results (1.2% average improvement in DSC), while P4 used around one tenth of computing time for similar C. Future Work loss of accuracy with respect to P1 (1.6% average loss in In addition to the results and analyses presented in this paper DSC). Furthermore, clinical centres usually lack dedicated on multi-scanner cardiac image segmentation, we also provide hardware for deep learning models thus increasing even more the M&Ms dataset open-access for the community, which the segmentation time. In this sense, a good equilibrium can be downloaded from the M&Ms website.6 It represents between accuracy and processing time needs to be attained, one of the most heterogeneous datasets ever compiled in with methods such as P4 serving as a good example with cardiac image analysis, comprising CMRs from a variety of a competitive performance and a prediction rate of around imaging protocols and cardiology units, and including a range 3 images per second. of cardiovascular diseases as distinct as coronary heart disease, In summary, the main findings are: cardiomyopathies, abnormal right ventricle or myocarditis. a) Exhaustive data augmentation reduced considerably the We thus hope the dataset will be of high value for the commu- domain gap, although the results were still more stable nity to address a number of research topics in the field, such as within the domains used during training. multi-scanner image registration, multi-structure segmentation, b) Domain adaptation did not result in better performance cardiac quantification, motion analysis and image synthesis. when compared to nnUNet models trained with spatial and intensity-driven data augmentation. 6 www.ub.edu/mnms CAMPELLO et al.: MULTI-CENTRE, MULTI-VENDOR AND MULTI-DISEASE CARDIAC SEGMENTATION 3553 It is important to note that a follow-up challenge is being Berkeley, CA 94720 USA (e-mail: fanwei_kong@berkeley.edu; organised on multi-centre, multi-vendor and multi-disease car- shadden@berkeley.edu). Jorge Corral Acero is with the Institute of Biomedical Engi- diac diagnosis. The diagnoses for the 375 cases are being gath- neering, Department of Engineering Science, University of Oxford, ered from the different hospitals in a legally compliant manner Oxford OX3 7DQ, U.K. (e-mail: jor.corral@eng.ox.ac.uk). and the clinical information will be made available after the Vaanathi Sundaresan is with the Centre for the Functional MRI of the Brain, Nuffield Department of Clinical Neurosciences, end of the next challenge, thus allowing the community to University of Oxford, Oxford OX3 9DU, U.K. (e-mail: vaanathi. work on cardiac image analysis as well as on computer-aided sundaresan@ndcn.ox.ac.uk). diagnosis in a multi-centre setting. Note that the participants Mina Saber is with the Research and Development Division, Intixel Company S.A.E., Cairo 11585, Egypt (e-mail: mina.saber@intixel.com). had less than three months to implement, optimize and test Mustafa Elattar is with the Research and Development Division, Intixel their techniques, which did not allow to go beyond the Company S.A.E., Cairo 11585, Egypt, and also with the Medical Imaging existing state-of-the-art techniques in data augmentation and and Image Processing Group, Nile University, Giza 16453, Egypt (e-mail: melattar@nu.edu.eg). domain adaptation. With more time at their disposal beyond Hongwei Li is with the Department of Computer Science, Technische the constraints of the challenge, we expect that researchers Universität München, 80333 Munich, Germany, and also with Orbem will have a valuable resource with the M&Ms dataset to GmbH, 85748 Garching, Germany (e-mail: hongwei.li@tum.de). Bjoern Menze is with the Department of Computer Science, investigate, develop and test new theories and frameworks for Technische Universität München, 80333 Munich, Germany (e-mail: addressing the difficulties posed by domain-shift in cardiac bjoern.menze@tum.de). image analysis. Firas Khader and Christoph Haarburger are with ARISTRA GmbH, 10439 Berlin, Germany (e-mail: firas.khader96@gmail.com; christoph.haarburger@aristra.com). Cian M. Scannell is with the School of Biomedical Engineering and D. Conclusion Imaging Sciences, King’s College London, London WC2R 2LS, U.K. The M&Ms challenge is the first study to evaluate a range (e-mail: cian.scannell@kcl.ac.uk). Mitko Veta is with the Department of Biomedical Engineering, of deep learning solutions for the automated segmentation of Eindhoven University of Technology, 5612 Eindhoven, The Netherlands multi-centre, multi-vendor and multi-disease cardiac images. (e-mail: m.veta@tue.nl). The results show the promise of existing data augmentation Adam Carscadden and Kumaradevan Punithakumar are with the Department of Radiology and Diagnostic Imaging, University and domain adaptation methods, but also calls for further of Alberta, Edmonton, AB T6G 2R3, Canada, and also with the research to develop highly generalizable solutions given the Servier Virtual Cardiac Centre, Mazankowski Alberta Heart Insti- inherent heterogeneity in cardiac imaging between centres, tute, Edmonton, AB T6G 2B7, Canada (e-mail: carscadd@ualberta.ca; punithak@ualberta.ca). vendors and protocols. More generally, there is a need for Xiao Liu is with the School of Engineering, The University of more research and development to realise the much-needed Edinburgh, Edinburgh EH9 3FB, U.K. (e-mail: xiao.liu@ed.ac.uk). shift from single-centre image analysis towards multi-domain Sotirios A. Tsaftaris is with the School of Engineering, The University of Edinburgh, Edinburgh EH9 3FB, U.K., and also with The Alan Turing approaches that will enable wider translation and usability Institute, London NW1 2DB, U.K. (e-mail: s.tsaftaris@ed.ac.uk). of future artificial intelligence tools in cardiac imaging and Xiaoqiong Huang and Xin Yang are with the School of Biomedical clinical cardiology. Engineering, Shenzhen University, Shenzhen 518037, China, and the Medical UltraSound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen 518037, China (e-mail: xqiong0920@gmail.com; ACKNOWLEDGMENT yangxinknow@gmail.com). The authors would like to thank NVIDIA and the Barcelona Lei Li is with the School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China (e-mail: lilei.sky@sjtu.edu.cn). Super Computing (BSC) Centre for providing the necessary Xiahai Zhuang is with the School of Data Science, Fudan University, computational resources for this work. Shanghai 200433, China (e-mail: zxh@fudan.edu.cn). David Viladés and Martín L. Descalzo are with the Cardiac Imag- Víctor M. Campello, Polyxeni Gkontra, Cristian Izquierdo, Carlos ing Unit, Cardiology Service, Hospital de la Santa Creu i Sant Pau, Martín-Isla, and Karim Lekadir are with the Artificial Intelligence in Universitat Autònoma de Barcelona, 08193 Barcelona, Spain (e-mail: Medicine Laboratory (BCN-AIM), Departament de Matemàtiques dvilades@santpau.cat; mdescalzo@santpau.cat). i Informàtica, Universitat de Barcelona, 08007 Barcelona, Spain (e-mail: Andrea Guala and José F. Rodríguez-Palomares are with the victor.campello@ub.edu; polyxeni.gkontra@ub.edu; c.izquierdo@ CIBERCV, Department of Cardiology, Vall d’Hebron Institut de Recerca, ub.edu; carlos.martinisla@ub.edu; karim.lekadir@ub.edu). Hospital Universitari Vall d’Hebron, Universitat Autònoma de Barcelona, Alireza Sojoudi is with Circle Cardiovascular Imaging Pvt., Ltd., 08193 Barcelona, Spain (e-mail: andrea.guala86@gmail.com; Calgary, AB T2P 3T6, Canada (e-mail: alireza.sojoudi@circlecvi.com). jfrodriguezpalomares@gmail.com). Peter M. Full and Klaus Maier-Hein are with the Division of Lucia La Mura is with the Department of Advanced Biomedical Medical Image Computing, German Cancer Research Center, Sciences, University of Naples Federico II, 80138 Naples, Italy (e-mail: 69120 Heidelberg, Germany (e-mail: p.full@dkfz-heidelberg.de; lucia.lamura@hotmail.it). K.Maier-Hein@dkfz-heidelberg.de). Matthias G. Friedrich, Ria Garg, Julie Lebel, and Filipe Yao Zhang is with the Institute of Computing Technology, Henriques are with the Department of Medicine and Diagnostic Chinese Academy of Sciences, Beijing 100190, China (e-mail: Radiology, McGill University, Montreal, QC H3A 0G4, Canada zhangyao215@mails.ucas.ac.cn). (e-mail: matthias.friedrich@mcgill.ca; ria.garg@muhc.mcgill.ca; Zhiqiang He is with Lenovo Ltd., Beijing 100085, China (e-mail: julie.lebel@muhc.mcgill.ca; filipe.henriques@muhc.mcgill.ca). hezq@lenovo.com). Mahir Karakas and Ersin Çavuş are with the Department Jun Ma is with the Department of Mathematics, Nanjing Uni- of Cardiology, University Heart & Vascular Center Hamburg, versity of Science and Technology, Nanjing 210094, China (e-mail: 20251 Hamburg, Germany, and also with the German Center for junma@njust.edu.cn). Cardiovascular Research (DZHK), 10785 Berlin, Germany (e-mail: Mario Parreño is with the PRHLT Research Center, Universitat Politèc- m.karakas@uke.de; e.cavus@uke.de). nica de València, 46022 Valencia, Spain (e-mail: maparla@prhlt.upv.es). Steffen E. Petersen is with the Barts Heart Centre, Barts Health Alberto Albiol is with the iTeam Research Institute, Universitat Politèc- NHS Trust, London E1 1BB, U.K., and also with the NIHR Barts nica de València, 46022 Valencia, Spain (e-mail: alalbiol@iteam.upv.es). Biomedical Research Centre, The William Harvey Research Institute, Fanwei Kong and Shawn C. Shadden are with the Department Queen Mary University of London, London E1 4NS, U.K. (e-mail: of Mechanical Engineering, University of California at Berkeley, s.e.petersen@qmul.ac.uk). 3554 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 40, NO. 12, DECEMBER 2021 Sergio Escalera is with the Departament de Matemàtiques [20] J. Ma, “Histogram matching augmentation for domain adaptation with i Informàtica, Universitat de Barcelona, 08007 Barcelona, Spain, and also application to multi-centre, multi-vendor and multi-disease cardiac with the Computer Vision Center, Universitat Autònoma de Barcelona, image segmentation,” in Statistical Atlases and Computational Models 08193 Barcelona, Spain (e-mail: sergio.escalera.guerrero@gmail.com). of the Heart. M&Ms and EMIDEC Challenges. Cham, Switzerland: Santi Seguí is with the Departament de Matemàtiques i Informàtica, Springer, 2021, pp. 177–186. Universitat de Barcelona, 08007 Barcelona, Spain (e-mail: [21] M. Parreño, R. Paredes, and A. Albiol, “Deidentifying MRI data santi.segui@ub.edu). domain by iterative backpropagation,” in Statistical Atlases and Compu- tational Models of the Heart. M&Ms and EMIDEC Challenges. Cham, R EFERENCES Switzerland: Springer, 2021, pp. 277–286. [22] F. Kong and S. C. Shadden, “A generalizable deep-learning approach for [1] I. Cetin et al., “A radiomics approach to computer-aided diagnosis with cardiac magnetic resonance image segmentation using image augmenta- cardiac cine-MRI,” in Statistical Atlases and Computational Models tion and attention U-Net,” in Statistical Atlases and Computational Mod- of the Heart. ACDC and MMWHS Challenges. Cham, Switzerland: els of the Heart. M&Ms and EMIDEC Challenges. Cham, Switzerland: Springer, 2018, pp. 82–90. Springer, 2021, pp. 287–296. [2] C. Martín-Isla et al., “Image-based cardiac diagnosis with machine [23] J. C. Acero, V. Sundaresan, N. Dinsdale, V. Grau, and M. Jenkinson, learning: A review,” Frontiers Cardiovascular Med., vol. 7, p. 1, “A 2-step deep learning method with domain adaptation for multi-centre, Jan. 2020. multi-vendor and multi-disease cardiac magnetic resonance segmenta- [3] X. Albà, K. Lekadir, M. Pereañez, P. Medrano-Gracia, A. A. Young, and tion,” in Statistical Atlases and Computational Models of the Heart. A. F. Frangi, “Automatic initialization and quality control of large-scale M&Ms and EMIDEC Challenges. Cham, Switzerland: Springer, 2021, cardiac MRI segmentations,” Med. Image Anal., vol. 43, pp. 129–141, pp. 196–207. Jan. 2018. [24] M. Saber, D. Abdelrauof, and M. Elattar, “Multi-center, multi-vendor, [4] W. Bai, W. Shi, C. Ledig, and D. Rueckert, “Multi-atlas segmentation and multi-disease cardiac image segmentation using scale-independent with augmented features for cardiac MR images,” Med. Image Anal., multi-gate UNET,” in Statistical Atlases and Computational Models vol. 19, no. 1, pp. 98–109, Jan. 2015. of the Heart. M&Ms and EMIDEC Challenges. Cham, Switzerland: [5] C. Chen et al., “Deep learning for cardiac image segmenta- Springer, 2021, pp. 259–268. tion: A review,” Frontiers Cardiovascular Med., vol. 7, p. 25, [25] H. Li, J. Zhang, and B. Menze, “Generalisable cardiac structure seg- Mar. 2020. mentation via attentional and stacked image adaptation,” in Statistical [6] O. Bernard et al., “Deep learning techniques for automatic MRI Atlases and Computational Models of the Heart. M&Ms and EMIDEC cardiac multi-structures segmentation and diagnosis: Is the problem Challenges. Cham, Switzerland: Springer, 2021, pp. 297–304. solved?” IEEE Trans. Med. Imag., vol. 37, no. 11, pp. 2514–2525, [26] F. Khader, J. Schock, D. Truhn, F. Morsbach, and C. Haarburger, Nov. 2018. [7] F. Isensee, P. F. Jaeger, P. M. Full, I. Wolf, S. Engelhardt, and “Adaptive preprocessing for generalization in cardiac MR image seg- K. H. Maier-Hein, “Automatic cardiac disease assessment on cine-MRI mentation,” in Statistical Atlases and Computational Models of the via time-series segmentation and domain specific features,” in Statistical Heart. M&Ms and EMIDEC Challenges. Cham, Switzerland: Springer, Atlases and Computational Models of the Heart. ACDC and MMWHS 2021, pp. 269–276. Challenges. Cham, Switzerland: Springer, 2018, pp. 120–129. [27] C. M. Scannell, A. Chiribiri, and M. Veta, “Domain-adversarial learning [8] I. Cetin et al., “Radiomics signatures of cardiovascular risk factors in for multi-centre, multi-vendor, and multi-disease cardiac MR image cardiac MRI: Results from the UK Biobank,” Frontiers Cardiovascular segmentation,” in Statistical Atlases and Computational Models of the Med., vol. 7, p. 232, Nov. 2020. Heart. M&Ms and EMIDEC Challenges. Cham, Switzerland: Springer, [9] W. Bai et al., “Automated cardiovascular magnetic resonance image 2021, pp. 228–237. analysis with fully convolutional networks,” J. Cardiovascular Magn. [28] A. Carscadden, M. Noga, and K. Punithakumar, “A deep convolutional Reson., vol. 20, no. 1, p. 65, Dec. 2018. neural network approach for the segmentation of cardiac structures from [10] P. V. Tran, “A fully convolutional neural network for cardiac segmen- MRI sequences,” in Statistical Atlases and Computational Models of the tation in short-axis MRI,” 2016, arXiv:1604.00494. [Online]. Available: Heart. M&Ms and EMIDEC Challenges. Cham, Switzerland: Springer, http://arxiv.org/abs/1604.00494 2021, pp. 250–258. [11] Q. Tao et al., “Deep Learning–based method for fully automatic [29] X. Liu, S. Thermos, A. Chartsias, A. O’Neil, and S. A. Tsaftaris, “Disen- quantification of left ventricle function from cine MR images: A tangled representations for domain-generalized cardiac segmentation,” in multivendor, multicenter study,” Radiology, vol. 290, no. 1, pp. 81–88, Statistical Atlases and Computational Models of the Heart. M&Ms and Jan. 2019. EMIDEC Challenges. Cham, Switzerland: Springer, 2021, pp. 187–195. [12] M. Khened, V. A. Kollerathu, and G. Krishnamurthi, “Fully convo- [30] X. Huang et al., “Style-invariant cardiac image segmentation with test- lutional multi-scale residual DenseNets for cardiac segmentation and time augmentation,” in Statistical Atlases and Computational Models automated cardiac diagnosis using ensemble of classifiers,” Med. Image of the Heart. M&Ms and EMIDEC Challenges. Cham, Switzerland: Anal., vol. 51, pp. 21–45, Jan. 2019. Springer, 2021, pp. 305–315. [13] C. Chen et al., “Improving the generalizability of convolutional neural [31] L. Li et al., “Random style transfer based domain generalization network-based segmentation on CMR images,” Frontiers Cardiovascular networks integrating shape and spatial information,” in Statistical Atlases Med., vol. 7, p. 105, Jun. 2020. and Computational Models of the Heart. M&Ms and EMIDEC Chal- [14] P. Radau, Y. Lu, K. Connelly, G. Paul, A. J. W. G. Dick, and G. Wright, lenges. Cham, Switzerland: Springer, pp. 208–218. “Evaluation framework for algorithms segmenting short axis cardiac [32] Z. Liu et al., “Remove appearance shift for ultrasound image segmenta- MRI,” MIDAS J.-Cardiac MR Left Ventricle Segmentation Challenge, tion via fast and universal style transfer,” in Proc. IEEE 17th Int. Symp. 2009. [Online]. Available: http://hdl.handle.net/10380/3070 Biomed. Imag. (ISBI), Apr. 2020, pp. 1824–1828. [15] A. Suinesiaputra et al., “A collaborative resource to build consensus for [33] F. Isensee, P. F. Jaeger, S. A. A. Kohl, J. Petersen, and K. H. Maier-Hein, automated left ventricular segmentation of cardiac MR images,” Med. “nnU-Net: A self-configuring method for deep learning-based biomed- Image Anal., vol. 18, no. 1, pp. 50–62, Jan. 2014. ical image segmentation,” Nature Methods, vol. 18, no. 2, pp. 203–211, [16] C. Petitjean et al., “Right ventricle segmentation from cardiac MRI: Feb. 2021. A collation study,” Med. Image Anal., vol. 19, no. 1, pp. 187–202, 2015. [34] H. Li, A. Zhygallo, and B. Menze, “Automatic brain structures seg- [17] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional net- mentation using deep residual dilated U-Net,” 2018, arXiv:1811.04312. works for biomedical image segmentation,” in Proc. Int. Conf. Med. [Online]. Available: http://arxiv.org/abs/1811.04312 Image Comput. Comput.-Assist. Intervent., in Lecture Notes in Computer [35] O. Oktay et al., “Attention U-Net: Learning where to look for the Science. Cham, Switzerland: Springer, 2015, pp. 234–241. pancreas,” 2018, arXiv:1804.03999. [Online]. Available: http://arxiv. [18] P. M. Full, F. Isensee, P. F. Jäger, and K. Maier-Hein, “Studying org/abs/1804.03999 robustness of semantic segmentation under domain shift in cardiac [36] A. Chartsias et al., “Disentangled representation learning in cardiac MRI,” in Statistical Atlases and Computational Models of the Heart. image analysis,” Med. Image Anal., vol. 58, Dec. 2019, Art. no. 101535. M&Ms and EMIDEC Challenges. Cham, Switzerland: Springer, 2021, [37] X. Huang and S. Belongie, “Arbitrary style transfer in real-time with pp. 238–249. adaptive instance normalization,” in Proc. IEEE Int. Conf. Comput. Vis. [19] Y. Zhang et al., “Semi-supervised cardiac image segmentation via label (ICCV), Oct. 2017, pp. 1510–1519. propagation and style transfer,” in Statistical Atlases and Computa- [38] D. Shanmugam, D. Blalock, G. Balakrishnan, and J. Guttag, “When and tional Models of the Heart. M&Ms and EMIDEC Challenges. Cham, why test-time augmentation works,” 2020, arXiv:2011.11156. [Online]. Switzerland: Springer, 2021, pp. 219–227. Available: http://arxiv.org/abs/2011.11156