European Journal of Nuclear Medicine and Molecular Imaging (2021) 48:3444–3456 https://doi.org/10.1007/s00259-021-05244-z ORIGINAL ARTICLE Convolutional neural networks for PET functional volume fully automatic segmentation: development and validation in a multi-center setting Andrei Iantsen1 · Marta Ferreira2 · Francois Lucia1 · Vincent Jaouen1 · Caroline Reinhold3 · Pietro Bonaffini3 · Joanne Alfieri4 · Ramon Rovira5 · Ingrid Masson6 · Philippe Robin7 · Augustin Mervoyer6 · Caroline Rousseau8 · Frédéric Kridelka9 · Marjolein Decuypere9 · Pierre Lovinfosse10 · Olivier Pradier1 · Roland Hustinx2 · Ulrike Schick1 · Dimitris Visvikis1 · Mathieu Hatt1 Received: 8 September 2020 / Accepted: 7 February 2021 / Published online: 27 March 2021 © The Author(s) 2021 Abstract Purpose In this work, we addressed fully automatic determination of tumor functional uptake from positron emission tomography (PET) images without relying on other image modalities or additional prior constraints, in the context of multicenter images with heterogeneous characteristics. Methods In cervical cancer, an additional challenge is the location of the tumor uptake near or even stuck to the bladder. PET datasets of 232 patients from five institutions were exploited. To avoid unreliable manual delineations, the ground truth was generated with a semi-automated approach: a volume containing the tumor and excluding the bladder was first manually determined, then a well-validated, semi-automated approach relying on the Fuzzy locally Adaptive Bayesian (FLAB) algorithm was applied to generate the ground truth. Our model built on the U-Net architecture incorporates residual blocks with concurrent spatial squeeze and excitation modules, as well as learnable non-linear downsampling and upsampling blocks. Experiments relied on cross-validation (four institutions for training and validation, and the fifth for testing). Results The model achieved good Dice similarity coefficient (DSC) with little variability across institutions (0.80±0.03), with higher recall (0.90±0.05) than precision (0.75±0.05) and improved results over the standard U-Net (DSC 0.77±0.05, recall 0.87±0.02, precision 0.74±0.08). Both vastly outperformed a fixed threshold at 40% of SUVmax (DSC 0.33±0.15, recall 0.52±0.17, precision 0.30±0.16). In all cases, the model could determine the tumor uptake without including the bladder. Neither shape priors nor anatomical information was required to achieve efficient training. Conclusion The proposed method could facilitate the deployment of a fully automated radiomics pipeline in such a challenging multicenter context. Keywords Convolutional neural network · PET · Segmentation · Cervical cancer · U-Net Introduction to provide functional information on organs and tissues, as well as disease abnormalities. Static PET images provide Combined positron emission tomography/computed tomog- semi-quantitative information regarding the distribution of a raphy (PET/CT) imaging is widely used in clinical practice radiotracer uptake. In oncology, fluorodeoxyglucose (FDG) PET imaging is routinely relied upon for diagnosis, staging, D. Visvikis and M. Hatt contributed equally to this work. treatment planning, and therapy follow-up [20]. In clinical applications, nuclear medicine physicians This article is part of the Topical Collection on Advanced Image Analyses (Radiomics and Artificial Intelligence) carry out qualitative assessments of PET/CT images, which is typically sufficient for detecting and anatomically  Andrei Iantsen locating lesions. For radiotherapy treatment planning, andrei.iantsen@inserm.fr radiation oncologists manually draw boundaries on fused PET/CT images to determine the gross target volume (GTV) Extended author information available on the last page of the article. of a tumor, in order to subsequently deliver a specific dose Eur J Nucl Med Mol Imaging (2021) 48:3444–3456 3445 to the target. The boundary of the target should be defined as is typically a labor-intensive, time-consuming task with low accurately as possible to maximize the coverage of the target reproducibility, due to the high intra- and interobserver and minimize the dose delivered to surrounding healthy variability [19]. tissues and nearby organs-at-risk (OAR). There have been a number of algorithms proposed More quantitative assessment of FDG uptake in PET for PET image segmentation, accounting at different images can also be performed. For instance, radiomics anal- degrees for some of the limitations referred to above yses [23] aim at extracting clinically relevant measurements [20]. For example, thresholding-based methods, the most through the calculation of numerous image-derived features simple image segmentation techniques, work on the features (e.g., intensity, shape, and textural). Such measures assumption that different tissue types have specific uptake can subsequently be used to build models predictive of out- ranges; therefore, segmentation can be done by comparing come or for assessing changes in tumors before, during, and individual voxel intensities with a set of thresholds. More after treatment in order to better evaluate response to ther- advanced methods aim at exploiting statistical differences apy [7, 37, 38]. It has been shown in all image modalities between uptake regions and surrounding tissues. These including PET that the choice of the segmentation method in include different clustering and classification methods this step of the radiomics workflow can significantly affect trained on a set of features extracted from PET images, the extracted features [21, 31, 42, 46, 51]. In addition, it is as well as atlas-based [39, 45] and generative models recognized that in the absence of fully automated segmen- such as Gaussian mixture models (GMM) [2] and Fuzzy tation, this step is a crucial bottleneck and time-consuming Locally Adaptive Bayesian (FLAB) model [16]. Numerous step of any radiomics study, preventing such a process to be other common image segmentation algorithms have been expanded to very large datasets [23]. There is therefore a evaluated for this task using PET uptake only [20]. For need for a delineation method that is not only accurate and the vast majority of these published methods, it is usually robust, but also fully automated as well. assumed that the tumor has been previously isolated in a There are several challenges pertaining to PET image volume of interest (VOI), i.e., the input to the algorithm is segmentation [20]. First, PET images suffer from limited not the entire PET image but a sub-volume containing the spatial resolution (4–5 mm), comparatively to CT (below object of interest, that is usually manually determined after 1 mm) due to partial volume effects (PVE) that make visually detecting the tumor uptake in the whole image. boundaries between adjacent functional regions blurred and It should be emphasized that numerous approaches tried result in underestimated activity in small objects of interest. to improve PET segmentation by considering both PET Second, signal-to-noise ratio in PET images is inherently and CT modalities together, assuming an (almost) perfect low and affected by a vast array of factors, such as scanner correspondence between tumor functional uptake and tumor sensitivity, temporal resolution, acquisition mode, scan anatomical boundaries as determined on CT images using time, quantity and distribution of tracer, applied corrections co-segmentation approaches exploiting co-registered PET (e.g., scatter, attenuation, randoms) and reconstruction and CT images [9, 14, 34]. This assumption may not algorithm type (e.g., resolution recovery, time of flight) be true as radiotracer uptake and anatomical boundaries and parameters (e.g., number of iterations). All these issues can be uncorrelated. This also makes the method sensitive make things challenging in a multi-center context, i.e., when to registration issues in PET imaging, especially in body analyzing PET images acquired using different systems, regions affected by motion [20]. acquisition protocols, and reconstruction settings. Third, Convolutional neural networks (CNNs) have been the wide variability in shapes and heterogeneity of lesion successfully applied to different medical imaging tasks uptakes might reduce the generalization of segmentation [35], such as reconstruction [32, 43], denoising [10, methods to only some specific cases. 44], segmentaton [13, 40], and classification [6]. Most An important aspect in medical image segmentation is segmentation studies rely on U-Net [47] that is arguably that the true boundary of the object of interest (ground the most popular network for semantic segmentation, and truth) is impossible to determine without a complete focus on anatomical modalities such as magnetic resonance histopathological analysis of an excised tumor, which can imaging (MRI) [8, 40] and CT [41, 48]. The limited number typically be performed only in a small number of cases. of papers dedicated to PET segmentation usually assume a In PET, even with a very robust protocol, this approach correspondence between functional and anatomical regions can only provide approximate co-registration between in combined PET/CT or PET/MRI imaging [12, 26, 52– the histopathology slides and the corresponding 3D PET 54]. The ground truth is usually obtained through manual slices [20]. One way to overcome this is the use of a delineation performed on multimodal images (e.g., training consensus of several manual segmentations by experts as a a network to reproduce delineations performed by radiation surrogate of truth [20]. Unfortunately, manual segmentation oncologist that perform this manually on fused PET/CT 3446 Eur J Nucl Med Mol Imaging (2021) 48:3444–3456 images). Guo et al. [13] included PET imaging within the anatomical proximity between the cervix and the blad- a CNN-based multimodal image segmentation framework der that generally has similar FDG uptake in PET scans, using PET, MR (T1 and T2), and CT images of a conventional techniques (e.g., thresholding, region- and publicly available soft tissue sarcoma dataset of 50 patients. boundary-based methods) provide poor results if applied Gross tumor volumes were manually annotated in all to the whole image without additional prior knowledge four imaging modalities. Different fusion networks were or constraints, which is why a VOI excluding physiolog- used for feature-, classifier-, and decision-level fusion, ical uptake usually needs to be provided as input to the demonstrating improved performance at a feature level method. For instance, the use of FLAB, as described in fusion [13]. the radiomic study above [37], requires an expert to first Considerably less attention has been dedicated to pro- manually define a VOI containing the tumor but exclud- cessing PET images as a stand-alone modality. Moreover, ing the bladder, in which FLAB is then applied to delineate a majority of studies have used only datasets with small the tumor. However, this step can be quite labor-intensive cohorts of patients from one or two centers and manual and time-consuming, especially when the tumor uptake and delineation as a surrogate of truth. Under these circum- bladder are very close to each other, hindering the potential stances, some previously published results might be less clinical translation of these segmentation tools, and in turn generalizable due to high heterogeneity of PET images the use of the predictive radiomics-based models. A fully caused, for instance, by scanner type, reconstruction algo- automated segmentation step, without the manual deter- rithm, and applied post-processing that vary across centers. mination of the VOI, is therefore highly desirable in that Huang et al. [26] applied U-Net with minor modifications context. for head and neck cancer gross tumor volume segmenta- The purpose of our study was thus to propose a U- tion on PET/CT images. Results were obtained for a dataset Net-based model for the fully automated delineation (i.e., of 22 patients using manual segmentation as a surrogate of without the need for visual detection of their location and truth. Blanc-Durand et al. [3] evaluated U-Net for glioma manual determination of a VOI) of 3D functional primary segmentation on PET images with the fluoroethyl-tyrosine tumor volumes in PET images only, especially in the (FET) tracer. Their dataset contained only 37 patients with specific context of cervical cancer where the pathological manually segmented lesions. Leung et al. [33] used a modi- uptake of interest is located close to a physiological one fied U-Net trained on simulated PET images and fine-tuned that should not be included (here, the bladder). A secondary using a clinical dataset of 160 patients with manual delin- objective was to train the network on a reliable ground eations for lung cancer segmentation. In cervical cancer, truth obtained through accurate and robust PET semi- Chen et al. [5] proposed to combine a 2D CNN and a automated segmentation instead of manual delineation. A post-processing step that relies on prior anatomical infor- final objective was to train and evaluate the performance mation on the tumor roundness and its position relative of our model under standard clinical imaging conditions, to the bladder. The choice of the 2D network was dic- considering a multi-center patient cohort without any tated by the limited size of the available dataset that con- prior standardization in the data acquisition or image tained 1176 slices from 50 patients, and the surrogate of reconstruction processes. truth was also obtained through manual delineation. Within the scope of the recent MICCAI challenge on automatic PET tumor functional volume delineation, the CNN-based Materials and methods method reached the highest score compared to twelve other approaches, among them some of the current state of the art PET images and training relying on FLAB-derived [22]. ground truth In this paper, we focused our experiments on cervical cancer. Approximately 570,000 cases of cervical cancer Our first objective is to achieve fully automated determina- and 311,000 deaths from the disease occurred in 2018 and tion of the functional uptake boundaries in PET images only, this type of cancer was the fourth most common cancer without relying on assumptions regarding its correspon- in women worldwide [1]. Recently, predictive models rely- dence with anatomical boundaries and to avoid registration ing on textural features from tumor volumes in PET images issues, which are important in the case of cervical cancer were able to identify the subset of patients that will suf- due to the elastic nature of organs in this body region. We fer from recurrence after treatment with clinically relevant decided to train and evaluate the proposed model exclu- accuracy [37, 46]. This obviously requires accurate delin- sively on real clinical images, contrary to recent recom- eation of the tumor volume in the PET images and this is mendations by the task group 211 of the AAPM (American achieved without the use of the associated CT image. Due to Association of Physicists in Medicine) dedicated on PET Eur J Nucl Med Mol Imaging (2021) 48:3444–3456 3447 Fig. 1 Proposed Encoder-Decoder Network with residual blocks. The number of output channels is depicted under blocks of each group auto-segmentation, which advises to rely on a combination by brachytherapy. Collected images considerably differed of simulated, phantom, and clinical images [20]. Indeed, in acquisition protocols (scanning duration per bed position, usually the number of clinical images available for train- injected radiopharmaceutical dose) and reconstruction ing and validation is small, and the surrogate of truth is (algorithms, use of time of flight information, resolution questionable when only manual delineations from experts modeling, voxel and matrix sizes) (see Table 1). Data are available. In such a context, results obtained on large were corrected for randoms and scatter in all cases. All datasets of simulated and/or phantom images can indeed reconstructed images were corrected for attenuation using increase the confidence in the results obtained in a smaller the associated low-dose CT. amount of clinical cases with less reliable surrogate of truth. An objective of our work is to train the network However, in the present work, we exploited a large dataset using a reliable ground truth excluding the bladder uptake. of images that were processed by experts using a semi- Segmentation of the tumor volumes to generate the ground automated approach (see section below detailing how the truth was performed on PET images using the FLAB ground truth was generated) for the purpose of radiomics- algorithm [16] applied in a semi-automated manner: first, a based outcome modeling studies. In addition, one objective VOI containing the tumor uptake and excluding the nearby of this work is to evaluate the ability of the proposed bladder and other physiological uptakes was manually approach to deal with fully automated tumor uptake delin- defined by the user. The FLAB algorithm was then run eation when it is located close to a physiological one that within that VOI to generate a segmentation mask that should not be included. Simulated or phantom images cor- was reviewed by the user. If this mask was deemed responding to this specific context are currently not avail- unsatisfactory, the user had the option to re-run the able in large amounts to train and evaluate a CNN-based algorithm after specifying different values of initialization approach. parameters in order to obtain a more satisfying result. We collected a dataset of 232 FDG PET images Finally, the results for all tumors were reviewed and in of patients from five institutions, all with histologically some cases (<5%) manually edited before being validated proven cervical cancer, with clinical stage IB1-IVB1 by two experts with more than 15 years of clinical practice. (see Supplementary Fig. 1). All images contained the Given that FLAB in such a context has been demonstrated abdominopelvic cavity and were acquired for diagnostic to provide accurate and reliable results in numerous studies and staging purposes before chemoradiotherapy followed [11, 21, 49], including for complex heterogeneous cases [17, 19] and over different scanner model and reconstruction algorithms, especially compared to manual delineation 1 According to the staging system of the International Federation of [18], we consider this ground truth sufficiently reliable Gynecology and Obstetrics (FIGO). for the purpose of training and evaluating the proposed 3448 Eur J Nucl Med Mol Imaging (2021) 48:3444–3456 Table 1 Summary of patients, including the different characteristics of the scanners, and associated reconstruction methods and parameters Institution Number Scanner models Voxel sizes (mm3 ) Reconstruction Time per bed FDG total of patients methods position (s) dose (MBq) University Hospital 69 Siemens Biograph 4.073×4.073×2.027 PSF+TOF 120 ± 17 253 ± 80 of Brest, France Integrated Centre 18 Siemens Biograph 4.073×4.073×2.027 PSF+TOF 202 ± 32 210 ± 56 for Oncology 5 GE Discovery STE 4.688 × 4.688 × 3.27 3D IR (ICO), France McGill University 7 GE Discovery 710 3.646 × 3.646 × 3.27 VPFXS 212 ± 30 398 ± 81 Health center, 19 GE Discovery ST 5.469 × 5.469 × 3.27 OSEM Canada Hospital of the 24 Philips Gemini TF 4×4×4 BLOB-OS-TF 109 ± 21 228 ± 50 Holy Cross and Saint Paul, Spain University Hospital 90 Philips Gemini TF 4×4×4 BLOB-OS-TF 73 ± 16 260 ± 32 of Liège, Belgium approach. Although FLAB was applied only within the [25]. Due to the high memory consumption working manually determined VOI, we then registered the obtained with 3D images, we switched from using batch norm segmentation mask onto the entire PET image used as input layers to instance normalization (instance norm) that to the network for training and testing. was shown to work better in a small-batch regime [50]. 3. We replaced max pooling operations in the encoder Network architecture of the network by learnable downsampling blocks (Fig. 1, red blocks), which consist of one 3 × 3 × The widely used 3D U-Net model [4] served as the basis 3 strided convolutional layer, the instance norm, the for our network design. Although not the main objective ReLU activation, and the scSE module. Similarly, we of the present work, we nonetheless introduced three implemented upsampling blocks in the decoder of the optimizations beyond the standard U-Net model: network using a 3 × 3 × 3 transposed convolution instead (Fig. 1, green blocks). To reduce memory 1. Original U-Net consists of conventional convolutional consumption and increase the receptive field of the blocks composed of a 3 × 3 × 3 convolution, a network, we implemented the first downsampling block normalization layer (e.g., batch norm), and a ReLU with a kernel size of 7 × 7 × 7 right after the input. activation function as a basic element of the network. The last convolutional layer followed by the sigmoid We chose to rely upon a residual block with full pre- activation function to produce the model output was activation [24] supplemented by a concurrent spatial applied with a kernel size of 1 × 1 × 1. and channel squeeze and excitation (scSE) module (Fig. 1, gray blocks). 2. An important part of the proposed architecture is the Experimental settings integration of squeeze and excitation (SE) blocks that aim at providing the option to compute weights for Data preprocessing the feature maps, so the network can put more or less attention on some of them. We implemented SE blocks The PET images exhibited a large variability of voxel sizes within the full pre-activation residual blocks, namely (see Table 1) that can adversely affect the model perfor- a specific modification called concurrent spatial SE mance since CNNs cannot natively interpret spatial dimen- (scSE) that has been shown to perform better for image sions with different scales. Therefore, we first interpolated segmentation tasks [48]. In order to include the scSE all PET images and corresponding segmentation masks to a module in the residual block, we followed the same common resolution of 4×4×2mm3 through the use of linear approach that was applied in SE-ResNet architectures interpolation. A slice thickness of 2 mm was chosen to retain Eur J Nucl Med Mol Imaging (2021) 48:3444–3456 3449 small image details that could be lost if interpolated at a of binary segmentation, the loss function for one training larger voxel size. Linear interpolation was chosen after com- example can be written as parison with other techniques including Nearest Neighbor N and B-spline, which led to decreased performance. 2 iyi ŷi + 1 L(y, ŷ) = 1 − N  2 (1) PET image intensities can exhibit a high variability in 2 i yi + N i ŷi + 1 both within image and between images. In order to reduce these variabilities and use them as the input for the CNNs, where yi ∈ {0, 1} is the the binary label for the i-th we applied Z-score normalization for each scan separately, voxel; ŷi ∈ [0, 1] predicted probability for the i-th voxel. with the mean and the standard deviation computed based Additionally, we applied Laplacian smoothing by adding +1 only on voxels with non-zero intensities corresponding to to the numerator and the denominator in the loss function the body region. to avoid the zero division in cases when only one class is represented in the training example. Data augmentation Multi-center cross-validation Due to the large variability in shapes, sizes, and heterogene- ity of tumor uptakes in PET images, data augmentation can Cross-validation is probably the simplest and most widely play a useful role in model training. To aid the model learn used method for estimating the expected prediction error of features invariant to affine transformations that are realistic, a model on an independent test sample [15]. Importantly, we applied mirroring on the axial plane, rotations in ran- cross-validation is based on the assumption that data dom directions with the angle uniformly sampled from the samples in the train and test folds are drawn from range [5, 15] degrees along the random set of axes, and scal- the same distribution. However, as already mentioned in ing with a random factor between 0.8 and 1.2. In order to section “Network architecture,” no standardization in the increase the diversity in lesion shapes, we relied on elas- acquisition or reconstruction protocols were implemented tic deformations. Gamma correction with γ sampled from across the five institutions in which the images were the uniform distribution between 0.8 and 1.2, and contrast collected. In addition, different PET/CT imaging devices stretching between 0 and 0.8–1.2 of the original range of with variable overall performance were used in these values was applied to adjust voxel intensity distributions. To centers. Therefore, in order to obtain a more reliable improve model robustness, we also added Gaussian noise to estimate of the model performance, we implemented 5-fold training samples. The standard deviation of the noise was cross-validation where each fold was composed only of equal to 0.1–0.2 standard deviations of the training sam- samples from one of the 5 centers. This simulated a “real- ple. All augmentation methods were applied independently life scenario” in which data from one or several centers are during model training with a probability of 0.2. used for training and evaluating a model that is then used in yet another center. For each cross-validation split of the Training procedure data, we randomly set aside 20% of training samples to tune hyperparameters of the model and to assess the model Due to the large size of PET images, we trained the model performance during training. on randomly extracted patches of 128 × 128 × 64 voxels with a batch size of 2. Since all PET images corresponded Evaluation metrics to the abdominopelvic cavity with a number of axial slices ranging from 77 to 192, the chosen patch size was large Aside from the DSC metric quantifying global volume enough to cover a significant part of the input PET image overlap, we used precision (a.k.a. positive predictive value) for all patients. P and recall R (a.k.a. sensitivity) to further evaluate model We trained the model for 400 epochs using Adam performance, as recommended by the TG211 [20], where optimizer with β1 = 0.9 and β2 = 0.99 for exponential DSC can be written as the harmonic mean of precision and decay rates for moment estimates. We applied a cosine recall: annealing schedule [36], gradually reducing the learning P ·R rate from lrmax = 10−4 to lrmin = 10−6 for every 25 DSC = 2 (2) P +R epochs and performing the adjustment at each epoch. Considering the fact that the Dice similarity coefficient Using these metrics, we compared our proposed network (DSC) is one of the most common metrics used for the to the standard U-Net (StdU-Net) as a baseline model. evaluation in medical image segmentation, we trained the In addition, a comparison was made with the use of a model with the Soft Dice Loss. Based on [40], in case fixed thresholding method (based on 40% of the maximum 3450 Eur J Nucl Med Mol Imaging (2021) 48:3444–3456 Table 2 Segmentation results obtained on the different test folds with the use of cross-validation Metrics Model Test fold Average Brest (n=69) Nantes (n=23) Montreal (n=26) Barcelona (n=24) Liège (n=90) DSC T 40 0.33 ± 0.36 0.57 ± 0.41 0.37 ± 0.31 0.22 ± 0.24 0.18 ± 0.22 0.33 ± 0.15 StdU-Net 0.68 ± 0.20 0.79 ± 0.12 0.77 ± 0.13 0.83 ± 0.10 0.79 ± 0.13 0.77 ± 0.05 Proposed 0.77 ± 0.15 0.81 ± 0.13 0.77 ± 0.21 0.84 ± 0.11 0.79 ± 0.13 0.80 ± 0.03 Precision T 40 0.30 ± 0.37 0.56 ± 0.43 0.29 ± 0.28 0.18 ± 0.21 0.14 ± 0.21 0.30 ± 0.16 StdU-Net 0.61 ± 0.24 0.75 ± 0.15 0.74 ± 0.16 0.81 ± 0.17 0.79 ± 0.18 0.74 ± 0.08 Proposed 0.69 ± 0.20 0.73 ± 0.16 0.77 ± 0.22 0.81 ± 0.18 0.77 ± 0.19 0.75 ± 0.05 Recall T 40 0.48 ± 0.43 0.74 ± 0.35 0.65 ± 0.40 0.38 ± 0.36 0.38 ± 0.37 0.52 ± 0.17 StdU-Net 0.88 ± 0.15 0.87 ± 0.10 0.85 ± 0.15 0.90 ± 0.09 0.84 ± 0.14 0.87 ± 0.02 Proposed 0.93 ± 0.10 0.96 ± 0.04 0.83 ± 0.19 0.91 ± 0.08 0.87 ± 0.14 0.90 ± 0.05 The proposed model was compared to the standard U-Net model and the fixed thresholding method in terms of DSC, precision and recall. The mean and standard deviation of each metric on the test folds are computed across corresponding data samples. Average results are reported across the test folds standardized uptake within the tumor, denoted from here the model architecture. For instance, we observed in our onwards as T40), still widely used in the literature despite experiments that applying b-spline interpolation for image its obvious limitations [20, 22]. resampling instead of linear interpolation deteriorated both models’ performance by an average of nearly 8.5% on the test folds. Results and discussion Both models achieved higher recall (between 0.83 and 0.93 on average) than precision (between 0.61 and 0.81) The results of all methods are summarized in Table 2 in all test folds, showing a trend in overestimating the and Supplementary Table 1. As expected, T40 obtained ground truth rather than underestimating it. One of the poor performance across all test folds compared to stdU- most challenging aspects pertaining to cervical cancer Net and the proposed model. On average, our proposed segmentation in PET images is to distinguish the tumor model outperformed its StdU-Net counterpart in terms of uptake from the adjacent bladder uptake. In all cases, DSC (0.80 vs. 0.77), with a slighlty smaller spread (0.03 even when the tumor was very close to the tumor, the vs. 0.05). The largest difference between the proposed proposed approach was capable of address this problem method and its standard counterpart was measured on the independently of the size, location, and shape of the tumor “Brest” test fold, where StdU-Net demonstrated relatively uptake (see examples in Fig. 3). However, the wider spread poorer performance (0.77 vs. 0.68). However, on the of results on the two largest test folds (Brest, Liège) (see other test folds, both models achieved closer results. The Fig. 2) could mean that the applied data augmentation superiority of the proposed model was due to a better techniques are not able to completely mimic all possible recall (0.90 vs. 0.87), whereas the difference in terms of variations in presented PET images and alternatives should precision was smaller (0.75 vs. 0.74). Kolmogorov-Smirnov be investigated, such as, for example, relying on realistic and Wilcoxon signed-rank tests both indicated that the simulated PET images to add more data for training. In difference in predictions of two models was significant (α = addition, due to our 5-fold evaluation scheme based on 0.05) only for all evaluation metrics on the Brest test fold, clinical centers, the size of training sets (and as a result and for recall on “Nantes” (see Supplementary Table 1). the variety of encountered examples) varied substantially This finding is in line with previous observations (e.g., holding out Liege yields 146 training cases whereas that, when properly tuned, the standard U-Net model holding out Nantes yields 209), which could also contribute can provide highly competitive results in many image in explaining these differences (Fig. 3). segmentation tasks, especially in medical imaging. For Analyzing predictions of the models, we identified a example, top-ranked results were obtained in recent number of outliers2 in each test fold (see examples in segmentation challenges using the ordinary U-Net model Fig. 4). When considering the DSC metric, the total number [28–30]. Under these circumstances, each step in the entire of outliers in the entire dataset was equal to 15 (12 for pipeline (data preprocessing, data augmentation, training procedure, etc.) may have a much larger impact on the 2A data point xi from a dataset X = {x1 , . . . , xn } is an outlier, if model performance than a careful or complex re-design of xi < q1 + 1.5(q3 − q1 ), where qi is the i-th empirical quartile of X. Eur J Nucl Med Mol Imaging (2021) 48:3444–3456 3451 (a) DSC (b) Precision (c) Recall Fig. 2 Box plots of the results on the test folds StdU-Net) and varies from 2 to 4 across the test folds. perfectly registered histopathological spatial information, In most cases, the model failed to accurately segment this is the best we can achieve with a single segmentation images with relatively small tumor regions (see Fig. 4a, d). method, which obviously provides imperfect results in a More specifically, 11 outliers corresponded to cases where small number of cases (for instance highly heterogeneous the tumor size was less than 200 voxels (i.e., 6.4 cm3 ), or very small and low contrast cases) [22]. An even whereas the average value across the dataset was 1160 better approach would consist in generating several manual voxels (i.e., 37.12 cm3 ). The other source of errors in the delineations by experts (at least three) in addition to model predictions is the presence of surrounding tissues the results of FLAB (other algorithms with proven good with a relatively high uptake that can be misclassified as the performance [22] could be added too) and generating a tumor (see Fig. 4b, c, e). statistical consensus of all these segmentation results. This The performance was affected by tumor volume (see would provide the proposed model an even more reliable Supplementary Table 2 and Supplementary Fig. 2): the ground truth to learn from, but it would be considerably lowest results were obtained in the first decile group3 with more time-consuming and tedious, especially for generating DSC = 0.56 compared to the performance obtained on the numerous manual delineations. Alternatively, our larger tumor volumes (significantly higher between 0.71 approach consisting in training the network on rigorously and 0.85). This happened due to precision that was steeply determined ground-truth masks could be reproduced by increasing along with the tumor size (0.44 to 0.90), whereas relying on other semi-automated methods with similar recall remained relatively stable (between 0.81 and 0.94). demonstrated levels of performance [20, 22]. Once trained, Examining the impact of the tumor contrast4 , we found the proposed network can be applied to new data that the proposed model demonstrated the worst results instantaneously, without the need for user intervention on low contrast images (see Supplementary Table 3 and beyond checking and validating the output result. Supplementary Fig. 3). The decile group with the lowest Unlike [5], we did not rely on any post-processing tumor contrast had DSC = 0.67 and recall = 0.77, techniques built on prior anatomical information. First, which were significantly different from the results on based on the segmentation results of the proposed model, it other groups (0.77 to 0.84, and 0.88 to 0.93, respectively). appears able to natively learn the anatomic position of the Investigating the relation between FIGO stages and the tumor relative to the bladder from training samples without model performance, we did not find significant differences additional a priori guidance. Second, the assumption about with DSC ranging from 0.75 to 0.82 (see Supplementary the tumor roundness contradicts numerous examples in our Table 4 and Supplementary Fig. 4). dataset, especially those with heterogeneous distributions It should be emphasized that although we used a (see Fig. 3b, e). Although in the present case we focused previously well-validated approach to define the ground- on PET-only delineation, the proposed model can be truth, this remains a surrogate of truth. In the absence of trained using multiple different modalities as input. It might be beneficial in specific cases, such as dealing 3 The first decile group corresponds to 10% of patients with the with small and/or low-contrast tumors, to extract additional smallest tumors. information from associated CT or MRI modalities. For 4 We roughly estimated tumor contrast as the ratio between average instance, in the context of the MICCAI 2020 Head and tumor intensity and average intensity of the body region. Neck Tumor segmentation challenge (HECKTOR), we 3452 Eur J Nucl Med Mol Imaging (2021) 48:3444–3456 (a) Brest (b) Nantes (c) Montreal (d) Barcelona (e) Liege (0.80, 0.70, 0.95) (0.84, 0.73, 0.98) (0.81, 0.73, 0.90) (0.88, 0.82, 0.94) (0.81, 0.77, 0.87) Fig. 3 Examples of the model predictions in each test fold. Axial slices. (first row) Input images, (second row) input images with ground truth segmentation, (last row) input images with predicted segmentation. Evaluation metrics for whole scans are provided in format (DSC, precision, recall) (a) Brest (b) Nantes (c) Montreal (d) Barcelona (e) Liege (0.34, 0.21, 0.87) (0.34, 0.20, 0.97) (0.16, 0.09, 0.90) (0.59, 0.42, 1.0) (0.40, 0.25, 1.0) Fig. 4 Examples of outliers in each test fold. Axial slices. (first row) Input images, (second row) input images with ground truth segmentation, (last row) input images with predicted segmentation. Evaluation metrics for whole scans are provided in format (DSC, precision, recall) Eur J Nucl Med Mol Imaging (2021) 48:3444–3456 3453 applied a similar U-Net-based model to delineate lesions in segmentation tasks in different medical imaging modalities combined PET/CT images, reaching 1st rank performance and could facilitate the deployment of fully automated with DSC of 0.76 [27]. However, the main challenge in radiomics pipelines. that case is to have a reliable ground truth determined on fused multimodal data, which could prove quite difficult Supplementary Information The online version contains supplemen- tary material available at https://doi.org/10.1007/s00259-021-05244-z. in the cervical region due to anatomical deformations and differences between PET and CT datasets.5 Funding A. Iantsen has received a PhD grant from the PREDICT With respect to our original objectives, our results Innovative Training Network funded by the Marie Sklodowska-Curie obtained with the use of multi-center cross-validation Actions, part of the EU’s Horizon 2020 Programme [grant agreement allow concluding that the designed model is able to no. 766276]. provide similar performance on PET images from different institutions and is robust to variations in scanner types, Declarations reconstruction algorithms, and post-processing methods. Ethics approval All procedures performed in studies involving human In addition, it allows for fully automated delineation of participants were in accordance with the ethical standards of the the tumor uptake without the need to exclude the bladder institutional and/or national research committee and with the 1964 uptake, either manually or through the incorporation of Helsinki declaration and its later amendments or comparable ethical additional prior information or constraints. standards. The retrospective collection of PET images from the five centers was approved by the following ethics committees: “The Institutional Review Board of Liège University Hospital” for images collected in Liège, “The Ethics Committee of the Hospital de la Santa Conclusion Creu i Sant Pau” for images collected in Barcelona, “The Institutional Review Board of Brest University Hospital” for images collected in Brest and Nantes, and “The McGill University Health Center Research In this work, we trained a modified U-Net model for Ethics Board” for those collected in McGill. fully automatic tumor uptake delineation in PET images in a multi-center context, without the need for additional Informed consent All patients provided signed permission for the use anatomical information or prior constraints. The ability of of their clinical data for scientific purposes and informed consent for the proposed model to learn and perform well for this the anonymous publication of data. task was demonstrated in PET images of 232 patients collected from five institutions. The ground-truth labels Conflict of interest The authors have no competing interests. for all patients were generated by experts with the use Open Access This article is licensed under a Creative Commons of a semi-automated algorithm, to reduce observer-related Attribution 4.0 International License, which permits use, sharing, variability and to avoid relying on manual delineations. adaptation, distribution and reproduction in any medium or format, as We presented a versatile pipeline that includes appropriate long as you give appropriate credit to the original author(s) and the data preprocessing and augmentation, design of the model source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in architecture beyond the standard U-Net model, and an this article are included in the article’s Creative Commons licence, optimized training procedure. We mimicked a typical unless indicated otherwise in a credit line to the material. If material clinical scenario and conducted all experiments in a multi- is not included in the article’s Creative Commons licence and your center context. The designed model obtained good average intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from accuracy for all considered institutions with very small the copyright holder. To view a copy of this licence, visit http:// standard deviation (DSC of 0.80 ± 0.03) without requiring creativecommonshorg/licenses/by/4.0/. any change in the pipeline. It slightly improved accuracy over the standard U-Net model, although both approaches provided good results and largely outperformed the fixed References threshold approach. The described approach managed to avoid including the bladder uptake in the resulting 1. Arbyn M, Weiderpass E, Bruni L, de Sanjosé S, Saraiya segmentation without the need for additional anatomical M, Ferlay J, Bray F. Estimates of incidence and mortality information (for instance using the CT image) or priors of cervical cancer in 2018: a worldwide analysis. Lancet Glob Health. 2020;8(2):191–203. such as shape constraints, and can therefore achieve fully 2. Aristophanous M, Penney BC, Martel MK, Pelizzari CA. A automated delineation of the tumor uptake without the gaussian mixture model for definition of lung tumor volumes need for any user intervention. It can be implemented in positron emission tomography. Med Phys. 2007;34(11):4223– with minimal modifications to solve a variety of other 4235. 3. Blanc-Durand P, Van Der Gucht A, Schaefer N, Itti E, Prior JO. Automatic lesion detection and segmentation of 18F-FET PET 5 In the HECKTOR challenge, manual delineation performed by a in gliomas: A full 3D U-Net convolutional neural network study. single expert was used as the ground truth. PLoS One. 2018. 3454 Eur J Nucl Med Mol Imaging (2021) 48:3444–3456 4. Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger lung cancer tumor delineation. J Nucl Med. 2011;52(11):1690– O. 3D U-Net: Learning dense volumetric segmentation from 1697. sparse annotation. In: International Conference on Medical Image 20. Hatt M, Lee JA, Schmidtlein C, Naqa I, Caldwell C, De Computing and Computer-Assisted Intervention, pp 424–432. Bernardi E, Lu W, Das S, Geets X, Gregoire V, Jeraj R, 2016. MacManus M, Mawlawi O, Nestle U, Pugachev A, Schöder 5. Chen L, Shen C, Zhou Z, Maquilan G, Albuquerque K, Folkert H, Shepherd T, Spezi E, Visvikis D, Zaidi H, Kirov MR, Wang J. Automatic PET cervical tumor segmentation by A. Classification and evaluation strategies of auto-segmentation combining deep learning and anatomic prior. Phys Med Biol. approaches for PET: Report of AAPM task group no. 211. Med 2019. Phys. 2017;44(6):1–42. 6. Coudray N, Ocampo PS, Sakellaropoulos T, Narula N, 21. Hatt M, Laurent B, Fayad H, Jaouen V, Visvikis D, Le Rest CC. Snuderl M, Fenyö D, Moreira AL, Razavian N, Tsirigos Tumour functional sphericity from PET images: prognostic value A. Classification and mutation prediction from non-small cell in NSCLC and impact of delineation method. Eur J Nucl Med Mol lung cancer histopathology images using deep learning. Nat Med. Imaging. 2018;45(4):630–641. 2018;24(10):1559–1567. 22. Hatt M, Laurent B, Ouahabi A, Fayad H, Tan S, Li L, Lu 7. Desseroit M, Visvikis D, Tixier F, Majdoub M, Perdrisot W, Jaouen V, Tauber C, Czakon J, Drapejkowski F, Dyrka W, R, Guillevin R, Cheze Le Rest C, Hatt M. Development of a Camarasu-Pop S, Cervenansky F, Girard P, Glatard T, Kain M, nomogram combining clinical staging with (18)f-FDG PET/CT Yao Y, Barillot C, Visvikis D. The first MICCAI challenge on image features in non-small-cell lung cancer stage i-III. Eur J Nucl PET tumor segmentation. Med Image Anal. 2018;44:177–195. Med Mol Imaging. 2016;43(8):1477–1485. 23. Hatt M, Le Rest C, Tixier F, Badic B, Schick U, Visvikis 8. Dolz J, Gopinath K, Yuan J, Lombaert H, Desrosiers C, Ben D. Radiomics: d are also images. J Nucl Med. 2019;60(Suppl Ayed I. HyperDense-Net: a hyper-densely connected CNN for 2):38–44. multi-modal image segmentation. IEEE Transactions on Medical 24. He K, Zhang X, Ren S, Sun J. Identity mappings in deep residual Imaging. 2018. networks. In: European conference on Computer Vision (ECCV). 9. El Naqa I, Yang D, Apte A, Khullar D, Mutic S, Zheng 2016. J, Bradley JD, Grigsby P, Deasy J. Concurrent multimodality 25. Hu J, Shen L, Albanie S, Sun G, Wu E. Squeeze-and-excitation image segmentation by active contours for radiotherapy treatment networks. arXiv preprint arXiv:170901507. 2017. planning. Med Phys. 2007;34(12):4738–4749. 26. Huang B, Chen Z, Wu PW, Ye Y, Feng ST, Wong CYO, Zheng 10. Gong K, Guan J, Liu CC, Qi J. PET image denoising using a L, Liu Y, Wang T, Li Q, Huang B. Fully automated delineation deep neural network through fine tuning. IEEE Transactions on of gross tumor volume for head and neck cancer on PET-CT using Radiation and Plasma Medical Sciences. 2019;3(2):153–161. deep learning. A dual-center study. Contrast Media & Molecular 11. Grootjans W, Usmanij EA, Oyen WJ, van der Heijden EH, Imaging. 2018. Visser EP, Visvikis D, Hatt M, Bussink J, de Geus-Oei 27. Iantsen A, Visvikis D, Hatt M. Squeeze-and-excitation L. Performance of automatic image segmentation algorithms for normalization for automated delineation of head and neck primary calculating total lesion glycolysis for early response monitoring tumors in combined PET and CT images. HECKTOR 2020. in non-small cell lung cancer patients during concomitant In: Andrearczyk V, Oreiller V, and Depeursinge A, editors. chemoradiotherapy. Radiother Oncol. 2016;119(3):473–479. Switzerland: Springer Nature; 2021. p. 1–7. 12. Guo Z, Guo N, Gong K, Zhong S, Li Q. Gross tumor volume 28. Isensee F, Maier-Hein KH. An attempt at beating the 3D U-Net. segmentation for head and neck cancer radiotherapy using deep arXiv preprint arXiv:190802182. 2019. dense multi-modality network. Phys Med Biol 64(20). 2019. 29. Isensee F, Kickingereder P, Wick W, Bendszus M, Maier-Hein 13. Guo Z, Li X, Huang H, Guo N, Li Q. Deep learning-based image KH. No New-Net. arXiv preprint arXiv:180910483. 2018. segmentation on multimodal medical imaging. IEEE Transactions 30. Isensee F, Jaeger PF, Kohl S, Petersen J, Maier-Hein on Radiation and Plasma Medical Sciences. 2019;3(2):162–169. KH. nnU-Net: a self-configuring method for deep learning-based 14. Hanzouli-Ben Salah H, Lapuyade-Lahorgue J, Bert J, Benoit biomedical image segmentation. Nat Methods. 2020. D, Lambin P, Van Baardwijk A, Monfrini E, Pieczynski 31. Kocak B, Ates E, Durmaz ES, Ulusan MB, Kilickesmez W, Visvikis D, Hatt M. A framework based on hidden Markov O. Influence of segmentation margin on machine learning- trees for multimodal PET/CT image co-segmentation. Med Phys. based high-dimensional quantitative CT texture analysis: a 2017;44(11):5835–5848. reproducibility study on renal clear cell carcinomas. Eur Radiol. 15. Hastie T, Tibshirani R, Friedman J. The elements of statistical 2019;29(9):4765–4775. learning: Data Mining, Inference, and Prediction. Springer 32. Lee H, Lee J, Kim H, Cho B, Cho S. Deep-neural-network-based Science & Business Media. 2009. sinogram synthesis for sparse-view CT image reconstruction. 16. Hatt M, Cheze le Rest C, Turzo A, Roux C, Visvikis IEEE Transactions on Radiation and Plasma Medical Sciences. D. A fuzzy locally adaptive Bayesian segmentation approach 2019;3(2):109–119. for volume determination in PET. IEEE Trans Med Imaging. 33. Leung KH, Marashdeh W, Wray R, Ashrafinia S, Pomper MG, 2009;28:881–893. Rahmim A, Jha AK. A physics-guided modular deep-learning 17. Hatt M, Cheze le Rest C, Descourt P, Dekker A, De Ruysscher based automated framework for tumor segmentation in PET. arXiv D, Oellers M, Lambin P, Pradier O, Visvikis D. Accurate preprint arXiv:14090473. 2020. automatic delineation of heterogeneous functional volumes in 34. Li L, Lu W, Tan Y, Tan S. Variational PET/CT tumor co- positron emission tomography for oncology applications. Int J segmentation integrated with PET restoration. IEEE Transactions Radiat Oncol Biol Phys. 2010;77(1):301–308. on Radiation and Plasma Medical Sciences. 2019;4(1):37–49. 18. Hatt M, Cheze-le Rest C, Albarghach N, Pradier O, Visvikis D. 35. Litjens G, Tand Bejnordi KBE, Setio A, Ciompi F, Ghafoorian PET functional volume delineation: a robustness and repeatability M, van der Laak J, van Ginneken B, Sánchez C. A survey study. Eur J Nucl Med Mol Imaging. 2011;38(4):663–672. on deep learning in medical image analysis. Med Image Anal. 19. Hatt M, Cheze-le Rest C, van Baardwijk A, Lambin 2017;42:60–68. P, Pradier O, Visvikis D. Impact of tumor size and tracer 36. Loshchilov I, Hutter F. SGDR: Stochastic gradient descent with uptake heterogeneity in (18)F-FDG PET and CT non-small-cell warm restarts. arXiv preprint arXiv:160803983. 2016. Eur J Nucl Med Mol Imaging (2021) 48:3444–3456 3455 37. Lucia F, Visvikis D, Vallières M, Desseroit M, Miranda O, 46. Reuzé S, Orlhac F, Chargari C, Nioche C, Limkin E, Riet F, Robin P, Bonaffini PA, Alfieri J, Masson I, Mervoyer A, Rein- Escande A, Haie-Meder C, Dercle L, Gouy S, Buvat I, Deutsch hold C, Pradier O, Hatt M, Schick U. External validation of a E, Robert C. Prediction of cervical cancer recurrence using combined PET and MRI radiomics model for prediction of recur- textural features extracted from 18f-FDG PET images acquired rence in cervical cancer patients treated with chemoradiotherapy. with different scanners. Oncotarget. 2017;8(26):43169–43179. Eur J Nucl Med Mol Imaging. 2019;46(4):864–877. 47. Ronneberger O, Fischer P, Brox T. U-Net: convolutional 38. Majdoub M, Hoeben BAW, Troost EGC, Oyen WJG, networks for biomedical image segmentation. In: International Kaanders JHAM, Cheze Le Rest C, Visser EP, Visvikis conference on medical image computing and computer-assisted D, Hatt M. Prognostic value of head and neck tumor intervention, pp 234–241. 2015. proliferative sphericity from 3’-deoxy-3’-[18F] fluorothymidine 48. Roy AG, Navab N, Wachinger C. Concurrent spatial and positron emission tomography. IEEE Transactions on Radiation channel ‘squeeze & excitation’ in fully convolutional networks. and Plasma Medical Sciences. 2018;2:33–40. In: International conference on medical image computing and 39. Martins S, Bragantini J, Falcão A, Yasuda C. Atlas-based computer-assisted intervention, pp 421–429. 2018. multiorgan segmentation for dynamic abdominal PET. Med Phys. 49. Supiot S, Rousseau C, Dore M, Cheze-Le-Rest C, Kandel- 2019;46(11):4940–4950. Aznar C, Potiron V, Guerif S, Paris F, Ferrer L, Campion L, 40. Milletari F, Navab N, Ahmadi SA. V-Net: fully convolutional neu- Meingan P, Delpon G, Hatt M, Visvikis D. Evaluation of tumor ral networks for volumetric medical image segmentation. In: Fourth hypoxia prior to radiotherapy in intermediate-risk prostate cancer International conference on 3D Vision (3DV), pp 565–571. using 18F-fluoromisonidazole PET/CT: a pilot study. Oncotarget. 2016. 2018;9(11):10005–10015. 41. Oktay O, Schlemper J, Le Folgoc L, Lee M, Heinrich M, 50. Wu Y, He K. Group normalization. arXiv preprint Misawa K, Mori K, McDonagh S, Hammerla NY, Kainz B, arXiv:180308494. 2018. Glocker B, Rueckert D. Attention U-Net: learning where to look 51. Zhang X, Zhong L, Zhang B, Zhang L, Du H, Lu L, Zhang for the pancreas. arXiv preprint arXiv:180403999. 2018. S, Yang W, Feng Q. The effects of volume of interest delineation 42. Pfaehler E, Beukinga R, de Jong JR, Slart RJA, Slump on MRI-based radiomics analysis: evaluation with two disease CH, Dierckx RA, Boellaard R. Repeatability of 18 f-FDG PET groups. Cancer Imaging 19(1). 2019. radiomic features: a phantom study to explore sensitivity to image 52. Zhao L, Lu Z, Jiang J, Zhou Y, Wu Y, Feng Q. Automatic reconstruction settings, noise, and delineation method. Med Phys. nasopharyngeal carcinoma segmentation using fully convolutional 2019;46(2):665–678. networks with auxiliary paths on dual-modality PET-CT images. 43. Qin C, Schlemper J, Caballero J, Price AN, Hajnal JV, Rueckert J Digit Imaging. 2019;32(3):462–470. D. Convolutional recurrent neural networks for dynamic MR image 53. Zhao X, Li L, Lu W, Tan S. Tumor co-segmentation in reconstruction. IEEE Trans Med Imaging. 2019;38(1):280–290. PET/CT using multi-modality fully convolutional neural network. 44. Ramon AJ, Yang Y, Pretorius PH, Johnson KL, King Phys Med Biol. 2018;64(1):015011. https://doi.org/10.1088/1361- MA, N WM. Improving diagnostic accuracy in low-dose SPECT 6560/aaf44b. myocardial perfusion imaging with convolutional denoising 54. Zhong Z, Kim Y, Zhou L, Plichta K, Allen B, Buatti J, networks. IEEE Trans Med Imaging. 2020;39(9):2893–2903. Wu X. 3D fully convolutional networks for co-segmentation of https://doi.org/10.1109/TMI.2020.2979940. tumors on PET-CT images. Proc IEEE Int Symp Biomed Imaging. 45. Ren S, Laub P, Lu Y, Naganawa M, Carson RE. Atlas- 2018;52(11):228–231. based multiorgan segmentation for dynamic abdominal PET. IEEE Transactions on Radiation and Plasma Medical Sciences. Publisher’s note Springer Nature remains neutral with regard to 2019;4(1):50–62. jurisdictional claims in published maps and institutional affiliations. 3456 Eur J Nucl Med Mol Imaging (2021) 48:3444–3456 Affiliations Andrei Iantsen1 · Marta Ferreira2 · Francois Lucia1 · Vincent Jaouen1 · Caroline Reinhold3 · Pietro Bonaffini3 · Joanne Alfieri4 · Ramon Rovira5 · Ingrid Masson6 · Philippe Robin7 · Augustin Mervoyer6 · Caroline Rousseau8 · Frédéric Kridelka9 · Marjolein Decuypere9 · Pierre Lovinfosse10 · Olivier Pradier1 · Roland Hustinx2 · Ulrike Schick1 · Dimitris Visvikis1 · Mathieu Hatt1 1 LaTIM, INSERM, UMR 1101, University Brest, Brest, France 2 GIGA-CRC in vivo Imaging, University of Liège, Liège, Belgium 3 Department of Radiology, McGill University Health Centre (MUHC), Montreal, Canada 4 Department of Radiation Oncology, McGill University Health Centre (MUHC), Montreal, Canada 5 Gynecology Oncology and Laparoscopy Department, Hospital de la Santa Creu i Sant Pau, Barcelona, Spain 6 Department of Radiation Oncology, Institut de Cancérologie de l’Ouest (ICO), Nantes, France 7 Nuclear Medicine Department, University Hospital, Brest, France 8 Nuclear Medicine Department, Institut de Cancérologie de l’Ouest (ICO), Nantes, France 9 Division of Oncological Gynecology, University Hospital of Liège, Liège, Belgium 10 Division of Nuclear Medicine and Oncological Imaging, University Hospital of Liège, Liège, Belgium