This is the accepted version of the journal article: Gil Resina, DeboraGil, Debora; Esteban Lansaque, Antonio; Borràs, Agnés; [et al.]. «Enhancing virtual bronchoscopy with intra-operative data using a multi-objective GAN». International Journal of Computer Assisted Radiology and Surgery, Vol. 14, Suppl. 1 (June 2018), p. S1- S194. This version is available at https://ddd.uab.cat/record/257867 under the terms of the license Enhancing virtual bronchoscopy with intra-operative data using a multi-objective GAN D.Gil1 ,A.Esteban-Lansaque1 , A.Borras1 and C.Sánchez1 1. Purpose maps are given by auto-encoders trained to opti- mize: State of the art methods in computer vision need huge amounts of data with unambiguous annota- `(Gr , Gv , Dr , Dv ) = (2) tions for their training. In the context of medical `GAN (Gv , Gr , Dr , Dv , V, R) + λ`cyc (Gv , Gr ) imaging this is, in general, a very difficult task due to limited access to clinical data, the time required The term `GAN measures how good are Gv , Gr for manual annotations and variability across ex- transferring images from one domain to the other perts. The particular field of intervention guiding one, while `cyc is a ”cycle consistency loss” intro- has the extra difficulty of intra-operative recordings duced to force bijective mappings. probably requiring the alteration of standard pro- The minimization problem is solved by adversar- tocols. ial training as: Virtual endoscopy could be used to train classi-   fiers and validate image processing methods if its G∗r , G∗v = min max `(Gr , Gv , Dr , Dv ) (3) Gr ,Gv Dr ,Dv appearance was comparable (in texture and color) to the actual appearance of intra-operative record- In this manner, G∗r and G∗v are optimized so that ings. Modern techniques for artistic style trans- Gr , Gv minimize (2) while the adversarial Dr , Dv fer could be used to endow virtual endoscopic im- maximize it. ages with the content and texture of intra-operative We propose to consider separately the optimiza- videos provided that stylized images preserved the tion of each of the terms in ` and pose adversarial anatomical structure of, both, intra-operative im- training as the following multi-objective optimiza- ages and simulated data. tion [2] problem: This work addresses the generation of realistic en- doscopic images using intra-operative data to aug- G∗r , G∗v = min (`1 , `2 ) = min (`cyc , `GAN ) = (4) Gr ,Gv Gr ,Gv ment the appearance of virtual endoscopy. In par- ticular, we use cycleGAN [1] in a multi-objective min (`cyc (Gr , Gv ), max `GAN (Gv , Gr , Dr , Dv )) Gr ,Gv Dr ,Dv optimization scheme to map virtual images into the intra-operative domain preserving their anatomical The solution to (4) is computed from the Pareto content. front [2] consisting in the set of dominating config- urations that outperform in any of the objectives without degrading at least one of the other ones. 2. Methods The set of Pareto epochs achieve the best trade- off between the objective functions and, thus, are Given two domains V irtual, V , and Real, R, equivalent from the point of view of the cycleGAN. style transfer learns two (bijective) maps (Gr , Gv ) The epoch from the Pareto front best suited for aug- from one domain onto the other one: mentation of virtual endoscopic images is selected as the one that minimizes the L2 -difference: Gr : V irtual → Real Gv : Real → V irtual (1) `Cont = mean (||v − Gr (v)||2 ) + (5) v∈V with the map composition Gr (Gv ) and Gv (Gr ) be- ing the identity on each domain. Following [1], mean (||r − Gv (r)||2 ) r∈R Preprint submitted to Journal of LAT EX Templates March 30, 2022 Figure 1: Virtual bronchoscopy enhanced using our method, (a), 200th epoch network, (b) and the least cost one, (c). for mean, mean denoting the average values on the preservation of virtual anatomical content. To val- v∈V r∈R training sets of virtual, V , and real, R, images. idate the visual quality of virtual bronchoscopies enhanced using this network, we used simulations generated from 4 additional CT scans acquired from 3. Results a set of different patients. Simulations were also enhanced using the 200th epoch network and the network achieving the least value of the cost (2). We trained our method on a set of 5 ultrathin Figure 1 shows representative images of virtual intra-operative recordings (defining the set of im- bronchoscopies enhanced using our method (fig.1 ages of the Real domain, R) and 5 virtual broncho- (a)), the 200th epoch network (fig.1 (b)) and the scopies (defining the set of images of the V irtual least cost one (fig.1 (c)). For each case, we show domain, V ). Intra-operative videos were acquired two consecutive frames of the enhanced virtual se- during biopsy interventions at Hospital Bellvitge quence which should be very similar in appearance (Barcelona, Spain) using an Olympus Exera III and content. Least cost images have sudden dark HD Ultrathin videobronchoscope. Virtual bron- artifacts, while the 200th epoch yields highly un- choscopies were generated using an own developed stable images that do not always match the origi- software from CT scans acquired for different pa- nal anatomy. The GAN trained using the proposed tients with an Aquilion ONE (Toshiba Medical Sys- multi-objective strategy provides a stable appear- tems, Otawara, Japan) using slice thickness and in- ance in images which are the most consistent with terval of 0.5 and 0.4 mm respectively. For each the original anatomical content of virtual images. intra-operative video or virtual simulation, we ran- domly selected 500 images for training cycleGAN from scratch. 4. Conclusions After 200 epochs, our multi-objective approach selected epoch 50 as the one achieving the best Visual inspection indicates that the proposed compromise between intr-operative appearance and multi-objective approach to GANs improves the 2 anatomical content of stylized virtual videos keep- ing an intra-operative appearance. Future quanti- tative analysis and observations will be conducted to analyze the performance of the multi-objective GAN in detail. Acknowledgment The research leading to these results has re- ceived funding from the European Unions Hori- zon 2020 research and innovation programme un- der the Marie Sk?odowska-Curie grant agreement No 712949 (TECNIOspring PLUS) and from the Agency for Business Competitiveness of the Gov- ernment of Catalonia. References [1] J.-Y. Zhu, T. Park, P. Isola, A. A. Efros, Unpaired image-to-image translation using cycle-consistent adver- sarial networks, arXiv preprint arXiv:1703.10593. [2] K. Miettinen, Nonlinear multiobjective optimization, Vol. 12, Springer Science & Business Media, 2012. 3