Received July 10, 2020, accepted August 7, 2020, date of publication August 21, 2020, date of current version September 14, 2020. Digital Object Identifier 10.1109/ACCESS.2020.3018590 Intraoperative Extraction of Airways Anatomy in VideoBronchoscopy DEBORA GIL , ANTONIO ESTEBAN-LANSAQUE, AGNÉS BORRÀS, ESMITT RAMÍREZ, AND CARLES SÁNCHEZ RAMOS Computer Vision Center, Department of Computer Science, Universitat Autònoma de Barcelona, 08193 Bellaterra, Spain Corresponding authors: Debora Gil (debora@cvc.uab.cat) and Carles Sánchez Ramos (csanchez@cvc.uab.cat) This work was supported in part by Spanish projects under Grant RTI2018- 095209 -B-C21 and Grant FIS-G64384969, in part by the Generalitat de Catalunya under Grant 2017-SGR-1624, in part by the CERCA-Programme, in part by the European Union Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie under Grant 712949 (TECNIOspring PLUS), and in part by the Agency for Business Competitiveness of the Government of Catalonia.. The work of Debora Gil was supported by Serra Hunter Fellow. The work of Esmitt Ramírez was supported by the Ministry of Economy, Industry and Competitiveness, Spain, under Grant BES-2016-078042. ABSTRACT A main bottleneck in bronchoscopic biopsy sampling is to efficiently reach the lesion navigating across bronchial levels. Any guidance system should be able to localize the scope position during the intervention with minimal costs and alteration of clinical protocols. With the final goal of an affordable image-based guidance, this work presents a novel strategy to extract and codify the anatomical structure of bronchi, as well as, the scope navigation path from videobronchoscopy. Experiments using interventional data show that our method accurately identifies the bronchial structure. Meanwhile, experiments using simulated data verify that the extracted navigation path matches the 3D route. INDEX TERMS Bronchial anatomy representation, videobronchoscopy, lung cancer, biopsy guidance. I. INTRODUCTION The 30% undiagnosed pulmonary lesions need CT follow-up Lung cancer early-stage detection increases the survival or futile surgery procedures such as thoracoscopies, which rate over 5-years from 38% to 67% [1]. Currently, can- induces patient anxiety, radiation exposure, invasive surgery cer diagnosis can only be achieved by analysis of tissue along with associated pain, disability and rarely death. Diag- extracted from the lesion usually sampled using ultrathin nostic yield could be improved reducing radiation and costs bronchoscopic navigation. Biopsy sampling using videobron- by developing intervention support systems able to guide the choscopy is a two-stage procedure. First, the intervention bronchoscopist to the lesion. is planned off-line using Virtual Bronchoscopy (VB) [2] to During the past years, several technologies have been compute from computed tomography (CT) data the shortest developed for on-line guiding the bronchoscopist through path across bronchial levels to each nodule. Second, the bron- the planned path. Existing systems can be split into purely choscopist tries to reproduce the pre-planned route by visual image-based navigation systems and systems, like electro- identification of bronchial levels and branch orientation in the magnetic navigation, that use specific tools that provide addi- intra-operative bronchoscopy video. tional information helpful in the guidance process. Even for expert bronchoscopists it is difficult to reach distal Electromagnetic Navigation Bronchoscopy [4] (ENB) is a lesions due to the lung’s anatomical structure. Conventional medical procedure designed to localize and guide both bron- bronchoscopic diagnostic procedures are visually guided choscope and bronchoscopic tools through the bronchial tree using radiating fluoroscopy which renders a suboptimal 34% by means of electromagnetic waves. The main disadvantage of positive results for lesions <2 cm [3]. New endoscopy of ENB is that it increases the cost of interventions, lacks techniques (like electromagnetic navigation) are expensive, of rotational information and might not be accurate enough require either manual intervention or special gadgets, only due to interferences between the electromagnetic waves and increase diagnostic yield to 70%, and still radiate the patient. human tissues. Image-based navigation systems try to put into correspondence video bronchoscopy images and VB images The associate editor coordinating the review of this manuscript and using multimodal registration techniques [5]–[7]. Unfortu- approving it for publication was Carmelo Militello . nately, the synchronization of simulated navigation with the This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ 159696 VOLUME 8, 2020 D. Gil et al.: Intraoperative Extraction of Airways Anatomy in VideoBronchoscopy actual intra-operative video is a challenging task that often a representation of the full anatomy navigated during the requires manual corrections during intervention. intervention. This work contributes to the identification of the In recent years, several alternatives to classical intensity- scope position during intervention in two aspects: based registration methods have begun to be explored. These 1) Intra-operative Extraction of the Anatomy Explo- new image-based methods include Simultaneous Localiza- tion and Mapping (SLAM) [6], [8], Convolutional Neural red. The anatomy of airways can be represented using Networks (CNNs) to compute 3D point clouds from depth a tree data structure with nodes representing bronchi maps learned from images [9] and synchronization based on branching points [10]. In bronchoscopy videos, airways anatomical structure is projected into a collection of visual features that can be identified in, both, CT scans an luminal regions arranged in a hierarchy of inclusions in videos [10], [11]. SLAM methods estimate a 3D map of the unknown envi- case frames show different bronchial levels [11]. The ronment together with the camera position from a set of fea- tracking of such hierarchy across the video defines a ture points matched in consecutive frames. A main challenge tree that represents the global anatomy of the patient observed during the intervention. In order that this in endoscopic explorations is the matching step, since images on-line exploration tree encodes all the bifurcations lack of enough texture and salient points. In this context, traversed during the intervention, its nodes codify the ORBSLAM [6] achieves higher performance with respect to other state-of-the-art monocular SLAM approaches because traversed luminal regions as ellipses and its edges the it uses feature detectors adapted to the specific surgical condi- branching levels hierarchy. tions of endoscopic interventions. ORBSLAM performance 2) Codification of the Scope Navigation Path. Nodes also keep a flag to indicate whether they are currently in bronchoscopic interventions is not clear due to the specific tracked, in order to identify the anatomy that it is scope forward-backward motion along the camera viewpoint observed at each time. The intra-operative navigation direction and sudden rapid motions. In a recent work [8], the authors presented an improved version of VSLAM to path is codified as the list of the roots of the sub-tree improve point matching in case of sudden motion of the representing such observed anatomy. Using this rep- bronchoscope. A main inconvenience of SLAM approaches resentation, the current position of the scope inside airways could be matched to the anatomical structure as navigation systems is that they still require registration of the lung extracted from a CT [10], [14]. of the point cloud to a segmentation of bronchi in order to localize the scope inside airways. Despite excellent results of CNNs in the medical imag- II. EXTRACTION OF THE ON-LINE EXPLORATION TREE ing domain [12], [13], a main inconvenience is the limited Our strategy for the extraction of the anatomy observed in annotated data available. In the particular field of interven- a bronchoscopy (sketched in fig.1) has 3 main steps. The tion guiding, gathering annotated data has the extra diffi- first step is to track the anatomy observed in a single frame culty of intra-operative recordings probably requiring the across the video (fig.1 (1)). To do so, the anatomical hierarchy alteration of standard protocols. The deep learning approach of ellipses extracted using [11] is matched to the current presented in [9] uses phantom synthetic data to train a on-line tree using a measure of anatomical similarity based deep SLAM that provides the depth of the bronchoscopic on ellipses overlap. Second, the exploration tree is updated image. Like classic SLAM approaches, the position of depending on the type of match between the tracked tree and the camera inside airways requires further processing of the image hierarchy (fig.1 (2)). In case all ellipses in the the data. hierarchy are matched, we consider that the scope has not A feasible alternative which does not require huge anno- change the bronchial level and update the anatomical infor- tated datasets in order to train complex methods is the use of mation of the nodes with the ellipse hierarchy parameters they anatomical landmarks [10]. In [10], lumen center lines were have been matched to. In case of a partial match between used to codify the route followed by the bronchoscope and one of the tracked node children and the ellipse root of the indicate the path that needs to be followed to reach a target hierarchy, we consider that the scope is approaching a deeper lesion. bronchial level and add a new level to the on-line tree. Finally, A navigation system based on anatomical landmarks the exploration path is dynamically obtained as the sequence requires the codification of airways main anatomy in, both, of tracked roots of the on-line tree (fig.1 (3)). In case of adding intra-operative videos and CT-scans. This work focuses in the a new tree level to the current tracked root, we have a forward intra-operative identification of a route planned on CT-scans motion (F) of the scope and the current tracked root is added using a codification of airways main anatomy. to the exploration path. In case the children of the current This work extends [11] to obtain the bronchial path fol- tracked root are unmatched, we consider the scope moves lowed during a bronchoscopic exploration. In [11] bronchial backwards (B) and the root is removed from the exploration anatomy was encoded in single videobronchoscopy frames path. as a hierarchy of ellipses representing the bronchial lev- The next sections give details about the computation of els observed in each frame. In the presented work we each of the steps required for the extraction of the on-line track such hierarchy across frames to dynamically extract exploration tree. VOLUME 8, 2020 159697 D. Gil et al.: Intraoperative Extraction of Airways Anatomy in VideoBronchoscopy FIGURE 1. Main steps in the extraction of the on-line exploration tree. 159698 VOLUME 8, 2020 D. Gil et al.: Intraoperative Extraction of Airways Anatomy in VideoBronchoscopy i i i i i A. ANATOMY TRACKING parameters (xj j , yjj , ajj , bjj , θj j ), j = 1, 2, then its elliptical Each node of the exploration tree stores the shape and position i region, namely Enjj (x, y), fulfills the following inequality: in the image of the luminal region it represents. Luminal shape is codified by the parameters of an ellipse representing i (x cos(θj j )−y sin(θj j )) − xj j i i i the luminal region: center position, (x, y), mayor and minor Enjj (x, y) = i axis, (a, b), and its orientation, θ. Such ellipses are obtained ajj from a hierarchy of MSER regions computed using [11]. The i i i (y cos(θj j ) + x sin(θj j )) − yjj lumen position in the image frame is codified by the image + ≤1 (2) i quadrant, Q, the ellipse representing the lumen is in. Image bjj quadrants are stored in order to provide guidance instructions indicating the bronchi to follow at each bronchial level. Nodes It follows that the VOE defining the similarity between ni11 also store temporal information. In particular, the number of and ni22 is given by: frames, NFr, it has been tracked and a boolean flag (labelled active) indicating whether the node corresponds to a lumen sim(ni11 , ni22 ) := Eni11 (x, y) ∩ Eni22 (x, y) that is currently being observed in the video frame (active = True) or it corresponds to a bifurcation seen in previous 2 · |(Eni11 (x, y) < 1) ∩ (Eni22 (x, y) < 1)| = (3) frames (active = False). A given node is activated when it |Eni11 (x, y) < 1| + |Eni22 (x, y) < 1| has been tracked (i.e. matched to an ellipse in the hierarchy of MSER regions) for at least NFrMx frames and deactivated where | · | indicates the number of pixels of the mask approx- otherwise. imating the ellipse area. In order to track (and update) the on-line exploration tree, the active nodes sub-tree is matched to the ellipse B. EXPLORATION TREE UPDATING hierarchy. To do so, we use a tree edit distance [15] that In order to update the exploration tree, we compute the takes into account airways anatomy given by their elliptical edit distance transformation between the active sub-tree, T1 , representation. defined by the active nodes and all their children and the The tree edit distance computes the optimal set of edit ellipse hierarchy encoded as a tree, T2 . The active sub-tree operations that transforms a tree T1 into a tree T2 . In the is updated depending on the edit operation (identity, delete or case of trees representing bronchial anatomy, aside the iden- insert) that it should be applied to transform it to the hierarchy tity transformation, we have two edit operations, insert and tree. delete. Node deletion corresponds to traversing a bronchial For those nodes having a match (identity edit operation) level, while node insertion indicates that the bronchoscope is to one of the ellipses in the hierarchy, we update all their approaching a deeper bronchial bifurcation. values. The anatomical parameters (ellipse and quadrant) are In order to define a criterion for the selection of the optimal set to the values of the ellipse they have been matched to. The transformation, each edit operation is assigned a cost which number of tracked frames, NFr, is increased by one and the is used to compute the total cost as the sum of the costs of all flag active is updated if NFr ≥ NFrMx . We observe that a full edit operations of the transformation. Then, the edit distance match of all nodes in T1 indicates that we are still in the same is the transformation of minimal cost. In our case, the identity bronchial level. has zero cost and the cost of the other edit operations (insert For those nodes in the active subtree that should be and delete) is set to 1. Nodes in T1 are deleted and inserted deleted, the anatomical information remains unchanged, depending on whether they are matched to a node in T2 or not. NFr, is decreased by one and the flag active is updated if A node in T1 is deleted if it can not be matched to any node NFr < NFrMx . A deactivation of the deleted nodes indicates in T2 . All unmatched nodes in T2 are, then, inserted into T1 . the possibility of a change in the bronchial level. If the Node matching is given in terms of a similarity measure. If we deactivated node is the root, the scope is approaching a deeper Nj bronchial level, while in case of deactivating the children, note by (nij )i=1 the nodes of tree Tj , j = 1, 2, and sim(·, ·) the the scope is moving backwards to a previous bronchial level. similarity measure, then ni11 is matched to ni22 if: We observe that in the first case, the active sub-tree could n o have two roots, one for each children of the deactivated ni22 = max sim(ni11 , ni2 ), for sim(ni11 , ni2 ) ≥ Th (1) i root. Finally, the unmatched nodes of the hierarchy are inserted being Th a tolerance parameter on a minimum similarity into the active sub-tree as new nodes with anatomical values between nodes. In case sim(ni11 , ni2 ) < Th, ∀i = 1, . . . N2 , equal to the values of the hierarchy nodes’ ellipses, NFr = 1 then ni11 is deleted. and active = False. We observe that the inserted nodes will In our case, the similarity between nodes is given by the be children of an active root and,, thus, indicate the scope is overlap between the ellipses they represent. This overlap is moving forward towards the next bronchial level of the active computed as the volumetric overlap error (VOE) between the root. The final exploration tree is given only by those nodes i masks of the nodes’ ellipses. If njj represents an ellipse with that have been active at least once. VOLUME 8, 2020 159699 D. Gil et al.: Intraoperative Extraction of Airways Anatomy in VideoBronchoscopy FIGURE 2. Definition of the exploration path from the active roots in forward motion. C. EXPLORATION PATH EXTRACTION The nodes of the active sub-tree are indicated with a blue The sequence of active roots provides information about the frame with the root in double line. We also show the ellipses current position of the scope, as well as, the path navigated so the the active sub-tree nodes represent in a bronchoscopy far. The exploration path is codified as a list of the nodes that image for better interpretation of the anatomical changes that have been single roots in the active sub-tree. The quadrants take place during level traversal. In fig.2 (a), the scope is of these nodes codify the navigation path. We add an extra placed at a point of Node2 bronchi that allows the visualiza- attribute to the nodes in the list in order to indicate the tion of the next level luminal areas, represented as the light direction of the scope motion at the moment of traversing the ellipses included in Node2 dark ellipse. As the scope moves bifurcation: F for forward motion, B for backwards motion. forward (fig.2(b)), Node2 lumen does not show any more in The position of the scope is given by the last node inserted in images and, thus, it is deactivated. At this moment, the active the exploration path list. In particular, the level of the last node sub-tree has two roots, Node4 and Node5. Finally, in fig.2(c), inside the whole exploration tree defines the current bronchial the scope has entered into Node5 bronchial level, so that this level. node becomes the only root of the active sub-tree and is added The exploration path list is computed as follows. The path to the path list. is initialized with the root (which represents the tracheal entry point) of the full exploration tree with forward motion. Nodes III. EXPERIMENTS are added to the list as new levels are traversed using the Our experiments have been designed to evaluate our method following criterion. In forward motion, each time the scope in two aspects: approaches a deeper level, two new children are added to the current active root which is the last node inserted in the 1) Assessment of the Exploration Tree. The extrac- path list. At the final approach phase, such root is deactivated tion of the on-line exploration tree has been tested and its children become active roots. The moment the scope on 8 interventional videos acquired at Hospital de traverses to the next bronchial level, only the root repre- Bellvitge (Barcelona, Spain) using an Olympus Exera senting the lumen the scope has entered into remains active. III HD Ultrathin videobronchoscope. Videos were In backwards motion, we would have the inverse sequence of acquired during 4 biopsy sampling procedures. For activations/desactivations. In any case, traversing a bronchial each procedure, 2 different videos were recorded, one level can be detected as an increment in the number of active navigating a lower lobe and the other one navigating an roots followed by a decrement. Every time this condition upper lobe. is satisfied, the current active root is added to the path list. In order to validate the extraction of the on-line explo- Forward and backward motion is determined depending on ration tree, a clinical expert visually inspected each of whether the added active root is a children or a parent of the interventional videos to create a Ground Truth (GT) the last inserted node. In case of being a children, we have tree with all the bronchi seen across the video. The a forward motion to a deeper level, while in case of being extracted on-line tree was compared to this GT tree a parent, the scope is moving backwards to a level already using a standard tree edit distance in order to com- visited. pute false positives, FP, false negatives, FN, preci- Figure 2 graphically sketches the process of adding a new sion and recall of the on-line exploration tree. False node to the exploration path list in case of forward motion. positives correspond to structures wrongly identified 159700 VOLUME 8, 2020 D. Gil et al.: Intraoperative Extraction of Airways Anatomy in VideoBronchoscopy as bronchial lumens, while false negatives are missed luminal regions. 2) Assessment of the Exploration Path. To assess the exploration path, we have computed the path list in VB to compared it to the list of bifurcations traversed by the VB camera [16]. Virtual bronchoscopies were generated using an own developed software from CT scans [16] and augmented with intra-operative appear- ance using the method presented in [17], [18]. For each FIGURE 3. Preprocessing for ORBSLAM input. Virtual image, left, and the processed for ORBSLAM, right. virtual bronchoscopy of a patient, four virtual explo- rations were generated, covering the four main lobes: left and right upper lobes, noted LUL, RUL, and left and right lower lobes, noted LLL, RLL. Exploration The computation of the exploration tree requires setting the paths reached between the sixth and twelfth bronchial values of the number of frames, NFrMx , for node activation level. Simulations were performed using central navi- and deactivation and the threshold, Th, determining node gation without rotation around the scope. matching in (1). The number of frames NFrMx was set to We have compared the quality of our exploration path NFrMx = 5, while the threshold was set to Th = 0.2 in order to the paths computed using [14]. Following [19], to manage sudden abrupt motions. the metrics used for the comparison of both methods Several parameters were tuned in order to find the optimal are True Positives Nodes (TPN) and True Path Rep- performance of the ORBSLAM. The algorithm is based on resentations (TPR). For a given exploration, a node the extraction of FAST keypoints and ORB descriptors. The is considered to be a TPN if its label coincides with keypoints are extracted following a pyramid scheme using the GT node label. The number of consecutive TPN different scales. The scaling factor between pyramid levels achieved from the 1st node divided by the path node is 1.2 and the number of levels used is 8. For each level, length defines TPR. We have used a T-test to detect the image is divided in a grid. At each cell, FAST keypoints significant differences across methods and Confidence are extracted imposing a minimum response. Firstly an initial Intervals (CIs) at significance α = 0.05 to report aver- threshold is imposed. The value of the initial threshold is age precision and recall ranges. iniThFAST = 9. If no corners are detected a lower value is Using the same metrics and database we also did a imposed. The minimum value imposed is minThFAST = 2. comparison with a SLAM state of the art method [6], which has been proven to perform well in scope track- TABLE 1. Exploration tree assessment. Quality numbers for each patient and video. ing in videobronchoscopy [8] In order to obtain the exploration path from ORBSLAM camera tracking, the path described by ORBSLAM camera 3D posi- tion was registered to the path described by the vir- tual camera which follows the center line of airways. A node representing a traversal of a bronchial level was considered correctly identified and considered TPN if the range of positions the virtual camera traverses a new bronchial level intersects the range of positions the ORBSLAM camera traverses the same bronchial level. Also, and with the aim of increasing the perfor- mance of ORBSLAM, we computed virtual explo- A. ASSESSMENT OF THE EXPLORATION TREE ration paths transformed to enhance edges and corners Table 1 reports, for each patient and path, false positives, of triangles. ORBSLAM needs to identify key points FP, false negatives, FN, the total number of nodes of the GT between frames to track the camera. Since our virtual exploration tree, precision and recall. The last row reports images have been computed to have an appearance as total numbers for the 8 paths. Our on-line exploration tree close as possible to intra-operative videos, there might has an overall recall of 92% and a precision of 82%, but its not be enough key points to a good performance of performance substantially varies across patients and videos. ORBSLAM even using the method described in [6]. Meanwhile Patient 2 and 3 reach 100% in precision with Thus, we did not apply [17] and used flat illumina- 90.7% of average recall, in the remaining cases the average tion in virtual simulations. The right image in figure 3 precision drops to 68.7% with only a slightly increase in shows an example of a frame transformed to enhance average recall (94.3%). In particular, Patient 4 has the lowest corners and edges in comparison to the realistic virtual precision scores, with only 46.67% in the first video. Most frame used to validate our method shown in the right FPs are due to shines in bubbles and sudden abrupt changes image of the figure. in scope motion, which mainly appear at most distal levels. VOLUME 8, 2020 159701 D. Gil et al.: Intraoperative Extraction of Airways Anatomy in VideoBronchoscopy Meanwhile FNs are mostly attributed to substantial deviation TABLE 4. Comparison of TPN and TPR obtained for ORBSLAM and the proposed on-line path. in central navigation, which is a main requirement for the assumption of inclusion of lumens from different bronchial levels. Table 2 reports the average, µ, standard deviation, σ , and CIs for the precision and recall. The confidence interval for the recall indicates that our algorithm detects most of the bronchi in all explorations (σ = 0.0759). In terms of method and ORBSLAM equal to [30.1183, 57.4840] and precision, its CI indicates that the precision varies depending [0.4922, 47.20] respectively, between the method proposed in on the patient due to the reasons explained in Table 1. this work and [6]. Further, we note that that ORSLAM inter- vals for TPN and TPR are the same. Both issues are due to the TABLE 2. Precision and recall statistics for the on-line exploration tree fact that ORBSLAM underestimates depth during navigation extraction. and, thus, the moment it does not reach a given level, it is unable to catch up and deeper levels are also missed. Depth underestimation could be attributed to tracking key points which are not good descriptors of bronchial anatomy. In fact, ORBSLAM is only able to detect the first 1-4 bifurcations on average. From the point of view of intervention guidance, TABLE 3. Comparison of TPN and TPR obtained for [14] and the proposed these levels are the least relevant ones since bronchoscopists on-line path. have little difficulties identifying proximal airways. Another issue influencing ORBSLAM performance is that, like other motion estimation methods, it requires a minimum amount of motion in consecutive frames to properly estimate camera position. Since motion is more pronounced at main airways, initial bifurcations are better reconstructed and reached by B. ASSESSMENT OF THE EXPLORATION PATH ORBSLAM. At deeper levels, scope motion is more subtle Table 3 reports CIs for average TPN and TPR percentages and, thus, ORBSLAM systematically underestimates camera for both methods and p-values of the T-test for the difference. position. The detection rate of the correct bifurcations along the whole path (TPN) is significantly different with a CI for the dif- IV. CONCLUSION ference equal to [1.44, 19.03] between the method proposed With the final goal of an image-based navigation system in this work and [14]. This improvement is due to the fact for bronchoscopy guidance, we have presented a method that the proposed algorithm not only detects luminal regions for the extraction and codification of the bronchial anatomy but it also encodes the hierarchy relationships between them. and exploration path from videobronchoscopy. Our method Hierarchy relationships allow to increase the robustness when bases on a graph encoding the hierarchy of bronchial levels luminal regions are not detected in some frames. Concerning traversed during the exploration. This simple representation the percentage of correct paths reached from the trachea of the geometry of airways allows the localization of the (TPR), there are not significant differences with a CI for the scope and the reconstruction of the path navigated during difference equal to [−16.39, 20.92]. Although, our algorithm the intervention. This is an advantage over SLAM methods outperforms the approach from [14] in terms of TPN, there that require further registration of the reconstructed 3D point are not significant differences due to the TPR metric itself. cloud to an anatomical reconstruction of the patient’s airways Given a path with all the bifurcations but the first one correct, in order to locate the scope. the TPR value is 0 since there is none correct value starting Experiments conducted in interventional videos show that from the trachea. In both cases binary trees are used to encode our method is highly accurate retrieving bronchial anatomy the path followed during the intervention. Even so, lungs (recall = 90%). However, there is a substantial variability in does not contain only binary bifurcations, they also contains precision across patients due to bubbles, shines and abrupt ternary bifurcations which are not well defined in binary changes in bronchial level due to patient cough. Bubbles trees. Such ternary bifurcations appear in the proximal levels introduce false lumen detections in the extraction of MSER leading to low TPR values. Although there is a problem of regions, which could introduce false nodes and levels in representation of such ternary bifurcations, proximal levels the exploration tree. This could be solved by either using are not important for a guiding system as doctors does not alternative descriptors of the lumen (like deep features) or get lost. pre-processing images to remove shines. Abrupt motion due Table 4 reports CIs for average TPN and TPR percent- to patient cough is prone to change the level of the scope at ages for both methods and p-values of the T-test for the most distal airways which, like all trackers, is usually missed. difference. The detection rate of TPN and TPR is signif- This artifact is common to all trackers and it is a limita- icantly different with a CI for the difference between our tion of the technique. The impact of abrupt motion could 159702 VOLUME 8, 2020 D. Gil et al.: Intraoperative Extraction of Airways Anatomy in VideoBronchoscopy be minimized if image guidance was complemented with [16] E. Ramírez, C. Sánchez, A. Borràs, M. Diez-Ferrer, A. Rosell, and D. Gil, inertial sensors deployed in the working channel in a hybrid ‘‘BronchoX: Bronchoscopy exploration software for biopsy intervention planning,’’ Healthcare Technol. Lett., vol. 5, no. 5, pp. 177–182, Oct. 2018. system. [17] D. Gil, A. Esteban-Lansaque, A. Borras, and C. Sanchez, ‘‘Enhancing Comparison to existing guidance methods based on image virtual bronchoscopy with intra-operative data using a multi-objective analysis, shows that the proposed on-line tree outperforms the GAN,’’ IJCAR, vol. 14, no. 1, p. S7, 2019. [18] A. Esteban-Lansaque, C. Sanchez, A. Borras, and D. Gil, ‘‘Augmentation existing methods based on landmarks [14] and also SLAM of virtual endoscopic images with intra-operative data using content-nets,’’ approaches [6]. BioRxiv, 2019, Art. no. 681825. The results of this work are so promising, that encourage [19] E. Ramírez, C. Sánchez, A. Borrás, M. Diez-Ferrer, A. Rosell, and D. Gil, ‘‘Image-based bronchial anatomy codification for biopsy guiding in video testing the system in clinical premises. This is a current work bronchoscopy,’’ in OR 2.0 Context-Aware Operating Theaters, Computer in cooperation with Hospital Germans Trias i Pujol from Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Barcelona, Spain. Image Analysis. Cham, Switzerland: Springer, 2018, pp. 214–222. ACKNOWLEDGMENT The Titan X Pascal used for this research was donated by the NVIDIA Corporation. The D. Gil would like to dedicate this DEBORA GIL received the Ph.D. degree in math- work to her mother, E. R. Enfedaque. ematics from the Computer Science Department, UAB. She is currently a Professor with the Com- REFERENCES puter Science Department, UAB, and in charge of [1] A. Jemal, F. Bray, M. M. Center, J. Ferlay, E. Ward, and D. Forman, the Interactive and Augmented Modeling (IAM, ‘‘Global cancer statistics,’’ CA: A Cancer J. Clinicians, vol. 61, no. 2, iam.cvc.uab.es) research group in the Computer pp. 69–90, Mar. 2011. Vision Center (CVC). She has a wide multidis- [2] P. J. Reynisson, H. O. Leira, T. N. Hernes, E. F. Hofstad, M. Scali, ciplinary experience and is an expert in mathe- H. Sorger, T. Amundsen, F. Lindseth, and T. Langø, ‘‘Navigated bron- matical and statistical modeling of heterogeneous choscopy: A technical review,’’ J. Bronchol. Intervent. Pulmonol., vol. 21, data in clinical (diagnosis and intervention) decision support systems. She no. 3, pp. 242–264, 2014. has coauthored more than 60 articles in journals (41 indexed in JCR) and [3] E. F. Donnelly, ‘‘Technical parameters and interpretive issues in screening over 100 international conferences. She has been IP of several national computed tomography scans for lung cancer,’’ J. Thoracic Imag., vol. 27, (seven competitive and five with private companies) and international no. 4, pp. 224–229, Jul. 2012. (WP Leader in E-pilots, H2020-CS2-CFP08-2018-01 and a Coordinator of [4] T. R. Gildea, P. J. Mazzone, D. Karnak, M. Meziane, and A. C. Mehta, H2020 Topiomics-ATTRACT). ‘‘Electromagnetic navigation diagnostic bronchoscopy: A prospective study,’’ J. Respiratory Crit. Care Med., vol. 174, no. 9, pp. 982–989, 2006. [5] X. Luó, M. Feuerstein, D. Deguchi, T. Kitasaka, H. Takabatake, and K. Mori, ‘‘Development and comparison of new hybrid motion tracking for bronchoscopic navigation,’’ Med. Image Anal., vol. 16, no. 3, pp. 577–596, Apr. 2012. ANTONIO ESTEBAN-LANSAQUE received the [6] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos, ‘‘ORB-SLAM: A Ph.D. degree in computer science from the Com- versatile and accurate monocular SLAM system,’’ IEEE Trans. Robot., puter Science Department, UAB. He is currently vol. 31, no. 5, pp. 1147–1163, Oct. 2015. an Temporal Associate Professor with the Com- [7] R. Eberhardt, N. Kahn, D. Gompelmann, M. Schumann, C. P. Heussel, puter Science Department, UAB. His main contri- and F. J. Herth, ‘‘LungPoint—A new approach to peripheral lesions,’’ butions have been the development of intelligent J. Thoracic Oncol., vol. 5, no. 10, pp. 1559–1563, 2010. bronchoscopic support systems. [8] C. Wanga, M. Odab, and Y. Hayashi, ‘‘Visual SLAM for bronchoscope tracking and bronchus reconstruction in bronchoscopic navigation,’’ Proc. SPIE, vol. 10951, Mar. 2019, Art. no. 109510A. [9] M. Visentini-Scarzanella, T. Sugiura, T. Kaneko, and S. Koto, ‘‘Deep monocular 3D reconstruction for assisted navigation in bronchoscopy,’’ Int. J. Comput. Assist. Radiol. Surg., vol. 12, no. 7, pp. 1089–1099, Jul. 2017. [10] C. Sánchez, M. Diez-Ferrer, J. Bernal, F. J. Sánchez, A. Rosell, and D. Gil, ‘‘Navigation path retrieval from videobronchoscopy using bronchial branches,’’ in Proc. Workshop Clin. Image-Based Procedures. Cham, Switzerland: Springer, 2015, pp. 62–70. [11] A. Esteban-Lansaque, C. Sánchez, A. Borrás, M. Diez-Ferrer, A. Rosell, AGNÉS BORRÀS received the degree in com- and D. Gil, ‘‘Stable anatomical structure tracking for video-bronchoscopy puter science engineering and the Ph.D. degree in navigation,’’ in Proc. Workshop Clin. Image-Based Procedures. Cham, computer vision from the Universitat Autonoma de Switzerland: Springer, 2016, pp. 18–26. Barcelona (UAB), the master’s degree in computer [12] J.-P. Charbonnier, E. M. V. Rikxoort, A. A. A. Setio, game design from UAB, and the Ph.D. degree in C. M. Schaefer-Prokop, B. V. Ginneken, and F. Ciompi, ‘‘Improving motion graphics from the Open University of Cat- airway segmentation in computed tomography using leak detection with alonia (UOC). Her research interests comprise the convolutional networks,’’ Med. Image Anal., vol. 36, pp. 52–60, Feb. 2017. [13] A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, and S. image analysis, virtual environments, augmented Thrun, ‘‘Dermatologist-level classification of skin cancer with deep neural reality, and 3D representation. She has published networks,’’ Nature, vol. 542, no. 7639, pp. 115–118, Feb. 2017. 23 scientific papers, six in indexed journals (four [14] C. Sánchez, A. Esteban-Lansaque, A. Borrás, M. Diez-Ferrer, A. Rosell, of them Q1). She has worked as a Collaborate Professor at the Image and and D. Gil, ‘‘Towards a videobronchoscopy localization system from air- Multimedia Technology Center (CITM) and as a Consultant at UOC. She is way centre tracking,’’ in Proc. VISIGRAPP (VISAPP), 2017, pp. 352–359. a member of the R + D Department of the Computer Vision Center (CVC) [15] M. Pawlik and N. Augsten, ‘‘Tree edit distance: Robust and memory- and she has a strong experience in technological transfer. She has participated efficient,’’ Inf. Syst., vol. 56, pp. 157–173, Mar. 2016. in 12 projects and she has an European Inventor’s Patent. VOLUME 8, 2020 159703 D. Gil et al.: Intraoperative Extraction of Airways Anatomy in VideoBronchoscopy ESMITT RAMÍREZ received the M.Sc. degree CARLES SÁNCHEZ RAMOS received the Ph.D. in computer science from the Central Univer- degree in computer science from the Computer sity of Venezuela (UCV). He is a member of Science Department, UAB. the Interactive and Augmented Modeling (IAM, He is currently a Professor with the Com- iam.cvc.uab.es) research group in the Computer puter Science Department, UAB, and a member Vision Center, UAB. He is also a member of the of the Interactive and Augmented Modeling (IAM, Computer Graphics Center (CCG), UCV. He has iam.cvc.uab.es) research group in the Computer coauthored more than 20 papers in journals and Vision Center (CVC). He has been working in international conferences. He currently holds a research projects as well technological transfer for fellowship granted by the Ministry of Economy, more than eight years. He has long experience in Industry and Competitiveness, Spain. His main researches are focused on medical imaging, anatomy modeling, and image processing applied in the medical imaging and 3D rendering algorithms. development of intelligent endoscopic support systems. He has coauthored more than ten papers in journals and over 20 international conferences. He currently has a TecnioSpring Plus fellow and has been IP of two national projects and participated in 12. 159704 VOLUME 8, 2020