Received September 18, 2020, accepted October 11, 2020, date of publication October 19, 2020, date of current version November 5, 2020. Digital Object Identifier 10.1109/ACCESS.2020.3032024 Co-Training for On-Board Deep Object Detection GABRIEL VILLALONGA AND ANTONIO M. LÓPEZ PEÑA , (Member, IEEE) Computer Vision Center (CVC), Universitat Autònoma de Barcelona (UAB), 08193 Bellaterra, Barcelona, Spain Computer Science Department, Universitat Autònoma de Barcelona (UAB), 08193 Bellaterra, Barcelona, Spain Corresponding author: Gabriel Villalonga (gvillalonga@cvc.uab.es) The authors acknowledge the financial support received for this work from the Spanish TIN2017-88709-R (MINECO/AEI/FEDER, UE) project. Antonio acknowledges the financial support to his general research activities given by ICREA under the ICREA Academia Program. The authors acknowledge the support of the Generalitat de Catalunya CERCA Program as well as its ACCIO agency to CVC’s general activities. ABSTRACT Providing ground truth supervision to train visual models has been a bottleneck over the years, exacerbated by domain shifts which degenerate the performance of such models. This was the case when visual tasks relied on handcrafted features and shallow machine learning and, despite its unprecedented performance gains, the problem remains open within the deep learning paradigm due to its data-hungry nature. Best performing deep vision-based object detectors are trained in a supervised manner by relying on human-labeled bounding boxes which localize class instances (i.e. objects) within the training images. Thus, object detection is one of such tasks for which human labeling is a major bottleneck. In this article, we assess co-training as a semi-supervised learning method for self-labeling objects in unlabeled images, so reducing the human-labeling effort for developing deep object detectors. Our study pays special attention to a scenario involving domain shift; in particular, when we have automatically generated virtual-world images with object bounding boxes and we have real-world images which are unlabeled. Moreover, we are particularly interested in using co-training for deep object detection in the context of driver assistance systems and/or self-driving vehicles. Thus, using well-established datasets and protocols for object detection in these application contexts, we will show how co-training is a paradigm worth to pursue for alleviating object labeling, working both alone and together with task-agnostic domain adaptation. INDEX TERMS Co-training, domain adaptation, vision-based object detection, ADAS, self-driving. I. INTRODUCTION data must be pursued. Thus, we can find different families Since more than two decades ago, machine learning (ML) has of algorithms, or paradigms, which address human-labeling been the enabling technology to solve computer vision tasks. minimization under different working assumptions. We con- In the last decade, traditional ML, i.e. based on relatively tinue this introduction by reviewing them in subsection I-A, shallow classifiers and hand-crafted features, has given way while subsection I-B highlights how domain shifts exacerbate to deep learning (DL). Thanks to DL models based on con- human labeling requirements. With the human-labeling bot- volutional neural networks (CNNs), DL approaches signifi- tleneck problem introduced, subsection I-C summarizes our cantly outperformed traditional ML in all kinds of computer focus when addressing such a problem. In particular, we out- vision tasks, such as image classification, object detection, line our application context and the proposed approach for semantic segmentation, etc. A major key for the usefulness reducing human annotation effort in such a context. Finally, of DL models is to train them in a supervised way. In other subsection I-D highlights the main contributions presented words, the raw data (still images and videos) need to be sup- in this article as well as the organization of the rest of its plemented by ground truth; nowadays, usually collected via contents. crowd-sourced labeling. In practice, due to the data-hungry nature of CNNs, data labeling is considered a major bottle- A. PARADIGMS TO MINIMIZE HUMAN LABELING neck. Therefore, approaches to minimize labeling effort are For instance, active learning (AL) approaches [1], [34], [35] of high interest; or put in another way, approaches to automat- assume an initial model and an unlabeled dataset to be labeled ically leverage the large amounts of available unlabeled raw by a human (oracle) following an interactive procedure. In particular, the current model processes the unlabeled data The associate editor coordinating the review of this manuscript and providing so-called pseudo-labels (at image, object, or pixel approving it for publication was Yongqiang Zhao . level); then, these results are inspected, either automatically This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. VOLUME 8, 2020 For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/ 194441 G. Villalonga, A. M. López Peña: Co-Training for On-Board Deep Object Detection or directly by the oracle, to select the next data to be labeled. Among the SSL approaches, co-training [5], [15] has Afterwards, the model is trained again, and the process been explored for deep image classification [8], [29] with repeated until fulfilling a stop criterion. AL assumes that promising results. However, up to the best of our knowl- with less labeling budget than the required to have the oracle edge, it remains unexplored for deep object detection. Note labeling data at random, one can train models that perform that, while image classification aims at assigning image-level at least similarly. In practice, it is usually hard to clearly class/attribute labels, object detection is more challenging outperform the oracle’s random labeling strategy. In contrast since it requires to localize and classify objects in images, to AL, other approaches do not assume human oracles in the i.e. placing a bounding box (BB) per object, together with labeling loop. For instance, this is the case of semi-supervised the class label assigned to it. Moreover, object detection from learning (SSL) algorithms [6], [38], [40], which assume the on-board images, i.e. detection of vehicles, pedestrians, etc., availability of a large number or raw unlabeled data together is especially difficult since, to the inherent intra-class differ- with a relatively small number of labeled data. Then, a model ences (e.g. due to vehicle models, pedestrian clothes, etc.), must be trained using the unlabeled and labeled data (with- acquisition conditions add a vast variability because the out human intervention), with the goal of being more accu- objects appear in a large range of distances to the camera rate than if only the labeled data were used. The so-called (resolution and focus variations), under different illumina- self-training and co-training algorithms, which are of our tion conditions (from strong shadows to direct sunlight), and main interest in this article, fall in the SSL paradigm [40]. they usually move (blur, occlusion, view angle, and pose In AL and SSL, the most common idea is to efficiently variations). label the unlabeled data, either via an oracle (AL) or auto- Originally, co-training relies on the agreement of two matically (SSL). Self-supervised learning (SfSL) follows a trained models performing predictions from different fea- different approach which consists in providing supervision tures (views) of the data [5]. These predictions are taken as in the form of additional (relatively simple) tasks, known as pseudo-labels to improve the models incrementally. Later, pretext tasks, for which automatic ground truth can be easily prediction disagreement was shown to improve co-training generated (e.g. solving jigsaw puzzles [12], [19], [21]). results in applications related to natural language process- ing [15], [39]. Thus, in this article, we propose a co-training B. THE DOMAIN ADAPTATION PROBLEM algorithm inspired by prediction disagreement. On the other Within these approaches, the most classical assumption is hand, since co-training was proposed as a SSL method aiming that the labeled and the unlabeled data are drawn from the at avoiding the drift problem of self-training [5], which incre- same distribution and one aims at using the unlabeled data mentally re-trains a single model from its previously most (via annotations or pretext tasks) to solve the same tasks for confident predictions, we also elaborate a strong self-training which the labeled data was annotated. However, in practice, baseline for our deep object detection problem. we may require to solve new tasks, leading to transfer learn- We assess the effectiveness of the two self-labeling meth- ing (TL) [46], [54], or solving the same tasks in a new domain, ods, i.e. self-training and co-training, in two different practi- leading to domain adaptation (DA) [10], [43], [47]. Beyond cal situations. First, following most classical SSL, we will specific techniques to tackle TL or DA, we can leverage solu- assume access to a small dataset of labeled images, X l , tions/ideas from AL, SSL, or SfSL. For instance, AL [41], together with a larger dataset of unlabeled images X u ; where SSL [20], [55], and SfSL [49] algorithms have been used to here the labels are object BBs with class labels. Second, address DA problems. In this article, we use SSL (comparing we will address a relevant setting that resorts DA. In par- self-training and co-training) to address DA too. ticular, we will assume access to a dataset of virtual-world images with automatically generated object labels. Therefore, C. THE FOCUS OF THIS WORK eventually, X l can be larger than X u , since labeling these From the application viewpoint, in this article, we focus images does not require human intervention; however, X u is on vision-based on-board deep object detection. Note that composed of real-world images, thus, between X l and X u this is a very relevant visual task for driving, since detect- there is a domain shift [41]. In this case, we also compare ing objects (e.g. vehicles, pedestrians, etc.) along the self-labeling techniques with task-agnostic DA, in particular, route of a vehicle is a key functionality for perception- using GAN-based image-to-image translation [53]. As espe- decision-action pipelines in the context of both advanced cially relevant cases, we will focus on on-board detection of driver assistance systems (ADAS) and self-driving vehicles. vehicles and pedestrians using a deep CNN architecture. Our Moreover, nowadays, most accurate vision-based models experiments will show how, indeed, our co-training algorithm to detect such objects are based on deep CNN architec- is a good SSL alternative for on-board deep object detec- tures [23], [24], [30], [31]. In addition, in this application tion. Co-training clearly boosts detection accuracy in regimes context, it is possible to acquire innumerable quantities of raw where the size of X l is just the 5%/10% of the labeled images, for instance, from cameras installed in fleets of cars. data used to train an upper-bound object detector. Moreover, Thus, methods to minimize the effort of manually labeling under the presence of domain shift, we will see how image- them are of great relevance. to-image GAN-based translation and co-training complement 194442 VOLUME 8, 2020 G. Villalonga, A. M. López Peña: Co-Training for On-Board Deep Object Detection each other, allowing to reach almost upper-bound perfor- as a new pseudo-labeled dataset X l̂ , which in turn is used mances without human labeling. to retrain φ. The pseudo-labeling/retraining cycle is repeated until reaching some stop criterion, when φ is expected to be D. CONTRIBUTIONS AND ORGANIZATION more accurate than at the beginning of the process. The main Hence, the main contributions of this article are: (1) propos- differences between self-labeling algorithms arise from how ing a co-training algorithm for deep object detection; (2) desi- X l̂ is formed and used to retrain φ in each cycle. gning this algorithm to allow addressing domain shift via GAN-based image-to-image translation; (3) showing its effectiveness by developing a strong self-training baseline 1) SELF-TRAINING and relying on publicly available evaluation standards and In self-training, introduced by Yarowsky [50] for word sense on-board image datasets. Alongside, we will also contribute disambiguation, X l̂ is formed by collecting the most confi- with the public release of the virtual-world dataset we gen- dent predictions of φ in X u , updating X u to contain only the erated for our experiments. To show these achievements remaining unlabeled data. Then, φ is retrained using super- we organize the rest of the paper as follows. Section II vision from X l ∪ X l̂ , i.e. pseudo-labels are taken as ground reviews the most related works to ours, highlighting common- truth. Before the DL era, Rosenberg et al. [33] already showed alities and differences. Section III details our self-training promising results when applying self-training to eye detection and co-training algorithms. Section IV draws our exper- in face images. More recently, Jeong et al. [18], proposed imental setting, and discuss the obtained results. Finally, an alternative to collect X l̂ for deep object detection, which Section V summarizes the presented work, suggesting lines of consists of adding a consistency loss for training φ as well as continuation. eliminating predominant backgrounds. If Iui is an unlabeled image, the consistence loss is based on the idea that φ(Iui ) and II. RELATED WORK φ((Iui ) ), where () stands for horizontal mirroring, must pro- Our main goal is to train a vision-based deep object detec- vide corresponding detections. Experiments are conducted on tor without relying on human labeling. Following our work PASCAL VOC and MS-COCO datasets, and results are on line [26], we propose to leverage labeled images from virtual pair with other state-of-the-art methods combining AL and worlds and unlabeled ones from the real world, in such a self-training [42]. The reader is referred to [28] for a review way that we can automatically label the real-world images on loss-based SSL methods for deep image classification. by progressively re-training a deep object detector that sub- On the other hand, note that this SSL variations are not agnos- stitutes human annotators. Note that the labeling step is tic to φ, since its training loss is modified. This is also the case needed due to the domain shift between virtual and real-world in [25], where, in the context of deep image classification, images, otherwise, we could just use the virtual-world images the activation functions composing φ must be replaced by to train object detectors and, afterwards, deploy them in Hermite polynomials. In this article, we use a self-training the real world expecting a reliable performance. In particu- strategy as SSL baseline for on-board deep object detection, lar, we explore the combination of the co-training idea for thus, keeping agnosticism regarding φ. self-labeling objects and GAN-based image-to-image trans- lation in the role of task-agnostic DA. On the other hand, our proposal is agnostic to the object detection architecture 2) CO-TRAINING supporting self-labeling. Co-training was introduced by Blum and Mitchell [5] in Accordingly, in the remaining of this section, we first the context of web-page classification, as alternative to the review related works on self-labeling (subsection II-A), self-training of Yarowsky [50], in particular, to avoid model including self-training and co-training; next, we review drift. In this case, two models, φ1 and φ2 , are trained on dif- related works on DA (subsection II-B). In both sections, ferent conditionally independent features of the data, called we relate our proposal to the reviewed works. views, assuming that each view is sufficiently good to learn an accurate model. Each model is trained by following the same A. SELF-LABELING idea as with self-training, but in each cycle the data samples Self-labeling algorithms are examples of SSL wrappers [40], self-labeled by φ1 and φ2 are aggregated together. Soon, which work as meta-learners for supervised ML algorithms. co-training was shown to outperform other state-of-the-art The starting point consists of a labeled dataset, X l , and an methods including self-training [27], and the conditional unlabeled dataset, X u , where it is supposed that the cardinal- independence assumption was shown not to be essential in ity of X u is significantly larger than the cardinality of X l , and practice [4], [44], [45]. Later, in the context of sentence seg- both datasets are drawn from the same domain, D. The goal mentation, Guz et al. [15] introduced the disagreement idea is to learn a predictive model, φ, whose accuracy would be for co-training, which was refined by Tur [39] to jointly tackle relatively poor by only using X l but becomes significantly DA in the context of natural language processing. In this case, higher by also leveraging X u . Briefly, self-labeling is an samples self-labeled with high confidence by φi but with low incremental process where φ is first trained with X l ; then X u confidence by φj , i, j ∈ {1, 2}, i 6= j, are considered as is processed using φ in a way that the predictions are taken part of the new pseudo-labeled data in each cycle. In fact, VOLUME 8, 2020 194443 G. Villalonga, A. M. López Peña: Co-Training for On-Board Deep Object Detection disagreement-based SSL became a subject of study on its own either φS or XSl,tr to train φT . For a review of the DA corpus at that time [52]. we advise the reader to consult [10], [43], [47]. Before the irruption of DL, Levin et al. [22] applied In this article, we assume an UDA setting. Moreover, our co-training to detect vehicles in video-surveillance images, source data comes from a virtual world with automatically so removing background and using different training data to generated labels, so we have XSl,tr . Since we aim at assessing generate different views. More recently, Qiao et al. [29] used self-labeling by co-training to address the UDA problem, a co-training setting for deep image classification, based on we have X l = XSl,tr , X u = XT , thus, X l̂ ⊆ XT . several views. Each view corresponds to a different CNN, φi , Following this line of work, Kim et al. [20] addressed USD trained by including samples generated to be mutually adver- for deep object detection by proposing a combination of a sarial. The idea is to use different training data for each φi to weak self-training and a special treatment of backgrounds prevent them to prematurely collapse in the same network. via a loss component used during the training of the object This implies to link the training of the φi ’s at the level of detector, with PASCAL VOC as XS and art-style datasets the loss function. In this article, we force the use of different (Clipart1K, Watercolor2K, Comic2K) as XT . Zou et al. [56] training data for each φi , without linking their training at the also use a self-training strategy for UDA in the context of deep level of loss functions, again, keeping co-training agnostic image classification and semantic segmentation (including to the used φi . Moreover, we address objected detection, a virtual-to-real setting), where the core idea is to perform which involves not only predicting class instances as in image confidence and model regularization of the trained classi- classification, but also localizing them within the images, fiers. Since in our experimental setting we will use Faster so that the background becomes a large source of potential R-CNN to obtain φ, it is also worth to mention the work false positives. by Chen et al. [9], where an UDA method was specifically Finally, to avoid confusion, it is worth to mention the designed for Faster R-CNN and demonstrated on car detec- so-called co-teaching, recently introduced by Han et al. [16], tion under a virtual-to-real setting too. Since Faster R-CNN and its variant co-teaching+ introduced by Yu et al. [51]. is a two-stage classifier, the proposed UDA involves an These algorithms are designed to address situations with image-level adaptation for the region proposal stage, and noisy labels on X l , both demonstrated on deep image classi- an instance-level adaptation for the BB prediction stage. fication. Indeed, these algorithms stem ideas from co-training Finally, focusing on traditional ML and Amazon reviews (the classical one from [16], the disagreement-based one datasets, Chen et al. [7] showed that co-training is a promising from [51]), however, reproducing the words of Han et al. [16], algorithm for UDA, providing better performance than self- co-training is designed for SSL, and co-teaching is for learn- training. ing with noisy (ground truth) labels (LNL); as LNL is not a Accordingly, beyond self-training-style and model-specific special case of SSL, we cannot simply translate co-training strategies for UDA, in this article, we are interested in from one problem setting to another problem setting. assessing co-training as a meta-learning UDA strategy. To the best of our knowledge, this is an under-explored and rel- evant setting. Moreover, even we are going to use Faster B. DOMAIN ADAPTATION R-CNN because its outstanding accuracy, our proposal nei- ML algorithms assume that training and testing data are ther is specific for it, nor requires modifying its losses. drawn from the same domain, D. When this is not the case, In other words, our co-training-based UDA works at the the trained models suffer from domain shift. In other words, training-data level. This allows to complement it with other we have data, XS , drawn from a source domain, DS , as well UDA working at the same level. In particular, we combine as data, XT , drawn from a target domain, DT , DT 6 = DS . it with GAN-based image-to-image translation, since such We can assume that XS = XStr ∪ XStt , XStr ∩ XStt = ∅, where task-agnostic approach can transform XSl,tr to be more sim- XStr is used to train some predictive model φS . It turns out ilar to XT before starting co-training. By using the Cycle- that the prediction accuracy of φS in XStt is much higher than GAN implementation of Zhu et al. [53], we will see how, in XT , a phenomenon known as domain shift. Addressing this indeed, GAN-based image-to-image translation combined problem is the goal of DA techniques, under the assumption with co-training outperforms the use of each method in that there is some (unknown) correlation between DS and DT , isolation. since DA is not possible otherwise. The core idea is to use XT to obtain a new model, φT , being clearly more accurate than φS in DT . While doing this, III. METHODS the human-based labeling effort in XT must be minimized. In this section, we detail our self-training and co-training Supervised DA (SDA) assumes access to a relatively small set meta-learning proposals as Algorithms 1 and 2, respectively. of labeled target-domain data XTl,tr ⊂ XT . If we do not have In addition, Figures 1 and 2 provide a visual representation access to XSl,tr , then the challenge is to leverage from φS and of these algorithms, highlighting their main components with XTl,tr to obtain φT . Otherwise, we can combine XSl,tr and XTl,tr the corresponding data flow. Our main interest is to assess the to train φT . In unsupervised DA (UDA), XT is unlabeled; thus, performance of co-training in vision-based object detection, we address the more challenging situations of using XT with but we need also to develop a strong self-training baseline. 194444 VOLUME 8, 2020 G. Villalonga, A. M. López Peña: Co-Training for On-Board Deep Object Detection Algorithm 1 Self-Labeling by Self-Training Algorithm 2 Self-Labeling by Co-Training input : Labeled images: Xl = {< Ili , Yil >} input : Labeled images: X l = {< Ili , Yil >} Unlabeled images: X u = {Iui } Unlabeled images: X u = {Iui } Object detection architecture: 8 Object detection architecture: 8 8 Training hyper-par.: H8 8 Training hyper-par.: H8 Slf-tr. hyp.-p.: Hsl = {T , N , n, Hstp [, Hseq ]} Co-tr. hyp.-p.: Hsl = {T , N , n, m, Hstp [, Hseq ]} output: New labeled images: X l̂ = {< Il̂i , Yil̂ >} output: New labeled images: X l̂ = {< Il̂i , Yil̂ >} < X l̂ , k > ← < ∅, 0 >; < X1l̂ , X2l̂ , k > ← < ∅, ∅, 0 >; φ ← TrainDetector(8, H8 , X l , X l̂ ); < X1l , X2l > ← < X l , (X l ) >; l̂ Xnew ← RunDetector(φ, X u , T ); φ1 ← TrainDetector(8, H8 , X1l , X1l̂ ); repeat φ2 ← TrainDetector(8, H8 , X2l , X2l̂ ); l̂ ← X l̂ ; Xold new l̂ X1,new ← RunDetector(φ1 , X u , T ); l̂ X↑ ← Select(↑, n, Rand(Xnew l̂ , N [, H , k])); seq l̂ X2,new ← RunDetector(φ2 , X u , T ); X l̂ ← Fuse(X l̂ , X↑l̂ ); repeat l̂ ← X l̂ φ ← TrainDetector(8, H8 , X l , X l̂ ); Xold 1,new ; l̂ Xnew ← RunDetector(φ, X u , T ); X1,↑ 1,new , N [, Hseq , k])); l̂ ← Select(↑, m, Rand(X l̂ l̂ , X l̂ , k++); until Stop?(Hstp , Xold 2,new , N [, Hseq , k])); l̂ ← Select(↑, m, Rand(X l̂ X2,↑ new l̂ X ← Xnew ;l̂ X1,↓ ← Select(↓, n, RunDetector(φ1 , X2,↑ l̂ l̂ , T )); return X l̂ ; l̂ ← Select(↓, n, RunDetector(φ , X l̂ , T )); X2,↓ 2 1,↑ X1l̂ ← Fuse(X1l̂ , X1,↓ l̂ ); X2l̂ ← Fuse(X2l̂ , X2,↓ l̂ ); φ1 ← TrainDetector(8, H8 , X1l , X1l̂ ); φ2 ← TrainDetector(8, H8 , X2l , X2l̂ ); l̂ X1,new ← RunDetector(φ1 , X u , T ); l̂ X2,new ← RunDetector(φ2 , X u , T ); until Stop?(Hstp , Xold 1,new , k++); l̂ , X l̂ l̂ X l̂ ← X1,new ; FIGURE 1. Self-training main components. We use the same notation as in Algorithm 1. The data in green is labeled, the one in dark grey is unlabeled, the transition of both colors represents pseudo-labeled data. return X l̂ ; The blue boxes correspond to the main components involved in self-training according to Algorithm 1. Since they share functional components, we first introduce those and then detail how they are used for both self-training and co-training. Finally, we will see how, depending on the input data, these SSL algorithms can be used even in a context where there is a domain shift between already existing labeled images and the images to be labeled automatically. To refer to both, self-training and co-training indistinctly, we will use the term self-labeling in the rest of this section. A. SELF-LABELING FUNCTIONAL COMPONENTS 1) INPUT AND OUTPUT PARAMETERS Since we follow a SSL setting [40], we assume access to a set of images (e.g. acquired on-board a car) with each object of interest (e.g. vehicles and pedestrians) labeled by a BB and a class label, as well as to a set of unlabeled images. The former is denoted as X l = {< Ili , Yil >}, FIGURE 2. Co-training main components. We use the same notation as in Algorithm 2, but we have introduced two dummy variables for the sake of where Ili is a particular image of the labeled set, being Yil l̂ , X l̂ ). The data in green is labeled, the one in dark grey is clarity (X1,t 2,t its corresponding labeling information; i.e., for each object unlabeled, the transition of both colors represents pseudo-labeled data. of interest in Ili , Yil includes its BB and its class label. The The blue boxes correspond to the main components involved in co-training according to Algorithm 2. The light grey bounding box is unlabeled set is X u = {Iui }, where Iui is a particular unlabeled executed just once. image. We assume also a given object detection architecture to perform self-labeling, denoted as 8, with corresponding we expect to obtain a new set of automatically labeled images, training hyper-parameters denoted as H8 . After self-labeling, which we denote as X l̂ = {< Il̂i , Yil̂ >}, where Il̂i ∈ X u and VOLUME 8, 2020 194445 G. Villalonga, A. M. López Peña: Co-Training for On-Board Deep Object Detection has Yil̂ as so-called pseudo-labels, which in this case consists If they are provided, it means that X l̂ consists of sequences of a BB and a class label for each detected object in Il̂i . The and we want to ensure not to return self-labeled consecutive variables introduced so far are generic in SSL, i.e. they are frames, since they may contain very similar samples com- not specific of our proposals. We denote as Hsl the specific ing from the same object instances and we want to favor set of hyper-parameters we require for self-labeling. In fact, variability in the samples. Moreover, in this way we can Hsl includes {T , N , n, Hstp , Hseq } as common parameters minimize the inclusion of spurious false positives that may for both self-training and co-training, but the latter requires persist for several frames. In this case, Hseq = {1t 1 , 1t 2 }; an additional parameter m that we will explain in the con- where 1t 1 is the minimum distance between frames with text of co-training. If K is the number of classes to be pseudo-labels generated in the current cycle, k, and 1t 2 is considered, then T = {t1 , . . . , tK } is a set of per-class the minimum distance between frames with pseudo-labels detection thresholds, normally used by object detectors to generated in cycle k and frames with pseudo-labels gen- ensure a minimum per-class confidence to accept potential erated in previous cycles (< k). 1t 1 condition is applied detections. Since self-labeling must perform object detection first, then 1t 2 one. The final random selection is per- on unlabeled images, these thresholds are needed. During formed on the frames remaining after applying these distance a self-labeling cycle, N self-labeled images are randomly conditions. selected, from which n are kept to retrain the object detector. Self-training and co-training use different criteria to select 5) Select(s, n, X l̂ ) : Xsl̂ such n self-labeled images. Hstp consists of the parameters This function returns a sub-set Xsl̂ ⊂ X l̂ of m self-labeled required to evaluate whether self-labeling should stop or not. images, selected according to a criterion s. If s =↑, the top n Finally, Hseq is an optional set of parameters, only required most confident images are selected; while for s =↓, the n least if X u consists of sequences rather than isolated images. Note confident images are selected. We resume the confidence of that, in this case, we can easily avoid to introduce too similar an image from the confidences of its detections. For the sake training samples coming from the same object instances in of simplicity, since X l̂ = {< Il̂i , Yil̂ >}, we can assume consecutive frames, which has shown to be useful in AL approaches [2]. that Yil̂ not only contains the BB and class label for each detected object in Il̂i , but also the corresponding detection 2) TrainDetector(8, H8 , X l , X l̂ ) : φ confidence. Then, the confidence assigned to Il̂i is the average This function returns an object detector, φ, by training a of the confidences in Yil̂ . CNN architecture 8 (e.g. Faster R-CNN [31]) according to the training hyper-parameters H8 (e.g. optimizer, learning l̂ , X l̂ l̂ 6) Fuse(Xold new ) : X rate, mini-batch size, training iterations, etc.). Note that this The goal of this function is to avoid redundant detections is just the standard manner of training φ, but as part of our l̂ l̂ self-labeling procedures, we control the provided training between the self-labeled sets Xold and Xnew and preserv- data. In particular, we use labels (in X l ) and pseudo-labels ing different ones, producing a new set of self-labeled (in X l̂ ) indistinctly. However, we only consider background images, X l̂ . Thus, on the one hand, X l̂ will contain all the self-labeled images in Xoldl̂ ∪ X l̂ l̂ l̂ samples based on X l , since, during self-labeling, X l̂ can new − Xold ∩ Xnew ; on the l̂ l̂ other hand, for those images in Xold ∩ Xnew , only the detec- contain false negatives which could be erroneously taken as hard negatives when training φ. Despite this control over the l̂ tions in Xnew are kept, so the corresponding information is training data, note that we neither require custom modifica- added to X l̂ . tions of the loss function used to train φ, nor architectural modifications of 8. l̂ , X l̂ 7) Stop?(Hstp , Xold new , k) : Boolean This function decides whether to stop self-labeling or not. 3) RunDetector(φ, X u , T ) : X l̂ The decision relies on two conditions. The first one is if a This function returns a set of self-labeled images, X l̂ , minimum number of self-labeling cycles, Kmin , have been obtained by running the object detector φ on the unlabeled performed, where current number is k. The second condition set of images X u ; in other words, the pseudo-labels corre- l̂ and X l̂ , computed as the relies on the similarity of Xold new spond to the object detections. Each detection D consists l̂ in the role mAP (mean average precision) [11] by using Xold of a BB B and a class label c; moreover, being a detec- l̂ of ground truth and Xnew in the role of results to be evaluated. tion implies that the confidence φ(D) fulfils the condition φ(D) ≥ tc , tc ∈ T . We compute the absolute value difference between the mAP at current cycle and previous one, keeping track of these differences in a sliding window fashion. If these differences 4) Rand(X l̂ , N[, Hseq , k]) : Xsl̂ are below a threshold, T1mAP , for more than a given number This function returns a set Xsl̂ of N self-labeled images ran- of consecutive cycles, 1K , self-labeling will stop. Therefore, domly chosen from X l̂ . Hseq and k are optional parameters. Hstp = {Kmin , T1mAP , 1K }. 194446 VOLUME 8, 2020 G. Villalonga, A. M. López Peña: Co-Training for On-Board Deep Object Detection B. SELF-TRAINING of φ1 since it is the detector using X l (the original labeled Algorithm 1 describes self-training based on the funct- dataset). ional components introduced in Section III-A. In the Another essential question is how each Xil̂ is obtained. following, we highlight several points of this As we have mentioned before, we follow the idea of dis- algorithm. agreement. Thus, from the random set of images self-labeled As Select(, , ) is run in the loop, it implies that in each by φi , i.e. Xi,new l̂ l̂ the m with overall , i ∈ {1, 2}, we set to Xi,↑ cycle the n most confident self-labeled images are kept for higher confident pseudo-labels; then, the images in Xi,↑ l̂ are retraining φ. Moreover, potentially redundant object detec- processed by φj , j ∈ {1, 2}, i 6= j, and the n with the overall tions arising from different cycles are resumed by running l̂ . Finally, each lower confident pseudo-labels are set to Xj,↓ the Fuse(, ) function. As is called Fuse(, ) in the loop, when l̂ is fused with the X l̂ accumulated from previous cycles Xi,↓ a previous detection and a new one overlap enough, the new i one is kept since it is based on the last trained version of φ, to update Xil̂ . Therefore, in each cycle, each φi is retrained which is expected to be more accurate than previous ones. with the images containing the less confident pseudo-labels When calling Select(, , ), not all the available self-labeled for current φi , selected among the images containing the images are considered, but just a random set of them of most confident pseudo-labels for current φj . In this way, size N ; which are chosen taking into account the selection the detectors trust each other, and use the samples that are conditions introduced in case of working with sequences. more difficult for them for improving. This prevents the same highly accurate detections to be Note also that the most costly processing in self-training systematically selected across cycles, which would prevent and co-training cycles is the TrainDetector(, , , ) procedure. the injection of variability when retraining φ. The random It is called once per cycle for self-training, and twice for selection over the entire X u was also proposed in the original co-training. However, in the latter case the two executions co-training algorithm [5]. As we have mentioned before, can run totally in parallel, therefore, with the proper hard- when running TrainDetector(, , , ), background samples are ware resources, self-training and co-training cycles can be only collected from X l to avoid introducing false positives. performed in a very similar time. Moreover, we set H8 to ensure that all the training sam- ples are visited at least once (mini-batch size × number of D. SELF-LABELING FOR UDA iterations ≥ number of training images). Therefore, we can Once introduced our self-training and co-training algorithms, think of X l ∪ X l̂ as a mixing of data, where X l acts as we see how using them for UDA is just a matter of pro- a regularizing factor during training, which aims to prevent viding the proper input parameters; i.e., X l = XSl,tr , φ to become an irrecoverably bad detector. Thinking in a X u = XT , being XSl,tr the source-domain labeled dataset, virtual-to-real UDA setting, we note that from traditional ML and XT the target-domain unlabeled one. We will draw the algorithms to modern deep CNNs, mixing virtual and real former from a virtual world, and the latter from the real data has been shown to be systematically beneficial across world. different computer vision tasks [17], [32], [36], [41]. Finally, we can see that to run the stopping criterion we rely on the full self-labeled data available at the beginning and end of each IV. EXPERIMENTS cycle, which results from applying the last available version A. EXPERIMENTAL SETUP of φ to the full X u set. As real-world datasets we use KITTI [11] and Waymo [37], in the following denoted as K, W, respectively. The former is a well-established standard born in the academia; the latter is C. CO-TRAINING a new contribution from the industry. To work with K, we fol- Algorithm 2 describes our co-training proposal based on low the splits introduced in [48], which are based on isolated the functional components introduced in Section III-A. images, and guarantee that training and testing images are not In the following, we highlight several points of this highly correlated; i.e., training and testing images are not just algorithm. different frames sampled from the same sequences. On the Our co-training strategy tries to make φ1 and φ2 different other hand, W dataset contains video sequences. Following by training them on different data. Note how each φi is trained the advice for its use, we have randomly selected part of on Xil and Xil̂ , i ∈ {1, 2}, at each cycle. The Xil ’s do not the sequences for training and the rest for testing. Taking K change across cycles and we have X1l = X l and X2l = (X l ) , as reference, we focus on daytime and non-adverse weather thus, one of the labeled sets is a horizontal mirroring of conditions in all the cases. In addition, we have accommo- the other. Moreover, the Xil̂ ’s are expected to be different dated W images to the resolution of K ones, i.e. 1240 × 375 from each other and changing from cycle to cycle. On the pixels, by just cropping away their upper part (which mainly other hand, even technically not using the same exact data to shows sky) and keeping the vertically centered 1240 columns. train φ1 and φ2 , there will be a cycle when they converge and Following KITTI benchmark moderate settings, we set the are not able to improve each other. This is what we check to minimum BB height to detect objects to 25 pixels and, analo- stop, i.e. as for self-training, but focusing on the performance gously, 50 pixels for W. Moreover, since W contains 3D BBs, VOLUME 8, 2020 194447 G. Villalonga, A. M. López Peña: Co-Training for On-Board Deep Object Detection as for the virtual-world objects (Figure 3), we obtain 2D BBs TABLE 1. Datasets (X ): train (X tr ) and test (X tt ) information, from them. In other words, the resulting 2D BBs account for X = X tr ∪ X tt , X tr ∩ X tt = ∅. We show the number of images/frames (sequences), vehicle BBs, pedestrian BBs, and whether the datasets occluded object areas, which is not the case for the 2D BBs consists of video sequences or not. directly available with W. FIGURE 4. Main components of Faster R-CNN: feature extractor (FE), region proposal network (RPN), and region-based CNN (RCNN). Their responsibilities are outlined in parenthesis and elaborated in the main text. We use VGG16 as FE. Blue boxes are blocks of neural network layers with input dimensions indicated as . Grey boxes are algorithmic steps to return BBs, candidates (RPN) or detections (RCNN). This Faster R-CNN architecture is detailed in [13]. have chosen Faster R-CNN [31] because it provides a com- FIGURE 3. Top images: virtual-world patches showing 3D BBs framing petitive detection accuracy and is very well known by com- vehicles and pedestrians. Bottom image: projecting 3D BBs as 2D BBs for different views of a pickup, with instance segmentation as visual puter vision practitioners [23]. For the sake of completeness, reference. Figure 4 shows the main components of Faster R-CNN, namely, the Feature Extractor (FE), the region proposal net- work (RPN), and the region-based CNN (RCNN). We use the In order to perform our experiments, as set of virtual-world implementation available in the Detectron framework [13], images (V), we have used a dataset that we generated inter- with VGG16 as FE. The RPN component generates object nally around two years ago as a complement for our former candidates from the so-called bottleneck of FE (i.e. Con- SYNTHIA dataset [32]. We did not have the opportunity to vBlock5). Conceptually, these candidates can be understood release it at that moment, but it will be publicly available as BBs which can potentially contain objects of interest. to complement this article. We generated this data follow- Using these candidates and the same bottleneck, the RCNN ing KITTI parameters: same image resolution, daytime and component performs the final object classification and BB non-adverse weather, and isolated images. As in [32], we did regression (a refinement of the BBs proposed by RPN). For not focus on obtaining photo-realistic images. However, for comprehensive details the reader can refer to [13]. At train- generating V, we included standard visual post-effects such ing time, we always initialize VGG16 (FE) with ImageNet as anti-aliasing, ambient occlusion, depth of field, eye adap- weights since it is a recommended best practice. The weights tation, blooming and chromatic aberration. Later, we will see of the RPN and RCNN components are randomly initialized. how the domain shift between V and K/W, is similar to the We run each Faster R-CNN training for 40,000 iterations one between K and W; thus, being V a proper virtual-world using SGD optimizer. The learning rate starts at 0.001 and dataset for our study. Figure 3 shows virtual-world images we have used a decay of 0.1 at iterations 30,000 and 35,000. with 3D BBs framing vehicles and pedestrians, illustrating Each iteration uses a mini batch of two images. In terms of also how the 3D BBs are projected as 2D BBs covering Algorithms 1 and 2, these settings correspond to H8 . occluded object areas. As GAN-based image-to-image translation method we use For each dataset, Table 1 summarizes the number of the CycleGAN implementation in [53]. A GAN training runs images/frames and objects (vehicles and pedestrians) used for for 40 epochs using a weight of 1.0 for the identity mapping training and testing our object detectors. loss. In order to be able to use full resolution images, the train- Since our proposal does not assume a particular object ing of the GAN has been done patch-wise, with patch sizes detection architecture, for performing the experiments we of 300 × 300 pixels. The rest of parameters have been set 194448 VOLUME 8, 2020 G. Villalonga, A. M. López Peña: Co-Training for On-Board Deep Object Detection FIGURE 5. From left to right: images from V, corresponding images in VGK (i.e. processed by the V → K GAN), and images from K. Last column is just a visual reference since, obviously, there is no a one-to-one correspondence between V and K. FIGURE 6. From left to right: images from V, corresponding images in VGW (i.e. processed by the V → W GAN), and images from W. For visual comparison, the two top rows of this figure and those in Figure 5, start with the same images in V. as recommended in [53]. We train a V → K transforming The ground truth present in V is used as ground truth for VG . GAN, GK , and another for V → W, GW . Accordingly, For qualitative examples of image-to-image transformations, VGK = GK (V) denotes the set of virtual-world images trans- we refer to Figures 5 and 6, using GK and GW , respectively. formed by GK and, analogously, we have VGW = GW (V). Table 2 summarizes the rest of hyper-parameters used We will use the notation VG to refer to any of these sets. to perform our experiments. Note that, regarding our VOLUME 8, 2020 194449 G. Villalonga, A. M. López Peña: Co-Training for On-Board Deep Object Detection TABLE 2. Self-training and co-training hyper-parameters as defined in TABLE 5. UDA results for V → {K, W}, i.e. virtual to real. ASource Algorithms 1 and 2. We use the same values for both, as well as to work (adapted source) refers to VG ∈ {VGK , VGW }. X l ,tr refers to the fully with KITTI (K) and Waymo (W) datasets, except for Hseq which only labeled target-domain training set. X l̂ ,tr consists of the same images as applies to W. N, n, m, 1t 1 , and 1t 2 are set in number-of-images units, X l ,tr , but self-labeled by either self-taining (Slf-T) or co-training (Co-T). Kmin and 1K in number-of-cycles, T1 runs in [0..100]. We use the mAP Just as reference, we also show the domain shift between K and W. same confidence detection threshold for vehicles and pedestrians, which ? According to these results, as upper bound for K we take the detector runs in [0..1]. ( ) Only used in co-training, however, for m = ∞, it has no based on X l ,tr &VG , while for W it is the detector based on X l ,tr . effect. We refer to the main text for more details. TABLE 3. SSL results for K and W. We assess vehicle (V) and pedestrian (P) detection, according to the mAP metric. From X tr ∈ {Ktr , W tr }, we preserve the labeling information for a randomly chosen p% of its images, while it is ignored for the rest. We report results for p = 100 (all labels are used), p = 5 and p = 10. Slf-T and Co-T stand for self-training and co-training, resp., which refers to how images were self-labeled from the respective unlabeled training sets. TABLE 4. Number of self-labeled vehicles and pedestrians applying self-training and co-training, for the SSL (5% & 10%) and UDA (Source & ASource) settings, for KITTI (K) and Waymo (W). In parenthesis we indicate the percentage of false positives. The top block corresponds to ground truth labels in the full training sets, and the percentages used for SSL. After removing false positives, in each block of rows, the corresponding 1X shows how many more objects are labeled by co-training compared to self-training. self-labeling approaches, we use the same values for K and W, with the only exception that for W we also con- sider that it is composed of video sequences. Moreover, we also use the same values for hyper-parameters shared by self-training and co-training. We relied on the meaning of the hyper-parameters to set them with reasonable values. FIGURE 7. Eventual detection performance (mAP) of self-training and co-training as a function of the stopping cycle, in the UDA setting. Upper Then, during the experiments of the 5% with self/co-training and lower bounds are included as visual reference. We refer to the main (see Table 3) we did visual inspection of the self-labeled text for more details. images to ensure the methods were working well. In this process, we noted that, since the confidence thresholds (T ) co-training it was better to send all the images self-labeled were already avoiding too erroneous self-labeled images, for by the detector φi to detector φj , i, j ∈ {1, 2}, i 6= j, so that 194450 VOLUME 8, 2020 G. Villalonga, A. M. López Peña: Co-Training for On-Board Deep Object Detection FIGURE 8. Examples of Self/Co-training + ASource. Red BBs are from the ground truth of Ktr , and green BBs are predicted. Each block of two columns with the same underlying image compares self-training (left column of the block) and co-training (right column of the block). Top row corresponds to detections in Cycle 0, when, in these examples, the only available training data is VGK (so it is the same for self-training and co-training). The following rows, top to bottom, correspond to detections from cycles 1, 10, and 20, respectively, when self-labeled images are incrementally added to the training set. FIGURE 9. Analogous to Figure 8 for W tr and VGW . φj can select the n most difficult for it among them. This is Table 3 shows the results when using the 100% of the respec- equivalent to set m = ∞ in Algorithm 2 (so nullifying the tive real-world training data, i.e. X l = X tr ∈ {Ktr , W tr }, parameter). as well as when randomly selecting the 5% or the 10% of Algorithms 1 and 2 return self-labeled images, i.e. X l̂ . For X tr as X l set, meaning that these subsets are created once our experiments, we use X l̂ together with the input labeled and frozen for all the experiments. Table 4 shows the number set, X l , to train a final Faster R-CNN object detector out of of object instances induced by the random selection in each the self-labeling cycles but using the same training settings. case. In Table 3, the 100% case shows the upper-bound This detector is the one used for testing. Finally, to measure performance, and the 5% and 10% act as lower bounds. object detection performance, we follow the KITTI bench- We can see that, in all the cases, the self-labeling mark mean average precision (mAP) metric [11]. methods outperform the lower bounds. For vehicles (V), self-training and co-training perform similarly for the 5% lower bound, while for the 10% self-training performs bet- B. RESULTS ter than co-training in K but worse in W. For pedestri- We start by assessing the self-labeling algorithms in a pure ans (P), co-training always performs better. Looking at the SSL setting, working only with either the K or W dataset. vehicle-pedestrian combined (V&P) performance, we see VOLUME 8, 2020 194451 G. Villalonga, A. M. López Peña: Co-Training for On-Board Deep Object Detection TABLE 6. Results for two new settings: (/FP) assuming we remove the self-labeled false positives; (/FP+BB) assuming that, in addition, for the self-labeled instances, we change the predicted BB by the corresponding one in the ground truth. The 1X rows show differences between these variants and the respective original one (i.e. neither removing the FP nor adjusting the BBs). The bottom block of rows remarks the differences between the best self-labeling (Co-T+ASource, including /FB and /FB+BB cases) and the upper bound. significant improvements over the lower bounds. Interest- ingly, for the 10% setting, co-training even outperforms the upper bound for pedestrians in W. In fact, in this case, the corresponding V&P performance is just 1.55 points below FIGURE 10. Examples of misalignment between ground truth BBs (red) the upper bound. The same setting in K, improves 5.3 points and self-labeled ones (green). Occlusion is the underlying problem, giving rise to shorter BBs (top and middle) or BBs fusing several instances in over the lower bound, but is 7.4 points below the upper bound. one (bottom). These experiments show that our self-labeling algorithms, especially co-training, are performing the task of SSL reason- ably well, which encourages to address the UDA challenge training set of W). Thus, without any manual training data with them. labeling, we are almost reaching upper-bound performance. Table 5 shows the UDA results for V → K and V → W, A second observation is that, indeed, co-training brings thus, training with V acts as lower bound. Just as reference, additional improvements on top of image-to-image trans- we also show the results of training on K and testing on W, lation; in terms of V&P, 5.77 points for K and 4.94 and vice versa. The former case shows a similar domain shift for W. Note that, when applying them separately to the as when training on V. The latter case shows a significant virtual-world images (source domain), both co-training and lower shift from V to K, than from W to K. Thus, we think CycleGAN (ASource) show similar performance improve- that V offers a realistic use case to assess virtual-to-real UDA. ments for W but co-training outperforms CycleGAN in K; The 1X rows at the bottom block of Table 5 summarize in terms of V&P, co-training improves 7.22 points for K and numerically the main insights. 6.32 for W, while CycleGAN improves 4.45 and 6.62 points, A first observation is that, by combining GAN-based respectively. Interestingly, CycleGAN performs better than image-to-image translation and co-training, we obtain signif- co-training for vehicles and it is the opposite for pedestri- icant performance improvements over the lower bounds in all ans. In any case, using both together outperforms them in cases; in terms of V&P, 10.22 points for K and 11.56 for W. isolation. In fact, the gap to reach upper-bound performances, is rel- A third observation is that co-training consistently out- atively small compared to the improvement over the lower performs self-training. Moreover, in Figure 7 we can see bounds; in terms of V&P, such gap is of 0.54 points for K that this is also consistent along self-labeling cycles (we and of 2.04 for W. Note that for K, the upper bound comes show the UDA setting). It plots curves illustrating how the from the X l,tr &VG setting (i.e. training on the full labeled self-training and co-training strategies would perform as a training set of K plus VGK ); while for W, the upper bound function of the stopping cycle. We collect the respective comes from the X l,tr setting (i.e. training on the full labeled self-labeled images at different cycles (x-axis) and train an 194452 VOLUME 8, 2020 G. Villalonga, A. M. López Peña: Co-Training for On-Board Deep Object Detection FIGURE 11. Results for Ktt (top ‘Source → Co-T + Asource’ block) and W tt (bottom ‘Source → Co-T + Asource’ block). Red BBs are the ground truth, and green ones are the detections done by the detector indicated at the first column of each row. object detector as these cycles were determined as the stop- so collecting the corresponding mAP (y-axis) per each ping ones. Then, we assess the performance of the detector considered cycle. We see how self-training curves oscillate in the corresponding testing set (either from K or W), more around the respective lower bounds, while co-training VOLUME 8, 2020 194453 G. Villalonga, A. M. López Peña: Co-Training for On-Board Deep Object Detection ones keep improving performance as we train more cycles. co-training as SSL strategy for self-labeling objects in Moreover, in Table 4, we can see how co-training sys- images. Moreover, we have focused on the challenging sce- tematically self-labels more correct object instances than nario where the initial labeled set is generated automatically self-training (1X rows show the increment for SSL and UDA in a virtual world; thus, co-training must actually perform settings). UDA. We have proposed a specific co-training algorithm Figures 8 and 9 show qualitative examples of how which is agnostic to the particular object detector used for self-training and co-training progress in Ktr and W tr images, self-labeling. We have devised a comprehensive set of exper- respectively; in this case, starting with VGK and VGW as iments addressing the challenging task of on-board vehicle corresponding initial labeled training sets. Both self-labeling and pedestrian detection, using de facto standards such as strategies improve over the starting point, since they can cor- KITTI and Waymo datasets, together with a virtual-world rect BB localization errors, remove false positives, and recov- dataset introduced in this article. Our qualitative results ering from false negatives. In some cases, self-training and allow us to conclude that co-training and GAN-based image- co-training show the same final right result, but co-training to-image translation complement each other up to allow the reaches it in earlier cycles, while in other cases co-training training of object detectors without manual labeling, while shows better final results. still reaching almost the same performance as by totally As summarized in Table 6, with additional experiments, relying on human labeling for obtaining upper-bound perfor- we have further analyzed co-training results. We repeat the mances. These results show that the self-labeled objects are training and evaluation of all the detectors developed for both sufficient to train a well-performing object detector, but also the SSL and UDA settings, considering two variants in the that improving BB adjustment is convenient to improve its respective training sets. In (/FP) we remove the false positives performance. Accordingly, our future work will address this from the self-labeled data. In (/FP+BB), in addition, for those point, for instance, by developing a multi-modal co-training self-labeled instances that are true positives, we replace the which jointly explores RGB images (as in this article) as BBs predicted during self-labeling by the corresponding BB well as depth information based on monocular depth estima- ground truth. In this way, we can incrementally analyze the tion [14], where object borders may be better localized. effect of false positives and BB adjustment accuracy. Table 6 shows how the impact of having false positives ACKNOWLEDGMENT as training data is not as strong as one may think a priori. The authors would like to thank several past collaborators and For instance, in Table 4 we can see that co-training with visiting interns of the Computer Vision Center (CVC). They CycleGAN (Co-T + ASource) has output a 28.8% of false thank Dr. Hamed H. Aghdam for starting discussions on the vehicles in W and a 3.1% in K, a large difference; however, use of co-training for computer vision tasks. They also thank Table 6 indicates that removing them results in a vehicle the interns Mojtaba Valipour and, especially, Sahil Gupta, for detection improvement of just 0.71 points in the former case, helping in the early stages of the research, when they were and 0.17 in the latter. This may be linked with the fact that programming exploratory ideas under their supervision. They SGD optimization in deep neural networks (DNNs) tend to also thank Jose L. Gómez for the help in the synthetic data prioritize learning patterns instead of noise [3]. generation. In general, as Table 6 shows, a better BB adjustment has a higher impact than removing false positives, especially REFERENCES for vehicles. In fact, for the higher performing detector, i.e. [1] Y. Abramson and Y. Freund, ‘‘SEmi-automatic VIsuaL LEarning co-training with CycleGAN, this adjustment would allow (SEVILLE): A tutorial on active learning for visual object recognition,’’ to even outperform the upper bound. This improvement is in Proc. Int. Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2005. [2] H. H. Aghdam, A. Gonzalez-Garcia, A. Lopez, and J. Weijer, ‘‘Active mainly coming from the detection of vehicles. For instance, learning for deep detection neural networks,’’ in Proc. IEEE/CVF Int. Conf. Figure 10 shows examples of the typical BB misalignment we Comput. Vis. (ICCV), Oct. 2019, pp. 3672–3680. have found, mainly due to occluded vehicle instances. Note [3] D. Arpit, S. Jastrzebsk, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, also that the (/FP+BB) results indicate that non-self-labeled T. Maharaj, A. Fischer, A. Courville, Y. Bengio, and S. Lacoste-Julien, ‘‘A closer look at memorization in deep networks,’’ in Proc. Int. Conf. objects (false negatives) do not cause any loss of perfor- Mach. Learn. (ICML), 2017, pp. 233–242. mance, unless they would result in better BB adjustments. [4] M.-F. Balcan, A. Blum, and K. Yang, ‘‘Co-training and expansion: Towards We finish by showing qualitative results on Ktt and W tt bridging theory and practice,’’ in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2004, pp. 89–96. (Figure 11), for different object detectors. Comparing ground [5] A. Blum and T. Mitchell, ‘‘Combining labeled and unlabeled data with co- truth BBs and detections, we appreciate how it is confirmed training,’’ in Proc. 11th Annu. Conf. Comput. Learn. Theory - COLT, 1998, what is expected from the quantitative results, i.e. co-training pp. 92–100. [6] O. Chapelle, B. Schölkopf, and A. Zien, Semi-Supervised Learning. Cam- combined with GAN-based image-to-image translation is bridge, MA, USA: MIT Press, 2006. providing the most accurate results. [7] M. Chen, K. Weinberger, and J. Blitzer, ‘‘Co-training for domain adaptation,’’ in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2011, V. CONCLUSION pp. 2456–2464. [8] M. Chen, K. Weinberger, and Y. Chen, ‘‘Automatic feature decomposition Motivated by the burden of manual data labeling when for single view co-training,’’ in Proc. Int. Conf. Mach. Learn. (ICML), addressing vision-based object detection, we have explored 2011, pp. 953–960. 194454 VOLUME 8, 2020 G. Villalonga, A. M. López Peña: Co-Training for On-Board Deep Object Detection [9] Y. Chen, W. Li, C. Sakaridis, D. Dai, and L. Van Gool, ‘‘Domain [31] S. Ren, K. He, R. Girshick, and J. Sun, ‘‘Faster R-CNN: Towards adaptive faster R-CNN for object detection in the wild,’’ in real-time object detection with region proposal networks,’’ IEEE Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2018, Trans. Pattern Anal. Mach. Intell., vol. 39, no. 6, pp. 1137–1149, pp. 3339–3348. Jun. 2017. [10] G. Csurka, ‘‘A comprehensive survey on domain adaptation for visual [32] G. Ros, L. Sellart, J. Materzynska, D. Vazquez, and A. M. Lopez, ‘‘The applications,’’ in Domain Adaptation in Computer Vision Applica- SYNTHIA dataset: A large collection of synthetic images for semantic tions (Advances in Computer Vision and Pattern Recognition). Cham, segmentation of urban scenes,’’ in Proc. IEEE Conf. Comput. Vis. Pattern Switzerland: Springer, 2017, ch. 1. Recognit. (CVPR), Jun. 2016, pp. 3234–3243. [11] A. Geiger, P. Lenz, and R. Urtasun, ‘‘Are we ready for autonomous driving? [33] C. Rosenberg, M. Hebert, and H. Schneiderman, ‘‘Semi-supervised The KITTI vision benchmark suite,’’ in Proc. IEEE Conf. Comput. Vis. self-training of object detection models,’’ in Proc. 7th IEEE Pattern Recognit., Jun. 2012, pp. 3354–3361. Workshops Appl. Comput. Vis. (WACV/MOTION), vol. 1, Jan. 2005, [12] S. Gidaris, P. Singh, and N. Komodakis, ‘‘Unsupervised representation pp. 29–36. learning by predicting image rotations,’’ in Proc. Int. Conf. Learn. Rep- [34] S. Roy, A. Unmesh, and V. Namboodiri, ‘‘Deep active learning for object resent. (ICLR), 2018, pp. 1–16. detection,’’ in Proc. Brit. Mach. Vis. Conf. (BMVC), 2018, p. 91. [13] R. Girshick, I. Radosavovic, G. Gkioxari, P. Dollár, and K. He. (2018). [35] B. Settles, ‘‘Active Learning,’’ Synthesis Lectures on Artificial Intelligence Detectron. [Online]. Available: https://github.com/facebookresearch/ and Machine Learning, vol. 6. San Rafael, CA, USA: Morgan & Claypool, detectron 2012, pp. 1–114. [14] C. Godard, O. M. Aodha, M. Firman, and G. Brostow, ‘‘Digging into self- [36] C. R. de Souza, A. Gaidon, Y. Cabon, N. Murray, and A. M. López, ‘‘Gener- supervised monocular depth estimation,’’ in Proc. IEEE/CVF Int. Conf. ating human action videos by coupling 3D game engines and probabilistic Comput. Vis. (ICCV), Oct. 2019, pp. 3828–3838. graphical models,’’ Int. J. Comput. Vis., vol. 128, no. 5, pp. 1505–1536, [15] U. Guz, D. Hakkani-Tür, S. Cuendet, and G. Tur, ‘‘Co-training May 2020. using prosodic and lexical information for sentence segmentation,’’ [37] P. Sun et al., ‘‘Scalability in perception for autonomous driving: Waymo in Proc. Conf. Int. Speech Commun. Assoc. (INTERSPEECH), 2007, open dataset,’’ in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. pp. 1–4. (CVPR), Jun. 2020, pp. 2446–2454. [16] B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. W. Tsang, and M. Sugiyama, ‘‘Co-teaching: Robust training of deep neural networks with [38] I. Triguero, S. García, and F. Herrera, ‘‘Self-labeled techniques for semi- extremely noisy labels,’’ in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), supervised learning: Taxonomy, software and empirical study,’’ Knowl. Inf. 2018, pp. 8527–8537. Syst., vol. 42, no. 2, pp. 245–284, Feb. 2015. [17] N. Jaipuria, X. Zhang, R. Bhasin, M. Arafa, P. Chakravarty, S. Shrivastava, [39] G. Tur, ‘‘Co-adaptation: Adaptive co-training for semi-supervised learn- S. Manglani, and V. N. Murali, ‘‘Deflating dataset bias using synthetic data ing,’’ in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), augmentation,’’ in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Apr. 2009, pp. 3721–3724. Workshops (CVPRW), Jun. 2020, pp. 772–773. [40] J. E. van Engelen and H. H. Hoos, ‘‘A survey on semi-supervised learning,’’ [18] J. Jeong, S. Lee, J. Kim, and N. Kwak, ‘‘Consistency-based semi- Mach. Learn., vol. 109, no. 2, pp. 373–440, Feb. 2020. supervised learning for object detection,’’ in Proc. Adv. Neural Inf. Process. [41] D. Vazquez, A. M. Lopez, J. Marin, D. Ponsa, and D. Geronimo, Syst. (NeurIPS), 2019, pp. 10759–10768. ‘‘Virtual and real world adaptation for pedestrian detection,’’ IEEE [19] D. Kim, D. Cho, D. Yoo, and I. S. Kweon, ‘‘Learning image representations Trans. Pattern Anal. Mach. Intell., vol. 36, no. 4, pp. 797–809, by completing damaged jigsaw puzzles,’’ in Proc. IEEE Winter Conf. Appl. Apr. 2014. Comput. Vis. (WACV), Mar. 2018, pp. 793–802. [42] K. Wang, X. Yan, D. Zhang, L. Zhang, and L. Lin, ‘‘Towards human- [20] S. Kim, J. Choi, T. Kim, and C. Kim, ‘‘Self-training and adversarial machine cooperation: Self-supervised sample mining for object detection,’’ background regularization for unsupervised domain adaptive one-stage in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2018, object detection,’’ in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), pp. 1605–1613. Oct. 2019, pp. 6092–6101. [43] M. Wang and W. Deng, ‘‘Deep visual domain adaptation: A survey,’’ [21] A. Kolesnikov, X. Zhai, and L. Beyer, ‘‘Revisiting self-supervised visual Neurocomputing, vol. 312, pp. 135–153, Oct. 2018. representation learning,’’ in Proc. IEEE/CVF Conf. Comput. Vis. Pattern [44] W. Wang and Z.-H. Zhou, ‘‘Analyzing co-training style algorithms,’’ in Recognit. (CVPR), Jun. 2019, pp. 1920–1929. Proc. Eur. Conf. Mach. Learn. (ECML), 2007, pp. 454–465. [22] Levin, Viola, and Freund, ‘‘Unsupervised improvement of visual detectors [45] W. Wang and Z.-H. Zhou, ‘‘A new analysis of co-training,’’ in Proc. Int. using cotraining,’’ in Proc. 9th IEEE Int. Conf. Comput. Vis., Oct. 2003, Conf. Mach. Learn. (ICML), 2010, pp. 1135–1142. p. 626. [46] K. Weiss, T. Khoshgoftaar, and D. Wang, ‘‘A survey of transfer learning,’’ [23] L. Liu, W. Ouyang, X. Wang, P. Fieguth, J. Chen, X. Liu, and J. Big Data, vol. 3, no. 9, 2016. M. Pietikäinen, ‘‘Deep learning for generic object detection: A survey,’’ [47] G. Wilson and D. J. Cook, ‘‘A survey of unsupervised deep domain Int. J. Comput. Vis., vol. 128, no. 2, pp. 261–318, Feb. 2020. adaptation,’’ ACM Trans. Intell. Syst. Technol., vol. 11, no. 5, pp. 1–46, [24] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. Berg, Sep. 2020. ‘‘SSD: Single shot multibox detector,’’ in Proc. Eur. Conf. Comput. Vis. [48] Y. Xiang, W. Choi, Y. Lin, and S. Savarese, ‘‘Data-driven 3D voxel patterns (ECCV), 2016, pp. 21–37. for object category recognition,’’ in Proc. IEEE Conf. Comput. Vis. Pattern [25] V. S. Lokhande, S. Tasneeyapant, A. Venkatesh, S. N. Ravi, and Recognit. (CVPR), Jun. 2015, pp. 1903–1911. V. Singh, ‘‘Generating accurate pseudo-labels in semi-supervised learning [49] J. Xu, L. Xiao, and A. M. Lopez, ‘‘Self-supervised domain adaptation and avoiding overconfident predictions via Hermite polynomial activa- for computer vision tasks,’’ IEEE Access, vol. 7, pp. 156694–156706, tions,’’ in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Nov. 2019. Jun. 2020, pp. 11435–11443. [26] A. M. López, G. Villalonga, L. Sellart, G. Ros, D. Vázquez, J. Xu, J. Marín, [50] D. Yarowsky, ‘‘Unsupervised word sense disambiguation rivaling super- and A. Mozafari, ‘‘Training my car to see using virtual worlds,’’ Image Vis. vised methods,’’ in Proc. 33rd Annu. meeting Assoc. Comput. Linguistics, Comput., vol. 68, pp. 102–118, Dec. 2017. 1995, pp. 189–196. [27] K. Nigam and R. Ghani, ‘‘Analyzing the effectiveness and applicability [51] X. Yu, B. Han, J. Yao, G. Niu, I. Tsang, and M. Sugiyama, ‘‘How does of co-training,’’ in Proc. 9th Int. Conf. Inf. Knowl. Manage. CIKM, 2000, disagreement help generalization against label corruption,’’ in Proc. Int. pp. 86–93. Conf. Mach. Learn. (ICML), 2019, pp. 7164–7173. [28] A. Oliver, A. Odena, C. Raffel, E. Cubuk, and I. Goodfellow, ‘‘Realistic [52] Z.-H. Zhou and M. Li, ‘‘Semi-supervised learning by disagreement,’’ evaluation of deep semi-supervised learning algorithms,’’ in Neural Inf. Knowl. Inf. Syst., vol. 24, no. 3, pp. 415–439, Sep. 2010. Process. Syst. (NeurIPS), 2018, pp. 3235–3246. [53] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, ‘‘Unpaired image-to-image [29] S. Qiao, W. Shen, Z. Zhang, B. Wang, and A. Yuille, ‘‘Deep co-training translation using cycle-consistent adversarial networks,’’ in Proc. IEEE Int. for semi-supervised image recognition,’’ in Proc. Eur. Conf. Comput. Vis. Conf. Comput. Vis. (ICCV), Oct. 2017, pp. 2223–2232. (ECCV), Sep. 2018, pp. 135–152. [54] F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, [30] J. Redmon and A. Farhadi, ‘‘YOLO9000: Better, faster, stronger,’’ in and Q. He, ‘‘A comprehensive survey on transfer learning,’’ 2019, Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, arXiv:1911.02685. [Online]. Available: http://arxiv.org/abs/1911. pp. 7263–7271. 02685 VOLUME 8, 2020 194455 G. Villalonga, A. M. López Peña: Co-Training for On-Board Deep Object Detection [55] Y. Zou, Z. Yu, B. Kumar, and J. Wang, ‘‘Unsupervised domain adaptation ANTONIO M. LÓPEZ PEÑA (Member, IEEE) for semantic segmentation via class-balanced self-training,’’ in Proc. Eur. is currently the Principal Investigator of the Conf. Comput. Vis. (ECCV), Sep. 2018, pp. 289–305. Autonomous Driving Lab, Computer Vision Cen- [56] Y. Zou, Z. Yu, X. Liu, B. V. K. V. Kumar, and J. Wang, ‘‘Confidence ter (CVC), Universitat Autònoma de Barcelona regularized self-training,’’ in Proc. IEEE/CVF Int. Conf. Comput. Vis. (UAB). He has also a tenure position as an (ICCV), Oct. 2019, pp. 5982–5991. Associate Professor at the Computer Science Department, UAB. He has a long trajectory car- rying research at the intersection of computer vision, computer graphics, machine learning, and autonomous driving. He has been deeply involved GABRIEL VILLALONGA received the B.Sc. in the creation of the SYNTHIA dataset and the CARLA open-source simu- degree in computer science from the Universi- lator for democratizing autonomous driving research. He is actively working tat Autònoma de Barcelona (UAB), in 2014, and hand-on-hand with industry partners to bring state-of-the-art techniques to the M.Sc. degree in computer vision jointly from the field of autonomous driving. He was granted by the Catalan ICREA UAB, UPC, UPF, and UOC universities, in 2015. Academia Program. He is currently pursuing the Ph.D. degree in com- puter science focusing on tasks related to percep- tion for autonomous driving. He performed his B.Sc. final project on on-board 3-D pedestrian detection at UAB. He did his M.Sc. dissertation on pedestrian behaviour analysis. 194456 VOLUME 8, 2020