This is the accepted version of the journal article: Xiao, Yi; Codevilla Moraes, Felipe; Gurram, Akhil; [et al.]. «Multimodal end-to-end autonomous driving». IEEE Transactions on Intelligent Trans- portation Systems, Vol. 23, issue 1 (Jan. 2022), p. 537-547. 11 pàg. DOI 10.1109/TITS.2020.3013234 This version is available at https://ddd.uab.cat/record/274828 under the terms of the license 1 Multimodal End-to-End Autonomous Driving Yi Xiao, Felipe Codevilla, Akhil Gurram, Onay Urfalioglu, Antonio M. López Abstract—A crucial component of an autonomous vehicle (AV) through a data-driven supervised learning process as is char- is the artificial intelligence (AI) is able to drive towards a desired acteristic of modern AI. End-to-end driving models can accept destination. Today, there are different paradigms addressing high-level navigation commands [38–41], or be restricted to the development of AI drivers. On the one hand, we find modular pipelines, which divide the driving task into sub-tasks specific navigation sub-tasks such as lane keeping [42–44] and such as perception and maneuver planning and control. On longitudinal control [45]. the other hand, we find end-to-end driving approaches that Driving paradigms are highly relying on convolutional try to learn a direct mapping from input raw sensor data neural netwoks (CNNs). In this context, one of the main to vehicle control signals. The later are relatively less studied, advantages of modular pipelines is the ability to explain the but are gaining popularity since they are less demanding in terms of sensor data annotation. This paper focuses on end-to- decisions of the AI driver in terms of its modules; which end autonomous driving. So far, most proposals relying on this is more difficult for pure end-to-end driving models [46– paradigm assume RGB images as input sensor data. However, 48]. However, developing some of the critical modules of AVs will not be equipped only with cameras, but also with active the modular paradigm requires hundreds of thousands of sensors providing accurate depth information (e.g., LiDARs). supervised data samples [49, 50], e.g. raw sensor data with Accordingly, this paper analyses whether combining RGB and depth modalities, i.e. using RGBD data, produces better end-to- ground truth (GT). Since the GT is most of the times provided end AI drivers than relying on a single modality. We consider manually (e.g. annotation of object bounding boxes [51], multimodality based on early, mid and late fusion schemes, both pixel-level delineation of semantic classes [52]), this is an in multisensory and single-sensor (monocular depth estimation) important bottleneck for this paradigm. Conversely, end-to-end settings. Using the CARLA simulator and conditional imitation approaches are able to learn CNN-based models for driving learning (CIL), we show how, indeed, early fusion multimodality outperforms single-modality. from raw sensor data (i.e. without annotated GT) and associ- ated supervision in terms of vehicle’s variables (e.g. steering Index Terms—Multimodal scene understanding, End-to-end angle, speed, geo-localization and orientation [37, 53, 54]); autonomous driving, Imitation learning. note that such supervision does not require human intervention in terms of explicitly annotating the content of the raw sen- I. I NTRODUCTION sor data. Moreover, end-to-end models are demonstrating an UTONOMOUS vehicles (AVs) are core for future mobil- unreasonable effectiveness in practice [36, 39, 43, 45], which A ity. Thus, it is essential to develop artificial intelligence (AI) for driving AVs. Two main paradigms are under research, makes worth to go deeper in their study. Although AVs will be multisensory platforms, equipping namely, modular pipelines and end-to-end driving. and maintaining on-board synchronized heterogeneous sensors The modular paradigm attaches to the traditional divide-and- is quite expensive nowadays. As a consequence, most end-to- conquer engineering principle, since AI drivers rely on mod- end models for driving rely only on vision [35–37, 39, 42, 43, ules with identifiable responsibilities; for instance, to provide 45, 55–58], i.e. they are visuomotor models. This is not bad in environmental perception [1, 2], as well as route planning and itself, after all, human drivers mainly rely on vision. However, maneuver control [3, 4]. Perception itself is already especially multimodality has shown better performance in key perception complex, since it involves sub-tasks such as object detection sub-tasks such as object detection [7–10, 59–63], tracking [15], [5–10] and tracking [11–15], traffic sign recognition [16], and semantic segmentation [21, 22]. Thus, it is worth exploring semantic segmentation [17–22], monocular depth estimation multimodality for end-to-end driving. [23–26], SLAM and place recognition [27–33], etc. Accordingly, in this paper we address the question can an The end-to-end driving paradigm focuses on learning holis- end-to-end driving model be improved by using multimodal tic models is able to directly map raw sensor data into control sensor data over just relying on a single modality? In par- signals for maneuvering AVs [34–37], i.e. without forcing ticular, we assume color images (RGB) and depth (D) as explicit sub-tasks related to perception or planning. Thus, single modalities, and RGBD as multimodal data. Due to its advocating for learning to perceive and act simultaneously, as capability of accepting high level commands, this study is humans do. Moreover, such sensorimotor models are obtained based on the CNN architecture known as conditional imitation learning (CIL) [39]. We explore RGBD from the perspective Yi, Felipe, and Antonio are with the Computer Vision Center (CVC) of early, mid and late fusion of the RGB and D modalities. and the Univ. Autònoma de Barcelona (UAB). Akhil and Onay are with Moreover, as in many recent works on end-to-end driving Huawei GRC in Munich. CVC members acknowledge the financial support by the Spanish TIN2017-88709-R (MINECO/AEI/FEDER, UE). Antonio M. [39–41, 48, 56, 57, 64], our experiments rely on the CARLA López acknowledges the financial support by ICREA under the ICREA simulator [65]. Academia Program. Felipe Codevilla acknowledges Catalan AGAUR for his The presented results show that multimodal RGBD end- FI grant 2017FI-B1-00162. Yi Xiao acknowledges the Chinese Scholarship Council (CSC), grant number 201808390010. We also thank the Generalitat to-end driving models outperform their single-modal counter- de Catalunya CERCA Program, as well as its ACCIO agency. parts. Moreover, early fusion shows better performance than 2 mid and late fusion. On the other hand, multisensory RGBD B. End-to-end driving (i.e. based on camera and LiDAR) still outperforms monocular RGBD; however, we conclude that it is worth pursuing this Pomerleau presented ALVINN three decades ago [34], special case of single-sensor multimodal end-to-end models. a sensorimotor fully-connected shallow neural network that We present the work as follows. Sect. II reviews the related was able to perform end-to-end road following assuming no literature. Sect. III presents the used CIL architecture from obstacles. ALVINN controlled a CMU’s van, NAVLAB, along the point of view of early, mid, and late fusion. Sect. IV a 400m straight path, at ∼ 2 Km/h and under good weather summarizes the experimental setting and the obtained results. conditions. Although the addressed scenario is extremely sim- Finally, Sect. V draws the main conclusions and future work. ple compared to driving in real traffic, it was already necessary to simulate data for training the sensorimotor model and, II. R ELATED WORK in fact, camera images (30 × 32 pixels, blue channel) were This section focuses on two main related topics: multimodal already combined by early fusion with laser range finder data perception and end-to-end driving models learned by imita- (8 × 32 depth cells). LeCun et al. [35] trained end-to-end a 6- tion. layer CNN for off-road obstacle avoidance using image pairs (from a stereo rig) as input. Such CNN was able to control a A. Multimodality 50cm-length four-wheel truck, DAVE, for avoiding obstacles Object detection is one of the perception tasks for which at a speed of ∼ 7 Km/h. During data collection for training, multimodality has received most attention. Enzweiler et al. the truck was remotely controlled by a human operator, thus, [66] developed a pedestrian detector using hand-crafted fea- the CNN was trained according to imitation learning in our tures and shallow classifiers combined as a mixture-of-experts terminology (or teleoperation-based demonstration [70]). More (MoE), where multimodality relies on image luminance and recently, Bojarski et al. [36] developed a vision-based end-to- stereo depth. Gonzalez et al. [60] detected vehicles, pedes- end driving CNN which was able to control the steering wheel trians and cyclists—vulnerable road users (VRUs)—, using of a real car in different traffic conditions. Still, lane and road a multimodal MoE based on space-time calibrated RGB and changing are not considered, neither stop-and-go maneuvers LiDAR depth. Chen et al. [61] used calibrated RGB and Li- since throttle and brake are not controlled. DAR depth as input for a CNN-based detector of vehicles and These pioneering works inspired new proposals based on VRUs, which is a current trend [8, 9, 61–63]. Some of these imitation learning for CNNs. Eraqi et al. [44] applied vision- works are inspired by Faster R-CNN [5], since they consist of a based end-to-end control of the steering angle (neither throttle first stage for proposing regions potentially containing objects nor break), focusing on including temporal reasoning by of interest, and a second stage performing the classification of means of long short-term memory recurrent neural networks those regions to provide final object detections; i.e. following a (LSTMs). Training and testing were done in the Comma.ai mid-level (deep) fusion scheme where CNN layers of features dataset [53]. George et al. [45] applied similar ideas for from the different modalities are fused in both stages [61–63]. controlling the speed of the car. Xu et al. [37] presented the Other alternatives are early fusion at raw data level [9], late BDD dataset and focused on vision-based prediction of the fusion of independent detectors [8, 9], or just using different steering angle using a fully convolutional network (FCN) and modalities at separated steps of the detection pipeline [7]. a LSTM, forcing semantic segmentation as auxiliary training Other approaches focus on multispectral appearance, as in Li task. Innocenti et al. [42] performed vision-based end-to-end et al. [10], where different fusion schemes for RGB and Far steering angle prediction for lane keeping on private datasets, Infrared (FIR) calibrated images are compared. and Chen et al. [43] in the Comma.ai dataset. All these studies and recent surveys [67, 68] show that Affordances have been proposed as intermediate tasks be- detection accuracy increases with multimodality. Therefore, tween enviromental perception and prediction of the vehicle more perception tasks have been addressed under the multi- control parameters [55, 56]. Affordances do not require to modal approach. Dimitrievski et al. [15] proposed a pedestrian solve perception sub-tasks such as explicit object detection, tracker that fuses camera and LiDAR detections to solve the etc; but they form a compact set of factors that influence data association step of their tracking-by-detection approach. driving according to prior human knowledge. Chen et al. Schneider et al. [21] proposed a CNN architecture for semantic [55] evaluated them on the TORCS simulator [71], so in segmentation which performs a mid-level fusion of RGB and car racing conditions (no pedestrians, no intersections, etc.) stereo depth, leading to a more accurate segmentation on under clean and dry weather; while Sauer et al. [56] used the small objects. Ha et al. [22] also proposed a mid-level RGB CARLA simulator, which supports regular traffic conditions and FIR fusion approach in a CNN architecture for semantic under different lighting and weather [65]. Muller et al. [57] segmentation. Piewak et al. [69] used a mid-level fusion of developed a vision-based CNN with an intermediate road LiDAR and camera data to produce a Stixel representation segmentation task for learning to perform vehicle maneuvers of the driving scene, showing improved accuracy in terms of in a semantic space; the driving policy consists of predicting geometry and semantics of the resulting representation. waypoints within the segmented road and applying a low-level In this paper, rather than focusing on individual perception PID controller afterwards. Training and testing are done in tasks such as object detection, tracking or semantic segmenta- CARLA, but neither incorporating other vehicles nor pedes- tion, we challenge multimodality in the context of end-to-end trians. Using LiDAR data, Rhinehart et al. [64] combined driving, exploring early, mid and late fusion schemes. imitation learning and model-based reinforcement learning to 3 predict expert-like vehicle trajectories, relying on CARLA but approach can be considered a type of mid-level fusion. without dynamic traffic participants. In contrast to these multimodal end-to-end driving ap- These end-to-end driving models do not accept high-level proaches, we assess early, mid and late level fusion schemes navigation instructions such as turn left at the next intersection without forcing intermediate representations which are not (without providing explicit distance information), which can trivial to obtain (e.g. semantic segmentation is an open prob- come from a global planner or just as voice commands from lem in itself). Moreover, we run CARLA benchmark [65], a passenger of the AV. Hubschneider et al. [38] proposed which includes dynamic actors (vehicles and pedestrians) and to feed a turn indicator in the vision-based CNN driving generalization conditions (unseen town and weather). We show model by concatenating it with features of a mid-level fully that CIL and early fusion produce state-of-the-art results. connected layer of the CNN. Codevilla et al. [39] proposed a more effective method, in which a vision-based CNN con- III. M ULTIMODAL F USION sisting of an initial block agnostic to particular navigation We first detail CIL [39], and then show how we adapt it to instructions, and a second block branched according to a leverage from multimodal perception data. subset of navigation instructions (at next intersection turn- left/turn-right/go-straight, or just keep lane). In the first block, vehicle information is also incorporated as mid-level feature A. Base CIL architecture of the CNN; in particular, current speed is used since the Fig. 1 shows the CNN implementing CIL. The observations CNN controls the steering angle, throttle, and break (Yang (CIL’s input) are twofold, perception data, p, and vehicle’s et al. [58] also reported the usefulness of speed feedback in state measurements, m. The action (CIL’s output), a, consists end-to-end driving). Experiments are performed in CARLA of vehicle controls for maneuvering. CIL includes a CNN for different traffic situations (including other vehicles and block to extract perception features, P (p); and a block of pedestrians), lighting and weather conditions. The overall fully connected layers to extract measurement features M (m). approach is termed as conditional imitation learning (CIL). A joint layer of features is formed by appending P (p) and In fact, Muller et al. and Sauer et al. leveraged from CIL. M (m); which is further processed by a new fully connected Liang et al. [41] also used CIL as imitation learning stage layer to obtain the joint features J(

), or just before refining the resulting model by applying reinforcement J(p, m) simplifying the notation. Up to this point of the neural learning. Wang et al. [40] used CIL too, but incorporating network, the processing done with the observations is common ego-vehicle heading information at the same CNN-layer level to any driving maneuver/action. However, many times, the au- as speed. tonomous vehicle reaches ambiguous situations which require These works focus on vision-based end-to-end driving. to incorporate informed decisions. For instance, when reaching Here, we explore muldimodal end-to-end driving based on a cross intersection, without incorporating a route navigation RGB and depth; which can be complementary to most of the command (e.g. from a global trajectory plan), the vehicle cited papers. Without losing generality, we chose CIL as core could only take a random decision about turning or going CNN architecture due to its effectiveness and increasing use. straight. Thus, the end-to-end driving CNN must incorporate Focusing on multimodality, Sobh et al. [72] used CARLA high-level commands, c, such as ‘in the next intersection turn to propose a CIL-based driving approach modified to process left’, or ‘turn right’, or ‘go straight’. Moreover, a will take camera and LiDAR data. In this case, the information fusion very different values depending on c. Thus, provided c takes is done by a mid-level approach; in particular, before fusion, discrete values, having specialized neural network layers for RGB images are used to generate a semantic segmentation each maneuver can be more accurate a priori. All this is which corresponds to one of the information streams reaching achieved in the CIL proposal by incorporating fully connected the fusion layers, and there are two more independent streams maneuver/action branches, Ac , selected by c (both during CNN based on LiDAR, one encoding a bird view and the other a training and vehicle self-driving). polar grid mapping. Khan et al. [73] also used CARLA to We follow the CIL architecture proposed in [39]. Therefore, propose an end-to-end driving CNN based on RGB and depth p is a RGB image of 200 × 88 pixels and 8 bits at each color images, which predicts only the steering angle, assuming that channel, m is a real value with the current speed of the vehicle, neither other vehicles nor pedestrians are present. In a first and a consists of three real-valued signals which set the next step, the CNN is trained only using depth information (taken maneuver in terms of steering angle, throttle, and brake. Thus, as the Z-buffer produced by UE4, the game engine behind the idea is to perform vision-based self-driving, as well as tak- CARLA). This CNN has an initial block of layers (CNN ing into account the vehicle speed to apply higher/lower throt- encoder) that outputs depth-based features, which are later tle and brake for the same perceived traffic situation. In [39], used to predict the steering angle with a second block of fully the focus is on handling intersections, then the considered c connected layers. In a second step, this angle-prediction block values are {turn-left, turn-right, go-straight, continue}, where is discarded and replaced by a new fully connected one. This the last refers to just keep driving in the current lane and the new block relies on the fusion of the depth-based features and others inform about what to do when reaching next intersection a semantic segmentation produced by a new CNN block that (which is an event detected by the own CNN). Accordingly, processes the RGB image paired with the depth image. During there are four branches Ac . Therefore, if we term by F the training, semantic segmentation is conditioned to depth-based end-to-end driver, we have F (p, m, c) = Ac (J(p, m)). As features due to the fusion block and back-propagation. This shown in [39], this manner of explicitly taking into account 4 P (p) c Conv. FC Branch Concatenation Image Steering Brake Throttle J (p, m) p Measurement M (m) m Ac Action a Fig. 1. CIL branched architecture: vehicle maneuvers (actions) in the form of the triplet , depend on a high-level route navigation command (branch selector) running on {turn-left, turn-right, go-straight, continue}, as well as on the world observations in the form of perception data (e.g. a RBG image) and vehicle state measurements (e.g. speed). high-level navigation commands is more effective than other multi-task learning helps to obtain more accurate CIL net- alternatives. works. In particular, we add an additional branch of three fully connected layers to predict current vehicle speed from B. Fusion schemes the perception data features P (p). This prediction relies on Fig. 2 illustrates how we fuse RGB and depth information a L1 loss `sp (s, sgt ) = |s − sgt |, where s is the predicted following mid, early and late fusion approaches. speed and sgt is the ground truth speed which, in this case, Early fusion: with respect to the original CIL we only is already available since it corresponds to the measurement change the number of channels of p from three (RGB) to four used as input. Speed prediction is only used during training. (RGBD). The CIL network only changes the first convolutional Thus, all networks, i.e. both single- and multimodal, are layer of P (p) to accommodate for the extra input channel, the trained according to the same total loss `(a, agt , w; c; s, sgt ) = rest of the network is equal to the original. β`act (a, agt , w; c) + (1 − β)`sp (s, sgt ), where β is used to Mid fusion:we replicate twice the perception processing balance the relevance of `act and `sp losses. P (p). One of the P (p) blocks processes only RGB images, the other one only depth images. Then, we build the joint IV. E XPERIMENTS feature vector

which is further pro- We start by summarizing the environment we use for our cessed to obtain J(RGB, D, m). From this point, the branched experiments, i.e. CARLA (Sect. IV-A). Next, we describe part of CIL is the same as in the original architecture. the driving benchmark available in CARLA (Sect. IV-B), the Late fusion: we replicate twice the full CIL architecture. dataset we use for training our AI drivers (Sect. IV-C), and Thus, RGB and depth channels are processed separately, the training protocol that we follow (Sect. IV-D). Finally, we but the measurements are shared as input. Hence, we run present and discuss the obtained results (Sect. IV-E). Ac (J(RGB, m)) and Ac (J(D, m)), and their outputs are concatenated and further processed by a module of fully connected layers, the output of which conveys the final action A. Environment values. Note that this is a kind of mixture-of-experts approach, In order to conduct our experiments, we rely on the open where the two experts are jointly trained. source driving simulator CARLA [65]. There are several As is common practice in the literature, we assume a pixel- reasons for this. First, many recent previous works on end-to- level correspondence of all channels and normalize all of them end driving rely on CARLA [39–41, 48, 56, 57, 64]; thus, we to be in the same magnitude range (we normalize depth values can compare our results with the previous literature. Second, to match the range of color channels, i.e. [0 . . . 255]). it seems that for some scenarios the end-to-end paradigm may need exponentially more training samples than the modular C. Loss function one [76], so there is a trade-off between collecting driving Given a predicted action a, its ground truth agt , and a runs (for the end-to-end paradigm) and manually annotating vector Pn of weights w, we use the L1 loss `act (a, agt , w) = on-board acquired data (for the modular paradigm) which, gt i |wi (ai −ai )|, with n = 3 (steering angle, throttle, brake). together with the gigantic effort needed to demonstrate that Note that when computing a, only one Ac branch is active at a an AV outperforms human drivers, really encourages to rely time. In particular, the one selected by the particular command on simulators during the development of AI drivers [77]. Yet, a c associated to the current input data (p, m). We make this third and core reason is specific for end-to-end driving models. fact explicit by changing the notation to `act (a, agt , w; c). In particular, in [78] it is demonstrated that current offline In addition, as in other computer vision problems addressed evaluation metrics (i.e. based on static datasets) for assessing by deep learning [74, 75], we empirically found that using end-to-end driving models do not correlate well-enough with 5 P (RGB) P (RGBD) c J (RGBD, m) J (RGB, m) RGB RGBD M (m) Measurement M (m) Ac Ac Action a m Measurement c (1) Early Fusion (EF) m M (m) P (RGB) Action a J (D, m) c RGB Depth P (D) P (D) J (RGB, D, m) Ac (3) Late Fusion (LF) Depth c A Action a Measurement M (m) m Conv. FC Branch Concatenation Steering Brake Throttle (2) Mid Fusion (MF) Fig. 2. Network Architectures - we explore RGBD from the perspective of early, mid and late fusion of the RGB and Depth (D) modalities. (1) Early Fusion: the raw RGB and D channels are the input of the CIL architecture; (2) Mid Fusion: intermediate CIL feature layers from RGB and D streams are fused; (3) Late Fusion: the output (maneuver controls) of the RGB and D CIL streams are fused to output the final values after further neural processing. Fig. 3. Bird-view road maps of Town 1 (left) and Town 2 (right). actual driving, an observation also seen in [79]; therefore, it is really important to evaluate these driving models in an on- board driving regime, which is possible in a realistic simulator such as CARLA. Briefly, CARLA contains two towns (Fig. 3) based on two-directional roads with turns and intersections, buildings, vegetation, urban furniture, traffic signs, traffic lights, and dynamic objects such as vehicles and pedestrians. Town 1 deploys 2.9Km of road and 11 intersections, while Town 2 contains 1.4Km of road and 8 intersections. The different Fig. 4. Top, from Town 1: clear noon (left) and clear after rain (right). Middle, from Town 1: heavy rain noon (left) and clear sunset (right). Bottom, from towns can be travelled under six different weather conditions Town 2: wet cloudy noon (left) and soft rainy sunset (right). (Fig. 4): ‘clear noon’, ‘clear after rain’, ‘heavy rain noon’, and ‘clear sunset’, ‘wet cloudy noon’ and ‘soft rainy sunset’. •navigation with dynamic obstacles. B. Driving benchmark For each driving task, an AI driver is assessed over a total of ET driving episodes. Each episode has its own starting CARLA was deployed with a benchmarking infrastructure and destination points with an associated topological route. An for assessing the performance of AI drivers [65]. Thus, it has episode is considered as successful if the AI driver completes been used in the related literature since then and we follow it the route within a time budget. Collisions do not lead to here too. Four increasingly difficult driving tasks are defined: the termination of an episode unless the AV runs in time- • straight: the destination point is straight ahead from the out as a consequence. If we term as ES the total number starting point but no dynamic objects are present; of successfully completed episodes by the assessed AI driver, • one turn: destination is one turn away from the starting then its success rate is defined as 100 × (ES /ET ). ET is point, no dynamic objects; determined by the selected town and weather conditions. • navigation: no restriction on the location of the destina- Table I shows how the benchmark organizes towns and tion and starting points, no dynamic objects; weather conditions for training, validation, and testing; where, 6 Fig. 5. From left to right: original RGB image, semantic segmentation ground truth (for the five considered classes), CARLA depth ground truth, post-processed to be closer to the capabilities of an active depth Sensor, and monocular depth estimation from a model trained using such a depth. TABLE I TABLE II T RAINING , VALIDATION , AND TESTING SETTINGS . T RAINING IS BASED ON VP FOR FIVE TRAINING RUNS FOR RGB ONLY, D EPTH (D) ONLY, AND A PRE - RECORDED DATASET. VALIDATION AND TESTING ARE BASED ON RGBD COMBINED BY EARLY (EF), MID (MF), OR LATE (LF) FUSION . ACTUAL DRIVING EPISODES . G REY MEANS ‘ NOT USED ’. D EPTH : FROM AN ACTIVE SENSOR OR ESTIMATED FROM RGB IMAGES . Training Validation Testing RGB Active Estimation (dataset) (episodes) (episodes) D EF MF LF D EF Wet cloudy noon Soft rainy sunset Towns 1 48 74 91 61 60 51 42 2 36 67 71 71 63 49 44 Clear noon 1&2 Towns 3 46 73 75 58 67 46 51 Clear after rain 1&2 4 40 68 71 74 60 59 46 Town 1 Clear sunset 5 36 68 77 52 62 51 49 Heavy rain noon irrespective of the town and weather, validation and testing resolution, with associated ground truth (see Fig. 5) consist- is always based on episodes, not in pre-recorded datasets, ing of corresponding images of dense depth and pixel-wise while training requires pre-recording a dataset. Validation is semantic classes (semantic segmentation), as well as meta- performed to select a driving model among those trained as information consisting of the high-level commands provided different trials from the same training dataset, while testing is by the navigation system (continue in lane, at next intersection performed for actually benchmarking the selected models. go straight, turn left or turn right), and car information such Regarding town and weather conditions, the benchmark as speed, steering angle, throttle, and braking. In this work, establishes four main town-weather blocks under which the we use perfect semantic segmentation to develop an upper- four driving tasks must be tested, assuming 25 episodes for bound driver. Since we focus on end-to-end driving, the each considered weather. Therefore, for each block, the ET twelve semantic classes of CARLA are mapped to five which value is different as we can deduce from Table I. In particular, we consider sufficient to develop such an upper-bound. In these are the town-weather blocks defined in the benchmark particular, we keep the original road-surface, vehicle, and with their respective ET value: pedestrian, while lane-marking and sidewalk are mapped as Training conditions: driving (i.e. running the episodes) in lane-limits (Town 1 and Town 2 only have roads with one go the same conditions as the training set (Town 1, four weather and one return lane, separated by double continuous lines), conditions), thus, ET = 100; and the remaining seven classes are mapped as other. New town: driving under the four weather conditions of the Focusing on depth information, as is common in the litera- training set but in Town 2, ET = 100; ture, we assume that RGB images have associated dense depth New weather: driving in Town 1 but under the two weather information; for instance, Premebida et al. [80] obtained it conditions not seen at training time, ET = 50; from LiDAR point clouds. In CARLA, the depth ground truth New town & weather: driving in conditions not included in is extremely accurate since it comes directly from the Z-buffer the training set (Town 2, two weather conditions), ET = 50. used during simulation rendering. In particular, depth values run from 0 to 1,000 meters and are codified with 24 bits, which means that depth precision is of ∼ 1/20 mm. This distance C. Training dataset range coverage and depth precision is far beyond from what In order to train our CNNs, we use the same dataset as even active sensors can provide. Therefore, we post-process in [78] since it corresponds to 25h of driving in Town 1, depth data to make it more realistic. In particular, we take as balancing weather conditions (Table I). Briefly, this dataset a realistic sensor reference the Velodyne information of KITTI was collected by a hard-coded auto-pilot with access to all the dataset [51]. First we trim depth values to consider only those privileged information of CARLA required for driving like an within the 1 to 100 meters interval, i.e. pixels of the depth expert. The auto-pilot kept a constant speed of 35 km/h when image with values outside this range are considered as not driving straight and reduced the speed when making turns. having depth information. Second, we re-quantify the depth Images were recorded at 10fps from three cameras: a central values to have an accuracy of ∼ 4 cm. Third, we perform forward-facing one and two lateral cameras facing 30◦ left inpainting to fill-in the pixels with no information. Finally, we and right. The central camera is the only one used for self- apply a median filter to avoid having perfect depth boundaries driving, while the images coming from the lateral cameras are between objects. The new depth images are used both during used only at training time to simulate episodes of recovering training and testing. Fig. 5 shows an example of a depth image from driving errors as can be done with real cars [36] (the from CARLA and its corresponding post-processed version. protocol for injecting noise follows [39]). Overall, the dataset During a training run we use Adam optimizer with 120 contains ∼ 2.5 millions of RGB images of 800 × 600 pixels training samples per iteration (minibatch), an initial learning 7 TABLE III M EAN AND STANDARD DEVIATION OF SUCCESS RATES ON THE ORIGINAL CARLA B ENCHMARK , BY RUNNING IT THREE TIMES . CIL BASED ON PERFECT SEMANTIC SEGMENTATION (SS) ACTS AS UPPER BOUND . E XCLUDING SS, FOR MODELS TESTED UNDER THE SAME ENVIRONMENT AND TRAFFIC CONDITIONS , WE SHOW IN BOLD THE HIGHER MEANS AND WE UNDERLINE SIMILAR SUCCESS RATES CONSIDERING STANDARD DEVIATIONS TOO . Active Estimated Active Estimated Task SS RGB D EF MF LF D EF SS RGB D EF MF LF D EF Training Conditions New Town Straight 98.00 ± 1.73 96.33 ± 1.53 98.67 ± 1.53 98.33 ± 0.58 92.33 ± 2.08 99.00 ± 0.00 92.33 ± 1.15 97.33 ± 1.15 100.00 ± 0.00 84.00 ± 2.00 94.33 ± 0.58 96.33 ± 0.58 87.00 ± 1.00 77.00 ± 0.00 78.33 ± 1.53 71.67 ± 2.08 One turn 100.00 ± 0.00 95.00 ± 0.00 92.00 ± 0.00 99.00 ± 0.00 91.67 ± 2.08 90.33 ± 0.58 84.67 ± 1.15 96.33 ± 1.53 96.67 ± 0.58 68.00 ± 1.00 74.33 ± 2.52 79.00 ± 1.73 78.00 ± 2.65 58.67 ± 2.08 46.33 ± 1.15 47.00 ± 1.00 Navigation 96.00 ± 0.00 89.00 ± 2.00 89.33 ± 2.08 92.67 ± 1.15 90.67 ± 1.15 93.67 ± 0.58 75.33 ± 1.15 94.33 ± 0.58 96.00 ± 0.00 59.67 ± 3.06 85.33 ± 1.15 90.00 ± 2.00 80.67 ± 0.58 52.33 ± 0.58 45.67 ± 3.06 46.67 ± 3.06 Nav.Dynamic 92.00 ± 1.00 84.00 ± 2.00 82.67 ± 0.58 89.33 ± 0.58 78.33 ± 2.89 89.00 ± 2.65 71.00 ± 1.00 89.67 ± 1.15 99.33 ± 0.58 54.33 ± 3.79 70.33 ± 1.15 84.33 ± 2.52 73.67 ± 2.52 55.67 ± 2.31 44.33 ± 2.52 46.67 ± 4.04 New Weather New Town & Weather Straight 100.00 ± 0.00 84.00 ± 0.00 99.33 ± 1.15 96.00 ± 2.00 94.67 ± 3.06 96.00 ± 0.00 92.00 ± 2.00 84.67 ± 1.15 100.00 ± 0.00 84.67 ± 1.15 97.33 ± 1.15 97.33 ± 2.31 88.67 ± 1.15 97.33 ± 1.15 78.00 ± 0.00 89.33 ± 1.15 One turn 100.00 ± 0.00 76.67 ± 4.16 94.67 ± 2.31 94.67 ± 2.31 94.00 ± 2.00 92.00 ± 2.00 93.33 ± 2.31 80.67 ± 1.15 96.00 ± 0.00 66.67 ± 4.62 72.67 ± 1.15 82.67 ± 2.31 69.33 ± 3.06 67.33 ± 2.31 62.67 ± 1.15 64.00 ± 3.46 Navigation 95.33 ± 1.15 72.67 ± 2.31 89.33 ± 1.15 91.33 ± 2.31 90.67 ± 3.06 96.00 ± 0.00 73.33 ± 2.31 80.67 ± 5.03 96.00 ± 0.00 57.33 ± 6.11 84.00 ± 3.46 92.67 ± 3.06 78.67 ± 3.06 72.67 ± 1.15 55.33 ± 6.11 60.67 ± 2.31 Nav.Dynamic 92.67 ± 1.15 68.67 ± 4.62 90.00 ± 2.00 86.00 ± 4.00 80.67 ± 3.06 92.67 ± 3.06 76.67 ± 4.16 77.33 ± 6.11 98.00 ± 2.00 46.67 ± 6.43 69.33 ± 2.31 94.00 ± 0.00 73.33 ± 3.06 73.33 ± 2.31 54.00 ± 4.00 49.33 ± 3.06 TABLE IV sky and very close areas (top and bottom part of the channels), S UCC . RATE COMPARISON WITH PREVIOUS METHODS ( SEE MAIN TEXT ). and down-scaled to finally obtain channels of 200 × 88 pixel Task MP RL CAL CIRL MT Active EF MP RL CAL CIRL MT Active EF resolution. In our initial experiments, we found that traditional Training Conditions New Town photometric and geometric recipes for data augmentation were Straight 98 89 100 98 98 98.33 ± 0.58 92 74 93 100 100 96.33 ± 0.58 One turn 82 34 97 97 87 99.00 ± 0.00 61 12 82 71 81 79.00 ± 1.73 not providing better driving models, thus, we do not use them. Navigation 80 14 92 93 81 92.67 ± 1.15 24 3 70 53 72 90.00 ± 2.00 Nav.dynamic 77 7 83 82 81 89.33 ± 0.58 24 2 64 41 53 84.33 ± 2.52 New Weather New Town & Weather Straight 100 86 100 100 100 96.00 ± 2.00 50 68 94 98 96 97.33 ± 2.31 E. Experimental results One turn 95 16 96 94 88 94.67 ± 2.31 50 20 72 82 82 82.67 ± 2.31 Navigation Nav.dynamic 94 89 2 2 90 82 86 80 88 80 91.33 ± 2.31 86.00 ± 4.00 47 44 6 4 68 64 68 62 78 62 92.67 ± 3.06 94.00 ± 0.00 We start the analysis of the experimental results by looking at Table II, which is produced during training and selection TABLE V of the best CIL networks. We focus first on RGB data as I NFRACTIONS ON DYNAMIC NAVIGATION IN NEW TOWN & WEATHER . well as depth based on the post-processed CARLA depth ground truth, termed here as active depth (Sect. IV-C) since Km per Event RGB Active D Active EF Sidewalk 0.86 ± 0.10 35.80 ± 1.30 16.76 ± 5.54 its accuracy and covered depth range is characteristic of active Infraction Opposite lane 0.73 ± 0.04 1.65 ± 0.24 3.29 ± 1.96 sensors (e.g. LiDAR). We see that the best (among five training Driven Km (Perfect driving: 17.30 Km) 13.62 ± 0.67 35.80 ± 1.30 20.22 ± 0.54 runs) validation performance VP is 48% when using RGB data only. So we will use the corresponding CIL model as RGB- based driver in the following experiments. Analogously, for rate of 0.0002, decreased to the half each 50K iterations. Mini- the case of using only active depth (D), the best CIL reports a batches are balanced in terms of per Ac branch samples. We performance of 74%. The best performances for early fusion set w = (0.5, 0.45, 0.05) to weight the control signals (action) (EF), mid fusion (MF), and late fusion (LF) are 91%, 74% in the loss function. Action and speed losses are balanced and 67%, respectively. Again, we take the corresponding CIL by β = 0.95. For selecting the best intermediate model of a models as drivers for the following experiments. training run, we do 500K iterations monitoring a validation Table III reports the performance of the selected models performance measurement, VP , each 100K iterations (thus, according to the original CARLA benchmark. We have in- five times). The intermediate model with highest VP is selected cluded a model trained on perfect semantic segmentation (SS) as the resulting model of the training run. Since CIL models according to the five classes considered here for self-driving are trained from the scratch, variability is expected in their (see Fig. 5). Thus, we consider this model as an upper bound. performance. Thus, for each type of model we perform five Indeed, its performance is most of the times ≥ 96, reaching training runs, finally selecting the model with the highest VP 100 several times. This also confirms that the CIL model is among those resulting from the five training runs. able to drive properly in CARLA conditions provided there Using Table I as reference, we define VP to balance training- is a proper input. We can see that, indeed, active depth is a validation differences in terms of town and weather conditions. powerful information for end-to-end driving by itself, clearly In particular, we use VP = 0.25Vw + 0.25Vt + 0.50Vwt ; where outperforming RGB in non-training conditions. However, in Vw is the success rate when validating in Town 1 and ‘soft most of the cases RGBD outperforms the use of only RGB rainy sunset’ weather (not included in training data), Vt is or only D. The most clear case is for new town and weather a success rate when validating in Town 2 (not included in with dynamic objects, i.e. for the most challenging conditions, training data) and ‘clear noon’ weather (included in training where RGB alone reaches a success rate of 46.67 ± 6.43, D data), and Vwt stands for success rate when validating in alone 69.33±2.31, but together the success rate is 94.00±0.00 Town 2 and ‘soft rainy sunset’ (neither town, nor weather are for early fusion. For a new town (irrespective of the weather part of the training data). Therefore, note that VP is a weighted conditions) early fusion clearly outperforms mid and late success rate based on 75 episodes. fusion. In any case, it is clear that multimodality improves CIL performance with respect to a single modality, which is D. Training protocol the main question we wanted to answer in this paper. All CIL models in this paper rely on the same training In order to further analyse the goodness of multimodal- protocol, partially following [39]. In all our CIL models ity, we compare it to previous single-modality methods (see original sensor channels (R/G/B/D) are trimmed to remove Sect. II). Not all the corresponding papers provide details 8 about the training methodology or training datasets; thus, this models beyond the static metrics currently used, which are comparison is solely based on the reported performances on agnostic to the task in which depth estimation is going to be the original CARLA benchmark and must be taken only as used. Note that even for evaluating the driving performance an additional reference about the goodness of multimodality. of end-to-end driving models in itself, it has been shown that Early fusion, is the smaller CNN architecture in terms of relying only on static evaluations may be misleading [78, 79]. weights, thus, we are going to focus on it for this comparison. Finally, for the RGB, Active D and EF models, we assess Table IV shows the results. MP and RL stand for modular per- additional infractions for new town and weather conditions ception and reinforcement learning, respectively. The reported with dynamic objects. Table V shows the driven Km per results are reproduced from [65]. CAL stands for conditional infraction of each model. Note that not all such infractions affordance learning and the results are reproduced from [56]. imply an accident stopping the AV. For instance, the AV can CIRL stands for controllable imitative reinforcement learning run into an opposite lane a bit without crashing with other and the results are reproduced from [41]. Finally, MT stands vehicles. As a reference, we also show the amount of driven for multi-task learning, and the results are reproduced from Km in which these measurements are based. All models are [48]. We see how, in presence of dynamic traffic participants, supposed to complete the same testing routes (i.e., same total the RGBD early fusion (with active depth) is the model with Km), termed as perfect driving in Table V. However, if a higher success rate on the original CARLA benchmark. On the model fails to follow the right path at an intersection the route other hand, such an early fusion approach can be combined would be recomputed, thus, it will need more Km to reach the with CAL or CIRL methods, they are totally compatible. We destination. On the contrary, if it fails to complete the routes, think that this comparison with previous works reinforces the the driven Km will be lower. We see that RGB and Active idea that multimodality can help end-to-end driving. EF models are not far, but Active D failed too much at taking Once it is clear that multimodality is beneficial for end-to- the right path at intersections. RGB performs the worst in all end driving, we can raise the question of whether monocular metrics. The Active D model does not run over the sidewalk depth estimation [23, 24, 81, 82] can be as effective as depth and uses the curbside as a cue that also helps on lane keeping coming from active sensors in this context. In the former except at intersections. Active D shows a good equilibrium case, it would consists on a multisensory multimodal approach, between RGB and Active depth single-modality models. while the later case would correspond to a single-sensor mul- timodal approach since both RGB and depth come from the V. C ONCLUSION same camera sensor (depth is estimated from RGB). In order to In this paper, we compare single- and multimodal perception carry out a proof-of-concept, we use our own monocular depth data for end-to-end driving. As multimodal perception data estimation model [24] (it was state-of-the-art at the moment we focus on RGB and depth, since they are usually available of its publication) fine-tuned on CARLA training dataset. in autonomous vehicles through the presence of cameras and More specifically, the dataset used for training the multimodal active sensors such as LiDAR. As end-to-end driving model CIL models is also used to fine-tune our monocular depth we use branched conditional imitation learning (CIL). Relying estimation model, i.e. using the post-processed depth channels on a well-established simulation environment, CARLA, we and corresponding RGB images. During training, we monitor assess the driving performance of single-modal (RGB, depth) the regression loss until it is stable, we do not stop training CIL models, as well as multimodal CIL models according based on the performance on validation episodes. Figure 5 to early, mid, and late fusion paradigms. In all cases, the shows and example of monocular depth estimation. depth information available in CARLA is post-processed to Analogously to the experiments shown so far, we train a obtain a more realistic range of distances and depth accuracy. CIL model based on the estimated depth as well as on the This depth is also used to train a depth estimation model corresponding multimodal (RGBD) fusion. In order to reduce so that the experiments cover multimodality not only based the burden of experiments, we use early fusion since it is on a multisensory setting (RGB and active depth) but also the best performing for the active depth case. The training based on a single-sensor setting (RGB and estimated depth). performances for model selection can be seen in Table II. We Overall, the experiments clearly allow us to conclude that use the CIL models of VP 59% and 51%, respectively. In multimodality (RGBD) is indeed a beneficial approach for validation terms, such performances are already clearly worse end-to-end driving. In fact, we plan to follow this line of work than the analogous based on active depth. Table III shows the in the near future, focusing on the single-sensor setting since results on the original CARLA benchmark. Indeed, these are better estimation models are required in order to compete with worse than using active depth, however, still when remaining the multisensory setting. In addition, we also plan to consider in the training conditions monocular-based EF outperforms other sources of multi-modality usually available in modern depth and RGB alone, and in fact shows similar performance vehicles, such as GNSS information which, even usually being as active depth. This is not the case when we change from noisy, eventually can complement direct scene sensing. training conditions since monocular depth estimation itself does not perform equally well in this case, and so happens R EFERENCES to EF. However, we think that this single-sensor multimodal [1] U. Franke, “Autonomous driving,” in Computer Vision in Vehicle Tech- setting is really worth to pursuit. Moreover, although it is out nology, 2017. [2] J. Janai, F. Güney, A. Behl, and A. Geiger, “Computer vision of the scope of this paper, we think that performing end-to-end for autonomous vehicles: Problems, datasets and state-of-the-art,” driving may be a good protocol for evaluating depth estimation arXiv:1704.05519, 2017. 9 [3] B. Paden, M. Cáp, S. Z. Yong, D. S. Yershov, and E. Frazzoli, “A [28] K. Tateno, F. Tombari, I. Laina, and N. Navab, “CNN-SLAM: Real-time survey of motion planning and control techniques for self-driving urban dense monocular SLAM with learned depth prediction,” in Int. Conf. on vehicles,” IEEE Trans. on Intelligent Vehicles, vol. 1, no. 1, pp. 33–55, Computer Vision and Pattern Recognition (CVPR), 2017. 2016. [29] X. Chen, H. Zhang, H. Lu, J. Xiao, Q. Qiu, and Y. Li, “Robust SLAM [4] W. Schwarting, J. Alonso, and D. Rus, “Planning and decision-making system based on monocular vision and LiDAR for robotic urban search for autonomous vehicles,” Annual Reviews of Control, Robotics, and and rescue,” in Safety, Security and Rescue Robotics (SSRR), 2017. Autonomous Systems, vol. 1, pp. 187–210, May 2018. [30] Y.-S. Shin, Y. S. Park, and A. Kim, “Direct visual SLAM using sparse [5] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: towards depth for camera-LiDAR system,” in Inter. Conf. on Robotics and real-time object detection with region proposal networks,” in Neural Automation (ICRA), 2018. Information Processing Systems (NIPS), 2015. [31] K. Qiu, Y. Ai, B. Tian, B. Wang, and D. Ca, “Siamese-ResNet: [6] B. Yang, W. Luo, and R. Urtasun, “PIXOR: Real-time 3D object implementing loop closure detection based on siamese network,” in detection from point clouds,” in Int. Conf. on Computer Vision and Intelligent Vehicles Symposium (IV), 2018. Pattern Recognition (CVPR), 2018. [32] H. Yin, L. Tang, X. Ding, Y. Wang, and R. Xiong, “LocNet: global [7] C. R. Qi, W. Liu, C. Wu, H. Su, and L. J. Guibas, “Frustum PointNets localization in 3D point clouds for mobile vehicles,” in Intelligent for 3D object detection from RGB-D data,” in Int. Conf. on Computer Vehicles Symposium (IV), 2018. Vision and Pattern Recognition (CVPR), 2018. [33] J. Zhu, Y. Ai, B. Tian, D. Cao, and S. Scherer, “Visual place recognition [8] A. Asvadi, L. Garrote, C. Premebida, P. Peixoto, and U. J. Nunes, in long-term and large-scale environment based on CNN feature,” in “Multimodal vehicle detection: fusing 3D-LIDAR and color camera Intelligent Vehicles Symposium (IV), 2018. data,” Pattern Recognition Letters, vol. 115, pp. 20–29, November 2018. [34] D. Pomerleau, “ALVINN: An autonomous land vehicle in a neural [9] A. Pfeuffer and K. Dietmayer, “Optimal sensor data fusion architecture network,” in Neural Information Processing Systems (NIPS), 1989. for object detection in adverse weather conditions,” in Inter. Conf. on [35] Y. LeCun, U. Muller, J. Ben, E. Cosatto, and B. Flepp, “Off-road Information Fusion (FUSION), 2018. obstacle avoidance through end-to-end learning,” in Neural Information [10] C. Li, D. Song, R. Tong, and M. Tang, “Illumination-aware Faster R- Processing Systems (NIPS), 2005. CNN for robust multispectral pedestrian detection,” Pattern Recognition, [36] M. Bojarski, D. D. Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, vol. 85, pp. 161–171, January 2019. L. D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang, J. Zhao, and [11] W. Choi, “Near-online multi-target tracking with aggregated local flow K. Zieba, “End to end learning for self-driving cars,” arXiv:1712.00409, descriptor,” in Inter. Conf. on Computer Vision (ICCV), 2015. 2016. [12] Y. Xiang, A. Alahi, and S. Savarese, “Learning to track: Online multi- [37] H. Xu, Y. Gao, F. Yu, and T. Darrell, “End-to-end learning of driving object tracking by decision making,” in Inter. Conf. on Computer Vision models from large-scale video datasets,” in Int. Conf. on Computer (ICCV), 2015. Vision and Pattern Recognition (CVPR), 2017. [13] N. Wojke, A. Bewley, and D. Paulus, “Simple online and realtime [38] C. Hubschneider, A. Bauer, M. Weber, and J. M. Zollner, “Adding tracking with a deep association metric,” in Inter. Conf. on Image navigation to the equation: Turning decisions for end-to-end vehicle Processing (ICIP), 2017. control,” in Intelligent Transportation Systems Conference (ITSC) Work- [14] G. Bhat, J. Johnander, M. Danelljan, F. S. Khan, and M. Felsberg, shops, 2017. “Unveiling the power of deep tracking,” in European Conf. on Computer [39] F. Codevilla, M. Müller, A. M. López, V. Koltun, and A. Dosovitskiy, Vision (ECCV), 2018. “End-to-end driving via conditional imitation learning,” in Inter. Conf. [15] M. Dimitrievski, P. Veelaert, and W. Philips, “Behavioral pedestrian on Robotics and Automation (ICRA), 2018. tracking using a camera and LiDAR sensors on a moving vehicle,” [40] Q. Wang, L. Chen, and W. Tian, “End-to-end driving simulation via Sensors, vol. 19, no. 2, 2019. angle branched network,” arXiv:1805.07545, 2018. [16] Z. Zhu, D. Liang, S. Zhang, X. Huang, B. Li, and S. Hu, “Traffic-sign [41] X. Liang, T. Wang, L. Yang, and E. Xing, “CIRL: Controllable imitative detection and classification in the wild,” in Int. Conf. on Computer Vision reinforcement learning for vision-based self-driving,” in European Conf. and Pattern Recognition (CVPR), 2016. on Computer Vision (ECCV), 2018. [17] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for [42] C. Innocenti, H. Lindén, G. Panahandeh, L. Svensson, and N. Moham- semantic segmentation,” in Int. Conf. on Computer Vision and Pattern madiha, “Imitation learning for vision-based lane keeping assistance,” Recognition (CVPR), 2015. in Intelligent Transportation Systems Conference (ITSC), 2017. [18] H. Noh, S. Hong, and B. Han, “Learning deconvolution network for [43] Z. Chen and X. Huang, “End-to-end learning for lane keeping of self- semantic segmentation,” in Inter. Conf. on Computer Vision (ICCV), driving cars,” in Intelligent Vehicles Symposium (IV), 2017. 2015. [44] H. M. Eraqi, M. N. Moustafa, and J. Honer, “End-to-end deep learning [19] F. Yu and V. Koltun, “Multi-scale context aggregation by dilated for steering autonomous vehicles considering temporal dependencies,” convolutions,” in Inter. Conf. on Learning Representation (ICLR), 2016. in Neural Information Processing Systems (NIPS)ML for ITS WS, 2017. [20] J. Uhrig, M. Cordts, U. Franke, and T. Brox, “Pixel-level encoding and [45] L. George, T. Buhet, E. Wirbel, G. Le-Gall, and X. Perrotton, “Imitation depth layering for instance-level semantic labelling,” in German Conf. learning for end to end vehicle longitudinal control with forward on Pattern Recognition (GCPR), 2016. camera,” in Neural Information Processing Systems (NIPS) Imitation [21] L. Schneider, M. Jasch, B. Fröhlich, T. Weber, U. Franke, M. Pollefeys, Learning WS, 2018. and M. Rätsch, “Multimodal neural networks: RGB-D for semantic [46] M. Bojarski, P. Yeres, A. Choromanska, K. Choromanski, B. Firner, segmentation and object detection,” in Scandinavian Conference on L. Jackel, and U. Muller, “Explaining how a deep neural network trained Image Analysis (SCIA), 2017. with end-to-end learning steers a car,” arXiv:1704.07911, 2017. [22] Q. Ha, K. Watanabe, T. Karasawa, Y. Ushiku, and T. Harada, “MFNet: [47] J. Kim and J. Canny, “Interpretable learning for self-driving cars by Towards real-time semantic segmentation for autonomous vehicles with visualizing causal attention,” in Inter. Conf. on Computer Vision (ICCV), multi-spectral scenes,” in Int. Conf. on Intelligent Robots and Systems 2017. (IROS), 2017. [48] Z. Li, T. Motoyoshi, K. Sasaki, T. Ogata, and S. Sugano, “Rethinking [23] C. Godard, O. Aodha, and G. Brostow, “Unsupervised monocular depth self-driving: Multi-task knowledge for better generalization and accident estimation with left-right consistency,” in Int. Conf. on Computer Vision explanation ability,” arXiv:1809.11100, 2018. and Pattern Recognition (CVPR), 2017. [49] C. Sun, A. Shrivastava, S. Singh, and A. Gupta, “Revisiting unreasonable [24] A. Gurram, O. Urfalioglu, I. Halfaoui, F. Bouzaraa, and A. M. Lopez, effectiveness of data in deep learning era,” in Inter. Conf. on Computer “Monocular depth estimation by learning from heterogeneous datasets,” Vision (ICCV), 2017. in Intelligent Vehicles Symposium (IV), 2018. [50] J. Hestness, S. Narang, N. Ardalani, G. Diamos, H. Jun, H. Kianinejad, [25] Y. Gan, X. Xu, W. Sun, and L. Lin, “Monocular depth estimation with M. M. A. Patwary, Y. Yang, and Y. Zhou, “Deep learning scaling is affinity, vertical pooling, and label enhancement,” in European Conf. on predictable, empirically,” arXiv:1604.07316, 2017. Computer Vision (ECCV), 2018. [51] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous [26] H. Fu, M. Gong, C. Wang, K. Batmanghelich, and D. Tao, “Deep ordinal driving? the KITTI vision benchmark suite,” in Int. Conf. on Computer regression network for monocular depth estimation,” in Int. Conf. on Vision and Pattern Recognition (CVPR), 2012. Computer Vision and Pattern Recognition (CVPR), 2018. [52] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benen- [27] G. Bresson, Z. Alsayed, L. Yu, and S. Glaser, “Simultaneous localization son, U. Franke, S. Roth, and B. Schiele, “The Cityscapes dataset for and mapping: A survey of current trends in autonomous driving,” IEEE semantic urban scene understanding,” in Int. Conf. on Computer Vision Trans. on Intelligent Vehicles, vol. 2, no. 3, pp. 194–220, 2017. and Pattern Recognition (CVPR), 2016. 10 [53] E. Santana and G. Hotz, “Learning a driving simulator,” [78] F. Codevilla, A. M. López, V. Koltun, and A. Dosovitskiy, “On offline arXiv:1608.01230, 2016. evaluation of vision-based driving models,” in European Conf. on [54] W. Maddern, G. Pascoe, C. Linegar, and P. Newman, “1 Year, 1000 Computer Vision (ECCV), 2018. Km: The Oxford RobotCar dataset,” Inter. Journal of Robotics Research [79] A. Bewley, J. Rigley, Y. Liu, J. Hawke, R. Shen, V.-D. Lam, and (IJRR), vol. 36, no. 1, pp. 3–15, 2017. A. Kendall, “Learning to drive from simulation without real world [55] C. Chen, A. Seff, A. L. Kornhauser, and J. Xiao, “DeepDriving: Learning labels,” arXiv:1812.03823v2, 2018. affordance for direct perception in autonomous driving,” in Inter. Conf. [80] C. Premebida, J. Carreira, J. Batista, and U. Nunes, “Pedestrian detection on Computer Vision (ICCV), 2015. combining rgb and dense LiDAR,” in Int. Conf. on Intelligent Robots [56] A. Sauer, N. Savinov, and A. Geiger, “Conditional affordance learning and Systems (IROS), 2014. for driving in urban environments,” in Conf. on Robot Learning (CoRL), [81] Y. Luo, J. Ren, M. Lin, J. Pang, W. Sun, H. Li, and L. Lin, “Single 2018. view stereo matching,” in Int. Conf. on Computer Vision and Pattern [57] M. Müller, A. Dosovitskiy, B. Ghanem, and V. Koltun, “Driving policy Recognition (CVPR), 2018. transfer via modularity and abstraction,” in Conf. on Robot Learning [82] D. Xu, W. Wang, H. Tang, H. Liu, N. Sebe, and E. Ricci, “Structured (CoRL), 2018. attention guided convolutional neural fields for monocular depth estima- [58] Z. Yang, Y. Zhang, J. Yu, J. Cai, and J. Luo, “End-to-end multi-modal tion,” in Int. Conf. on Computer Vision and Pattern Recognition (CVPR), multi-task vehicle control for self-driving cars with visual perceptions,” 2018. in Inter. Conf. on Pattern Recognition (ICPR), 2018. [59] A. González, Z. Fang, Y. Socarras, J. Serrat, D. Vázquez, J. Xu, and A. M. López, “Pedestrian detection at day/night time with visible and fir cameras: A comparison,” Sensors, vol. 16, no. 6, 2016. [60] A. González, D. Vázquez, A. M. López, and J. Amores, “On-board object detection: Multicue, multimodal, and multiview random forest of local experts,” IEEE Trans. on Cybernetics, vol. 47, no. 11, pp. 3980– 3990, 2017. [61] X. Chen, H. Ma, J. Wan, B. Li, and T. Xia, “Multi-view 3D object detection network for autonomous driving,” in Int. Conf. on Computer Vision and Pattern Recognition (CVPR), 2017. [62] J. Ku, M. Mozifian, J. Lee, A. Harakeh, and S. Waslander, “Joint 3D proposal generation and object detection from view aggregation,” in Int. Conf. on Intelligent Robots and Systems (IROS), 2018. [63] Y. Wu, H. Qin, T. Liu, H. Liu, and Z. Wei, “A 3D object detection based on multi-modality sensors of USV,” Applied Sciences, vol. 9, no. 3, 2019. [64] N. Rhinehart, R. McAllister, and S. Levine, “Deep imitative models for flexible inference, planning, and control,” arXiv:1810.06544, 2018. [65] A. Dosovitskiy, G. Ros, F. Codevilla, A. López, and V. Koltun, “CARLA: An open urban driving simulator,” in Conf. on Robot Learning (CoRL), 2017. [66] M. Enzweiler and D. Gavrila, “A multilevel mixture-of-experts frame- work for pedestrian classification,” IEEE Trans. on Image Processing, vol. 20, no. 10, pp. 2967–2979, 2011. [67] E. Arnold, O. Y. Al-Jarrah, M. Dianati, S. Fallah, D. Oxtoby, and A. Mouzakitis, “A survey on 3d object detection methods for au- tonomous driving applications,” IEEE Trans. on Intelligent Transporta- tion Systems, January 2019. [68] D. Feng, C. Haase-Schutz, L. Rosenbaum, H. Hertlein, F. Duffhauss, C. Glaser, W. Wiesbeck, and K. Dietmayer, “Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges,” arXiv:1902.07830, 2019. [69] F. Piewak, P. Pinggera, M. Enzweiler, D. Pfeiffer, and M. Zöllner, “Improved semantic stixels via multimodal sensor fusion,” arXiv:1809.08993, 2018. [70] B. Argall, S. Chernova, M. Veloso, and B. Browning, “A survey of robot learning from demonstration,” Robotics and Autonomous Systems, vol. 57, no. 5, pp. 469–483, 2009. [71] B. Wymann, E. Espié, C. Guionneau, C. Dimitrakakis, R. Coulom, and A. Sumner, “TORCS, The Open Racing Car Simulator,” http://www. torcs.org. [72] I. Sobh, L. Amin, S. Abdelkarim, K. Elmadawy, M. Saeed, O. A. Valeo, M. Gamal, and A. El-Sallab, “End-to-end multi-modal sensors fusion system for urban automated driving,” in Neural Information Processing Systems (NIPS) MLITS WS, 2018. [73] Q. Khan, T. Schön, and P. Wenzel, “Towards self-supervised high level sensor fusion,” arXiv:1902.04272, 2019. [74] C. Doersch and A. Zisserman, “Multi-task selfsupervised visual learn- ing,” in Inter. Conf. on Computer Vision (ICCV), 2017. [75] A. Kendall, Y. Gal, and R. Cipolla, “Multi-task learning using uncer- tainty to weigh losses for scene geometry and semantics,” in Int. Conf. on Computer Vision and Pattern Recognition (CVPR), 2018. [76] S. Halev-Shwartz and A. Shashua, “On the sample complexity of end-to- end training vs. semantic abstraction training,” arXiv:1604.06915, 2016. [77] N. Kalra and S. M. Paddock, “Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliabil- ity?” Transportation Research Part A: Policy and Practice, vol. 94, p. 182–193, December 2016.