International Journal of Computer Vision (2019) 127:1643–1658 https://doi.org/10.1007/s11263-019-01226-9 Slanted Stixels: A Way to Represent Steep Streets Daniel Hernandez-Juarez1 · Lukas Schneider2 · Pau Cebrian1 · Antonio Espinosa1 · David Vazquez3 · Antonio M. López1,4 · Uwe Franke2 · Marc Pollefeys5 · Juan C. Moure1 Received: 20 February 2018 / Accepted: 30 August 2019 / Published online: 14 September 2019 © The Author(s) 2019 Abstract This work presents and evaluates a novel compact scene representation based on Stixels that infers geometric and semantic information. Our approach overcomes the previous rather restrictive geometric assumptions for Stixels by introducing a novel depth model to account for non-flat roads and slanted objects. Both semantic and depth cues are used jointly to infer the scene representation in a sound global energy minimization formulation. Furthermore, a novel approximation scheme is introduced in order to significantly reduce the computational complexity of the Stixel algorithm, and then achieve real-time computation capabilities. The idea is to first perform an over-segmentation of the image, discarding the unlikely Stixel cuts, and apply the algorithm only on the remaining Stixel cuts. This work presents a novel over-segmentation strategy based on a fully convolutional network, which outperforms an approach based on using local extrema of the disparity map. We evaluate the proposed methods in terms of semantic and geometric accuracy as well as run-time on four publicly available benchmark datasets. Our approach maintains accuracy on flat road scene datasets while improving substantially on a novel non-flat road dataset. Keywords Stereo vision · Stixel world · Self-driving cars · Scene understanding · Automotive vision · Intelligent vehicles 1 Introduction This success has led to increased interest in the model from the intelligent vehicles community over the past years Autonomous vehicles, advanced driver assistance systems, The Stixel world has been successfully used for representing robots and other intelligent devices need to understand their traffic scenes, as introduced in Pfeiffer and Franke (2011). It environment. For this purpose, both geometric (distance) and has shown its potential particularly in the Bertha-Benz drive semantic (classification) sources of information are useful. (Ziegler et al. 2014b), where it has been successfully applied We want to represent these inputs in a very compact model for visual scene understanding in autonomous driving. This and compute them in real-time to serve as a building block success has led to increased interest in the model from the of higher-level modules, such as localization and planning. intelligent vehicles community over the past years (Schnei- der et al. 2016; Hernandez-Juarez et al. 2017a; Benenson et al. 2011; Cordts et al. 2014, 2017; Ignat 2016; Levi et al. Communicated by Tae-Kyun Kim, Stefanos Zafeiriou, Ben Glocker, 2015; Carrillo and Sutherland 2016; Hernandez-Juarez et al. Stefan Leutenegger. 2017b). Authors Daniel Hernandez-Juarez and Lukas Schneider have con- The Stixel world defines a compact medium-level repre- tributed equally to this work. sentation of dense 3D disparity data obtained from stereo vision using rectangles, the so called Stixels, as elements. B Daniel Hernandez-Juarez Stixels are classified either as ground-like planes, upright dhernandez0@gmail.com objects or sky, which are important geometric elements found 1 Universitat Autònoma de Barcelona (UAB), Barcelona, Spain in man-made environments. This representation transforms 2 Daimler AG R&D, Böblingen, Germany millions of disparity pixels to hundreds or thousands of Stix- 3 els. At the same time, most scene structures, such as free Element AI, Montreal, Canada space and obstacles, which are relevant for autonomous driv- 4 Computer Vision Center (CVC), Barcelona, Spain ing tasks, are adequately represented. 5 ETH Zürich, Zürich, Switzerland 123 1644 International Journal of Computer Vision (2019) 127:1643–1658 Fig. 1 The proposed approach: pixel-wise color, semantic and depth arbitrary scenarios such as San Francisco city. The optional over- information serve as input to our Slanted Stixels model, which is a segmentation in the top-right yields significant speed gains while nearly compact semantic representation of a 3D scene that accurately handles retaining the depth and semantic accuracy (Color figure online) The idea behind the Stixel model is that planar surfaces ing an over-segmentation strategy that can be applied to any are dominant in man-made environments and they can be Stixel model proposed so far. An earlier version of our work modeled using this assumption. Scene structure found in (Hernandez-Juarez et al. 2017b) proposed a simple over- urban environments can be modeled with certain constraints, segmentation strategy that provided faster execution at the e.g. the sky is above the horizon line and objects usually expense of decreasing the accuracy of the model. This paper lie on the ground. Generally, the geometric constraints of introduces a novel over-segmentation approach based on a a scene are tied to the vertical direction. Hence, the envi- Fully Convolutional Network (FCN) that outperforms the ronment can be modeled as a column-wise segmentation previous strategy, and achieves similar speedup results but of the image with a 3D stick-like shape, i.e. a set of Stix- retaining most of the accuracy of the original version. An els, c.f. Fig. 1. The segmentation of the image is estimated overview of our method is shown in Fig. 1. by solving a column-wise energy minimization problem, Our main contributions are: (1) a novel depth model to taking depth and semantic cues as inputs as well as a pri- accurately represent arbitrary kinds of slanted surfaces into ori information that is used to regularize the solution c.f. the Stixel representation; (2) a novel over-segmentation prior Fig. 1. designed to reduce the run-time of the method; (3) an effec- The Stixel model has been successfully used for auto- tive over-segmentation strategy based on a shallow Fully motive vision applications either to decrease parsing time, Convolutional Network; (4) a new synthetic dataset with increase accuracy or both. We can find examples of works non-flat roads that includes pixel-level semantic and depth using the Stixel representation in different topics such as ground-truth, which is publicly available1 ; and (5) an in- object recognition (Benenson et al. 2012; Li et al. 2016), depth evaluation in terms of run-time as well as semantic and building a grid map over time (Muffert et al. 2014) and depth accuracy carried out on this novel dataset and several for autonomous driving (Ziegler et al. 2014b). Specifically, real-world benchmarks. Compared to the existing state-of- for motion planning in the context of autonomous driv- the-art approaches, our method substantially improves the ing, the Stixel model has been used c.f. (Ziegler et al. depth accuracy in non-flat road scenarios. 2014b, a) to model the geometric constraints of a given The remainder of this paper is structured as follows. Sec- scene. tion 2 reviews the state of the art. Section 3 presents the We propose a new depth model that is able to accurately new Stixel formulation. We present two over-segmentation represent arbitrary kinds of slanted objects and non-flat roads. methods in Sect. 4. Section 5 explains the experiments we The improved Stixel representation outperforms the original carried on and discusses their results. Finally, we state our Stixel model in scenarios with non-flat roads, while keep- conclusions in Sect. 6. ing the same accuracy on flat road scenes. The induced extra computational complexity is reduced by incorporat- 1 http://synthia-dataset.net. 123 International Journal of Computer Vision (2019) 127:1643–1658 1645 2 Related Work this drawback by incorporating a novel plane model together with effective priors on the plane parameters. Our proposed method introduces a novel Stixel-based scene Our proposal of using Stixels cuts is related to Cordts representation that is able to account for non-flat roads, c.f. et al. (2014): they use fast object detectors for different Fig. 2. We also devise an approximation to reduce the compu- object classes, e.g. Viola-Jones cascade detector (Viola and tational complexity of the underlying Dynamic Programming Jones 2001), to produce top and bottom Stixel cuts that are algorithm. used as prior information, which is then integrated into the First, we will comment on works proposing different road Stixel algorithm. They prove that using object-level knowl- scene models. Occupancy grid maps are models used to rep- edge provides significant accuracy improvements. Instead, resent the surroundings of the vehicle (Dhiman et al. 2014; we leverage semantic information as pixel-level knowledge Muffert et al. 2014; Nuss et al. 2015; Thrun 2002). Typi- in our model for the same purpose of improving accu- cally, a grid in bird’s eye perspective is defined and used to racy. Semantic segmentation identifies the objects and other detect occupied grid cells and then, from this information, to elements of the image, e.g. walls or sidewalks, providing extract the obstacles, drivable area, and unobservable areas pixel-level information, instead of boxes around the objects. from range data. These grids and the Stixel world both repre- Also, semantic segmentation requires a single predictor, sent the 2D image in terms of column-wise stripes allowing while the method proposed by Cordts et al. (2014) needs to capture the camera data in a polar fashion. Also, the Stixel a detector trained for each object class. In contrast, we define data model is similar to the forward step usually found in a Stixel cut prior to generate an over-segmentation of the occupancy grid maps (Cordts et al. 2017). However, the Stixel optimal Stixel cuts in order to speed up the execution of the inference method in the image domain presents important algorithm. differences compared to classical grid-based approaches. There are some methods (Benenson et al. 2011; Ignat Our work builds upon the proposal from Schneider et al. 2016; Levi et al. 2015), that represent simplified scene mod- (2016): they use semantic cues in addition to depth to extract els with a single Stixel per column. The advantage of these a Stixel representation, which is able to provide a rich yet approaches is that the computational complexity of the under- compact representation of the traffic scene. However, their lying algorithms is linear, but they cannot represent some model assumes a constant road slant and is therefore limited complex scenarios found in the real world, e.g. a pedestrian to flat road scenarios. In contrast, our proposal overcomes and a building in the same column. Recent work by Carrillo and Sutherland (2016) uses edge- based disparity maps to compute Stixels. Their method is fast but they show that it gives inferior accuracy compared to the original Stixel model (Pfeiffer et al. 2013). Levi et al. (2015) firstly introduced the use of an FCN in Stixel-based methods. A single RGB image feeds the FCN to estimate the bottom of the first non-road Stixel, i.e. closest obstacle. We use an FCN for a entirely different objective: to extract a Stixel cut over-segmentation that accelerates the execution of the algorithm. Moreover, the input of our FCN (a) is a disparity map obtained from a stereo camera. Finally, there are some works proposing fast implemen- tations for Stixel computation. The FPGA implementation from Muffert et al. (2014) runs at 25 Hz with a Stixel width of 5 pixels, but the authors do not indicate the image resolution. Hernandez-Juarez et al. (2017a) present a GPU-accelerated implementation that runs at 26 Hz for a Stixel width of 5 pixels and image resolution of 1024 × 440 pixels, com- puted using a Semi-Global Matching (SGM) (Hirschmüller 2008) stereo algorithm. We propose a novel approximation that accelerates the computation by reducing the algorithmic (b) complexity. Accordingly, our proposal could benefit from the aforementioned FPGA- or GPU-accelerated implemen- Fig. 2 Scene representation obtained by our method of a challeng- tations. ing street environment with a slanted road. Both geometric (top) and semantic (bottom) representations are shown (Color figure online) 123 1646 International Journal of Computer Vision (2019) 127:1643–1658 3 Stixel Model The joint Stixel segmentation and labeling problem is carried out via optimization of the column-wise posterior The Stixel world is a compressed representation of a 3D distribution P(S: | M : ) defined over a Stixel segmentation scene that preserves its relevant structure. Since the structure S: given all measurements M : from that particular image in street environments is dominant in the vertical domain, column. In the following, we drop the column indexes for the Stixel world leverages this idea to model a scene with- ease of notation. We obtain Stixel width > 1 as illustrated out taking into account the horizontal neighborhood. This e.g. in Fig. 1 by down-sampling of the inputs, this width is assumption leads to an efficient inference method and also fixed and is chosen to reduce the computational complex- allows the inference to be performed on all columns in par- ity during inference, however heavy down-sampling leads to allel. degradation in accuracy (Cordts et al. 2017). The Stixel world is defined as a segmentation of image A Stixel column segmentation consists of an arbitrary columns into stick-like super-pixels with class labels and a number N of Stixels Si , each representing four random vari- 3D planar depth model c.f. Fig. 3. We consider three struc- ables: the Stixel extent via bottom Vib and top Vit row, as well tural classes: object, ground and sky. These classes have as its class Ci and geometric depth model G i . Thereby, the properties that are derived from an underlying 3D model: number of Stixels itself is a random variable that is optimized for object Stixels the distance is roughly constant and usu- jointly during inference. To this end, the posterior probability ally lie on the ground, for sky Stixels the distance is infinite is defined by means of the unnormalized prior and likelihood and for ground Stixels we favor planes with accordance to distributions the expected ground. The Stixel world has several properties that are useful 1 P(S | M ) = P̃(M | S) P̃(S) (1) for higher-level processing stages: (1) it is a medium-level Z scene representation that significantly reduces the quantity of where Z is the normalizing partition function. Transformed elements, e.g. from millions of pixels to hundreds of Stixels, to log-likelihoods via while keeping an abstract representation of physical extent, depth and semantics; (2) the representation is based upon a P(S = s | M = m) = − log(e−E (s,m) ) (2) street model; (3) the representation is not high-level because an object is represented by more than one Stixel horizontally and it can be split in more than one Stixel vertically too, where E(·) is the energy function, E data (·) is the likelihood e.g. occlusions and slanted objects such as cars viewed from term and E prior (·) is the prior term. behind. E(s, m) = E data (s, m) + E prior (s) (3) 3.1 Data Term The likelihood term E data (·) thereby rates how well the mea- surements mv at pixel v fit to the overlapping Stixel si  N E data (s, m) = E sti xel (si , m) i=1 vi t (4)  N  = E pi xel (si , mv ) . i=1 v=v b i This pixel-wise energy is further split in a semantic and a depth term E pi xel (si , mv ) = E disp (si , dv ) + wl · E sem (si , lv ) . (5) The parameter wl controls the influence of the semantic data Fig. 3 Example of input disparity measurements (black lines) and out- term. The input is provided by an FCN that delivers normal- put Stixels encoded with semantic colors (colored lines) for a typical  scene column (right). Adapted from Cordts et al. (2017) (Color figure ized semantic scores lv (ci ) with ci lv (ci ) = 1 for all classes online) ci at pixels v. The semantic energy favors semantic classes of 123 International Journal of Computer Vision (2019) 127:1643–1658 1647 the Stixel that fit to the observed pixel-level semantic input and sky Stixels), assumptions per Stixel, e.g. for object Stix- (Schneider et al. 2016). The semantic likelihood term is els: μ(si , v) = constant. This paper introduces a new plane depth model that relaxes E sem (si , lv ) = −log(lv (ci )) . (6) the previous assumptions in favor of a more accurate depth representation. The new model is formulated such that it The depth model is designed to represent the different nicely interacts with this well founded and experimentally characteristics of the different geometric classes, i.e. object, validated depth sensor model. To this end, we formulate the ground and sky Stixels. Furthermore, the model enforces depth model μ(si , v) using two random variables defining a multiple stacked Stixels in cases of objects with the same plane in the disparity space that evaluates to the disparity in class but different depths. row v via Our depth input is a dense disparity map, each pixel is assigned a disparity value or is masked as invalid i.e. μ(si , v) = bi · v + ai . (10) dv ∈ {0...dmax , dinvalid }. The depth term is defined by means of a probabilistic and generative sensor model Pv (·) that con- Note that we assume narrow Stixels and thus can neglect siders the accordance of the depth measurement dv at row v one plane parameter, i.e. the roll. to the Stixel si This model is a generalization of the previous class- specific depth models used in previous works, allowing for a E disp (si , dv ) = − log(Pv (Dv = dv | Si = si )) . (7) more flexible representation of the scene because of the extra free parameter c.f. Fig. 4. The way of modeling the different Invalid dinv disparity measurements have to be handled, Stixel classes i.e. object, ground and sky is through priors, as therefore, a prior probability of a valid disparity value is explained in Sect. 3.2.5. Also, to completely understand the defined as pval details about the inference, we suggest to read Sect. 3.3.  pval Pv,val (Dv | Si ) if dv = dinv 3.2 Prior Term Pv (Dv | Si ) = (8) (1 − pval ) otherwise The prior captures knowledge about the segmentation inde- where Pv,val (Dv | Si ) is the measurement model of valid pendent from measurements, in this section we define the disparities only. It is comprised of a constant outlier prob- priors used for this model, they are based on Cordts et al. ability pout and a Gaussian sensor noise model for valid (2017). The Markov property is used so that the prior reduces measurements with confidence cv to pair-wise relations between subsequent Stixels. Accord- ingly, the prior is computed as   cv (dv −μ( si ,v )) 2 pout 1 − pout − σ ( si ) Pv,val (Dv | Si ) = + e (9)  N ZU Z G (si ) E prior (s) = E pair (si , si−1 ) + E f ir st (s1 ) . (11) i=2 that is centered at the expected disparity μ(si , v) given by the depth model of the Stixel, where Z U and Z G (si ) nor- In the next sections, where different priors are introduced, malize the distributions. Similarly to Pfeiffer et al. (2013), E pair (si , si−1 ) is the summation of all these priors. However, we use the confidence of the depth estimates cv to influence E f ir st (s1 ) does not include pairwise terms, i.e. the shape of the distribution. The Gaussian models the typ- ical disparity noise and the uniform distribution makes the E f ir st (s1 ) = E mc (s1 ) + E seg f ir st (s1 ) + E seglast (s1 ) model more robust to outliers, which is weighted by pout . (12) + E top≥bottom (si ) + E plane (s1 ) The standard deviation σ (si ) models the noise of the stereo matching algorithm and depends on the class ci . 3.2.1 Model Complexity Prior 3.1.1 New Depth Model A model complexity term favors solutions composed of fewer Stixels and thus invokes costs for each Stixel in the column The depth model defines the 3D outline of a Stixel using segmentation S: very few parameters per Stixel and reflects our assump- tions on the surrounding scene. Both, data term (c.f. eq. (9)) E mc (si ) = Cmc . (13) and priors (c.f. Sect. 3.2) have a significant impact on the inferred depth model. In previous formulations, the three There is a trade-off between compactness and accuracy. A different geometric classes were designed using restrictive high Cmc parameter would lead to a very compact segmen- constant height (ground Stixels) and constant depth (object tation i.e. few Stixels. However, a representation with few 123 1648 International Journal of Computer Vision (2019) 127:1643–1658 Fig. 4 Comparison of original (Schneider et al. 2016) (top) and our Slanted Stixels (bottom): due to the fixed slant in the original formulation, the road surface is not well represented as illustrated on the top-left figure. The novel model is capable of reconstructing the whole scene accurately ⎧ Stixels is more likely to have lower accuracy, e.g. a solution ⎪ ⎪ ⎪α− − + βgravit y Δg if Δg < 0 comprised of one Stixel the size of the whole column would ⎨ gravit y + + E gravit y (si , si−1 ) = αgravit y + βgravit y Δg if Δg > 0 result in a huge disparity and semantic error. ⎪ ⎪ ⎪ ⎩0 otherwise 3.2.2 Segmentation Priors (18) The model has to enforce that all pixels are assigned to exactly where Δg = μs (vib , gi ) − μs (vi−1t ,g i−1 ) is the difference one Stixel, i.e. non-overlapping Stixels, Stixels extend over between the object Stixel disparity μs (vib , gi ) at it’s bottom all the column and Stixels are connected. Therefore, the first pixel vib and the disparity of the ground Stixel μs (vi−1 t ,g i−1 ) priors are defined to comply with the following rules: The at the top row vi . It only applies for si being an object and t first Stixel must begin in row 1 and the last Stixel must end si−1 being a ground Stixel. in row h, i.e. The depth ordering prior penalizes a combination of two staggered object Stixels when the upper of the two is closer  (in distance to the car) than the lower one. ∞ if vib = 1, i = 1 E seg f ir st (si ) = (14) ⎧ 0 otherwise ⎨αor d +βor d (gi −gi−1 ) if gi > gi−1  E or d (si , si−1 ) = . ∞ if = h, i = n vit ⎩0 otherwise E seglast (si ) = . (15) 0 otherwise (19) A novel prior is introduced in this paper: the ground gap Furthermore, every Stixel must be connected to the next prior penalizes two consecutive ground Stixels when the bot- one and the Stixel top row must be greater than the bottom tom disparity of the upper Stixel i.e. disparity at row vib and row, i.e. the disparity of the lower Stixel at row vib do not match.  ⎧ ⎪ − − if vib = vi−1 +1 ⎪αgap + βgap Δgap if Δgap < 0 0 t E connection (si , si−1 ) = (16) ⎪ ⎨ ∞ otherwise + + β+ Δ  E gap (si , si−1 ) = αgap gap gap if Δgap > 0 (20) ⎪ ⎪ 0 if vib ≤ vit ⎪ ⎩ E top≥bottom (si ) = . (17) 0 otherwise ∞ otherwise where Δgap = gs (vib , gi ) − gs (vib , gi−1 ). These structural 3.2.3 Structural Priors priors do not enforce their assumptions. Instead, they penal- ize unusual combinations, e.g. a flying object (gravity prior), The gravity prior penalizes a flying object i.e. an object Stixel traffic signs (ordering prior). not lying on top of the previous ground Stixel, 123 International Journal of Computer Vision (2019) 127:1643–1658 1649 3.2.4 Transition Priors argmin E(s, m) = argmin E sti xel (si , m) + E plane (si ) . ai ,bi ai ,bi These priors define the knowledge regarding the transition (23) between a pair of Stixels. Thus, we minimize the global energy function with respect to the plane parameters of all Stixels and all geometric classes E transition (si , si−1 ) = γci ,ci−1 (21) independently. We can find an optimal solution of the result- ing weighted least squares problem in closed form. However, where γci ,ci−1 is the transition cost between previous Stixel we still need to compare the Stixel measurements to our new class ci−1 to current Stixel class ci . This is defined via a two- plane depth model. Therefore, the complexity added to the dimensional transition matrix for all combinations of classes original formulation is another quadratic term in the image γci ,ci−1 . Only first order relations are modeled in order to height. infer efficiently. 3.4 Stixel Cut Prior 3.2.5 Plane Prior The Stixel inference process described so far requires the estimation of the cost for each possible Stixel in an image. In this paper, we propose a new additional prior term that However, many Stixels can be trivially discarded, e.g. in uses the specific properties of the three geometric classes. image regions with homogeneous depth and semantic input, We expect the two random variables A, B representing the making it possible to avoid the computation steps associated plane parameters of a Stixel to be Gaussian distributed, i.e. to the calculation of these. 2 2 We propose a novel prior that exploits hypothesis genera- ai − μaci bi − μbci tion to significantly reduce the computational burden of the E plane (si ) = + −log(Z ) . (22) σcai σcbi inference task. To this end, we formulate a new prior simi- lar to Cordts et al. (2014); however, instead of Stixel bottom This prior favors planes in accordance to the expected 3D and top probabilities, we incorporate generic likelihoods for layout corresponding to the geometric class. For instance, pixels being the cut between two Stixels. object Stixels are expected to have an approximately constant We leverage this additional information adding a novel disparity, i.e. μbobject = 0. The expected road slant μbground prior term for a Stixel si can be set using prior knowledge or a preceding road surface  E cut (si ) = − log cvi (cut) (24) detection. For sky Stixels we expect infinite distance i.e. 0 disparity, therefore, we set μasky = μbsky = 0. where cvi (cut) is the confidence for a cut at vi , thus The standard deviations σcai and σcbi are used in order to cvi (cut) = 0 implies that there is no cut between two Stixels enforce the assumptions for each Stixel class, i.e. the more at row v. confident we are that object Stixels have constant distance, As described in Pfeiffer (2014), we can design a recursive the closer to 0 we would set σobject b . The same applies for definition of the optimization problem in order to solve the ground Stixels: if we know the road is not slanted, we can rely problem using a Dynamic Programming scheme. In order to on the expected previous road model and set σgr b ound → 0. simplify our description, we use a special notation to refer to For sky Stixels, it does not make sense to have a disparity Stixels: obbt = {v b , v t , object}. Similarly, O B k is defined as a → 0 and σ b → 0. different to 0. Therefore, we set σsky sky the minimum aggregated cost of the best segmentation from Note that the novel formulation is a strict generalization position 0 to k. The Stixel at the end of the segmentation of the original method, since they are equivalent, e.g. if the associated with each minimum cost is denoted as obk . We slant is fixed, i.e. σobject b → 0, μbobject = 0. next show a recursive definition of the problem: ⎧ ⎪ ⎪ E data (ob0k ) + E prior (ob0k ) 3.3 Inference ⎪ ⎪ ⎪ ⎪ E data (obkx ) + E prior (obkx , ob x−1 ) ⎪ ⎪ ⎪ ⎪ The sophisticated energy function defined in Sect. 3 is opti- ⎪ ⎨ + O B x−1 ∀x ∈ cuts, x ≤ k mized via Dynamic Programming as in Pfeiffer and Franke O B k = min E data (obkx ) + E prior (obkx , gr x−1 ) . ⎪ ⎪ (2011). However, we must also optimize jointly for the novel ⎪ ⎪ + G R ∀x ∈ cuts, x ≤ k x−1 ⎪ ⎪ depth model. When optimizing for the plane parameters ⎪ ⎪ ⎪ ⎪ E (ob k) + E prior (obx , sk k x−1 ) ai , bi of a certain Stixel si , it becomes apparent that all other ⎪ ⎩ data x optimization parameters are independent of the actual choice + SK x−1 ∀x ∈ cuts, x ≤ k of the plane parameters. We can thus simplify (25) 123 1650 International Journal of Computer Vision (2019) 127:1643–1658 4 Generation of the Stixel Cut Prior G The previous section explained how to use a Stixel cut prior O to reduce the computational complexity of the Stixel infer- ence. The idea is that many Stixel cuts could be trivially discarded, e.g. in image regions with homogeneous depth S and semantic input. We can save a lot of computation by not vit = 1 2 3 processing those unlikely Stixel cuts. The goal is to devise a cvi (cut) = 1 1 1 fast method to generate an over-segmentation of the optimal Stixel cuts. And, if those optimal cuts are included in the generated hypothesis, then the Stixel algorithm will provide (a) Stixel graph representation the same output as in the original case, but doing much fewer computation steps. We propose two methods to generate Stixel cuts. The first G method is a simple strategy that uses some mathematical con- cepts to identify points of interest c.f. Sect. 4.1. It is a very fast approach, but misses some of the optimal Stixel cuts O and, therefore, the final accuracy of the Stixel inference is reduced. The second method uses a shallow Fully Convolu- S tional Network (FCN) that is trained on the disparity map to vit = 1 2 3 infer likely Stixel cuts c.f. Sect. 4.2. This strategy is also very cvi (cut) = 1 0 1 fast, since the FCN is small, and is able to provide almost all of the optimal Stixel cuts. For both methods, we leverage semantic segmentation information by including the edges (b) Pruned graph using Stixel cut prior of the semantic image into the set of the generated Stixel Fig. 5 Stixel inference illustrated as shortest path problem on a directed cuts. acyclic graph: the Stixel segmentation is computed by finding the short- est path from the source (left gray node) to the sink (right gray node). The vertices represent Stixels with colors encoding their geometric class, i.e. 4.1 Time Series Compression ground, object and sky. Only the incoming edges of ground nodes are shown for simplicity. Adapted from Cordts et al. (2017) (Color figure online) The first method to generate Stixel cuts is based on the work of Ignat (2016), and has linear time complexity and linear memory requirements. In their work, each column of the disparity map is treated independently as a time series, i.e. We only show the case for object Stixels, but the other a signal with measurements on equal intervals of time. They cases are solved similarly. Also, G R k and S K k stand for first perform an extreme points detection step that generates ground and sky respectively. The base case problem, i.e. a list of possible Stixel cuts, and then apply subsequent filters segmenting a column of the single pixel at the bottom, is to this list in order to generate the final Stixel segmentation. defined: O B 0 = E data (ob00 ) + E prior (ob00 ). Our method As we want to obtain an over-segmentation containing all trusts that all the optimal cuts will be included in our the optimal Stixel cuts, we only use the first step of their over-segmentation [cuts in Eq. (25)], therefore, only those proposal. positions are checked as Stixel bottom and top. This reduces The detection of extreme points is based on techniques the complexity of the Stixel estimation problem for a single for time series compression (Fink and Gandhi 2011). A time column to O(h × h ), where h is the number of over- series can be compressed by selecting local extreme points, segmentation cuts computed for this column, h is image i.e. maxima and minima of a function within a range. The height and h h. assumption is that local extreme points are enough to find The computational complexity reduction becomes appar- the important parts of the signal, and the rest would be unim- ent in Fig. 5. As stated in Cordts et al. (2017), the inference portant points or noise. problem can be interpreted as finding the shortest path in a In Ignat (2016) only left and right extrema are selected, directed acyclic graph. Our approach prunes all the vertices while other kinds of extrema are discarded. Given a time associated with the Stixel’s top row not included according series {t1 , t2 , . . . , ti , . . . , tn−1 , tn } and point ti with 1 < i < to the Stixel cut prior, c.f. Fig. 5b. n, the definition of left and right minimum is as follows (the definition of maxima is symmetric): 123 International Journal of Computer Vision (2019) 127:1643–1658 1651 Fig. 6 Generated Stixel cuts (highlighted in red) using the left and right extrema as defined in Ignat (2016), and also cuts generated from semantic segmentation. Stixel cut density is 30%, equivalent to a 3.3× reduction in vertical resolution (Color figure online) Fig. 7 Generated Stixel cuts (highlighted in red) for the FCN-based method. Stixel cut density is 31.5%, equivalent to a 3.2× reduction in vertical resolution (Color figure online) – ti is left minimum if ti < ti−1 and there is t j such that j > i and ti = . . . = t j < t j+1 . – ti is right minimum if ti < ti+1 and there is t j such that j < i and t j−1 > t j = . . . = ti . Similarly, we generate Stixel cuts by finding left and right extrema and the first and last points of the sequence of pixels in the column. The example in Fig. 6 illustrates the method. The predicted Stixel cuts are indicated in red color. In the example the vertical resolution is reduced around 3.3 times, which implies reduced computational work for the Stixel inference task. 4.2 FCN-Based Method We propose a novel shallow deep neural network c.f. Fig. 8 that generates a set of promising Stixel cuts from depth Fig. 8 Definition of the proposed Fully Convolutional Network for generating Stixel cuts images c.f. Fig. 7. We follow the proposal in Jasch et al. (2018): we use disparities instead of depth. We have experi- mentally found that adding the RGB image to the input of the neural network does not improve the accuracy of the method, than the total number of potential Stixel cuts (which is the compared to the simpler and faster strategy of directly adding height of the image). Also, the computational work required the edges of the semantic image into the set of the generated for the network inference must be small, ideally similar to the Stixel cuts. Time Series method proposed in Sect. 4.1. In the remainder We design the network to provide an over-segmentation of this section, we will first discuss the proposed network of the optimal Stixel cuts that should be significantly smaller architecture, and then describe the data and training strategy. 123 1652 International Journal of Computer Vision (2019) 127:1643–1658 4.2.1 Network Architecture As discussed previously c.f. Sect. 3.2.1, the definition of the parameters of the Stixel model represent a trade- Our proposal is based on the architecture described by off between compactness and accuracy. Since we need an Schneider et al. (2017). They present a multi-modal FCN over-segmentation of the optimal Stixel cuts, we adjust the designed for semantic segmentation with a mid-level fusion parameters of the model to be conservative and to favor accu- architecture that exploits complementary input cues, i.e. racy versus compactness. RGB and disparity data. Their design includes the Network The idea of using the Stixel model as a way to train a in Network (NiN) method proposed by Lin et al. (2013). fast and simple neural network to approximate the optimal Our proposal inherits the network branch that processes the Stixel segmentation is inspired by model distillation tech- disparity data and discards the branch on the RGB data, niques (Bucila et al. 2006). The comparatively slow Dynamic which is described in detail in Fig. 8. The proposed FCN Programming method to solve the probabilistic model is used is a very shallow network with three consecutive NiNs, and to transfer the knowledge inside the complex model to a fast a final deconvolution that recovers the desired resolution of and compact FCN that approximates the optimal Stixel cuts. the Stixel cuts. The output of the FCN is a binary image indicating whether or not there is a Stixel cut for that pixel. 4.2.3 Training Strategies Since our problem is to classify each pixel of our input dis- 4.2.2 Training Data parity map as cut or not-cut, we use cross-entropy as the loss function that must be minimized. The distribution of We trained the proposed FCN using disparity maps gener- cut/not-cut is strongly biased in our input and, accordingly, ated from images in the Synthia synthetic dataset (Ros et al. we introduce a class-balancing weight in the loss function, 2016) and from images in a real-data sequence (6757 images) similarly to Xie and Tu (2017). These weights cause the FCN recorded in San Francisco, c.f. Fig. 9. In both cases, the to generate wider edges c.f. Fig. 7. This is useful, since the disparity maps are generated from the left and right RGB FCN roughly detects the Stixel cut positions, and the precise images using a stereo matching algorithm (Hirschmüller detection is left to the Stixel inference. 2008). This is the expected situation in a realistic scenario, We set the learning rate to 10−8 and the batch size to five: where the SGM algorithm in the perception pipeline gener- four of those inputs are Synthia images and one of them is ates the disparity map and feeds the FCN that produces the a real-data image. The missing disparities are encoded as Stixel over-segmentation. −1. Input normalization is done by subtracting the mean The ground-truth for the training data (the expected value from the disparity map. We initialize the FCN with Stixel cuts) is generated as a combination of methods. In the weights used in Schneider et al. (2017), since semantic the case of the annotated synthetic dataset, which contains segmentation is a similar problem. both pixel-level semantic and instance-level annotations, the ground-truth includes, as desired Stixel cuts, the boundaries of the instances and the semantic classes in the image (as in Cordts et al. 2017). Finally, the Stixel cuts associated to dis- 5 Experiments parity changes are obtained by running the Stixel inference This section assesses the accuracy and run-time of our pro- method. In the real-data sequence, we only perform this last posal. A previous concern is to verify that our method not step because we lack ground-truth. only improves the representation of scenes with non-flat roads, but also maintains the accuracy for scenes containing only flat roads. For that purpose, we present datasets of syn- thetic and real data to evaluate our proposal in Sect. 5.1. We introduce inputs, metrics, baselines, and other experimen- tal details in Sect. 5.2. Finally, quantitative and qualitative results are reported in Sect. 5.3. 5.1 Datasets As our Stixel model represents geometric and semantic infor- mation, we must evaluate the accuracy of our method for Fig. 9 Sample image from the real-data sequence used for Stixel cut both. For that purpose, we select Ladicky (Ladicky et al. generation. Stixel cut ground-truth is highlighted in red (Color figure 2014), an annotated subset of KITTI (Geiger et al. 2012), online) which is, to the best of our knowledge, the only dataset con- 123 International Journal of Computer Vision (2019) 127:1643–1658 1653 Fig. 10 The SYNTHIA-SF Dataset. A sample frame (left) with its depth (center) and semantic labels (right) taining both dense semantic labels and depth ground-truth. that we use to evaluate both depth and semantic accuracy in It consists of a set of 60 images with 0.5 MP resolution that non-flat roads. we use for evaluating Stixel semantic and depth accuracy. We follow the suggestion given by the author (Ladicky et al. 5.2 Experiment Details 2014) to ignore the three rarest object classes, which leaves us with 8 classes. 5.2.1 Metrics Additionally, for training our semantic segmentation FCN, we use publicly available semantic annotations on other parts We evaluate our proposed method in terms of semantic and of KITTY (Kundu et al. 2014; He and Upcroft 2013; Sengupta depth accuracy using two metrics. The depth accuracy is et al. 2013; Xu et al. 2013; Zhang et al. 2015). Our total train- obtained as the rate of outliers of the disparity estimates, ing set is composed of 676 images, where we harmonized the the standard metric used to evaluate on KITTI benchmark object classes used by the different authors to the previously (Geiger et al. 2012). An outlier is a disparity estimation with mentioned set suggested by Ladicky et al. (2014). This har- an absolute error larger than 3 px or a relative deviation larger monization and data processing is the same applied in the than 5% compared to the ground-truth. The semantic accu- previous work (Schneider et al. 2016) to allow for fair com- racy is evaluated with the average Intersection-over-Union parison. (IoU) over all classes, which is also a standard measure for In order to further evaluate disparity accuracy we use the semantic segmentation (Everingham et al. 2015). We mea- training data of the well-known stereo challenge KITTI 2015 sure the number of Stixels generated per image to quantify the (Geiger et al. 2012). This dataset provides a set of 200 images complexity of the obtained representation. Finally, we evalu- with sparse disparity ground-truth obtained from a laser scan- ate the inference speed of the algorithm using the Frame-rate ner. There is no suitable semantic ground-truth available for (Hz) metric, which helps us estimate if our system is capa- this dataset. ble of real-time performance. All the execution times of Furthermore, we also evaluate semantic accuracy using Stixels and SGM are obtained using a multi-threaded imple- Cityscapes (Cordts et al. 2016), a highly complex dataset mentation running on standard consumer hardware: Intel with dense annotations of 19 classes on ∼ 3000 images for i7-6800K. The semantic segmentation FCN frame-rate esti- training and 500 images for validation that we use for testing. mations are obtained using Maxwell NVidia Titan X. The Unfortunately, all the above datasets were generated in flat Stixel frame-rate includes the over-segmentation approach. road environments. Hence, they only help us validate that we Note that Stixel frame-rate is variable if we use an over- are not decreasing our accuracy for this kind of environments. segmentation method, i.e. it will depend on the number of In order to compare the accuracy of competing algorithms on Stixel cuts removed, therefore we provide a representative non-flat road scenarios, we need a new dataset. frame-rate. Similarly to Cordts et al. (2017), we can maxi- Therefore, we introduce a new synthetic dataset inspired mize the throughput of the system by computing SGM and by Ros et al. (2016). This dataset has been generated with the Semantic Segmentation in parallel, then the system would purpose of evaluating our proposed model; however, it con- run with one frame delay. tains enough information to be useful in additional related tasks, such as object recognition, semantic and instance seg- 5.2.2 Baseline mentation, among others. SYNTHIA-San Francisco (SYNTHIA-SF) consists of Semantic Stixels (Schneider et al. 2016) serve as our compar- photo-realistic frames rendered from a virtual city and comes ison baseline, as they achieve state-of-the-art results in terms with precise pixel-level depth and semantic annotations for of Stixel accuracy. We provide the accuracy of our new dis- 19 classes c.f. Fig. 10. This new dataset contains 2224 images parity model, c.f. Sect. 3. Finally, we evaluate the complexity 123 1654 International Journal of Computer Vision (2019) 127:1643–1658 Table 1 Accuracy of our methods compared to Semantic Stixels (Schneider et al. 2016), raw SGM and FCN Metric Dataset Input No over-segmentation Fast: time series Fast: FCN SGM FCN Sem. Stixels Ours Sem. Stixels Ours Sem. Stixels Ours Disp Error (%) Ladicky 16.66 − 17.38 16.84 17.60 17.01 17.44 16.84 KITTI 15 11.01 − 11.05 11.21 11.9 11.9 11.21 11.24 SYNTHIA-SF 11.06 − 29.33 12.99 30.60 14.20 31.12 14.19 IoU (%) Ladicky – 69.8 66.2 66.1 66.0 66.0 66.2 66.1 Cityscapes – 66.7 65.4 65.8 64.9 65.0 65.5 65.6 SYNTHIA-SF – 48.1 46.0 48.5 45.7 48.0 47.0 48.6 We evaluate on four datasets: Ladicky (Ladicky et al. 2014), KITTI 15 (Geiger et al. 2012), Cityscapes (Cordts et al. 2016) and SYNTHIA-SF using these metrics: Disparity Error (less is better) and Intersection over Union (more is better) c.f. Sects. 5.1 and 5.2.1. Fast versions are detailed in Sects. 4.1 and 4.2. Significantly best results are highlighted in bold Table 2 Number of Stixels Dataset Input No over-segmentation Fast: time series Fast: FCN (103 ) generated by our methods compared to Semantic Stixels SGM/FCN Sem. Stixels Ours Sem. Stixels Ours Sem. Stixels Ours (Schneider et al. 2016) and raw input (total number of pixels) Ladicky 454 0.6 0.6 0.6 0.6 0.6 0.6 KITTI 15 452 0.7 0.7 0.7 0.7 0.7 0.7 Cityscapes 2k 1.4 1.5 1.3 1.4 1.4 1.5 SYNTHIA-SF 2k 1.5 1.7 1.2 1.3 1.3 1.3 We evaluate on four datasets: Ladicky (Ladicky et al. 2014), KITTI 15 (Geiger et al. 2012), Cityscapes (Cordts et al. 2016) and SYNTHIA-SF c.f. Sect. 5.1. Fast versions are detailed in Sects. 4.1 and 4.2 60 54.3 Table 3 Per-stage report of frame-rate of our pipeline for a stereo pair of resolution 1242 × 375. OS stands for Over-segmentation Frame-rate (Hz) 38.9 Stage Frame-rate (Hz) 40 35.2 33.1 SGM 55 19.4 Semantic segmentation 47.6 20 Our Stixels (OS: time series) 116 4.7 Our Stixels (OS: FCN) 130 0 Our Stixels (No OS) 61 No over-segmentation Time Series FCN Total (OS: time series) 20.92 Total (OS: FCN) 21.33 Semantic Stixels Ours Total (No OS) 18 Fig. 11 Frame-rate of our method (only the Stixel computation step and SGM run-time using a CPU Intel i7-6800K. For the Semantic Segmen- the corresponding over-segmentation approach) compared to Semantic tation method, a Maxwell NVidia Titan X is used. Note that Stixel Stixels (Schneider et al. 2016) for SYNTHIA-SF (image resolution of frame-rate is variable if we use an over-segmentation method, therefore 1920×1080) on a multi-threaded CPU implementation (Intel i7-6800K) we provide a representative run-time. The total frame-rate is reported computed with a Stixel width of 8 pixels and equivalent down-sampling as the sum of the stages in the v-direction. Different methods of over-segmentation are also com- pared, these are: Time Series c.f. Sect. 4.1, FCN c.f. Sect. 4.2 model used in Schneider et al. (2016) without retraining, to of the fast approach defined in Sect. 3.4, with the two over- allow for comparison. For the same reason, we set Stixel segmentation techniques presented in Sects. 4.1 and 4.2. width to 8 px. The same down-sampling is applied in the vertical direction. The rest of the parameters used are taken 5.2.3 Input from Schneider et al. (2016). We use the camera parameters obtained after calibration As input, we use disparity images obtained via SGM to set the expected values of μagr ound and μbgr ound . For object (Hirschmüller 2008) and pixel-level semantic labels com- Stixels, we set σobject b → 0, μbobject = 0 because the dis- puted by an FCN (Long et al. 2015). We use the same FCN parity is too noisy for the slanted object model. Finally, 123 International Journal of Computer Vision (2019) 127:1643–1658 1655 since sky Stixels can not have slanted surfaces, we set: Cityscapes datasets in Table 1. These results indicate that the μasky = 0, μasky = 0, σsky a → 0, σ b → 0. sky novel and more flexible model does not harm the accuracy In order to improve the computational efficiency of our in such scenarios. approach, we use the two Fast Stixel over-segmentation We also observe that our novel model is able to accurately methods presented in Sect. 4.1, labeled as Time Series, and represent non-flat scenarios in contrast to the original Stixel Sect. 4.2, labeled as FCN. approach, yielding a substantially increased depth accuracy of more than 16% c.f. when comparing Semantic Stixels to 5.3 Results Ours for the SYNTHIA-SF dataset in Table 1. Additionally, to verify that our method equally works also on real data, we The quantitative results of our proposals and baselines, as provide a video of the Stixel 3D representation of a challeng- described in Sect. 3, are shown in Tables 1 and 2 and Fig. 11. ing non-flat road scene as supplementary material. Results The first observation is that our method achieves compara- also improve in terms of semantic accuracy, which we explain ble or slightly better results on all datasets with flat roads c.f. as a consequence of the joint semantic and depth inference compare Semantic Stixels to Ours for Ladicky, KITTI 15 and that benefits from a better depth model. Fig. 12 Exemplary outputs on real data: in all cases with non-flat roads sifies the road as sidewalk due to erroneous semantic input. However, our model correctly represents the scene, while retaining accuracy on the original approach reconstructs a wall in this case, highlighted by a objects. The last example shows a failure case, where our approach clas- red circle. This could lead to an emergency break (Color figure online) 123 1656 International Journal of Computer Vision (2019) 127:1643–1658 A perfect over-segmentation method would find all opti- As future work, we would like to focus on circumvent- mal cuts, and consequently, it would have the same accuracy ing the limitations of our method. Namely, (1) the verti- as not using any over-segmentation. cal/column independence assumed by the model is clearly not Our novel approach Fast: FCN has an accuracy almost true. A more global representation, e.g. super-pixels that span equal to not using any over-segmentation method (in all cases vertically and horizontally, would be more compact and less but one). Note that, our proposed approach Fast: FCN is prone to errors; (2) some surfaces are not well represented by superior to Fast: Time Series method in all cases c.f. when a linear model, e.g. cars. A more complex depth model and comparing both methods for the SYNTHIA-SF dataset in specific models for each semantic class could represent more Table 1. faithfully the scene. Nonetheless, a model with more free Both over-segmentation methods increase the error for our variables could also lead to a bad representation because of challenging SYNTHIA-SF dataset; we think this is because the noise; (3) the proposed over-segmentation algorithm has of the difficult road Stixel cuts in these scenes, c.f. compare a non-predictable run-time. And this is a bad characteristic No over-segmentation to Fast methods in Table 1. for a real-time system. The worst-case scenario, i.e. no Stixel All variants are compact representations of the surround- cuts removed, is as slow as not using over-segmentation at ing, since the complexity of the Stixel representation is small all (although very unlikely); (4) in case of movement of the compared to the high resolution input images, c.f. Table 2. stereo rig during operation, there could be an offset in roll Our last observation is that the proposed Fast variants effectively breaking the vertical world assumption. improve the run-time of the original Stixel approach by up to 2×, and also improve the novel Slanted Stixel approach Acknowledgements This work has been partially supported by Span- ish TIN2017-84553-C2-1-R (MINECO/AEI/FEDER, UE). We also by up to 7×, with only a slight drop in depth accuracy c.f. thank the Generalitat de Catalunya CERCA Program, the 2017-SGR- Fig. 11. The benefit increases with higher resolution input 1597 and 2017-SGR-313 projects, as well as the ACCIO agency. We images due to the quadratic and cubic computational com- also acknowledge SEBAP for the internship funding program. Anto- plexity of the original and slanted Stixel inference methods, nio M. López acknowledges the financial support by the Spanish TIN2017-88709-R (MINECO/AEI/FEDER, UE), and by ICREA under respectively. We also detail per-stage run-time c.f. Table 3 the ICREA Academia Program. Finally, we thank Francisco Molero for completeness. and Marc García at CVC/UAB for the generation of the SYNTHIA-SF In addition to the quantitative evaluation presented before, dataset. we have visually inspected many of the obtained Stixel rep- Open Access This article is distributed under the terms of the Creative resentations, to check the qualitative differences between our Commons Attribution 4.0 International License (http://creativecomm proposal and the previous work. Figure 12 illustrates some ons.org/licenses/by/4.0/), which permits unrestricted use, distribution, of these examples, in which the scenes with non-flat roads and reproduction in any medium, provided you give appropriate credit are correctly represented and all the objects in the scenario to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. are identified by our proposal, while the previous model pro- duces an incomplete road representation, or even generates non-existing objects at some road positions. References Benenson, R., Mathias, M., Timofte, R., & Gool, L. J. V. (2012). Fast 6 Conclusions stixel computation for fast pedestrian detection. In Proceedings, Part III Computer vision—ECCV 2012. Workshops and demon- strations—Florence, Italy, October 7–13, 2012 (pp 11–20). https:// This paper presented a novel depth model for the Stixel world doi.org/10.1007/978-3-642-33885-4_2. that is able to account for non-flat roads and slanted objects in Benenson, R., Timofte, R., & Gool, L. J. V. (2011). Stixels estimation a compact representation that overcomes the previous restric- without depth map computation. In IEEE international conference on computer vision workshops, ICCV 2011 Workshops, Barcelona, tive constant height and depth assumptions. This change Spain, November 6–13, 2011 (pp. 2010–2017). https://doi.org/10. in the way Stixels are represented is required for difficult 1109/ICCVW.2011.6130495. environments that are found in many real-world scenarios. Bucila, C., Caruana, R., & Niculescu-Mizil, A. (2006). Model compres- Moreover, in order to significantly reduce the computational sion. In Proceedings of the twelfth ACM SIGKDD international conference on knowledge discovery and data mining, Philadel- complexity of the extended model, a novel approximation phia, PA, USA, August 20–23, 2006 (pp. 535–541). https://doi. has been introduced that consists of checking only reason- org/10.1145/1150402.1150464. able Stixel cuts inferred using fast methods. We showed in Carrillo, D. A. P., & Sutherland, A. (2016). Fast obstacle detection extensive experiments on several related datasets that our using sparse edge-based disparity maps. In Fourth international conference on 3D Vision (3DV), 2016, IEEE (pp. 66–72). depth model is able to better represent slanted road scenes, Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benen- and that our approximation is able to reduce the run-time son, R., Franke, U., Roth, S., & Schiele, B. (2016). The cityscapes drastically, with only a slight drop in accuracy. dataset for semantic urban scene understanding. In 2016 IEEE con- 123 International Journal of Computer Vision (2019) 127:1643–1658 1657 ference on computer vision and pattern recognition, CVPR 2016, Li, X., Flohr, F., Yang, Y., Xiong, H., Braun, M., Pan, S., Li, K., & Las Vegas, NV, USA, June 27–30, 2016 (pp. 3213–3223). https:// Gavrila, D. M. (2016). A new benchmark for vision-based cyclist doi.org/10.1109/CVPR.2016.350. detection. In 2016 IEEE intelligent vehicles symposium, IV 2016, Cordts, M., Rehfeld, T., Schneider, L., Pfeiffer, D., Enzweiler, M., Roth, Gotenburg, Sweden, June 19–22, 2016 (pp 1028–1033). https:// S., et al. (2017). The stixel world: A medium-level representation doi.org/10.1109/IVS.2016.7535515. of traffic scenes. Image and Vision Computing,. https://doi.org/10. Lin, M., Chen, Q., & Yan, S. (2013). Network in network. In Proceed- 1016/j.imavis.2017.01.009. ings of the international conference on learning representations Cordts, M., Schneider, L., Enzweiler, M., Franke, U., & Roth, S. (2014). (ICLR). Object-level priors for stixel generation. In Proceedings pattern Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional net- recognition - 36th German conference, GCPR 2014, Münster, works for semantic segmentation. In IEEE conference on computer Germany, September 2–5, 2014 (pp. 172–183). https://doi.org/10. vision and pattern recognition, CVPR 2015, Boston, MA, USA, 1007/978-3-319-11752-2_14. June 7–12, 2015 (pp. 3431–3440). https://doi.org/10.1109/CVPR. Dhiman, V., Kundu, A., Dellaert, F., & Corso, J. J. (2014). Modern MAP 2015.7298965. inference methods for accurate and fast occupancy grid mapping Muffert, M., Schneider, N., & Franke, U. (2014). Stix-fusion: A prob- on higher order factor graphs. In ICRA. abilistic stixel integration technique. In Canadian conference on Everingham, M., Eslami, S. M. A., Gool, L. J. V., Williams, C. K. I., computer and robot vision, CRV 2014, Montreal, QC, Canada, Winn, J. M., & Zisserman, A. (2015). The pascal visual object May 6-9, 2014, pp 16–23. https://doi.org/10.1109/CRV.2014.11. classes challenge: A retrospective. International Journal of Com- Nuss, D., Yuan, T., Krehl, G., Stuebler, M., Reuter, S., & Dietmayer, puter Vision, 111(1), 98–136. https://doi.org/10.1007/s11263- K. (2015). Fusion of laser and radar sensor data with a sequen- 014-0733-5. tial Monte Carlo Bayesian occupancy filter. In IEEE intelligent Fink, E., & Gandhi, H. S. (2011). Compression of time series by vehicles symposium (IV). extracting major extrema. Journal of Experimental & Theoretical Pfeiffer, D. (2014). The stixel world—a compact medium-level rep- Artificial Intelligence, 23(2), 255–270. resentation for efficiently modeling three-dimensional environ- Geiger, A., Lenz, P., & Urtasun, R. (2012). Are we ready for autonomous ments. Ph.D. thesis, Hu Berlin. driving? the KITTI vision benchmark suite. In Conference on com- Pfeiffer, D., & Franke, U. (2011). Towards a global optimal multi-layer puter vision and pattern recognition. stixel representation of dense 3D data. In Proceedings of British He, H., & Upcroft, B. (2013). Nonparametric semantic segmentation for machine vision conference, BMVC 2011, Dundee, UK, August 29– 3D street scenes. In 2013 IEEE/RSJ international conference on September 2, 2011 (pp 1–12). https://doi.org/10.5244/C.25.51. intelligent robots and systems, Tokyo, Japan, November 3–7, 2013 Pfeiffer, D., Gehrig, S., & Schneider, N. (2013). Exploiting the power of (pp. 3697–3703). https://doi.org/10.1109/IROS.2013.6696884. stereo confidences. In 2013 IEEE conference on computer vision Hernandez-Juarez, D., Espinosa, A., Moure, J. C., Vázquez, D., & and pattern recognition, Portland, OR, USA, June 23–28, 2013 López, A. M. (2017). GPU-accelerated real-time stixel computa- (pp 297–304). https://doi.org/10.1109/CVPR.2013.45. tion. In 2017 IEEE winter conference on applications of computer Ros, G., Sellart, L., Materzynska, J., Vazquez, D., & Lopez, A. (2016). vision, WACV 2017, Santa Rosa, CA, USA, March 24–31, 2017 The SYNTHIA Dataset: A large collection of synthetic images for (pp. 1054–1062). https://doi.org/10.1109/WACV.2017.122. semantic segmentation of urban scenes. In CVPR. Hernandez-Juarez, D., Schneider, L., Espinosa, A., Vázquez, D., López, Schneider, L., Cordts, M., Rehfeld, T., Pfeiffer, D., Enzweiler, M., A. M., Franke, U., Pollefeys, M., & Moure, J. C. (2017). Slanted Franke, U., Pollefeys, M., & Roth, S. (2016). Semantic stixels: stixels: Representing san francisco’s steepest streets. In British Depth is not enough. In 2016 IEEE intelligent vehicles symposium, Machine Vision Conference, BMVC 2017, London, UK, September IV 2016, Gotenburg, Sweden, June 19–22, 2016 (pp 110–117). 4–7, 2017. https://doi.org/10.1109/IVS.2016.7535373. Hirschmüller, H. (2008). Stereo processing by semiglobal matching Schneider, L., Jasch, M., Fröhlich, B., Weber, T., Franke, U., Pollefeys, and mutual information. IEEE Transactions on Pattern Analy- M., & Rätsch, M. (2017). Multimodal neural networks: Rgb-d sis and Machine Intelligence, 30(2), 328–341. https://doi.org/10. for semantic segmentation and object detection. In Scandinavian 1109/TPAMI.2007.1166. conference on image analysis, Berlin: Springer (pp 98–109). Ignat, O. (2016). Disparity image segmentation for free-space detec- Sengupta, S., Greveson, E., Shahrokni, A., & Torr, P. H. S. (2013). tion. In 2016 IEEE 12th international conference on intelligent Urban 3D semantic modelling using stereo vision. In 2013 IEEE computer communication and processing (ICCP) (pp. 217–224). international conference on robotics and automation, Karlsruhe, https://doi.org/10.1109/ICCP.2016.7737150. Germany, May 6–10, 2013 (pp 580–585). https://doi.org/10.1109/ Jasch, M., Weber, T., & Rätsch, M. (2018). Fast and robust RGB-D scene ICRA.2013.6630632. labeling for autonomous driving. Journal of Computers, 13(4), Thrun, S. (2002). Robotic mapping: A survey. In G. Lakemeyer & B. 393–400. Nebel (Eds.), Exploring artificial intelligence in the new mille- Kundu, A., Li, Y., Dellaert, F., Li, F., & Rehg, J. M. (2014). Joint seman- nium. Burlington: Morgan Kaufmann. tic segmentation and 3D reconstruction from monocular video. In Viola, P., & Jones, M. (2001). Rapid object detection using a boosted 13th European conference computer vision—ECCV 2014, Zurich, cascade of simple features. In Proceedings of the 2001 IEEE Switzerland, September 6–12, 2014, Proceedings, Part VI (pp. computer society conference on Computer vision and pattern 703–718). https://doi.org/10.1007/978-3-319-10599-4_45. recognition, 2001. CVPR 2001, IEEE, (vol. 1, pp. I–I). Ladicky, L., Shi, J., & Pollefeys, M. (2014). Pulling things out of per- Xie, S., & Tu, Z. (2017). Holistically-nested edge detection. Interna- spective. In 2014 IEEE conference on computer vision and pattern tional Journal of Computer Vision, 125(1–3), 3–18. https://doi. recognition, CVPR 2014, Columbus, OH, USA, June 23–28, 2014 org/10.1007/s11263-017-1004-z. (pp. 89–96). https://doi.org/10.1109/CVPR.2014.19. Xu, P., Davoine, F., Bordes, J., Zhao, H., & Denoeux, T. (2013). Infor- Levi, D., Garnett, N., & Fetaya, E. (2015). Stixelnet: A deep convo- mation fusion on oversegmented images: An application for urban lutional network for obstacle detection and road segmentation. In scene understanding. In Proceedings of the 13. IAPR international Proceedings of the British machine vision conference 2015, BMVC conference on machine vision applications, MVA 2013, Kyoto, 2015, Swansea, UK, September 7–10, 2015 (pp. 109.1–109.12). Japan, May 20–23, 2013 (pp 189–193). URL http://www.mva- https://doi.org/10.5244/C.29.109. org.jp/Proceedings/2013USB/papers/08-04.pdf. 123 1658 International Journal of Computer Vision (2019) 127:1643–1658 Zhang, R., Candra, S. A., Vetter, K., & Zakhor, A. (2015). Sensor fusion Publisher’s Note Springer Nature remains neutral with regard to juris- for semantic segmentation of urban scenes. In IEEE international dictional claims in published maps and institutional affiliations. conference on robotics and automation, ICRA 2015, Seattle, WA, USA, 26–30 May, 2015 (pp. 1850–1857). https://doi.org/10.1109/ ICRA.2015.7139439. Ziegler, J., Bender, P., Dang, T., & Stiller, C. (2014). Trajectory planning for bertha—A local, continuous method. In 2014 IEEE intelligent vehicles symposium proceedings, Dearborn, MI, USA, June 8–11, 2014 (pp. 450–457). Ziegler, J., Bender, P., Schreiber, M., Lategahn, H., Strauss, T., Stiller, C., Dang, T., Franke, U., Appenrodt, N., Keller, C. G., Kaus, E., Herrtwich, R. G., Rabe, C., Pfeiffer, D., Lindner, F., Stein, F., Erbs, F., Enzweiler, M., Knoppel, C., Hipp, J., Haueis, M., Trepte, M., Brenk, C., Tamke, A., Ghanaat, M., Braun, M., Joos, A., Fritz, H., Mock, H., Hein, M., & Zeeb, E. (2014), Making bertha drive—an autonomous journey on a historic route. In IEEE intelligent trans- portation systems magazine (pp. 8–20). https://doi.org/10.1109/ MITS.2014.2306552. 123