This is the submitted version of the journal article: Boquet, Guillem; Morell Pérez, Antoni; Serrano García, Javier; [et al.]. «A variational autoencoder solution for road traffic forecasting systems : missing data imputation, dimension reduction, model selection and anomaly detection». Transportation Research Part C: Emerging Technologies, Vol. 115 (June 2020), art. 102622. DOI 10.1016/j.trc.2020.102622 This version is available at https://ddd.uab.cat/record/273834 under the terms of the license A variational autoencoder solution for road traffic forecasting systems: missing data imputation, dimension reduction, model selection and anomaly detection Guillem Boquet⇤, Antoni Morell, Javier Serrano, Jose Lopez Vicario Wireless Information Networking (WIN) Group Universitat Autònoma de Barcelona (UAB) Abstract E↵orts devoted to mitigate the e↵ects of road traffic congestion have been conducted since 1970s. Nowadays, there is a need for prominent solutions capable of mining information from messy and multidimensional road traffic data sets with few modeling constraints. In that sense, we propose a unique and versatile model to address di↵erent major challenges of traffic forecasting in an unsupervised manner. We formulate the road traffic forecasting problem as a latent variable model, assuming that traffic data is not generated randomly but from a latent space with fewer dimensions containing the underlying characteristics of traffic. We solve the problem by proposing a variational autoencoder (VAE) model to learn how traffic data are generated and inferred, while validating it against three di↵erent real-world traffic data sets. Under this framework, we propose an online unsupervised imputation method for unobserved traffic data with missing values. Additionally, taking advantage of the low dimension latent space learned, we compress the traffic data before applying a prediction model obtaining improvements in the forecasting accuracy. Finally, given that the model not only learns useful forecasting features but also meaningful characteristics, we explore the latent space as a tool for model and data selection and traffic anomaly detection from the point of view of traffic modelers. Keywords: Intelligent Transportation Systems, Traffic Forecasting, Missing Data Imputation, Dimension Reduction, Anomaly Detection, Model Selection 2019 MSC: 00-01, 99-00 1. Introduction Traveler Information Systems (ATIS). Those sys- tems have been continuously improving and will Due to recent developments in Intelligent Trans- 15 continue to with the expansion of technology and portation Systems (ITS), road traffic forecasting data provided by sensors in roads and vehicles, the conforms a vivid area of research, policy making envisaged Vehicle-to-Everything (V2X) paradigm. 5 and technology development. One of its founda- In that context, data-driven approaches like deep tions is to predict traffic characteristics, since traf- neural networks (DNN) have arisen as a prominent fic congestion generates important social, economic 20 solution as they are capable of mining information and environmental problems [1]. E↵orts devoted to from messy and multi-dimensional traffic data sets mitigate the e↵ects of traffic congestion have been with few modeling constraints [1, 2]. However, the 10 conducted from the 1970s to the present in traffic intrinsic characteristics of road traffic still makes flow management with the use of Advanced Traf- the forecast a challenging problem because of com- fic Management Systems (ATMS) and Advanced 25 plex spatial dependency on road networks [3], non- linear temporal dynamics with changing road con- ⇤ Corresponding ditions and inherent difficulties of long-term fore- author Email address: guillem.boquet@uab.cat (Guillem casting [4]. In addition to the forecasting problem, Boquet ) more challenges of equal magnitude derive from said Preprint submitted to Journal of LATEX Templates March 5, 2020 30 context. To name a few, the quality of data, arterial 2. how to extract useful features while compress- and network-level predictions, spatiotemporal fore- ing the traffic data casts and model selection techniques are identified 3. how to select the best model and data for a as current major challenges of future road traffic specific traffic estimation problem forecasting [1, 2]. 85 4. how to detect anomalous traffic 35 Two main categories can be distinguished in data-driven approaches that address the traffic fore- To answer that, in this work we learn the underly- casting problem: parametric and non-parametric ing structures that generate real-world traffic data. models. The parametric category is represented We assume that road traffic is not generated ran- by the autoregressive integrated moving average domly, but from a latent subspace, based on the as- 40 (ARIMA) [5], which remains popular (e.g., seasonal 90 sumption that there are strong spatiotemporal re- ARIMA coupled with the Kalman filter [6]) despite lationships and seasonality between points in the relying on the assumption of stationarity of traf- road network. We formulate it as a latent variable fic data. Non-parametric models are the k-nearest model that forces us to approximate the joint prob- neighbor algorithm (KNN) [7], support vector re- ability distribution via variational inference. Thus, 45 gression (SVR) [8] and K-means [9], but the cur- 95 we base our model on VAE, which we show is able rent trend is focused on DNN models that show to learn an approximation of the data distribution better better results due to advances in deep learn- of three di↵erent real-world traffic data set. Un- ing techniques [10, 11]. For instances, convolutional der this framework, the posterior distribution of the neural networks (CNN) [12] and recurrent neural latent space is forced to be continuous, which al- 50 networks (RNN) [13] are coupled into more complex 100 lows the model to decode plausible traffic samples and deeper networks to characterize the spatiotem- from every point in the subspace, therefore, to on- poral behavior of traffic and increase the accuracy line impute unobserved missing traffic data without of traffic forecasting systems [4, 14, 15, 16, 17]. Nev- supervision. In said subspace, traffic of the same ertheless, all these supervised methods are feed with class ends up closer together, allowing unsupervised 55 raw data without any or few preprocessing steps. 105 traffic classification and at the same time detecting All available data are used for training, which may anomalous traffic. The latent space dimensions are not be the best option for optimal forecasting per- constrained, which results in learning useful proper- formance. Data quality and missing values ??are ties of traffic to compress the data, that is, feature not considered, which is known to degrade system extraction. Moreover, since VAE is a generative 60 performance. In addition, an excessive number of 110 model, the model allows traffic modelers or prac- features available from data sources are used to titioners to sample from the learned distribution forecast, which is computational inefficient and un- to generate new traffic data and the possibility to dertakes the risk of over-fitting. explore into the meaningful latent representations. Now, using the contributions of this manuscript, In this manuscript, we propose a unified solution 115 a traffic modeler can implement a model to com- 65 to the aforementioned problems, which have been press the traffic data and efficiently forecast, im- addressed separately in the literature. Following pute missing values, select the best data and model the DNN trend, we propose to merge the recent for a specific problem and detect anomalous traffic advances in variational inference [18] with traffic data at the same time with no additional knowledge forecasting systems as we assume and show that 120 required. Before, if an ITS that already estimated 70 the traffic forecasting problem can be formulated traffic required addressing other traffic problems, as a latent variable model and resolved in an unsu- such as missing data imputation or data compres- pervised manner using the Variational Autoencoder sion, it was forced to implement other models, re- (VAE) framework [19, 20]. According to the best of sources and practitioners to solve it. authors’ knowledge, this is the first work that im- 125 In summary, our main contributions are: 75 plements a VAE-based model for traffic data. Here, we propose a unique and versatile framework to – The applicability of the VAE model for real- solve the following major challenges of road traf- world road traffic data as a unified solution to fic forecasting: for various traffic forecasting problems. 1. how to impute missing data to not deteriorate – A novel online multidimensional imputation 80 the performance of forecasting systems 130 method for missing values in road traffic data 2 based on learning the probability distribution characterize the data generation process and learn of the data given the observed values. underlying statistical patterns in traffic data. On the other hand, state-of-the-art imputation – A novel dimension reduction approach to traf- 180 methods from other research fields can be classi- fic data to improve efficiency and accuracy of fied [26] as either discriminative, such as multi- 135 forecasting systems by learning powerful char- ple imputation by chained equations (MICE) [27] acteristics of traffic in an unsupervised manner. and matrix completion [28], or generative meth- ods based on DNN. For example, [29] proposed – A novel tool for traffic modelers using pro- 185 an overcomplete denoising autoencoder (DAE) to jected data in a unsupervised learned subspace be able to reconstruct data by stochastically cor- with meaningful dimensions that can be used rupting it. Closer to our work, [30] and [31] pro- 140 for model and data selection and anomaly de- posed a RNN-based VAE which succeed at imput- tection. ing missing words from sentences. [32] applied a – Three case studies on real-world data from 190 deep sequential VAE with a Gaussian process prior USA (California) and UK (England) that val- in the latent space to capture temporal dynamics idates the usefulness of the proposed frame- to impute real-world medical data. Similarly, [33] 145 work. and [34] proposed also a generative model imputa- tion method but using generative adversarial net- 195 works (GAN). Contrary to GAN, our VAE-based 2. Related work approach possesses certain desirable properties for traffic forecasting systems, such as stable train- To the date of this manuscript, the most up-to- ing [35], interpretable encoder/decoder network [36] date reviews on road traffic forecasting are Laña et and outlier-robustness [37]. al. in [1] and Vlahogianni et al. in [2]. Our work 150 is related to the current applications of neural net- 200 Dimension reduction. The number of features works for road traffic forecasting in di↵erent ways. available from data sources jointly with the num- We divided the related work discussion accordingly ber of available data points in road networks are as we are proposing a transversal solution to di↵er- excessive. Forecasting with all those features can ent problems that have been addressed separately be computational inefficient and undertakes the risk 155 in literature. 205 of over-fitting. Therefore, it is essential to reduce the dimension of the feature space before applying a Missing data imputation. Road traffic forecasting prediction model [38]. Reduction of the data is done systems are deployed in scenarios where sensor and by learning the principal components or indepen- system failure are common. In these scenarios, the dent factors of a given data manifold, i.e., feature missing values ??are known to negatively a↵ect the 210 extraction. Recently, a systematic literature review 160 precision of the forecast [21] although they are of- of feature selection and extraction in spatiotempo- ten underestimate in current forecast models [1, 2]. ral traffic forecasting was reported in [39]. Note The current strategy is to preprocess the data by that feature extraction does not necessarily mean inferring the missing values from the known part reducing the dimension of the data space, that is, of the data. Three well-known imputation meth- 215 dimension reduction is a subclass of feature extrac- 165 ods in traffic forecasting are ARIMA, KNN and tion methods. The low-dimensional representation principal component analysis (PCA)-based meth- is traditionally obtained by PCA approaches that ods. [22] compared them among others and the re- had been widely used to extract the linear corre- sults showed that the probabilistic PCA is the most lations between the variables [40, 38]. In addition, e↵ective in terms of performance and implementa- 220 the least absolute shrinkage and selection operator 170 tion. More recently, [23] proposed a spatial context (LASSO) is a well-known technique used [41, 38]. sensing model based on an automated clustering On the other hand, [42] exploited compressed sens- analysis tool and the information provided by sur- ing to reduce the complexity of road networks prior rounding sensors. [24] proposed a model that com- to regression. In DNN data-driven approaches, bines long-short term memory (LSTM), SVR and 225 RNN and CNN are used to extract temporal and 175 collaborative filtering. With a similar approach to spatial characteristic within the regression model. ours, [25] proposed a Bayesian imputation model to [16] used a LSTM and CNN mixed with an atten- 3 tion layer but not as an independent layer to the a hybrid method of short-term traffic forecasting regression task like our proposal. Similarly to our using a self-organized Kohonen map as an initial 230 work, features learned from a stack of autoencoders 280 classifier where each class had an individually as- (SAE) have been previously used in literature to sociated tuned ARIMA model. But, according to improve traffic forecasting [10, 43]. Contrary to the best knowledge of the authors, there is no work the autoencoder, the VAE encourages the model that presents a VAE applied to the selection of the to generalize features and reconstruct samples as model or data. 235 an aggregation of those, forces the latent space to be continuous and is a generative model. Likewise, 285 Anomaly detection. One of the main applications other VAE approaches have been used successfully of urban traffic analysis lies in detecting anoma- for dimensionality reduction within other research lies from traffic data [51]. Recently, [51] and [52] fields, such as fault diagnosis [44] and towards se- reviewed on existing outlier detection techniques 240 quencing the RNA of individual cells [45]. in traffic data in three main categories: statisti- 290 cal, similarity-based, and based on pattern analy- Model explanatory power. The explanatory and sis. Among them, some find outliers in subspaces, representative power of models is valuable for traffic which is exactly what VAE can provide. In [53], modelers to obtain information on how transporta- dimensionality reduction is performed by PCA and tion networks behave and evolve. Some e↵orts have a kNN-based outlier detection is applied in the de- 245 been devoted to explain the behavior of the mod- 295 rived subspace. On the contrary, the VAE is based els in the literature as a second derivative of traf- on a DNN that has greater modeling capabilities. fic forecast. For example, [46] analyzed the spatial In fact, a linear autoencoder learns to span the features captured by CNN through characterizing same subspace as PCA. In the learned latent space the information that retained layer by layer. [41] by our proposed model, traffic samples are clus- 250 discussed how the input variables relate to the pre- 300 tered and projections close to each are forced to dicted output using the coefficients of the fitted lin- have similar reconstructions that help in the detec- ear model. In that context, our proposal learns the tion of outliers. Moreover, when our proposal is probability distribution of the given traffic data and trained with much more normal samples than the a low-dimension continuous latent space. Thus, we anomalous ones, the reconstruction errors of nor- 255 exploit these features as a tool to perform model se- 305 mal data are relatively higher than those of anoma- lection and anomaly detection for traffic forecasting lous data. Therefore, the VAE loss function pro- systems. vides an anomaly score function which can be ex- ploited as an anomaly detection technique. In that Model selection. There is no best method that suits regard, VAE-based outlier detection methods had all situations in traffic forecast [47], which implies 310 been used successfully in other research fields: [54] 260 an applicability at a higher level of the method added a supervised method to the VAE approach to choose the most suitable model given the char- to enhance detection of seen anomalies without de- acteristics of the forecasting problem [1]. Traffic grading the performance for unseen anomalies on modelers frequently face several optimization chal- real industrial data, [55] proposed a a RNN-based lenges related to model selection, while there are 315 VAE to detect anomalies on robot time series data 265 no clear baselines to find the best method and its and [56] proposed an anomaly detection method configuration [48]. According to the best of au- based on a reconstruction probability derived from thors’ knowledge, few works are related to the traf- the VAE loss function. fic forecast context. [49] proposed a metamodeling technique to optimize both algorithm selection and 3. Methodology 270 hyperparameter setting and [48] explored the use of Auto-WEKA, an automatic algorithm selection 320 In this section, we formulate the traffic forecast- method. On the contrary, we approach the prob- ing problem as a latent variable model. We propose lem from a data perspective. We provide a tool a VAE implementation to learn how traffic data based on the clustering of data in the learned la- are generated and to learn an approximation of the 275 tent space to select the data from which the best probability distribution of the data. Then, we pro- forecasting model will be built to solve a specific 325 pose a procedure to impute missing values. Finally, problem. Similar to our approach, [50] proposed we exploit the learned latent subspace proposing a 4 Intelligent Transportation System Proposed VAE Framework Anomaly Detection Underlying traffic Road Traffic Forecasting Traffic big data characteristics Missing Value Imputation Recognition Generative model model Historical (encoder) (decoder) Spatial Temporal Latent space Feature Extraction Traffic Network Knowledge Future state of road traffic network Compressed data Compressed Model & Data data Dimension Reduction Selection Figure 1: Interrelation of the proposed VAE model with di↵erent traffic forecasting tasks. The proposal is a unified framework that aims to solve several important challenges of ITS in road traffic forecasting. tool for model and data selection, dimension reduc- strong spatio-temporal relationships exist between tion and anomaly detection. A general framework road network’s points [1]. For instances, due to diagram that links the model to the di↵erent traffic seasonality, it is possible to discern between a work 330 prediction tasks is depicted in Fig. 1. day or not just by observing how morning traffic 365 develops through time and space.. Therefore, this 3.1. Problem definition can be seen as a generative model, where data can Let X = {x(i) }N i=1 be a historic traffic data set be generated from a latent manifold with fewer di- composed of N observed traffic variables or traffic mensions, or as a reconstruction problem, where samples with an unknown ground-truth probabil- applying a function to a corrupted input x̃ derives 335 ity distribution, x(i) ⇠ pgt (x). Let each element 370 the actual input. within x(i) represent a value of a traffic variable as- As with the forecasting problem, this function sociated with time and space, x(i) 2 Rn⇥d where can be approximated using DNN to obtain an es- n is the number of past traffic variables and d the timate x̂ of the actual input. Next, based on the number of traffic sensors deployed into the road net- previous assumption, we formulate the problem as 340 work. The method does not use information about 375 a latent variable model and solve that using a deep the position of the sensors along the route. Hence- learning variational inference approach. forth, the superscript (i) denoting the i -th sample is omitted to avoid clutter, except in cases where some 3.2. Latent variable model ambiguity may exist. Note that x is real-valued, 345 which is intended to represent traffic variables such Let {z (i) }N i=1 be the set of vectors composed of as speed, flow, density, etc., hence, the methodol- continuous random latent variables defining a low- ogy presented during this section will be derived dimensional representation of the significant factors accordingly. Likewise, let y 2 Rm denote the fu- of variation in X, z 2 RJ with J ⌧ dim(x). Thus, ture state of m  d subset of sensors in the time whenever a traffic sample x is feed to the model, 350 horizon of h samples. The traffic forecast problem z will represent its underlying characteristics. Let can be modeled as y = f ⇤ (x), where forecast sys- f denote a deterministic function derived from a tems aim to make an accurate estimate of y from x, neural network that maps z to the data space and while the challenge remains on deriving a function that closely resembles f ⇤ . X ⇡ X̂ = f (z; ✓) , (1) 355 Suppose we want to infer the traffic behavior dur- ing the next two hours or that we have a partially the generative model (Fig. 2) parametrized with ✓, occluded traffic sample due to a sensor or system where X̂ is the estimate of X. Our motivation is failure. Missing data could be anything if there is to learn f that minimizes the error between X and no underlying structure (or subspace) from which X̂, which is equivalent to maximize the probabil- 360 the data are generated. In that sense, we know that ity distribution of the data p✓ (X) in terms of ✓. 5 Generated True Underlying traffic distribution distribution characteristics pgt (X) Generative model (neural net) Latent space Traffic data space Figure 2: The generative model. The goal is to learn a parametric model capable of producing M new traffic samples {x(j) }M j=1 , x (j) ⇠ p (X) ⇡ p (X). That is, let the model learn the ground-truth distribution of the data p (X) from a ✓ gt gt random variable z with a simple distribution that captures the underlying characteristics of the traffic in the given road traffic network. Black dots are the observed traffic samples x(i) . Because we assumed that x is generated by a ran- dom process involving z, this could be solved by Recognition Generative integrating over the joint probability distribution model model Z (encoder) (decoder) q (z|x) z p✓ (x|z) p✓ (x) = p✓ (x|z) p(z) dz (2) x x̂ Stochastic PN Traffic sample representations Reconstruction while maximizing log p✓ (X) = i=1 log p✓ (x(i) ), a maximum likelihood problem. Note that by max- Figure 3: The VAE framework adds a recognition model 380 imizing (2) we hope to discover a meaningful rep- q to approximate the intractable true posterior distribution p(z|x). This can be optimized via stochastic gradient de- resentation for the traffic data x in terms of latent scent (SGD) as the input and output of the model should features given by p(z|x). Unfortunately, the inte- be the same. The model might not learn the exact data gral of the marginal likelihood requires computing distribution pgt (X) but an approximation, (3). the intractable true posterior or sampling-based so- 385 lutions, which are too costly [19, 20]. To circum- vent this, (1) is treated as an optimization problem where the first term is the expected reconstruction adding a recognition model q to approximate the in- error and the second term is the Kullback-Leibler tractable true posterior p(z|x). Consequently, the 400 (KL) divergence of the approximate posterior from whole data model may be viewed as consisting of the prior. 390 two parts (Fig. 3): 3.3. Model implementation – the generative model p✓ (x, z) = p(z) p✓ (x|z), The aforementioned model is implemented using which produces a distribution over the possible VAE [19, 20]: the recognition model using a neural corresponding values of x given z and 405 network encoder with weights and biases and the – the added recognition model q (z|x) parame- generative model using a neural network decoder 395 terized with , that given a traffic sample x, with weights and biases ✓, Fig. 3. We model the produces a distribution over the possible values encoder and decoder using a multilayer perceptron of z from which x could have been generated. (MLP), considering that during this work we will be 410 experimenting with speed and flow data separately. From there, a variational lower bound on the However, thanks to the versatility and continuous marginal likelihood can be derived which does not development of neural networks, any architecture depend on the intractable p✓ (z|x) or p✓ (x). This could be used as part of the encoder/decoder and yields the known ELBO function [19, 20], which can even stack several traffic variables as input to the be optimized in terms of and ✓ at the same time 415 network. For example, [14] treated x as an image to minimizing exploit the spatial and temporal correlation infor- ⇥ ⇤ mation between road network’s points using CNN. L(✓, ) = Ez⇠q (z|x) log p✓ (x|z) + DKL q (z|x) || p(z) (3) Encoder. We let the encoder model a multivari-  log p✓ (X) , ate Gaussian with a diagonal covariance struc- 6 ture, q (z|x) = N (z|µz , diag( 2z )). The encoder ||x x̂|| 2 is implemented using a 1 hidden layer MLP with = {W1 , W2 , W3 , b1 , b2 , b3 }, whose outputs are the x̂ mean µz and s.d. z of q (z|x) (called the code or latent representation of the data): Decoder ✓ henc = LReLU (W 1 x + b1 ) 2 µz + ✏ ⌘z µz = W 2 henc + b2 DKL N (µz , diag( z )) || N (0, I) z (4) z = ReLU (W 3 henc + b3 ) + b , + where REctified Linear activation Unit (ReLU ) and Leaky ReLU (LReLU ) are nonlinear activa- µz z ✏ ⇠ N (0, I) 420 tion functions and z is filtered through a ReLU while lower bounded (b = 1e 5 ) to help for numer- henc Encoder ical stability during training, since the covariance is positive definite. x Figure 4: Feed forward pass of the network using the Decoder. Likewise, we let the decoder model a reparametrization trick for training via SGD. Red and pur- multivariate Gaussian, p✓ (x|z) = N (x|µx , I). The ple show the loss and non linear activation layers, respec- decoder is implemented using a 1 hidden layer MLP tively. ✓ and are updated on the backward pass with the with ✓ = {W4 , W5 , b4 , b5 }, whose output is defined backpropagation of the error. by analytically derived [19] resulting in hdec = LReLU (W 4 z + b4 ) (5) J x̂ = µx = W 5 hdec + b5 , 1X 2 DKL q (z|x) || p(z) = (1+log j µj 2 j 2 ), 2 j=1 where its input are codes sampled from the pos- (6) 425 terior z ⇠ q (z|x) and x̂ is computed using only where J is the dimension of z and j indicates each µx . Note that during training via SGD we can- component of the encoder moments µz and z . In not directly propagate gradients w.r.t. through addition, we estimate the expectation of the recon- the sampling operator. Thus, we make use of the struction error in (3) using a single sample from reparametrization trick in [19] as an equivalent sam- q (z|x), as it is enough in practice when using 430 pling procedure that avoids derivation of z, Fig. 4. SGD. We minimize the l2 norm between x and x̂ Now, we let the prior over the latent variables be analogously to maximize log p✓ (x|z), since the vari- the centered isotropic multivariate Gaussian p(z) = ance for the recognition model is fixed to 1 and N (z|0, I). This jointly with the KL term in (3) al- speed and flow data are real-valued. Hence, the lows for a continuous latent space and assumes that objective function (3) to minimize becomes 435 latent representations of samples are iid. Notice J that any distribution in d dimensions can be gener- 2 1X 2 ated by taking a set of d variables that are normally L(✓, ) = ||x x̂||2 (1+log j µj 2 j 2 ), 2 j=1 distributed and mapping them through non-linear functions. Thus, the model can learn to generate (7) 440 any distribution of traffic data from the Gaussian 445 where the second term (6) is a regularization or assumption on the prior. In addition, this may lead penalty term during training in an autoencoder into learning more disentangled features because z sense. components are orthogonal, which may help traffic modelers to explore and interpret the latent space. 3.4. Model optimization As both the prior and the approximated poste- Regarding the encoder/decoder architecture, [57] rior are Gaussian, the KL divergence in (3) can be 450 stated that simple decoders like a conditional uni- 7 modal Gaussian decoder (5) typically results in rep- yielded to better results, despite not optimizing the resentations that are good at capturing the global proper lower bound during the early stages of train- structure but fail at capturing more complex local 505 ing. structure. To address this, autoregressive models Finally, since road traffic networks evolve with 455 (e.g., PixelRNN [58] or PixelCNN [59]) may be in- time, the reconstruction error of new traffic sam- tegrated with VAE and used as encoder/decoder ples can be used as an indicator of when to ad- like in the Variational Lossy Autoencoder [60], just the model to new data. A high reconstruc- or use the Channel-Recurrent Variational Autoen- 510 tion error would mean that samples reconstructed coder [61] that uses recurrent connections across conform to a di↵erent data distribution than the 460 CNN channels to circumvent the simplification of already learned by the model. VAE’s latent space. These models may be a power- ful tool for traffic forecasting [62] because they are 3.5. Missing data imputation good at capturing local statistics [63]. However, The first implication of Sec. 3.3 model is that even that the architecture presented in Sec. 3.3 is 515 new unobserved traffic samples with missing values 465 not that complex, we show in the experimentation can be reconstructed from the learned p✓ (x|z). A section that is enough to solve general road traffic corrupted data sample x̃ can be reconstructed once forecast problems with the data dealt in this work. the whole network is trained on historical data min- This is aligned with recent findings suggesting that imizing (7). The imputation procedure depicted in Gaussian encoder/decoder assumptions do not re- 520 Fig. 5 consists of: 470 duce the e↵ectiveness of VAEs [64]. Regarding the definition of p(z), we let p(z) = – random initialize the missing values, N (z|0, I) for our purpose, which has the following computational and implementation benefits: (i) the – sample from the recognition model, i.e., encode samples of z can be drawn from a simple distribu- x̃ sampling from z ⇠ N (µz , z ) where µz and 475 tion, (ii) it forces a continuous latent space and (iii) z are given by the encoder (4) and the KL divergence is given in closed form. However, by doing so, we assumed that latent representations 525 – sample from the generative model, i.e., map of samples are iid, which for many datasets, such as back the resulting z to the data space using de- time-series of images, can be a strong assumption coder (5) to obtain a reconstructed data sam- 480 [65]. ple x̂. Nevertheless, increasing the complexity of the This procedure can be iterated until convergence, model leads to several issues identified in litera- 530 simulating a Markov chain that has been shown in ture [18]. Therefore, the model should match the [20] that converges to the true marginal distribu- complexity of the problem and data because this tion of missing values given observed values. In 485 substantially increases the complexity of model im- practice, a more straightforward method is to sam- plementation, training and tuning. Ladder Varia- ple only using the mean, i.e., z = µz , which leads tional Autoencoder (LVAE) [66] were proposed to 535 to similar results. Note that (6) forces the model train deeper architectures for more representational to be able to decode plausible traffic samples from power, but it is known that a VAE with powerful de- every point in the latent space that has a reason- 490 coding capabilities tend to ignore latent space and able probability under the prior. On the contrary, use only the decoding distribution to represent the an autoencoder without the latent variable model entire data set [67]. We found that in most cases 540 would have learned a latent space which may not a straightforward implementation ignored the la- be continuous or allow interpolation. tent space, that is, q (z|x) was learned by setting 495 q (z|x) ⇠ p(z) thus bringing the KL term close to zero (the posterior collapse problem [68]). Note 3.6. Dimension reduction that a model that encodes useful information in The second implication is that the learned latent the latent variable z will have a non-zero KL di- space can be exploited in several di↵erent ways that vergence term. To prevent that, we modified the 545 are of interest to traffic forecasting systems. The 500 training objective (7) by weighting the second term latent space defined by z is forced to capture use- with 2 [0, 1] and increasing its value on each ful information about the data because z is limited epoch during training. This annealing strategy [30] to having a dimension smaller than x. Therefore, 8 Random Projection initialize to latent space missing values µz z Encoder z ⇠ N (µz , z) Decoder z Iterate for convergence Figure 5: Reconstruction of a corrupted sample using the imputation procedure of Sec. 3.5 on PeMS data. y-axis shows 3 hours of 5-minute samples. x-axis represents 31 sensors. The colored variable is the traffic speed [km/h]. VAE is learning the principal components or inde- learn meaningful (humanly interpretable) and dis- 550 pendent factors of the highly non-linear latent man- entangled latent dimensions with VAE [72, 73]. ifold of the given traffic data set. Recall that a lin- ear autoencoder minimizing the mean squared error 3.7. Model selection (MSE) learns to span the same subspace as PCA. 590 The latent space can be exploited as a tool for This can be exploited as an unsupervised dimension the selection of models and data, since similar data 555 reduction or feature extraction independent layer is encoded closer in the latent space. Traffic sam- for traffic forecasting systems. On the one hand, ples are clustered in an unsupervised manner in the the data can be compressed using the encoder to latent space learned by VAE. This can be used to store and reconstruct them when necessary using 595 distinguish between work days, weekends, holidays, the decoder. To aim for the lowest possible dimen- anomalous days, etc. or to compare the traffic from 560 sion of z (i.e., maximum compression) one may find di↵erent road traffic networks and time periods. it empirically or rely on the recently published ar- This explanatory power makes the model adapt- ticle [69], which inspects the mutual information able and responsive to dynamic traffic and road evolution between layers. On the other hand, fea- 600 environment changes over time. Traffic modelers tures learned may be used by a regression layer to may use the tool as an indicator of model perfor- 565 improve traffic estimation as the compressed infor- mance against new data, thus, the need to train mation filters out useless information and allows a new model, or to gain deeper knowledge of the data-driven models to easily learn. In this case, traffic behavior by exploring the latent space. In the performance is less conditioned to the dimen- 605 that way, accuracy of traffic forecasting systems can sions of z since in practice we have obtained similar be enhanced by splitting the data into the classes 570 results for di↵erent latent space dimensions, except learned by the model and fitting a separate model to with very small or very large dimensions (the reader each class [50]. This can be done by projecting the is referred to [70] for a theoretical explanation of new data into the learned subspace and comparing that). Finally, the whole procedure consists of: 610 it with new data using clustering algorithms [53]. – pre-train the model to reconstruct its input in Further, modelers can visually search for correla- 575 an unsupervised manner tions and seasonality by using visualization tech- niques of high-dimension data sets such as PCA or – use the pre-trained encoder as input to a re- t-SNE [74], as we show in the experimentation sec- gression model for supervised traffic estimation 615 tion. – fine-tune the entire network if the regression 3.8. Anomaly detection model is a DNN (we found that yields better 580 results than fixing the weights and biases of the The anomaly detection with VAE can be done encoder), if not, supervise train with the latent online and o✏ine. A simple but powerful approach representations. is to visually compare projected samples in latent 620 space, which may be useful for traffic modelers. For It is worth mentioning that, unlike SAE [10, 43], example, by projecting the samples in the latent stacking several VAE in our framework would not space using PCA and displaying them colored by 585 lead to an enhanced representation power [71]. type of day, the modeler can see if a Tuesday sample Also, currently, researchers have put an e↵ort to deviates significantly from his cluster, which may 9 625 mean that an anomaly is occurring or that it is a perform forecast on complex networks [14]. Dur- holiday if it’s closer to Sunday’s cluster. 675 ing experimentation all code was written in Python On the other hand, a more interesting scenario with the help of Tensorflow library for DNN cod- for ITS is to detect anomalies automatically. For ing. All testing was performed with an Intel Xeon statistical methods, key statistics are used when W-2123 + 4 ⇥ NVIDIA GeForce RTX 2080 Ti on 630 anomalies are detected if the statistic exceeds a cer- an Ubuntu server. tain threshold value. If anomalies are labeled, one may project a sample to the latent space, compute 680 4.1. Data description the Euclidean distance of the sample to its class cen- troid and then establish the threshold, for example PeMS. We collected data from 31 loop detectors 635 by means of the AUROC. If anomalies are not la- installed on a south-bound section of Interstate 5 (I- beled, one may assume that clusters are Gaussian 5), Fig. 6b. Traffic data are available from the free- distributed and set the threshold proportional to way Performance Measurement System (PeMS) of the s.d. or use kernel density estimation setting a 685 the California Department of Transportation (Cal- minimum probability threshold. Nevertheless, VAE trans)1 that has been widely used in traffic forecast- 640 inherently provides the two typically steps of statis- ing literature. Detectors used span spaced equally tical anomaly detection techniques: dimension re- apart 82 km of the highway in San Diego County, duction and an statistical anomaly criterion. VAE concretely from post mile (PM) 1.1 to 52.3. Each provides a probability measure with the KL diver- 690 detector reports the speed, occupancy and flow, gence term in (7) rather than a reconstruction er- which are aggregated into 5-minute intervals includ- 645 ror as an anomaly score function. Probabilities ing a reliable measure of data quality showing the are more objective than reconstruction errors and percent of observed samples. Incorrect values are do not require model specific thresholds for judg- filtered out, while missing samples are imputed us- ing anomalies [56]. When the VAE is trained with 695 ing linear regression [75]. Data collected covers the far more normal samples than anomalous ones, the two-year period from 2015 until 2017. 650 VAE learns to model the distribution of normal traffic data, thus a traffic sample can be detected as UKM1. We gathered traffic speed and flow data2 anomalous if it statistically deviates from what the from 19 junctions (J27 to J1) of the English M1 sec- model has learned [54]. This particularly suits the tion from Nottingham to London, Fig. 6a. The M1 traffic domain because traffic data sets are usually 700 is a major motorway of the Strategic Road Network 655 imbalanced, samples are only labeled by days and (SRN) which runs between London to Leeds in the most of the anomalies are still unseen. We have United Kingdom. The data are averaged between explored some of this suggestions under Sec. 7 and junctions and aggregated into 15-minute intervals. left the others as future work. Junctions span 210 km and consist of di↵erent road 705 lengths. Speeds are estimated using a combination of sources, including automatic number plate recog- 4. Real-world traffic data sets nition (ANPR) cameras, in-vehicle global position- ing systems (GPS) and inductive loops built into 660 We gathered and cleaned three di↵erent kind of the road surface. Data collected covers the four- real-world data that are described in this section to 710 year period from 2011 until 2015. later evaluate the proposed methodology, Fig. 6. It should be noted that there is a lack of bench- UKM4. We gathered from the same source the mark data sets in traffic forecasting literature that traffic speed and flow data from 19 junctions (J22 665 has been identified as a problem to compare dif- to J2) of the English M4 section from Bristol to ferent proposals [1]. The three datasets come from London, Fig. 6a. The SRN’s M4 motorway con- highways, however, please keep in mind that the 715 nects London to South Wales. Similarly to UKM1, method can also be applied to any road traffic net- junctions span 180 km and consist of di↵erent road work and urban areas. Its success depends on the lengths. Data collected goes from 2011 until 2015. 670 encoder/decoder architecture chosen to mine the existing relations between sensors. In that sense, literature has shown that neural networks are ca- 1 http://pems.dot.ca.gov pable of mining spatio-temporal characteristics to 2 https://data.gov.uk 10 Nottingham Oceanside San Diego Bristol London Tijuana (a) M1 and M4 highways. (b) Interstate 5 highway. Figure 6: Approximate location of the traffic sensors in (a) England and (b) California. 5. Experimentation: Imputation method In this section, we evaluated our proposal as an 720 imputation method by using a defined set of syn- thetically generated missing data while determin- ing to what extent an improvement on the imputed values yields an enhanced accuracy of the subse- quent traffic forecast model. Because imputation 725 requirements may vary depending on the final ap- plication [23], we evaluated the final performance of the whole traffic forecasting system instead of mea- Figure 7: Distribution over the PeMS data set of the induced suring the distance between the real data and its missing values (white fields). Each column shows two years of 5-minute samples of mean speed data corresponding to reconstruction. There are cases in which improving each detector. 730 the data imputation does not necessarily mean that prediction will improve, e.g., when there is sufficient information in the observed data for the traffic fore- In other words, averaged values with more than casting system to estimate. The reader may think 25% of invalid samples were treated as missing val- on the increase in root mean squared error (RMSE) ues. The data distribution of Fig. 7 resulted from 735 when the reconstruction is the same as the original said assumption in which 11.28% of the speed data but shifted by one value. 755 is missing. Fig.7 shows a Not Missing at Random (NMAR) pattern where consecutive missing values 5.1. Inducing missingness are found in not so random time instants and sen- Although the original PeMS data contained miss- sors. This is consistent with real-world missingness ing values, we could not directly use those for types analyzed in literature [26, 29, 76]. 740 evaluation as their values were previously imputed 760 Additionally, we investigated the robustness of by PeMS [75]. Instead, we considered the PeMS the system against higher shares of missing val- data quality measure and produced artificial miss- ues by removing additional observations from the ing data on the test data, that we will refer to data following a Missing Completely at Random as PeMS-NMAR. All 5-min data samples available (MCAR) pattern. There might be cases where the 745 on PeMS are attached to a data quality measure. 765 improvement in imputation accuracy is large but Those samples are the average of 30-sec samples in the improvements in estimation accuracy may not 5 minutes and the quality measure shows the per- be significant. This might be because there is suffi- centage of valid samples during that time. In this cient information in the observed data for the traffic section, we considered all 5-minute samples that did forecasting system to estimate. Thus, the predic- 750 not meet a 75% quality measure as missing values. 770 tion accuracy was evaluated for 10%, 20% and 40% 11 missing data proportion on the test data which we 820 as the estimation. On the original test data with- will refer as PeMS-MCAR-(%). out missing values, RL showed a 34.8% and 25.3% improvement on RMSE and MAPE, respectively, 5.2. Evaluation task which was considered as a benchmark and enough Fig. 8 shows the scenario considered divided for the evaluation purpose. 775 into two parts: an imputation layer that prepro- cesses corrupt speed traffic samples that are then 825 5.3. Performance and discussion fed separately to a regression layer to estimate fu- The proposed VAE implementation showed an ture traffic speed. In interest of faster training, we RMSE improvement of 69.6%, 52.6% and 39.5% set the RL to estimate 1 hour ahead (h = 12) traf- over RL, PCA and AE on NMAR test speed data, 780 fic speed of sensor number 15 (m = 1), the one respectively. Likewise, VAE showed superior per- presenting less corrupted data (0.07%). The last 3 830 formance for each di↵erent missing value propor- hours of speed samples were used (n = 36) as input tion on MCAR. For example, on MCAR–40, VAE (x 2 R36⇥31 ). Evaluation was done on all possible showed an RMSE improvement of 54.9%, 18.7% 3-hour speed traffic samples of PeMS-NMAR and and 17.3% over RL, PCA and AE, respectively. The 785 PeMS-MCAR from 2016 (105360 samples) while main di↵erence between VAE and AE is that a reg- the rest was used for training (105072 samples). 835 ularizing term on the objective function is imposed Each experiment was conducted 10 times and we on the former to force the model to learn a con- reported the mean of RMSE and mean absolute tinuous latent space. Results indicate that learn- percentage error (MAPE) of the prediction task as ing the p✓ (X) helps to infer missing data as the 790 the performance metrics in Table 1. model is able to decode plausible unseen data sam- 840 ples from every point in the latent space that has a Imputation layer (IL). We compared our pro- reasonable probability under the prior, which val- posal (VAE) against a 1-hidden layer non-linear idates our initial assumption. We also found that autoencoder (AE) and PCA. Details of VAE im- non-linearity helps to impute missing values when plementation are found in Sec. 3.3 and Fig. 8. larger gaps of missing data are found (NMAR pat- 795 We set z = µz for simplicity. The VAE model 845 tern). Looking at the VAE and AE performance had 1,298,724 trainable parameters and the aver- against PCA in Table 1 on NMAR data, the linear age time to convergence was 22 minutes using one model performs poorly. However, no relevant di↵er- GPU. Regarding the AE, ReLU was used for each ences were found between PCA and AE on MCAR. layer except for the output. We trained both au- In this case, the PCA performs similarly to AE be- 800 toencoders with a batch size of 128 using a random 850 cause of the MCAR pattern which implies less con- validation split of 10% for earlystopping and 512 secutive missing values and the linear model can neurons per layer. The latent space dimension was perform better. Another interesting finding is that first arbitrary set to 100. We used Adam [77] op- VAE performed better in NMAR than MCAR–10 timizer with a learning rate of 5e 5 . Input was even when the missing data proportion of the for- 805 normalized to zero mean and unit variance and all 855 mer is greater, which makes the proposed method missing values were treated as zero prior to each more suitable for real-world data set where mostly imputation method for fair comparison. NMAR patterns are found. We also varied the la- tent space dimension and provided some results on Regression layer (RL). A 2-hidden layer MLP Table 1, where the compression factors applied on was trained where each layer was composed of 100 860 the data are shown between parenthesis near each 810 neurons with sigmoids activations. However, note IL method. Results showed that accuracy increased that it could be another type of model, not nec- jointly with the compression factor but to a certain essarily a DNN. l2 regularization was used on the extent. Constraining the latent space dimensions weights to prevent overfitting. Input was normal- forces the network to learn better features until the ized to zero mean and unit variance. The MSE was 865 space becomes small enough. Same thing happened 815 minimized using SGD with default Adam. We use while increasing the dimensions. This suggested a 10% random split of the training set as the valida- the existence of a lower and higher bound where tion set for early stopping of the training procedure. only an insignificant improvement can be observed, The RL showed better performance compared to a which led us to conclude that the optimal latent naive approach, where the last input sample is used 870 space dimension should be empirically defined as a 12 µz x̃ x̂ x̂ enc dec h h Hidden cell ... L z L S S R Hidden cell + ReLU L L S S y 100 L Hidden cell + LeakyReLU L L S S ... z Hidden cell + Sigmoid ... ... ... ... S R ... ... ... L L S S Input cell ... 100 512 R 512 100 100 Output cell 1116 100 1116 1116 Gaussian sampling cell Imputation Layer Regression Layer Figure 8: The production traffic forecasting system evaluated in Sec. 5. First, the imputation layer imputes the missing values of the 1116 dimension input. The VAE approach is showed following the imputation procedure of Sec. 3.5 using z = µz . Then, an independent regression layer estimates the future traffic speed of one sensor using the reconstructed sample. Three di↵erent imputation layers based on VAE, AE and PCA were evaluated computing the RMSE and MAPE of the prediction task. Table 1: RMSE [km/h] — MAPE [%] results on the estimation of 1 hour ahead traffic speed of sensor number 15 over the PeMS test speed data. The first row shows the performance of the RL alone and should be compared to the results when an IL is added, i.e., IL + RL. The compression factor value is shown between parenthesis near each imputation method which was computed as the ratio between the dimensions of the data space and the latent space. MCAR–(%) indicates the proportion of generated missing data. In bold are the results closer to the performance of the RL on the Original data containing no missing values (the closer the better). Original NMAR MCAR–10 MCAR–20 MCAR–40 RL 5.53 — 3.04 19.37 — 13.50 27.24 — 20.05 30.07 — 22.75 33.28 — 26.20 PCA (11.16) + RL N/A 12.42 — 7.82 10.68 — 6.79 14.35 — 9.40 18.46 — 12.84 AE (11.16) + RL N/A 9.74 — 5.69 10.69 — 6.91 14.02 — 9.46 18.16 — 12.92 VAE (11.16) + RL N/A 5.89 — 3.23 8.98 — 5.52 11.79 — 7.46 15.01 — 9.78 VAE (22.32) + RL N/A 8.70 — 5.27 8.58 — 5.28 10.61 — 6.64 11.98 — 7.70 VAE (111.6) + RL N/A 7.71 — 4.53 7.86 — 4.58 8.57 — 5.03 9.18 — 5.38 hyperparameter or by means of mutual information 100 features from a 4464 and 1368 input data space between layers like recently suggested in [69]. depending on the data set. 890 Feature extraction layer (FL). We compared 6. Experimentation: Dimension reduction our proposal (VAE) against an autoencoder (AE) and PCA similarly to Sec. 5. The implemented Under this section, we experimented with our VAE is shown in Fig. 9 and described in Sec. 3.6. 875 proposal as a data compression tool. Our goal was Training was done using KL cost annealing to avoid to explore if the subspace learned by VAE results 895 the posterior collapse problem. We set the initial in representative and powerful features of the traf- weight of KL cost term to be zero and increased it fic data that can be used to perform traffic fore- at each training step k as k = 0.0001 ⇥ 1.05k 1 , cast. To that aim, we set a more complex problem 2 [0, 1]. Gradients were clipped by their l2 norm 880 and we aimed to estimate 1 hour ahead speed of to 0.5 for training stability. The VAE models had all the network sensors using the last 12 hours of 900 37,810,744/12,445,216 trainable parameters for the data of PeMS and the last 18 hours of UKM1 and 4464/1368 input and the average time to conver- UKM4. Similarly to Sec. 5, we considered a fea- gence was 37/25 minutes using one GPU. Regard- ture extraction layer and a regression layer but, in ing the AE, the hidden layers were composed of 885 this case, models were evaluated on PeMS, UKM1 4096 neurons followed by ReLU activations. We and UKM4 test data. The latent space dimension 905 trained both using Adam(5e 5 ) with a batch size was set to 100, thus models were forced to extract of 128 and earlystopping. Input was normalized to 13 x µz henc z y ... L z Hidden cell L 100 R Hidden cell + ReLU L ... ... ... z L Hidden cell + LeakyReLU + DO ... ... ... ... ... R L Hidden cell + ELU + BN + DO ... 100 100 31 4096 R 1024 1024 1024 Input cell 4464/1368 100 Output cell Feature Extraction Layer Regression Layer Figure 9: The production forecasting system evaluated in Sec. 6. Here, we show the encoder of an independently pre-trained VAE used as a feature extraction layer prior to long-term wide-network traffic forecast. Like in Sec. 5, we used directly µz as the feature vector. The AE and PCA approaches were also evaluated as feature extraction layers. Note that the 4464 or 1368 input dimension is reduced down to 100 features to perform the forecast. DO stands for a Dropout layer with a drop probability of 0.5 and BN for a Batch Normalization layer. zero mean and unit variance. the same time for cloud computing and storage as 940 significant compression factors are achieved. Regression layer (RL). We trained a non linear 3-hidden layer MLP with 1024 neurons per layer Our proposal is intended to be a tool, indepen- 910 plus an Exponential Linear Units (ELU ) activation dent from the model used in the prediction part. function, a batch normalization (BN) layer and a However, care must be taken in choosing the fore- dropout layer to avoid overfitting, Fig. 9. That casting model because compressed data can degrade is, 2,234,399 trainable parameters not counting BN 945 the performance of models that exploit the spatial layers. Labels were normalized to zero mean and or temporal structure of the data. Recall that we 915 unit variance. The MSE was minimized using SGD let p(z) = N (z|0, I), i.e., latent components are and Adam(1e 5 ) with a batch size of 128. The best orthogonal. Therefore, it is safe to conclude that generalization was selected as the final model using using a convolutional or recurrent approach to fore- a validation split of 10%. 950 cast using the compressed data will not lead to im- provement over MLP since the latent representa- 6.1. Performance and discussion tions of the data do not keep the temporal or spatial 920 Main results are reported in Table 2. First two structure. In that sense, we made some testing with rows show that the tuned RL improved accuracy for LSTM and CNN prediction models that confirmed all data sets. The RL performed the forecast from 955 a degradation on the accuracy performance. 4464 samples input for PeMS and 1368 for the other During training, we found that increasing the data sets, which equals to 12 and 18 hours of data number of hidden layers derived in VAE ignoring 925 respectively. The rest of the rows show the models most of the latent space. Instead, using one layer evaluated that first projected the data to a 100 di- with a higher number of neurons led to learning mension subspace which was then used as input to 960 better features for the traffic forecast task. In that train another RL with the same architecture. The sense, the KL cost annealing and the dropout layer data compression factor was 44,64 on PeMS and also proved to be useful. The former helped to 930 13,68 on UKM1 and UKM4. In Table 2, VAE out- avoid the posterior collapse problem and the later performs all the compared models. Although the to prevent overfitting of the model. Here, we spent improvement is slight, below 5% on the RMSE, it 965 the same time tuning each FL independently of the even exceeds the performance of the original RL RL and results were considered enough to validate for all the data sets despite having significantly re- the VAE as a prominent solution for dimension re- 935 duced the space dimension of the input. Therefore, duction of traffic data. However, we believe that the introduction of non-linearities and the latent there exists room for improvement on optimizing variable model of VAE is well suited to extract use- 970 the VAE model for this specific data set which is ful features to perform traffic forecasting while at beyond the scope of this manuscript, e.g., using hy- 14 Table 2: RMSE [km/h] values of 10 experiment runs of the one. Those di↵erences at the time of fitting the RL forecast are reported showing the dimension reduction results on the test data of PeMS (4464 space dimension), forecast model can influence its performance since UKM1 and UKM4 (1368 space dimension) data sets. 1010 the 2012 data may not be beneficial for predicting 2014 traffic, as pointed out by the model. Model RL input PeMS UKM1 UKM4 4464; 1368 Evaluation details. We divided the data sets into Naive R 10.83 13.06 15.24 RL R4464; 1368 7.51 11.11 10.73 weekends and weekdays. For each data set a speed forecasting model was fitted to see if the overall PCA + RL R100 7.61 11.38 10.56 1015 performance of the two separate models was an AE + RL R100 7.63 11.28 10.37 improvement over a single model for the whole VAE + RL R100 7.49 10.89 10.23 week. To that aim, we set the following prob- lem: predict 1 hour ahead of all network sensors using the last 3 hours of data. We implemented the perparameter optimization techniques. 1020 following DNN model: Dropout(0.5)–MLP(512)– LeakyReLU–MLP(512)–LeakyReLU, trained to 7. Experimentation: Model selection and minimize the MSE using l2 regularization on the anomaly detection weights, Adam and normalized inputs. Three new data sets were made from whole week (WW ), just 975 In this section, we experimented with the repre- 1025 weekdays (WD) and just weekends (WE ) for PeMS, sentational power of the VAE model and its learned UKM1 and UKM4. The whole last year of each latent space. To that aim, we trained the same data set was used for evaluation and the rest for model of Sec. 6, but using unique day samples training. We fitted the DNN model to each of the of traffic flow and speed for all of the three data new data sets, resulting in three di↵erent models: 980 sets. Then, we projected the data to the learned 1030 MLP–WW, MLP–WD and MLP–WE, respectively. subspace and analyzed it from the point of view of Finally, the RMSE performance of the MLP–WW traffic modelers with the goal to improve the predic- model was used as a benchmark and we reported tion accuracy of a traffic forecasting system. Note the RMSE improvement in % of the rest of the mod- that the model can be used as an unsupervised tool els with respect to it. 985 to learn insights about traffic data without previous knowledge of the road traffic network. 1035 Results. The results of MLP–WD and MLP–WE models are shown in Table 3. From these results, it 7.1. Model selection can be concluded that predicting the speed by using Fig. 10 shows the two principal components (PC) two separate models for weekdays and weekends in of the latent space. The pattern of flow and speed UKM1 and UKM4 shows little improvement over 990 di↵ers between weekends and weekdays, even a sep- 1040 the results of the models for the whole week. On arate cluster for Fridays can be clearly distinguished the other hand, in the case of PeMS, training a from the flow. The flow is classified similarly for the separated model only on weekend data improves the three data sets, instead, the model classifies speed RMSE by 17.7% on weekend test data. However, di↵erently for PeMS rather than for the rest of data no improvement on the WD data was found by the 995 sets. Only in UKM1, the model can cluster between 1045 MLP–WD meaning that the performance resembles speed samples from Saturday and Sunday as speed to the MLP–WW model. The latter model, which has more complex behavior than flow. In PeMS, the was trained on WW data, mainly learns how traffic weekend cluster is more separated from the week- behaves on weekdays because weekend samples are days cluster suggesting a greater di↵erence between imbalanced w.r.t weekday samples. Those results 1000 both and the possibility that two specific models for 1050 are related to the cluster separation that exhibit each cluster perform better than a global one. In the two classes in the latent space, which can be UKM4, the model also clearly identified two clus- seen in the two-dimensional visualization of Fig. 10. ters which are distinguished by di↵erent instants of More precisely, the euclidean distances in the latent time, Fig. 11. A similar trend can be slightly ap- space (R100 ) between weekday and weekend cluster 1005 preciated for UKM1. The data from 2012 and 2013 1055 centroids of PeMS, UKM1 and UKM4 are 87.3, are classified in the upper cluster, while the data for 65.6 and 62.7, respectively. PeMS ’ clusters are the the years 2011 and 2014 are classified in the lower ones that the VAE model projected more separated 15 Flow - PeMS Speed - PeMS 6 6 75 5 5 50 50 Day of the week Day of the week 4 4 25 PC2 PC2 3 0 3 0 2 2 25 50 1 1 50 0 0 50 0 50 100 50 0 50 100 PC1 PC1 (a) Explained var.: 55.8 % (b) Explained var.: 27.6 % Flow - UKM1 Speed - UKM1 6 100 6 75 5 5 50 50 Day of the week Day of the week 4 4 25 PC2 PC2 3 3 0 0 2 2 25 1 1 50 50 0 0 50 0 50 100 50 0 50 100 PC1 PC1 (c) Explained var.: 52.9 % (d) Explained var.: 29.4 % Flow - UKM4 Speed - UKM4 75 6 6 75 5 50 5 50 Day of the week Day of the week 4 25 4 25 PC2 PC2 3 3 0 0 2 2 25 25 1 1 50 50 0 0 50 0 50 50 0 50 PC1 PC1 (e) Explained var.: 49.8 % (f) Explained var.: 22.6 % Figure 10: Traffic flow and speed samples projected to the latent space and colored by day of the week (0-6: Monday to Sunday). Day samples were projected to a 100-dimension latent space learned by the VAE model in an unsupervised manner. Then, the two principal components (PC) of the projected data were plotted with the help of PCA. The cumulative explained variance of PC1 and PC2 is shown below each figure, which means that the other dimensions that are not seen still capture more traffic characteristics. Only in PeMS, holidays samples are plotted with a star marker. 16 Speed - UKM4 Flow - UKM4 1 Normalized distance to class centroid 75 3 75 50 50 25 2 25 Year PC2 PC2 0 0 1 25 25 50 0 50 0 50 0 50 50 0 50 PC1 PC1 Figure 11: 2D PCA visualization of one-day UKM4 speed Figure 12: 2D PCA visualization of one-day UKM4 flow samples projected to the latent space (R100 ) and colored by samples projected to the latent space (R100 ). Samples are year (0-3: 2011 to 2014). The clustering shows that traffic colored proportionally to the distance (normalized between behavior changed between years. 0 and 1) to their corresponding cluster centroid, which high- lights the anomalous traffic samples (darker). See Fig. 10e Table 3: RMSE improvement [%] results of MLP-WD and for comparison. MLP-WE models w.r.t. MLP-WW on the test data from all three data sets. The highlighted cells indicate the type of test data on which the model was intended to improve the performance. plotted the samples corresponding to holidays with a star marker on PeMS in Fig. 10a and 10b. Al- Data set Split type MLP–WD MLP–WE though holidays can be considered non-anomalies, 1075 it is more likely that during these the behavior of WW -7.0 -45.0 PeMS WD 0.1 -52.0 the traffic will deviate from the usual. First thing WE -26.0 17.7 that Fig. 10a shows is that the majority of the hol- iday days behave like Sundays, which confirms a WW -1.6 -17.6 common and known fact of most road networks. UKM1 WD 1.0 -21.3 WE -7.1 2.2 1080 In Fig. 10a, a few of the samples are projected in the middle between the workday and weekend WW 0.5 -20.9 UKM4 WD 3.8 -26.0 clusters, thus we inspected more closely those sam- WE -6.9 3.0 ples because we did not have anomalous traffic la- beled. We visually compared the Monday and Sun- 1085 day samples closer to their centroid against the holi- apart, that is, that were considered more dissimilar. day sample (Monday) placed between both clusters This validates the latent space as an indicator of in Fig. 10a. This simplifies the analysis because 1060 the performance of separated models for di↵erent the three data points compared vary greatly along classes of data. Therefore, the VAE model can be the x-axis (PC1), while the variability in the y- used by traffic modelers as a tool to decide when 1090 axis (PC2) is much smaller. In this case, PC1 and it is best to make use of di↵erent models instead of PC2 represented the 55.8% of the variance of the one unique model to predict traffic. 100 features learned by VAE. The anomaly is that the targeted sample does not behave like a Mon- day or Sunday which should be expected because 1065 7.2. Anomaly detection 1095 the sample is labeled as a holiday Monday. To un- There are two interesting scenarios for anomaly derstand what caused the anomaly, we investigated detection in traffic forecasting: o✏ine and online. the latent space by not varying PC2 and compar- O✏ine anomaly detection is helpful for traffic mod- ing the three mentioned traffic samples which pro- elers to analyze historic data. Training a VAE duced a variation only on PC1. Upon investiga- 1070 model with unique day samples leads to Fig. 10-like 1100 tion, we found an increase of traffic flow around images which can be used to detect anomalies. We sensor 9 for all three samples, which means that 17 PC2 is modeling where the traffic intensity is lo- to automatically detect anomalies in terms of im- cated in the road traffic network. Contrary, the 1155 plementation cost and accuracy will likely depend main di↵erence was the intensity of traffic flow and on the data under consideration and needs further 1105 the peak hours. The intensity decreased propor- exploration, hence, will be left as future work. tionally from the Monday sample to Sunday while a light peak hour moved from morning to the af- 8. Conclusion ternoon, meaning that PC1 component is modeling those traffic features. Therefore, we can conclude In this paper, we proposed a transversal solution 1110 that the anomaly was the intensity of the flow and 1160 for road traffic forecasting systems based on the as- when happened, not where it was located. We are sumption that traffic data can be generated from not able to justify this behavior as we do not have a manifold with reduced dimensions. We formu- the data labeled. However, this anomaly may be lated the forecasting problem as a latent variable explained by the e↵ect of non-traffic features (e.g., model and proposed the variational autoencoder 1115 weather conditions, unusual events, etc.). That 1165 (VAE) as a method to unsupervisely learn an ap- said, traffic modelers may consider the holiday sam- proximation of the probability distribution of the ple as an anomaly and plan accordingly to absorb traffic data. This was evaluated on three di↵erent the specific traffic intensity at noon on that holiday. real-world traffic data sets addressing some current Furthermore, a traffic modeler could answer ques- major challenges of traffic forecasting and obtain- 1120 tions such as what the holiday would have been like 1170 ing relevant results. First, the proposed model was on a Wednesday? by exploring the latent space, used as an imputation method, showing significant because the model learned meaningful dimensions improvements on unseen traffic samples with miss- plus it is a generative model with a continuous la- ing values. Secondly, we showed that the model tent space. can learn useful features for traffic forecasting sys- 1125 On the other hand, one could elaborate more 1175 tems, allowing for dimension reduction of traffic complex analytical methods for automatic anomaly data without loss in accuracy on the forecast. In detection, instead of inspecting it visually. For ex- fact, a regression model trained with data com- ample, Fig. 12 shows the same samples of Fig. 10e, pressed by a factor of 44,6 using the VAE exceeded but colored proportionally to the Euclidean dis- the performance of the same model trained with 1130 tance to their respective class centroids. A quick vi- 1180 the raw data. Third, we exploited the learned la- sual comparison shows that all holidays and anoma- tent space from the point of view of traffic model- lies are distinguished in darker color without previ- ers in order to improve traffic forecasting systems. ous knowledge or labeled data, validating the via- Without previous knowledge of the road traffic net- bility of the approach. In cases where the model can works, projecting the data in said space allowed us 1135 clearly cluster data (e.g., the flow from UKM1;4 ), 1185 to hypothesize and gain insights about the traffic a threshold should be defined by means of the AU- data that were later validated with experimenta- ROC for labeled anomaly data or by assuming that tion, concluding that our proposal can be used as clusters are Gaussian distributed and setting the a tool for model and data selection and anomaly threshold proportional to the s.d., for unlabeled detection. 1140 data. Similarly, in [53] dimensionality reduction is performed by PCA and then kNN outlier detection 1190 Acknowledgment is applied. Nevertheless, note that same method applied to UKM4 ’s speed, Fig. 10f, would not lead This research is supported by the Catalan to such clear results (at least in 2D) due to the Government under Project 2017 SGR 1670 and 1145 topology of the clusters and the higher complex- the Spanish Government under Project TEC2017- ity of speed data. Instead of using the Euclidean 84321-C4-4-R co-funded with European Union distance, a metric can be derived from (7) that pro- 1195 ERDF funds. vides an anomaly score function in terms of proba- bilities [56]. The intuition behind it is that anoma- References 1150 lous data will have higher reconstruction errors be- [1] I. Lana, J. Del Ser, M. Velez, E. I. Vlahogianni, Road cause VAE is trained with far more normal samples traffic forecasting: Recent advances and new chal- than anomalous ones, that is, VAE learns to model lenges, IEEE Intelligent Transportation Systems Mag- normal traffic data. Finally, the best methodology 1200 azine 10 (2) (2018) 93–109. 18 [2] E. I. Vlahogianni, M. G. Karlaftis, J. C. Golias, Short- 777–785. term traffic forecasting: Where we are and where were [18] C. Zhang, J. Butepage, H. Kjellstrom, S. Mandt, Ad- going, Transportation Research Part C: Emerging Tech- vances in variational inference, IEEE transactions on nologies 43 (2014) 3–19. pattern analysis and machine intelligence. 1205 [3] E. I. Vlahogianni, M. G. Karlaftis, J. C. Golias, Sta- 1270 [19] D. P. Kingma, M. Welling, Auto-encoding variational tistical methods for detecting nonlinearity and non- bayes, arXiv preprint arXiv:1312.6114. stationarity in univariate short-term time-series of traf- [20] D. J. Rezende, S. Mohamed, D. Wierstra, Stochas- fic volume, Transportation Research Part C: Emerging tic backpropagation and approximate inference in deep Technologies 14 (5) (2006) 351–367. generative models, arXiv preprint arXiv:1401.4082. 1210 [4] Y. Li, R. Yu, C. Shahabi, Y. Liu, Di↵usion convo- 1275 [21] T. Pamula, Impact of data loss for prediction of traf- lutional recurrent neural network: Data-driven traffic fic flow on an urban road using neural networks, IEEE forecasting. Transactions on Intelligent Transportation Systems. [5] M. S. Ahmed, A. R. Cook, Analysis of freeway traffic [22] Y. Li, Z. Li, L. Li, Missing traffic data: comparison time-series data by using Box-Jenkins techniques, no. of imputation methods, IET Intelligent Transport Sys- 1215 722, 1979. 1280 tems 8 (1) (2014) 51–57. [6] M. Lippi, M. Bertini, P. Frasconi, Short-term traffic [23] I. Laña, I. I. Olabarrieta, M. Vélez, J. Del Ser, On the flow forecasting: An experimental comparison of time- imputation of missing data for road traffic forecasting: series analysis and supervised learning, IEEE Trans- New insights and novel techniques, Transportation re- actions on Intelligent Transportation Systems 14 (2) search part C: emerging technologies 90 (2018) 18–33. 1220 (2013) 871–882. 1285 [24] L. Li, J. Zhang, Y. Wang, B. Ran, Missing value im- [7] F. G. Habtemichael, M. Cetin, Short-term traffic flow putation for traffic-related time series data based on a rate forecasting based on identifying similar traffic pat- multi-view learning method, IEEE Transactions on In- terns, Transportation Research Part C: Emerging Tech- telligent Transportation Systems. nologies 66 (2016) 61–78. [25] X. Chen, Z. He, L. Sun, A bayesian tensor decompo- 1225 [8] Y.-S. Jeong, Y.-J. Byon, M. M. Castro-Neto, S. M. 1290 sition approach for spatiotemporal traffic data imputa- Easa, Supervised weighting-online learning algorithm tion, Transportation Research Part C: Emerging Tech- for short-term traffic flow prediction, IEEE Trans- nologies 98 (2019) 73–84. actions on Intelligent Transportation Systems 14 (4) [26] R. J. Little, D. B. Rubin, Statistical analysis with miss- (2013) 1700–1707. ing data, Vol. 333, John Wiley & Sons, 2014. 1230 [9] R. Garcı́a-Ródenas, M. L. López-Garcı́a, M. T. 1295 [27] S. v. Buuren, K. Groothuis-Oudshoorn, mice: Multi- Sánchez-Rico, An approach to dynamical classification variate imputation by chained equations in r, Journal of daily traffic patterns, Computer-Aided Civil and In- of statistical software (2010) 1–68. frastructure Engineering 32 (3) (2017) 191–212. [28] H.-F. Yu, N. Rao, I. S. Dhillon, Temporal regularized [10] Y. Lv, Y. Duan, W. Kang, Z. Li, F.-Y. Wang, et al., matrix factorization for high-dimensional time series 1235 Traffic flow prediction with big data: A deep learning 1300 prediction, in: Advances in neural information process- approach., IEEE Trans. Intelligent Transportation Sys- ing systems, 2016, pp. 847–855. tems 16 (2) (2015) 865–873. [29] L. Gondara, K. Wang, Multiple imputation us- [11] X. Ma, Z. Tao, Y. Wang, H. Yu, Y. Wang, Long short- ing deep denoising autoencoders, arXiv preprint term memory neural network for traffic speed predic- arXiv:1705.02737. 1240 tion using remote microwave sensor data, Transporta- 1305 [30] S. R. Bowman, L. Vilnis, O. Vinyals, A. M. Dai, tion Research Part C: Emerging Technologies 54 (2015) R. Jozefowicz, S. Bengio, Generating sentences from a 187–197. continuous space, arXiv preprint arXiv:1511.06349. [12] J. Zhang, Y. Zheng, D. Qi, Deep spatio-temporal resid- [31] M. Jang, S. Seo, P. Kang, Recurrent neural network- ual networks for citywide crowd flows prediction., in: based semantic variational autoencoder for sequence- 1245 AAAI, 2017, pp. 1655–1661. 1310 to-sequence learning, Information Sciences 490 (2019) [13] N. Laptev, J. Yosinski, L. E. Li, S. Smyl, Time-series 59–73. extreme event forecasting with neural networks at uber, [32] V. Fortuin, G. Rätsch, S. Mandt, Multivariate time in: International Conference on Machine Learning, series imputation with variational autoencoders, arXiv no. 34, 2017, pp. 1–5. preprint arXiv:1907.04155. 1250 [14] X. Ma, Z. Dai, Z. He, J. Ma, Y. Wang, Y. Wang, Learn- 1315 [33] J. Yoon, J. Jordon, M. van der Schaar, Gain: Miss- ing traffic as images: a deep convolutional neural net- ing data imputation using generative adversarial nets, work for large-scale transportation network speed pre- arXiv preprint arXiv:1806.02920. diction, Sensors 17 (4) (2017) 818. [34] C. Shang, A. Palmer, J. Sun, K.-S. Chen, J. Lu, J. Bi, [15] Z. Zhao, W. Chen, X. Wu, P. C. Chen, J. Liu, Lstm Vigan: Missing view imputation with generative adver- 1255 network: a deep learning approach for short-term traf- 1320 sarial networks, in: Big Data (Big Data), 2017 IEEE fic forecast, IET Intelligent Transport Systems 11 (2) International Conference on, IEEE, 2017, pp. 766–775. (2017) 68–75. [35] I. Tolstikhin, O. Bousquet, S. Gelly, B. Schoelkopf, [16] Q. Liu, B. Wang, Y. Zhu, Short-term traffic speed fore- Wasserstein auto-encoders, arXiv preprint casting based on attention convolutional neural network arXiv:1711.01558. 1260 for arterials, Computer-Aided Civil and Infrastructure 1325 [36] A. Brock, T. Lim, J. M. Ritchie, N. Weston, Neural Engineering 33 (11) (2018) 999–1016. photo editing with introspective adversarial networks, [17] R. Yu, Y. Li, C. Shahabi, U. Demiryurek, Y. Liu, Deep arXiv preprint arXiv:1609.07093. learning: A generic approach for extreme condition traf- [37] B. Dai, Y. Wang, J. Aston, G. Hua, D. Wipf, Connec- fic forecasting, in: Proceedings of the 2017 SIAM Inter- tions with robust pca and the role of emergent sparsity 1265 national Conference on Data Mining, SIAM, 2017, pp. 1330 in variational autoencoder models, The Journal of Ma- 19 chine Learning Research 19 (1) (2018) 1573–1614. scale traffic data, in: 2015 IEEE International Confer- [38] S. Yang, S. Qian, Understanding and predicting ence on Digital Signal Processing (DSP), IEEE, 2015, travel time with spatio-temporal features of network pp. 507–510. traffic flow, weather and incidents, arXiv preprint [54] Y. Kawachi, Y. Koizumi, N. Harada, Complementary 1335 arXiv:1901.06766. 1400 set variational autoencoder for supervised anomaly de- [39] D. Pavlyuk, Feature selection and extraction in spa- tection, in: 2018 IEEE International Conference on tiotemporal traffic forecasting: a systematic literature Acoustics, Speech and Signal Processing (ICASSP), review, European Transport Research Review 11 (1) IEEE, 2018, pp. 2366–2370. (2019) 6. [55] M. Sölch, J. Bayer, M. Ludersdorfer, P. van der 1340 [40] Q. Li, H. Jianming, Z. Yi, A flow volumes data com- 1405 Smagt, Variational inference for on-line anomaly de- pression approach for traffic network based on principal tection in high-dimensional time series, arXiv preprint component analysis, in: 2007 IEEE Intelligent Trans- arXiv:1602.07109. portation Systems Conference, IEEE, 2007, pp. 125– [56] J. An, S. Cho, Variational autoencoder based anomaly 130. detection using reconstruction probability, Special Lec- 1345 [41] N. G. Polson, V. O. Sokolov, Deep learning for short- 1410 ture on IE 2 (2015) 1–18. term traffic flow prediction, Transportation Research [57] A. B. L. Larsen, S. K. Sønderby, H. Larochelle, Part C: Emerging Technologies 79 (2017) 1–17. O. Winther, Autoencoding beyond pixels us- [42] N. Mitrovic, M. T. Asif, J. Dauwels, P. Jaillet, Low- ing a learned similarity metric, arXiv preprint dimensional models for compressed sensing and predic- arXiv:1512.09300. 1350 tion of large-scale traffic data, IEEE Transactions on 1415 [58] A. v. d. Oord, N. Kalchbrenner, K. Kavukcuoglu, Intelligent Transportation Systems 16 (5) (2015) 2949– Pixel recurrent neural networks, arXiv preprint 2954. arXiv:1601.06759. [43] H.-F. Yang, T. S. Dillon, Y.-P. P. Chen, Optimized [59] A. van den Oord, N. Kalchbrenner, L. Espeholt, structure of the traffic flow forecasting model with a O. Vinyals, A. Graves, et al., Conditional image gen- 1355 deep learning approach, IEEE transactions on neural 1420 eration with pixelcnn decoders, in: Advances in Neural networks and learning systems 28 (10) (2017) 2371– Information Processing Systems, 2016, pp. 4790–4798. 2381. [60] X. Chen, D. P. Kingma, T. Salimans, Y. Duan, [44] G. San Martin, E. López Droguett, V. Meruane, M. das P. Dhariwal, J. Schulman, I. Sutskever, P. Abbeel, Chagas Moura, Deep variational auto-encoders: A Variational lossy autoencoder, arXiv preprint 1360 promising tool for dimensionality reduction and ball 1425 arXiv:1611.02731. bearing elements fault diagnosis, Structural Health [61] W. Shang, K. Sohn, Y. Tian, Channel-recurrent autoen- Monitoring (2018) 1475921718788299. coding for image modeling, in: 2018 IEEE Winter Con- [45] D. Wang, J. Gu, Vasc: dimension reduction and visual- ference on Applications of Computer Vision (WACV), ization of single-cell rna-seq data by deep variational IEEE, 2018, pp. 1195–1204. 1365 autoencoder, Genomics, proteomics & bioinformatics 1430 [62] D. Pavlyuk, Short-term traffic forecasting using mul- 16 (5) (2018) 320–331. tivariate autoregressive models, Procedia Engineering [46] Y. Wu, H. Tan, L. Qin, B. Ran, Z. Jiang, A hybrid deep 178 (2017) 57–66. learning based traffic flow prediction method and its un- [63] G. Ostrovski, W. Dabney, R. Munos, Autoregres- derstanding, Transportation Research Part C: Emerg- sive quantile networks for generative modeling, arXiv 1370 ing Technologies 90 (2018) 166–180. 1435 preprint arXiv:1806.05575. [47] J. Van Lint, C. Van Hinsbergen, Short-term traffic [64] B. Dai, D. Wipf, Diagnosing and enhancing VAE mod- and travel time prediction models, Artificial Intelligence els, in: International Conference on Learning Represen- Applications to Critical Transportation Issues 22 (1) tations, 2019. (2012) 22–41. URL https://openreview.net/forum?id=B1e0X3C9tQ 1375 [48] J. S. Angarita-Zapata, I. Triguero, A. D. Masegosa, 1440 [65] F. P. Casale, A. Dalca, L. Saglietti, J. Listgarten, A preliminary study on automatic algorithm selec- N. Fusi, Gaussian process prior variational autoen- tion for short-term traffic forecasting, in: International coders, in: Advances in Neural Information Processing Symposium on Intelligent and Distributed Computing, Systems, 2018, pp. 10369–10380. Springer, 2018, pp. 204–214. [66] C. K. Sønderby, T. Raiko, L. Maaløe, S. K. Sønderby, 1380 [49] E. I. Vlahogianni, Optimization of traffic forecast- 1445 O. Winther, Ladder variational autoencoders, in: Ad- ing: Intelligent surrogate modeling, Transportation Re- vances in neural information processing systems, 2016, search Part C: Emerging Technologies 55 (2015) 14–23. pp. 3738–3746. [50] M. Van Der Voort, M. Dougherty, S. Watson, Com- [67] A. Alemi, B. Poole, I. Fischer, J. Dillon, R. A. Saurous, bining kohonen maps with arima time series models to K. Murphy, Fixing a broken elbo, in: International Con- 1385 forecast traffic flow, Transportation Research Part C: 1450 ference on Machine Learning, 2018, pp. 159–168. Emerging Technologies 4 (5) (1996) 307–318. [68] A. Razavi, A. v. d. Oord, B. Poole, O. Vinyals, Prevent- [51] Y. Djenouri, A. Belhadi, J. C.-W. Lin, D. Djenouri, ing posterior collapse with delta-vaes, arXiv preprint A. Cano, A survey on urban traffic anomalies detection arXiv:1901.03416. algorithms, IEEE Access. [69] S. Yu, J. C. Principe, Understanding autoencoders with 1390 [52] Y. Djenouri, A. Zimek, Outlier detection in urban traf- 1455 information theoretic concepts, Neural Networks. fic data, in: Proceedings of the 8th International Con- [70] B. Dai, D. Wipf, Diagnosing and enhancing vae models, ference on Web Intelligence, Mining and Semantics, arXiv preprint arXiv:1903.05789. ACM, 2018, p. 3. [71] S. Zhao, J. Song, S. Ermon, Infovae: Information [53] T. T. Dang, H. Y. Ngan, W. Liu, Distance-based k- maximizing variational autoencoders, arXiv preprint 1395 nearest neighbors outlier detection method in large- 1460 arXiv:1706.02262. 20 [72] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glo- rot, M. Botvinick, S. Mohamed, A. Lerchner, beta- vae: Learning basic visual concepts with a constrained variational framework, in: International Conference on 1465 Learning Representations, 2017. [73] C. P. Burgess, I. Higgins, A. Pal, L. Matthey, N. Wat- ters, G. Desjardins, A. Lerchner, Understanding disen- tangling in -vae, arXiv preprint arXiv:1804.03599. [74] L. v. d. Maaten, G. Hinton, Visualizing data using t-sne, 1470 Journal of machine learning research 9 (Nov) (2008) 2579–2605. [75] C. Chen, J. Kwon, P. Varaiya, The quality of loop data and the health of californias freeway loop detectors, PeMS Development Group. 1475 [76] J. Van Lint, S. Hoogendoorn, H. J. van Zuylen, Ac- curate freeway travel time prediction with state-space neural networks under missing data, Transportation Research Part C: Emerging Technologies 13 (5-6) (2005) 347–369. 1480 [77] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980. 21